Add feature to make quicklisp tree available to ASDF without quicklisp being loaded

[eugeneia]

We discussed this briefly on #lisp, together with a hint by quicklisp-bootstrap this would allow users to use Quicklisp-installed software through ASDF without having loaded Quicklisp. 29cb626 is so that ASDF won't find any old systems.

Consider this a request for comments, as I am unequipped to evaluate the possible impact of these changes.

[quicklisp]

Dist cleaning can't happen automatically until the LispWorks issue in #131 is resolved.

[quicklisp]

Quicklisp blocks some .asd files from some projects from being visible to ASDF, for various reasons.

Also, although multiple dists are not common, Quicklisp has a system to decide which systems have priority when multiple systems are present with the same name.

An asdf :tree option is too coarse to cover these cases.

[quicklisp]

So, if you want to do this and be faithful to what you'd get from within Quicklisp, I think you'd have to dump an index file that iterates through the local-project-directories indexes, iterates through (ql-dist:provided-systems t), writes out the data to a master index, and then includes code that can lookup systems based on that index. It would possibly be nice to do that within a single file.

I'm not very familiar with the ASDF registry configuration language. If it has a way to address individual system files and not "any file within this directory or tree", you could perhaps write a config file directly, too.

[eugeneia]

Hmm this seems to be a tough nut...

@fare sorry to bother you, but a comment on this issue from the ASDF perspective would be welcome.

[fare]

One solution that avoids the cost of scanning while allowing Quicklisp to blacklist .asd files that shouldn't be exposed (e.g. from systems that include old versions of dependencies) is to have Quicklisp generate a .cl-source-registry.cache listing exactly the files it wants to expose, then add the tree to the source-registry.

Limitation: the cl-source-registry.cache is only supported since ASDF release 3.1.4.

[eugeneia]

@xach I propose the following:

• add a function that generates dists/cl-source-registry.cache based on (ql-dist:provided-systems t)
• after installation, offer the user to generate an ASDF config that adds dists/ as a tree (and local-projects/, I am thinking it would be OK to expect users to configure their non-standard ql:*local-project-directories* themselves), and generate the initial cache
• make sure the user is only offered the above if ASDF satisfies the version requirement (3.1.4)
• after each ql:update-dist, probe for the cache, and if it exists regenerate it to reflect the changes

Does that sound about right?

[fare]

Alternatively, Quicklisp could ship asdf 3.1.4 (or 3.1.7, or the latest) instead of the antique and useless asdf 2.26. Thus, it could rely on the cl-source-registry.cache and/or other modern ASDF features.

Goodbye 2012, hello 2017!

[quicklisp]

Where is the syntax of the cache file documented?

[fare]

In the ASDF manual, section "Caching Results"
https://common-lisp.net/project/asdf/asdf/Caching-Results.html#Caching-Results

[eugeneia]

Happy new year! :-)

@xach Just to clarify, I volunteer to make the proposed changes, I just want to make sure I am not missing something fundamental.

[fare]

Note that you may have to update the cache every time Quicklisp installs a new package.

[eugeneia]

@fare Is that so? What is the difference from ASDF's perspective if a system is not "registered" as opposed to its asd file not being present? I figured we cache the dists as they are and just let ASDF run into a missing file if a system is not installed, we would/should get the generic "system not found" error?

I.e. the cache would contain paths to asd files that don't necessarily exist (but will once the system in question is installed).

[fare]

Problem is, if you register a file that doesn't exist, it may not be worse than the system being absent, but it may shadow other entries somewhere else in your source registry.

[fare]

That's not a big issue if you're at the very end of the source registry, but can be quite confusing if you aren't.

[quicklisp]

@eugeneia I am open to a function that generates a cache file. You could call it something like (ql:configure-asdf-system-cache).

For local-projects, it would be good if the ASDF configuration matched the local-projects search algorithm, where shorter paths take precedence over longer paths, if two identically named system files are in the tree.

I don't want such caches generated automatically after updates or installs, though. I wouldn't be opposed to a hook mechanism that could be used to auto-generate, but that's another can of worms.

[quicklisp]

Also, installed-systems rather than provided-systems could be a better starting point. And it would also help to take the numeric preference mechanism into account - not sure if installed-systems already does that or not, I'd have to review the code.

[fare]

I can make ASDF 3.3 use the same algorithm as local-projects, but it's a bit too late for ASDF 3.2.

How do you break conflicts between paths of the same length?

[quicklisp]

string<.

[eugeneia]

I don't see a problem with shadowing related using provided-systems for the cache, neither do I see the direct issue with local-projects precedence. From my perspective, duplicate systems are an issue unsolved by a particular precedence algorithm.

• Quicklisp prefers local-projects over provided-systems, this can be mirrored in an ASDF configs
• for shadowed systems I think the situation is more deterministic than say with only caching installed-systems, and thus better
• duplicates in local-projects is really a user issue, its great to have deterministic behavior (bonus: shared between QL and ASDF), but I doubt anyone will knowingly resolve name conflicts by adjusting the path length

I am thinking maybe each dist should just include a static cl-source-registry.cache off the bat, including all (eventually) provided-systems.

[quicklisp]

I like the idea of a static ASDF cache file, but that does mean some variance from Quicklisp's preference system.

The idea behind the preference system is that if you have two dists with the same system, you can choose which system is loaded by (ql:quickload "system") with some granularity. If dists A and B both provide systems X, Y, and Z, you can specify a preference for X from A, Y from B, and Z from A. Or if that doesn't work for your needs, you can change the preference in a persistent way.

This is not much of an issue today, because there is in effect only one popular dist, but in the future, it may be that more dists are available and resolving precedence is more important.

That preference would be busted by just including a static ASDF system cache file.

[eugeneia]

How are these preferences specified/expressed?

[quicklisp]

This is in the context of ql-dist.

Each dist, release, and system object has an associated preference value, an integer. It is available via (preference object) and set with (setf (preference object) 42).

When searching for a system or release, if more than one of the same name is found across all dists, the one with the highest preference number is returned first.

If a system does not have a specified preference value, the preference value for its release is used. If the release does not have a specified value, the value for its dist is used.

This allows preference to be established at three different levels of granularity.

The idea being that in a world with many dists available, you could say something like "Get all cffi systems from dist X, but everything else from dist Y" in a concise way.

The initial preference value of a dist the universal-time of its installation. That means more recently installed dists have a higher preference. I was planning to make it easy to specify relative preferences (e.g. (prefer object :more-than other-object) or something), because who wants to mess around with integers as big as universal-times?

But anyway, that system affects what Quicklisp returns when there is overlap.

[fare]

So the cache file (or equivalently, a symlink farm) should be generated after the preferences are modified?

[quicklisp]

Whenever a preference changes, the cache is potentially invalid. (It is only actually invalid if there are duplicate names, and the relative preference of same-named objects are switched.)

[fare]

I am not actively handling ASDF anymore. Paging @rpgoldman for the ASDF discussion.

[rpgoldman]

I'm open to helping with this, but it sounds like it's been dormant for three years.

[eugeneia]

I have since went another route and currently don’t have a use case for this. Also not sure this is the way to go. 🤷‍♂️️

Speaking for myself, feel free to direct resources elsewhere! :-)

[Hexstream]

Ah, vendoring Quicklisp. Hilarious!

…

On Thu, Jan 9, 2020, 9:48 AM Max Rottenkolber ***@***.***> wrote: I have since went another route <https://mr.gy/blog/lisp-vendoring-quicklisp-nix.html> and currently don’t have a use case for this. Also not sure this is the way to go. 🤷‍♂️ ️ — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#137?email_source=notifications&email_token=AAADFSAW6XYJHNVLIHJZSGDQ442NXA5CNFSM4CZQMZV2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEIQRRVI#issuecomment-572594389>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAADFSCNFBXYUPS2PQ6CTPDQ442NXANCNFSM4CZQMZVQ> .