Source files aren't listed in dist-info/RECORD for editable installs.

[fofoni]

OS version

Ubuntu 20.04

Python version

3.7.13

Pip version

22.2.2

Guide link

https://packaging.python.org/tutorials/packaging-projects

Problem description

I configured my project exactly like the tutorial, but then before building a wheel and uploading, I want to develop locally with an editable install. I run pip install -e . and then a .dist-info directory and a .pth file are created inside my virtualenv's site-packages. Inside the .dist-info directory, the RECORD file lists only metadata files (readme, license, etc) but no source files. I tested, and if I make a regular install instead (pip install .) then the RECORD file will correctly list all .py files. Also, with the editable install, the package can be imported and works as expected. I'm using hatchling as the build backend, but I tested with flit and the behavior is the same.

Is the lack of source files in the RECORD expected for an editable install?

Context: as far as I understand, the lack of .py files in dist-info/RECORD is what is preveting importlib_metadata.packages_distributions() from listing my package. I have some other local packages installed with pip install -e which use setuptools/setup.py, and these appear in the site-packages as .egg-link files. These are correctly listed by importlib_metadata.packages_distributions(). Is this a bug in importlib_metadata?

[pfmoore]

RECORD should only hold files that are installed into site-packages, (because those are what uninstalling should remove) so this seems correct to me.

If importlib_metadata.packages_distributions is expecting to find the linked files from the editable install in there, that's an issue with importlib_metadata (either it's a bug that can be fixed, or the packages_distributions function doesn't support editable installs) which should be raised with that project.

[fofoni]

That's the clarification I wanted, thank you!

[merwok]

(RECORD should only contain installed files, which means modules in site-packages + scripts, headers, data files)

[pfmoore]

@merwok Exactly, but in an editable install, the modules are located in the source tree, and only a pointer to those files is installed into the site-packages (that pointer may be a .pth file, or a redirector that gets added to sys.meta_path, or something else).

The key here is that RECORD must not include the files in the source tree, or otherwise pip uninstall will remove them...

I suspect, though, that importlib.metadata doesn't expect a layout like that for some reason.

[layday]

importlib.metadata looks for Python files listed in the RECORD to map packages to their distributions since there's no other standards-based way to get that information.

[merwok]

Ah, I missed the key word of editable installs here! I understand now why you mentioned files in site-packages only. Thanks for your patience 🙂

[pfmoore]

@layday I suspected that was the case. In which case it probably needs to document the limitation somehow, in order to help people hitting this sort of issue.

The implementation appears to look for a top_level.txt file (not standardised, setuptools-specific) or infer the value from what's in RECORD. But I imagine that documenting what are meant to be internal details is not ideal, so maybe just "the approach used may not work for some editable installs" is enough.

[layday]

python/importlib_metadata#402