There's an openQA test run on updates which does the following: * Set up a side repo with the update in it * Update the SUT from that repo, reboot * Create mock config which also uses that repo, and a live kickstart which also uses that repo * Install some packages into the mock chroot and then copy the kickstart into it (not using --old-chroot or --enable-networking) * Try to create a live image in the mock chroot, this time using --old-chroot and --enable-networking, using that mock config and live kickstart this process is designed to replicate as close as possible what Koji does when creating official live images. When run on https://bodhi.fedoraproject.org/updates/FEDORA-2020-8c3c5881b2 - systemd-246.4-2.fc33 - that test fails at the point where we try to run the livemedia-creator command in the mock chroot, with this error: ... File "/usr/lib/python3.9/site-packages/mockbuild/buildroot.py", line 286, in _copy_config shutil.copy2(orig_conf_file, etcdir) File "/usr/lib64/python3.9/shutil.py", line 432, in copy2 copyfile(src, dst, follow_symlinks=follow_symlinks) File "/usr/lib64/python3.9/shutil.py", line 261, in copyfile with open(src, 'rb') as fsrc, open(dst, 'wb') as fdst: FileNotFoundError: [Errno 2] No such file or directory: '/var/lib/mock/fedora-33-x86_64/root/etc/resolv.conf' The same test run on earlier and later updates does not fail, suggesting it is this update specifically that causes the problem. The change in the update is this: https://src.fedoraproject.org/rpms/systemd/c/b98737cec484f1c8dcb839e97192809b99f77c89?branch=f33 AFAIK, that's making systemd %post try harder (on initial install, not on update) to create /etc/resolv.conf as a symlink to /run/systemd/resolve/stub-resolv.conf . Which I guess may not exist in a mock environment, which may be the issue here. Here's the failure: https://openqa.fedoraproject.org/tests/664882
This also appears to have broken today's Rawhide compose: https://pagure.io/releng/failed-composes/issue/1761
Thank you for the report, Adam. What SUT stands for? > https://src.fedoraproject.org/rpms/systemd/c/b98737cec484f1c8dcb839e97192809b99f77c89?branch=f33 Great analysis, that indeed triggered the problems. This all is caused by (to me) unexpected shutil.copy*() behavior: $ mkdir /tmp/a $ ln -s non-existing-file /tmp/a/resolv.conf $ mkdir /tmp/b $ echo content > /tmp/b/resolv.conf $ ipyton3 In [1]: import shutil In [2]: shutil.copy2("/tmp/b/resolv.conf", "/tmp/a") Out[2]: '/tmp/a/resolv.conf' In [3]: Do you really want to exit ([y]/n)? y $ cat /tmp/a/non-existing-file content IOW, shutil.copy() follows symlinks in dst=, and creates the non-existing file. Seems like we have to unlink the destination file first.
Heh, shutil.copy2() even has a follow_symlinks= argument, but it doesn't work! With follow_symlinks=False, the symlink in /tmp/a is still followed. _copy_config() already removes the destination path, but it uses os.path.exists() which returns false for broken symlinks. Maybe use os.access() instead?
There's e.g. https://bugs.python.org/issue34374 - and per the report, /bin/cp indeed behaves the same, except for the "dangling" symlink situation: $ cp b/resolv.conf a/resolv.conf cp: not writing through dangling symlink 'a/resolv.conf' I reported that.
> _copy_config() already removes the destination path, but it uses os.path.exists() > which returns false for broken symlinks. Maybe use os.access() instead? That sounds racy (as well as the current behavior). Is it any better to just try to remove the destination file, and ignore FileNotFoundError?
https://github.com/rpm-software-management/mock/pull/631
So, I guess we need a new mock build for Rawhide at least. If we are still planning to have the change in F33, we need a new mock build for F33 as well and it should be added to the systemd update, I think.
FEDORA-2020-f7ce2b3a71 has been submitted as an update to Fedora 32. https://bodhi.fedoraproject.org/updates/FEDORA-2020-f7ce2b3a71
FEDORA-2020-8b8b3ade7b has been submitted as an update to Fedora 33. https://bodhi.fedoraproject.org/updates/FEDORA-2020-8b8b3ade7b
FEDORA-EPEL-2020-07a27d3e56 has been submitted as an update to Fedora EPEL 8. https://bodhi.fedoraproject.org/updates/FEDORA-EPEL-2020-07a27d3e56
FEDORA-EPEL-2020-0996fb7a3c has been submitted as an update to Fedora EPEL 7. https://bodhi.fedoraproject.org/updates/FEDORA-EPEL-2020-0996fb7a3c
FEDORA-2020-f7ce2b3a71 has been pushed to the Fedora 32 testing repository. In short time you'll be able to install the update with the following command: `sudo dnf upgrade --enablerepo=updates-testing --advisory=FEDORA-2020-f7ce2b3a71` You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2020-f7ce2b3a71 See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.
FEDORA-EPEL-2020-0996fb7a3c has been pushed to the Fedora EPEL 7 testing repository. You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-EPEL-2020-0996fb7a3c See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.
FEDORA-EPEL-2020-07a27d3e56 has been pushed to the Fedora EPEL 8 testing repository. You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-EPEL-2020-07a27d3e56 See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.
FEDORA-2020-ef6074cb37 has been pushed to the Fedora 31 testing repository. In short time you'll be able to install the update with the following command: `sudo dnf upgrade --enablerepo=updates-testing --advisory=FEDORA-2020-ef6074cb37` You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2020-ef6074cb37 See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.
FEDORA-2020-8b8b3ade7b has been pushed to the Fedora 33 testing repository. In short time you'll be able to install the update with the following command: `sudo dnf upgrade --enablerepo=updates-testing --advisory=FEDORA-2020-8b8b3ade7b` You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2020-8b8b3ade7b See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.
FEDORA-2020-f7ce2b3a71 has been pushed to the Fedora 32 stable repository. If problem still persists, please make note of it in this bug report.
FEDORA-2020-ef6074cb37 has been pushed to the Fedora 31 stable repository. If problem still persists, please make note of it in this bug report.
FEDORA-2020-8b8b3ade7b has been pushed to the Fedora 33 stable repository. If problem still persists, please make note of it in this bug report.
FEDORA-EPEL-2020-0996fb7a3c has been pushed to the Fedora EPEL 7 stable repository. If problem still persists, please make note of it in this bug report.
FEDORA-EPEL-2020-07a27d3e56 has been pushed to the Fedora EPEL 8 stable repository. If problem still persists, please make note of it in this bug report.