Skip to content

Conversation

@sgaraev
Copy link
Contributor

@sgaraev sgaraev commented Oct 9, 2025

When upgrading to a new version, if there are two or more extensions enabled, all extensions are deleted due to incorrect verification logic. Only the last downloaded extension remains. This makes it impossible to boot the system after a reboot in closed environments due to lack of access to https://update.release.flatcar-linux.net

How to use

Testing done

An image was created in the flatcar-4426 branch, which was added with extensions to the local nginx file server. The package was created on the local nebraska server.
With extensions enabled:

# butane.yaml:
variant: flatcar
version: 1.0.0
storage:
  files:
    - path: /etc/flatcar/enabled-sysext.conf
      mode: 0644
      contents:
        inline: |
          python
          zfs
    - path: /etc/flatcar/update.conf
      overwrite: true
      mode: 0644
      contents:
        inline: |
          MACHINE_ALIAS="test-vm"
          GROUP=beta
          SERVER=http://172.18.0.5/v1/update/

# boot from https://bincache.flatcar-linux.net/images/amd64/4230.2.3+nightly-20251008-2100/flatcar_production_qemu_image.img 
with internet enabled


localhost ~ # ls -lah /etc/flatcar/sysext/
total 25M
drwxr-xr-x. 2 root root 4.0K Oct  9 18:17 .
drwxr-xr-x. 1 root root 4.0K Oct  9 18:17 ..
-rw-r--r--. 1 root root  20M Oct  9 18:17 flatcar-python-4230.2.3+nightly-20251008-2100.raw
-rw-r--r--. 1 root root 4.1M Oct  9 18:17 flatcar-zfs-4230.2.3+nightly-20251008-2100.raw

# Update
sudo update_engine_client -update

localhost ~ # ls -lah /etc/flatcar/sysext/
total 49M
drwxr-xr-x. 2 root root 4.0K Oct  9 18:24 .
drwxr-xr-x. 1 root root 4.0K Oct  9 18:24 ..
-rw-r--r--. 1 root root  20M Oct  9 18:17 flatcar-python-4230.2.3+nightly-20251008-2100.raw
-rw-r--r--. 1 root root  21M Oct  9 18:24 flatcar-python-4426.1.0+nightly-20251007-2100.raw
-rw-r--r--. 1 root root 4.1M Oct  9 18:17 flatcar-zfs-4230.2.3+nightly-20251008-2100.raw
-rw-r--r--. 1 root root 4.0M Oct  9 18:24 flatcar-zfs-4426.1.0+nightly-20251007-2100.raw

Without extensions enabled:

# butane.yaml:
variant: flatcar
version: 1.0.0
storage:
  files:
    - path: /etc/flatcar/update.conf
      overwrite: true
      mode: 0644
      contents:
        inline: |
          MACHINE_ALIAS="test-vm"
          GROUP=beta
          SERVER=http://172.18.0.5/v1/update/

# boot from https://bincache.flatcar-linux.net/images/amd64/4230.2.3+nightly-20251008-2100/flatcar_production_qemu_image.img

Flatcar Container Linux by Kinvolk beta 4230.2.3+nightly-20251008-2100 for QEMU
core@localhost ~ $ ls -lah /usr/share/flatcar/enabled-sysext.conf
ls: cannot access '/usr/share/flatcar/enabled-sysext.conf': No such file or directory
core@localhost ~ $ ls -lah /etc/flatcar/enabled-sysext.conf
ls: cannot access '/etc/flatcar/enabled-sysext.conf': No such file or directory

core@localhost ~ $ sudo update_engine_client -update
System reboot in 5 minutes!                                                    
  
LAST_CHECKED_TIME=1760035063
PROGRESS=0.000000
CURRENT_OP=UPDATE_STATUS_UPDATED_NEED_REBOOT
NEW_VERSION=4426.1.0+nightly-20251007-2100
NEW_SIZE=467964215
I20251009 18:38:09.153322  1718 update_engine_client.cc:198] Update succeeded -- reboot needed.

core@localhost ~ $ sudo reboot

core@localhost ~ $ cat /etc/os-release 
NAME="Flatcar Container Linux by Kinvolk"
ID=flatcar
ID_LIKE=coreos
VERSION=4426.1.0+nightly-20251007-2100
VERSION_ID=4426.1.0
BUILD_ID=nightly-20251007-2100
SYSEXT_LEVEL=1.0
PRETTY_NAME="Flatcar Container Linux by Kinvolk 4426.1.0+nightly-20251007-2100 (Oklo)"
ANSI_COLOR="38;5;75"
HOME_URL="https://flatcar.org/"
BUG_REPORT_URL="https://issues.flatcar.org"
FLATCAR_BOARD="amd64-usr"
CPE_NAME="cpe:2.3:o:flatcar-linux:flatcar_linux:4426.1.0+nightly-20251007-2100:*:*:*:*:*:*:*"
  • Changelog entries added in the respective changelog/ directory (user-facing change, bug fix, security fix, update)
  • Inspected CI output for image differences: /boot and /usr size, packages, list files for any missing binaries, kernel modules, config files, kernel modules, etc.

@sgaraev sgaraev requested a review from a team as a code owner October 9, 2025 19:18
When upgrading to a new version, if there are two or more extensions
enabled, all extensions are deleted due to incorrect verification logic.
Only the last downloaded extension remains. This makes it impossible
to boot the system after a reboot in closed environments due to lack of
access to https://update.release.flatcar-linux.net

Signed-off-by: Sergey Garaev <[email protected]>
Copy link
Member

@pothos pothos left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks a lot!
Yes, that logic was copy pasted from the OEM case where there is only one but of course it won't work in the loop…

@pothos pothos merged commit 85eea2c into flatcar:main Oct 10, 2025
1 check passed
pothos added a commit to flatcar/scripts that referenced this pull request Oct 10, 2025
@pothos
Copy link
Member

pothos commented Oct 10, 2025

I plan to backport this, and hope that not too many things have changed 🤞 since Stable

pothos added a commit to flatcar/scripts that referenced this pull request Oct 13, 2025
pothos added a commit to flatcar/scripts that referenced this pull request Oct 14, 2025
pothos added a commit to flatcar/scripts that referenced this pull request Oct 14, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants