ADMIN Network & Security

Focus
Guide

AlmaLinux
A New Path
ABI compatibility with RHEL

ELevate from CentOS 7

Easing the migration process

The AlmaLinux Build System

Automating package development

An HPC Solution
3 real-world use cases

Past, Present, and Future

An interview with the AlmaLinux team
US$ 7.95

www . admin - magazine . com

 Table of Contents Fo c u s o n A l m aL i n u x

ADMIN Network & Security

AlmaLinux
Contact Info
Dear Readers:
Editor in Chief
Joe Casad, [email protected] If you are looking for a free, open source replacement for
Associate Editor
Amy Pettle
the now discontinued CentOS Linux, AlmaLinux OS has you
Copy Editor covered. Offering binary compatibility with Red Hat Enterprise
Aubrey Vaughn Linux, AlmaLinux OS provides a community-driven enterprise
Layout and Graphic Design
Dena Friesen, Lori White Linux solution that features the long-term stability required
Advertising for production-grade environments. In this focus guide, we
Jessica Pryor, [email protected]
introduce you to AlmaLinux OS and its role in the enterprise
Marketing Communications
Gwen Clark, [email protected] Linux ecosystem. We'll show you how to get started with this
Publisher forever-free distribution.
Brian Osborn
Customer Service / Subscription
Email: [email protected] Table of Contents
Phone: 1-866-247-2802
(toll-free from the US and Canada)
4 Introduction
www.admin-magazine.com
AlmaLinux OS places an emphasis on its community to deliver a no-cost
While every care has been taken with the
content of the m
­ agazine, the publishers Red Hat Enterprise Linux alternative.
cannot be held responsible for the accuracy
of the information contained within it or any 6 RHEL Compatibility
­consequences arising from the use of it. Recent policy changes at Red Hat have upturned the RHEL clone community.
Copyright and Trademarks © 2024 Linux AlmaLinux charts a new path by shifting to binary compatibility and away from
New Media USA, LLC
being a downstream RHEL build.
Cover Illustration © tatkhagata, 123RF.com
No material may be reproduced in any form
9 HPC Use Cases
whatsoever in whole or in part without the AlmaLinux steps forward as a free, open source enterprise Linux solution for
written permission of the p­ ublishers. It is HPC systems.
assumed that all correspondence sent, for
example, letters, email, faxes, photographs,
articles, drawings, are s­ upplied for
12 AlmaLinux Build System
publication or license to third parties on a The AlmaLinux Build System lets you build, test, sign, and release packages
non-exclusive worldwide basis by Linux New from a single interface.
Media unless otherwise stated in writing.
All brand or product names are trademarks 16 AlmaLinux’s ELevate Migration Tool
of their respective owners. Contact us if we CentOS 7 reached end of life in June 2024, forcing users to look for a free
haven’t credited your copyright; we will
always correct any oversight. enterprise Linux alternative. AlmaLinux’s ELevate migration tool can help
Published by Linux New Media USA, LLC,
ease the transition.
4840 Bob Billings Parkway, Ste 104, Law-
rence, KS 66049, USA 20 An Interview with AlmaLinux OS
Represented in Europe and other territories AlmaLinux team members discuss their quest to deliver a forever-free,
by: Sparkhaus Media GmbH, Bialasstr. 1a, community-driven enterprise Linux distribution.
85625 Glonn, Germany.

w w w. a d m i n - m aga z i n e .co m Fo cu s o n A l m aLi n ux 3

FO C U S O N A L M A L I N U X Introduction

A community-owned enterprise Linux solution

Introducing AlmaLinux
AlmaLinux OS places an emphasis on its community to deliver a no-cost Red Hat Enterprise Linux alternative. By Amy Pettle
Alma means “soul” in many Latin CentOS carried on as a free down- from profit motivation and answerable
languages, making it a fitting name for stream distribution of RHEL until De- to both its users and those impacted
a distribution that places so much em- cember 2020 when Red Hat announced by its decisions. To achieve this, Alma-
phasis on Linux’s soul: its community. that CentOS would be replaced with Linux needed to be completely owned
Jack Aboutboul, AlmaLinux Commu- CentOS Stream, an upstream develop- and governed by its members. By giving
nity Manager, says the name Alma- ment version of RHEL with half the users a voice, AlmaLinux could remedy
Linux “ties into our core belief that lifespan. CentOS 8, the current version something they felt was missing in the
AlmaLinux’s community of individuals at the time, would have its support life original CentOS community, as well as
and organizations are the ‘soul’ that cycle cut short to December 31, 2021, avoid the problems with CentOS’s origi-
powers and drives us forward” [1]. As instead of the planned 2029 end of life nal organizational structure that made it
a community owned and governed en- (EOL), and CentOS 7 reached EOL on possible for it to be acquired by Red Hat
terprise Linux distribution, AlmaLinux schedule in June 2024. in the first place. Bringing AlmaLinux
offers a long-term, stable, production- With thousands of servers worldwide users and contributors into governance
grade environment for users looking across a wide range of industries run- and key decision-making ultimately
for a free, binary-compatible alternative ning CentOS 7 and 8, the Red Hat would make the distro more sustainable
to Red Hat Enterprise Linux (RHEL). announcement had a huge impact. For in the long term.
In this focus guide, we’ll introduce you users needing a stable enterprise Linux Consequently, shortly after the first
to AlmaLinux OS and its role in the ever- solution, CentOS Stream wasn’t a via- stable release of AlmaLinux OS, Cloud-
evolving enterprise Linux space. You’ll ble replacement due to its development Linux transferred development and gov-
learn about AlmaLinux’s compatibility status and short lifespan. Addition- ernance to the AlmaLinux OS Founda-
with RHEL and how it has adapted in ally, removing a free alternative such tion, a nonprofit with 501(c)6 status. To
its short lifetime to RHEL source code as CentOS essentially functioned as a guarantee support of the project, Cloud-
restrictions. We interview three organi- tax on innovation because only larger Linux also pledged $1 million in annual
zations in the high-performance com- companies can typically afford a sub- funding. Keeping his initial promise
puting (HPC) world, providing insights scription-based enterprise Linux. With of a truly community-owned project,
on how AlmaLinux meets the needs of CentOS Linux EOL looming, enterprise Seletskiy stepped down from the Alma-
HPC users. We also cover two custom Linux users needed a free alternative, Linux OS Foundation board in October
AlmaLinux tools that can also be used and the Linux community stepped up 2021, being replaced by open source
by the broader enterprise Linux com- to deliver a solution. community leader benny Vasquez.
munity: the ELevate migration tool and The AlmaLinux OS Foundation
the AlmaLinux Build System. Finally, Community Driven membership includes contributors,
we talk to AlmaLinux team members users, mirror providers, and sponsors.
to learn more about AlmaLinux’s AlmaLinux was one of these solutions. Individual memberships are free and
journey so far as well as its future. Igor Seletskiy, CEO at CloudLinux require minimal effort to engage. In a
Before getting into all of that, we’ll (which maintains its own commercial talk at AlmaLinux Day: Tokyo in 2023,
give you a brief history of AlmaLinux, Linux distro), set out to collaborate Vasquez, the current board chair, noted
facts about the project, and how the with the Linux community to create that “Anyone who has an interest in
“soul” of AlmaLinux influences ev- AlmaLinux as a successor to CentOS the future of AlmaLinux should have a
erything it does. promising 1:1 RHEL compatibility. voice in the future of AlmaLinux.” The
In early February 2021, AlmaLinux foundation has made a specific effort to
In the Beginning announced a beta release, followed make the AlmaLinux community a wel-
shortly thereafter by the first stable coming space for the CentOS commu-
Before AlmaLinux OS, CentOS was release at the end of March 2021. By nity by setting up ground rules to make
the free enterprise Linux alternative of the end of the first year, AlmaLinux it a safe space to contribute. They also
Lead Image © Author, 123RF.com

choice. The CentOS project started in had delivered three releases. welcome all levels of users – the com-
2004 as a downstream version of RHEL What sets AlmaLinux apart from the munity is not just for advanced users.
in response to Red Hat turning the once other free RHEL-compatible distros is Currently, the AlmaLinux OS Foun-
free RHEL into a commercial product. its focus on community governance. dation board consists of nine board
In 2014, Red Hat acquired CentOS by When considering a CentOS alternative, members. To prevent AlmaLinux being
hiring several of its developers. AlmaLinux looked to create a distro free taken over by a single organization, no

4 FOCUS ON ALMALINUX W W W. A D M I N - M AGA Z I N E .CO M

 Introduction FO C U S O N A L M A L I N U X

two board members can be from the The Future Is Bright RHEL as possible, any bugs found in
same organization. The board also ac- AlmaLinux should also be in RHEL.
tively seeks to include a greater variety Users looking for a free enterprise- If the bug is in CentOS Stream, mem-
of voices driving the future of Alma­ grade Linux distro have found a home bers are encouraged to report the bug
Linux. While the initial board mem- in AlmaLinux. In the past year, Alma- to CentOS Stream Bugzilla and col-
bers were appointed to set up a mem- Linux adoption has seen exponential laborate with CentOS Stream where
bership structure, elections are now growth [2]. Likewise, membership possible. At the end of the day, this
held regularly. To be eligible for the in the AlmaLinux OS Foundation has approach improves the enterprise
board, you must be nominated by a grown to 350+ members, with over Linux ecosystem for everybody.
member and be a member of the foun- 25 corporate sponsors.
dation yourself for three months with You can keep up with what is hap- Getting Started
good standing. Whenever AlmaLinux pening at AlmaLinux by joining the
acquires a new Platinum sponsor, the AlmaLinux OS Foundation Chat [3] AlmaLinux OS 9.4 is the current version
foundation’s bylaws allow the board to or signing up for AlmaLinux’s mailing at the time of writing, but Alma­Linux
also add two new board members. lists [4], covering a range of topics OS 8.10 is supported through 2029 for
such as release and security announce- those who need RHEL 8 compatibility.
Shake Up ments, community discussion and To download AlmaLinux for free, visit
help, development discussion, and the AlmaLinux website [8]. Other
Things hummed along until June 2023 mirror assistance for mirror users and download options include an ISO from
when Red Hat announced that RHEL- owners. You can also follow along on AlmaLinux mirror sites [9], a .torrent
related source code would be restricted AlmaLinux Forums [5], Fosstodon, file, and CHECKSUM options. AlmaL-
to subscribers, who were prohibited reddit, GitHub, and Docker Hub. With inux supports four architectures: Intel/​
from redistributing the code due to an AlmaLinux Account [6], you can AMD (x86_64), ARM64 (aarch64), IBM
RHEL licensing agreements. Moving create a single account to access an ar- PowerPC (ppc64le), and IBM Z (s390x),
forward, CentOS Stream would be the ray of AlmaLinux sites and services. along with images for cloud, containers,
only source code repository for RHEL For those looking to get more involved, Live media, Vagrant boxes, LXC/​LXD,
code. In terms of 1:1 compatibility, there are several Special Interest Group Raspberry Pi, and WSL.
this posed problems for AlmaLinux be- (SIGs) [7] that bring individuals and Now that you’ve learned a little bit
cause CentOS Stream is not production organizations with common interests about AlmaLinux’s origin story, you are
grade and includes updates not yet together to work on specific projects. ready to get started with AlmaLinux OS.
included in RHEL. AlmaLinux is also looking for people to Turn the page and read on to learn what
After consulting with their community, translate their website as well as help this community-owned, forever free
AlmaLinux chose what they like to call with testing, industry events, and an- enterprise Linux distro can do for you! n
the No Drama route. They responded swering community questions.
by moving away from 1:1 binary com- Info
patibility, which potentially poses legal The Greater Good [1] “Looking Back, Leaping Forward: A look back
issues, to application binary interface at the first year for AlmaLinux and what the
(ABI) compatibility. Instead of build- Keeping with their community focus, future holds” by Jack Aboutboul, AlmaLinux
ing downstream of RHEL, they are AlmaLinux also contributes to the blog, February 2, 2022, [https://​­almalinux.​
now building alongside RHEL. With greater enterprise Linux community. ­org/​­blog/​­looking‑​­back‑​­leaping‑forward‑a‑​
ABI compatibility, AlmaLinux guaran- Aboutboul says, “We believe that giv- ­look‑​­back‑​­at‑​­the‑first‑year‑for‑​­almalinux‑​
tees that if something works in RHEL ing back to the ecosystem is vital for ­and‑​­what‑the‑future‑holds/]
it will work in AlmaLinux – if not, it’s the health of open source software” [1]. [2] “A Look Back at 2023 for AlmaLinux” by
a bug that needs to be fixed. AlmaLinux has consciously created benny Vasquez, AlmaLinux blog, December 13,
Getting the source code has become AlmaLinux tools that can be used by 2023, https://almalinux.org/blog/2023-12-
a little more complicated. The new the community at large. AlmaLinux’s 14-2023-highlights/
release cycle initially lagged RHEL ELevate not only helps AlmaLinux us- [3] AlmaLinux OS Foundation Chat:
by about a week, though current Al- ers migrating from CentOS, but it also [chat.almalinux.org]
maLinux releases are being updated works for other RHEL derivatives, in- [4] Mailing lists: [https://​­lists.​­almalinux.​­org]
within days. No longer being bound cluding Rocky Linux and Oracle Linux. [5] Forums: [https://​­almalinux.​­discourse.​­group]
by 1:1 compatibility with RHEL also The AlmaLinux Build System was de- [6] AlmaLinux Accounts:
has its benefits. With ABI compat- veloped to create, sign, test, and release [https://​­accounts.​­almalinux.​­org]
ibility, AlmaLinux can now address versions of AlmaLinux, but the tool can [7] SIGs: [https://​­wiki.​­almalinux.​­org/​­sigs/]
security and bug fixes (e.g., Zenbleed, also be used for other projects. [8] Download AlmaLinux:
Downfall, Inception, regreSSHion, When it comes to bugs, AlmaLinux [https://​­almalinux.​­org/​­get‑almalinux/]
etc.) more quickly than RHEL and takes an upstream approach. Because [9] AlmaLinux mirrors: [https://​­mirrors.​
other RHEL clones. AlmaLinux aims to be as close to ­almalinux.​­org/​­isos/​­x86_64/​­9.​­3.​­html]

W W W. A D M I N - M AGA Z I N E .CO M FO CU S O N A L M A L I N UX 5
 FO C U S O N A L M A L I N U X RHEL Compatibility

AlmaLinux promises continued RHEL compatibility

Friendly Fork
Recent policy changes at Red Hat have upturned the RHEL clone community. enterprise Linux solution. (See the
“New Path Forward” box for our
AlmaLinux charts a new path by shifting to binary compatibility and away interview with benny Vasquez,
from being a downstream RHEL build. By Amy Pettle AlmaLinux OS Foundation Chair, to
learn why they chose this route.)

When Red Hat discontinued CentOS version of RHEL that contains ex- 1:1 vs. ABI Compatibility
and replaced it with CentOS Stream perimental packages, would now
in late 2020, AlmaLinux stepped be the sole repository for public In 1:1 compatibility, a clone dis-
forward to build a community RHEL-related source code releases. tribution provides an exact copy
downstream version of Red Hat Because Red Hat’s subscription of RHEL’s functionality, behavior,
Enterprise Linux (RHEL). In a desire agreement prohibits customers from and binary compatibility, including
to fill this void in the enterprise redistributing code, this move ap- bug-to-bug compatibility. It is an
Linux ecosystem, CloudLinux col- peared to put an end to downstream exact replica of RHEL minus RHEL’s
laborated with the community to builds like AlmaLinux as well as branding and trademarks.
develop AlmaLinux OS as a down- other RHEL clones like Rocky Linux With ABI compatibility, AlmaLinux
stream build of RHEL. After the and Oracle Linux. guarantees that all apps developed
first stable release in March 2021, Some were quick to predict the for RHEL or its clones will run on
CloudLinux turned governance of demise of these RHEL clones, but AlmaLinux without any modifica-
AlmaLinux OS over to the nonprofit AlmaLinux, Rocky Linux, and oth- tions or extra work on the part of
AlmaLinux OS Foundation. From ers quickly charted a path forward. the user. AlmaLinux will not be an
there, AlmaLinux chugged along for While Rocky Linux and the newly exact copy, but it will include ker-
over two years providing the enter- formed OpenELA (founded by nel and application compatibility.
prise Linux community with a for- Oracle, SUSE, and CIQ) have prom- This also means that AlmaLinux
ever-free Linux distro while offering ised to retain 1:1 compatibility with will not guarantee bug-to-bug
Photo by Alex Kondratiev on Unsplash

long-term stability and a production RHEL, citing their rights under the compatibility. While some us-
grade platform. GPL, AlmaLinux is forging a different ers might find bugs not found in
That all changed in June 2023 path forward. RHEL, AlmaLinux also has the op-
when Red Hat announced that AlmaLinux plans to maintain ap- portunity to include bug fixes not
RHEL-related source code would plication binary interface (ABI) yet addressed by Red Hat, as well
be restricted to Red Hat’s customer compatibility to continue to provide as possibly offer new features not
portal. CentOS Stream, an upstream the community with a forever-free available in RHEL.

6 FOCUS ON ALMALINUX W W W. A D M I N - M AGA Z I N E .CO M

 RHEL Compatibility FO C U S O N A L M A L I N U X

Adjustments from these sources. To achieve ABI pose the biggest challenge because
compatibility, AlmaLinux will use AlmaLinux can no longer pull these
Prior to Red Hat moving RHEL CentOS Stream (the upstream ver- updates from Red Hat without vio-
source code behind a paywall, any sion of RHEL still available to the lating licensing agreements. Mov-
security update or bug fix in RHEL public) and then get additional code ing forward, AlmaLinux plans to
resulted in Red Hat publishing the from Red Hat Universal Base Images pull kernel updates from various
corresponding code to a public re- (UBIs) and upstream Linux code. In other sources, and, if all else fails,
pository. AlmaLinux then integrated a recent talk at All Things Open [1], the Oracle releases (which are also
this updated code into their own Vasquez noted that 99 percent of the based on RHEL).
build and test system, produced a packages would match RHEL source On the upside, AlmaLinux can now
new RPM package manager, and then code. Of this 99 percent, 75 percent include comments in their patches
published the code in the AlmaLinux will be built from CentOS Stream for greater transparency. Users will
repositories. or UBI images, while approximately see where the patch comes from,
Instead of updates and patches com- 24 percent will require manual which was not an option before.
ing from a single repository, Alma- patching. Finally, AlmaLinux now asks users
Linux now must gather them from The remaining one percent that who find bugs in AlmaLinux to at-
multiple sources and then compare, differs from RHEL lies in the ker- tempt to test and replicate the prob-
test, and build the new release nel patches. These kernel updates lem in CentOS Stream in order to let

New Path Forward

We talked to benny Vasquez, chair of the AlmaLinux OS Foundation, also announced two additional repositories. One for testing and one
about their decision to shift to ABI compatibility in the wake of the for new packages that aren’t available in our upstream or in EPEL.
changes at Red Hat. This also opens the door for other features and improvements that
we could add back in or change, as our users need. We have already
Linux Magazine (LM): What prompted AlmaLinux to choose ABI seen greater community involvement, especially around these ideas.
over 1:1 compatibility with RHEL?
LM: Does the ABI route pose any extra challenges?
benny Vasquez (bV): The short answer is our users. Overwhelm‑
ingly, our users made it clear that they chose AlmaLinux for its ease bV: The obvious one is that building from CentOS Stream sources
of use, the security and stability that it provides, and the backing of takes more effort, but I think the more important challenge (and the
a diverse group of sponsors. All of that together meant that we didn’t one that will only be solved with consistency over time) is the one of
need to lock ourselves into copying RHEL, and we could continue to proving that we will be able to deliver on the promise. With a commu‑
provide what our users needed. nity like ours, rebuilding someone else’s code doesn’t take as much
effort. Technically, building from Stream takes more time for sure,
Moreover, we needed to consider what our sponsors would be able to
but the public perception is that it will lead to greater divergence
help us provide, and how we could best serve the downstream projects
from RHEL. I think folks will be seriously happy about what they find
that now rely on AlmaLinux. The rippling effects of any decision that
as we release the new versions, namely, the consistency, stability,
we make are beyond measure at this point, so we consider all aspects
and security that they’ve come to expect from us.
of our impact and then move forward with confidence and intention.
LM: How did AlmaLinux’s mission of improving the Linux ecosystem
LM: Since you are no longer bound to conform to 1:1 compatibility,
for everyone influence this decision?
what do you see in AlmaLinux’s future?
bV: We strongly believe that the soul of open source means work‑
bV: We will continue on our goal of becoming the home for all users
ing together, providing value where there is a gap, and helping each
other solve problems. If we participate in an emotional reaction to that need enterprise Linux for free, but in the next year I expect
a business’s change, we will then be distracted and potentially hurt that we will see an expansion in the number of kernels we support
users and the enterprise Linux ecosystem overall. By remaining and see some new and exciting SIGs spun up around other features
focused on what is best (though not easiest), and adapting to the or use cases, as the community continues to standardize on how to
ecosystem as it is today, we will provide a better and more stable achieve their goals collectively.
operating system.
LM: What do you think your relationship with Red Hat will look like
LM: What opportunities does the ABI route offer over 1:1 compatibility? moving forward?

bV: By liberating ourselves from the 1:1 promise, we have been bV: Ultimately our goal is to improve the enterprise Linux ecosys‑
able to do a few small things that have proven to be a good testing tem, and we’ll welcome anyone who is actively working toward that
ground for what will come in the future. Specifically, we shipped a goal. We have loved seeing the positive infusion of energy that the
couple of smallish, but extremely important, security patches ahead AlmaLinux users have been able to build on and are excited to see
of Red Hat, offering quicker security to the users of AlmaLinux. We that continue to expand through the entire ecosystem.

W W W. A D M I N - M AGA Z I N E .CO M FO CU S O N A L M A L I N UX 7
FO C U S O N A L M A L I N U X RHEL Compatibility

developers correct the issue in the RHEL or Extra Packages for Enter- CentOS Stream, Fedora, and Linux
right place. prise Linux (EPEL, a set of extra in general.
software packages maintained by At the time of writing, AlmaLinux has
New Additions the Fedora SIG that are not avail- announced the first releases using the
able in RHEL or CentOS Stream). new build process, beta versions of
No longer bound to 1:1 compatibil- Synergy is available for AlmaLinux AlmaLinux 8.9 and 9.3, so you can
ity, AlmaLinux can set its own pri- 8 and 9 as well as all enterprise see for yourself how ABI compatibil-
orities rather than following RHEL’s Linux users (e.g., RHEL, Rocky ity works. n
lead. AlmaLinux now has the oppor- Linux, Oracle Linux, CentOS
tunity to include features that meet Stream). Once accepted to EPEL,
the needs of its community, whether these packages will be removed Info
that is fixing bugs faster (like the from Synergy. At the time of writ- [1] benny Vasquez talk at All Things Open 23:
AMD microcode exploits [2]) or ing, current packages include the [https://​­www.​­youtube.​­com/​­watch?​
adding new features. Pantheon Desktop Environment ­v=Jjda39dlu7I]
In August 2023, AlmaLinux added and the Warpinator app. Community [2] AMD microcode exploits:
two new repositories, Testing and members can request packages [https://​­www.​­amd.​­com/​­en/​­resources/​
Synergy [3]. Testing, currently via the AlmaLinux Packaging chat ­product‑security/​­bulletin/​­amd‑sb‑7005.​­html]
available for AlmaLinux 8 and 9, channel in Mattermost [4]. [3] Testing and Synergy repositories:
offers security updates before they [https://​­almalinux.​­org/​­blog/​­new‑​­repositories‑​
are approved and implemented Conclusion ­for‑almalinux‑​­os‑synergy‑and‑testing/]
upstream. AlmaLinux has invited [4] Packaging chat channel:
community members to help test Despite Red Hat making it more diffi- [https://​­chat.​­almalinux.​­org/​­login?​­redirect_
these updates. (As per usual, Test- cult to use RHEL code, AlmaLinux has to=%2Falmalinux%2Fchannels%2Fengine
ing is not recommended for produc- adjusted course, relying on ABI com- eringpackaging]
tion machines.) patibility to deliver a RHEL alternative
Synergy contains packages re- for the enterprise Linux ecosystem. Author
quested by community members Moving forward, AlmaLinux plans Amy Pettle is an editor for ADMIN and Linux
that currently aren’t available in to continue contributing upstream to Magazine.

8 FOCUS ON ALMALINUX W W W. A D M I N - M AGA Z I N E .CO M

 HPC Use Cases FO C U S O N A L M A L I N U X

An open source enterprise Linux alternative

AlmaLinux and HPC

AlmaLinux steps forward as a free, open source enterprise Linux solution for HPC systems. By Amy Pettle

Linux dominates the world of high Luckily, AlmaLinux [3], an open also helps users build more secure
performance computing (HPC), with source, community-driven Linux, environments by allowing the integra-
CentOS, a free alternative to Red Hat quickly arose to fill this gap by de- tion of security tools like OpenSCAP
Enterprise Linux (RHEL), historically livering a CentOS alternative in early and Center for Internet Security (CIS)
being the system of choice for HPC 2021. Originally a downstream build Benchmarks, a set of guidelines that
clusters [1] used in noncommercial of RHEL with 1:1 (or bug-to-bug) help secure systems, software, and
settings like universities and research compatibility, AlmaLinux pivoted to networks against evolving cyberthreats.
institutions. being application binary interface With help from CloudLinux, Alma-
That all changed in late 2020 when (ABI) compatible in the summer Linux 9 OS also has achieved Federal
Red Hat (a subsidiary of IBM) an- of 2023 when Red Hat announced Information Process Standard (FIPS)
nounced it would replace CentOS restrictions to RHEL-related source 140-3 validation, which means Alma-
with CentOS Stream. Whereas code. Today, AlmaLinux provides Linux 9 is using approved algorithms
CentOS sat downstream of RHEL HPC users with a production grade, for encryption, hashing, assigning,
making it a reliable alternative to enterprise operating system that and so on.
RHEL, CentOS Stream sits upstream offers the stability and security re- AlmaLinux’s move to ABI compatibil-
and serves as a developmental quired in HPC environments. ity has freed it from being bound to
platform for Red Hat partners and In terms of stability, each AlmaLinux bug-to-bug compatibility with RHEL.
community members who want to release has a 10-year life cycle, and As a result, bug fixes can potentially
contribute to RHEL. Compared to AlmaLinux attempts to update re- be addressed more quickly than in
CentOS’s 10-year life cycle, CentOS leases within one business day of RHEL. In her 2023 recap [4], benny
Stream has a 5-year life cycle and RHEL updates. In addition, AlmaL- Vasquez, AlmaLinux OS Foundation
updates software versions more fre- inux’s nonprofit status, with a guar- Chair, noted “As a result of this change
quently, making it a poor choice for anteed $1 million in sponsorship from we have introduced new repositories
Lead Image © jussang, 123RF.com

anyone who needs long-term sup- CloudLinux and other sponsors, as- for AlmaLinux and have been able
port and consistency. Consequently, sures HPC users that AlmaLinux will to ship critical security and bug fixes
CentOS Stream is not intended for have the necessary funding for future sooner than any other enterprise Linux
production environments [2]. It updates. distro.”
lacks the reliability required for the AlmaLinux uses firewalld, iptables, and As a community-governed
HPC ecosystem. SELinux for security mechanisms. It nonprofit, the AlmaLinux OS

W W W. A D M I N - M AGA Z I N E .CO M FO CU S O N A L M A L I N UX 9
 FO C U S O N A L M A L I N U X HPC Use Cases

Foundation owns and manages the Germany running AlmaLinux 9.2 software after Red Hat restricted the
operating system, intellectual prop- (November 2023) [8] source code access.”
erty, and direction of the project. n #187 “Alex” at University of
Foundation membership is open to Erlangen-Nuremburg – NHR@FAU, CERN
AlmaLinux OS users, project contrib- Germany running AlmaLinux 8.5
utors, mirror maintainers/​sponsors, (November 2023) [9] CERN [14], the European Organiza-
service providers to the community, n #213 “Fritz” at University of tion for Nuclear Research, is also
and official project partners [5]. Not Erlangen-Nuremburg – NHR@FAU, using AlmaLinux. In addition to
only do members vote on the Alma- Germany running AlmaLinux 8.5 RHEL, CERN’s IT department began
Linux Board of Directors, but they (November 2023) [10] supporting AlmaLinux in production
also help decide the direction of n #416 “VSC-5” at Vienna University a little over a year ago.
AlmaLinux, including any important of Technology, Austria running Dr. Arne Wiebalck, Head of Compute
issues that may arise. Ultimately, AlmaLinux 8.5 (November 2023) [11] and Devices Group at CERN IT, said
the community, not a corporation, n #467 “Marvin” (Scalable GPU they started offering AlmaLinux in
owns AlmaLinux. Partition) at University of Bonn, December 2022 for CERN use cases
As a result of AlmaLinux’s stability, Germany running AlmaLinux 9.1 in “close collaboration with other
security, and community gover- (November 2023) [12] laboratories from the High Energy
nance, HPC community members n #483 “amplitUDE (GPU Partition)” Physics community.” By early 2023,
have turned to AlmaLinux, includ- at University of Duisburg-Essen, the appropriate infrastructure was in
ing MEGWARE, CERN, and Uni- Germany running AlmaLinux 9.1 place to support AlmaLinux.
versity College London (UCL). We (June 2023) [13] CERN mainly uses AlmaLinux for
had the opportunity to interview Due to updates and security patches services offering interactive access to
all three companies about their since making the list in 2023, the non-CERN sites as well as for virtual
adoption of AlmaLinux in an HPC VSC-5, Alex, and Fritz clusters cur- machine and container images that
environment. rently use AlmaLinux 8.9, while the will be distributed outside of CERN.
Marvin and amplitUDE clusters have CERN also coordinates the Worldwide
TOP500 with MEGWARE updated to AlmaLinux 9.3. Large Hadron Collider (LHC) Comput-
Being free and open source makes ing Grid. This global computing infra-
MEGWARE [6], a leading European AlmaLinux a good fit for MEGWARE’s structure handles the vast amount of
supercomputer specialist, develops HPC clusters, which are used exclu- data generated by the LHC. Wiebalck
and installs HPC systems and Linux sively for research and education, as notes:
clusters. Several of MEGWARE’s super- they offer a free alternative to RHEL.
computers have made the TOP500 [7], Großöhme notes, “Especially our “As not all participating sites have
a biannual ranked list of the world’s academic customers won’t spend ad- support contracts for an enterprise
most powerful non-distributed com- ditional money for subscription and/​ Linux distribution, the deployment
puter systems. or support fees for each node.” of a meaningful fraction of the CERN
In looking for a CentOS replace- Another advantage to AlmaLinux compute capacity with a free and open
ment in December 2020, a customer is its quick release cycle, even after source Linux distribution is an im-
pointed MEGWARE toward Alma- Red Hat made it more difficult to ac- portant aspect of CERN’s role in this
Linux, which already had a beta cess the official RHEL source code. community.”
version available. MEGWARE started Großöhme says:
their first HPC cluster with AlmaLinux When looking for an enterprise Linux
in March 2021 and have been using “AlmaLinux provides updates and clone, Wiebalck said:
it for almost all of their HPC clus- security patches very quickly – the last
ter deployments since then. Peter major update to AlmaLinux 9 was “The CERN IT Linux team did a thor-
Großöhme, Chief Operating Officer released within a few days after the ough review of the characteristics of
and Head of Engineering at MEGWARE, official announcement by Red Hat. the various options. Eventually, the
says “If there is no preference from This is an outstanding performance long life cycles for major versions, the
the customer’s side, MEGWARE goes compared to previous CentOS commu- extended architecture support, the
with AlmaLinux.” nity releases and other current RHEL- rapid release cycle, the upstream
As for the TOP500 list, six MEGWARE based Linux distributions.” community contributions, and the
HPC clusters running AlmaLinux support for security advisory data
made the TOP500 list in 2023, demon- Finally, Großöhme appreciates the made AlmaLinux the logical choice
strating AlmaLinux’s viability in the way the AlmaLinux team works. He for our use cases.”
HPC marketplace: likes “the whole team and their mind-
n #84 “Goethe-NHR” at Goethe set” and, in particular, AlmaLinux’s Because the LHC has multi-year run
University Frankfurt – NHR@SW, willingness “to fight for open source phases, CERN needs a system that

10 FOCUS ON ALMALINUX W W W. A D M I N - M AGA Z I N E .CO M

 HPC Use Cases FO C U S O N A L M A L I N U X

offers stability and predictability. For AlmaLinux for about six months as potentially fixing bugs ahead of
CERN, Wiebalck says “the setup of their development platform to deploy RHEL and other RHEL alternatives.
AlmaLinux as a 501(c)(6) nonprofit tools onto their HPC and private Both HPC cluster manufacturers and
organization to ensure the processes cloud RHEL Systems. research institutions have turned to
are open, transparent, and allow License management played an im- AlmaLinux as a RHEL alternative for
for participation reduces the risk of portant role in UCL’s decision to use their HPC environments.
surprises and was clearly among the AlmaLinux. Because AlmaLinux is
reasons to go with AlmaLinux.” very close to RHEL, it offers a good Info
AlmaLinux’s move to ABI compatibil- platform for testing deployments mi- [1] C entOS and HPC clusters:
ity (rather than bug-to-bug compat- nus “the complicated and often fragile [https://​­uwm.​­edu/​­hpc/​­software‑​­and‑​
ibility) and the subsequent potential license management tools used by ­operating‑systems/]
ability to address bugs and security Red Hat.” Owain Kenway, Head of Re- [2] CentOS Stream: [https://​­www.​­redhat.​­com/​
issues before releases in other offer- search Computing at UCL ARC, notes: ­en/​­resources/​­centos‑stream‑checklist]
ings gave AlmaLinux an advantage [3] AlmaLinux: [https://​­almalinux.​­org]
over other options. “We actually have sufficient licenses [4] “A look back at 2023 for AlmaLinux”
Finally the AlmaLinux community to cover the development systems by benny Vasquez, December 13, 2023,
in general was a draw for CERN. running RHEL should we want to [https://​­almalinux.​­org/​­blog/​­2023‑12‑14‑​
Wiebalck said: do that. However the pains of man- ­2023‑highlights/]
aging Satellite and Subscription [5] AlmaLinux OS Foundation membership:
“The CERN IT Linux team felt ex- Manager on what are often very [https://​­almalinux.​­org/​­members/]
tremely welcome from day one; we temporary cloud VMs (cattle not [6] MEGWARE: [https://​­www.​­megware.​­com/​­en/]
could openly discuss issues and con- pets!) mean that it is much more [7] TOP500: [https://​­top500.​­org/​­lists/​
cerns (e.g., around support for older painless to use AlmaLinux for these ­top500/​­2023/​­11/]
custom-built hardware) and always activities.” [8] Goethe-NHR:
felt the AlmaLinux team was trying to [https://​­top500.​­org/​­system/​­180175/]
find the best solution, balancing our Kenway also uses AlmaLinux in [9] Alex:
specific needs with the requirements teaching because of licensing issues. [https://​­top500.​­org/​­system/​­180075/]
from the community as a whole in a He finds that students can easily get [10] Fritz:
very transparent way.” “legitimate access” to AlmaLinux [https://​­top500.​­org/​­system/​­180074/]
where they can’t to RHEL. [11] VSC-5:
In fact, AlmaLinux welcomed Alex Irib- AlmaLinux’s governance structure [https://​­top500.​­org/​­system/​­180056/]
arren, a systems engineer at CERN and also appealed to UCL. Being a uni- [12] Marvin:
leader of CERN’s Cloud and Linux Plat- versity, “the ethos of a ‘community [https://​­top500.​­org/​­system/​­180161/]
form Engineering teams, to the Alma- run’ project” aligned well with UCL. [13] amplitUDE (GPU Partition):
Linux OS Foundation board in Decem- Kenway also appreciates the way [https://​­top500.​­org/​­system/​­180159/]
ber 2023. In the announcement [15], AlmaLinux has handled the chal- [14] CERN: [https://​­home.​­cern]
Vasquez pointed out that Iribarren’s lenges presented by changes in ac- [15] “Meet our two new board members!”
experience “managing a large and di- cess to RHEL’s source code in 2023. by benny Vasquez, December 19, 2023,
verse user base and his insights into the Regarding UCL’s experience with [https://​­almalinux.​­org/​­blog/​­2023‑12‑19‑​
High Energy Physics community will AlmaLinux, Kenway says “we are ­jun‑and‑alex‑join‑board/]
help the AlmaLinux OS Foundation to extremely happy with it.” [16] UCL ARC: [https://​­www.​­ucl.​­ac.​­uk/​
meet the needs of the scientific com- ­advanced‑research‑computing/​
munity and contribute to the overall Conclusion ­advanced‑research‑computing‑centre]
success of the project’s mission.”
AlmaLinux is filling the void left by The Author
University College London CentOS for academic and research Amy Pettle is an editor for ADMIN and Linux
facilities in the HPC ecosystem. Magazine. She started out in tech publishing
At University College London (UCL), Not only is AlmaLinux stable and with C/C++ Users Journal over 20 years ago
the Advanced Research Computing secure, but its move to ABI com- and has worked on various Linux New Media
(ARC) Centre [16] has been using patibility offers the added perk of publications.

W W W. A D M I N - M AGA Z I N E .CO M FO CU S O N A L M A L I N UX 11
 FO C U S O N A L M A L I N U X AlmaLinux Build System

Exploring the AlmaLinux Build System

Package Packer
The AlmaLinux Build System lets you build, test, sign, and release system at other, third-party Git reposi-
tories, which makes it suitable for
packages from a single interface. By Joe Casad many in-house DevOps development
settings.
When IBM announced that it and releasing them to public reposi-
was restricting access to Red Hat tories.” In other words, the goal is How Does It Work?
Enterprise (RHEL) source code and to assist with every phase of the
moving CentOS upstream, the distros package development process, rely- The AlmaLinux Build System auto-
that depended on RHEL and CentOS ing on automation to reduce human mates the process of building, sup-
source code were sent scrambling. It error and minimize manpower porting, and managing packages.
is still a little unclear whether IBM’s requirements. The vision is for something that is
moves are legal and consistent with The AlmaLinux Build System is a more than a build tool, with support
the GNU Public License (GPL), but free software project that is avail- for testing, signing, and releasing
the litigation to sort it out could able on GitHub. Other Linux dis- software packages.
take years, and in the meantime, the tributions are welcome to use the If the AlmaLinux project needed a
derivatives need a solution. AlmaLinux Build System as a tool build system to interact with source
One enterprise distribution that for building and managing pack- code originating from a Red Hat en-
weathered the storm quite smoothly ages. You can also point the build vironment, you might be wondering
was AlmaLinux [1] (see the box
entitled “Where Do They Get Their Where Do They Get Their Code?
Code?”). If you ask the AlmaLinux AlmaLinux was envisioned as a free alterna- IBM currently restricts access to some, but not
developers, they will say that one rea- tive to RHEL, which comes with a subscrip- all, RHEL source code. Some source code is
son for their success in navigating the tion fee and other corporate licensing available through the Red Hat Universal Base
transition to the post-RHEL era is the arrangements. AlmaLinux and other RHEL Image (UBI) [3]. AlmaLinux uses as much of
AlmaLinux Build System [2]. derivatives used source code from Red Hat the Red Hat UBI code as it can, but a majority
The AlmaLinux Build System repositories as the basis for building an of the code comes from the CentOS Stream
Lead Image © stylephotographs, 123RF.com

evolved from an earlier system independent distro. It is important to note project [4]. IBM did not eliminate all access
used by CloudLinux. (CloudLinux that Red Hat does not own the source code in to CentOS; they just moved it upstream, so the
is a contributor to the AlmaLinux the sense that the term own is used with pro- code does not include some of the final bug
project.) The developers refer to prietary software. Because Linux and most fixes and updates that go into the final version
their build system as “a project of the code included with it are open source of RHEL. AlmaLinux uses some code from the
designed to automate processes of and licensed under the GNU Public License CentOS Stream project and performs its own
(GPL), the code is available for others to use fixes and updates. They also pull code from
building distribution and packages,
and modify. other upstream sources when necessary.
testing packages, signing packages,

12 FOCUS ON ALMALINUX W W W. A D M I N - M AGA Z I N E .CO M

 AlmaLinux Build System FO C U S O N A L M A L I N U X

why they didn’t just use Koji [5], n Terraform – an infrastructure-as- interface or text-based commands.
the freely available build tool as- code tool used to build simulated Support for command-line process-
sociated with Red Hat’s Fedora environments for package testing ing creates the possibility for script-
project. The answer given by the de- n PGP – an encryption utility that ing and other custom automation
velopers is that, although Koji is an provides signing services for pack- scenarios.
effective tool, the AlmaLinux project age verification At the center of the system is the
had a much broader vision. For one n Git – a source code repository Build System Master Service. The
thing, they wanted to integrate ad- system Master Service receives commands
ditional package formats (Koji is Git isn’t actually part of the build sys- from the user and sets the process
limited to RPMs). They also wanted tem itself, but it is an integral part of in motion, creating, restarting, and
to provide a complete, integrated the ecosystem, providing source code deleting builds and communicating
pipeline to manage a package from for building packages and communi- with the rest of the system via API
the build phase, to testing, to sign- cating with the build system through calls. Responsibilities of the mas-
ing the package, and finally to re- an API. ter service include requesting and
lease. The AlmaLinux Build System Figure 1 shows the complete sys- receiving source code from the Git
includes controls that allow the user tem at a glance. Users interact server and assigning tasks to the
to specify where to release pack- through either a graphical user build nodes.
ages, and it is one of
the first build systems
to support modular-
ity. A module is a
collection of packages
that occur together,
such as the packages
in a single applica-
tion or an operating
system component.
Support for modular-
ity lets you treat the
packages together,
thus saving steps
and streamlining the
configuration.
Like other build plat-
forms, the AlmaLinux
Build System is not a
monolithic application
but a combination of
back-end tools behind
a single, unified inter-
face. Some of the tools
incorporated into the
AlmaLinux Build Sys-
tem include:
n Mock – a tool
for building RPM
packages
n Pulp – a content
repository for or-
ganizing and dis-
tributing software
packages
n NGINX – a web
server that serves
as an interface for
managing access to
the build system Figure 1: The AlmaLinux Build System at a glance.

W W W. A D M I N - M AGA Z I N E .CO M FO CU S O N A L M A L I N UX 13
 FO C U S O N A L M A L I N U X AlmaLinux Build System

builds appears in the main screen

(Figure 2). A menu on the left of-
fers options for creating a new build,
new release, or new distribution.
Click on the Details link for a config-
ured build to view the build settings.
You can choose an architecture (Fig-
ure 3) or view the artifacts associ-
ated with the build (Figure 4).
To create a new build, choose New
build in the main view. In subsequent
menus, you can select a platform and
choose architecture(s), set options
for the Mock build tool, and specify
whether to build for a Secure Boot
system.
Once you have configured the build
settings, select the project (the pack-
ages) you would like to build (Fig-
Figure 2: Your first view shows the builds configured for the system. ure 5) and click on Create Build.

Another important component of First Look Testing and Later Steps

the build system is Pulp [6], which
provides artifact storage for newly- When you log in to the AlmaLinux The AlmaLinux Test System
built packages and other products Build System, a view of configured (ALTS) [8] included with the build
of the build process. According to
AlmaLinux Community Manager SBOMs
Jack Aboutboul, “the master service On May 12, 2021, the Biden administration help investigators identify and trace security
is the brain, and Pulp is the heart” released Presidential Executive Order 14028 risks that might affect the package. If a com-
of the build system. As you can see “Improving the Nation’s Cybersecurity” [9]. ponent used in building the package turns up
One of the important features of that order with a critical vulnerability, it will be easy to
in Figure 1, Pulp is essential to the
is the stipulation that software packages for spot the problem and to know that the pack-
later stages of the process, provid-
software used by the US government should age needs an update.
ing packages for signing and testing, include a bill of materials for all the code AlmaLinux was the first Linux distribution to
and forwarding finished packages provided in the package. This Software Bill notarize and provide an SBOM for all source
for release. Much of the power of of Materials (SBOM) is described as a “list and components. The AlmaLinux developers
the AlmaLinux Build System is in its of ingredients” for the software package. created an SBOM generation utility and inte-
ability to oversee the testing, signing, The idea is that providing an accurate list of grated it into the AlmaLinux Build System. You
ingredients used for building the package will can find the alma-sbom utility on GitHub [10].
and release phases of the develop-
ment process.

Getting the Code

The AlmaLinux Build System uses
the Gitea software development ser-
vice [7] to communicate with the
Git server. Gitea is described as an
all-in-one service for managing a Git
environment, including “code review,
team collaboration, package registry
and CI/​CD.” The AlmaLinux team
has developed a gitea-listener tool
for interfacing with Gitea and the Git
repository. The AlmaLinux Build Sys-
tem also supports Fedora Community
Repository Platform format (COPR),
which makes it easy to add alterna-
tive repositories to the system. Figure 3: The AlmaLinux Build Systems supports several hardware architectures.

14 FOCUS ON ALMALINUX W W W. A D M I N - M AGA Z I N E .CO M

 AlmaLinux Build System FO C U S O N A L M A L I N U X

environment automates packet test- design with support for scripting [5] Koji: [https://​­koji.​­build/]
ing in realistic conditions. ALTS first opens a range of possibilities for [6] Pulp Project: [https://​­pulpproject.​­org/]
launches a clean test environment adapting the build system for other [7] Gitea: [https://​­github.​­com/​­go‑gitea/​­gitea]
(for instance, a Docker container) projects. n [8] AlmaLinux Test System:
using Terraform to recreate a real- [https://​­github.​­com/​­AlmaLinux/​­alts]
istic setting that models actual pro- [9] Presidential Executive Order 14028:
duction conditions. Once the envi- Info [https://​­www.​­whitehouse.​­gov/​
ronment is in place, ALTS attempts [1] AlmaLinux: [https://​­almalinux.​­org/] ­briefing‑room/​­presidential‑actions/​­2021/​
to install the package, and, if the [2] AlmaLinux Build System: [https://​­github.​ ­05/​­12/​­executive‑​­order‑on‑improving‑the‑​
installation is successful, begins a ­com/​­AlmaLinux/​­build‑system] ­nations‑cybersecurity/]
series of integrity checks predefined [3] Red Hat Universal Base Image: [10] alma-sbom: [https://​­github.​­com/​
by the user. [https://​­catalog.​­redhat.​­com/​­software/​ ­AlmaLinux/​­alma‑sbom]
Results of the tests are then for- ­base‑images]
warded to the Pulp artifacts store [4] CentOS Stream: Joe Casad
in the form of test logs and reports, [https://​­www.​­centos.​­org/​­centos‑stream/] Joe Casad is the editor in chief of Linux Magazine.
and the results are then available to
the user through the web interface.
Approved packages are then signed
and marked for release. The build
system lets you define and select
specific channels for the software
release, and the verification sys-
tem allows the receiver to trace the
authenticity back to the original
source code.

Conclusion
The stability and versatility of the
AlmaLinux Build System has given
the developers a head start on
achieving the project’s ambitious
goals while avoiding much of the
wheel spinning that often comes
with putting a distribution together.
AlmaLinux was recently chosen as Figure 4: Viewing the artifacts associated with the build.
a standard Linux distribution for
Fermilab and the CERN European
laboratory for particle physics. The
AlmaLinux project was also the first
enterprise Linux to offer a complete
Software Bill of Materials (SBOM)
for every package (see the box en-
titled “SBOMs”).
The AlmaLinux team is busy right
now using the AlmaLinux Build
System to create, sign, test, and re-
lease the next version AlmaLinux,
but the developers also want to
sure make the system is available to
other users and other projects. The
user interface makes it easy to in-
corporate other source code reposi-
tories, and the testing, signing, and
release components support cus-
tomization for alternative projects
and applications. An API-driven Figure 5: Adding a project to the build.

W W W. A D M I N - M AGA Z I N E .CO M FO CU S O N A L M A L I N UX 15
 FO C U S O N A L M A L I N U X AlmaLinux’s ELevate Migration Tool

Migrating from CentOS 7

ELevate to the Future

CentOS 7 reached end of life in June 2024, forcing users to look for a free enterprise Linux alternative.
AlmaLinux’s ELevate migration tool can help ease the transition. By Rubén Llorente
For years, CentOS was the operating immediately after this announce- convert an existing CentOS 7.x install
system of choice for users who ment. One of these alternatives, to the 8.x version of a RHEL derivative.
needed a free enterprise Linux solu- AlmaLinux [2], quickly stepped in Developed by AlmaLinux to be agnos-
tion. CentOS offered a predictable as a replacement for CentOS by of- tic, ELevate supports upgrade paths
life cycle and a long lifespan for fering 1:1 bug compatibility with for multiple RHEL derivatives (see
each release, making it a reliable RHEL by early 2021, eventually Figure 1). You can also use ELevate
alternative to the commercial Red moving to application binary inter- as a regular upgrade tool.
Hat Enterprise Linux (RHEL) dis- face compatibility in 2023 when Red Until recently, ELevate only supported
tribution. That all changed in 2020 Hat restricted access to RHEL source official operating system repositories.
when Red Hat replaced CentOS 8 code. Today, AlmaLinux provides an If your CentOS install used “extra”
with CentOS Stream and announced alternative for former CentOS users repositories, such as the popular Extra
that CentOS 7 would reach end of as a forever-free, community-gov- Packages for Enterprise Linux (EPEL)
life on June 30, 2024. erned, production-grade platform repository, you could only migrate
Despite its name, CentOS Stream focused on long-term stability. using ELevate Testing. ELevate now
does not serve as a replacement for While an alternative free enterprise offers third-party EPEL support for
CentOS. While CentOS sat down- Linux solution like AlmaLinux is good migrating from CentOS 7 to Alma-
stream of RHEL, making it a reliable news in light of the fast approaching Linux 8 in the stable release [5].
replacement for RHEL, CentOS CentOS end of life, administrators still Furthermore, ELevate has added sup-
Stream sits upstream and serves as need to migrate their existing CentOS port for other repositories (Imunify,
a developmental platform for RHEL systems to a new enterprise Linux KernelCare, MariaDB, NGINX, and
Lead Image © sergeyback, 123RF.com

contributors. As a result, Red Hat distribution. Migration isn’t exactly a PostgreSQL) for all supported systems
has warned that CentOS Stream is trivial task. Luckily, AlmaLinux has (AlmaLinux, CentOS, EuroLinux,
not considered stable for production an answer for this migration problem: Oracle Linux, and Rocky Linux).
environments [1]. ELevate [3]. By combining Red Hat’s With CentOS 7’s end of life quickly ap-
As one would expect from the open Leapp [4] framework with a com- proaching, system administrators need
source community, CentOS alter- munity-created migration metadata to develop a migration plan. Alma-
natives started popping up almost library and service, ELevate lets you Linux’s ELevate tool can help ease the

16 FOCUS ON ALMALINUX W W W. A D M I N - M AGA Z I N E .CO M

 AlmaLinux’s ELevate Migration Tool FO C U S O N A L M A L I N U X

pain of migration. In this article I will As mentioned earlier, ELevate uses it to upgrade CentOS 7 into
show you how to use ELevate to mi- Red Hat’s upgrade tool, Leapp. In AlmaLinux:
grate from CentOS 7 to AlmaLinux 8. order to migrate to AlmaLinux,
you must fetch Leapp, along with $ sudo yum install ‑y leapp‑upgradeU
Precautions Before the patch files that will configure leapp‑data‑almalinux

Upgrading
This article intends to be a quickstart
guide on ELevate to help you upgrade
away from CentOS 7 to AlmaLinux
quickly and painlessly. Before you get
started, it is important you adopt the
necessary precautions and make the
required preparations.
While ELevate is being used to up-
grade production systems in the wild,
caution is still advised. At the very
least, you should back up your sys-
tems before you get started, just in
case. AlmaLinux recommends doing
a trial run in a sandbox to verify the
migration works in your environment
before migrating a production system.
The upgrade process will be done
in-place, which means AlmaLinux
will be installed over your CentOS Figure 1: AlmaLinux supports multiple upgrade paths. Notice that ELevate does not support
instance directly. The machine being upgrading from Oracle Linux 8 to Oracle Linux 9. Upgrading to CentOS Stream 9 is still
subjected to conversion will need to experimental at the time of writing.
be rebooted more than once.
Only single-step upgrades are sup-
ported. This means that if you intend
to migrate to AlmaLinux 9, you will
need to migrate to AlmaLinux 8 first
and then upgrade from there.

Gathering the Tools

Before you start migrating, you first
must ensure your CentOS instance is
up to date. I like to reboot after sys-
tem updates to ensure every installed
service loads the updated compo-
nents as follows:

$ sudo yum update ‑y

$ sudo reboot

Next, you need to install ELevate.

You can install the standard release
from the associated AlmaLinux re-
pository, which you can enable as
follows:

$ sudo yum install ‑y U

http://repo.almalinux.org/elevate/U
elevate‑release‑latest‑elU Figure 2: This failure message is to be expected. In order to proceed further, you need to
$(rpm ‑‑eval %rhel).noarch.rpm fix the problems listed in /var/log/leapp/answerfile.

W W W. A D M I N - M AGA Z I N E .CO M FO CU S O N A L M A L I N UX 17
 FO C U S O N A L M A L I N U X AlmaLinux’s ELevate Migration Tool

Figure 3: The issues listed in /var/log/leapp/answerfile must be resolved before the upgrade process can commence.

Listing 1: Common Fixes after the Pre-Upgrade You will need check the current SELinux state with
01 sudo rmmod pata_acpi
to reboot manu- the following command:
02 echo PermitRootLogin yes | sudo tee ‑a /etc/ssh/sshd_config ally afterwards
03 sudo leapp answer ‑‑section remove_pam_pkcs11_module_check.confirm=True (Figure 4). The $ getenforce
machine will boot
into your new If the output of the above command
Ready to Migrate AlmaLinux 8 distribution. If you run is Enforcing, SELinux is enabled and
into any additional issues, please see there is nothing left to do. If not, you
It is a good idea to ensure your ap- the ELevate Frequent Issues page on can set SELinux to enforcing mode
plications will run on AlmaLinux the AlmaLinux website [6]. by editing /etc/selinux/config and
before you migrate. AlmaLinux pro- setting the variable SELINUX within the
vides a pre-upgrade check to deter- Verifying SELinux State config file to enforcing. You then need
mine the feasibility of the migration. to reboot your system for the configu-
When migrating from CentOS 7, you Running leapp will set SELinux to ration to take effect.
are likely to get a failure message at permissive mode during the upgrade
this stage, which is to be expected in order to carry it out. This should Conclusion
(Figure 2). Make sure you invoke not be an issue for most users. Nev-
the preupgrade checks with the fol- ertheless, if you had SELinux enabled As CentOS’s end of life draws near,
lowing command: in CentOS 7, you might want to have you are going to need a migration
it enabled in AlmaLinux 8. You can plan for moving forward. ELevate
$ sudo leapp preupgrade

During the pre-upgrade stage, a

report will be saved to /var/log/
leapp/leapp‑report.txt with a list
of dangerous actions that might oc-
cur during the migration. Some sug-
gestions are also given. Most likely,
a file will be generated in /var/log/
leapp/answerfile with questions
that require a true/​false answer.
These questions must be answered
before the upgrade can take place
(Figure 3). You can answer these
questions by either editing the file
directly or using leapp answer. List-
ing 1 shows the bare minimum fixes
you should perform immediately
after running preupgrade in order to
satisfy the answerfile from a CentOS 7
install.
Once you are ready, launch the up-
grade by issuing the final command:

$ sudo leapp upgrade Figure 4: Once leapp upgrade is finished, the only thing left to do is to reboot your machine.

18 FOCUS ON ALMALINUX W W W. A D M I N - M AGA Z I N E .CO M

 AlmaLinux’s ELevate Migration Tool FO C U S O N A L M A L I N U X

helps you migrate from CentOS to then using ELevate to move forward [2] AlmaLinux: [https://​­almalinux.​­org]
AlmaLinux, or another RHEL-com- to AlmaLinux 9 immediately, which [3] ELevate: [https://​­almalinux.​­org/​­elevate/]
patible distribution, with relative will give you a distribution sup- [4] Red Hat Leapp: [https://​­www.​­redhat.​­com/​
ease. During testing, I discovered ported until 2032. ­en/​­resources/​­leapp‑explained‑detail]
that ELevate could reliably upgrade As a community-driven Linux OS, [5] EPEL support: [https://​­almalinux.​­org/​­blog/​
a CentOS 7 machine, which hosted AlmaLinux welcomes contributions to ­2024‑01‑31‑elevate‑updates/]
a Dockerized web application and the ELevate project. For more informa- [6] ELevate Frequent Issues:
some EPEL services, using nothing tion, see the “Get Involved” box [7]. n [https://​­wiki.​­almalinux.​­org/​­elevate/​
but the steps described in this ar- ­ELevate‑​­frequent‑issues.​­html#​­leapp‑​
ticle (Figure 5). ­upgrade‑error‑caused‑by‑​­important‑​
Your new AlmaLinux 8 install Info ­modules‑from‑kernel‑drivers]
should be able to serve you well un- [1] CentOS Stream in production use: [7] ELevate Contribution Guide:
til 2029, when it reaches end of life. [https://​­www.​­redhat.​­com/​­en/​­resources/​ [https://​­wiki.​­almalinux.​­org/​­elevate/​
However, you can upgrade before ­centos‑stream‑checklist] ­Contribution‑guide.​­html]

Author
Rubén Llorente is an OpenBSD enthusiast
and a mechanical engineer whose job is to
ensure that the security measures of the IT
infrastructure of a small clinic are both law
compliant and safe.

Get Involved
If you’d like to contribute to ELevate, Alma‑
Linux has launched the Package Evolution
Service (PES). PES collects bug reports
and issues from users as well as supplies
information on migrating from one RHEL-
compatible distribution to another. For more
information on how to contribute to PES, see
the ELevate Contribution Guide [7].
Figure 5: Welcome to AlmaLinux! This is what a successful migration looks like.

W W W. A D M I N - M AGA Z I N E .CO M FO CU S O N A L M A L I N UX 19
 Fo c u s o n A l m a L i n u x Interview: AlmaLinux

An interview with AlmaLinux OS

Freedom Fighter
AlmaLinux team members discuss their quest to deliver a forever-free,
community-driven enterprise Linux distribution. By Amy Pettle

Since its debut in 2021, AlmaLinux When setting out to fill the void left possible with AlmaLinux. That’s why
OS has stepped up to provide a free by CentOS, what key considerations we decided that an independent,
enterprise Linux alternative in the drove AlmaLinux’s development? nonprofit organization had to own all
wake of CentOS’s demise. In the ever- the trademarks and copyrights, and
changing enterprise Linux ecosystem, benny Vasquez: One of our core be- all of the related management, code,
AlmaLinux OS’s journey has not been liefs is that the long-term health of build, and release infrastructure. We
without a few challenges to delivering the enterprise Linux ecosystem relies didn’t want a single person, group, or
a drop-in replacement for Red Hat En- on a stable, secure, compatible, no- company to be able to overly influ-
terprise Linux (RHEL). Throughout this cost distribution with a long life cycle. ence the project or to be able to (or
quest, AlmaLinux OS has put its focus We knew we wanted to learn from be forced to, as with CentOS) sell it
front and center on the needs of its the amazing open source projects that to the highest bidder years down the
community. To learn more about have thrived before us, while also road. We were users of and counted
AlmaLinux OS’s journey, as well as making sure we anticipated the next on CentOS for nearly 20 years, and
what the future may hold, we talk to evolution of open source. Ensuring our goal is to build an organization
benny Vasquez, AlmaLinux OS Foun- AlmaLinux couldn’t become a single- that will successfully serve its com-
dation Chair and Marketing Sig Lead; vendor open source project and en- munity for even longer.
Andrew Lukoshko, AlmaLinux OS suring diversity in both contributors
Lead Architect and Core SIG Lead; and and funding would be critical to our What are the AlmaLinux OS
Elkhan Mammadli, Cloud SIG Lead. long-term success. Foundation’s goals? How does the
foundation measure the success of
Shortly after AlmaLinux’s first these goals?
stable release, CloudLinux turned
over governance to the AlmaLinux benny: The AlmaLinux OS Foundation
OS Foundation. Why was this impor- was set up with the goal of providing
tant and what are the advantages of central ownership and a central point
a community-governed nonprofit of collaboration for the community
for AlmaLinux? around AlmaLinux OS. We enable the
community to achieve its goals with
benny: There was a lot of uncertainty AlmaLinux however they want to and
and doubt after the announcement connect those who want to work on
of CentOS’s end of life. Some of it similar projects. We see evidence of
was “how will I solve the urgent our success as the number of individ-
problem,” but immediately beyond ual contributors grows, and the con-
benny Vasquez is the Chair of that was “what can be done so we tinued excitement around AlmaLinux
aren’t left stranded again.” We see the leads to the expansion of what we’re
the AlmaLinux OS Foundation,
problem being in motivations: Red helping solve.
with a long history in business Hat isn’t bad, but at the end of the
and community building and an day it’s a corporation, and corpora- How has AlmaLinux sponsorship
undying love of open source. tions have to make money. When Red and adoption grown since 2021?
benny also serves as the Hat acquired CentOS, CentOS moved
from “Project” to “Product” and we benny: We announced our inten-
Marketing SIG Lead.
wanted to ensure that was never tions in January of 2021 with just

20 Focus on AlmaLinux w w w. a d m i n - m aga z i n e .co m

 Interview: AlmaLinux FO C U S O N A L M A L I N U X

CloudLinux as our sponsor, and by evidence of the importance of a free security. Knowing that AlmaLinux
the time we released our first build and open source operating system in OS is compatible with RHEL means
that March we had a handful of spon- the enterprise Linux ecosystem. that they benefit from the massive
sors. Those companies were primarily ecosystem that grew around CentOS
providing hardware or infrastructure Does the AlmaLinux OS Foundation Linux including software support
that we needed to build the operat- plan to expand hardware support in and stacks of guides and community
ing system. We soon had additional the future? knowledge. They also see both an
sponsors that backed us with capital active user community and an ever-
and personnel, too. We now have Andrew Lukoshko: Yes, of course! In growing list of sponsors as evidence
25+ tiered sponsor members, 100+ terms of architecture, we just added of a strong project. And, obviously,
mirror sponsors, and are adding more support for Raspberry Pi 5 and are because the project is free and you
every month, with all of our sponsors always looking at the needs of our can use images or pick us as the OS
returning for multiple years. users and enabling expansion to on all of the cloud providers, the
meet those needs. RISC-V support is barrier to getting started with Alma-
Adoption has been steadily growing, one of the things our community is Linux is nearly nonexistent. It’s an
and we see evidence of that adop- working on now, as we continue to easy drop-in replacement for anyone
tion in a variety of ways. We’ve see excitement and adoption expand- that is using a RHEL-compatible distro
seen the number of mirrors in our ing there. We also brought forward already.
mirror system grow from 170 at the hardware support we had seen dep-
end of 2021 to just over 350 today. recated in CentOS Stream and RHEL With Project ELevate, even if you
Nearly all of those are donated by for AlmaLinux 8.10 and 9.4, which are still running CentOS Linux 6,
organizations in education, science, brought back support for more than 7, or 8, you can upgrade to a sup-
for-profit corporations, or even indi- 140 devices that were needed by our ported operating system without
viduals that understand and believe community. having to migrate your data. And
in our goals. We are seeing nearly with AlmaLinux-deploy you can
one million devices reporting home We will continue to expand and migrate from any existing enterprise
to that mirror system from across fix hardware support, even outside Linux operating system to Alma-
the globe. of upstream if necessary. Recently, Linux in minutes, with just one
we had a report by a community reboot.
We’re also elated to have been chosen member that their storage controller
as the upstream source for at least didn’t work properly even though The ease with which you can get
four operating systems (that we it was supported and recognized by to AlmaLinux, and the speed with
know of). The fact that these oper- the driver. The fix was already avail- which we’re releasing both major
ating systems rely on us is further able, widely tested, and adopted versions and security updates, makes
upstream, but it hadn’t been brought us a very attractive enterprise option.
into RHEL yet. We applied this fix to The fact that we have partnered with
the AlmaLinux kernel to resolve the major trusted brands for those who
issue and now maintain this patch in need enterprise support makes it easy
every kernel update. to convince those who need the life-
line beyond community support.
It’s important to say that you don’t
have to be a big company to impact With community-governance being
AlmaLinux’s development. Anyone a key component of AlmaLinux’s
reporting an obvious and responsibly identity, how can AlmaLinux users
reported concern is enough to make get involved?
us consider fixing it in AlmaLinux.
benny: As an open source project,
What makes AlmaLinux attractive there are tons of ways to get involved,
Andrew Lukoshko has served as to enterprise Linux users? no matter what skills you have or
want to grow. We have special inter-
AlmaLinux OS Architect and Release
benny: Enterprise users have used est groups (SIGs) for everything from
Engineering Lead from day one. He a free version of enterprise Linux creating and managing Cloud images
is also the Core SIG Lead. Andrew for a very long time, and it’s clear to attending events or writing guides,
has 20+ years of Linux experience why. Ten years of support for an from contributing translations of our
and 10+ years of building RHEL OS allows them to lower their website to building out the infrastuc-
migration-related administrative ture used for everything our SIGs do.
source code-based products.
costs without compromising their The quickest way to find your spot is

W W W. A D M I N - M AGA Z I N E .CO M FO CU S O N A L M A L I N UX 21
 FO C U S O N A L M A L I N U X Interview: AlmaLinux

to go to [almalinux.​­org/​­contribute] today (without having full access bug that is causing virtual machine
and see what fits! to the RHEL sources) is more chal- (VM) instability that Red Hat is de-
lenging than when we started. Alma- laying, or a patch for a security flaw
AlmaLinux has hosted two Linux is not just binary compatible that drastically impacts multitenant
AlmaLinux Days, one in Tokyo with RHEL, but it also has to in- systems (of which AlmaLinux has
in December 2023 and another in clude all of the security fixes and an extraordinarily high percent-
Germany in March 2024. What bug fixes that are included in RHEL age), we can now much more reli-
happens at an AlmaLinux Day event? to keep AlmaLinux systems secure ably serve our community. We’re
and stable. always ready to make AlmaLinux
benny: Our two AlmaLinux Days more secure and stable, no matter
have been focused on what the lo- It’s entirely possible, and has proven the reason for the delay or omission
cal community is asking about! The fortuitous to us, that we now recre- upstream.
first one was focused on educat- ate RHEL using publicly accessible
ing the open source community in resources. We find this code primarily In the past year, AlmaLinux auto-
Japan about AlmaLinux and how in UBI images and CentOS Stream mated testing with openQA and
AlmaLinux can be the solution for sources. If needed, we have even Test­infra. Tell us about how Alma-
them. AlmaLinux Day: Germany, pulled from upstream code. In the Linux adapted openQA and Testinfra
on the other hand, was for con- year since Red Hat restricted access to to work with AlmaLinux.
necting with our community in its source code, we still match RHEL’s
Europe and showcasing the ways code up to 99.99 percent. Elkhan Mammadli: openQA is an
that Alma­Linux is being used across amazing open source project that
the world. We typically address An opportunity that has come from we learned about at FOSDEM in
the high-level discussions that are this is the opportunity to deviate 2023, and we knew it was already
relevant to AlmaLinux users at that from RHEL where our community used for testing in Fedora. It was so
time, and then get into the deeper needs it. We are intensely focused close to exactly what we wanted,
subjects that are plaguing users at on meeting the needs of our users, we were excited to be able to con-
that exact moment. and this change has brought about tribute to the project so that we
a new way for us to meet those could adopt it. Building and main-
What prompted AlmaLinux Days? needs. We are now able to deviate taining an RPM for openQA allows
Do you have plans for more of these in very small, but incredibly impor- us to help other operating systems
events in the future? tant, ways from RHEL. As we are no adopt testing best practices more
longer limited to only what Red Hat
benny: Our first AlmaLinux Day last releases, we can now pull in urgent
year was prompted by our users in security patches or add support for
Japan wanting to help spread the more hardware, as our community
word. Cybertrust Japan has been needs. However, we will only do
contributing to AlmaLinux for quite so when the changes we make are
some time, and they saw the need done without negatively impact-
for educating and spreading the ing our users, especially those who
word about AlmaLinux’s existence, chose us specifically as a free RHEL-
and why we are doing the things we compatible operating system. In-
are doing. We are always looking for novations are cool but we will never
ways to connect with our community forget our main goal – making sure
and would like to host an AlmaLinux AlmaLinux is a 100-percent RHEL-
Day in North America this year, in compatible distribution. Elkhan Mammadli, Cloud SIG
addition to another in Japan.
Lead, is a FLOSS enthusiast and
How has moving to binary com-
With AlmaLinux’s move to binary patibility improved AlmaLinux for contributor who maintains the
compatibility with RHEL after Red users? cloud, container, live media, and
Hat restricted access to the RHEL WSL images and automates the
source code, building AlmaLinux Andrew: Now we can bring not only
testing of ISOs and projects such
became more complicated. What extended hardware support but
challenges have you faced in build- also security fixes and bug fixes as ELevate and AlmaLinux-deploy
ing AlmaLinux? ahead of RHEL if our users need it. at the AlmaLinux OS Foundation.
We already have several examples Elkhan works as a Cloud Engineer
Andrew: Indeed, releasing updates of doing so in the first year of our
at CloudLinux.
and even new AlmaLinux versions freedom. Whether it’s a fix for a

22 FOCUS ON ALMALINUX W W W. A D M I N - M AGA Z I N E .CO M

 Interview: AlmaLinux FO C U S O N A L M A L I N U X

easily. Adding support for enter- the speed with which we release benny: AlmaLinux remains hy-
prise Linux virtualization is a boon our updates. We use openQA to perfocused on our users, but we
to the whole ecosystem, too. We ensure the accuracy and strength don’t dictate to the community
also expanded support to include of AlmaLinux OS images, allowing what AlmaLinux will be. A lot of
s390x, AArch64, and ppc64le vir- us to deliver them speedily. The the exciting bits of our future will
tualization, so that we could test time and energy we historically depend on what our users will ask
across all of the architectures that needed for manual testing can be for, and who can or will contrib-
we support. spent on expanding test coverage ute to the project. Whether those
and addressing new test cases as requests come primarily from HPC
For Testinfra, it was as simple as add- they come up. We are getting our users, hobbyists, enterprise users,
ing support for AlmaLinux, as our openQA implementation to a point students, downstream operating
distro wasn’t yet supported. Contrib- where we can welcome feedback systems, or software providers will
uting that to Testinfra was a painless and new tests from our community, shape the future of what AlmaLinux
process, and we continue to be grate- but anyone can already help us is. I hope to see more forks and re-
ful for the incredible work that’s been by reporting bugs and asking for builds of AlmaLinux in the future,
done in that project. new tests to be done. We also use with wider hardware support, out-of-
Testinfra in our non-OS projects the-box support for exotic filesystems,
How does automated testing benefit like testing ELevate scenarios and virtualization components, and data-
AlmaLinux? Do you have plans to testing our Cloud and Container bases. I am also excited to see even
expand openQA and Testinfra test- images. more diverse sponsors joining us, as
ing for AlmaLinux in the future? more companies see the benefit in
What do you see on the horizon for making sure that the software they
Elkhan: Automated testing is critical AlmaLinux in the next three to five depend on has the resources it needs
to the success of AlmaLinux, and years? to stick around. n

W W W. A D M I N - M AGA Z I N E .CO M FO CU S O N A L M A L I N UX 23