Fedora CI
Fedora Continuous Integration Portal
Why
Vision
There are hundreds of packages which make up the Operating System. Making sure that they all work together as a whole is not an easy task. This becomes even harder as the number of packages and their inter-dependencies grows. An extensive testing is required before a new version of the operating system is released to ensure it is stable enough. That is the past.
Imagine an Always Ready Operating System which consists of packages which are constantly kept in a good shape. Integrated and stable thanks to an extensive test coverage which is continuously executed upon changes in individual packages, in this way allowing to prepare a new release in much shorter time, or even in no time.
Imagine an operating system distribution which you could release at any moment. This is where we are heading. Here comes the CI, Continuous Integration, as an invaluable tool to ensure everything is working together as expected in every point of time.
Manifesto
Continuous integration aims to ensure broken changes are revealed as soon as possible and do not affect other developers, packagers, maintainers or users. The feedback that continuous integration provides is vital for fast paced agile delivery of software. Late testing, long after a change occurs, does not scale to the pace of Fedora. Learn the goals, terminology and rules for a working CI in the manifesto.
How
There are three main pieces of the puzzle to get this nicely working: A process which clearly defines how to discover and execute tests, a set of tools which help to efficiently implement the process and the tests themselves.
Process
Enabling Tests
Tests can be enabled using the Test Metadata Specification implemented by the Test Management Tool which provides a number of features to efficiently work with tests and is now the recommended way. The Standard Test Interface is still supported but has been obsoleted.
-
Test Metadata Specification (recommended)
-
Standard Test Interface (obsoleted)
Gating
When a test fails, CI can prevent the broken change from affecting other packages. That gating happens in Bodhi. Greenwave is used together with ResultsDB and WaiverDB to make the gating decision.
Notifications
Fedora Notifications have been adjusted to notify by default every packager when any step of the CI pipeline fails on one of the package they maintain. So if you are a kernel maintainer and a commit made to the kernel dist-git repository fails to compose an OSTree, FMN will notify you of it.
Bodhi includes the CI results in its update page, just as it already includes tests results from taskotron.
Messages
Various tools involved in the CI automation are sending updates about the progress of the testing to the message bus. Consistent format of the CI messages allows to build simpler services which will provide useful features for everyone.
-
Fedora CI Messages … CI messages specification
-
Datagrepper … search sent messages (e.g. by topic)
-
fedmsg bus … messages sent by the pipeline
-
topics … list of fedmsg topics related to CI
Tools
Test Management Tool
The tmt
tool provides a user-friendly way to work with tests.
You can comfortably create new tests, safely and easily run tests across different environments,
review test results, debug test code and enable tests in the CI using a consistent and concise config.
Standard Test Roles
Standard Test Roles were implemented to enable both automation tools and developers in their local environments to easily execute tests. This set of ansible roles supports various frameworks and allows to execute tests against different test subjects (such as classic rpm package, docker container or Atomic Host). Standard Test Roles have been obsoleted by the Test Management Tool.
Frameworks
As Standard Test Interface does not define test framework which should be used for testing it is possible to choose the most suitable framework for your project. Here are some examples:
Pipeline
The testing Pipeline detects tests for enabled packages, executes the test coverage and gathers the results.
Tests
The core of the CI success are reliable tests of a good quality, well selected, stable, organized and continuously maintained.
Test Types
In general it makes sense to store tests as close to the upstream as possible. So what are the appropriate test types recommended for testing the Always Ready Operating System?
-
Basic functionality tests
-
Integration tests
For unit tests it usually makes more sense to store them directly within the upstream project repository. However, in some cases it might be worth to fetch tests for Fedora CI from the upstream repository as well.
Test Code
Test code can be stored directly in the dist-git or fetched from another repository. The minimal config to enable a simple smoke test looks like this:
execute: script: foo --version
Tests from a shared repository can be enabled in this way:
discover: how: fmf url: https://src.fedoraproject.org/tests/shell execute: how: tmt
See the Test Management Tool documentation for more details and examples.
Test Execution
Executing tests is as simple as running a single command:
tmt run
Learn more in the Execute Tests section.
Shared Responsibility
The ownership and maintenance of tests should be shared between QE & Devel. For tests that reside in the rpm namespace, QE can use pull requests to create/update tests. Likewise in the tests namespace, both QE and Devel will have commit rights, both QE and Devel should review and sign-off with each commit.
More
Contact
If you have questions or would like to get involved:
-
Fedora Chat: Fedora CI room
-
Mailing list: [email protected] (archive)
Links
Here are some additional related links:
-
Test Management Tool … Fedora CI docs for tmt
-
tmt Guide … upstream tmt guide
-
Standard Test Interface … definition of the process
-
Standard Test Roles … set of ansible roles
-
Tests … executing and adding tests
-
CI SIG … Continuous Integration Special Interest Group
Want to help? Learn how to contribute to Fedora Docs ›