Fedora CoreOS Cincinnati SOP
Cincinnati is the update service/backend for Fedora CoreOS (FCOS) machines. This SOP describes how to access and how to troubleshoot it.
Details
- Source
-
https://github.com/coreos/fedora-coreos-cincinnati
- Playbook
- Location
-
OpenShift cluster (production): https://console-openshift-console.apps.ocp.fedoraproject.org/
- Project
-
coreos-cincinnati: https://console-openshift-console.apps.ocp.fedoraproject.org/k8s/cluster/projects/coreos-cincinnati
- Deployment
- Containers
-
-
fcos-graph-builder
(GB - raw updates graph) -
fcos-policy-engine
(PE - frontend handling client requests)
-
- Routes
-
-
coreos-updates-raw
(GB web service) -
coreos-updates-raw-status
(GB status and metrics) -
coreos-updates
(PE web service) -
coreos-updates-status
(PE status and metrics)
-
Upgrades
Building a new version
FCOS Cincinnati is built as container image directly from source, referencing a pinned git commit.
In order to build a new version, you will first have to find the
relevant commit (i.e. the latest on the main
branch) at
https://github.com/coreos/fedora-coreos-cincinnati .
Once you have identified the target commit, these are the steps to build a new container image:
-
update the
fcos_cincinnati_build_git_sha
playbook variable inroles/openshift-apps/coreos-cincinnati/vars/staging.yml
-
update the
fcos_cincinnati_build_git_sha
playbook variable inroles/openshift-apps/coreos-cincinnati/vars/production.yml
-
commit and push the update to the
fedora-infra/ansible
repository -
SSH to
batcave01.iad2.fedoraproject.org
-
run
sudo rbac-playbook openshift-apps/coreos-cincinnati.yml
using your FAS password and your second-factor OTP -
schedule a new build by running
sudo rbac-playbook -t build openshift-apps/coreos-cincinnati.yml
Deploying a new version
Once the target commit has been built into a container image, these are the steps to deploy the new image:
-
update the
fcos_cincinnati_deploy_git_sha
playbook variable inroles/openshift-apps/coreos-cincinnati/vars/staging.yml
-
update the
fcos_cincinnati_deploy_git_sha
playbook variable inroles/openshift-apps/coreos-cincinnati/vars/production.yml
-
commit and push the update to the
fedora-infra/ansible
repository -
SSH to
batcave01.iad2.fedoraproject.org
-
run
sudo rbac-playbook openshift-apps/coreos-cincinnati.yml
using your FAS password and your second-factor OTP
Things that could go wrong
Application build is stuck
Issues in the underlying OpenShift cluster may result in builds being permanently stuck.
If a build does not complete within a reasonable amount of time (i.e. 15 minutes):
-
go to the build overview at https://console-openshift-console.apps.ocp.fedoraproject.org/k8s/ns/coreos-cincinnati/builds
-
click on the build
-
cancel it through the "Cancel Build" button
-
go back to the build overview page
-
schedule a new build through the "Start Build" button
Want to help? Learn how to contribute to Fedora Docs ›