Red Hat OpenShift Container

Platform (RHOCP) with RHEL KVM

on IBM Z and LinuxONE
Version: 4.7
April 21, 2021
–
Holger Wolf – [email protected]
Product Owner – Red Hat OpenShift Container Platform on IBM Z and LinuxONE

Dr. Wolfgang Voesch – [email protected]

Iteration Manager – Red Hat OpenShift Container Platform on IBM Z and LinuxONE

Installing Red Hat OpenShift Container Platform with RHEL 8.3 KVM on IBM Z and LinuxONE / © 2021 IBM Corporation
Financing Available: IBM Global Financing provides numerous payment options to help you
acquire the technology you need to grow your business. For more information, visit:
ibm.com/financing.

© Copyright IBM Corporation 2021.

IBM Systems, New Orchard Road Armonk, NY 10504.

IBM, the IBM logo, and ibm.com are trademarks of International Business Machines Corp., registered in
many jurisdictions worldwide. Other product and service names might be trademarks of IBM or other
companies. A current list of IBM trademarks is available on the web at “Copyright and trademark
information” at ibm.com/legal/copytrade.shtml. This document is current as of the initial date of
publication and may be changed by IBM at any time. Not all offerings are available in every country in
which IBM operates. The performance data and client examples cited are presented for illustrative
purposes only. Actual performance results may vary depending on specific configurations and
operating conditions.

THE INFORMATION IN THIS DOCUMENT IS PROVIDED “AS IS” WITHOUT ANY WARRANTY, EXPRESS
OR IMPLIED, INCLUDING WITHOUT ANY WARRANTIES OF MERCHANTABILITY, FITNESS FOR A
PARTICULAR PURPOSE AND ANY WARRANTY OR CONDITION OF NON-INFRINGEMENT.

IBM products are warranted according to the terms and conditions of the agreements under which they
are provided. The client is responsible for ensuring compliance with laws and regulations applicable to
it. IBM does not provide legal advice or represent or warrant that its services or products will ensure
that the client is in compliance with any law or regulation. Statements regarding IBM’s future direction
and intent are subject to change or withdrawal without notice, and represent goals and objectives only.
Outline

1. Red Hat OpenShift Container Platform (RHOCP) v4.7

a) Overview
b) RHOCP on RHEL KVM

2. System requirements KVM

3. Install process RHOCP on RHEL KVM
a) Prerequisites
b) Cluster installation
c) Post cluster installation
4. Configuring the KVM environment for larger clusters

5. Questions

Installing Red Hat OpenShift Container Platform with RHEL 8.3 KVM on IBM Z and LinuxONE / © 2021 IBM Corporation 3
Where can you download RHOCP?

try.openshift.com
cloud.redhat.com

RHOCP 4.7 on Z was released on 02/24/2021 – Full Support

RHOCP 4.6 on Z was released on 10/27/2020 – Extended Full Support
RHOCP 4.5 on Z was released on 07/30/2020 – Maintenance Support
RHOCP 4.4 on Z was released on 06/22/2020 – Out of Support
RHOCP 4.3 on Z was released on 04/30/2020 – Out of Support
RHOCP 4.2 on Z was released on 02/11/2020 – Out of Support
Installation instructions for RedHat OpenShift on KVM & zVM for Z and LinuxONE:
https://docs.openshift.com/container-platform/4.7/installing/installing_ibm_z/installing-ibm-z-kvm.html
https://docs.openshift.com/container-platform/4.7/installing/installing_ibm_z/installing-ibm-z.html

Release Notes:
https://docs.openshift.com/container-platform/4.7/release_notes/ocp-4-7-release-notes.html
Installing Red Hat OpenShift Container Platform with RHEL 8.3 KVM on IBM Z and LinuxONE / © 2021 IBM Corporation 4
What‘s new in RHOCP 4.7?

Platform Support Storage Options Networking

• New hypervisor support RHEL • Emulated Devices (EDEVICES) • Support for OVN
8.3 KVM: (z/VM) Kubernetes
Ø Install RHOCP as virtual Ø Allows to install RHOCP on EDEV in Ø As a second default CNI
machines z/VM and attach EDEV as PV provider

Ø Support for connected and • Multipath Support (only direct

disconnected install attached SCSI)
Ø Only supported with a fresh
installatioin

Installing Red Hat OpenShift Container Platform with RHEL 8.3 KVM on IBM Z and LinuxONE / © 2021 IBM Corporation 5
 Overview on RHOCP on RHEL
8.3 KVM
Storage and Network minimal
setup?

Installing Red Hat OpenShift Container Platform with RHEL 8.3 KVM on IBM Z and LinuxONE / © 2021 IBM Corporation 6
RHOCP – A simplified cluster overview
RHOCP Cluster

RHOCP RHOCP RHOCP

Data
RHOCP RHOCP
Control Control Control Compute Compute Warehouse
Plane Plane Plane Node Node

CoreOS CoreOS CoreOS CoreOS CoreOS

Hypervisor
LPAR 1 LPAR 2

IBM Z or IBM LinuxONE

Cores (IFLs) Network (OSA, RoCE) I/O (FICON / FCP)

Shared
FCP/SCSI ECKD/DASD
FS
Installing Red Hat OpenShift Container Platform with RHEL 8.3 KVM on IBM Z and LinuxONE / © 2021 IBM Corporation 7
http://public.dhe.ibm.com/software/dw/linux390/docu/RHOCP-reference-architecture.pdf
 Storage Option

Host Storage RHOCP SDN

Contains host environment and
hosting qcow2 images for the
guests
OCP OCP OCP OCP OCP
Could be a large enough Control Control Control Compute Compute
FCP/DASD or LVM

Guest Storage
qcow2 files to host the guest
storage resides on host and
attached to guest in guest
definition XML
QCOW2 QCOW2
QCOW2
LVM
Hypervisor KVM QCOW2
QCOW2

LPAR
FCP/SCSI ECKD/DASD
Installing Red Hat OpenShift Container Platform with RHEL 8.3 KVM on IBM Z and LinuxONE / © 2021 IBM Corporation 8
http://public.dhe.ibm.com/software/dw/linux390/docu/RHOCP-reference-architecture.pdf
 Network topology for HA with OSA and Linux Bonding

Benefits: RHOCP SDN

Using Linux Bonding , the network
HA and enhanced bandwidth can be
achieved
OCP OCP OCP OCP OCP
Guests are connected via MacVTap Control Control Control Compute Compute
directly to the bond device

Reference: Virt Bond Virt Bond

Virt Bond Virt Bond Virt Bond
http://www.redbooks.ibm.com/redbooks/pdf
s/sg248463.pdf
Virtualization Cookbook for IBM Z Volume 5:
KVM page 92

https://www.ibm.com/docs/de/linux-on-
systems?topic=devices-direct-interfaces BOND
Hypervisor KVM

LPAR
OSA/ RoCE OSA/ RoCE
Installing Red Hat OpenShift Container Platform with RHEL 8.3 KVM on IBM Z and LinuxONE / © 2021 IBM Corporation 9
http://public.dhe.ibm.com/software/dw/linux390/docu/RHOCP-reference-architecture.pdf
 Network topology for HA with Linux Bridge

Benefits: RHOCP SDN

Bridge can work as an
isolated network including
firewall rules
OCP OCP OCP OCP OCP
Higher complexity == more Control Control Control Compute Compute
freedom

Virt Nic Virt Nic Virt Nic Virt Nic Virt Nic

BOND Bridge
Hypervisor KVM

LPAR
OSA/ RoCE OSA/ RoCE
Installing Red Hat OpenShift Container Platform with RHEL 8.3 KVM on IBM Z and LinuxONE / © 2021 IBM Corporation 10
http://public.dhe.ibm.com/software/dw/linux390/docu/RHOCP-reference-architecture.pdf
 What are the
hardware and software
requirements?

Installing Red Hat OpenShift Container Platform with RHEL 8.3 KVM on IBM Z and LinuxONE / © 2021 IBM Corporation 11
System Requirements KVM
Area Minimum Preferred
LPAR/Processor Options 1 LPAR – 3 IFLs SMT enabled 3 LPARs – 6 IFLs SMT enabled

LPAR Network Options 1 RoCE or OSA 2 RoCE or OSA

Hypervisor RHEL 8.3+ KVM – 1 Hypervisor LPAR RHEL 8.3+ KVM – 3 Hypervisor LPARs
3 Control Plane Nodes 3 Contol Plane Nodes (1 per instance)
2 Compute Nodes 3 Infra node
1 Temp Bootstrap System 6 Compute Nodes
1 Temp Bootstrap System
Memory 16 GB for Control Plane Nodes 16 GB+ for Control Plane Nodes
8 GB for Compute Nodes 8 GB+ for Compute Nodes
16 GB for Bootstrap Node (temporary) 8 GB+ for Infra Nodes
16 GB for Bootstrap Node (temporary)
Disk Attachement Hypervisor 120 GB per Node -> min. 840 GB 120 GB per Node -> 840 GB per hypervisor for
DASD / FCP scaling
Disk Attachment Node qcow2 qcow2

Network configuration direct OSA, RoCE 2 OSA/RoCE as bonded devices

Shared Storage (NFSv4) 100 GB 100 GB

Installing Red Hat OpenShift Container Platform with RHEL 8.3 KVM on IBM Z and LinuxONE / © 2021 IBM Corporation 12
Preferred IBM Z System Requirements

DNS
Compute
LPAR
Load Balancer Control
Compute RHEL 8.3 - KVM

DHCP
Compute
HTTP/S LPAR
Bootstrap Control
Compute
RHEL 8.3 - KVM
Gateway

Compute
Install VM / LPAR
KVM Host Control RHEL 8.3 - KVM
Compute

Install Control Compute

Plane Plane Plane

Red Hat OpenShift Container Platform Cluster

https://docs.openshift.com/container-platform/4.7/installing/installing_ibm_z/installing-ibm-z-kvm.html
Installing Red Hat OpenShift Container Platform with RHEL 8.3 KVM on IBM Z and LinuxONE / © 2021 IBM Corporation 13
Minimum IBM Z System Requirements

DNS

Load Balancer Control

DHCP Compute

HTTP/S LPAR
Bootstrap Control RHEL 8.3 - KVM
Gateway
Compute

Install VM /
KVM Host Control

Install Control Compute

Plane Plane Plane

Red Hat OpenShift Container Platform Cluster

https://docs.openshift.com/container-platform/4.7/installing/installing_ibm_z/installing-ibm-z-kvm.html
Installing Red Hat OpenShift Container Platform with RHEL 8.3 KVM on IBM Z and LinuxONE / © 2021 IBM Corporation 14
Install Virtual Machine (VM) options

DNS

Load Balancer Control

DHCP Compute
Virtual Machine LPAR
HTTP/S LPAR RHEL 8.3 z/VM
Bootstrap Control RHEL 8.3 - KVM
Gateway
Compute

Install VM /
KVM Host Control 90x 4
3
s /x 6
x86
Install virtual machine (VM) Install Control Compute
• Runs: Plane Plane Plane
openshift-install get from https://mirror.openshift.com/pub/openshift-v4/s390x/clients/ocp/latest/
• RHEL 8.3 KVM host (s390x) Red Hat OpenShift Container Platform Cluster
• Virtual machine RHEL 8.3 on RHEL 8.3 KVM (s390x) Mac
• Virtual machine RHEL 8.3 on z/VM (s390x)
• Mac (x64)
https://docs.openshift.com/container-platform/4.7/installing/installing_ibm_z/installing-ibm-z-kvm.html
Installing Red Hat OpenShift Container Platform with RHEL 8.3 KVM on IBM Z and LinuxONE / © 2021 IBM Corporation 15
Proof of Concept IBM Z System Requirements

DNS

Load Balancer Control

DHCP Compute

HTTP/S LPAR
Bootstrap Control RHEL 8.3 - KVM
Gateway
Compute

Install VM /
KVM Host Control

Install Control Compute

Bastion VM Plane Plane Plane

Red Hat OpenShift Container Platform Cluster

UPI with Bastion VM:

• multiarch-ci-playbooks: not officially supported
https://github.com/multi-arch/multiarch-ci-playbooks
Installing Red Hat OpenShift Container Platform with RHEL 8.3 KVM on IBM Z and LinuxONE / © 2021 IBM Corporation 16
 Install Sequence

DNS

Load Balancer Control

Compute
prepare DHCP prepare prepare install Bootstrap Control create configure
prereqs. install VMs VMs Compute cluster cluster
HTTP/S Control
Install Control Compute
Gateway Plane Plane Plane
Red Hat OpenShift Container Platform Cluster
Install VM /
KVM Host

Red Had OpenShift Container Platform Installation Process

Pre-requisite: installed RHEL 8.3+ KVM

Installing Red Hat OpenShift Container Platform with RHEL 8.3 KVM on IBM Z and LinuxONE / © 2021 IBM Corporation 17
 Step 0 of 14: Install Options – Considerations (1 of 4)
DNS
Load Balancer prepare install Control
VMs VMs Compute
prepare DHCP prepare Bootstrap Control create configure
prereqs. HTTP/S install Compute cluster cluster
prepare install Control
Gateway VMs VMs
Install Control Compute
Install VM / Plane Plane Plane
KVM Host

“Fast track installation” “Full installation”

• With a prepacked RHCOS QCOW2 disk image. • Of RHCOS on a QCOW2 disk image.
• Advantage • Advantage
• Fast installation • Full installation with more options (slower)
• Less files • allows static IPs
• Requires: • Requires:
• DHCP • HTTP/S server
• Not required: • Optional / not required (depending on setup) :
• HTTP/S server • DHCP
https://docs.openshift.com/container-platform/4.7/installing/installing_ibm_z/installing-ibm-z-kvm.html#installation-user-infra-machines-iso-ibm-z_kvm_installing-ibm-z-kvm
https://github.com/openshift/openshift-docs/pull/31704
Installing Red Hat OpenShift Container Platform with RHEL 8.3 KVM on IBM Z and LinuxONE / © 2021 IBM Corporation 18
 Step 0 of 14: Preparing Prerequisites (2 of 4)
DNS
Load Balancer prepare install Control
VMs VMs Compute
prepare DHCP prepare Bootstrap Control create configure
prereqs. HTTP/S install Compute cluster cluster
prepare install Control
Gateway VMs VMs
Install Control Compute
Install VM / Plane Plane Plane
KVM Host

0. UPI – User Provisioned Infrastructure:

Create an environment overview with all HW

1. Provision the required load balancers.

2. Configure the ports for your machines.
3. Configure DNS.
4. Ensure network connectivity.
5. Option: configure HTTP/S server
6. Option: configure DHCP
7. Download images (next slide)
https://docs.openshift.com/container-platform/4.7/installing/installing_ibm_z/installing-ibm-z-kvm.html#installation-infrastructure-user-infra_installing-ibm-z-kvm
Installing Red Hat OpenShift Container Platform with RHEL 8.3 KVM on IBM Z and LinuxONE / © 2021 IBM Corporation 19
 Step 0 of 14: Install Binaries (3 of 4)
DNS
Load Balancer prepare install Control
VMs VMs Compute
prepare DHCP prepare Bootstrap Control create configure
prereqs. HTTP/S install Compute cluster cluster
prepare install Control
Gateway VMs VMs
Install Control Compute
Install VM / Plane Plane Plane
KVM Host

“Fast track installation”: “Full installation”:

Install: Install:
openshift-install-<os>-4.7.z.tar.gz openshift-install-<os>-4.7.z.tar.gz
openshift-client-<os>-4.7.z.tar.gz openshift-client-<os>-4.7.z.tar.gz

Images: Images:
rhcos-4.7.z-s390x-qemu.s390x.qcow2.gz rhcos-4.y.z-s390x-live-initramfs.s390x.img
rhcos-4.y.z-s390x-live-kernel-s390x
rhcos-4.y.z-s390x-live-rootfs.s390x.img

https://mirror.openshift.com/pub/openshift-v4/s390x/clients/ocp/latest/
https://mirror.openshift.com/pub/openshift-v4/s390x/dependencies/rhcos/latest/latest/
Installing Red Hat OpenShift Container Platform with RHEL 8.3 KVM on IBM Z and LinuxONE / © 2021 IBM Corporation 20
 Step 0 of 14: Side Note (4 of 4)
DNS
Load Balancer prepare install Control
VMs VMs Compute
prepare DHCP prepare Bootstrap Control create configure
prereqs. HTTP/S install Compute cluster cluster
prepare install Control
Gateway VMs VMs
Install Control Compute
Install VM / Plane Plane Plane
KVM Host

Using the multiarch-ci-playbooks: use as is, not

production ready
• Ensure correct configuration in
groupvars/all.yml for devices, network, disks,
etc.
• IP-addresses have to be in subnet CIDR /24

• Delete residuals from possible previous installs:

• dir: ocp4-workdir
• binary: openshift-install, oc client

https://github.com/multi-arch/multiarch-ci-playbooks
Installing Red Hat OpenShift Container Platform with RHEL 8.3 KVM on IBM Z and LinuxONE / © 2021 IBM Corporation 21
 Step 1 of 14: Preparing the Cluster Installation (1 of 3)
DNS
Load Balancer prepare install Control
VMs VMs Compute
prepare DHCP prepare Bootstrap Control create configure
prereqs. HTTP/S install Compute cluster cluster
prepare install Control
Gateway VMs VMs
Install Control Compute
Install VM / Plane Plane Plane
KVM Host

Files on the involved machines before execution of openshift-install

• Install VM
• Binaries: oc client & openshift-install
• File: install-config.yaml

https://docs.openshift.com/container-platform/4.7/installing/installing_ibm_z/installing-ibm-z-kvm.html#installation-bare-metal-config-yaml_installing-ibm-z-kvm
Installing Red Hat OpenShift Container Platform with RHEL 8.3 KVM on IBM Z and LinuxONE / © 2021 IBM Corporation 22
 Step 1 of 14: Preparing the Cluster Installation (2 of 3)
DNS
Load Balancer prepare install Control
VMs VMs Compute
prepare DHCP prepare Bootstrap Control create configure
prereqs. HTTP/S install Compute cluster cluster
prepare install Control
Gateway VMs VMs
Install Control Compute
Install VM / Plane Plane Plane
KVM Host

On the Install VM Using the multiarch-ci-playbooks: use as is

• Follow the installation steps to create manifests • No cluster-scheduler configuration
• Modify cluster-scheduler
oc patch \
• Follow the installation steps to create ignition schedulers.config.openshift.io/cluster \
• Execution of openshift-install --type merge --patch \
• Important times for cluster creation: '{"spec":{"mastersSchedulable": false}}'
• Cluster has to be created within 24 hours after
ignition file creation

https://docs.openshift.com/container-platform/4.7/installing/installing_ibm_z/installing-ibm-z-kvm.html#installation-initializing-manual_installing-ibm-z-kvm
https://docs.openshift.com/container-platform/4.7/installing/installing_ibm_z/installing-ibm-z-kvm.html#installation-user-infra-generate-k8s-manifest-ignition_installing-ibm-z-kvm
https://docs.openshift.com/container-platform/4.7/backup_and_restore/disaster_recovery/scenario-3-expired-certs.html#dr-recovering-expired-certs
Installing Red Hat OpenShift Container Platform with RHEL 8.3 KVM on IBM Z and LinuxONE / © 2021 IBM Corporation 23
 Step 1 of 14: Preparing the Cluster Installation (3 of 3)
DNS
Load Balancer prepare install Control
VMs VMs Compute
prepare DHCP prepare Bootstrap Control create configure
prereqs. HTTP/S install Compute cluster cluster
prepare install Control
Gateway VMs VMs
Install Control Compute
Install VM / Plane Plane Plane
KVM Host

Files on the involved machines after execution of openshift-install

• Install VM (can be the same as the KVM host)
• Binaries: oc client & openshift-install
• File: install-config.yaml
• From install-config.yaml, openshift-install generated:
• bootstrap.ign
• master.ign
• worker.ign

Installing Red Hat OpenShift Container Platform with RHEL 8.3 KVM on IBM Z and LinuxONE / © 2021 IBM Corporation 24
 Step 2a of 14: Preparing the KVM Guests – Fast Track Installation (1 of 2)
DNS
Load Balancer prepare install Control
VMs VMs Compute
prepare DHCP prepare Bootstrap Control create configure
prereqs. HTTP/S install Compute cluster cluster
prepare install Control
Gateway VMs VMs
Install Control Compute
Install VM / Plane Plane Plane
KVM Host

Files on the involved machines before KVM guest creation

a) Fast track installation
• KVM host(s):
• Image: qcow2
• Ignition files: bootstrap.ign, master.ign, worker.ign
b) Full installation
• KVM host(s):
• Images: -
• HTTP/S server:
• Images: initramfs, kernel, rootfs
• Ignition files: bootstrap.ign, master.ign, worker.ign
Installing Red Hat OpenShift Container Platform with RHEL 8.3 KVM on IBM Z and LinuxONE / © 2021 IBM Corporation 25
 Step 2a of 14: Preparing the KVM Guests – Fast Track Installation (2 of 2)
DNS
Load Balancer prepare install Control
VMs VMs Compute
prepare DHCP prepare Bootstrap Control create configure
prereqs. HTTP/S install Compute cluster cluster
prepare install Control
Gateway VMs VMs
Install Control Compute
Install VM / Plane Plane Plane
KVM Host

On the KVM hypervisor

• Check that the disk has enough free space for the KVM guests
• Create the the disk for the KVM VMs
qemu-img create -f qcow2 -F qcow2 -b \
/{fully-qualified-path}/{rhcos-qemu}.qcow2 \
/{fully-qualified-path}/{vm-name}.qcow2 {disk-size}
• Example for the bootstrap:
qemu-img create -f qcow2 -F qcow2 –b \
/var/lib/libvirt/images/rhcos-4.7.0-s390x-qemu.s390x.qcow2 \
/var/lib/libvirt/images/bootstrap-0.qcow2 120G

https://docs.openshift.com/container-platform/4.7/installing/installing_ibm_z/installing-ibm-z-kvm.html#installation-user-infra-machines-iso-ibm-z_kvm_installing-ibm-z-kvm
Installing Red Hat OpenShift Container Platform with RHEL 8.3 KVM on IBM Z and LinuxONE / © 2021 IBM Corporation 26
 Step 3a of 14: Installation of KVM Guests – Fast Track Installation (1 of 2)
DNS
Load Balancer prepare install Control
VMs VMs Compute
prepare DHCP prepare Bootstrap Control create configure
prereqs. HTTP/S install Compute cluster cluster
prepare install Control
Gateway VMs VMs
Install Control Compute
Install VM / Plane Plane Plane
KVM Host

Install your KVM guests

virt-install --noautoconsole \
--connect qemu:///system \
--name {vn-name} \
--memory {memory} \
--vcpus {vcpus} \
--disk {disk} \
--import \
--network network={network},mac={mac} \
--qemu-commandline="-drive \
if=none,id=ignition,format=raw,file={ign-file},readonly=on \
-device virtio-blk,serial=ignition,drive=ignition"
https://docs.openshift.com/container-platform/4.7/installing/installing_ibm_z/installing-ibm-z-kvm.html#installation-user-infra-machines-iso-ibm-z_kvm_installing-ibm-z-kvm
Installing Red Hat OpenShift Container Platform with RHEL 8.3 KVM on IBM Z and LinuxONE / © 2021 IBM Corporation 27
 Step 3a of 14: Installation of KVM Guests – Fast Track Installation (2 of 2)
DNS
Load Balancer prepare install Control
VMs VMs Compute
prepare DHCP prepare Bootstrap Control create configure
prereqs. HTTP/S install Compute cluster cluster
prepare install Control
Gateway VMs VMs
Install Control Compute
Install VM / Plane Plane Plane
KVM Host

Install your KVM guests – example for the bootstrap

virt-install --noautoconsole \
--connect qemu:///system \
--name bootstrap-0 \
--memory 16384 \
--vcpus 4 \
--disk /var/lib/libvirt/images/bootstrap-0.qcow2 \
--import \
--network network=macvtap,mac=52:54:00:d0:f7:02 \
--qemu-commandline="-drive \
if=none,id=ignition,format=raw,file=bootstrap.ign,readonly=on \
-device virtio-blk,serial=ignition,drive=ignition"

Installing Red Hat OpenShift Container Platform with RHEL 8.3 KVM on IBM Z and LinuxONE / © 2021 IBM Corporation 28
 Step 2b of 14: Preparing the KVM Guests – Full Installation
DNS
Load Balancer prepare install Control
VMs VMs Compute
prepare DHCP prepare Bootstrap Control create configure
prereqs. HTTP/S install Compute cluster cluster
prepare install Control
Gateway VMs VMs
Install Control Compute
Install VM / Plane Plane Plane
KVM Host

Files on the involved machines before KVM guest creation

a) Fast track installation
• KVM host(s):
• Image: qcow2
• Ignition files: bootstrap.ign, master.ign, worker.ign
b) Full installation
• KVM host(s):
• Images: -
• HTTP/S server:
• Images: initramfs, kernel, rootfs
• Ignition files: bootstrap.ign, master.ign, worker.ign
Installing Red Hat OpenShift Container Platform with RHEL 8.3 KVM on IBM Z and LinuxONE / © 2021 IBM Corporation 29
 Step 3b of 14: Installation of KVM Guests – Full Installation (1 of 2)
DNS
Load Balancer prepare install Control
VMs VMs Compute
prepare DHCP prepare Bootstrap Control create configure
prereqs. HTTP/S install Compute cluster cluster
prepare install Control
Gateway VMs VMs
Install Control Compute
Install VM / Plane Plane Plane
KVM Host

Install your KVM guests

virt-install --noautoconsole \
--connect qemu:///system --name {vn-name} --memory {memory} --vcpus {vcpus} \
--disk /{fully-qualified-path}/{vn-name}.qcow2,size={size} \
--network network={network-name} --boot hd \
--location {http/s},kernel=rhcos-live-kernel-s390x,initrd=rhcos-live-initramfs.s390x.img \
--extra-args "rd.neednet=1 coreos.inst.install_dev=/dev/vda \
coreos.live.rootfs_url={http/s}/rhcos-live-rootfs.s390x.img \
coreos.inst.ignition_url={http/s}/{ign-file} \
ip={vm-ip}::{nameserver-ip}:{subnet-mask}:::none nameserver={nameserver-ip}" \
--wait
https://docs.openshift.com/container-platform/4.7/installing/installing_ibm_z/installing-ibm-z-kvm.html#installation-user-infra-machines-iso-ibm-z_kvm_installing-ibm-z-kvm
https://github.com/openshift/openshift-docs/pull/31704
Installing Red Hat OpenShift Container Platform with RHEL 8.3 KVM on IBM Z and LinuxONE / © 2021 IBM Corporation 30
 Step 3b of 14: Installation of KVM Guests – Full Installation (2 of 2)
DNS
Load Balancer prepare install Control
VMs VMs Compute
prepare DHCP prepare Bootstrap Control create configure
prereqs. HTTP/S install Compute cluster cluster
prepare install Control
Gateway VMs VMs
Install Control Compute
Install VM / Plane Plane Plane
KVM Host

Install your KVM guests – Example for the bootstrap

virt-install --noautoconsole \
--connect qemu:///system --name bootstrap-0 --memory 16384 –vcpus 4 \
--disk /var/lib/libvirt/images/bootstrap-0.qcow2,size=120G \
--network network=macvtap-mv1 --boot hd \
--location http://example.com:8080/assets/,kernel=rhcos-live-kernel-s390x,initrd=rhcos-
live-initramfs.s390x.img \
--extra-args "rd.neednet=1 coreos.inst.install_dev=/dev/vda \
coreos.live.rootfs_url=http://example.com:8080/assets/rhcos-live-rootfs.s390x.img \
coreos.inst.ignition_url=http://example.com:8080/ignition/bootstrap.ign \
ip=172.18.100.1::172.18.0.1:255.255.255.0:::none nameserver=172.18.0.1" \
--wait

Installing Red Hat OpenShift Container Platform with RHEL 8.3 KVM on IBM Z and LinuxONE / © 2021 IBM Corporation 31
 Step 4 of 14: Installation of KVM Guests
DNS
Load Balancer prepare install Control
VMs VMs Compute
prepare DHCP prepare Bootstrap Control create configure
prereqs. HTTP/S install Compute cluster cluster
prepare install Control
Gateway VMs VMs
Install Control Compute
Install VM / Plane Plane Plane
KVM Host

Start the KVM guests In case of error:

• Start at least bootstrap and all control plane 1. Read carefully
nodes with short delay only 2. Emergency shell, search/check for:
• Monitor the progress with • CPUs, memory
virsh console {vm-name} • Internet connectivity
• Working name resolution
Note: • Network connectivity between nodes
• Currently on IBM z15 and LinuxONE III the
Using the multiarch-ci-playbooks: use as is
kernel option dfltcc=off is required.
• Wait until playbook step:
[ Wait for bootstrap accessibility ]
Installing Red Hat OpenShift Container Platform with RHEL 8.3 KVM on IBM Z and LinuxONE / © 2021 IBM Corporation
• Timed out: no problem 32
 Step 5 of 14: Monitoring the Bootstrap
DNS
Load Balancer prepare install Control
VMs VMs Compute
prepare DHCP prepare Bootstrap Control create configure
prereqs. HTTP/S install Compute cluster cluster
prepare install Control
Gateway VMs VMs
Install Control Compute
Install VM / Plane Plane Plane
KVM Host

From install VM connect to bootstrap: In case of error:

• Command is presented: • Access to quay.io
journalctl -b -f -u \ • Pull Secret
release-image.service -u bootkube.service • Network connectivity
Monitor: • Wrong image format
• Expect to see many errors and failures • Wrong installer, e.g. OCP on X86
• As long as errors are counted, it is still fine.
• Wait for: etcd cluster formed

Installing Red Hat OpenShift Container Platform with RHEL 8.3 KVM on IBM Z and LinuxONE / © 2021 IBM Corporation 33
 Step 6 of 14: Ignition of Control Plane Nodes
DNS
Load Balancer prepare install Control
VMs VMs Compute
prepare DHCP prepare Bootstrap Control create configure
prereqs. HTTP/S install Compute cluster cluster
prepare install Control
Gateway VMs VMs
Install Control Compute
Install VM / Plane Plane Plane
KVM Host

• Bootstrap: one node cluster Start job for ignition

All three control plane nodes need to be active
[ 13.686667] ignition[715]: GET https://api-int.<name>.<domain>:22623/config/master: attempt #6
• [ ***] A start job is running for Ignition (fetch) (29s / no limit)[ 33.688872] ignition[715]: GET error: Get "https://api-
int.<name>.<domain>:22623/config/master": dial tcp: lookup api-int.<name>.<domain> on 192.168.79.1:53: read udp

simultaneously
192.168.79.0:50590->192.168.79.1:53: i/o timeout
[* ] A start job is running for Ignition (fetch) (34s / no limit)[ 38.689441] ignition[715]: GET https://api-
int.<name>.<domain>:22623/config/master: attempt #7
[*** ] A start job is running for Ignition (fetch) (54s / no limit)[ 58.690750] ignition[715]: GET error: Get "https://api-
• Do not reboot a control plane node int.<name>.<domain>:22623/config/master": dial tcp: lookup api-int.<name>.<domain> on 192.168.79.1:53: read udp
192.168.79.0:48059->192.168.79.1:53: i/o timeout
[ **] A start job is running for Ignition (fetch) (59s / no limit)[ 63.691024] ignition[715]: GET https://api-
int.<name>.<domain>:22623/config/master: attempt #8
[ **] A start job is running for Ignition (fetch) (1min 19s / no limit)[ 83.692592] ignition[715]: GET error: Get "https://api-

Error: start job for ignition while bootstrap is up:

int.<name>.<domain>:22623/config/master": dial tcp: lookup api-int.<name>.<domain> on 192.168.79.1:53: read udp
• 192.168.79.0:58831->192.168.79.1:53: i/o timeout
[*** ] A start job is running for Ignition (fetch) (1min 24s / no limit)[ 88.692908] ignition[715]: GET https://api-

• Network connectivity
int.<name>.<domain>:22623/config/master: attempt #9
[* ] A start job is running for Ignition (fetch) (1min 44s / no limit)[ 108.694466] ignition[715]: GET error: Get "https://api-
int.<name>.<domain>:22623/config/master": dial tcp: lookup api-int.<name>.<domain> on 192.168.79.1:53: read udp
192.168.79.0:39987->192.168.79.1:53: i/o timeout
[ *** ] A start job is running for Ignition (fetch) (1min 49s / no limit)[ 113.695028] ignition[715]: GET https://api-
int.<name>.<domain>:22623/config/master: attempt #10
[ **] A start job is running for Ignition (fetch) (2min 9s / no limit)[ 133.697541] ignition[715]: GET error: Get "https://api-
int.<name>.<domain>:22623/config/master": dial tcp: lookup api-int.<name>.<domain> on 192.168.79.1:53: read udp
192.168.79.0:34470->192.168.79.1:53: i/o timeout
[ *** ] A start job is running for Ignition (fetch) (2min 14s / no limit)[ 138.697372] ignition[715]: GET https://api-
int.<name>.<domain>:22623/config/master: attempt #11
[*** ] A start job is running for Ignition (fetch) (2min 34s / no limit)[ 158.698748] ignition[715]: GET error: Get "https://api-
int.<name>.<domain>:22623/config/master": dial tcp: lookup api-int.<name>.<domain> on 192.168.79.1:53: read udp
192.168.79.0:60220->192.168.79.1:53: i/o timeout
[*** ] A start job is running for Ignition (fetch) (2min 39s / no limit)[ 163.699509] ignition[715]: GET https://api-
Installing Red Hat OpenShift Container Platform with RHEL 8.3 KVM on IBM Z and LinuxONE / © 2021 IBM Corporation int.<name>.<domain>:22623/config/master: attempt #12 34
 Step 7 of 14: Ignition of Compute Nodes
DNS
Load Balancer prepare install Control
VMs VMs Compute
prepare DHCP prepare Bootstrap Control create configure
prereqs. HTTP/S install Compute cluster cluster
prepare install Control
Gateway VMs VMs
Install Control Compute
Install VM / Plane Plane Plane
KVM Host

• Compute nodes: wait until control plane nodes are up Start job for ignition
[ 13.686667] ignition[715]: GET https://api-int.<name>.<domain>:22623/config/worker: attempt #6
[ ***] A start job is running for Ignition (fetch) (29s / no limit)[ 33.688872] ignition[715]: GET error: Get "https://api-
int.<name>.<domain>:22623/config/worker": dial tcp: lookup api-int.<name>.<domain> on 192.168.79.1:53: read udp

Error: start job for ignition while bootstrap is up:

192.168.79.0:50590->192.168.79.1:53: i/o timeout
• [* ] A start job is running for Ignition (fetch) (34s / no limit)[ 38.689441] ignition[715]: GET https://api-
int.<name>.<domain>:22623/config/worker: attempt #7
[*** ] A start job is running for Ignition (fetch) (54s / no limit)[ 58.690750] ignition[715]: GET error: Get "https://api-
• Network connectivity int.<name>.<domain>:22623/config/worker": dial tcp: lookup api-int.<name>.<domain> on 192.168.79.1:53: read udp
192.168.79.0:48059->192.168.79.1:53: i/o timeout
[ **] A start job is running for Ignition (fetch) (59s / no limit)[ 63.691024] ignition[715]: GET https://api-
int.<name>.<domain>:22623/config/worker: attempt #8
[ **] A start job is running for Ignition (fetch) (1min 19s / no limit)[ 83.692592] ignition[715]: GET error: Get "https://api-

Add the compute nodes to the cluster

int.<name>.<domain>:22623/config/worker": dial tcp: lookup api-int.<name>.<domain> on 192.168.79.1:53: read udp
• 192.168.79.0:58831->192.168.79.1:53: i/o timeout
[*** ] A start job is running for Ignition (fetch) (1min 24s / no limit)[ 88.692908] ignition[715]: GET https://api-

• Check for pending csrs and approve

int.<name>.<domain>:22623/config/worker: attempt #9
[* ] A start job is running for Ignition (fetch) (1min 44s / no limit)[ 108.694466] ignition[715]: GET error: Get "https://api-
int.<name>.<domain>:22623/config/worker": dial tcp: lookup api-int.<name>.<domain> on 192.168.79.1:53: read udp

oc get csr
192.168.79.0:39987->192.168.79.1:53: i/o timeout
[ *** ] A start job is running for Ignition (fetch) (1min 49s / no limit)[ 113.695028] ignition[715]: GET https://api-
int.<name>.<domain>:22623/config/worker: attempt #10

oc adm certificate approve <csr-name> [ **] A start job is running for Ignition (fetch) (2min 9s / no limit)[ 133.697541] ignition[715]: GET error: Get "https://api-
int.<name>.<domain>:22623/config/worker": dial tcp: lookup api-int.<name>.<domain> on 192.168.79.1:53: read udp
192.168.79.0:34470->192.168.79.1:53: i/o timeout
[ *** ] A start job is running for Ignition (fetch) (2min 14s / no limit)[ 138.697372] ignition[715]: GET https://api-
int.<name>.<domain>:22623/config/worker: attempt #11
[*** ] A start job is running for Ignition (fetch) (2min 34s / no limit)[ 158.698748] ignition[715]: GET error: Get "https://api-
int.<name>.<domain>:22623/config/worker": dial tcp: lookup api-int.<name>.<domain> on 192.168.79.1:53: read udp
192.168.79.0:60220->192.168.79.1:53: i/o timeout
[*** ] A start job is running for Ignition (fetch) (2min 39s / no limit)[ 163.699509] ignition[715]: GET https://api-
Installing Red Hat OpenShift Container Platform with RHEL 8.3 KVM on IBM Z and LinuxONE / © 2021 IBM Corporation int.<name>.<domain>:22623/config/worker: attempt #12 35
 Step 8 of 14: Bootstrapping Complete
DNS
Load Balancer prepare install Control
VMs VMs Compute
prepare DHCP prepare Bootstrap Control create configure
prereqs. HTTP/S install Compute cluster cluster
prepare install Control
Gateway VMs VMs
Install Control Compute
Install VM / Plane Plane Plane
KVM Host

Monitor: a)
Control
bootkube.service: Succeeded. Compute
Control
Compute
a) Bootstrap not needed any more Control
• Save to shut down Control Compute
Plane Plane
b) Create third compute node from it
• Remove the VM b) Control Compute

• Reinstall with ignition: worker.ign Control Compute

• Check for pending csrs Control Compute
oc get csr
Control Compute
oc adm certificate approve <csr-name> Plane Plane
Installing Red Hat OpenShift Container Platform with RHEL 8.3 KVM on IBM Z and LinuxONE / © 2021 IBM Corporation 36
 Step 9 of 14: Completing the Cluster Installation
DNS
Load Balancer prepare install Control
VMs VMs Compute
prepare DHCP prepare Bootstrap Control create configure
prereqs. HTTP/S install Compute cluster cluster
prepare install Control
Gateway VMs VMs
Install Control Compute
Install VM / Plane Plane Plane
KVM Host

• Monitor nodes and cluster operators oc debug node/<nodename>

oc get nodes chroot /host
oc get clusteroperators toolbox
dbginfo.sh
• Cluster operator does not come up
oc describe clusteroperators <clusteroperator>
oc get pods –n <namespace>
oc get all –n <namespace>
oc logs <type>/<name> -n <namespace>
oc describe nodes
oc adm must-gather
https://docs.openshift.com/container-platform/4.7/installing/installing_ibm_z/installing-ibm-z-kvm.html#installation-ibm-z-troubleshooting-and-debugging_installing-ibm-z-kvm
Installing Red Hat OpenShift Container Platform with RHEL 8.3 KVM on IBM Z and LinuxONE / © 2021 IBM Corporation 37
 Step 10 of 14: Installation Times
DNS
Load Balancer prepare install Control
VMs VMs Compute
prepare DHCP prepare Bootstrap Control create configure
prereqs. HTTP/S install Compute cluster cluster
prepare install Control
Gateway VMs VMs
Install Control Compute
Install VM / Plane Plane Plane
KVM Host

Usual installation times

• Bootstrap accessible after 1 – 15 minutes
• including syncing disks
• not including formatting of disks
• Bootstrap complete after 5 – 10 minutes
• Install process duration: 20 – 60 minutes

• Cluster should to run for 24 hours

https://docs.openshift.com/container-platform/4.7/installing/installing_ibm_z/installing-ibm-z-kvm.html#installation-installing-bare-metal_installing-ibm-z-kvm
https://docs.openshift.com/container-platform/4.7/backup_and_restore/disaster_recovery/scenario-3-expired-certs.html
https://www.openshift.com/blog/enabling-openshift-4-clusters-to-stop-and-resume-cluster-vms
Installing Red Hat OpenShift Container Platform with RHEL 8.3 KVM on IBM Z and LinuxONE / © 2021 IBM Corporation 38

Disclaimer: Values should only be taken as a reference and might vary depending on configuration and environment.
 Step 11 of 14: Configuring the Cluster (1 of 3)
DNS
Load Balancer prepare install Control
VMs VMs Compute
prepare DHCP prepare Bootstrap Control create configure
prereqs. HTTP/S install Compute cluster cluster
prepare install Control
Gateway VMs VMs
Install Control Compute
Install VM / Plane Plane Plane
KVM Host

Preferred registry config:

• Managed registry:
oc patch configs.imageregistry.operator.openshift.io cluster \
--type merge --patch '{"spec":{"managementState":"Managed"}}’

• Prove of concept:
oc patch configs.imageregistry.operator.openshift.io cluster \
--type merge --patch '{"spec":{"storage":{"emptyDir":{}}}}’

• Production, with NFS

oc patch configs.imageregistry.operator.openshift.io cluster \
--type merge --patch '{"spec":{"storage":{"pvc":{"claim":""}}}}'

https://docs.openshift.com/container-platform/4.7/installing/installing_ibm_z/installing-ibm-z-kvm.html#registry-configuring-storage-baremetal_installing-ibm-z-kvm
https://kubernetes.io/docs/concepts/storage/volumes/#emptydir
Installing Red Hat OpenShift Container Platform with RHEL 8.3 KVM on IBM Z and LinuxONE / © 2021 IBM Corporation 39
 Step 11 of 14: Configuring the Cluster (2 of 3)
DNS
Load Balancer prepare install Control
VMs VMs Compute
prepare DHCP prepare Bootstrap Control create configure
prereqs. HTTP/S install Compute cluster cluster
prepare install Control
Gateway VMs VMs
Install Control Compute
Install VM / Plane Plane Plane
KVM Host

Preferred cluster config:

• Add identity provider of choice
• Add cluster-admin and test it
• Remove kube:admin

https://docs.openshift.com/container-platform/4.7/authentication/understanding-identity-provider.html
https://docs.openshift.com/container-platform/4.7/authentication/remove-kubeadmin.html
Installing Red Hat OpenShift Container Platform with RHEL 8.3 KVM on IBM Z and LinuxONE / © 2021 IBM Corporation 40
 Step 11 of 14: Configuring the Cluster (3 of 3)
DNS
Load Balancer prepare install Control
VMs VMs Compute
prepare DHCP prepare Bootstrap Control create configure
prereqs. HTTP/S install Compute cluster cluster
prepare install Control
Gateway VMs VMs
Install Control Compute
Install VM / Plane Plane Plane
KVM Host

Preferred backups: Install VM:

• ssh keys <installation_directory>
|── auth
• Files
| |── kubeadmin-password
• etcd | |── kubeconfig
|── bootstrap.ign
|── master.ign
|── metadata.json
|── worker.ign

https://docs.openshift.com/container-platform/4.7/backup_and_restore/backing-up-etcd.html
Installing Red Hat OpenShift Container Platform with RHEL 8.3 KVM on IBM Z and LinuxONE / © 2021 IBM Corporation 41
 Step 12 of 14: Configure the Control Plane Nodes for Large Clusters
DNS
Load Balancer prepare install Control Compute Compute
VMs VMs
prepare DHCP prepare Bootstrap Control Compute create
Compute configure
prereqs. HTTP/S install cluster cluster
prepare install Control Compute Compute
Gateway VMs VMs
Install Control Compute Compute
Install VM / Plane Plane
KVM Host Compute Compute

• Guide for sizing of control plane nodes Compute Compute

Compute Compute
Number of
compute CPU cores Memory (GB) Compute Compute

nodes
Compute Plane
25 4 16
60 8 32

https://docs.openshift.com/container-platform/4.7/scalability_and_performance/recommended-host-practices.html#master-node-sizing_
Installing Red Hat OpenShift Container Platform with RHEL 8.3 KVM on IBM Z and LinuxONE / © 2021 IBM Corporation 42

Disclaimer: Values should only be taken as a starting guide and might vary depending on the actual installation and the importance of workload.
 Step 13 of 14: Configure Infra Nodes
DNS
Load Balancer prepare install Control Compute Compute
VMs VMs
prepare DHCP prepare Bootstrap Control Compute create
Compute configure
prereqs. HTTP/S install cluster cluster
prepare install Control Compute Compute
Gateway VMs VMs
Install Control Compute Compute
Install VM / Plane Plane
KVM Host Compute Compute

Infra
• Two to three infra nodes, depending on the Compute Compute

components Infra Compute Compute

• Monitoring service, router, image registry, Infra Compute Compute

cluster logging, … Infrastruc.

Plane Compute Plane
• Better network performance
• Free of charge subscription
• Guide for sizing Prometheus
• Depends on number of nodes and pods

https://docs.openshift.com/container-platform/4.7/machine_management/creating-infrastructure-machinesets.html
https://docs.openshift.com/container-platform/4.7/scalability_and_performance/scaling-cluster-monitoring-operator.html#prometheus-database-storage-requirements_cluster-monitoring-operator
https://www.linkedin.com/pulse/boosting-performance-using-infrastructure-nodes-your-cluster-miranda/
Installing Red Hat OpenShift Container Platform with RHEL 8.3 KVM on IBM Z and LinuxONE / © 2021 IBM Corporation 43

Disclaimer: Values should only be taken as a starting guide and might vary depending on the actual installation and the importance of workload.
 Step 14 of 14: Conclusion – Installing RHOCP with RHEL KVM on IBM Z
DNS
Load Balancer prepare install Control
VMs VMs Compute
prepare DHCP prepare Bootstrap Control create configure
prereqs. HTTP/S install Compute cluster cluster
prepare install Control
Gateway VMs VMs
Install Control Compute
Install VM / Plane Plane Plane
KVM Host

• Plan enough time • Read carefully

• Spend time on building the environment: UPI • Instructions
• Verify your environment • Errors
• Disks, CPUs, memory • Follow the steps closely
• NICs, internet connectivity, network connectivity • Be patient

https://docs.openshift.com/container-platform/4.7/installing/installing_ibm_z/installing-ibm-z-kvm.html
Installing Red Hat OpenShift Container Platform with RHEL 8.3 KVM on IBM Z and LinuxONE / © 2021 IBM Corporation 44
Red Hat OpenShift Container Platform with RHEL KVM on IBM Z
and LinuxONE

Questions

Holger Wolf – [email protected]

Product Owner – Red Hat OpenShift Container Platform on IBM Z and LinuxONE

Dr. Wolfgang Voesch – [email protected]

Iteration Manager – Red Hat OpenShift Container Platform on IBM Z and LinuxONE

Installing Red Hat OpenShift Container Platform with RHEL 8.3 KVM on IBM Z and LinuxONE / © 2021 IBM Corporation 45
Red Hat OpenShift Container Platform with RHEL KVM on IBM Z
and LinuxONE

Thank you

Holger Wolf – [email protected]

Product Owner – Red Hat OpenShift Container Platform on IBM Z and LinuxONE

Dr. Wolfgang Voesch – [email protected]

Iteration Manager – Red Hat OpenShift Container Platform on IBM Z and LinuxONE

Thanks to the “Red Hat OpenShift Container Platform on IBM Z and LinuxONE” team.

Installing Red Hat OpenShift Container Platform with RHEL 8.3 KVM on IBM Z and LinuxONE / © 2021 IBM Corporation 46
Installing Red Hat OpenShift Container Platform with RHEL 8.3 KVM on IBM Z and LinuxONE / © 2021 IBM Corporation 47