Proxmox VE Administration Guide 27 / 464

File /etc/apt/sources.list.d/ceph.list

deb http://download.proxmox.com/debian/ceph-nautilus buster main

3.1.7 Ceph Nautilus Test Repository

This Ceph repository contains the Ceph packages before they are moved to the main repository. It is used
to test new Ceph releases on Proxmox VE.

File /etc/apt/sources.list.d/ceph.list

deb http://download.proxmox.com/debian/ceph-nautilus buster test

3.1.8 Proxmox VE Ceph Luminous Repository For Upgrade

If Ceph is deployed this repository is needed for the upgrade from Proxmox VE 5.x to Proxmox VE 6.0. It
provides packages for the older Ceph Luminous release for Proxmox VE 6.0.
The Upgrade 5.x to 6.0 document explains how to use this repository in detail.

File /etc/apt/sources.list.d/ceph.list

deb http://download.proxmox.com/debian/ceph-luminous buster main

3.1.9 SecureApt

The Release files in the repositories are signed with GnuPG. APT is using these signatures to verify that all
packages are from a trusted source.
If you install Proxmox VE from an official ISO image, the key for verification is already installed.
If you install Proxmox VE on top of Debian, download and install the key with the following commands:

# wget http://download.proxmox.com/debian/proxmox-ve-release-6.x.gpg -O / ←-
etc/apt/trusted.gpg.d/proxmox-ve-release-6.x.gpg

Verify the checksum afterwards with:

# sha512sum /etc/apt/trusted.gpg.d/proxmox-ve-release-6.x.gpg

The output should be:

Proxmox VE Administration Guide 28 / 464

acca6f416917e8e11490a08a1e2842d500b3a5d9f322c6319db0927b2901c3eae23cfb5cd5df6facf2b
/etc/apt/trusted.gpg.d/proxmox-ve-release-6.x.gpg

or:

# md5sum /etc/apt/trusted.gpg.d/proxmox-ve-release-6.x.gpg

The output should be:

f3f6c5a3a67baf38ad178e5ff1ee270c /etc/apt/trusted.gpg.d/proxmox-ve-release ←-
-6.x.gpg

3.2 System Software Updates

Proxmox provides updates on a regular basis for all repositories. To install updates use the web-based GUI
or the following CLI commands:

# apt-get update
# apt-get dist-upgrade

Note
The APT package management system is very flexible and provides many features, see man apt-get,
or [Hertzog13] for additional information.

Tip
Regular updates are essential to get the latest patches and security related fixes. Major system upgrades
are announced in the Proxmox VE Community Forum.

3.3 Network Configuration

Network configuration can be done either via the GUI, or by manually editing the file /etc/network/interfac
which contains the whole network configuration. The interfaces(5) manual page contains the com-
plete format description. All Proxmox VE tools try hard to keep direct user modifications, but using the GUI
is still preferable, because it protects you from errors.
Once the network is configured, you can use the Debian traditional tools ifup and ifdown commands to
bring interfaces up and down.
Proxmox VE Administration Guide 29 / 464

3.3.1 Apply Network Changes

Proxmox VE does not write changes directly to /etc/network/interfaces. Instead, we write into a
temporary file called /etc/network/interfaces.new, this way you can do many related changes at
once. This also allows to ensure your changes are correct before applying, as a wrong network configuration
may render a node inaccessible.

Reboot Node to apply

With the default installed ifupdown network managing package you need to reboot to commit any pending
network changes. Most of the time, the basic Proxmox VE network setup is stable and does not change
often, so rebooting should not be required often.

Reload Network with ifupdown2

With the optional ifupdown2 network managing package you also can reload the network configuration
live, without requiring a reboot.

Note
ifupdown2 cannot understand OpenVSwitch syntax, so reloading is not possible if OVS interfaces are
configured.

Since Proxmox VE 6.1 you can apply pending network changes over the web-interface, using the Apply
Configuration button in the Network panel of a node.
To install ifupdown2 ensure you have the latest Proxmox VE updates installed, then

Warning
installing ifupdown2 will remove ifupdown, but as the removal scripts of ifupdown before version
0.8.35+pve1 have a issue where network is fully stopped on removal a you must ensure that you
have a up to date ifupdown package version.
a Introduced with Debian Buster: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=945877

For the installation itself you can then simply do:

apt install ifupdown2

With that you’re all set. You can also switch back to the ifupdown variant at any time, if you run into issues.

3.3.2 Naming Conventions

We currently use the following naming conventions for device names:

• Ethernet devices: en*, systemd network interface names. This naming scheme is used for new Proxmox
VE installations since version 5.0.
Proxmox VE Administration Guide 30 / 464

• Ethernet devices: eth[N], where 0 ≤ N (eth0, eth1, . . . ) This naming scheme is used for Proxmox VE
hosts which were installed before the 5.0 release. When upgrading to 5.0, the names are kept as-is.

• Bridge names: vmbr[N], where 0 ≤ N ≤ 4094 (vmbr0 - vmbr4094)

• Bonds: bond[N], where 0 ≤ N (bond0, bond1, . . . )

• VLANs: Simply add the VLAN number to the device name, separated by a period (eno1.50, bond1.30)

This makes it easier to debug networks problems, because the device name implies the device type.

Systemd Network Interface Names

Systemd uses the two character prefix en for Ethernet network devices. The next characters depends on the
device driver and the fact which schema matches first.

• o<index>[n<phys_port_name>|d<dev_port>] — devices on board

• s<slot>[f<function>][n<phys_port_name>|d<dev_port>] — device by hotplug id

• [P<domain>]p<bus>s<slot>[f<function>][n<phys_port_name>|d<dev_port>] — devices by bus id

• x<MAC> — device by MAC address

The most common patterns are:

• eno1 — is the first on board NIC

• enp3s0f1 — is the NIC on pcibus 3 slot 0 and use the NIC function 1.

For more information see Predictable Network Interface Names.

3.3.3 Choosing a network configuration

Depending on your current network organization and your resources you can choose either a bridged, routed,
or masquerading networking setup.

Proxmox VE server in a private LAN, using an external gateway to reach the internet

The Bridged model makes the most sense in this case, and this is also the default mode on new Proxmox
VE installations. Each of your Guest system will have a virtual interface attached to the Proxmox VE bridge.
This is similar in effect to having the Guest network card directly connected to a new switch on your LAN, the
Proxmox VE host playing the role of the switch.

Proxmox VE server at hosting provider, with public IP ranges for Guests

For this setup, you can use either a Bridged or Routed model, depending on what your provider allows.
Proxmox VE Administration Guide 31 / 464

Proxmox VE server at hosting provider, with a single public IP address

In that case the only way to get outgoing network accesses for your guest systems is to use Masquerading.
For incoming network access to your guests, you will need to configure Port Forwarding.
For further flexibility, you can configure VLANs (IEEE 802.1q) and network bonding, also known as "link
aggregation". That way it is possible to build complex and flexible virtual networks.

3.3.4 Default Configuration using a Bridge

Top of Rack Switch

1 2 3

eno1 eno1
eno1

Gateway, DHCP vmbr0 vmbr0

192.168.10.1 192.168.10.2/24 192.168.10.3/24
tap100i0 tap200i0 tap201i0

ens18 ens18 ens18

VM 100 VM 200 VM 201

192.168.10.100 192.168.10.200 192.168.10.201

Node: proxmox1 Node: proxmox2

Bridges are like physical network switches implemented in software. All virtual guests can share a single
bridge, or you can create multiple bridges to separate network domains. Each host can have up to 4094
bridges.
The installation program creates a single bridge named vmbr0, which is connected to the first Ethernet
card. The corresponding configuration in /etc/network/interfaces might look like this:

auto lo
iface lo inet loopback

iface eno1 inet manual

auto vmbr0
iface vmbr0 inet static
address 192.168.10.2
netmask 255.255.255.0
gateway 192.168.10.1
Proxmox VE Administration Guide 32 / 464

bridge-ports eno1
bridge-stp off
bridge-fd 0

Virtual machines behave as if they were directly connected to the physical network. The network, in turn,
sees each virtual machine as having its own MAC, even though there is only one network cable connecting
all of these VMs to the network.

3.3.5 Routed Configuration

Most hosting providers do not support the above setup. For security reasons, they disable networking as
soon as they detect multiple MAC addresses on a single interface.

Tip
Some providers allow you to register additional MACs through their management interface. This avoids
the problem, but can be clumsy to configure because you need to register a MAC for each of your VMs.

You can avoid the problem by “routing” all traffic via a single interface. This makes sure that all network
packets use the same MAC address.

Provider Gateway
198.51.100.1

ip_forward = 1
vmbr0 proxy_arp = 1 eno0
203.0.113.17/28 198.51.100.5/29
tap100i0

ens18 ens18 ens18

VM 100 VM 101 VM 101

203.0.113.18 203.0.113.19 203.0.113.20

Node: proxmox

A common scenario is that you have a public IP (assume 198.51.100.5 for this example), and an addi-
tional IP block for your VMs (203.0.113.16/29). We recommend the following setup for such situations:

auto lo
iface lo inet loopback
Proxmox VE Administration Guide 33 / 464

auto eno1
iface eno1 inet static
address 198.51.100.5
netmask 255.255.255.0
gateway 198.51.100.1
post-up echo 1 > /proc/sys/net/ipv4/ip_forward
post-up echo 1 > /proc/sys/net/ipv4/conf/eno1/proxy_arp

auto vmbr0
iface vmbr0 inet static
address 203.0.113.17
netmask 255.255.255.248
bridge-ports none
bridge-stp off
bridge-fd 0

3.3.6 Masquerading (NAT) with iptables

Masquerading allows guests having only a private IP address to access the network by using the host IP
address for outgoing traffic. Each outgoing packet is rewritten by iptables to appear as originating from
the host, and responses are rewritten accordingly to be routed to the original sender.

auto lo
iface lo inet loopback

auto eno1
#real IP address
iface eno1 inet static
address 198.51.100.5
netmask 255.255.255.0
gateway 198.51.100.1

auto vmbr0
#private sub network
iface vmbr0 inet static
address 10.10.10.1
netmask 255.255.255.0
bridge-ports none
bridge-stp off
bridge-fd 0

post-up echo 1 > /proc/sys/net/ipv4/ip_forward

post-up iptables -t nat -A POSTROUTING -s ’10.10.10.0/24’ -o eno1 ←-
-j MASQUERADE
post-down iptables -t nat -D POSTROUTING -s ’10.10.10.0/24’ -o eno1 ←-
-j MASQUERADE
Proxmox VE Administration Guide 34 / 464

Note
In some masquerade setups with firewall enabled, conntrack zones might be needed for outgoing connec-
tions. Otherwise the firewall could block outgoing connections since they will prefer the POSTROUTING
of the VM bridge (and not MASQUERADE).

Adding these lines in the /etc/network/interfaces can fix this problem:

post-up iptables -t raw -I PREROUTING -i fwbr+ -j CT --zone 1

post-down iptables -t raw -D PREROUTING -i fwbr+ -j CT --zone 1

For more information about this, refer to the following links:

Netfilter Packet Flow
Patch on netdev-list introducing conntrack zones
Blog post with a good explanation by using TRACE in the raw table

3.3.7 Linux Bond

Bonding (also called NIC teaming or Link Aggregation) is a technique for binding multiple NIC’s to a single
network device. It is possible to achieve different goals, like make the network fault-tolerant, increase the
performance or both together.
High-speed hardware like Fibre Channel and the associated switching hardware can be quite expensive. By
doing link aggregation, two NICs can appear as one logical interface, resulting in double speed. This is a
native Linux kernel feature that is supported by most switches. If your nodes have multiple Ethernet ports,
you can distribute your points of failure by running network cables to different switches and the bonded
connection will failover to one cable or the other in case of network trouble.
Aggregated links can improve live-migration delays and improve the speed of replication of data between
Proxmox VE Cluster nodes.
There are 7 modes for bonding:

• Round-robin (balance-rr): Transmit network packets in sequential order from the first available network
interface (NIC) slave through the last. This mode provides load balancing and fault tolerance.

• Active-backup (active-backup): Only one NIC slave in the bond is active. A different slave becomes
active if, and only if, the active slave fails. The single logical bonded interface’s MAC address is externally
visible on only one NIC (port) to avoid distortion in the network switch. This mode provides fault tolerance.

• XOR (balance-xor): Transmit network packets based on [(source MAC address XOR’d with destination
MAC address) modulo NIC slave count]. This selects the same NIC slave for each destination MAC
address. This mode provides load balancing and fault tolerance.

• Broadcast (broadcast): Transmit network packets on all slave network interfaces. This mode provides
fault tolerance.

• IEEE 802.3ad Dynamic link aggregation (802.3ad)(LACP): Creates aggregation groups that share the
same speed and duplex settings. Utilizes all slave network interfaces in the active aggregator group ac-
cording to the 802.3ad specification.
Proxmox VE Administration Guide 35 / 464

• Adaptive transmit load balancing (balance-tlb): Linux bonding driver mode that does not require any
special network-switch support. The outgoing network packet traffic is distributed according to the current
load (computed relative to the speed) on each network interface slave. Incoming traffic is received by one
currently designated slave network interface. If this receiving slave fails, another slave takes over the MAC
address of the failed receiving slave.

• Adaptive load balancing (balance-alb): Includes balance-tlb plus receive load balancing (rlb) for IPV4
traffic, and does not require any special network switch support. The receive load balancing is achieved by
ARP negotiation. The bonding driver intercepts the ARP Replies sent by the local system on their way out
and overwrites the source hardware address with the unique hardware address of one of the NIC slaves in
the single logical bonded interface such that different network-peers use different MAC addresses for their
network packet traffic.

If your switch support the LACP (IEEE 802.3ad) protocol then we recommend using the corresponding
bonding mode (802.3ad). Otherwise you should generally use the active-backup mode.
If you intend to run your cluster network on the bonding interfaces, then you have to use active-passive mode
on the bonding interfaces, other modes are unsupported.
The following bond configuration can be used as distributed/shared storage network. The benefit would be
that you get more speed and the network will be fault-tolerant.

Example: Use bond with fixed IP address

auto lo
iface lo inet loopback

iface eno1 inet manual

iface eno2 inet manual

iface eno3 inet manual

auto bond0
iface bond0 inet static
bond-slaves eno1 eno2
address 192.168.1.2
netmask 255.255.255.0
bond-miimon 100
bond-mode 802.3ad
bond-xmit-hash-policy layer2+3

auto vmbr0
iface vmbr0 inet static
address 10.10.10.2
netmask 255.255.255.0
gateway 10.10.10.1
bridge-ports eno3
bridge-stp off
bridge-fd 0
Proxmox VE Administration Guide 36 / 464

Top of Rack Switch 1 MLAG Top of Rack Switch 2

1 2 1 2

eno1 eno2 eno1 eno2

bond0 bond0
LACP LACP

bond0 bond0

vmbr0 vmbr0
10.10.10.2/24 10.10.10.3/24
tap100i0 tap100i0

ens18 ens18

VM 100 VM 200
10.10.10.100 10.10.10.200

Node: proxmox1 Node: proxmox2

Another possibility it to use the bond directly as bridge port. This can be used to make the guest network
fault-tolerant.

Example: Use a bond as bridge port

auto lo
iface lo inet loopback

iface eno1 inet manual

iface eno2 inet manual

auto bond0
iface bond0 inet manual
bond-slaves eno1 eno2
bond-miimon 100
bond-mode 802.3ad
bond-xmit-hash-policy layer2+3

auto vmbr0
iface vmbr0 inet static
address 10.10.10.2
netmask 255.255.255.0
gateway 10.10.10.1
bridge-ports bond0
bridge-stp off
bridge-fd 0
Proxmox VE Administration Guide 37 / 464

3.3.8 VLAN 802.1Q

A virtual LAN (VLAN) is a broadcast domain that is partitioned and isolated in the network at layer two. So it
is possible to have multiple networks (4096) in a physical network, each independent of the other ones.
Each VLAN network is identified by a number often called tag. Network packages are then tagged to identify
which virtual network they belong to.

VLAN for Guest Networks

Proxmox VE supports this setup out of the box. You can specify the VLAN tag when you create a VM.
The VLAN tag is part of the guest network configuration. The networking layer supports different modes to
implement VLANs, depending on the bridge configuration:

• VLAN awareness on the Linux bridge: In this case, each guest’s virtual network card is assigned to
a VLAN tag, which is transparently supported by the Linux bridge. Trunk mode is also possible, but that
makes configuration in the guest necessary.

• "traditional" VLAN on the Linux bridge: In contrast to the VLAN awareness method, this method is not
transparent and creates a VLAN device with associated bridge for each VLAN. That is, creating a guest on
VLAN 5 for example, would create two interfaces eno1.5 and vmbr0v5, which would remain until a reboot
occurs.

• Open vSwitch VLAN: This mode uses the OVS VLAN feature.

• Guest configured VLAN: VLANs are assigned inside the guest. In this case, the setup is completely
done inside the guest and can not be influenced from the outside. The benefit is that you can use more
than one VLAN on a single virtual NIC.

VLAN on the Host

To allow host communication with an isolated network. It is possible to apply VLAN tags to any network
device (NIC, Bond, Bridge). In general, you should configure the VLAN on the interface with the least
abstraction layers between itself and the physical NIC.
For example, in a default configuration where you want to place the host management address on a separate
VLAN.

Example: Use VLAN 5 for the Proxmox VE management IP with traditional Linux bridge

auto lo
iface lo inet loopback

iface eno1 inet manual

iface eno1.5 inet manual

auto vmbr0v5
iface vmbr0v5 inet static
address 10.10.10.2
Proxmox VE Administration Guide 38 / 464

netmask 255.255.255.0
gateway 10.10.10.1
bridge-ports eno1.5
bridge-stp off
bridge-fd 0

auto vmbr0
iface vmbr0 inet manual
bridge-ports eno1
bridge-stp off
bridge-fd 0

Example: Use VLAN 5 for the Proxmox VE management IP with VLAN aware Linux bridge

auto lo
iface lo inet loopback

iface eno1 inet manual

auto vmbr0.5
iface vmbr0.5 inet static
address 10.10.10.2
netmask 255.255.255.0
gateway 10.10.10.1

auto vmbr0
iface vmbr0 inet manual
bridge-ports eno1
bridge-stp off
bridge-fd 0
bridge-vlan-aware yes

The next example is the same setup but a bond is used to make this network fail-safe.

Example: Use VLAN 5 with bond0 for the Proxmox VE management IP with traditional Linux bridge

auto lo
iface lo inet loopback

iface eno1 inet manual

iface eno2 inet manual

auto bond0
iface bond0 inet manual
bond-slaves eno1 eno2
bond-miimon 100
bond-mode 802.3ad
bond-xmit-hash-policy layer2+3
Proxmox VE Administration Guide 39 / 464

iface bond0.5 inet manual

auto vmbr0v5
iface vmbr0v5 inet static
address 10.10.10.2
netmask 255.255.255.0
gateway 10.10.10.1
bridge-ports bond0.5
bridge-stp off
bridge-fd 0

auto vmbr0
iface vmbr0 inet manual
bridge-ports bond0
bridge-stp off
bridge-fd 0

3.4 Time Synchronization

The Proxmox VE cluster stack itself relies heavily on the fact that all the nodes have precisely synchronized
time. Some other components, like Ceph, also refuse to work properly if the local time on nodes is not in
sync.
Time synchronization between nodes can be achieved with the “Network Time Protocol” (NTP). Proxmox VE
uses systemd-timesyncd as NTP client by default, preconfigured to use a set of public servers. This
setup works out of the box in most cases.

3.4.1 Using Custom NTP Servers

In some cases, it might be desired to not use the default NTP servers. For example, if your Proxmox VE
nodes do not have access to the public internet (e.g., because of restrictive firewall rules), you need to setup
local NTP servers and tell systemd-timesyncd to use them:

File /etc/systemd/timesyncd.conf

[Time]
NTP=ntp1.example.com ntp2.example.com ntp3.example.com ntp4.example.com

After restarting the synchronization service (systemctl restart systemd-timesyncd) you should
verify that your newly configured NTP servers are used by checking the journal (journalctl --since
-1h -u systemd-timesyncd):

...
Oct 07 14:58:36 node1 systemd[1]: Stopping Network Time Synchronization...
Oct 07 14:58:36 node1 systemd[1]: Starting Network Time Synchronization...
Oct 07 14:58:36 node1 systemd[1]: Started Network Time Synchronization.
Proxmox VE Administration Guide 40 / 464

Oct 07 14:58:36 node1 systemd-timesyncd[13514]: Using NTP server ←-

10.0.0.1:123 (ntp1.example.com).
Oct 07 14:58:36 nora systemd-timesyncd[13514]: interval/delta/delay/jitter/ ←-
drift 64s/-0.002s/0.020s/0.000s/-31ppm
...

3.5 External Metric Server

In Proxmox VE, you can define external metric servers, which will periodically receive various stats about
your hosts, virtual guests and storages.
Currently supported are:

• Graphite (see http://graphiteapp.org )

• InfluxDB (see https://www.influxdata.com/time-series-platform/influxdb/ )

The external metric server definitions are saved in /etc/pve/status.cfg, and can be edited through the web
interface.
Proxmox VE Administration Guide 41 / 464

3.5.1 Graphite server configuration

The default port is set to 2003 and the default graphite path is proxmox.
By default, Proxmox VE sends the data over UDP, so the graphite server has to be configured to accept this.
Here the maximum transmission unit (MTU) can be configured for environments not using the standard 1500
MTU.
You can also configure the plugin to use TCP. In order not to block the important pvestatd statistic collec-
tion daemon, a timeout is required to cope with network problems.

3.5.2 Influxdb plugin configuration

Proxmox VE sends the data over UDP, so the influxdb server has to be configured for this. The MTU can
also be configured here, if necessary.
Here is an example configuration for influxdb (on your influxdb server):

[[udp]]
enabled = true
bind-address = "0.0.0.0:8089"
database = "proxmox"
batch-size = 1000
batch-timeout = "1s"

With this configuration, your server listens on all IP addresses on port 8089, and writes the data in the
proxmox database

3.6 Disk Health Monitoring

Although a robust and redundant storage is recommended, it can be very helpful to monitor the health of
your local disks.
Starting with Proxmox VE 4.3, the package smartmontools 1 is installed and required. This is a set of tools
to monitor and control the S.M.A.R.T. system for local hard disks.
1 smartmontools homepage https://www.smartmontools.org
Proxmox VE Administration Guide 42 / 464

You can get the status of a disk by issuing the following command:

# smartctl -a /dev/sdX

where /dev/sdX is the path to one of your local disks.

If the output says:

SMART support is: Disabled

you can enable it with the command:

# smartctl -s on /dev/sdX

For more information on how to use smartctl, please see man smartctl.
By default, smartmontools daemon smartd is active and enabled, and scans the disks under /dev/sdX and
/dev/hdX every 30 minutes for errors and warnings, and sends an e-mail to root if it detects a problem.
For more information about how to configure smartd, please see man smartd and man smartd.conf.
If you use your hard disks with a hardware raid controller, there are most likely tools to monitor the disks in
the raid array and the array itself. For more information about this, please refer to the vendor of your raid
controller.

3.7 Logical Volume Manager (LVM)

Most people install Proxmox VE directly on a local disk. The Proxmox VE installation CD offers several
options for local disk management, and the current default setup uses LVM. The installer let you select a
single disk for such setup, and uses that disk as physical volume for the Volume Group (VG) pve. The
following output is from a test installation using a small 8GB disk:

# pvs
PV VG Fmt Attr PSize PFree
/dev/sda3 pve lvm2 a-- 7.87g 876.00m

# vgs
VG #PV #LV #SN Attr VSize VFree
pve 1 3 0 wz--n- 7.87g 876.00m

The installer allocates three Logical Volumes (LV) inside this VG:

# lvs
LV VG Attr LSize Pool Origin Data% Meta%
data pve twi-a-tz-- 4.38g 0.00 0.63
root pve -wi-ao---- 1.75g
swap pve -wi-ao---- 896.00m
Proxmox VE Administration Guide 43 / 464

root
Formatted as ext4, and contains the operation system.

swap
Swap partition

data
This volume uses LVM-thin, and is used to store VM images. LVM-thin is preferable for this task,
because it offers efficient support for snapshots and clones.

For Proxmox VE versions up to 4.1, the installer creates a standard logical volume called “data”, which is
mounted at /var/lib/vz.
Starting from version 4.2, the logical volume “data” is a LVM-thin pool, used to store block based guest
images, and /var/lib/vz is simply a directory on the root file system.

3.7.1 Hardware

We highly recommend to use a hardware RAID controller (with BBU) for such setups. This increases perfor-
mance, provides redundancy, and make disk replacements easier (hot-pluggable).
LVM itself does not need any special hardware, and memory requirements are very low.

3.7.2 Bootloader

We install two boot loaders by default. The first partition contains the standard GRUB boot loader. The
second partition is an EFI System Partition (ESP), which makes it possible to boot on EFI systems.

3.7.3 Creating a Volume Group

Let’s assume we have an empty disk /dev/sdb, onto which we want to create a volume group named
“vmdata”.

Caution
Please note that the following commands will destroy all existing data on /dev/sdb.

First create a partition.

# sgdisk -N 1 /dev/sdb

Create a Physical Volume (PV) without confirmation and 250K metadatasize.

# pvcreate --metadatasize 250k -y -ff /dev/sdb1

Proxmox VE Administration Guide 44 / 464

Create a volume group named “vmdata” on /dev/sdb1

# vgcreate vmdata /dev/sdb1

3.7.4 Creating an extra LV for /var/lib/vz

This can be easily done by creating a new thin LV.

# lvcreate -n <Name> -V <Size[M,G,T]> <VG>/<LVThin_pool>

A real world example:

# lvcreate -n vz -V 10G pve/data

Now a filesystem must be created on the LV.

# mkfs.ext4 /dev/pve/vz

At last this has to be mounted.

Warning
be sure that /var/lib/vz is empty. On a default installation it’s not.

To make it always accessible add the following line in /etc/fstab.

# echo ’/dev/pve/vz /var/lib/vz ext4 defaults 0 2’ >> /etc/fstab

3.7.5 Resizing the thin pool

Resize the LV and the metadata pool can be achieved with the following command.

# lvresize --size +<size[\M,G,T]> --poolmetadatasize +<size[\M,G]> < ←-

VG>/<LVThin_pool>

Note
When extending the data pool, the metadata pool must also be extended.
Proxmox VE Administration Guide 45 / 464

3.7.6 Create a LVM-thin pool

A thin pool has to be created on top of a volume group. How to create a volume group see Section LVM.

# lvcreate -L 80G -T -n vmstore vmdata

3.8 ZFS on Linux

ZFS is a combined file system and logical volume manager designed by Sun Microsystems. Starting with
Proxmox VE 3.4, the native Linux kernel port of the ZFS file system is introduced as optional file system and
also as an additional selection for the root file system. There is no need for manually compile ZFS modules
- all packages are included.
By using ZFS, its possible to achieve maximum enterprise features with low budget hardware, but also high
performance systems by leveraging SSD caching or even SSD only setups. ZFS can replace cost intense
hardware raid cards by moderate CPU and memory load combined with easy management.
G ENERAL ZFS ADVANTAGES

• Easy configuration and management with Proxmox VE GUI and CLI.

• Reliable

• Protection against data corruption

• Data compression on file system level

• Snapshots

• Copy-on-write clone

• Various raid levels: RAID0, RAID1, RAID10, RAIDZ-1, RAIDZ-2 and RAIDZ-3

• Can use SSD for cache

• Self healing

• Continuous integrity checking

• Designed for high storage capacities

• Protection against data corruption

• Asynchronous replication over network

• Open Source

• Encryption

• ...
Proxmox VE Administration Guide 46 / 464

3.8.1 Hardware

ZFS depends heavily on memory, so you need at least 8GB to start. In practice, use as much you can get
for your hardware/budget. To prevent data corruption, we recommend the use of high quality ECC RAM.
If you use a dedicated cache and/or log disk, you should use an enterprise class SSD (e.g. Intel SSD DC
S3700 Series). This can increase the overall performance significantly.

Important
Do not use ZFS on top of hardware controller which has its own cache management. ZFS needs to
directly communicate with disks. An HBA adapter is the way to go, or something like LSI controller
flashed in “IT” mode.

If you are experimenting with an installation of Proxmox VE inside a VM (Nested Virtualization), don’t use
virtio for disks of that VM, since they are not supported by ZFS. Use IDE or SCSI instead (works also
with virtio SCSI controller type).

3.8.2 Installation as Root File System

When you install using the Proxmox VE installer, you can choose ZFS for the root file system. You need to
select the RAID type at installation time:

RAID0 Also called “striping”. The capacity of such volume is the sum of the capacities of all
disks. But RAID0 does not add any redundancy, so the failure of a single drive
makes the volume unusable.

RAID1 Also called “mirroring”. Data is written identically to all disks. This mode requires at
least 2 disks with the same size. The resulting capacity is that of a single disk.

RAID10 A combination of RAID0 and RAID1. Requires at least 4 disks.

RAIDZ-1 A variation on RAID-5, single parity. Requires at least 3 disks.

RAIDZ-2 A variation on RAID-5, double parity. Requires at least 4 disks.

RAIDZ-3 A variation on RAID-5, triple parity. Requires at least 5 disks.

The installer automatically partitions the disks, creates a ZFS pool called rpool, and installs the root file
system on the ZFS subvolume rpool/ROOT/pve-1.
Another subvolume called rpool/data is created to store VM images. In order to use that with the
Proxmox VE tools, the installer creates the following configuration entry in /etc/pve/storage.cfg:

zfspool: local-zfs
pool rpool/data
sparse
content images,rootdir
Proxmox VE Administration Guide 47 / 464

After installation, you can view your ZFS pool status using the zpool command:

# zpool status
pool: rpool
state: ONLINE
scan: none requested
config:

NAME STATE READ WRITE CKSUM

rpool ONLINE 0 0 0
mirror-0 ONLINE 0 0 0
sda2 ONLINE 0 0 0
sdb2 ONLINE 0 0 0
mirror-1 ONLINE 0 0 0
sdc ONLINE 0 0 0
sdd ONLINE 0 0 0

errors: No known data errors

The zfs command is used configure and manage your ZFS file systems. The following command lists all
file systems after installation:

# zfs list
NAME USED AVAIL REFER MOUNTPOINT
rpool 4.94G 7.68T 96K /rpool
rpool/ROOT 702M 7.68T 96K /rpool/ROOT
rpool/ROOT/pve-1 702M 7.68T 702M /
rpool/data 96K 7.68T 96K /rpool/data
rpool/swap 4.25G 7.69T 64K -

3.8.3 ZFS RAID Level Considerations

There are a few factors to take into consideration when choosing the layout of a ZFS pool. The basic building
block of a ZFS pool is the virtual device, or vdev. All vdevs in a pool are used equally and the data is striped
among them (RAID0). Check the zpool(8) manpage for more details on vdevs.

Performance

Each vdev type has different performance behaviors. The two parameters of interest are the IOPS (In-
put/Output Operations per Second) and the bandwidth with which data can be written or read.
A mirror vdev (RAID1) will approximately behave like a single disk in regards to both parameters when writing
data. When reading data if will behave like the number of disks in the mirror.
A common situation is to have 4 disks. When setting it up as 2 mirror vdevs (RAID10) the pool will have
the write characteristics as two single disks in regard of IOPS and bandwidth. For read operations it will
resemble 4 single disks.
A RAIDZ of any redundancy level will approximately behave like a single disk in regard of IOPS with a lot of
bandwidth. How much bandwidth depends on the size of the RAIDZ vdev and the redundancy level.
Proxmox VE Administration Guide 48 / 464

For running VMs, IOPS is the more important metric in most situations.

Size, Space usage and Redundancy

While a pool made of mirror vdevs will have the best performance characteristics, the usable space will be
50% of the disks available. Less if a mirror vdev consists of more than 2 disks, for example in a 3-way mirror.
At least one healthy disk per mirror is needed for the pool to stay functional.
The usable space of a RAIDZ type vdev of N disks is roughly N-P, with P being the RAIDZ-level. The RAIDZ-
level indicates how many arbitrary disks can fail without losing data. A special case is a 4 disk pool with
RAIDZ2. In this situation it is usually better to use 2 mirror vdevs for the better performance as the usable
space will be the same.
Another important factor when using any RAIDZ level is how ZVOL datasets, which are used for VM disks,
behave. For each data block the pool needs parity data which is at least the size of the minimum block size
defined by the ashift value of the pool. With an ashift of 12 the block size of the pool is 4k. The default
block size for a ZVOL is 8k. Therefore, in a RAIDZ2 each 8k block written will cause two additional 4k parity
blocks to be written, 8k + 4k + 4k = 16k. This is of course a simplified approach and the real situation will be
slightly different with metadata, compression and such not being accounted for in this example.
This behavior can be observed when checking the following properties of the ZVOL:

• volsize

• refreservation (if the pool is not thin provisioned)

• used (if the pool is thin provisioned and without snapshots present)

# zfs get volsize,refreservation,used <pool>/vm-<vmid>-disk-X

volsize is the size of the disk as it is presented to the VM, while refreservation shows the reserved
space on the pool which includes the expected space needed for the parity data. If the pool is thin provi-
sioned, the refreservation will be set to 0. Another way to observe the behavior is to compare the
used disk space within the VM and the used property. Be aware that snapshots will skew the value.
There are a few options to counter the increased use of space:

• Increase the volblocksize to improve the data to parity ratio

• Use mirror vdevs instead of RAIDZ

• Use ashift=9 (block size of 512 bytes)

The volblocksize property can only be set when creating a ZVOL. The default value can be changed in
the storage configuration. When doing this, the guest needs to be tuned accordingly and depending on the
use case, the problem of write amplification if just moved from the ZFS layer up to the guest.
Using ashift=9 when creating the pool can lead to bad performance, depending on the disks underneath,
and cannot be changed later on.
Mirror vdevs (RAID1, RAID10) have favorable behavior for VM workloads. Use them, unless your environ-
ment has specific needs and characteristics where RAIDZ performance characteristics are acceptable.
Proxmox VE Administration Guide 49 / 464

3.8.4 Bootloader

Depending on whether the system is booted in EFI or legacy BIOS mode the Proxmox VE installer sets
up either grub or systemd-boot as main bootloader. See the chapter on Proxmox VE host bootladers
Section 3.11 for details.

3.8.5 ZFS Administration

This section gives you some usage examples for common tasks. ZFS itself is really powerful and provides
many options. The main commands to manage ZFS are zfs and zpool. Both commands come with great
manual pages, which can be read with:

# man zpool
# man zfs

Create a new zpool

To create a new pool, at least one disk is needed. The ashift should have the same sector-size (2 power
of ashift) or larger as the underlying disk.

# zpool create -f -o ashift=12 <pool> <device>

To activate compression (see section Compression in ZFS):

# zfs set compression=lz4 <pool>

Create a new pool with RAID-0

Minimum 1 disk

# zpool create -f -o ashift=12 <pool> <device1> <device2>

Create a new pool with RAID-1

Minimum 2 disks

# zpool create -f -o ashift=12 <pool> mirror <device1> <device2>

Proxmox VE Administration Guide 50 / 464

Create a new pool with RAID-10

Minimum 4 disks

# zpool create -f -o ashift=12 <pool> mirror <device1> <device2> mirror < ←-

device3> <device4>

Create a new pool with RAIDZ-1

Minimum 3 disks

# zpool create -f -o ashift=12 <pool> raidz1 <device1> <device2> <device3>

Create a new pool with RAIDZ-2

Minimum 4 disks

# zpool create -f -o ashift=12 <pool> raidz2 <device1> <device2> <device3> ←-

<device4>

Create a new pool with cache (L2ARC)

It is possible to use a dedicated cache drive partition to increase the performance (use SSD).
As <device> it is possible to use more devices, like it’s shown in "Create a new pool with RAID*".

# zpool create -f -o ashift=12 <pool> <device> cache <cache_device>

Create a new pool with log (ZIL)

It is possible to use a dedicated cache drive partition to increase the performance(SSD).

As <device> it is possible to use more devices, like it’s shown in "Create a new pool with RAID*".

# zpool create -f -o ashift=12 <pool> <device> log <log_device>

Proxmox VE Administration Guide 51 / 464

Add cache and log to an existing pool

If you have a pool without cache and log. First partition the SSD in 2 partition with parted or gdisk

Important
Always use GPT partition tables.

The maximum size of a log device should be about half the size of physical memory, so this is usually quite
small. The rest of the SSD can be used as cache.

# zpool add -f <pool> log <device-part1> cache <device-part2>

Changing a failed device

# zpool replace -f <pool> <old device> <new device>

Changing a failed bootable device

Depending on how Proxmox VE was installed it is either using grub or systemd-boot as bootloader
(see Host Bootloader Section 3.11).
The first steps of copying the partition table, reissuing GUIDs and replacing the ZFS partition are the same.
To make the system bootable from the new disk, different steps are needed which depend on the bootloader
in use.

# sgdisk <healthy bootable device> -R <new device>

# sgdisk -G <new device>
# zpool replace -f <pool> <old zfs partition> <new zfs partition>

Note
Use the zpool status -v command to monitor how far the resilvering process of the new disk has
progressed.

With systemd-boot:

# pve-efiboot-tool format <new disk’s ESP>

# pve-efiboot-tool init <new disk’s ESP>

Note
ESP stands for EFI System Partition, which is setup as partition #2 on bootable disks setup by the Proxmox
VE installer since version 5.4. For details, see Setting up a new partition for use as synced ESP Setting
up a new partition for use as synced ESP.
Proxmox VE Administration Guide 52 / 464

With grub:

# grub-install <new disk>

3.8.6 Activate E-Mail Notification

ZFS comes with an event daemon, which monitors events generated by the ZFS kernel module. The daemon
can also send emails on ZFS events like pool errors. Newer ZFS packages ship the daemon in a separate
package, and you can install it using apt-get:

# apt-get install zfs-zed

To activate the daemon it is necessary to edit /etc/zfs/zed.d/zed.rc with your favourite editor, and
uncomment the ZED_EMAIL_ADDR setting:

ZED_EMAIL_ADDR="root"

Please note Proxmox VE forwards mails to root to the email address configured for the root user.

Important
The only setting that is required is ZED_EMAIL_ADDR. All other settings are optional.

3.8.7 Limit ZFS Memory Usage

It is good to use at most 50 percent (which is the default) of the system memory for ZFS ARC to prevent per-
formance shortage of the host. Use your preferred editor to change the configuration in /etc/modprobe.d/zfs
and insert:

options zfs zfs_arc_max=8589934592

This example setting limits the usage to 8GB.

Important
If your root file system is ZFS you must update your initramfs every time this value changes:

# update-initramfs -u
Proxmox VE Administration Guide 53 / 464

3.8.8 SWAP on ZFS

Swap-space created on a zvol may generate some troubles, like blocking the server or generating a high IO
load, often seen when starting a Backup to an external Storage.
We strongly recommend to use enough memory, so that you normally do not run into low memory situations.
Should you need or want to add swap, it is preferred to create a partition on a physical disk and use it
as swapdevice. You can leave some space free for this purpose in the advanced options of the installer.
Additionally, you can lower the “swappiness” value. A good value for servers is 10:

# sysctl -w vm.swappiness=10

To make the swappiness persistent, open /etc/sysctl.conf with an editor of your choice and add the
following line:

vm.swappiness = 10

Table 3.1: Linux kernel swappiness parameter values

Value Strategy
vm.swappiness = 0 The kernel will swap only to avoid an out of memory condition
vm.swappiness = 1 Minimum amount of swapping without disabling it entirely.
vm.swappiness = 10 This value is sometimes recommended to improve performance
when sufficient memory exists in a system.
vm.swappiness = 60 The default value.
vm.swappiness = 100 The kernel will swap aggressively.

3.8.9 Encrypted ZFS Datasets

ZFS on Linux version 0.8.0 introduced support for native encryption of datasets. After an upgrade from
previous ZFS on Linux versions, the encryption feature can be enabled per pool:

# zpool get feature@encryption tank

NAME PROPERTY VALUE SOURCE
tank feature@encryption disabled local

# zpool set feature@encryption=enabled

# zpool get feature@encryption tank

NAME PROPERTY VALUE SOURCE
tank feature@encryption enabled local
Proxmox VE Administration Guide 54 / 464

Warning
There is currently no support for booting from pools with encrypted datasets using Grub, and only
limited support for automatically unlocking encrypted datasets on boot. Older versions of ZFS
without encryption support will not be able to decrypt stored data.

Note
It is recommended to either unlock storage datasets manually after booting, or to write a custom unit to
pass the key material needed for unlocking on boot to zfs load-key.

Warning
Establish and test a backup procedure before enabling encryption of production data. If the as-
sociated key material/passphrase/keyfile has been lost, accessing the encrypted data is no longer
possible.

Encryption needs to be setup when creating datasets/zvols, and is inherited by default to child datasets. For
example, to create an encrypted dataset tank/encrypted_data and configure it as storage in Proxmox
VE, run the following commands:

# zfs create -o encryption=on -o keyformat=passphrase tank/encrypted_data

Enter passphrase:
Re-enter passphrase:

# pvesm add zfspool encrypted_zfs -pool tank/encrypted_data

All guest volumes/disks create on this storage will be encrypted with the shared key material of the parent
dataset.
To actually use the storage, the associated key material needs to be loaded and the dataset needs to be
mounted. This can be done in one step with:

# zfs mount -l tank/encrypted_data

Enter passphrase for ’tank/encrypted_data’:

It is also possible to use a (random) keyfile instead of prompting for a passphrase by setting the keylocation
and keyformat properties, either at creation time or with zfs change-key on existing datasets:

# dd if=/dev/urandom of=/path/to/keyfile bs=32 count=1

# zfs change-key -o keyformat=raw -o keylocation=file:///path/to/keyfile ←-

tank/encrypted_data

Warning
When using a keyfile, special care needs to be taken to secure the keyfile against unauthorized
access or accidental loss. Without the keyfile, it is not possible to access the plaintext data!
Proxmox VE Administration Guide 55 / 464

A guest volume created underneath an encrypted dataset will have its encryptionroot property set
accordingly. The key material only needs to be loaded once per encryptionroot to be available to all encrypted
datasets underneath it.
See the encryptionroot, encryption, keylocation, keyformat and keystatus proper-
ties, the zfs load-key, zfs unload-key and zfs change-key commands and the Encryption
section from man zfs for more details and advanced usage.

3.8.10 Compression in ZFS

When compression is enabled on a dataset, ZFS tries to compress all new blocks before writing them and
decompresses them on reading. Already existing data will not be compressed retroactively.
You can enable compression with:

# zfs set compression=<algorithm> <dataset>

We recommend using the lz4 algorithm, because it adds very little CPU overhead. Other algorithms like
lzjb and gzip-N, where N is an integer from 1 (fastest) to 9 (best compression ratio), are also avail-
able. Depending on the algorithm and how compressible the data is, having compression enabled can even
increase I/O performance.
You can disable compression at any time with:

# zfs set compression=off <dataset>

Again, only new blocks will be affected by this change.

3.8.11 ZFS Special Device

Since version 0.8.0 ZFS supports special devices. A special device in a pool is used to store meta-
data, deduplication tables, and optionally small file blocks.
A special device can improve the speed of a pool consisting of slow spinning hard disks with a lot of
metadata changes. For example workloads that involve creating, updating or deleting a large number of
files will benefit from the presence of a special device. ZFS datasets can also be configured to store
whole small files on the special device which can further improve the performance. Use fast SSDs for the
special device.

Important
The redundancy of the special device should match the one of the pool, since the special
device is a point of failure for the whole pool.

Warning
Adding a special device to a pool cannot be undone!
Proxmox VE Administration Guide 56 / 464

Create a pool with special device and RAID-1:

# zpool create -f -o ashift=12 <pool> mirror <device1> <device2> special ←-

mirror <device3> <device4>

Add a special device to an existing pool with RAID-1:

# zpool add <pool> special mirror <device1> <device2>

ZFS datasets expose the special_small_blocks=<size> property. size can be 0 to disable

storing small file blocks on the special device or a power of two in the range between 512B to 128K.
After setting the property new file blocks smaller than size will be allocated on the special device.

Important
If the value for special_small_blocks is greater than or equal to the recordsize (default
128K) of the dataset, all data will be written to the special device, so be careful!

Setting the special_small_blocks property on a pool will change the default value of that property
for all child ZFS datasets (for example all containers in the pool will opt in for small file blocks).

Opt in for all file smaller than 4K-blocks pool-wide:

# zfs set special_small_blocks=4K <pool>

Opt in for small file blocks for a single dataset:

# zfs set special_small_blocks=4K <pool>/<filesystem>

Opt out from small file blocks for a single dataset:

# zfs set special_small_blocks=0 <pool>/<filesystem>

3.9 Proxmox Node Management

The Proxmox VE node management tool (pvenode) allows to control node specific settings and resources.
Currently pvenode allows to set a node’s description and to manage the node’s SSL certificates used for
the API and the web GUI through pveproxy.
Proxmox VE Administration Guide 57 / 464

3.9.1 Wake-on-LAN

Wake-on-LAN (WoL) allows to switch on a sleeping computer in the network by sending a magic packet.
At least one NIC must support this feature and the respective option needs to be enabled in the computers
firmware (BIOS/UEFI) configuration. The option name can vary from Enable Wake-on-Lan to Power On By
PCIE Device, check your motherboards vendor manual, if unsure. ethtool can be used to check the WoL
configuration of <interface> by running:

ethtool <interface> | grep Wake-on

pvenode allows to wake sleeping members of a cluster via WoL using the command:

pvenode wakeonlan <node>

This broadcasts the WoL magic packet on UDP port 9, containing the MAC address of <node> obtained
from the wakeonlan property. The node specific wakeonlan property can be set by the following com-
mand:

pvenode config set -wakeonlan XX:XX:XX:XX:XX:XX

3.10 Certificate Management

3.10.1 Certificates for Intra-Cluster Communication

Each Proxmox VE cluster creates by default its own (self-signed) Certificate Authority (CA) and generates
a certificate for each node which gets signed by the aforementioned CA. These certificates are used for
encrypted communication with the cluster’s pveproxy service and the Shell/Console feature if SPICE is
used.
The CA certificate and key are stored in the Proxmox Cluster File System (pmxcfs) Chapter 6.

3.10.2 Certificates for API and Web GUI

The REST API and web GUI are provided by the pveproxy service, which runs on each node.
You have the following options for the certificate used by pveproxy:

1. By default the node-specific certificate in /etc/pve/nodes/NODENAME/pve-ssl.pem is used.

This certificate is signed by the cluster CA and therefore not automatically trusted by browsers and
operating systems.

2. use an externally provided certificate (e.g. signed by a commercial CA).

3. use ACME (Let’s Encrypt) to get a trusted certificate with automatic renewal, this is also integrated in
the Proxmox VE API and Webinterface.
Proxmox VE Administration Guide 58 / 464

For options 2 and 3 the file /etc/pve/local/pveproxy-ssl.pem (and /etc/pve/local/pveproxy-

which needs to be without password) is used.

Note
Keep in mind that /etc/pve/local is a node specific symlink to /etc/pve/nodes/NODENAME.

Certificates are managed with the Proxmox VE Node management command (see the pvenode(1) man-
page).

Warning
Do not replace or manually modify the automatically generated node certificate files in
/etc/pve/local/pve-ssl.pem and /etc/pve/local/pve-ssl.key or the cluster
CA files in /etc/pve/pve-root-ca.pem and /etc/pve/priv/pve-root-ca.key.

3.10.3 Upload Custom Certificate

If you already have a certificate which you want to use for a Proxmox VE node you can upload that certificate
simply over the web interface.

Note that the certificates key file, if provided, mustn’t be password protected.

3.10.4 Trusted certificates via Let’s Encrypt (ACME)

Proxmox VE includes an implementation of the Automatic Certificate Management Environment ACME pro-
tocol, allowing Proxmox VE admins to interface with Let’s Encrypt for easy setup of trusted TLS certificates
which are accepted out of the box on most modern operating systems and browsers.
Currently the two ACME endpoints implemented are the Let’s Encrypt (LE) production and its staging en-
vironment. Our ACME client supports validation of http-01 challenges using a built-in webserver and
validation of dns-01 challenges using a DNS plugin supporting all the DNS API endpoints acme.sh does.
Proxmox VE Administration Guide 59 / 464

ACME Account

You need to register an ACME account per cluster with the endpoint you want to use. The email address
used for that account will server as contact point for renewal-due or similar notifications from the ACME
endpoint.
You can register and deactivate ACME accounts over the web interface Datacenter -> ACME or using
the pvenode command line tool.

pvenode acme account register account-name [email protected]

Tip
Because of rate-limits you should use LE staging for experiments or if you use ACME for the first time.

ACME Plugins

The ACME plugins task is to provide automatic verification that you, and thus the Proxmox VE cluster under
your operation, are the real owner of a domain. This is the basis building block for automatic certificate
management.
The ACME protocol specifies different types of challenges, for example the http-01 where a webserver
provides a file with a certain value to prove that it controls a domain. Sometimes this isn’t possible, either
because of technical limitations or if the address a domain points to is not reachable from the public internet.
For such cases, one could use the dns-01 challenge. This challenge also provides a certain value, but
through a DNS record on the authority name server of the domain, rather than over a text file.
Proxmox VE Administration Guide 60 / 464

Proxmox VE supports both of those challenge types out of the box, you can configure plugins either over the
web interface under Datacenter -> ACME, or using the pvenode acme plugin add command.
ACME Plugin configurations are stored in /etc/pve/priv/acme/plugins.cfg. A plugin is available
for all nodes in the cluster.

Node Domains

Each domain is node specific. You can add new or manage existing domain entries under Node ->
Certificates, or using the pvenode config command.

After configuring the desired domain(s) for a node and ensuring that the desired ACME account is selected,
you can order your new certificate over the web-interface. On success the interface will reload after 10
seconds.
Renewal will happen automatically Section 3.10.7.

3.10.5 ACME HTTP Challenge Plugin

There is always an implicitly configured standalone plugin for validating http-01 challenges via the
built-in webserver spawned on port 80.
Proxmox VE Administration Guide 61 / 464

Note
The name standalone means that it can provide the validation on it’s own, without any third party
service. So, this plugin works also for cluster nodes.

There are a few prerequisites to use it for certificate management with Let’s Encrypts ACME.

• You have to accept the ToS of Let’s Encrypt to register an account.

• Port 80 of the node needs to be reachable from the internet.

• There must be no other listener on port 80.

• The requested (sub)domain needs to resolve to a public IP of the Node.

3.10.6 ACME DNS API Challenge Plugin

On systems where external access for validation via the http-01 method is not possible or desired, it is
possible to use the dns-01 validation method. This validation method requires a DNS server that allows
provisioning of TXT records via an API.

Configuring ACME DNS APIs for validation

Proxmox VE re-uses the DNS plugins developed for the acme.sh 2 project, please refer to its documenta-
tion for details on configuration of specific APIs.
The easiest way to configure a new plugin with the DNS API is using the web interface (Datacenter ->
ACME).

Choose DNS as challenge type. Then you can select your API provider, enter the credential data to access
your account over their API.

Tip
See the acme.sh How to use DNS API wiki for more detailed information about getting API credentials for
your provider.

As there are so many API endpoints Proxmox VE autogenerates the form for the credentials, but not all
providers are annotated yet. For those you will see a bigger text area, simply copy all the credentials
KEY=VALUE pairs in there.
2 acme.sh https://github.com/acmesh-official/acme.sh
Proxmox VE Administration Guide 62 / 464

DNS Validation through CNAME Alias

A special alias mode can be used to handle the validation on a different domain/DNS server, in case your
primary/real DNS does not support provisioning via an API. Manually set up a permanent CNAME record for
_acme-challenge.domain1.example pointing to _acme-challenge.domain2.example and
set the alias property in the Proxmox VE node configuration file to domain2.example to allow the DNS
server of domain2.example to validate all challenges for domain1.example.

Combination of Plugins

Combining http-01 and dns-01 validation is possible in case your node is reachable via multiple do-
mains with different requirements / DNS provisioning capabilities. Mixing DNS APIs from multiple providers
or instances is also possible by specifying different plugin instances per domain.

Tip
Accessing the same service over multiple domains increases complexity and should be avoided if possible.

3.10.7 Automatic renewal of ACME certificates

If a node has been successfully configured with an ACME-provided certificate (either via pvenode or via
the GUI), the certificate will be automatically renewed by the pve-daily-update.service. Currently,
renewal will be attempted if the certificate has expired already, or will expire in the next 30 days.

3.10.8 ACME Examples with pvenode

Example: Sample pvenode invocation for using Let’s Encrypt certificates

root@proxmox:~# pvenode acme account register default [email protected]

Directory endpoints:
0) Let’s Encrypt V2 (https://acme-v02.api.letsencrypt.org/directory)
1) Let’s Encrypt V2 Staging (https://acme-staging-v02.api.letsencrypt.org/ ←-
directory)
2) Custom
Enter selection: 1

Terms of Service: https://letsencrypt.org/documents/LE-SA-v1.2-November ←-

-15-2017.pdf
Do you agree to the above terms? [y|N]y
...
Task OK
root@proxmox:~# pvenode config set --acme domains=example.invalid
root@proxmox:~# pvenode acme cert order
Loading ACME account details
Placing ACME order
...
Status is ’valid’!
Proxmox VE Administration Guide 63 / 464

All domains validated!

...
Downloading certificate
Setting pveproxy certificate and key
Restarting pveproxy
Task OK

Example: Setting up the OVH API for validating a domain

Note
the account registration steps are the same no matter which plugins are used, and are not repeated here.

Note
OVH_AK and OVH_AS need to be obtained from OVH according to the OVH API documentation

First you need to get all information so you and Proxmox VE can access the API.

root@proxmox:~# cat /path/to/api-token

OVH_AK=XXXXXXXXXXXXXXXX
OVH_AS=YYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYY
root@proxmox:~# source /path/to/api-token
root@proxmox:~# curl -XPOST -H"X-Ovh-Application: $OVH_AK" -H "Content-type ←-
: application/json" \
https://eu.api.ovh.com/1.0/auth/credential -d ’{
"accessRules": [
{"method": "GET","path": "/auth/time"},
{"method": "GET","path": "/domain"},
{"method": "GET","path": "/domain/zone/*"},
{"method": "GET","path": "/domain/zone/*/record"},
{"method": "POST","path": "/domain/zone/*/record"},
{"method": "POST","path": "/domain/zone/*/refresh"},
{"method": "PUT","path": "/domain/zone/*/record/"},
{"method": "DELETE","path": "/domain/zone/*/record/*"}
]
}’
{"consumerKey":"ZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZ","state":" ←-
pendingValidation","validationUrl":"https://eu.api.ovh.com/auth/? ←-
credentialToken= ←-
AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA"}

(open validation URL and follow instructions to link Application Key with ←-
account/Consumer Key)

root@proxmox:~# echo "OVH_CK=ZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZ" >> /path/to/ ←-

api-token
Proxmox VE Administration Guide 64 / 464

Now you can setup the the ACME plugin:

root@proxmox:~# pvenode acme plugin add dns example_plugin --api ovh --data ←-
/path/to/api_token
root@proxmox:~# pvenode acme plugin config example_plugin
&#x250c;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x252c;&#x
&#x2502; key &#x2502; value &#x2502;
&#x255e;&#x2550;&#x2550;&#x2550;&#x2550;&#x2550;&#x2550;&#x2550;&#x2550;&#x256a;&#x
&#x2502; api &#x2502; ovh &#x2502;
&#x251c;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x253c;&#x
&#x2502; data &#x2502; OVH_AK=XXXXXXXXXXXXXXXX &#x2502;
&#x2502; &#x2502; OVH_AS=YYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYY &#x2502;
&#x2502; &#x2502; OVH_CK=ZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZ &#x2502;
&#x251c;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x253c;&#x
&#x2502; digest &#x2502; 867fcf556363ca1bea866863093fcab83edf47a1 &#x2502;
&#x251c;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x253c;&#x
&#x2502; plugin &#x2502; example_plugin &#x2502;
&#x251c;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x253c;&#x
&#x2502; type &#x2502; dns &#x2502;
&#x2514;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2534;&#x

At last you can configure the domain you want to get certitficates for and place the certificate order for it:

root@proxmox:~# pvenode config set -acmedomain0 example.proxmox.com,plugin= ←-

example_plugin
root@proxmox:~# pvenode acme cert order
Loading ACME account details
Placing ACME order
Order URL: https://acme-staging-v02.api.letsencrypt.org/acme/order ←-
/11111111/22222222

Getting authorization details from ’https://acme-staging-v02.api. ←-

letsencrypt.org/acme/authz-v3/33333333’
The validation for example.proxmox.com is pending!
[Wed Apr 22 09:25:30 CEST 2020] Using OVH endpoint: ovh-eu
[Wed Apr 22 09:25:30 CEST 2020] Checking authentication
[Wed Apr 22 09:25:30 CEST 2020] Consumer key is ok.
[Wed Apr 22 09:25:31 CEST 2020] Adding record
[Wed Apr 22 09:25:32 CEST 2020] Added, sleep 10 seconds.
Add TXT record: _acme-challenge.example.proxmox.com
Triggering validation
Sleeping for 5 seconds
Status is ’valid’!
[Wed Apr 22 09:25:48 CEST 2020] Using OVH endpoint: ovh-eu
[Wed Apr 22 09:25:48 CEST 2020] Checking authentication
[Wed Apr 22 09:25:48 CEST 2020] Consumer key is ok.
Remove TXT record: _acme-challenge.example.proxmox.com

All domains validated!

Creating CSR
Proxmox VE Administration Guide 65 / 464

Checking order status

Order is ready, finalizing order
valid!

Downloading certificate
Setting pveproxy certificate and key
Restarting pveproxy
Task OK

Example: Switching from the staging to the regular ACME directory

Changing the ACME directory for an account is unsupported, but as Proxmox VE supports more than one
account you can just create a new one with the production (trusted) ACME directory as endpoint. You can
also deactivate the staging account and recreate it.

Example: Changing the default ACME account from staging to directory using pvenode

root@proxmox:~# pvenode acme account deactivate default

Renaming account file from ’/etc/pve/priv/acme/default’ to ’/etc/pve/priv/ ←-
acme/_deactivated_default_4’
Task OK

root@proxmox:~# pvenode acme account register default [email protected]

Directory endpoints:
0) Let’s Encrypt V2 (https://acme-v02.api.letsencrypt.org/directory)
1) Let’s Encrypt V2 Staging (https://acme-staging-v02.api.letsencrypt.org/ ←-
directory)
2) Custom
Enter selection: 0

Terms of Service: https://letsencrypt.org/documents/LE-SA-v1.2-November ←-

-15-2017.pdf
Do you agree to the above terms? [y|N]y
...
Task OK

3.11 Host Bootloader

Proxmox VE currently uses one of two bootloaders depending on the disk setup selected in the installer.
For EFI Systems installed with ZFS as the root filesystem systemd-boot is used. All other deployments
use the standard grub bootloader (this usually also applies to systems which are installed on top of Debian).

3.11.1 Partitioning Scheme Used by the Installer

The Proxmox VE installer creates 3 partitions on the bootable disks selected for installation. The bootable
disks are:
Proxmox VE Administration Guide 66 / 464

• For Installations with ext4 or xfs the selected disk

• For ZFS installations all disks belonging to the first vdev:

– The first disk for RAID0

– All disks for RAID1, RAIDZ1, RAIDZ2, RAIDZ3
– The first two disks for RAID10

The created partitions are:

• a 1 MB BIOS Boot Partition (gdisk type EF02)

• a 512 MB EFI System Partition (ESP, gdisk type EF00)

• a third partition spanning the set hdsize parameter or the remaining space used for the chosen storage
type

grub in BIOS mode (--target i386-pc) is installed onto the BIOS Boot Partition of all bootable disks
for supporting older systems.

3.11.2 Determine which Bootloader is Used

Proxmox VE Administration Guide 67 / 464

The simplest and most reliable way to determine which bootloader is used, is to watch the boot process of
the Proxmox VE node.
You will either see the blue box of grub or the simple black on white systemd-boot.

Determining the bootloader from a running system might not be 100% accurate. The safest way is to run the
following command:

# efibootmgr -v

If it returns a message that EFI variables are not supported, grub is used in BIOS/Legacy mode.
If the output contains a line that looks similar to the following, grub is used in UEFI mode.

Boot0005* proxmox [...] File(\EFI\proxmox\grubx64.efi)

If the output contains a line similar to the following, systemd-bood is used.

Boot0006* Linux Boot Manager [...] File(\EFI\systemd\systemd-bootx64.efi ←-

)
Proxmox VE Administration Guide 68 / 464

3.11.3 Grub

grub has been the de-facto standard for booting Linux systems for many years and is quite well documented
3.

The kernel and initrd images are taken from /boot and its configuration file /boot/grub/grub.cfg
gets updated by the kernel installation process.

Configuration

Changes to the grub configuration are done via the defaults file /etc/default/grub or config snip-
pets in /etc/default/grub.d. To regenerate the /boot/grub/grub.cfg after a change to the
configuration run:

‘update-grub‘.

3.11.4 Systemd-boot

systemd-boot is a lightweight EFI bootloader. It reads the kernel and initrd images directly from the EFI
Service Partition (ESP) where it is installed. The main advantage of directly loading the kernel from the ESP
is that it does not need to reimplement the drivers for accessing the storage. In the context of ZFS as root
filesystem this means that you can use all optional features on your root pool instead of the subset which is
also present in the ZFS implementation in grub or having to create a separate small boot-pool 4 .
In setups with redundancy (RAID1, RAID10, RAIDZ*) all bootable disks (those being part of the first vdev)
are partitioned with an ESP. This ensures the system boots even if the first boot device fails. The ESPs are
kept in sync by a kernel postinstall hook script /etc/kernel/postinst.d/zz-pve-efiboot. The
script copies certain kernel versions and the initrd images to EFI/proxmox/ on the root of each ESP and
creates the appropriate config files in loader/entries/proxmox-*.conf. The pve-efiboot-tool
script assists in managing both the synced ESPs themselves and their contents.
The following kernel versions are configured by default:

• the currently running kernel

• the version being newly installed on package updates

• the two latest already installed kernels

• the latest version of the second-to-last kernel series (e.g. 4.15, 5.0), if applicable

• any manually selected kernels (see below)

The ESPs are not kept mounted during regular operation, in contrast to grub, which keeps an ESP mounted
on /boot/efi. This helps to prevent filesystem corruption to the vfat formatted ESPs in case of a
system crash, and removes the need to manually adapt /etc/fstab in case the primary boot device fails.
3 Grub Manual https://www.gnu.org/software/grub/manual/grub/grub.html
4 Booting ZFS on root with grub https://github.com/zfsonlinux/zfs/wiki/Debian-Stretch-Root-on-ZFS
Proxmox VE Administration Guide 69 / 464

Configuration

systemd-boot is configured via the file loader/loader.conf in the root directory of an EFI System
Partition (ESP). See the loader.conf(5) manpage for details.
Each bootloader entry is placed in a file of its own in the directory loader/entries/
An example entry.conf looks like this (/ refers to the root of the ESP):

title Proxmox
version 5.0.15-1-pve
options root=ZFS=rpool/ROOT/pve-1 boot=zfs
linux /EFI/proxmox/5.0.15-1-pve/vmlinuz-5.0.15-1-pve
initrd /EFI/proxmox/5.0.15-1-pve/initrd.img-5.0.15-1-pve

Manually keeping a kernel bootable

Should you wish to add a certain kernel and initrd image to the list of bootable kernels use pve-efiboot-tool
kernel add.
For example run the following to add the kernel with ABI version 5.0.15-1-pve to the list of kernels to
keep installed and synced to all ESPs:

pve-efiboot-tool kernel add 5.0.15-1-pve

pve-efiboot-tool kernel list will list all kernel versions currently selected for booting:

# pve-efiboot-tool kernel list

Manually selected kernels:
5.0.15-1-pve

Automatically selected kernels:

5.0.12-1-pve
4.15.18-18-pve

Run pve-efiboot-tool remove to remove a kernel from the list of manually selected kernels, for
example:

pve-efiboot-tool kernel remove 5.0.15-1-pve

Note
It’s required to run pve-efiboot-tool refresh to update all EFI System Partitions (ESPs) after a
manual kernel addition or removal from above.
Proxmox VE Administration Guide 70 / 464

Setting up a new partition for use as synced ESP

To format and initialize a partition as synced ESP, e.g., after replacing a failed vdev in an rpool, or when con-
verting an existing system that pre-dates the sync mechanism, pve-efiboot-tool from pve-kernel-help
can be used.

Warning
the format command will format the <partition>, make sure to pass in the right device/par-
tition!

For example, to format an empty partition /dev/sda2 as ESP, run the following:

pve-efiboot-tool format /dev/sda2

To setup an existing, unmounted ESP located on /dev/sda2 for inclusion in Proxmox VE’s kernel update
synchronization mechanism, use the following:

pve-efiboot-tool init /dev/sda2

Afterwards /etc/kernel/pve-efiboot-uuids should contain a new line with the UUID of the newly
added partition. The init command will also automatically trigger a refresh of all configured ESPs.

Updating the configuration on all ESPs

To copy and configure all bootable kernels and keep all ESPs listed in /etc/kernel/pve-efiboot-uuids
in sync you just need to run:

pve-efiboot-tool refresh

(The equivalent to running update-grub on systems being booted with grub).

This is necessary should you make changes to the kernel commandline, or want to sync all kernels and
initrds.

Note
Both update-initramfs and apt (when necessary) will automatically trigger a refresh.

3.11.5 Editing the Kernel Commandline

You can modify the kernel commandline in the following places, depending on the bootloader used:

Grub

The kernel commandline needs to be placed in the variable GRUB_CMDLINE_LINUX_DEFAULT in the file
/etc/default/grub. Running update-grub appends its content to all linux entries in /boot/grub/g
Proxmox VE Administration Guide 71 / 464

Systemd-boot

The kernel commandline needs to be placed as one line in /etc/kernel/cmdline. To apply your
changes, run pve-efiboot-tool refresh, which sets it as the option line for all config files in
loader/entries/proxmox-*.conf.
Proxmox VE Administration Guide 72 / 464

Chapter 4

Graphical User Interface

Proxmox VE is simple. There is no need to install a separate management tool, and everything can be done
through your web browser (Latest Firefox or Google Chrome is preferred). A built-in HTML5 console is used
to access the guest console. As an alternative, SPICE can be used.
Because we use the Proxmox cluster file system (pmxcfs), you can connect to any node to manage the
entire cluster. Each node can manage the entire cluster. There is no need for a dedicated manager node.
You can use the web-based administration interface with any modern browser. When Proxmox VE detects
that you are connecting from a mobile device, you are redirected to a simpler, touch-based user interface.
The web interface can be reached via https://youripaddress:8006 (default login is: root, and the password is
specified during the installation process).

4.1 Features

• Seamless integration and management of Proxmox VE clusters

• AJAX technologies for dynamic updates of resources

• Secure access to all Virtual Machines and Containers via SSL encryption (https)

• Fast search-driven interface, capable of handling hundreds and probably thousands of VMs

• Secure HTML5 console or SPICE

• Role based permission management for all objects (VMs, storages, nodes, etc.)

• Support for multiple authentication sources (e.g. local, MS ADS, LDAP, . . . )

• Two-Factor Authentication (OATH, Yubikey)

• Based on ExtJS 6.x JavaScript framework

Proxmox VE Administration Guide 73 / 464

4.2 Login

When you connect to the server, you will first see the login window. Proxmox VE supports various authen-
tication backends (Realm), and you can select the language here. The GUI is translated to more than 20
languages.

Note
You can save the user name on the client side by selecting the checkbox at the bottom. This saves some
typing when you login next time.

4.3 GUI Overview

Proxmox VE Administration Guide 74 / 464

The Proxmox VE user interface consists of four regions.

Header On top. Shows status information and contains buttons for most important actions.

Resource Tree At the left side. A navigation tree where you can select specific objects.

Content Panel Center region. Selected objects display configuration options and status here.

Log Panel At the bottom. Displays log entries for recent tasks. You can double-click on those
log entries to get more details, or to abort a running task.

Note
You can shrink and expand the size of the resource tree and log panel, or completely hide the log panel.
This can be helpful when you work on small displays and want more space to view other content.

4.3.1 Header

On the top left side, the first thing you see is the Proxmox logo. Next to it is the current running version of
Proxmox VE. In the search bar nearside you can search for specific objects (VMs, containers, nodes, . . . ).
This is sometimes faster than selecting an object in the resource tree.
To the right of the search bar we see the identity (login name). The gear symbol is a button opening the
My Settings dialog. There you can customize some client side user interface setting (reset the saved login
name, reset saved layout).
The rightmost part of the header contains four buttons:

Help Opens a new browser window showing the reference documentation.

Create VM Opens the virtual machine creation wizard.

Create CT Open the container creation wizard.

Logout Logout, and show the login dialog again.

Proxmox VE Administration Guide 75 / 464

4.3.2 My Settings

The My Settings window allows you to set locally stored settings. These include the Dashboard Storages
which allow you to enable or disable specific storages to be counted towards the total amount visible in the
datacenter summary. If no storage is checked the total is the sum of all storages, same as enabling every
single one.
Below the dashboard settings you find the stored user name and a button to clear it as well as a button to
reset every layout in the GUI to its default.
On the right side there are xterm.js Settings. These contain the following options:

Font-Family The font to be used in xterm.js (e.g. Arial).

Font-Size The preferred font size to be used.

Letter Spacing Increases or decreases spacing between letters in text.

Line Height Specify the absolute height of a line.

4.3.3 Resource Tree

This is the main navigation tree. On top of the tree you can select some predefined views, which changes
the structure of the tree below. The default view is Server View, and it shows the following object types:

Datacenter Contains cluster wide setting (relevant for all nodes).

Node Represents the hosts inside a cluster, where the guests runs.

Guest VMs, Containers and Templates.

Storage Data Storage.

Proxmox VE Administration Guide 76 / 464

Pool It is possible to group guests using a pool to simplify management.

The following view types are available:

Server View Shows all kind of objects, grouped by nodes.

Folder View Shows all kind of objects, grouped by object type.

Storage View Only show storage objects, grouped by nodes.

Pool View Show VMs and Containers, grouped by pool.

4.3.4 Log Panel

The main purpose of the log panel is to show you what is currently going on in your cluster. Actions like
creating an new VM are executed in background, and we call such background job a task.
Any output from such task is saved into a separate log file. You can view that log by simply double-click a
task log entry. It is also possible to abort a running task there.
Please note that we display most recent tasks from all cluster nodes here. So you can see when somebody
else is working on another cluster node in real-time.

Note
We remove older and finished task from the log panel to keep that list short. But you can still find those
tasks in the Task History within the node panel.

Some short running actions simply sends logs to all cluster members. You can see those messages in the
Cluster log panel.

4.4 Content Panels

When you select something in the resource tree, the corresponding object displays configuration and status
information in the content panel. The following sections give a brief overview of the functionality. Please refer
to the individual chapters inside the reference documentation to get more detailed information.
Proxmox VE Administration Guide 77 / 464

4.4.1 Datacenter

On the datacenter level you can access cluster wide settings and information.

• Search: it is possible to search anything in cluster ,this can be a node, VM, Container, Storage or a pool.

• Summary: gives a brief overview over the cluster health.

• Cluster: allows to create/join cluster and shows join information.

• Options: can show and set defaults, which apply cluster wide.

• Storage: is the place where a storage will add/managed/removed.

• Backup: has the capability to schedule Backups. This is cluster wide, so you do not care about where the
VM/Container are on your cluster at schedule time.

• Replication: shows replication jobs and allows to create new ones.

• Permissions: will manage user and group permission, LDAP, MS-AD and Two-Factor authentication can
be setup here.

• HA: will manage the Proxmox VE High-Availability

Proxmox VE Administration Guide 78 / 464

• Firewall: on this level the Proxmox Firewall works cluster wide and makes templates which are cluster
wide available.

• Support: here you get all information about your support subscription.

If you like to have more information about this see the corresponding chapter.

4.4.2 Nodes

Nodes in your cluster can be managed invidiually at this level.

The top header has useful buttons such as Reboot, Shutdown, Shell, Bulk Actions and Help. Shell has the
options noVNC, SPICE and xterm.js. Bulk Actions has the options Bulk Start, Bulk Stop and Bulk Migrate.

• Search: it is possible to search anything on the node, this can be a VM, Container, Storage or a pool.

• Summary: gives a brief overview over the resource usage.

• Notes: is where custom notes about a node can be written.

• Shell: logs you into the shell of the node.

Proxmox VE Administration Guide 79 / 464

• System: is for configuring the network, DNS and time, and also shows your syslog.

• Updates: will upgrade the system and inform you about new packages.

• Firewall: on this level is only for this node.

• Disks: gives you a brief overview about you physical hard drives and how they are used.

• Ceph: is only used if you have installed a Ceph server on your host. Then you can manage your Ceph
cluster and see the status of it here.

• Replication: shows replication jobs and allows to create new ones.

• Task History: here all past tasks are shown.

• Subscription: here you can upload you subscription key and get a system overview in case of a support
case.

4.4.3 Guests

There are two different kinds of guests and both can be converted to a template. One of them is a Kernel-
based Virtual Machine (KVM) and the other one a Linux Container (LXC). Generally the navigation is the
same, only some options are different.
In the main management center the VM navigation begins if a VM is selected in the left tree.
The top header contains important VM operation commands like Start, Shutdown, Reset, Remove, Migrate,
Console and Help. Some of them have hidden buttons like Shutdown has Stop and Console contains the
different console types SPICE, noVNC and xterm.js.
Proxmox VE Administration Guide 80 / 464

On the right side the content switches depending on the selected option.
On the left side. All available options are listed one below the other.

• Summary: gives a brief overview over the VM activity.

• Console: an interactive console to your VM.

• (KVM)Hardware: shows and set the Hardware of the KVM VM.

• (LXC)Resources: defines the LXC Hardware opportunities.

• (LXC)Network: the LXC Network settings.

• (LXC)DNS: the LXC DNS settings.

• Options: all guest options can be set here.

• Task History: here all previous tasks from the selected guest will be shown.

• (KVM) Monitor: is the interactive communication interface to the KVM process.

• Backup: shows the available backups from the selected guest and also create a backupset.

• Replication: shows the replication jobs for the selected guest and allows to create new jobs.

• Snapshots: manage VM snapshots.

• Firewall: manage the firewall on VM level.

• Permissions: manage the user permission for the selected guest.

Proxmox VE Administration Guide 81 / 464

4.4.4 Storage

In this view we have a two partition split-view. On the left side we have the storage options and on the right
side the content of the selected option will be shown.

• Summary: shows important information about storages like Usage, Type, Content, Active and Enabled.

• Content: Here all content will be listed grouped by content type.

• Permissions: manage the user permission for this storage.

Proxmox VE Administration Guide 82 / 464

4.4.5 Pools

In this view we have a two partition split view. On the left side we have the logical pool options and on the
right side the content of the selected option will be shown.

• Summary: show the description of the pool.

• Members: Here all members of this pool will listed and can be managed.

• Permissions: manage the user permission for this pool.

Proxmox VE Administration Guide 83 / 464

Chapter 5

Cluster Manager

The Proxmox VE cluster manager pvecm is a tool to create a group of physical servers. Such a group is
called a cluster. We use the Corosync Cluster Engine for reliable group communication, and such clusters
can consist of up to 32 physical nodes (probably more, dependent on network latency).
pvecm can be used to create a new cluster, join nodes to a cluster, leave the cluster, get status informa-
tion and do various other cluster related tasks. The Proxmox Cluster File System (“pmxcfs”) is used to
transparently distribute the cluster configuration to all cluster nodes.
Grouping nodes into a cluster has the following advantages:

• Centralized, web based management

• Multi-master clusters: each node can do all management tasks

• pmxcfs: database-driven file system for storing configuration files, replicated in real-time on all nodes
using corosync.

• Easy migration of virtual machines and containers between physical hosts

• Fast deployment

• Cluster-wide services like firewall and HA

5.1 Requirements

• All nodes must be able to connect to each other via UDP ports 5404 and 5405 for corosync to work.

• Date and time have to be synchronized.

• SSH tunnel on TCP port 22 between nodes is used.

• If you are interested in High Availability, you need to have at least three nodes for reliable quorum. All
nodes should have the same version.

• We recommend a dedicated NIC for the cluster traffic, especially if you use shared storage.

• Root password of a cluster node is required for adding nodes.