P ROXMOX VE A DMINISTRATION G UIDE

R ELEASE 6.3

November 25, 2020

Proxmox Server Solutions Gmbh
www.proxmox.com
Proxmox VE Administration Guide ii

Copyright © 2020 Proxmox Server Solutions Gmbh

Permission is granted to copy, distribute and/or modify this document under the terms of the GNU Free
Documentation License, Version 1.3 or any later version published by the Free Software Foundation; with
no Invariant Sections, no Front-Cover Texts, and no Back-Cover Texts.
A copy of the license is included in the section entitled "GNU Free Documentation License".
Proxmox VE Administration Guide iii

Contents

1 Introduction 1
1.1 Central Management . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2
1.2 Flexible Storage . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3
1.3 Integrated Backup and Restore . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3
1.4 High Availability Cluster . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3
1.5 Flexible Networking . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4
1.6 Integrated Firewall . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4
1.7 Hyper-converged Infrastructure . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4
1.7.1 Benefits of a Hyper-Converged Infrastructure (HCI) with Proxmox VE . . . . . . . . . . 4
1.7.2 Hyper-Converged Infrastructure: Storage . . . . . . . . . . . . . . . . . . . . . . . . 5
1.8 Why Open Source . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5
1.9 Your benefits with Proxmox VE . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5
1.10 Getting Help . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6
1.10.1 Proxmox VE Wiki . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6
1.10.2 Community Support Forum . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6
1.10.3 Mailing Lists . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6
1.10.4 Commercial Support . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6
1.10.5 Bug Tracker . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6
1.11 Project History . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6
1.12 Improving the Proxmox VE Documentation . . . . . . . . . . . . . . . . . . . . . . . . . . . 7
1.13 Translating Proxmox VE . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7
1.13.1 Translating with git . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8
1.13.2 Translating without git . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8
1.13.3 Testing the Translation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8
1.13.4 Sending the Translation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9
Proxmox VE Administration Guide iv

2 Installing Proxmox VE 10
2.1 System Requirements . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10
2.1.1 Minimum Requirements, for Evaluation . . . . . . . . . . . . . . . . . . . . . . . . . . 10
2.1.2 Recommended System Requirements . . . . . . . . . . . . . . . . . . . . . . . . . . 11
2.1.3 Simple Performance Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11
2.1.4 Supported Web Browsers for Accessing the Web Interface . . . . . . . . . . . . . . . 11
2.2 Prepare Installation Media . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12
2.2.1 Prepare a USB Flash Drive as Installation Medium . . . . . . . . . . . . . . . . . . . 12
2.2.2 Instructions for GNU/Linux . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12
2.2.3 Instructions for macOS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13
2.2.4 Instructions for Windows . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14
2.3 Using the Proxmox VE Installer . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14
2.3.1 Advanced LVM Configuration Options . . . . . . . . . . . . . . . . . . . . . . . . . . 21
2.3.2 Advanced ZFS Configuration Options . . . . . . . . . . . . . . . . . . . . . . . . . . 22
2.3.3 ZFS Performance Tips . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23
2.4 Install Proxmox VE on Debian . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23

3 Host System Administration 24

3.1 Package Repositories . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24
3.1.1 Proxmox VE Enterprise Repository . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25
3.1.2 Proxmox VE No-Subscription Repository . . . . . . . . . . . . . . . . . . . . . . . . . 25
3.1.3 Proxmox VE Test Repository . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25
3.1.4 Ceph Octopus Repository . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26
3.1.5 Ceph Octopus Test Repository . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26
3.1.6 Ceph Nautilus Repository . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26
3.1.7 Ceph Nautilus Test Repository . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27
3.1.8 Proxmox VE Ceph Luminous Repository For Upgrade . . . . . . . . . . . . . . . . . . 27
3.1.9 SecureApt . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27
3.2 System Software Updates . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28
3.3 Network Configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28
3.3.1 Apply Network Changes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29
3.3.2 Naming Conventions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29
3.3.3 Choosing a network configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . 30
3.3.4 Default Configuration using a Bridge . . . . . . . . . . . . . . . . . . . . . . . . . . . 31
3.3.5 Routed Configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32
3.3.6 Masquerading (NAT) with iptables . . . . . . . . . . . . . . . . . . . . . . . . . . 33
Proxmox VE Administration Guide v

3.3.7 Linux Bond . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 34

3.3.8 VLAN 802.1Q . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 37
3.4 Time Synchronization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 39
3.4.1 Using Custom NTP Servers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 39
3.5 External Metric Server . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 40
3.5.1 Graphite server configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41
3.5.2 Influxdb plugin configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41
3.6 Disk Health Monitoring . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41
3.7 Logical Volume Manager (LVM) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 42
3.7.1 Hardware . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 43
3.7.2 Bootloader . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 43
3.7.3 Creating a Volume Group . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 43
3.7.4 Creating an extra LV for /var/lib/vz . . . . . . . . . . . . . . . . . . . . . . . . 44
3.7.5 Resizing the thin pool . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 44
3.7.6 Create a LVM-thin pool . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 45
3.8 ZFS on Linux . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 45
3.8.1 Hardware . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 46
3.8.2 Installation as Root File System . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 46
3.8.3 ZFS RAID Level Considerations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 47
3.8.4 Bootloader . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 49
3.8.5 ZFS Administration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 49
3.8.6 Activate E-Mail Notification . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 52
3.8.7 Limit ZFS Memory Usage . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 52
3.8.8 SWAP on ZFS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 53
3.8.9 Encrypted ZFS Datasets . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 53
3.8.10 Compression in ZFS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 55
3.8.11 ZFS Special Device . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 55
3.9 Proxmox Node Management . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 56
3.9.1 Wake-on-LAN . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 57
3.10 Certificate Management . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 57
3.10.1 Certificates for Intra-Cluster Communication . . . . . . . . . . . . . . . . . . . . . . . 57
3.10.2 Certificates for API and Web GUI . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 57
3.10.3 Upload Custom Certificate . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 58
3.10.4 Trusted certificates via Let’s Encrypt (ACME) . . . . . . . . . . . . . . . . . . . . . . 58
3.10.5 ACME HTTP Challenge Plugin . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 60
Proxmox VE Administration Guide vi

3.10.6 ACME DNS API Challenge Plugin . . . . . . . . . . . . . . . . . . . . . . . . . . . . 61

3.10.7 Automatic renewal of ACME certificates . . . . . . . . . . . . . . . . . . . . . . . . . 62
3.10.8 ACME Examples with pvenode . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 62
3.11 Host Bootloader . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 65
3.11.1 Partitioning Scheme Used by the Installer . . . . . . . . . . . . . . . . . . . . . . . . 65
3.11.2 Determine which Bootloader is Used . . . . . . . . . . . . . . . . . . . . . . . . . . . 66
3.11.3 Grub . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 68
3.11.4 Systemd-boot . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 68
3.11.5 Editing the Kernel Commandline . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 70

4 Graphical User Interface 72

4.1 Features . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 72
4.2 Login . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 73
4.3 GUI Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 73
4.3.1 Header . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 74
4.3.2 My Settings . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 75
4.3.3 Resource Tree . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 75
4.3.4 Log Panel . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 76
4.4 Content Panels . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 76
4.4.1 Datacenter . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 77
4.4.2 Nodes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 78
4.4.3 Guests . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 79
4.4.4 Storage . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 81
4.4.5 Pools . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 82

5 Cluster Manager 83
5.1 Requirements . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 83
5.2 Preparing Nodes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 84
5.3 Create a Cluster . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 84
5.3.1 Create via Web GUI . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 84
5.3.2 Create via Command Line . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 85
5.3.3 Multiple Clusters In Same Network . . . . . . . . . . . . . . . . . . . . . . . . . . . . 85
5.4 Adding Nodes to the Cluster . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 85
5.4.1 Join Node to Cluster via GUI . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 86
5.4.2 Join Node to Cluster via Command Line . . . . . . . . . . . . . . . . . . . . . . . . . 86
5.4.3 Adding Nodes With Separated Cluster Network . . . . . . . . . . . . . . . . . . . . . 88
Proxmox VE Administration Guide vii

5.5 Remove a Cluster Node . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 88

5.5.1 Separate A Node Without Reinstalling . . . . . . . . . . . . . . . . . . . . . . . . . . 89
5.6 Quorum . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 91
5.7 Cluster Network . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 91
5.7.1 Network Requirements . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 91
5.7.2 Separate Cluster Network . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 92
5.7.3 Corosync addresses . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 95
5.8 Corosync Redundancy . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 96
5.8.1 Adding Redundant Links To An Existing Cluster . . . . . . . . . . . . . . . . . . . . . 96
5.9 Role of SSH in Proxmox VE Clusters . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 98
5.10 Corosync External Vote Support . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 99
5.10.1 QDevice Technical Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 99
5.10.2 Supported Setups . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 99
5.10.3 QDevice-Net Setup . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 100
5.10.4 Frequently Asked Questions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 101
5.11 Corosync Configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 101
5.11.1 Edit corosync.conf . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 102
5.11.2 Troubleshooting . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 103
5.11.3 Corosync Configuration Glossary . . . . . . . . . . . . . . . . . . . . . . . . . . . . 103
5.12 Cluster Cold Start . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 103
5.13 Guest Migration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 104
5.13.1 Migration Type . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 104
5.13.2 Migration Network . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 104

6 Proxmox Cluster File System (pmxcfs) 106

6.1 POSIX Compatibility . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 106
6.2 File Access Rights . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 107
6.3 Technology . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 107
6.4 File System Layout . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 107
6.4.1 Files . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 107
6.4.2 Symbolic links . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 108
6.4.3 Special status files for debugging (JSON) . . . . . . . . . . . . . . . . . . . . . . . . 108
6.4.4 Enable/Disable debugging . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 108
6.5 Recovery . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 108
6.5.1 Remove Cluster configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 108
6.5.2 Recovering/Moving Guests from Failed Nodes . . . . . . . . . . . . . . . . . . . . . . 109
Proxmox VE Administration Guide viii

7 Proxmox VE Storage 110

7.1 Storage Types . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 110
7.1.1 Thin Provisioning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 111
7.2 Storage Configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 111
7.2.1 Storage Pools . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 112
7.2.2 Common Storage Properties . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 112
7.3 Volumes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 113
7.3.1 Volume Ownership . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 114
7.4 Using the Command Line Interface . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 114
7.4.1 Examples . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 114
7.5 Directory Backend . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 116
7.5.1 Configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 117
7.5.2 File naming conventions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 117
7.5.3 Storage Features . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 118
7.5.4 Examples . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 118
7.6 NFS Backend . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 119
7.6.1 Configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 119
7.6.2 Storage Features . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 120
7.6.3 Examples . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 120
7.7 CIFS Backend . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 120
7.7.1 Configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 121
7.7.2 Storage Features . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 121
7.7.3 Examples . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 122
7.8 Proxmox Backup Server . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 122
7.8.1 Configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 122
7.8.2 Storage Features . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 123
7.8.3 Encryption . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 124
7.8.4 Example: Add Storage over CLI . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 124
7.9 GlusterFS Backend . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 125
7.9.1 Configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 125
7.9.2 File naming conventions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 125
7.9.3 Storage Features . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 126
7.10 Local ZFS Pool Backend . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 126
7.10.1 Configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 126
7.10.2 File naming conventions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 127
Proxmox VE Administration Guide ix

7.10.3 Storage Features . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 127

7.10.4 Examples . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 127
7.11 LVM Backend . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 128
7.11.1 Configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 128
7.11.2 File naming conventions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 128
7.11.3 Storage Features . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 129
7.11.4 Examples . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 129
7.12 LVM thin Backend . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 129
7.12.1 Configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 130
7.12.2 File naming conventions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 130
7.12.3 Storage Features . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 130
7.12.4 Examples . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 130
7.13 Open-iSCSI initiator . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 131
7.13.1 Configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 131
7.13.2 File naming conventions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 131
7.13.3 Storage Features . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 132
7.13.4 Examples . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 132
7.14 User Mode iSCSI Backend . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 132
7.14.1 Configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 132
7.14.2 Storage Features . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 132
7.15 Ceph RADOS Block Devices (RBD) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 133
7.15.1 Configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 133
7.15.2 Authentication . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 134
7.15.3 Storage Features . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 134
7.16 Ceph Filesystem (CephFS) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 135
7.16.1 Configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 135
7.16.2 Authentication . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 136
7.16.3 Storage Features . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 136

8 Deploy Hyper-Converged Ceph Cluster 138

8.1 Precondition . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 139
8.2 Initial Ceph installation & configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 141
8.3 Installation of Ceph Packages . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 143
8.4 Create initial Ceph configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 143
8.5 Ceph Monitor . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 144
8.5.1 Create Monitors . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 144
Proxmox VE Administration Guide x

8.5.2 Destroy Monitors . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 144

8.6 Ceph Manager . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 145
8.6.1 Create Manager . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 145
8.6.2 Destroy Manager . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 145
8.7 Ceph OSDs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 146
8.7.1 Create OSDs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 146
8.7.2 Destroy OSDs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 148
8.8 Ceph Pools . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 148
8.8.1 Create Pools . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 149
8.8.2 Destroy Pools . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 150
8.9 Ceph CRUSH & device classes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 150
8.10 Ceph Client . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 152
8.11 CephFS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 153
8.11.1 Metadata Server (MDS) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 153
8.11.2 Create CephFS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 154
8.11.3 Destroy CephFS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 155
8.12 Ceph maintenance . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 155
8.12.1 Replace OSDs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 155
8.12.2 Trim/Discard . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 156
8.12.3 Scrub & Deep Scrub . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 156
8.13 Ceph monitoring and troubleshooting . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 156

9 Storage Replication 157

9.1 Supported Storage Types . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 157
9.2 Schedule Format . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 158
9.2.1 Detailed Specification . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 158
9.2.2 Examples: . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 159
9.3 Error Handling . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 159
9.3.1 Possible issues . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 159
9.3.2 Migrating a guest in case of Error . . . . . . . . . . . . . . . . . . . . . . . . . . . . 160
9.3.3 Example . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 160
9.4 Managing Jobs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 161
9.5 Command Line Interface Examples . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 161
Proxmox VE Administration Guide xi

10 Qemu/KVM Virtual Machines 163

10.1 Emulated devices and paravirtualized devices . . . . . . . . . . . . . . . . . . . . . . . . . . 163
10.2 Virtual Machines Settings . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 164
10.2.1 General Settings . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 164
10.2.2 OS Settings . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 165
10.2.3 System Settings . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 165
10.2.4 Hard Disk . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 166
10.2.5 CPU . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 168
10.2.6 Memory . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 172
10.2.7 Network Device . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 174
10.2.8 Display . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 175
10.2.9 USB Passthrough . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 176
10.2.10BIOS and UEFI . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 176
10.2.11Inter-VM shared memory . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 177
10.2.12Audio Device . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 177
10.2.13VirtIO RNG . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 178
10.2.14Device Boot Order . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 178
10.2.15Automatic Start and Shutdown of Virtual Machines . . . . . . . . . . . . . . . . . . . 179
10.2.16SPICE Enhancements . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 180
10.3 Migration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 181
10.3.1 Online Migration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 181
10.3.2 Offline Migration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 182
10.4 Copies and Clones . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 182
10.5 Virtual Machine Templates . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 183
10.6 VM Generation ID . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 184
10.7 Importing Virtual Machines and disk images . . . . . . . . . . . . . . . . . . . . . . . . . . . 184
10.7.1 Step-by-step example of a Windows OVF import . . . . . . . . . . . . . . . . . . . . . 185
10.7.2 Adding an external disk image to a Virtual Machine . . . . . . . . . . . . . . . . . . . 185
10.8 Cloud-Init Support . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 186
10.8.1 Preparing Cloud-Init Templates . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 186
10.8.2 Deploying Cloud-Init Templates . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 188
10.8.3 Custom Cloud-Init Configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 189
10.8.4 Cloud-Init specific Options . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 189
10.9 PCI(e) Passthrough . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 191
10.9.1 General Requirements . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 191
Proxmox VE Administration Guide xii

10.9.2 Host Device Passthrough . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 193

10.9.3 SR-IOV . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 194
10.9.4 Mediated Devices (vGPU, GVT-g) . . . . . . . . . . . . . . . . . . . . . . . . . . . . 195
10.10Hookscripts . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 196
10.11Hibernation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 197
10.12Managing Virtual Machines with qm . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 197
10.12.1CLI Usage Examples . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 197
10.13Configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 198
10.13.1File Format . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 198
10.13.2Snapshots . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 198
10.13.3Options . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 199
10.14Locks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 223

11 Proxmox Container Toolkit 224

11.1 Technology Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 224
11.2 Container Images . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 225
11.3 Container Settings . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 227
11.3.1 General Settings . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 227
11.3.2 CPU . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 228
11.3.3 Memory . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 229
11.3.4 Mount Points . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 230
11.3.5 Network . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 233
11.3.6 Automatic Start and Shutdown of Containers . . . . . . . . . . . . . . . . . . . . . . 234
11.3.7 Hookscripts . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 235
11.4 Security Considerations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 235
11.4.1 AppArmor . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 235
11.5 Guest Operating System Configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 236
11.6 Container Storage . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 237
11.6.1 FUSE Mounts . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 238
11.6.2 Using Quotas Inside Containers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 238
11.6.3 Using ACLs Inside Containers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 238
11.6.4 Backup of Container mount points . . . . . . . . . . . . . . . . . . . . . . . . . . . . 239
11.6.5 Replication of Containers mount points . . . . . . . . . . . . . . . . . . . . . . . . . 239
11.7 Backup and Restore . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 239
11.7.1 Container Backup . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 239
11.7.2 Restoring Container Backups . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 239
Proxmox VE Administration Guide xiii

11.8 Managing Containers with pct . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 240

11.8.1 CLI Usage Examples . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 241
11.8.2 Obtaining Debugging Logs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 241
11.9 Migration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 242
11.10Configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 242
11.10.1File Format . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 243
11.10.2Snapshots . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 243
11.10.3Options . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 244
11.11Locks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 249

12 Software Defined Network 250

12.1 Installation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 250
12.2 Basic Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 250
12.3 Main configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 251
12.3.1 SDN . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 251
12.3.2 Zones . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 251
12.3.3 VNets . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 251
12.3.4 Controllers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 252
12.4 Zones Plugins . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 252
12.4.1 Common options . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 252
12.4.2 VLAN Zones . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 252
12.4.3 QinQ Zones . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 252
12.4.4 VXLAN Zones . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 253
12.4.5 EVPN Zones . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 253
12.5 Controllers Plugins . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 254
12.5.1 EVPN Controller . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 254
12.6 Local Deployment Monitoring . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 254
12.7 VLAN Setup Example . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 255
12.8 QinQ Setup Example . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 256
12.9 VXLAN Setup Example . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 258
12.10EVPN Setup Example . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 260

13 Proxmox VE Firewall 263

13.1 Zones . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 263
13.2 Configuration Files . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 263
13.2.1 Cluster Wide Setup . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 264
Proxmox VE Administration Guide xiv

13.2.2 Host Specific Configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 265

13.2.3 VM/Container Configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 267
13.3 Firewall Rules . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 268
13.4 Security Groups . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 270
13.5 IP Aliases . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 270
13.5.1 Standard IP Alias local_network . . . . . . . . . . . . . . . . . . . . . . . . . . 270
13.6 IP Sets . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 271
13.6.1 Standard IP set management . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 271
13.6.2 Standard IP set blacklist . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 271
13.6.3 Standard IP set ipfilter-net* . . . . . . . . . . . . . . . . . . . . . . . . . . . 271
13.7 Services and Commands . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 272
13.8 Default firewall rules . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 272
13.8.1 Datacenter incoming/outgoing DROP/REJECT . . . . . . . . . . . . . . . . . . . . . 272
13.8.2 VM/CT incoming/outgoing DROP/REJECT . . . . . . . . . . . . . . . . . . . . . . . . 273
13.9 Logging of firewall rules . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 274
13.9.1 Logging of user defined firewall rules . . . . . . . . . . . . . . . . . . . . . . . . . . . 274
13.10Tips and Tricks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 275
13.10.1How to allow FTP . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 275
13.10.2Suricata IPS integration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 275
13.11Notes on IPv6 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 276
13.12Ports used by Proxmox VE . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 276

14 User Management 277

14.1 Users . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 277
14.1.1 System administrator . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 277
14.2 Groups . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 278
14.3 API Tokens . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 278
14.4 Authentication Realms . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 278
14.4.1 Syncing LDAP-based realms . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 280
14.5 Two-factor authentication . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 281
14.5.1 Realm enforced two-factor authentication . . . . . . . . . . . . . . . . . . . . . . . . 281
14.5.2 User configured TOTP authentication . . . . . . . . . . . . . . . . . . . . . . . . . . 282
14.5.3 Server side U2F configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 282
14.5.4 Activating U2F as a user . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 283
14.6 Permission Management . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 283
14.6.1 Roles . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 283
Proxmox VE Administration Guide xv

14.6.2 Privileges . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 284

14.6.3 Objects and Paths . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 285
14.6.4 Pools . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 286
14.6.5 What permission do I need? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 286
14.7 Command Line Tool . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 287
14.8 Real World Examples . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 288
14.8.1 Administrator Group . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 288
14.8.2 Auditors . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 289
14.8.3 Delegate User Management . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 289
14.8.4 Limited API token for monitoring . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 289
14.8.5 Pools . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 290

15 High Availability 291

15.1 Requirements . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 292
15.2 Resources . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 293
15.3 Management Tasks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 293
15.4 How It Works . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 294
15.4.1 Service States . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 295
15.4.2 Local Resource Manager . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 296
15.4.3 Cluster Resource Manager . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 297
15.5 HA Simulator . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 298
15.6 Configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 299
15.6.1 Resources . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 299
15.6.2 Groups . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 301
15.7 Fencing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 303
15.7.1 How Proxmox VE Fences . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 304
15.7.2 Configure Hardware Watchdog . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 304
15.7.3 Recover Fenced Services . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 304
15.8 Start Failure Policy . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 305
15.9 Error Recovery . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 305
15.10Package Updates . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 306
15.11Node Maintenance . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 306
15.11.1Shutdown Policy . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 306
Proxmox VE Administration Guide xvi

16 Backup and Restore 308

16.1 Backup modes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 308
16.2 Backup File Names . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 310
16.3 Backup File Compression . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 310
16.4 Backup Encryption . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 310
16.5 Backup Retention . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 310
16.5.1 Prune Simulator . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 311
16.5.2 Retention Settings Example . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 312
16.6 Restore . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 312
16.6.1 Bandwidth Limit . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 312
16.7 Configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 313
16.8 Hook Scripts . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 315
16.9 File Exclusions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 316
16.10Examples . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 316

17 Important Service Daemons 318

17.1 pvedaemon - Proxmox VE API Daemon . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 318
17.2 pveproxy - Proxmox VE API Proxy Daemon . . . . . . . . . . . . . . . . . . . . . . . . . . . 318
17.2.1 Host based Access Control . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 318
17.2.2 SSL Cipher Suite . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 319
17.2.3 Diffie-Hellman Parameters . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 319
17.2.4 Alternative HTTPS certificate . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 319
17.2.5 COMPRESSION . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 320
17.3 pvestatd - Proxmox VE Status Daemon . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 320
17.4 spiceproxy - SPICE Proxy Service . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 320
17.4.1 Host based Access Control . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 320

18 Useful Command Line Tools 321

18.1 pvesubscription - Subscription Management . . . . . . . . . . . . . . . . . . . . . . . . . . . 321
18.2 pveperf - Proxmox VE Benchmark Script . . . . . . . . . . . . . . . . . . . . . . . . . . . . 321
18.3 Shell interface for the Proxmox VE API . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 322
18.3.1 EXAMPLES . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 322
18.3.2 Proxmox Node Management . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 322

19 Frequently Asked Questions 324

Proxmox VE Administration Guide xvii

20 Bibliography 327
20.1 Books about Proxmox VE . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 327
20.2 Books about related technology . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 327
20.3 Books about related topics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 328

A Command Line Interface 329

A.1 Output format options [FORMAT_OPTIONS] . . . . . . . . . . . . . . . . . . . . . . . . . 329
A.2 pvesm - Proxmox VE Storage Manager . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 330
A.3 pvesubscription - Proxmox VE Subscription Manager . . . . . . . . . . . . . . . . . . . . . 342
A.4 pveperf - Proxmox VE Benchmark Script . . . . . . . . . . . . . . . . . . . . . . . . . . . . 342
A.5 pveceph - Manage CEPH Services on Proxmox VE Nodes . . . . . . . . . . . . . . . . . . . 343
A.6 pvenode - Proxmox VE Node Management . . . . . . . . . . . . . . . . . . . . . . . . . . . 349
A.7 pvesh - Shell interface for the Proxmox VE API . . . . . . . . . . . . . . . . . . . . . . . . . 355
A.8 qm - Qemu/KVM Virtual Machine Manager . . . . . . . . . . . . . . . . . . . . . . . . . . . 357
A.9 qmrestore - Restore QemuServer vzdump Backups . . . . . . . . . . . . . . . . . . . . . . 385
A.10 pct - Proxmox Container Toolkit . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 386
A.11 pveam - Proxmox VE Appliance Manager . . . . . . . . . . . . . . . . . . . . . . . . . . . . 407
A.12 pvecm - Proxmox VE Cluster Manager . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 408
A.13 pvesr - Proxmox VE Storage Replication . . . . . . . . . . . . . . . . . . . . . . . . . . . . 411
A.14 pveum - Proxmox VE User Manager . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 415
A.15 vzdump - Backup Utility for VMs and Containers . . . . . . . . . . . . . . . . . . . . . . . . 428
A.16 ha-manager - Proxmox VE HA Manager . . . . . . . . . . . . . . . . . . . . . . . . . . . . 430

B Service Daemons 435

B.1 pve-firewall - Proxmox VE Firewall Daemon . . . . . . . . . . . . . . . . . . . . . . . . . . 435
B.2 pvedaemon - Proxmox VE API Daemon . . . . . . . . . . . . . . . . . . . . . . . . . . . . 436
B.3 pveproxy - Proxmox VE API Proxy Daemon . . . . . . . . . . . . . . . . . . . . . . . . . . . 437
B.4 pvestatd - Proxmox VE Status Daemon . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 437
B.5 spiceproxy - SPICE Proxy Service . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 438
B.6 pmxcfs - Proxmox Cluster File System . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 439
B.7 pve-ha-crm - Cluster Resource Manager Daemon . . . . . . . . . . . . . . . . . . . . . . . 439
B.8 pve-ha-lrm - Local Resource Manager Daemon . . . . . . . . . . . . . . . . . . . . . . . . 440

C Configuration Files 441

C.1 Datacenter Configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 441
C.1.1 File Format . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 441
C.1.2 Options . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 441

D Firewall Macro Definitions 444

E GNU Free Documentation License 458

Proxmox VE Administration Guide 1 / 464

Chapter 1

Introduction

Proxmox VE is a platform to run virtual machines and containers. It is based on Debian Linux, and completely
open source. For maximum flexibility, we implemented two virtualization technologies - Kernel-based Virtual
Machine (KVM) and container-based virtualization (LXC).
One main design goal was to make administration as easy as possible. You can use Proxmox VE on a
single node, or assemble a cluster of many nodes. All management tasks can be done using our web-based
management interface, and even a novice user can setup and install Proxmox VE within minutes.

User Tools
qm pvesm pveum ha-manager

pct pvecm pveceph pve-firewall

Services
pveproxy pvedaemon pvestatd pve-ha-lrm pve-cluster

VM VM
App App App App

Guest OS Guest OS Container Container

App App App App
Qemu

KVM Linux Kernel AppArmor cgroups

Proxmox VE Administration Guide 2 / 464

1.1 Central Management

While many people start with a single node, Proxmox VE can scale out to a large set of clustered nodes.
The cluster stack is fully integrated and ships with the default installation.

Unique Multi-Master Design

The integrated web-based management interface gives you a clean overview of all your KVM guests
and Linux containers and even of your whole cluster. You can easily manage your VMs and con-
tainers, storage or cluster from the GUI. There is no need to install a separate, complex, and pricey
management server.

Proxmox Cluster File System (pmxcfs)

Proxmox VE uses the unique Proxmox Cluster file system (pmxcfs), a database-driven file system for
storing configuration files. This enables you to store the configuration of thousands of virtual machines.
By using corosync, these files are replicated in real time on all cluster nodes. The file system stores
all data inside a persistent database on disk, nonetheless, a copy of the data resides in RAM which
provides a maximum storage size of 30MB - more than enough for thousands of VMs.
Proxmox VE is the only virtualization platform using this unique cluster file system.

Web-based Management Interface

Proxmox VE is simple to use. Management tasks can be done via the included web based manage-
ment interface - there is no need to install a separate management tool or any additional management
node with huge databases. The multi-master tool allows you to manage your whole cluster from any
node of your cluster. The central web-based management - based on the JavaScript Framework (Ex-
tJS) - empowers you to control all functionalities from the GUI and overview history and syslogs of each
single node. This includes running backup or restore jobs, live-migration or HA triggered activities.

Command Line
For advanced users who are used to the comfort of the Unix shell or Windows Powershell, Proxmox
VE provides a command line interface to manage all the components of your virtual environment. This
command line interface has intelligent tab completion and full documentation in the form of UNIX man
pages.

REST API
Proxmox VE uses a RESTful API. We choose JSON as primary data format, and the whole API is for-
mally defined using JSON Schema. This enables fast and easy integration for third party management
tools like custom hosting environments.

Role-based Administration
You can define granular access for all objects (like VMs, storages, nodes, etc.) by using the role based
user- and permission management. This allows you to define privileges and helps you to control
access to objects. This concept is also known as access control lists: Each permission specifies a
subject (a user or group) and a role (set of privileges) on a specific path.

Authentication Realms
Proxmox VE supports multiple authentication sources like Microsoft Active Directory, LDAP, Linux PAM
standard authentication or the built-in Proxmox VE authentication server.
Proxmox VE Administration Guide 3 / 464

1.2 Flexible Storage

The Proxmox VE storage model is very flexible. Virtual machine images can either be stored on one or
several local storages or on shared storage like NFS and on SAN. There are no limits, you may configure as
many storage definitions as you like. You can use all storage technologies available for Debian Linux.
One major benefit of storing VMs on shared storage is the ability to live-migrate running machines without
any downtime, as all nodes in the cluster have direct access to VM disk images.
We currently support the following Network storage types:

• LVM Group (network backing with iSCSI targets)

• iSCSI target

• NFS Share

• CIFS Share

• Ceph RBD

• Directly use iSCSI LUNs

• GlusterFS

Local storage types supported are:

• LVM Group (local backing devices like block devices, FC devices, DRBD, etc.)

• Directory (storage on existing filesystem)

• ZFS

1.3 Integrated Backup and Restore

The integrated backup tool (vzdump) creates consistent snapshots of running Containers and KVM guests.
It basically creates an archive of the VM or CT data which includes the VM/CT configuration files.
KVM live backup works for all storage types including VM images on NFS, CIFS, iSCSI LUN, Ceph RBD.
The new backup format is optimized for storing VM backups fast and effective (sparse files, out of order data,
minimized I/O).

1.4 High Availability Cluster

A multi-node Proxmox VE HA Cluster enables the definition of highly available virtual servers. The Proxmox
VE HA Cluster is based on proven Linux HA technologies, providing stable and reliable HA services.
Proxmox VE Administration Guide 4 / 464

1.5 Flexible Networking

Proxmox VE uses a bridged networking model. All VMs can share one bridge as if virtual network cables
from each guest were all plugged into the same switch. For connecting VMs to the outside world, bridges
are attached to physical network cards and assigned a TCP/IP configuration.
For further flexibility, VLANs (IEEE 802.1q) and network bonding/aggregation are possible. In this way it is
possible to build complex, flexible virtual networks for the Proxmox VE hosts, leveraging the full power of the
Linux network stack.

1.6 Integrated Firewall

The integrated firewall allows you to filter network packets on any VM or Container interface. Common sets
of firewall rules can be grouped into “security groups”.

1.7 Hyper-converged Infrastructure

Proxmox VE is a virtualization platform that tightly integrates compute, storage and networking resources,
manages highly available clusters, backup/restore as well as disaster recovery. All components are software-
defined and compatible with one another.
Therefore it is possible to administrate them like a single system via the centralized web management inter-
face. These capabilities make Proxmox VE an ideal choice to deploy and manage an open source hyper-
converged infrastructure.

1.7.1 Benefits of a Hyper-Converged Infrastructure (HCI) with Proxmox VE

A hyper-converged infrastructure (HCI) is especially useful for deployments in which a high infrastructure
demand meets a low administration budget, for distributed setups such as remote and branch office environ-
ments or for virtual private and public clouds.
HCI provides the following advantages:

• Scalability: seamless expansion of compute, network and storage devices (i.e. scale up servers and
storage quickly and independently from each other).

• Low cost: Proxmox VE is open source and integrates all components you need such as compute, storage,
networking, backup, and management center. It can replace an expensive compute/storage infrastructure.

• Data protection and efficiency: services such as backup and disaster recovery are integrated.

• Simplicity: easy configuration and centralized administration.

• Open Source: No vendor lock-in.

Proxmox VE Administration Guide 5 / 464

1.7.2 Hyper-Converged Infrastructure: Storage

Proxmox VE has tightly integrated support for deploying a hyper-converged storage infrastructure. You can,
for example, deploy and manage the following two storage technologies by using the Webinterface only:

• ceph: a both, self-healing and self-managing shared, reliable and highly scalable storage system. Check-
out how to manage ceph services on Proxmox VE nodes Chapter 8

• ZFS: a combined file system and logical volume manager with extensive protection against data corruption,
various RAID modes, fast and cheap snapshots - among other features. Find out how to leverage the power
of ZFS on Proxmox VE nodes Section 3.8.

Besides above, Proxmox VE has support to integrate a wide range of additional storage technologies. You
can find out about them in the Storage Manager chapter Chapter 7.

1.8 Why Open Source

Proxmox VE uses a Linux kernel and is based on the Debian GNU/Linux Distribution. The source code of
Proxmox VE is released under the GNU Affero General Public License, version 3. This means that you are
free to inspect the source code at any time or contribute to the project yourself.
At Proxmox we are committed to use open source software whenever possible. Using open source software
guarantees full access to all functionalities - as well as high security and reliability. We think that everybody
should have the right to access the source code of a software to run it, build on it, or submit changes back
to the project. Everybody is encouraged to contribute while Proxmox ensures the product always meets
professional quality criteria.
Open source software also helps to keep your costs low and makes your core infrastructure independent
from a single vendor.

1.9 Your benefits with Proxmox VE

• Open source software

• No vendor lock-in

• Linux kernel

• Fast installation and easy-to-use

• Web-based management interface

• REST API

• Huge active community

• Low administration costs and simple deployment

Proxmox VE Administration Guide 6 / 464

1.10 Getting Help

1.10.1 Proxmox VE Wiki

The primary source of information is the Proxmox VE Wiki. It combines the reference documentation with
user contributed content.

1.10.2 Community Support Forum

We always encourage our users to discuss and share their knowledge using the Proxmox VE Community
Forum. The forum is moderated by the Proxmox support team. The large user base is spread out all over
the world. Needless to say that such a large forum is a great place to get information.

1.10.3 Mailing Lists

This is a fast way to communicate with the Proxmox VE community via email.

• Mailing list for users: PVE User List

Proxmox VE is fully open source and contributions are welcome! The primary communication channel for
developers is the:

• Mailing list for developers: PVE development discussion

1.10.4 Commercial Support

Proxmox Server Solutions Gmbh also offers enterprise support available as Proxmox VE Subscription Ser-
vice Plans. All users with a subscription get access to the Proxmox VE Enterprise Repository, and—with
a Basic, Standard or Premium subscription—also to the Proxmox Customer Portal. The customer portal
provides help and support with guaranteed response times from the Proxmox VE developers.
For volume discounts, or more information in general, please contact [email protected].

1.10.5 Bug Tracker

Proxmox runs a public bug tracker at https://bugzilla.proxmox.com. If an issue appears, file your report there.
An issue can be a bug as well as a request for a new feature or enhancement. The bug tracker helps to keep
track of the issue and will send a notification once it has been solved.

1.11 Project History

The project started in 2007, followed by a first stable version in 2008. At the time we used OpenVZ for
containers, and KVM for virtual machines. The clustering features were limited, and the user interface was
simple (server generated web page).
Proxmox VE Administration Guide 7 / 464

But we quickly developed new features using the Corosync cluster stack, and the introduction of the new
Proxmox cluster file system (pmxcfs) was a big step forward, because it completely hides the cluster com-
plexity from the user. Managing a cluster of 16 nodes is as simple as managing a single node.
We also introduced a new REST API, with a complete declarative specification written in JSON-Schema.
This enabled other people to integrate Proxmox VE into their infrastructure, and made it easy to provide
additional services.
Also, the new REST API made it possible to replace the original user interface with a modern HTML5
application using JavaScript. We also replaced the old Java based VNC console code with noVNC. So
you only need a web browser to manage your VMs.
The support for various storage types is another big task. Notably, Proxmox VE was the first distribution to
ship ZFS on Linux by default in 2014. Another milestone was the ability to run and manage Ceph storage on
the hypervisor nodes. Such setups are extremely cost effective.
When we started we were among the first companies providing commercial support for KVM. The KVM
project itself continuously evolved, and is now a widely used hypervisor. New features arrive with each
release. We developed the KVM live backup feature, which makes it possible to create snapshot backups on
any storage type.
The most notable change with version 4.0 was the move from OpenVZ to LXC. Containers are now deeply
integrated, and they can use the same storage and network features as virtual machines.

1.12 Improving the Proxmox VE Documentation

Contributions and improvements to the Proxmox VE documentation are always welcome. There are several
ways to contribute.
If you find errors or other room for improvement in this documentation, please file a bug at the Proxmox bug
tracker to propose a correction.
If you want to propose new content, choose one of the following options:

• The wiki: For specific setups, how-to guides, or tutorials the wiki is the right option to contribute.

• The reference documentation: For general content that will be helpful to all users please propose your con-
tribution for the reference documentation. This includes all information about how to install, configure, use,
and troubleshoot Proxmox VE features. The reference documentation is written in the asciidoc format. To
edit the documentation you need to clone the git repository at git://git.proxmox.com/git/pve-docs
then follow the README.adoc document.

Note
If you are interested in working on the Proxmox VE codebase, the Developer Documentation wiki article
will show you where to start.

1.13 Translating Proxmox VE

The Proxmox VE user interface is in English by default. However, thanks to the contributions of the commu-
nity, translations to other languages are also available. We welcome any support in adding new languages,
translating the latest features, and improving incomplete or inconsistent translations.
Proxmox VE Administration Guide 8 / 464

We use gettext for the management of the translation files. Tools like Poedit offer a nice user interface to edit
the translation files, but you can use whatever editor you’re comfortable with. No programming knowledge is
required for translating.

1.13.1 Translating with git

The language files are available as a git repository. If you are familiar with git, please contribute according
to our Developer Documentation.
You can create a new translation by doing the following (replace <LANG> with the language ID):

# git clone git://git.proxmox.com/git/proxmox-i18n.git

# cd proxmox-i18n
# make init-<LANG>.po

Or you can edit an existing translation, using the editor of your choice:

# poedit <LANG>.po

1.13.2 Translating without git

Even if you are not familiar with git, you can help translate Proxmox VE. To start, you can download the
language files here. Find the language you want to improve, then right click on the "raw" link of this language
file and select Save Link As. . . . Make your changes to the file, and then send your final translation directly
to office(at)proxmox.com, together with a signed contributor license agreement.

1.13.3 Testing the Translation

In order for the translation to be used in Proxmox VE, you must first translate the .po file into a .js file.
You can do this by invoking the following script, which is located in the same repository:

# ./po2js.pl -t pve xx.po >pve-lang-xx.js

The resulting file pve-lang-xx.js can then be copied to the directory /usr/share/pve-i18n, on
your proxmox server, in order to test it out.
Alternatively, you can build a deb package by running the following command from the root of the repository:

# make deb

Important
For either of these methods to work, you need to have the following perl packages installed on your
system. For Debian/Ubuntu:
Proxmox VE Administration Guide 9 / 464

# apt-get install perl liblocale-po-perl libjson-perl

1.13.4 Sending the Translation

You can send the finished translation (.po file) to the Proxmox team at the address office(at)proxmox.com,
along with a signed contributor licence agreement. Alternatively, if you have some developer experience,
you can send it as a patch to the Proxmox VE development mailing list. See [Developer Documentation]
Proxmox VE Administration Guide 10 / 464

Chapter 2

Installing Proxmox VE

Proxmox VE is based on Debian. This is why the install disk images (ISO files) provided by Proxmox include
a complete Debian system (Debian 10 Buster for Proxmox VE version 6.x) as well as all necessary Proxmox
VE packages.
The installer will guide through the setup, allowing you to partition the local disk(s), apply basic system
configurations (for example, timezone, language, network) and install all required packages. This process
should not take more than a few minutes. Installing with the provided ISO is the recommended method for
new and existing users.
Alternatively, Proxmox VE can be installed on top of an existing Debian system. This option is only recom-
mended for advanced users because detailed knowledge about Proxmox VE is required.

2.1 System Requirements

We recommend using high quality server hardware, when running Proxmox VE in production. To further
decrease the impact of a failed host, you can run Proxmox VE in a cluster with highly available (HA) virtual
machines and containers.
Proxmox VE can use local storage (DAS), SAN, NAS, and distributed storage like Ceph RBD. For details see
chapter storage Chapter 7.

2.1.1 Minimum Requirements, for Evaluation

These minimum requirements are for evaluation purposes only and should not be used in production.

• CPU: 64bit (Intel EMT64 or AMD64)

• Intel VT/AMD-V capable CPU/Mainboard for KVM full virtualization support

• RAM: 1 GB RAM, plus additional RAM needed for guests

• Hard drive

• One network card (NIC)

Proxmox VE Administration Guide 11 / 464

2.1.2 Recommended System Requirements

• Intel EMT64 or AMD64 with Intel VT/AMD-V CPU flag.

• Memory: Minimum 2 GB for the OS and Proxmox VE services, plus designated memory for guests. For
Ceph and ZFS, additional memory is required; approximately 1GB of memory for every TB of used storage.

• Fast and redundant storage, best results are achieved with SSDs.

• OS storage: Use a hardware RAID with battery protected write cache (“BBU”) or non-RAID with ZFS
(optional SSD for ZIL).

• VM storage:

– For local storage, use either a hardware RAID with battery backed write cache (BBU) or non-RAID for
ZFS and Ceph. Neither ZFS nor Ceph are compatible with a hardware RAID controller.
– Shared and distributed storage is possible.

• Redundant (Multi-)Gbit NICs, with additional NICs depending on the preferred storage technology and
cluster setup.

• For PCI(e) passthrough the CPU needs to support the VT-d/AMD-d flag.

2.1.3 Simple Performance Overview

To get an overview of the CPU and hard disk performance on an installed Proxmox VE system, run the
included pveperf tool.

Note
This is just a very quick and general benchmark. More detailed tests are recommended, especially re-
garding the I/O performance of your system.

2.1.4 Supported Web Browsers for Accessing the Web Interface

To access the web-based user interface, we recommend using one of the following browsers:

• Firefox, a release from the current year, or the latest Extended Support Release

• Chrome, a release from the current year

• Microsoft’s currently supported version of Edge

• Safari, a release from the current year

When accessed from a mobile device, Proxmox VE will show a lightweight, touch-based interface.
Proxmox VE Administration Guide 12 / 464

2.2 Prepare Installation Media

Download the installer ISO image from: https://www.proxmox.com/en/downloads/category/iso-images-pve

The Proxmox VE installation media is a hybrid ISO image. It works in two ways:

• An ISO image file ready to burn to a CD or DVD.

• A raw sector (IMG) image file ready to copy to a USB flash drive (USB stick).

Using a USB flash drive to install Proxmox VE is the recommended way because it is the faster option.

2.2.1 Prepare a USB Flash Drive as Installation Medium

The flash drive needs to have at least 1 GB of storage available.

Note
Do not use UNetbootin. It does not work with the Proxmox VE installation image.

Important
Make sure that the USB flash drive is not mounted and does not contain any important data.

2.2.2 Instructions for GNU/Linux

On Unix-like operating system use the dd command to copy the ISO image to the USB flash drive. First find
the correct device name of the USB flash drive (see below). Then run the dd command.

# dd bs=1M conv=fdatasync if=./proxmox-ve_*.iso of=/dev/XYZ

Note
Be sure to replace /dev/XYZ with the correct device name and adapt the input filename (if ) path.

Caution
Be very careful, and do not overwrite the wrong disk!
Proxmox VE Administration Guide 13 / 464

Find the Correct USB Device Name

There are two ways to find out the name of the USB flash drive. The first one is to compare the last lines of
the dmesg command output before and after plugging in the flash drive. The second way is to compare the
output of the lsblk command. Open a terminal and run:

# lsblk

Then plug in your USB flash drive and run the command again:

# lsblk

A new device will appear. This is the one you want to use. To be on the extra safe side check if the reported
size matches your USB flash drive.

2.2.3 Instructions for macOS

Open the terminal (query Terminal in Spotlight).

Convert the .iso file to .img using the convert option of hdiutil for example.

# hdiutil convert -format UDRW -o proxmox-ve_*.dmg proxmox-ve_*.iso

Tip
macOS tends to automatically add .dmg to the output file name.

To get the current list of devices run the command:

# diskutil list

Now insert the USB flash drive and run this command again to determine which device node has been
assigned to it. (e.g., /dev/diskX).

# diskutil list
# diskutil unmountDisk /dev/diskX

Note
replace X with the disk number from the last command.

# sudo dd if=proxmox-ve_*.dmg of=/dev/rdiskX bs=1m

Note
rdiskX, instead of diskX, in the last command is intended. It will increase the write speed.
Proxmox VE Administration Guide 14 / 464

2.2.4 Instructions for Windows

Using Etcher

Etcher works out of the box. Download Etcher from https://etcher.io. It will guide you through the process of
selecting the ISO and your USB Drive.

Using Rufus

Rufus is a more lightweight alternative, but you need to use the DD mode to make it work. Download Rufus
from https://rufus.ie/. Either install it or use the portable version. Select the destination drive and the Proxmox
VE ISO file.

Important
Once you Start you have to click No on the dialog asking to download a different version of GRUB.
In the next dialog select the DD mode.

2.3 Using the Proxmox VE Installer

The installer ISO image includes the following:

• Complete operating system (Debian Linux, 64-bit)

• The Proxmox VE installer, which partitions the local disk(s) with ext4, xfs or ZFS and installs the operating
system.

• Proxmox VE Linux kernel with KVM and LXC support

• Complete toolset for administering virtual machines, containers, the host system, clusters and all neces-
sary resources

• Web-based management interface

Note
All existing data on the for installation selected drives will be removed during the installation process. The
installer does not add boot menu entries for other operating systems.

Please insert the prepared installation media Section 2.2 (for example, USB flash drive or CD-ROM) and
boot from it.

Tip
Make sure that booting from the installation medium (for example, USB) is enabled in your servers firmware
settings.
Proxmox VE Administration Guide 15 / 464

After choosing the correct entry (e.g. Boot from USB) the Proxmox VE menu will be displayed and one of
the following options can be selected:

Install Proxmox VE
Starts the normal installation.

Tip
It’s possible to use the installation wizard with a keyboard only. Buttons can be clicked by pressing the
ALT key combined with the underlined character from the respective button. For example, ALT + N to
press a Next button.

Install Proxmox VE (Debug mode)

Starts the installation in debug mode. A console will be opened at several installation steps. This helps
to debug the situation if something goes wrong. To exit a debug console, press CTRL-D. This option
can be used to boot a live system with all basic tools available. You can use it, for example, to repair a
degraded ZFS rpool Section 3.8 or fix the bootloader Section 3.11 for an existing Proxmox VE setup.

Rescue Boot
With this option you can boot an existing installation. It searches all attached hard disks. If it finds
an existing installation, it boots directly into that disk using the Linux kernel from the ISO. This can be
Proxmox VE Administration Guide 16 / 464

useful if there are problems with the boot block (grub) or the BIOS is unable to read the boot block
from the disk.

Test Memory
Runs memtest86+. This is useful to check if the memory is functional and free of errors.

After selecting Install Proxmox VE and accepting the EULA, the prompt to select the target hard disk(s) will
appear. The Options button opens the dialog to select the target file system.
The default file system is ext4. The Logical Volume Manager (LVM) is used when ext4 or xfs ist selected.
Additional options to restrict LVM space can be set (see below).
Proxmox VE can be installed on ZFS. As ZFS offers several software RAID levels, this is an option for
systems that don’t have a hardware RAID controller. The target disks must be selected in the Options
dialog. More ZFS specific settings can be changed under Advanced Options (see below).

Warning
ZFS on top of any hardware RAID is not supported and can result in data loss.
Proxmox VE Administration Guide 17 / 464

The next page asks for basic configuration options like the location, the time zone, and keyboard layout. The
location is used to select a download server close by to speed up updates. The installer usually auto-detects
these settings. They only need to be changed in the rare case that auto detection fails or a different keyboard
layout should be used.
Proxmox VE Administration Guide 18 / 464

Next the password of the superuser (root) and an email address needs to be specified. The password must
consist of at least 5 characters. It’s highly recommended to use a stronger password. Some guidelines are:

• Use a minimum password length of 12 to 14 characters.

• Include lowercase and uppercase alphabetic characters, numbers, and symbols.

• Avoid character repetition, keyboard patterns, common dictionary words, letter or number sequences, user-
names, relative or pet names, romantic links (current or past), and biographical information (for example
ID numbers, ancestors’ names or dates).

The email address is used to send notifications to the system administrator. For example:

• Information about available package updates.

• Error messages from periodic CRON jobs.

Proxmox VE Administration Guide 19 / 464

The last step is the network configuration. Please note that during installation you can either use an IPv4 or
IPv6 address, but not both. To configure a dual stack node, add additional IP addresses after the installation.
Proxmox VE Administration Guide 20 / 464

The next step shows a summary of the previously selected options. Re-check every setting and use the
Previous button if a setting needs to be changed. To accept, press Install. The installation starts to
format disks and copies packages to the target. Please wait until this step has finished; then remove the
installation medium and restart your system.
Proxmox VE Administration Guide 21 / 464

If the installation failed check out specific errors on the second TTY (‘CTRL + ALT + F2’), ensure that the
systems meets the minimum requirements Section 2.1.1. If the installation is still not working look at the how
to get help chapter Section 1.10.
Further configuration is done via the Proxmox web interface. Point your browser to the IP address given
during installation (https://youripaddress:8006).

Note
Default login is "root" (realm PAM) and the root password is defined during the installation process.

2.3.1 Advanced LVM Configuration Options

The installer creates a Volume Group (VG) called pve, and additional Logical Volumes (LVs) called root,
data, and swap. To control the size of these volumes use:

hdsize
Defines the total hard disk size to be used. This way you can reserve free space on the hard disk for
further partitioning (for example for an additional PV and VG on the same hard disk that can be used
for LVM storage).
Proxmox VE Administration Guide 22 / 464

swapsize
Defines the size of the swap volume. The default is the size of the installed memory, minimum 4 GB
and maximum 8 GB. The resulting value cannot be greater than hdsize/8.

Note
If set to 0, no swap volume will be created.

maxroot
Defines the maximum size of the root volume, which stores the operation system. The maximum
limit of the root volume size is hdsize/4.

maxvz
Defines the maximum size of the data volume. The actual size of the data volume is:
datasize = hdsize - rootsize - swapsize - minfree
Where datasize cannot be bigger than maxvz.

Note
In case of LVM thin, the data pool will only be created if datasize is bigger than 4GB.

Note
If set to 0, no data volume will be created and the storage configuration will be adapted accordingly.

minfree
Defines the amount of free space left in the LVM volume group pve. With more than 128GB storage
available the default is 16GB, else hdsize/8 will be used.

Note
LVM requires free space in the VG for snapshot creation (not required for lvmthin snapshots).

2.3.2 Advanced ZFS Configuration Options

The installer creates the ZFS pool rpool. No swap space is created but you can reserve some unpartitioned
space on the install disks for swap. You can also create a swap zvol after the installation, although this can
lead to problems. (see ZFS swap notes).

ashift
Defines the ashift value for the created pool. The ashift needs to be set at least to the sector-
size of the underlying disks (2 to the power of ashift is the sector-size), or any disk which might be
put in the pool (for example the replacement of a defective disk).

compress
Defines whether compression is enabled for rpool.
Proxmox VE Administration Guide 23 / 464

checksum
Defines which checksumming algorithm should be used for rpool.

copies
Defines the copies parameter for rpool. Check the zfs(8) manpage for the semantics, and why
this does not replace redundancy on disk-level.

hdsize
Defines the total hard disk size to be used. This is useful to save free space on the hard disk(s) for
further partitioning (for example to create a swap-partition). hdsize is only honored for bootable
disks, that is only the first disk or mirror for RAID0, RAID1 or RAID10, and all disks in RAID-Z[123].

2.3.3 ZFS Performance Tips

ZFS works best with a lot of memory. If you intend to use ZFS make sure to have enough RAM available for
it. A good calculation is 4GB plus 1GB RAM for each TB RAW disk space.
ZFS can use a dedicated drive as write cache, called the ZFS Intent Log (ZIL). Use a fast drive (SSD) for it.
It can be added after installation with the following command:

# zpool add <pool-name> log </dev/path_to_fast_ssd>

2.4 Install Proxmox VE on Debian

Proxmox VE ships as a set of Debian packages and can be installed on to of a standard Debian installation.
After configuring the repositories Section 3.1 you need to run the following commands:

# apt-get update
# apt-get install proxmox-ve

Installing on top of an existing Debian installation looks easy, but it presumes that the base system has been
installed correctly and that you know how you want to configure and use the local storage. You also need to
configure the network manually.
In general, this is not trivial, especially when LVM or ZFS is used.
A detailed step by step how-to can be found on the wiki.
Proxmox VE Administration Guide 24 / 464

Chapter 3

Host System Administration

The following sections will focus on common virtualization tasks and explain the Proxmox VE specifics re-
garding the administration and management of the host machine.
Proxmox VE is based on Debian GNU/Linux with additional repositories to provide the Proxmox VE related
packages. This means that the full range of Debian packages is available including security updates and
bug fixes. Proxmox VE provides it’s own Linux kernel based on the Ubuntu kernel. It has all the necessary
virtualization and container features enabled and includes ZFS and several extra hardware drivers.
For other topics not included in the following sections, please refer to the Debian documentation. The De-
bian Administrator’s Handbook is available online, and provides a comprehensive introduction to the Debian
operating system (see [Hertzog13]).

3.1 Package Repositories

Proxmox VE uses APT as its package management tool like any other Debian-based system. Repositories
are defined in the file /etc/apt/sources.list and in .list files placed in /etc/apt/sources.list.
Each line defines a package repository. The preferred source must come first. Empty lines are ignored. A #
character anywhere on a line marks the remainder of that line as a comment. The available packages from a
repository are acquired by running apt-get update. Updates can be installed directly using apt-get,
or via the GUI.

File /etc/apt/sources.list

deb http://ftp.debian.org/debian buster main contrib

deb http://ftp.debian.org/debian buster-updates main contrib

# security updates
deb http://security.debian.org/debian-security buster/updates main contrib

Proxmox VE additionally provides three different package repositories.

Proxmox VE Administration Guide 25 / 464

3.1.1 Proxmox VE Enterprise Repository

This is the default, stable, and recommended repository, available for all Proxmox VE subscription users. It
contains the most stable packages and is suitable for production use. The pve-enterprise repository
is enabled by default:

File /etc/apt/sources.list.d/pve-enterprise.list

deb https://enterprise.proxmox.com/debian/pve buster pve-enterprise

The root@pam user is notified via email about available updates. Click the Changelog button in the GUI to
see more details about the selected update.
You need a valid subscription key to access the pve-enterprise repository. Different support levels are
available. Further details can be found at https://www.proxmox.com/en/proxmox-ve/pricing.

Note
You can disable this repository by commenting out the above line using a # (at the start of the
line). This prevents error messages if you do not have a subscription key. Please configure the
pve-no-subscription repository in that case.

3.1.2 Proxmox VE No-Subscription Repository

This is the recommended repository for testing and non-production use. Its packages are not as heavily
tested and validated. You don’t need a subscription key to access the pve-no-subscription reposi-
tory.
We recommend to configure this repository in /etc/apt/sources.list.

File /etc/apt/sources.list

deb http://ftp.debian.org/debian buster main contrib

deb http://ftp.debian.org/debian buster-updates main contrib

# PVE pve-no-subscription repository provided by proxmox.com,

# NOT recommended for production use
deb http://download.proxmox.com/debian/pve buster pve-no-subscription

# security updates
deb http://security.debian.org/debian-security buster/updates main contrib

3.1.3 Proxmox VE Test Repository

This repository contains the latest packages and is primarily used by developers to test new features. To
configure it, add the following line to etc/apt/sources.list:
Proxmox VE Administration Guide 26 / 464

sources.list entry for pvetest

deb http://download.proxmox.com/debian/pve buster pvetest

Warning
The pvetest repository should (as the name implies) only be used for testing new features or
bug fixes.

3.1.4 Ceph Octopus Repository

Note
Ceph Octopus (15.2) was declared stable with Proxmox VE 6.3 and is the most recent Ceph release
supported. It will continue to get updates for the remaining life time of the 6.x release [?informaltable].

This repository holds the main Proxmox VE Ceph Octopus packages. They are suitable for production. Use
this repository if you run the Ceph client or a full Ceph cluster on Proxmox VE.

File /etc/apt/sources.list.d/ceph.list

deb http://download.proxmox.com/debian/ceph-octopus buster main

3.1.5 Ceph Octopus Test Repository

This Ceph repository contains the Ceph packages before they are moved to the main repository. It is used
to test new Ceph releases on Proxmox VE.

File /etc/apt/sources.list.d/ceph.list

deb http://download.proxmox.com/debian/ceph-octopus buster test

3.1.6 Ceph Nautilus Repository

Note
Ceph Nautlius (14.2) is the older supported Ceph version, introduced with Proxmox VE 6.0. It will continue
to get updates until end of Q2 2021, so you will eventually need to upgrade to Ceph Octopus.

This repository holds the main Proxmox VE Ceph Nautilus packages. They are suitable for production. Use
this repository if you run the Ceph client or a full Ceph cluster on Proxmox VE.