Mastering OpenVPN Master Building and Integrating Chapter 9 Troubleshooting

Troubleshooting and Tuning
Usually, it is quite easy to set up a VPN using OpenVPN. This is one of the most
attractive features of OpenVPN compared to other VPN solutions. However,
sometimes it is necessary to troubleshoot a non-working setup or to tune an existing
setup to gain performance.
Troubleshooting and tuning OpenVPN are often overlooked topics. The OpenVPN
log files on both client and server side provide a lot of information, but you have
to know how to read them. There are also quite a few common mistakes to make
when setting up the client and server configuration files. In this chapter, you will
learn how to interpret the OpenVPN log files and how to detect and fix some of
these common mistakes.
Finally, there is a large difference between a working setup and a setup that works
well. Your OpenVPN setup may be functioning correctly, yet users may still
complain about poor performance. Getting the maximum performance out of an
OpenVPN setup can seem like black magic. In this chapter, you will learn some of
this black magic.
The following topics will be covered in this chapter:
• How to read the log files

• Fixing common configuration mistakes
• Troubleshooting routing issues
• How to optimize performance using ping and iperf
Copyright © 2015. Packt Publishing, Limited. All rights reserved.
• Analyzing OpenVPN traffic using tcpdump
[ 275 ]
Keijser, Jan Just, and Crist, Eric F. Mastering OpenVPN : Master Building and Integrating Secure Private Networks Using OpenVPN, Packt Publishing, Limited, 2015. ProQuest Ebook
Central, http://ebookcentral.proquest.com/lib/linkgroup-ebooks/detail.action?docID=4191311.
Created from linkgroup-ebooks on 2023-10-18 17:38:15.
How to read the log files

Debugging a non-working setup can seem like a daunting task at first. Where should
you begin? Luckily, OpenVPN provides excellent logging and debugging facilities.
However, with increased logging verbosity, it also becomes increasingly difficult to
read these log files. The default log level of OpenVPN is 1, but it is recommended
that you set the verbosity to 3. This often gives the administrator enough information
to detect setup issues, while keeping the performance penalty to a minimum.
Setting the verbosity to 5 or higher is recommended only for debugging purposes,

as it will severely affect performance.
Every example in this book so far has included the setting verb 3. First, we will
go through both client and server log files for a working setup with this verbosity.
It is important to understand, and possibly even store, the log files of a working
connection. When trying to find a bug in a non-working setup, it is very useful to
compare the log files of the non-working case with those of the working setup.
Start the server using the default configuration file basic-udp-server.conf:

[root@server]# openvpn --config basic-udp-server.conf
Do not connect to the client yet. The server log file will now contain the following:
1 14:53:27 OpenVPN 2.3.6 x86_64-redhat-linux-gnu
[SSL (OpenSSL)] [LZO] [EPOLL] [PKCS11] [MH] [IPv6]
built on Dec 2 2014
2 14:53:27 library versions: OpenSSL 1.0.1e-fips 11 Feb 2013,
LZO2.03
3 14:53:27 Diffie-Hellman initialized with 2048 bit key
4 14:53:31 WARNING: this configuration may cache passwords in
memory -- use the auth-nocache option to prevent this
5 14:53:31 Control Channel Authentication: using
'/etc/openvpn/movpn/ta.key' as a OpenVPN static key
file
6 14:53:31 Outgoing Control Channel Authentication: Using 160
bit message hash 'SHA1' for HMAC authentication
7 14:53:31 Incoming Control Channel Authentication: Using 160

bit message hash 'SHA1' for HMAC authentication
8 14:53:31 Socket Buffers: R=[16777216->131072]
S=[16777216->131072]
[ 276 ]
Chapter 9
9 14:53:31 TUN/TAP device tun0 opened

10 14:53:31 TUN/TAP TX queue length set to 100
11 14:53:31 do_ifconfig, tt->ipv6=0, tt-did_ifconfig_ipv6_setup=0
12 14:53:31 /sbin/ip link set dev tun0 up mtu 1500
13 14:53:31 /sbin/ip addr add dev tun0 10.200.0.1/24
broadcast 10.200.0.255
14 14:53:31 GID set to nobody
15 14:53:31 UID set to nobody
16 14:53:31 UDPv4 link local (bound): [undef]
17 14:53:31 UDPv4 link remote: [undef]
18 14:53:31 MULTI: multi_init called, r=256 v=256
19 14:53:31 IFCONFIG POOL: base=10.200.0.2 size=252, ipv6=0
20 14:53:31 Initialization Sequence Completed
The timestamps at the front of each line have been abbreviated for the sake of clarity.
Let's go through this log file line by line:
• Lines 1 and 2 indicate the version and build date of OpenVPN itself, as well
as the libraries that OpenVPN depends on.
• Line 3 tells us that the server Diffie-Hellman parameters were initialized
successfully. The file specified in the server configuration line dh /etc/
openvpn/movpn/dh2048.pem was used for this.
• Line 4 is a warning that is almost always printed. There has been discussion
among the developers whether this line should be removed or not. In the
end, it was decided that, for security reasons, it is best to print out this
warning. Unless you are extremely paranoid about security, you can ignore
this warning line.
• Line 5 indicates that the control channel is protected using the tls-auth
configuration option, and that OpenVPN was able to successfully read the
specified file.
• Lines 6 and 7 tell us that two SHA1 keys are derived from the tls-auth
file, and are now used to sign (hash) the outgoing traffic and to check the
incoming traffic.
• Line 8 shows the size of the Receive (R) and Send (S) buffers that
OpenVPN uses. These parameters are useful only when tuning a working
setup, as we will see later in this chapter.
[ 277 ]
• Lines 9 and 10 show that OpenVPN was able to successfully open a tun
device and was able to set the packet queue depth for this device to 100.
• Line 11 to line 13 shows the IPv4 settings that are used for this server
configuration. They also list that no IPv6 settings were specified. The
settings listed here are a translation of the server configuration line server
10.200.0.0 255.255.255.0.
• Lines 14 and 15 are the result of specifying group nobody and user nobody
in the server configuration file, respectively.
• Lines 16 and 17 show that OpenVPN is listening for UDP traffic and is bound
to an undefined interface (0.0.0.0). This is the result of specifying proto udp
and bind in the server configuration file.
• Line 18 tells us that this is a multi-client setup with real and virtual address
table hash sizes of 256.
• Line 19 lists the range of pool addresses available to OpenVPN clients that
can connect to this server. It is also part of the translation of the server
configuration line server 10.200.0.0 255.255.255.0.
• Line 20 is the magical line that tells us that the server started up successfully
and that the initialization has completed. The server is now ready to accept
connections from incoming clients.
Next, we launch the client and watch the server-side log file:
[root@client]# openvpn --config basic-udp-client.conf
After that, we will go through the client-side log file as well:

21 15:30:37 <CLIENT-IP>:39086 TLS: Initial packet from
[AF_INET]<CLIENT-IP>:39086, sid=071ba589 7e9ff2a0
22 15:30:37 <CLIENT-IP>:39086 VERIFY OK: depth=1, C=ZA,
ST=Enlightenment, L=Overall, O=Mastering OpenVPN,
CN=Mastering OpenVPN, [email protected]
23 15:30:37 <CLIENT-IP>:39086 VERIFY OK: depth=0, C=ZA,
ST=Enlightenment, O=Mastering OpenVPN, CN=client3,
[email protected]
24 15:30:37 <CLIENT-IP>:39086 Data Channel Encrypt: Cipher
'BF-CBC' initialized with 128 bit key

25 15:30:37 <CLIENT-IP>:39086 Data Channel Encrypt: Using 160 bit
message hash 'SHA1' for HMAC authentication
[ 278 ]
Chapter 9
26 15:30:37 <CLIENT-IP>:39086 Data Channel Decrypt: Cipher

'BF-CBC' initialized with 128 bit key
27 15:30:37 <CLIENT-IP>:39086 Data Channel Decrypt: Using 160 bit
message hash 'SHA1' for HMAC authentication
28 15:30:37 <CLIENT-IP>:39086 Control Channel: TLSv1, cipher
TLSv1/SSLv3 DHE-RSA-AES256-SHA, 2048 bit RSA
29 15:30:37 <CLIENT-IP>:39086 [client3] Peer Connection Initiated
with [AF_INET]<CLIENT-IP>:39086
30 15:30:37 client3/<CLIENT-IP>:39086 MULTI_sva: pool returned
IPv4=10.200.0.2, IPv6=(Not enabled)
31 15:30:37 client3/<CLIENT-IP>:39086 MULTI: Learn: 10.200.0.2 →
client3/<CLIENT-IP>:39086
32 15:30:37 client3/<CLIENT-IP>:39086 MULTI: primary virtual IP
for client3/<CLIENT-IP>:39086: 10.200.0.2
33 15:30:39 client3/<CLIENT-IP>:39086 PUSH: Received control
message: 'PUSH_REQUEST'
34 15:30:39 client3/<CLIENT-IP>:39086 send_push_reply():
safe_cap=940
35 15:30:39 client3/<CLIENT-IP>:39086 SENT CONTROL [client3]:
'PUSH_REPLY,route-gateway 10.200.0.1,topology subnet,
ping 10,ping-restart 60,
ifconfig 10.200.0.2 255.255.255.0' (status=1)
Let's go through the new log entries:
• Line 21 indicates that an initial packet was received from the client with the
IP address <CLIENT-IP>. Normally, a full IPv4 address is listed here.
• Lines 22 and 23 show the verification process of the certificate provided by
the OpenVPN client. The important part in these log lines is VERIFY-OK.
• Lines 24 to 27 list the encryption and decryption cipher used, as well as
the SHA1 hashes used to hash incoming and outgoing traffic on the data
channel. BF-CBC (Blowfish Cipher Block Chaining) is the current default
cipher for OpenVPN.
• Line 28 shows the TLS cipher used to protect the OpenVPN control channel.
The cipher listed here is very similar to the encryption cipher used by a
secure web server.
[ 279 ]
• Line 29 indicates that the client client3 from the IP address <CLIENT-IP>
successfully passed the authentication process.
• Lines 30 to 32 list the pool address that will be assigned to this client.
• Lines 33 to 34 show that the client asked for configuration information
(PUSH REQUEST), and the reply from the server that it will send a
push_reply.
• Line 35 shows the contents of the push_reply message with all the
configuration information for this client. This line is extremely useful
when debugging an OpenVPN setup, as it shows most of the information
that an OpenVPN server has for a particular client, regardless of the
configuration file used.
Similarly, here's the client log file (note the timestamps and match them against the
timestamps from the server log file):
1 15:30:37 OpenVPN 2.3.6 x86_64-redhat-linux-gnu
[SSL (OpenSSL)] [LZO] [EPOLL] [PKCS11] [MH] [IPv6]
built on Dec 2 2014
2 15:30:37 library versions: OpenSSL 1.0.1e-fips 11 Feb 2013,
LZO 2.03
3 15:30:37 Control Channel Authentication: using
'/etc/openvpn/movpn/ta.key' as a OpenVPN static key
file
4 15:30:37 UDPv4 link local: [undef]
5 15:30:37 UDPv4 link remote: [AF_INET]<SERVER-IP>:1194
6 15:30:37 [Mastering OpenVPN Server] Peer Connection Initiated
with [AF_INET]<SERVER-IP>:1194
7 15:30:39 TUN/TAP device tun0 opened
8 15:30:39 do_ifconfig, tt->ipv6=0, tt-did_ifconfig_ipv6_setup=0
9 15:30:39 /sbin/ip link set dev tun0 up mtu 1500
10 15:30:39 /sbin/ip addr add dev tun0 10.200.0.2/24
broadcast 10.200.0.255
11 15:30:39 Initialization Sequence Completed
[ 280 ]
Chapter 9
Let's go through the new log entries:
• Lines 1 and 2 are very similar to the lines from the server log.
• Line 3 indicates that the control channel is protected by using the tls-auth
configuration option, and that OpenVPN was able to successfully read the
specified file.
• Lines 4 and 5 tell us that the client did not bind to a local IP address, and that
a UDP connection was established with the server at the IP address <SERVER-
IP> and port 1194.
• Line 6 lists that the connection with the OpenVPN server that identifies itself
as Mastering OpenVPN Server was established successfully. The name of
the server is retrieved from the common name of the server-side certificate.
• Line 7 tells us that OpenVPN was able to open the TUN device tun0.
• Lines 8 to10 list the IPv4 information that the server pushed towards this
client, and it shows that the IP address and netmask are set using the
standard Linux /sbin/ip command.
• Line 11 is again the magical line that tells us that the VPN connection was
established successfully, and that we can now securely communicate with
the OpenVPN server. However, as we will see later on, the error messages
may have yet to occur.
Detecting a non-working setup

An OpenVPN setup can fail for many reasons. In the next section, we will go through
a list of common failures. First, let's take a look at what is shown in the log files when
a connection attempt fails. Failures can occur very early in the connection attempt, or
even after the Initialization Sequence Completed line.
If we use the wrong tls-auth file, the connection will fail very early on. This
is exactly the reason to use a tls-auth file, as it will minimize the load on our
OpenVPN server when rogue clients attempt access. A client that attempts to
connect without specifying a tls-auth file will show up in the server logs as follows:
16:40:31 Authenticate/Decrypt packet error:
packet HMAC authentication failed
16:40:31 TLS Error: incoming packet authentication failed from

[AF_INET]<CLIENT-IP>:49956
[ 281 ]

Nothing else is reported about this client, as the OpenVPN server rejects the
connection attempt immediately. From the timestamps in the log file, we can see that
the client is increasing the delay time between connection attempts with every failed
connection. If no connection can be made in 60 seconds, then the client will abort:
TLS Error: TLS key negotiation failed to occur within 60 seconds (check
your network connectivity)
TLS Error: TLS handshake failed
The second connection failure will only become apparent after the connection seems
to have been successfully initialized. For this, we specify the use of a different
encryption cipher on one side, but forget to do so on the other. The client log file will
now show the following:
16:56:20 /sbin/ip link set dev tun0 up mtu 1500
16:56:20 /sbin/ip addr add dev tun0 10.200.0.2/24 broadcast 10.200.0.255
16:56:20 Initialization Sequence Completed
16:56:30 Authenticate/Decrypt packet error: cipher final failed
16:56:40 Authenticate/Decrypt packet error: cipher final failed
Thus, at first the connection seems to have been established successfully (lines 1 to 3),
but after 10 seconds, data channel encryption and decryption is failing.
[ 282 ]
Chapter 9
If the Windows GUI had been used in this case, the GUI icon would
have turned green but the VPN itself would not be functional!
Most configuration issues will be reported at either the server or the client end
during initialization. Routing issues, which are much more common, will usually not
be reported by OpenVPN. Hence, different troubleshooting techniques are required.
Fixing common configuration mistakes

When setting up an OpenVPN configuration, there are a few common mistakes
that are easily made. These configuration mistakes can be roughly divided up
into four categories:
• Certificate (PKI) errors and mismatches

• Option mismatches, such as tun versus tap, ciphers, and compression
• Insufficient privileges to run OpenVPN
• Routing mistakes
In this section, we will go through the first three of these categories. Routing
mistakes will be discussed later in this chapter.
Wrong CA certificate in the client

configuration
The client configuration file will almost always contain three lines like this:
ca ca.crt
cert client.crt
key client.key
These certificate and private key files were created in Chapter 3, PKIs and Certificates,
and are used extensively in subsequent chapters.
[ 283 ]
The CA file, however, does not need to specify the certificate authority that was
used to sign the client certificate file. It must be the public certificate of the certificate
authority that was used to sign the server certificate. If the server certificate was
signed by a different CA, then the client will refuse to connect to the server.
This can be seen in the client-side log file:
UDPv4 link remote: [AF_INET]<SERVER-IP>:1194
VERIFY ERROR: depth=1, error=self signed certificate in certificate
chain: C=ZA, ST=Enlightenment, L=Overall, O=Mastering OpenVPN,
CN=Mastering OpenVPN, [email protected]
TLS_ERROR: BIO read tls_read_plaintext error: error:14090086:SSL
routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify failed
TLS Error: TLS object -> incoming plaintext read error
TLS Error: TLS handshake failed
In this case, no errors are logged on the server side, as the client certificate is
considered valid by the server.
The only thing logged on the server is as follows:

<CLIENT-IP>:42472 TLS: Initial packet from
[AF_INET]<CLIENT-IP>:42472, sid=9a1e4a84 cdbb6926
[AF_INET]<CLIENT-IP>:51441, sid=17d3c89b 6999ae97
[AF_INET]<CLIENT-IP>:43513, sid=4609202f 4c91c23d
This shows the successive connection attempts that are made by the OpenVPN client.
How to fix
Ensure that the right CA file is specified in the client configuration file.
Client certificate not recognized by the server

If the client certificate is not recognized by the server, the server will refuse access to
it. This can happen if either the wrong (or a rogue) client certificate is used, or if the
client's certificate was revoked and the crl-verify option is specified in the server
configuration file.
[ 284 ]
Chapter 9
The following entries will show up in the server's log file if an unknown client
attempts to connect to the OpenVPN server:
[AF_INET]<CLIENT-IP>:57072, sid=a175f1be 6faed111
<CLIENT-IP>:57072 VERIFY ERROR: depth=0, error=unable to get
local issuer certificate: C=NL, O=Cookbook, CN=client1,
name=Cookbook Client, [email protected]
<CLIENT-IP>:57072 TLS_ERROR: BIO read tls_read_plaintext error:
error:140890B2:SSL routines:SSL3_GET_CLIENT_CERTIFICATE:
no certificate returned
<CLIENT-IP>:57072 TLS Error: TLS object -> incoming plaintext
read error
<CLIENT-IP>:57072 TLS Error: TLS handshake failed
The server cannot verify the client's certificate, as it does not recognize the CA
certificate that was used to sign it. It therefore refuses to allow this client to connect.
On the client side, no messages are printed in the log file for 60 seconds, after which
the initial handshake times out and a new connection attempt is made:
13:24:23 UDPv4 link local: [undef]
13:24:23 UDPv4 link remote: [AF_INET]<SERVER-IP>:1194
13:25:23 TLS Error: TLS key negotiation failed to occur within
60 seconds (check your network connectivity)
13:25:23 TLS Error: TLS handshake failed
13:25:23 SIGUSR1[soft,tls-error] received, process restarting
13:25:25 Control Channel Authentication: using
'/etc/openvpn/movpn/ta.key' as a OpenVPN static key file
13:25:25 UDPv4 link local: [undef]
13:25:25 UDPv4 link remote: [AF_INET]<SERVER-IP>:1194
[ 285 ]
How to fix
Ensure that the client's certificate is recognized by the server. This can be done either
by specifying the proper CA certificate in the server configuration file, or by adding
the CA certificate to a stacked CA certificate file in the server configuration file:
# cat CA1.crt CA2.crt > /etc/openvpn/movpn/ca-stack.pem
Next, use the following in the server configuration:

ca /etc/openvpn/movpn/ca-stack.pem.
This way, client-side certificates that are signed by either CA1.crt or CA2.crt will be
accepted by the server.
Of course, if this is a rogue client that is attempting to connect, then a more

appropriate solution might be to blacklist the IP address from which the
client is connecting.
Client certificate and private key mismatch

If the certificate and private key on the client do not match, then OpenVPN will
not even attempt to connect to the server. The following error will be printed in
the log file instead:
Cannot load private key file /etc/openvpn/movpn/client1.key:
error:0B080074:x509 certificate routines:
X509_check_private_key:key values mismatch
Error: private key password verification failed
Exiting due to fatal error
This problem can occur especially when the certificate and private key are renewed;
it is a common mistake to use the old private key with the new certificate.
How to fix
Ensure that the client's certificate and private key match. Surprisingly, there is no
easy-to-use tool for this. To find out if the certificate and private key belong together,
we can use the following commands and look for the modulus sections:
$ openssl x509 -text -noout -in client1.crt

[…]
Public Key Algorithm: rsaEncryption
Public-Key: (2048 bit)
[ 286 ]
Chapter 9
Modulus:
00:b2:17:bd:31:6d:56:d9:eb:c9:09:98:e2:c1:48:
c9:6a:e4:4a:6b:54:52:ea:1e:60:94:6b:cb:5e:d5:
a1:ef:83:05:f8:cf:a4:06:df:06:ee:d6:c8:75:65:
de:a7:96:68:a1:41:d1:9d:f0:2c:84:3f:ca:b9:d2:
e8:07:af:37:48:24:69:57:4e:09:70:66:47:6c:47:
36:4d:c9:29:13:eb:ed:c1:aa:cd:36:84:3c:55:18:
bc:ce:01:34:b5:89:04:dc:09:c5:ea:f2:57:9f:c2:
f5:c1:05:dd:66:4d:11:13:05:47:46:26:1a:55:18:
51:bd:89:65:ba:0d:89:bd:ea:03:58:5e:d3:d9:96:
a5:5e:2f:5f:b9:c8:88:fc:48:95:cb:4a:b2:12:3b:
b5:ed:4c:40:4c:50:8d:1d:eb:a5:c9:c0:e6:2c:ec:
01:0a:56:ac:db:9e:e7:56:f0:06:f7:ba:b6:ac:de:
41:d4:fb:b3:d6:f5:fe:13:b4:03:81:d9:f7:7c:2e:
60:2f:9c:5a:81:eb:2e:3a:e1:c4:8b:f8:b6:8d:2d:
f7:ec:7a:f6:2c:ff:af:1c:d2:7b:58:ca:9e:d1:f4:
ed:8a:7a:35:00:97:a3:35:dd:79:02:b4:79:9a:66:
3c:5e:c8:4d:87:eb:68:5d:45:29:73:70:7f:61:28:
67:b1
$ openssl rsa -text -noout -in client1.key

Private-Key: (2048 bit)
modulus:
00:b2:17:bd:31:6d:56:d9:eb:c9:09:98:e2:c1:48:
c9:6a:e4:4a:6b:54:52:ea:1e:60:94:6b:cb:5e:d5:
a1:ef:83:05:f8:cf:a4:06:df:06:ee:d6:c8:75:65:
de:a7:96:68:a1:41:d1:9d:f0:2c:84:3f:ca:b9:d2:
e8:07:af:37:48:24:69:57:4e:09:70:66:47:6c:47:
36:4d:c9:29:13:eb:ed:c1:aa:cd:36:84:3c:55:18:
bc:ce:01:34:b5:89:04:dc:09:c5:ea:f2:57:9f:c2:
f5:c1:05:dd:66:4d:11:13:05:47:46:26:1a:55:18:
51:bd:89:65:ba:0d:89:bd:ea:03:58:5e:d3:d9:96:
a5:5e:2f:5f:b9:c8:88:fc:48:95:cb:4a:b2:12:3b:
b5:ed:4c:40:4c:50:8d:1d:eb:a5:c9:c0:e6:2c:ec:
[ 287 ]
01:0a:56:ac:db:9e:e7:56:f0:06:f7:ba:b6:ac:de:
41:d4:fb:b3:d6:f5:fe:13:b4:03:81:d9:f7:7c:2e:
60:2f:9c:5a:81:eb:2e:3a:e1:c4:8b:f8:b6:8d:2d:
f7:ec:7a:f6:2c:ff:af:1c:d2:7b:58:ca:9e:d1:f4:
ed:8a:7a:35:00:97:a3:35:dd:79:02:b4:79:9a:66:
3c:5e:c8:4d:87:eb:68:5d:45:29:73:70:7f:61:28:
67:b1
[…]
If we look closely at the modulus from both the public key (certificate) and the
private key, we can see that they are equal. Thus, this certificate and private key
belong together.
When comparing moduli, it is often sufficient to compare the first

few bytes and then the last few bytes.
The auth and tls-auth key mismatch

The auth and tls-auth options are used to authenticate both control channel and
data channel packets using an HMAC signing algorithm. The default value for the
auth HMAC algorithm is SHA1, which uses 160-bit keys. There is no default value for
the tls-auth option, as it is not required. However, this option is recommended, as
it provides an extra layer of protection against DDoS attacks.
If the auth algorithm specified in the client and server configurations do not match,
then the server will not allow the client to begin the TLS security handshake.
Similarly, if the tls-auth files on the client and server mismatch, or if the wrong
direction parameter is given on either side, then the server will also not allow the
client to begin the TLS security handshake.
Normally, the following option is specified in the server configuration file:

tls-auth /etc/openvpn/movpn/ta.key 0
Correspondingly, on the client we have the following option:

[ 288 ]
Chapter 9
Here, the second parameter defines direction of the tls-auth keys used.
This parameter is not required, but it allows OpenVPN to use different hashing
(or HMAC) keys for incoming and outgoing traffic. The key used on the client to
sign outgoing traffic must match the key used on the server to verify incoming
traffic, and vice versa.
If the wrong tls-auth key file is used, or if direction is omitted or not correctly
specified, then the following entries will show up in the server log file:
Authenticate/Decrypt packet error: packet HMAC
authentication failed
TLS Error: incoming packet authentication failed from
In the meantime, the client will simply attempt to connect for 60 seconds before a
timeout occurs.
How to fix
Make sure that the same tls-auth file is used in both client and server configuration
files. Also, make sure that the direction parameter is specified correctly on both ends
(if used at all).
If you are still unsure of which HMAC keys are used for incoming and outgoing
connections, you can increase the log file verbosity to see the actual keys being
used by both client and server. Let's add the following to both client and server
configuration files:
verb 7
Now, both sides will print out a large amount of logging information at startup.
The lines to look for in the log file are on the server side:
Outgoing Control Channel Authentication:
Using 160 bit message hash 'SHA1' for HMAC authentication
HMAC KEY: 4660a714 7f4d33f9 d2f7c61a 9f1d5743 4bf9411e
HMAC size=20 block_size=20

Incoming Control Channel Authentication:
[ 289 ]

HMAC KEY: cd1f6d9c 88db5ec7 d7977322 e01d14f1 26ee4e22
The HMAC size=20 line corresponds to the fact that 160-bit message hashing using
SHA1 is used, as 160 bit is the same as 20 byte.
If the correct tls-auth file and direction parameter are used on the client side, we
will find the following:
HMAC KEY: cd1f6d9c 88db5ec7 d7977322 e01d14f1 26ee4e22
HMAC KEY: 4660a714 7f4d33f9 d2f7c61a 9f1d5743 4bf9411e
The incoming and outgoing control channel authentication keys are mirrored on the
client versus the server, ensuring proper TLS authentication.
The MTU size mismatch

OpenVPN uses two Maximum Transfer Unit (MTU) sizes:
• tun-mtu: This specifies the MTU setting of the tun adapter and specifies the
maximum size of each packet inside the VPN tunnel.
• link-mtu: This specifies the maximum size of each packet outside the tunnel.
This includes all padding, encryption, and authentication bits, but it is not
the actual packet size as it goes over the network. The actual packet size
cannot be determined beforehand, as the size of each packet can differ due to
compression and encryption algorithms.
[ 290 ]
Chapter 9
The default value of the tun-mtu parameter is 1,500 bytes, which is the default MTU
size of an Ethernet adapter as well. Under normal circumstances, we can use the
following formula to compute the link-mtu size from the tun-mtu size:
link-mtu = tun-mtu + constant
Here, constant depends on the configuration options used. Among the

configuration options that have an influence on this constant, we have the following:
• Compression options such as comp-lzo and comp-noadapt

• The initialization vector (IV) size of the encryption parameter of the
cipher option
• The fragment option that adds an extra byte
• The no-replay option that removes a byte
If we see link-mtu warning mismatch, this usually points to a misconfiguration

elsewhere in our client and server configuration files. As you will see in the
subsequent examples, a mismatch in link-mtu between the client and the server can
occur quite often. Normally, a VPN connection will not function correctly if there is a
link-mtu mismatch.
Resist the temptation to fix the link-mtu warning itself by

explicitly setting it. First, fix the other warnings that may have
caused the link-mtu warning to appear.
The link-mtu parameter is also of great value when tuning a VPN connection.
To get the maximum performance out of a VPN connection, we will need to ensure
that the packets are not fragmented by the operating system, as this will have a
drastic impact on performance. Over satellite-based links in particular, this can
decrease performance almost to a stand-still.
If a different MTU size is specified on the server side compared to the client side,
the following warning will appear in the log files:
WARNING: 'link-mtu' is used inconsistently,
local='link-mtu 1441', remote='link-mtu 1541'
WARNING: 'tun-mtu' is used inconsistently,
local='tun-mtu 1400', remote='tun-mtu 1500'
This shows that for a fairly default configuration, the link-mtu overhead is actually
41 bytes. Here, we have added to the client configuration file:
tun-mtu 1400
[ 291 ]
At this point, the VPN connection will function. Performance will be limited,
however, as packets need to be fragmented and reassembled. It is possible to trigger
an error with this setup by sending large ICMP packets with the do not fragment
flag set. On Linux/FreeBSD, this can be done by using the following command:
$ ping -M do -s 1450 10.200.0.2
On Windows, we use the following:

C:\> ping -f -l 1450 10.200.0.2
This will result in 100 percent packet loss for the ping command, and it will also
show up in the log file:
Authenticate/Decrypt packet error:
This error message may appear confusing at first, but it is caused by the fact that the
sending party constructed and signed a packet which is larger than 1,400 bytes. The
client receives only the first 1,400 bytes of this packet and checks the signature, which
fails. It then rejects the packet and prints out the error.
How to fix
Make sure that, if you want to use the tun-mtu option that it is specified in both
client and server configuration files.
The Cipher mismatch

The encryption cipher that is used for OpenVPN's data channel can be specified by
using the following option with a default setting of BF-CBC:
cipher aes-256-cbc
If a different cipher is specified in the client configuration file than in the server
configuration file, then the log files on both sides will print out a warning message
but the VPN connection will come up. However, as soon as any traffic passes over
it, it will fail to decrypt. We can see this in the following excerpt from the client-side
log file:

WARNING: 'cipher' is used inconsistently,
[ 292 ]
Chapter 9
local='cipher AES-256-CBC', remote='cipher BF-CBC'

WARNING: 'keysize' is used inconsistently,
local='keysize 256', remote='keysize 128'
[Mastering OpenVPN Server] Peer Connection Initiated
with [AF_INET]<SERVER-IP>:1194
TUN/TAP device tun0 opened
do_ifconfig, tt->ipv6=0, tt->did_ifconfig_ipv6_setup=0
/sbin/ip link set dev tun0 up mtu 1500
/sbin/ip addr add dev tun0 10.200.0.2/24 broadcast 10.200.0.255
Initialization Sequence Completed
Authenticate/Decrypt packet error: cipher final failed
The three warnings that are printed initially show both the different type and the
different size of the cipher used. The default Blowfish cipher uses 128-bit strength,
whereas the AES-256 cipher uses 256-bit strength, resulting in a slightly larger
encrypted packet (link-mtu of 1,541 bytes for Blowfish versus a link-mtu of 1,557
bytes for AES-256).
How to fix
Make sure that the same cipher is specified in both client and server configuration
files. As both client and server log files print out the expected cipher, it is a relatively
easy error to fix.
It is currently not possible to push a cipher from the server to the

client. This is on the wish list of the OpenVPN developers, but it
has a substantial impact on the code. It is not going to be added to
OpenVPN before Version 2.4 or even 2.5.
The Compression mismatch

OpenVPN has the ability to compress all VPN traffic that is on-the-fly. For certain
types of traffic, such as plain web traffic, this can improve the performance of the
VPN, but it does add extra overhead to the VPN protocol. For incompressible traffic,
this option actually slightly decreases performance.
The option used to specify compression currently is as follows:

comp-lzo [no|yes|adaptive]
[ 293 ]
Note that we do not need to specify the second parameter. The default value is
adaptive, if compression is used.
As we will learn in Chapter 10, Future Directions, this option will be superseded by a
more generic compression option, allowing for different compression mechanisms.
It is possible to push a compression option from the server to the client, but only if
a compression option has been specified in the client configuration file itself. If the
client configuration file does not contain such an option, then the VPN connection
will fail. The client log file will show the following:
WARNING: 'comp-lzo' is present in remote config but
missing in local config, remote='comp-lzo'
[Mastering OpenVPN Server] Peer Connection Initiated with
[AF_INET]<SERVER-IP>:1194
write to TUN/TAP : Invalid argument (code=22)
The server log file will list the same WARNING messages, and it will also show
decompression warnings:
client3/<CLIENT-IP>:45113 Bad LZO decompression header byte: 42
Odd but true: if we wait long enough, the client will restart due to
the compression errors and will attempt to reconnect. This time,
however, the connection will succeed, as the comp-lzo option
still resides in memory.
[ 294 ]
Chapter 9
How to fix
Make sure that, if you want to use compression, the comp-lzo option is specified in
both client and server configuration files. With the comp-lzo option in the client-side
configuration file, we can now control the type of compression used at the server
side using a push option. Use the following:
comp-lzo no
push "comp-lzo no"
This will turn off compression, but unfortunately this is not the same as not
specifying any compression method at all. This will hopefully be addressed in a
future release.
The fragment mismatch

One of the most commonly used tuning parameters is the fragment option. You will
learn more about this option in the section How to optimize performance by using ping
and iperf later in this chapter.
Like the comp-lzo option, the fragment option does not need to be specified on
either side. However, we cannot specify it only on one side; it must be configured
on both. If it is specified only on one side, then it must also be specified on the other.
Technically speaking, it is not even necessary to use the same value for the fragment
option on both sides, but it is recommended.
If the fragment option is not specified on the client side but it is used on the server
side, then the VPN connection will not function properly, as can be seen in the
client's log:
WARNING: 'mtu-dynamic' is present in remote config but
missing in local config, remote='mtu-dynamic'
[AF_INET]194.171.96.101:1194

write to TUN/TAP : Invalid argument (code=22)
[ 295 ]
Again, it will appear as if the VPN has come up (Initialization sequence

completed), but the log file will fill up with code=22 error messages.
Note that the warning actually lists mtu-dynamic, which is the deprecated name
of this feature.
How to fix
Make sure that, if you want to use the fragment option, then it is specified in both
client and server configuration files.
Note that, unlike the comp-lzo option, this feature cannot be pushed from the server
to the client.
The tun versus tap mismatch

The most common use case for using a tap style network is a bridged setup, as
we have learned in Chapter 6, Client/Server Mode with Tap Devices. Not all devices,
however, support a tap-style network. Most notably, all Android and iOS devices
lack this capability. Hence, if we connect such a device to a tap style OpenVPN
server, the server log file will list warnings from these clients:
<CLIENT-IP>:39959 WARNING: 'dev-type' is used inconsistently,
local='dev-type tap', remote='dev-type tun'
<CLIENT-IP>:39959 WARNING: 'link-mtu' is used inconsistently,
<CLIENT-IP>:39959 WARNING: 'tun-mtu' is used inconsistently,
local='tun-mtu 1532', remote='tun-mtu 1500'
Apart from these warnings, the server will not detect anything about the connecting
clients. On the client, similar warnings will be listed, along with this one:
WARNING: Since you are using --dev tun with a point-to-point topology,
the second argument to --ifconfig must be an IP address. You are using
something (255.255.255.0) that looks more like a netmask. (silence this
warning with --ifconfig-nowarn)
As we cannot push the subnet topology in a tap-style setup, the client falls back to a
default Net30-style network. This type of network is inherently incompatible with a

tap-style network, but apart from that, the client does not list any warnings or errors.
Even if we were to (erroneously) add a topology subnet to suppress this warning on

the client, the VPN would still not function correctly.
[ 296 ]
Chapter 9
How to fix
Make sure that the same type of network (tun or tap) is used on both sides. If
you must use Android or iOS devices, then you must set up a tun-style server
configuration, as these operating systems do not support a tap-style network.
The client-config-dir issues

In Chapter 4, Client/Server Mode with tun Devices, we learned about CCD files and their
uses in the section Client-specific configuration – CCD files. CCD files are generally used
to connect a client-side LAN to the server network by using an iroute statement.
Experience on the OpenVPN mailing lists and forums has shown that the client-
config-dir option is susceptible to error and misconfiguration. The three main
reasons for this are as follows:
• The CCD file or the directory in which it resides, cannot be read by

OpenVPN after it switches to a safe user, such as nobody.
• The client-config-dir option was specified without an absolute path.
• The name of the CCD file is not listed correctly. Normally, the name of a
CCD file is the same as the name from /CN= field of the client certificate,
without /CN= part and without any extension!
With the normal log level, OpenVPN does not complain if it cannot find or read a
CCD file. It simply treats the incoming connection as a standard connection, and thus
the required iroute statement is never reached.
The easiest way to debug this is to temporarily add an extra option to the server
configuration:
ccd-exclusive
Restart the server and have the client try to reconnect. If the server cannot read the
appropriate CCD file for a connecting client, it will refuse access. If this happens, we
know that the CCD file has not been read. If the client can connect, then there is a
different issue—most likely a routing issue.
Another way to see what the OpenVPN server does with CCD files is to increase the
log level to 4 and reconnect a client for which a CCD file is listed. The contents of this
CCD file for the client with the certificate /CN=client1 are as follows:
ifconfig-push 10.200.0.99 255.255.255.0
iroute 192.168.4.0 255.255.255.0
[ 297 ]
This instructs the OpenVPN server to assign the VPN IP address 10.200.0.99 to
this client, and to route the subnet 192.168.4.0./24 via this client. The server log
file now lists the following:
<CLIENT-IP>:38876 [client1] Peer Connection Initiated with [AF_
INET]<CLIENT-IP>:38876
client1/<CLIENT-IP>:38876 OPTIONS IMPORT: reading client specific options
from: /etc/openvpn/movpn/clients/client1
client1/<CLIENT-IP>:38876 MULTI: Learn: 10.200.0.99 -> client1/<CLIENT-
IP>:38876
client1/<CLIENT-IP>:38876 MULTI: primary virtual IP for client1/<CLIENT-
IP>:38876: 10.200.0.99
client1/<CLIENT-IP>:38876 MULTI: internal route 192.168.4.0/24 ->
client1/<CLIENT-IP>:38876 MULTI: Learn: 192.168.4.0/24 ->
If the highlighted line is not present, then the CCD file is not read. Also, the next
lines starting with MULTI: show how the OpenVPN server interprets the lines
found in the CCD file. This can be important to further debug any iroute issues.
How to fix
If the server process cannot read the CCD file, then check the permissions of the
full path to the file, including all subdirectories leading to it. Make sure the user
specified with the user option has permission to read all directories and the CCD
file itself.
Make sure that the client-config-dir option lists an absolute path instead of
a relative one. Also, if we are using the chroot option (see the manual page for
details), then make sure the client-config-dir directory is visible inside
chroot-jail.
Use the ccd-exclusive option to quickly determine if OpenVPN can read the CCD
file. If it can, then increase the log level on the server side to see how OpenVPN
interprets the statements found in the CCD file.
[ 298 ]
Chapter 9
No access to the tun device in Linux

If OpenVPN is started with insufficient privileges, or if OpenVPN is set to drop
root privileges and switch to another userid (for example, nobody), then access to
the tun device can be lost. This can also occur if OpenVPN is used in a virtualized
environment, such as OpenVZ or Virtual Private Server (VPS).
If OpenVPN is started with insufficient privileges, the VPN connection will not
come up at all:
UDPv4 link local: [undef]
[AF_INET]<SERVER-IP>:1194
ERROR: Cannot ioctl TUNSETIFF tun: Operation not permitted
(errno=1)
Check userid or use sudo to switch to a privileged user before starting OpenVPN.
The more common scenario when insufficient privileges are available is after
an automatic restart of the VPN connection. Consider the following client
configuration file:
client
proto udp
remote openvpnserver.example.com
port 1194
dev tun
nobind
remote-cert-tls server
ca /etc/openvpn/movpn/movpn-ca.crt
cert /etc/openvpn/movpn/client3.crt
key /etc/openvpn/movpn/client3.key
user nobody
group nobody
[ 299 ]
This is the basic configuration file with two lines added at the bottom. When we
launch the VPN connection using this configuration file, the connection comes up
properly, but a warning is printed:
WARNING: you are using user/group/chroot/setcon without persist-tun --
this may cause restarts to fail
Indeed, after the VPN connection needs to be restarted (for example, due to a bad
network connection), the restart will fail:
[Mastering OpenVPN Server] Inactivity timeout (--ping-restart),
restarting
Mon Jun 1 16:51:50 2015 /sbin/ip addr del dev tun0 10.200.0.2/24
RTNETLINK answers: Operation not permitted
Linux ip addr del failed: external program exited with error
status: 2
SIGUSR1[soft,ping-restart] received, process restarting
WARNING: you are using user/group/chroot/setcon without
persist-key -- this may cause restarts to fail
Error: private key password verification failed
Here, we see that OpenVPN failed to restart as the user nobody was not allowed to
read the private key that was used for this connection. If we had specified a user that
did have the right permissions, we would have seen a different error:
ERROR: Cannot ioctl TUNSETIFF tun: Operation not permitted
(errno=1)
Note that, during the restart, OpenVPN cannot shut down the existing tun device
or delete any system routes. This will also be the case if persist-tun is used, but in
that case it would be harmless.
[ 300 ]
Chapter 9
How to fix
Add the following options to the client configuration file if you are using the user
and/or group options as well:
persist-tun
persist-key
Make sure you start OpenVPN with sufficient privileges.
Also, make sure that OpenVPN has the right SELinux security context, or try
running OpenVPN with SELinux set to permissive or disabled mode:
# setenforcing permissive
Missing elevated privileges in Windows

With some older versions of the OpenVPN installer program for Windows, the right
privileges were not set for the OpenVPN GUI application.
For this particular example, a single route was pushed from the OpenVPN server to
all clients:
push "route 192.168.122.0 255.255.255.0"
On Windows Vista and higher, OpenVPN needs elevated privileges to be able to add
or remove system routes. If these privileges are not present, the VPN will usually be
initialized correctly and the GUI-icon will turn green:
[ 301 ]
We can even ping the OpenVPN server on its VPN server IP address. However, the
log file in the OpenVPN GUI will show some errors:
The first line is actually tricky:

Warning: cannot open -log file: .....: Access is denied
The tricky part is that as soon as we click on the Disconnect button, the log is
gone, as it could not be written to disk! This is caused by the fact that the default
log directory C:\Program Files\OpenVPN\log is accessible only to a user with
elevated privileges.
The last few lines in the log file tell us that OpenVPN was unable to add the route
that was pushed by the server. Again, this is caused by the fact that the OpenVPN
program was run with insufficient privileges.
[ 302 ]
Chapter 9
How to fix
After restarting the OpenVPN GUI with elevated privileges (turn on Run as
Administrator), the route is added correctly. This can be seen from the routing table:
The pushed route, 192.168.122.0/24, is now present in the routing table, using the
server's VPN IP address 10.200.0.1 as the gateway.
Troubleshooting routing issues

Most of the questions asked on the OpenVPN e-mail lists and user forums are
actually routing questions. Setting up a VPN connection is one thing, but integrating
it into your existing network is quite another. To a novice, the difficult part is to see
where OpenVPN stops and where routing begins. This section is intended as a
step-by-step guide to troubleshoot routing issues in a fairly basic OpenVPN setup.
[ 303 ]
Consider the following network plan:
• The network at the main office location needs to be made accessible to a

secondary office and to people working from home
• The servers in the secondary office needs to be made accessible to the IT
department of the main office
• The people working from home only need to access the computer resources
at the main office
For this, an OpenVPN server is set up at the main office, with the employees
connecting as regular VPN clients, and with the secondary office connecting as a
special client, disclosing its own network.
Drawing a detailed picture

Before creating the configuration files for OpenVPN, draw a detailed picture of the
network layout, including all subnets, IP addresses, gateway IP addresses, interface
names, and more.
Site A LAN: Site B LAN:

192.168.3.0/24 gateway2: gateway1: 172.31.1.0/24
192.168.3.1 172.31.1.254
Internet
/WAN
VPN
Client Server
Lan: 192.168.3.17 Lan: 172.31.1.2
tun0: 10.200.0.200 tun0: 10.200.0.1
The public IP addresses used are not listed in this picture, but it is recommended to
do so. Also, the connections from people working from home are not included, but
they will connect to the public IP address of gateway1 in the preceding picture.
On gateway1, a port forwarding rule is added, so incoming and outgoing UDP

traffic on port 1194 is forwarded to the OpenVPN server at 172.31.1.2:1194.
As we need to disclose the network in the secondary office, we will also need to
use a client-config-dir file with the appropriate iroute statement.
[ 304 ]
Chapter 9
The server and client configuration files for this setup are already listed in Chapter 4,
Client/Server Mode with tun Devices, with some minor IP address changes. The new set
of configuration files is as follows:
proto udp
port 1194
dev tun
server 10.200.0.0 255.255.255.0
dh /etc/openvpn/movpn/dh2048.pem
ca /etc/openvpn/movpn/movpn-ca.crt
cert /etc/openvpn/movpn/server.crt
key /etc/openvpn/movpn/server.key
persist-key
persist-tun
keepalive 10 60
topology subnet
user nobody
group nobody
verb 3
daemon
log-append /var/log/openvpn.log
push "route 172.31.1.0 255.255.255.0"

client-config-dir /etc/openvpn/movpn/clients
route 192.168.3.0 255.255.255.0 10.200.0.1
[ 305 ]
This file is saved as movpn-09-01-server.conf. For the OpenVPN client

in the secondary office, a special certificate is created with the name /
CN=SecondaryOffice. The corresponding CCD file, therefore has the name /etc/
openvpn/movpn/clients/SecondaryOffice. Its contents are as follows:
ifconfig-push 10.200.0.200 255.255.255.0

iroute 192.168.3.0 255.255.255.0
For all clients, the basic-udp-client.conf or basic-udp-client.ovpn

configuration file can be used. This, by the way, shows the flexibility of OpenVPN's
configuration files. In most cases, there is no need to change the client configuration
files, even if the network layout on the server side is modified, or a secondary
network is brought into the VPN.
Next, we start the OpenVPN server and the secondary office client, and we make
sure that the CCD file is picked up. The VPN client in the secondary office can ping
the OpenVPN server at its VPN IP address, and so can a test user at home.
At this point, the VPN is working, but routing is not.
Start in the middle and work your way

outward
The most efficient method to troubleshoot this setup is to consider the VPN link
as the middle, and then work outwards step-by-step until all parts of the network
are connected. First, there are some tests to perform on the OpenVPN client in the
secondary office. For almost all tests, a simple ping command will suffice.
Note that it does not make sense to move on to the second test if the first test is
failing, and similarly, to the third test, if the second test is not yet working:
• Can the client reach the server's VPN IP address?

This should function; otherwise, there is a problem with our VPN. It could
be a very restrictive firewall/iptables setup on the server. The VPN server
IP should be private (RFC1918, typically) and, thus, won't be routable across
the general internet.
• Can the client reach the server's LAN IP address?

If this is not functioning, then most likely there is a firewall or iptables rule
that is blocking access. Check the inbound rules or try disabling the firewall
rules for debugging.
[ 306 ]
Chapter 9
• Can the client reach the server-side gateway IP address?

If not, then check the answers to the following questions:
°° Is IP forwarding enabled on the server?

°° Is there a firewall/iptables rule blocking forwarding access to the
server from a particular IP range?
°° Is there a firewall rule on the server-side gateway blocking access
from non-LAN IP addresses? (This would be a good security policy.)
If so, then it needs to be adjusted to allow traffic coming from the
VPN IP 10.200.0.0/24.
°° Is there a return route on the gateway to tell it where packets
originating from the VPN should go back to? Packets with
destination address in the 10.200.0.0/24 range should be forwarded
to the OpenVPN server at IP 172.31.1.2 on the router gateway1.
Note that this is usually not the case. The actual syntax for adding
such a route to the gateway depends on the model and firmware of
the router used.
• Can the client reach another server on the server-side LAN?
°° Does this server on the server-side LAN have the proper gateway
as the default gateway?
°° Is there a firewall rule on the server blocking access from non-LAN-
IP addresses? (This would actually be a good security policy!)
After making sure that the client can reach all machines on the server-side LAN, it is
time to make sure that the reverse is also true. Ensure that the OpenVPN server can
reach all machines on the LAN behind the secondary client. The tests to perform are
very similar:
• Can the server reach the client's VPN IP address?

This should function; otherwise, there is a problem with our VPN. It could
be a very restrictive firewall/iptables setup on the client. However, at this
point it is highly unlikely to be a problem. Better safe than sorry, though,
so let's test this.
• Can the server reach the client's LAN IP address?

If this is not functioning, then most likely there is a firewall/iptables rule
that is blocking access. Check the inbound rules.
[ 307 ]
• Can the server reach the client-side gateway IP address?

°° Is IP forwarding enabled on the secondary office client?

Is there a firewall/iptables rule blocking forwarding access on the
client from a particular IP range?
°° Is there a firewall rule on the client-side gateway blocking access
from non-LAN IP addresses? (This would be a good security policy.)
VPN IP 10.200.0.1.
°° Is there a return route on the gateway in the secondary office to
tell it where packets originating from the VPN should go back to?
Packets with the source address 10.200.0.1 should be forwarded
to the OpenVPN client at IP 192.168.3.17 on the router gateway2.
Note that this is usually not the case. The actual syntax for adding
such a route to the gateway depends on the model and firmware
of the router used. Also, note that we only allow packets from the
OpenVPN server itself to pass through, as all other clients do not
require access to this network.
• Can the OpenVPN server reach another machine on the client-side LAN?

°° Does this server on the client-side LAN have the proper gateway as
the default gateway?
°° Is there a firewall rule on the server blocking access from non-LAN-
IP addresses? (This would actually be a good security policy!)
At this point, the OpenVPN client in the secondary office should be able to reach
all machines in the server-side LAN, and the OpenVPN server in the main office
should be able to reach all machines in the client-side LAN. There is only one more
step: make sure that the servers on the server-side LAN can reach the servers on the
client-side LAN and vice versa. Again, there are four tests to perform, starting on a
machine on the server-side LAN:
• Can this machine reach the OpenVPN client's VPN IP address?

This should function, as the client can reach this machine, as a result of the
fourth test. However, better safe than sorry, so let's test this.
[ 308 ]
Chapter 9
• Can this machine reach the client's LAN IP address?

If this is not functioning, then most likely there is a firewall or iptables rule
that is blocking access. Check the inbound rules on the OpenVPN client.
• Can the server-side LAN machine reach the client-side gateway IP address?
°° Is IP forwarding enabled on the secondary office client? Is there a

firewall/iptables rule blocking forwarding access on the client for
a particular IP range? Note that packets coming from the machine
on the server-side LAN will have a different source address
(172.31.1.X) to that of the OpenVPN server itself (10.200.0.1).
°° Is there a firewall rule on the client-side gateway blocking access
from non-LAN-IP addresses? (This would be a good security policy.)
server-side LAN IP range 172.31.1.0/24. Similarly, a firewall rule
may need to be added on the server-side gateway to allow traffic
coming from the client-side LAN IP range 192.168.3.0/24.
°° Is there a return route on the gateway in the secondary office to tell it
where packets originating from the VPN should go back to? Packets
with source address 172.31.1.0/24 should be forwarded to the
OpenVPN client at IP 192.168.3.17 on the router gateway2. Note
that this is usually not the case.
• Can the server-side LAN machine reach another machine on the

client-side LAN?
°° Does the server on the client-side LAN have the proper gateway as
the default gateway?
°° Is there a firewall rule on the client-side machine blocking access
from non-LAN-IP addresses? (This would actually be a good
security policy!)
By methodically working through all of these steps, we can resolve almost all routing
issues. In some cases, more advanced debugging techniques may be required. This
may require us to temporarily disable firewall rules, so special care should be taken
before attempting this.
[ 309 ]
Find a time to temporarily disable firewall

There have been too many cases on the OpenVPN mailing lists of people not being
able to get routing to work, where it turned out to be a firewall or iptables rule that
was too restrictive. There is no need to disable all firewall rules, but if you get stuck
at one of the twelve steps listed previously, then try disabling the firewall related to
the device that you cannot reach or that you are sending traffic from.
If you need to make use of a NATted setup then make sure that
you do not disable the NATting rules.
If all else fails, use tcpdump

The low-level networking tool tcpdump is a great tool for testing connectivity. In
order to debug routing issues, we can use tcpdump to see if any traffic is arriving
at or leaving a particular network interface, and we can check the source and
destination addresses of this traffic. On a Windows client or server, it might be
easier to run Wireshark (http://www.wireshark.org), which provides similar
functionality, including a GUI.
In the twelve steps listed previously, the following tcpdump statements can help:
1. Run tcpdump -nnel -i tun0 on the server to see if any traffic is coming in
over the VPN at all.
2. Run tcpdump -nnel -i eth0 on the server (where eth0 is the LAN
interface of the server in use) to see if any traffic is coming in on the LAN
interface at all. If not, then most likely a firewall rule is dropping inbound
traffic on the tunnel interface.
3. Run tcpdump -nnel -i eth0 on the server to see if any traffic is leaving the
LAN interface with the following:
source address = 10.200.0.200
destination address = 172.31.1.254
Also, check whether we can see return traffic from the server-side gateway
with the source and destination addresses reversed.
[ 310 ]
Chapter 9
4. Again, run tcpdump -nnel -i eth0 on the server to see if any traffic is
leaving the LAN interface with the following packet headers:
source address = 10.200.0.200
destination address = 172.31.1.XXX
Here, 172.31.1.XXX is the IP address of the machine we are trying to reach

on the server-side LAN. Is there any return traffic seen?
And so on and so on for the remaining steps!
How to optimize performance by using

ping and iperf
Getting maximum performance out of an OpenVPN setup can be difficult to achieve.
On a clean Ethernet network, the default settings of OpenVPN are fairly good.
However, some tuning is required on gigabit-speed networks.
When an ADSL or cable-modem connection is used, performance is also usually

pretty good. However, under certain circumstances, the performance of our
OpenVPN tunnel can fall way behind the performance of the normal network.
These circumstances are almost always ISP-dependent, but nevertheless, it is
worthwhile exploring how to increase performance.
The key to optimizing performance is to have good tools for measuring performance
in the first place. Two basic, yet invaluable, tools for measuring network performance
are ping and iperf. The iperf tool is readily available on Linux, FreeBSD, and Mac
OS. There are ports available for Windows and even Android.
Using ping
Using ping, we can determine the optimal MTU size for our network. Most network
operators now provide their customers with an Ethernet-style MTU of 1,500 bytes.
This leads to an effective packet payload of 1,472 byes. The remaining 28 bytes are
the TCP/IP overhead for things like the source and destination address.
However, if there is a network between the client and the server that has a lower
MTU, then it can greatly increase performance to reduce the size of the OpenVPN
packets to just below that size. To find out what the maximum transfer size is for our
network, we use the following:
$ ping -M do -s 1472 www.example.org
[ 311 ]
On Windows, we use the following:

C:\> ping -f -l 1472 www.example.org
This will send ICMP packets to a remote server of our choice, with the do not
fragment flag set, instructing the network routers not to break up this packet into
smaller bits. If there is a network between the client and the server with a smaller
MTU, then the ping command will fail:
$ ping -M do -s 1472 www.example.org
PING www.example.org (IP) 1472(1500) bytes of data.
ping: local error: Message too long, mtu=1480
This tells us that performance will most likely improve if we use either a fragment
size of 1,480 or an MTU size that is 1,480 bytes instead of the default value of 1500.
Note that this is not a guarantee—only by measuring the actual VPN performance
we will know what the impact actually is.
Using iperf
By using iperf, we can measure the performance of a network, both inside and
outside a VPN tunnel. This will give us valuable insight into how much bandwidth
we are wasting by using a VPN tunnel.
Before measuring the performance of the VPN tunnel itself, always try to measure
the performance of the normal network. It will be quite hard to make the VPN
perform better than the underlying network.
First, launch iperf on the server with the following command:

$ iperf -s
Next, launch iperf on the client with the following command:

$ iperf -c openvpn.example.org
On the cable network that was used for testing, the result is as follows:
[ 312 ]
Chapter 9
This is actually the upload speed of the cable connection used. We can now test the
performance of the VPN tunnel over the same network:
[ 3] 0.0-10.8 sec 5.25 MBytes 4.09 Mbits/sec
Repeating the measurement yields very similar numbers, so it is fair to state that
the performance of the VPN tunnel is a few percent below that of the underlying
network. This actually makes sense, as the use of a VPN does introduce some
overhead for the encapsulation, encryption, and authentication (signing) of the
original data. It will be hard to further optimize this network.
Similarly, for the download speed of the cable connection used, we find that the
performance of the VPN tunnel is a few percent lower:
Performance of the underlying network is shown as follows:

Now, compare this to the VPN tunnel:

Again, we see a 4.5 percent drop in performance.
We can now use the fragment and mssfix parameters to see if we can increase
performance. There will be a little bit trial-and-error work in order to find the sweet
spot for a particular setup. It is not known what the exact sweet spot will be in
advance, but the method for finding it is always the same. Now, add the options to
both client and server configuration files:
fragment X
mssfix
By doing this and by varying X, we get the following results:
X (bytes) Download (Mbps) Upload (Mbps)

1200 37.9 3.94
1300 38.1 4.01
1400 38.4 4.04
1472 38.8 4.06
1500 37.6 3.98
<none> 39.0 4.09
[ 313 ]
We can conclude that OpenVPN's default settings are actually the sweet spot for
this network. We could repeat this exercise by varying the tun-mtu parameter, but
we would find the same result. However, it is advisable to first tune performance
by using the fragment parameter, as this parameter has less influence on the
forwarding of packets.
Gigabit networking
We will now perform the same procedure on a non-utilized gigabit Ethernet
network. The iperf performance of the underlying network is 950 Mbps up
and down.
When we launch the OpenVPN server using the basic-udp-server.conf

configuration and attach a client to it by using the basic-udp-client.conf
configuration file, we achieve the following iperf performance:
[ ID] Interval Transfer Bandwidth
[ 5] 0.0-10.0 sec 193 MBytes 161 Mbits/sec
There is now a clear drop in performance. Unfortunately, lowering the fragment

parameter does not help us here. With fragment 1200, we achieve 149 Mbps and
115 Mbps, respectively.
On high-speed networks, it also makes sense to experiment with the encryption

cipher. The servers used in this example are both capable of fast AES instructions,
thanks to the AES-NI extension that is present on the CPUs (a 2 GHz Xeon E5 2620
and a 3.5 GHz (turbo) Xeon E5 2643, respectively). Let's add the following:
cipher aes-256-cbc
We now get the following result:

On a capable CPU, the cipher has a large impact on performance. As OpenVPN is

a monolithic program, a high number of cores do not help at all. The clock speed of
the CPU is a dominant factor. By connecting a 3.8 GHz (turbo) Core i7 laptop to the
3.5 GHz Xeon E5-2643 server, we achieve a much higher throughput using the exact
same configuration:
[ 314 ]
Chapter 9
Thus, if you want to set up an OpenVPN tunnel across a high-speed network, then
the best advice is to use high clock speed CPUs that have support for the AES-NI
instruction set. With such a setup, it is possible to achieve network speeds of over 500
Mbps in both directions.
Analyzing OpenVPN traffic by using

tcpdump
The low-level networking tool tcpdump, or its GUI equivalent Wireshark, is a last
resort tool for troubleshooting network issues and network performance. In this
section, we will walk through the process of capturing and analyzing the encrypted
network traffic produced by OpenVPN.
First, we set up our standard OpenVPN network using the basic-udp configuration
files. On the client, there is also a web server running. We will use the wget
command on the server side to retrieve a file from the web server so that we can look
at the resulting network traffic.
We run tcpdump on the Ethernet interface and capture the network traffic while
doing a wget outside the tunnel:
wget -O /dev/null https://CLIENT-IP/test1
The resulting tcpdump output is as follows (modified for the sake of clarity):
[ 315 ]
As we can see, there are 13 packets to transfer a 5 KB text file. Most of these packets
were used to set up and tear down the connection, but there are four large packets
that were used to actually transfer the data. The first three of the four packets are
1,514 bytes in size, which is the maximum size of an Ethernet packet.
Next, we run the same wget command inside the tunnel. We now observe the
encrypted traffic on the Ethernet adapter:
Here, we see 22 packets being captured. The first and last two packets are OpenVPN
heartbeat packets and can be ignored. The remaining 18 packets are the encrypted
equivalent of the packets shown in the first tcpdump output. As we can see here, the
length of the packet is slightly smaller, and especially the payload of each packet is
quite a bit smaller: the largest UDP payload packet is 1,445 bytes. These 1,445 bytes
contain the encrypted and signed data from the wget command. In our setup, we did
not specify a fragment parameter, which means that OpenVPN 2.3 will default to an
internal fragmentation of 1,450 bytes.
The total size of each packet never exceeds 1,487 bytes, which is fairly close to
optimal: normally, packets should not exceed the MTU size, which is 1,500 bytes.
[ 316 ]
Chapter 9
This tcpdump screen dump also shows that there is no fragmentation occurring other
than within OpenVPN. This is good, as we want to avoid packet fragmentation by
the operating system, or the network, for maximum performance. If we had seen
packet fragmentation here, then this would have been an excellent indication that we
needed to add extra fragmentation in our OpenVPN configuration.
Let's take a look at what happens if we add fragment 1400 to our setup. We restart
the server and client and rerun the wget command:
With fragment 1400 added to our setup, we can see in the tcpdump output that the
packet payload is now 1,397 bytes in size, which is very close to the limit of 1,400.
We can also see that more packets are now needed to transfer the 5 KB text file over
the tunnel, which means a decrease in performance. From this screenshot, we can
conclude that we should remove the parameter again.
From the preceding screenshot and the previous one, we can also deduce that each
OpenVPN packets incurs a 42-byte overhead. This overhead partly contributes to the
overhead incurred by using any VPN solution. It does comprise the entire overhead,
as all network packets need to contain overhead information about the source
address, destination address, packet type, checksums, flags, and many more.
[ 317 ]
Finally, let's take a look at the contents of an actual encrypted OpenVPN packet. For
this, it is handy to use the Wireshark tool (http://www.wireshark.org). Wireshark
basically provides a GUI on top of the low-level tcpdump tool. It can decode the
contents of most types of network traffic, as we can see in the following screenshot
(the screenshot has been anonymized for privacy reasons):
This screenshot tells us the following things:
• The actual packet size is 1487 bytes.

• It contains Ethernet and IPv4 headers, like any network packet on an
Ethernet network.
• This is an OpenVPN packet with source port 35400 and destination
port 1194, which means that it is travelling from the client to the server.
It actually is one of the encrypted packets of the 5 KB file transfer from
the client to the server.
• The payload of the packet is an OpenVPN data packet (format version 1),
with a payload size of 1,487 bytes. Note that tcpdump reported 1,488 bytes
earlier, but Wireshark can decode the payload and see that the first byte is
an OpenVPN opcode.
This packet will be received by OpenVPN, checked for authentication, decrypted,

and decompressed (if we had specified). The resulting unencrypted packet is then
forwarded to the operating system's routing tables, which decide where to route the
packet to. In our case, the packet will remain on the server and will be given to the
wget process.
[ 318 ]
Chapter 9
Summary
In this chapter, you learned some basic techniques for troubleshooting and tuning
OpenVPN. You also got an insight into reading the client and server log files. You
learned how to detect and fix some of the most often-made mistakes. Most questions
on the OpenVPN e-mail list are about routing issues, therefore we discussed
detecting and fixing routing problems. Finally, there is a large difference between a
working setup and a well-working setup, so we looked at examples on how to detect
and address performance issues.
Of course, OpenVPN is not perfect, and therefore your non-working setup could also
be caused by a bug in OpenVPN itself. There are several channels for reporting bugs,
including an e-mail list ([email protected]), an IRC channel
(#openvpn on freenode.net IRC), and a forum website (https://forums.openvpn.
net). You can also report feature requests or wish lists to these channels, some of
which might make it into a future version of OpenVPN.
In the next chapter, you will learn what is new in the upcoming OpenVPN releases.
You will also learn what are the currently known issues with the OpenVPN codebase
and you will learn about the plans in place to address these issues.
[ 319 ]

Mastering OpenVPN Master Building and Integrating Chapter 9 Troubleshooting

Uploaded by

Copyright:

Available Formats

Mastering OpenVPN Master Building and Integrating Chapter 9 Troubleshooting

Uploaded by

Document Information

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Mastering OpenVPN Master Building and Integrating Chapter 9 Troubleshooting

Uploaded by

Copyright:

Available Formats

Troubleshooting and Tuning

The following topics will be covered in this chapter:

• How to read the log files

• Analyzing OpenVPN traffic using tcpdump

How to read the log files

Setting the verbosity to 5 or higher is recommended only for debugging purposes,

Start the server using the default configuration file basic-udp-server.conf:

7 14:53:31 Incoming Control Channel Authentication: Using 160

9 14:53:31 TUN/TAP device tun0 opened

Let's go through this log file line by line:

After that, we will go through the client-side log file as well:

'BF-CBC' initialized with 128 bit key

26 15:30:37 <CLIENT-IP>:39086 Data Channel Decrypt: Cipher

Let's go through the new log entries:

Let's go through the new log entries:

Detecting a non-working setup

16:40:31 TLS Error: incoming packet authentication failed from

16:40:33 TLS Error: incoming packet authentication failed from

but after 10 seconds, data channel encryption and decryption is failing.

Fixing common configuration mistakes

• Certificate (PKI) errors and mismatches

Wrong CA certificate in the client

The only thing logged on the server is as follows:

Client certificate not recognized by the server

Next, use the following in the server configuration:

Of course, if this is a rogue client that is attempting to connect, then a more

Client certificate and private key mismatch

$ openssl x509 -text -noout -in client1.crt

$ openssl rsa -text -noout -in client1.key

When comparing moduli, it is often sufficient to compare the first

The auth and tls-auth key mismatch

Normally, the following option is specified in the server configuration file:

Correspondingly, on the client we have the following option:

HMAC size=20 block_size=20

Incoming Control Channel Authentication:

The MTU size mismatch

Here, constant depends on the configuration options used. Among the

• Compression options such as comp-lzo and comp-noadapt

If we see link-mtu warning mismatch, this usually points to a misconfiguration

Resist the temptation to fix the link-mtu warning itself by

local='tun-mtu 1400', remote='tun-mtu 1500'

On Windows, we use the following:

The Cipher mismatch

WARNING: 'link-mtu' is used inconsistently,

local='cipher AES-256-CBC', remote='cipher BF-CBC'

It is currently not possible to push a cipher from the server to the

The Compression mismatch

The option used to specify compression currently is as follows:

The fragment mismatch

do_ifconfig, tt->ipv6=0, tt->did_ifconfig_ipv6_setup=0

Again, it will appear as if the VPN has come up (Initialization sequence

The tun versus tap mismatch

default Net30-style network. This type of network is inherently incompatible with a

Even if we were to (erroneously) add a topology subnet to suppress this warning on

The client-config-dir issues

• The CCD file or the directory in which it resides, cannot be read by

No access to the tun device in Linux

Make sure you start OpenVPN with sufficient privileges.

Missing elevated privileges in Windows