SSH SERVER

GOAL
Con gure secure command-line service on remote systems, using OpenSSH

OBJECTIVES
• Log in to a remote system and run commands using ssh
• Con gure key-based authentication for a user account to log in to remote systems securely without a
password
• Restrict direct logins as root and disable password-based authentication for the OpenSSH service

SECTIONS
• Accessing the Remote Command Line with SSH (and Guided Exercise
• Con guring SSH Key-Based Authentication (and Guided Exercise
• Customizing OpenSSH Service Con guration (and Guided Exercise

LAB
Con guring and Securing SS
fi
fi

fi
fi

fi
.

 ACCESSING THE REMOTE COMMAND LINE WITH SS

OBJECTIVE
After completing this section, you should be able log into a remote system and run command
using ssh

WHAT IS OPENSSH
OpenSSH implements the Secure Shell or SSH protocol in the Red Hat Enterprise Linux systems
The SSH protocol enables systems to communicate in an encrypted and secure fashion over a
insecure network
You can use the ssh command to create a secure connection to a remote system, authenticate as
speci c user, and get an interactive shell session on the remote system as that user. You may als
use the ssh command to run an individual command on the remote system without running a
interactive shell

SECURE SHELL EXAMPLE

The following ssh command would log you in on the remote server remotehost using th
same user name as the current local user. In this example, the remote system prompts you t
authenticate with that user's password

[user01@host ~]$ ssh remotehos

user01@remotehost's password: redha
...output omitted..
[user01@remotehost ~]

You can the exit command to log out of the remote system

[user01@remotehost ~]$ exi

logou
Connection to remotehost closed
[user01@host ~]

The next ssh command would log you in on the remote server remotehost using the user nam
user02. Again, you are prompted by the remote system to authenticate with that user's password

[user01@host ~]$ ssh user02@remotehos

user02@remotehost's password: shadowma
...output omitted..
[user02@remotehost ~]

This ssh command would run the hostname command on the remotehost remote system as th
user02 user without accessing the remote interactive shell

[user01@host ~]$ ssh user02@remotehost hostnam

user02@remotehost's password: shadowma
remotehost.lab.example.co
[user01@host ~]

Notice that the preceding command displayed the output in the local system's terminal

IDENTIFYING REMOTE USER

The w command displays a list of users currently logged into the computer. This is especially usefu
to show which users are logged in using ssh from which remote locations, and what they are doing

[user01@host ~]$ ssh user01@remotehos

user01@remotehost's password: redha
[user01@remotehost ~]$
16:13:38 up 36 min, 1 user, load average: 0.00, 0.00, 0.0
t

fi
.

 USER TTY FROM LOGIN@ IDLE JCPU PCPU WHA

user02 pts/0 172.25.250.10 16:13 7:30 0.01s 0.01s -bas
user01 pts/1 172.25.250.10 16:24 3.00s 0.01s 0.00s
[user02@remotehost ~]

The preceding output shows that the user02 user has logged in to the system on the pseudotermina
0 at 16:13 today from the host with the 172.25.250.10 IP address, and has been idl
at a shell prompt for seven minutes and thirty seconds. The preceding output also shows that th
user01 user has logged in to the system on the pseudo-terminal 1 and has been idle since sinc
last three seconds after executing the w command

SSH HOST KEY

SSH secures communication through public-key encryption. When an SSH client connects to a
SSH server, the server sends a copy of its public key to the client before the client logs in. This i
used to set up the secure encryption for the communication channel and to authenticate the serve
to the client
When a user uses the ssh command to connect to an SSH server, the command checks to see if i
has a copy of the public key for that server in its local known hosts les. The system administrato
may have pre-con gured it in /etc/ssh/ssh_known_hosts, or the user may have a ~/.ssh
known_hosts le in their home directory that contains the key
If the client has a copy of the key, ssh will compare the key from the known hosts les for tha
server to the one it received. If the keys do not match, the client assumes that the network traf
to the server could be hijacked or that the server has been compromised, and seeks the user'
con rmation on whether or not to continue with the connection
**********************
NOT
Set the StrictHostKeyChecking parameter to yes in the user-speci c ~/.ssh
con g le or the system-wide /etc/ssh/ssh_con g to cause the ss
command to always abort the SSH connection if the public keys do not match
If the client does not have a copy of the public key in its known hosts les, the ssh comman
will ask you if you want to log in anyway. If you do, a copy of the public key will be saved in you
~/.ssh/known_hosts le so that the server's identity can be automatically con rmed in th
future
***************************

[user01@host ~]$ ssh newhos

The authenticity of host 'remotehost (172.25.250.12)' can't be established
ECDSA key ngerprint is SHA256:qaS0PToLrqlCO2XGklA0iY7CaP7aPKimerDoaUkv720
Are you sure you want to continue connecting (yes/no)? ye
Warning: Permanently added 'newhost,172.25.250.12' (ECDSA) to the list of know
hosts
user01@newhost's password: redha
...output omitted..
[user01@newhost ~]

SSH Known Hosts Key Managemen

If a server's public key is changed because the key was lost due to hard drive failure, or replaced fo
some legitimate reason, you will need to edit the known hosts les to make sure the entry for th
old public key is replaced with an entry with the new public key in order to log in without errors
Public keys are stored in the /etc/ssh/ssh_known_hosts and each users' ~/.ssh
known_hosts le on the SSH client. Each key is on one line. The rst eld is a list of hostname
and IP addresses that share that public key. The second eld is the encryption algorithm for th
key. The last eld is the key itself

[user01@host ~]$ cat ~/.ssh/known_host

remotehost,172.25.250.11 ecdsa-sha2-nistp25
AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBOsEi0e+FlaNT6jul8Ag5N
+RViZl0yE2w6iYUr+1fPtOIF0EaOgFZ1LXM37VFTxdgFxHS3D5WhnIfb+68zf8+w

Each remote SSH server that you conect to stores its public key in the /etc/ssh directory in le
with the extension .pub

[user01@remotehost ~]$ ls /etc/ssh/*key.pu

/etc/ssh/ssh_host_ecdsa_key.pub /etc/ssh/ssh_host_ed25519_key.pub /etc/ssh
ssh_host_rsa_key.pu
fi
fi
.

fi
fi
.

fi
fi
fi
.

fi
$

fi
.

fi
h

fi
.

fi
.

fi
fi
/

fi
=

fi
n

fi
/

fi
.

fi
j

fi
d

fi
.

 NOT
It is a good practice to add entries matching a server's ssh_host_*key.pu
les to your ~/.ssh/known_hosts le or the system-wide /etc/ssh
ssh_known_hosts le

CONFIGURING SSH KEY-BASED AUTHENTICATIO

OBJECTIVE
After completing this section, you should be able to con gure a user account to use key-base
authentication to log in to remote systems securely without a password

SSH KEY-BASED AUTHENTICATIO

You can con gure an SSH server to allow you to authenticate without a password by using keybase
authentication. This is based on a private-public key scheme
To do this, you generate a matched pair of cryptographic key les. One is a private key, the othe
a matching public key. The private key le is used as the authentication credential and, like
password, must be kept secret and secure. The public key is copied to systems the user wants t
connect to, and is used to verify the private key. The public key does not need to be secret
You put a copy of the public key in your account on the server. When you try to log in, the SS
server can use the public key to issue a challenge that can only be correctly answered by using th
private key. As a result, your ssh client can automatically authenticate your login to the serve
with your unique copy of the private key. This allows you to securely access systems in a way tha
doesn't require you to enter a password interactively every time

Generating SSH Key

To create a private key and matching public key for authentication, use the ssh-keyge
command. By default, your private and public keys are saved in your ~/.ssh/id_rsa an
~/.ssh/id_rsa.pub les, respectively

[user@host ~]$ ssh-keyge

Generating public/private rsa key pair
Enter le in which to save the key (/home/user/.ssh/id_rsa): Ente
Created directory '/home/user/.ssh'
Enter passphrase (empty for no passphrase): Ente
Enter same passphrase again: Ente
Your identi cation has been saved in /home/user/.ssh/id_rsa
Your public key has been saved in /home/user/.ssh/id_rsa.pub
The key ngerprint is
SHA256:vxutUNPio3QDCyvkYm1oIx35hmMrHpPKWFdIYu3HV+w [email protected]
The key's randomart image is
+---[RSA 2048]----
|
|..
|ooo
|.=oo.
|o+=SE.
| ..O o + * +
|.+% O . + B .
|=*oO . . + *
|++. . +.
+——[SHA256]-----

If you do not specify a passphrase when ssh-keygen prompts you, the generated private ke
is not protected. In this case, anyone with your private key le could use it for authentication. I
you set a passphrase, then you will need to enter that passphrase when you use the private ke
for authentication. (Therefore, you would be using the private key's passphrase rather than you
password on the remote host to authenticate.
You can run a helper program called ssh-agent which can temporarily cache your private ke
passphrase in memory at the start of your session to get true passwordless authentication. Thi
will be discussed later in this section
The following example of the ssh-keygen command shows the creation of the passphraseprotecte
private key alongside the public key
fi
|

fi
|

fi
|

fi
|

fi
+

fi
fi
S

fi
.

fi
r

fi
fi
.

fi
N

 [user@host ~]$ ssh-keygen -f .ssh/key-with-pas

Generating public/private rsa key pair
Enter passphrase (empty for no passphrase)
Enter same passphrase again
Your identi cation has been saved in .ssh/key-with-pass
Your public key has been saved in .ssh/key-with-pass.pub
The key ngerprint is
SHA256:w3GGB7EyHUry4aOcNPKmhNKS7dl1YsMVLvFZJ77VxAo [email protected]
The key's randomart image is
+---[RSA 2048]----
| . + =.o ...
| = B XEo o.
| . o O X =....
| = = = B = o.
|= + * * S .
|.+ = o + .
|+.
|
|
+----[SHA256]-----

The -f option with the ssh-keygen command determines the les where the keys are saved. I
the preceding example, the private and public keys are saved in the /home/user/.ssh/keywith
pass /home/user/.ssh/key-with-pass.pub les, respectively

***************************
WARNIN
During further SSH keypair generation, unless you specify a unique le name, yo
are prompted for permission to overwrite the existing id_rsa and id_rsa.pu
les. If you overwrite the existing id_rsa and id_rsa.pub les, then you mus
replace the old public key with the new one on all the SSH servers that have your ol
public key
Once the SSH keys have been generated, they are stored by default in the .ssh/ directory o
the user's home directory. The permission modes must be 600 on the private key and 644 on th
public key
***********************************

Sharing the Public Ke

Before key-based authentication can be used, the public key needs to be copied to the destinatio
system. The ssh-copy-id command copies the public key of the SSH keypair to the destinatio
system. If you omit the path to the public key le while running ssh-copy-id, it uses the defaul
/home/user/.ssh/id_rsa.pub le

user@host ~]$ ssh-copy-id -i .ssh/key-with-pass.pub user@remotehos

/usr/bin/ssh-copy-id: INFO: Source of key(s) to be installed: "/home/user/.ssh
id_rsa.pub
/usr/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to lte
out any that are already installe
/usr/bin/ssh-copy-id: INFO: 1 key(s) remain to be installed -- if you are prompte
now it is to install the new key
user@remotehost's password: redha
Number of key(s) added:
Now try logging into the machine, with: "ssh 'user@remotehost'
and check to make sure that only the key(s) you wanted were added

After the public key is successfully transferred to a remote system, you can authenticate to th
remote system using the corresponding private key while logging in to the remote system ove
SSH. If you omit the path to the private key le while running the ssh command, it uses the defaul
/home/user/.ssh/id_rsa le

[user@host ~]$ ssh -i .ssh/key-with-pass user@remotehos

Enter passphrase for key '.ssh/key-with-pass': redhatpas
...output omitted..
[user@remotehost ~]$ exi
logou
Connection to remotehost closed
[user@host ~]
fi
[

fi
|

"

fi
|

fi
:

fi
.

fi
fi
fi
.

"

fi
.

fi
t

fi
r

fi
b

 Using ssh-agent for Non-interactive Authenticatio

If your SSH private key is protected with a passphrase, you normally have to enter the passphras
to use the private key for authentication. However, you can use a program called ssh-agent t
temporarily cache the passphrase in memory. Then any time that you use SSH to log in to anothe
system with the private key, ssh-agent will automatically provide the passphrase for you. Thi
is convenient, and can improve security by providing fewer opportunities for someone "shoulde
sur ng" to see you type the passphrase in
Depending on your local system's con guration, if you initially log in to the GNOME graphica
desktop environment, the ssh-agent program might automatically be started and con gured fo
you
If you log in on a text console, log in using ssh, or use sudo or su, you will probably need to star
ssh-agent manually for that session. You can do this with the following command

[user@host ~]$ eval $(ssh-agent

Agent pid 1015
[user@host ~]

*****************************
NOT
When you run ssh-agent, it prints out some shell commands. You need to ru
these commands to set environment variables used by programs like ssh-add t
communicate with it. The eval $(ssh-agent) command starts ssh-agent an
runs those commands to automatically set those environment variables for that shel
session. It also displays the PID of the ssh-agent process
*************************

Once ssh-agent is running, you need to tell it the passphrase for your private key or keys. Yo
can do this with the ssh-add command
The following ssh-add commands add the private keys from /home/user/.ssh/id_rsa (th
default) and /home/user/.ssh/key-with-pass les, respectively

[user@host ~]$ ssh-ad

Identity added: /home/user/.ssh/id_rsa ([email protected]
[user@host ~]$ ssh-add .ssh/key-with-pas
Enter passphrase for .ssh/key-with-pass: redhatpas
Identity added: .ssh/key-with-pass ([email protected]

After successfully adding the private keys to the ssh-agent process, you can invoke an SS
connection using the ssh command. If you are using any private key le other than the defaul
/home/user/.ssh/id_rsa le, then you must use the -i option with the ssh command t
specify the path to the private key le
The following example of the ssh command uses the default private key le to authenticate to a
SSH server

[user@host ~]$ ssh user@remotehos

Last login: Fri Apr 5 10:53:50 2019 from host.example.co
[user@remotehost ~]

The following example of the ssh command uses the /home/user/.ssh/key-with-pass (nondefault
private key le to authenticate to an SSH server. The private key in the following exampl
has already been decrypted and added to its parent ssh-agent process, so the ssh comman
does not prompt you to decrypt the private key by interactively entering its passphrase

[user@host ~]$ ssh -i .ssh/key-with-pass user@remotehos

Last login: Mon Apr 8 09:44:20 2019 from host.example.co
[user@remotehost ~]

When you log out of the session that started ssh-agent, the process will exit and your th
passphrases for your private keys will be cleared from memory
fi
.

fi
$

fi
)

fi
s

fi
.

fi
m

fi
n

fi
d

fi
.

 CUSTOMIZING OPENSSH SERVICE CONFIGURATIO

OBJECTIVE
After completing this section, you should be able to restrict direct logins as root and disabl
password-based authentication for the OpenSSH service

CONFIGURING THE OPENSSH SERVE

OpenSSH service is provided by a daemon called sshd. Its main con guration le is /etc/ssh
sshd_con g
The default con guration of the OpenSSH server works well. However, you might want to mak
some changes to strengthen the security of your system. There are two common changes yo
might want to make. You might want to prohibit direct remote login to the root account, and yo
might want to prohibit password-based authentication (in favor of SSH private key authentication)

PROHIBIT THE SUPERUSER FROM LOGGING IN USIN

SS
It is a good practice to prohibit direct login to the root user account from remote systems. Som
of the risks of allowing direct login as root include
• The user name root exists on every Linux system by default, so a potential attacker only ha
to guess the password, instead of a valid user name and password combination. This reduce
complexity for an attacker
• The root user has unrestricted privileges, so its compromise can lead to maximum damage t
the system
• From an auditing perspective, it can be hard to track which authorized user logged in as roo
and made changes. If users have to log in as a regular user and switch to the root account, thi
generates a log event that can be used to help provide accountability
The OpenSSH server uses the PermitRootLogin con guration setting in the /etc/ssh
sshd_con g con guration le to allow or prohibit users logging in to the system as root

PermitRootLogin ye

With the PermitRootLogin parameter to yes, as it is by default, people are permitted to log in a
root. To prevent this, set the value to no. Alternatively, to prevent password-based authenticatio
but allow private key-based authentication for root, set the PermitRootLogin parameter t
without-password
The SSH server (sshd) must be reloaded for any changes to take effect

[root@host ~]# systemctl reload ssh

****************************
IMPORTAN
The advantage of using systemctl reload sshd command is that it tells ssh
to re-read its con guration le rather than completely restarting the service.
systemctl restart sshd command would also apply the changes, but woul
also stop and start the service, breaking all active SSH connections to that host
************************

PROHIBITING PASSWORD-BASED AUTHENTICATIO

FOR SS
Allowing only private key-based logins to the remote command line has various advantages
• Attackers cannot use password guessing attacks to remotely break into known accounts on th
system
• With passphrase-protected private keys, an attacker needs both the passphrase and a copy o
the private key. With passwords, an attacker just needs the password
• By using passphrase-protected private keys in conjunction with ssh-agent, the passphrase i
exposed less frequently since it is entered less frequently, and logging in is more convenient fo
the user
The OpenSSH server uses the PasswordAuthentication parameter in the /etc/ssh
H

fi
fi
.

fi
fi
s

fi
.

fi
fi
d

fi
.

fi
.

fi
.

 sshd_con g con guration le to control whether users can use password-based authenticatio
to log in to the system

PasswordAuthentication ye

The default value of yes for the PasswordAuthentication parameter in the /etc/ssh
sshd_con g con guration le causes the SSH server to allow users to use password-base
authentication while logging in. The value of no for PasswordAuthentication prevents user
from using password-based authentication
Keep in mind that whenever you change the /etc/ssh/sshd_con g le, you must reload th
sshd service for changes to take effect

**************************
IMPORTAN
Remember, if you turn off password-based authentication for ssh, you need to hav
a way to ensure that the user's ~/.ssh/authorized_keys le on the remot
server is populated with their public key, so that they can log in
*********************************
fi
fi
T

fi
fi
.

fi
fi
.

fi
.

fi
fi
e

 ANALYZING AND STORING LOG

GOAL
Locate and accurately interpret logs of system events for troubleshooting purposes

OBJECTIVES
• Describe the basic logging architecture used by Red Hat Enterprise Linux to record events
• Interpret events in relevant syslog les to troubleshoot problems or review system status
• Find and interpret entries in the system journal to troubleshoot problems or review system status
• Con gure the system journal to preserve the record of events when a server is rebooted
• Maintain accurate time synchronization using NTP and con gure the time zone to ensure correct time
stamps for events recorded by the system journal and logs

SECTIONS
• Describing System Log Architecture (and Quiz
• Reviewing Syslog Files (and Guided Exercise
• Reviewing System Journal Entries (and Guided Exercise
• Preserving the System Journal (and Guided Exercise
• Maintaining Accurate Time (and Guided Exercise

LAB
Analyzing and Storing Log

fi

fi
)

fi
.

 DESCRIBING SYSTEM LOG ARCHITECTUR

OBJECTIVE
After completing this section, you should be able to describe the basic logging architecture use
by Red Hat Enterprise Linux to record events

SYSTEM LOGGIN
Processes and the operating system kernel record a log of events that happen. These logs are use
to audit the system and troubleshoot problems
Many systems record logs of events in text les which are kept in the /var/log directory. Thes
logs can be inspected using normal text utilities such as less and tail
A standard logging system based on the Syslog protocol is built into Red Hat Enterprise Linux
Many programs use this system to record events and organize them into log les. The systemdjournal
and rsyslog services handle the syslog messages in Red Hat Enterprise Linux 8
The systemd-journald service is at the heart of the operating system event loggin
architecture. It collects event messages from many sources including the kernel, output from th
early stages of the boot process, standard output and standard error from daemons as they star
up and run, and syslog events. It then restructures them into a standard format, and writes the
into a structured, indexed system journal. By default, this journal is stored on a le system tha
does not persist across reboots
However, the rsyslog service reads syslog messages received by systemd-journald fro
the journal as they arrive. It then processes the syslog events, recording them to its log les o
forwarding them to other services according to its own con guration
The rsyslog service sorts and writes syslog messages to the log les that do persist acros
reboots in /var/log. The rsyslog service sorts the log messages to speci c log les based o
the type of program that sent each message, or facility, and the priority of each syslog message
In addition to syslog message les, the /var/log directory contains log les from other service
on the system. The following table lists some useful les in the /var/log directory

NOT
Some applications do not use syslog to manage their log messages, althoug
typically, they do place their log les in a subdirectory of /var/log. For example, th
Apache Web Server saves log messages to les in a subddirectory of the /var/lo
directory
E

fi
.

fi
fi
fi
.

fi
fi
fi
.

fi
fi
h

fi
fi
E

fi
.

fi
s

 REVIEWING SYSLOG FILE

OBJECTIVE
After completing this section, you should be able to interpret events in relevant syslog les t
troubleshoot problems or review system status

LOGGING EVENTS TO THE SYSTE

Many programs use the syslog protocol to log events to the system. Each log message i
categorized by a facility (the type of message) and a priority (the severity of the message)
Available facilities are documented in the rsyslog.conf(5) man page
The following table lists the standard eight syslog priorities from highest to lowest

Overview of Syslog Priorities

The rsyslog service uses the facility and priority of log messages to determine how to handl
them. This is con gured by rules in the /etc/rsyslog.conf le and any le in the /etc
rsyslog.d directory that has a le name extension of .conf. Software packages can easily ad
rules by installing an appropriate le in the /etc/rsyslog.d directory
Each rule that controls how to sort syslog messages is a line in one of the con guration les. Th
left side of each line indicates the facility and severity of the syslog messages the rule matches
The right side of each line indicates what le to save the log message in (or where else to delive
the message). An asterisk (*) is a wildcard that matches all values
For example, the following line would record messages sent to the authpriv facility at any priorit
to the le /var/log/secure

authpriv.* /var/log/secur

Log messages sometimes match more than one rule in rsyslog.conf. In such cases, on
message is stored in more than one log le. To limit messages stored, the key word none in th
priority eld indicates that no messages for the indicated facility should be stored in the given le
Instead of logging syslog messages to a le, they can also be printed to the terminals of all loggedi
users. The rsyslog.conf le has a setting to print all the syslog messages with the emer
priority to the terminals of all logged-in users
fi
fi
fi
e

fi
:

fi
fi
fi
fi
fi
.

fi
M

fi
fi
.

fi
e

fi
s

fi
e

 SAMPLE RULES OF RSYSLOG

LOG FILE ROTATIO

The logrotate tool rotates log les to keep them from taking up too much space in the l
system containing the /var/log directory. When a log le is rotated, it is renamed with a
extension indicating the date it was rotated. For example, the old /var/log/messages le ma
become /var/log/messages-20190130 if it is rotated on 2019-01-30. Once the old log le i
rotated, a new log le is created and the service that writes to it is noti ed
After a certain number of rotations, typically after four weeks, the oldest log le is discarded t
free disk space. A scheduled job runs the logrotate program daily to see if any logs need to b
rotated. Most log les are rotated weekly, but logrotate rotates some faster, or slower, or whe
they reach a certain size
Con guration of logrotate is not covered in this course. For more information, see th
logrotate(8) man page

ANALYZING A SYSLOG ENTR

Log messages start with the oldest message on top and the newest message at the end of the lo
le. The rsyslog service uses a standard format while recording entries in log les. The followin
example explains the anatomy of a log message in the /var/log/secure log le
fi
fi
fi
fi
.

fi
N

fi
Y

fi
.

fi
fi
.

fi
e

fi
fi
fi
n

 MONITORING LOG
Monitoring one or more log les for events is helpful to reproduce problems and issues. The tai
-f /path/to/ le command outputs the last 10 lines of the le speci ed and continues t
output new lines in the le as they get written
For example, to monitor for failed login attempts, run the tail command in one terminal an
then in another terminal, run the ssh command as the root user while a user tries to log in to th
system
In the rst terminal, run the following tail command
[root@host ~]# tail -f /var/log/secur
In the second terminal, run the following ssh command

[root@host ~]# ssh root@localhos

root@localhost's password: redha
...output omitted..
[root@host ~]

Return to the rst terminal and view the logs

...output omitted..
Feb 10 09:01:13 host sshd[2712]: Accepted password for root from 172.25.254.25
port 56801 ssh
Feb 10 09:01:13 host sshd[2712]: pam_unix(sshd:session): session opened for use
root by (uid=0

SENDING SYSLOG MESSAGES MANUALL

The logger command can send messages to the rsyslog service. By default, it sends th
message to the user facility with the notice priority (user.notice) unless speci ed otherwis
with the -p option. It is useful to test any change to the rsyslog service con guration
To send a message to the rsyslog service that gets recorded in the /var/log/boot.log lo
le, execute the following logger command

root@host ~]# logger -p local7.notice "Log entry created on host

fi
[

fi
.

fi
#

fi
.

fi
fi
t

"

fi
fi
4

fi
Y

fi
.

 REVIEWING SYSTEM JOURNAL ENTRIE

OBJECTIVE
After completing this section, you should be able to nd and interpret entries in the system journa
to troubleshoot problems or review system status
FINDING EVENT
The systemd-journald service stores logging data in a structured, indexed binary le called th
journal. This data includes extra information about the log event. For example, for syslog event
this includes the facility and the priority of the original message

*****************
IMPORTAN
In Red Hat Enterprise Linux 8, the /run/log directory stores the system journal b
default. The contents of the /run/log directory get cleared after a reboot. You ca
change this setting, and how to do so is discussed later in this chapter
******************

To retrieve log messages from the journal, use the journalctl command. You can use thi
command to view all messages in the journal, or to search for speci c events based on a wid
range of options and criteria. If you run the command as root, you have full access to the journal
Regular users can also use this command, but might be restricted from seeing certain messages.
T

fi
.

fi
.

fi
s

 The journalctl command highlights important log messages: messages at notice or warnin
priority are in bold text while messages at the error priority or higher are in red text
The key to successfully using the journal for troubleshooting and auditing is to limit journa
searches to show only relevant output
By default, journalctl -n shows the last 10 log entries. You can adjust this with an optiona
argument that speci es how many log entries to display. For the last ve log entries, run th
following journalctl command

Similar to the tail -f command, the journalctl -f command outputs the last 10 lines of th
system journal and continues to output new journal entries as they get written to the journal. T
exit the journalctl -f process, use the Ctrl+C key combination

To help troubleshoot problems, you might want to lter the output of the journal based on th
priority of the journal entries. The journalctl -p takes either the name or the number of
priority level and shows the journal entries for entries at that priority and above. The journalct
command understands the debug, info, notice, warning, err, crit, alert, and emer
priority levels
Run the following journalctl command to list journal entries at the err priority or higher
.

fi
:

fi
.

fi
g

 When looking for speci c events, you can limit the output to a speci c time frame. Th
journalctl command has two options to limit the output to a speci c time range, the --sinc
and --until options. Both options take a time argument in the format "YYYY-MM-DD hh:mm:ss
(the double-quotes are required to preserve the space in the option). If the date is omitted
the command assumes the current day, and if the time is omitted, the command assumes th
whole day starting at 00:00:00. Both options take yesterday, today, and tomorrow as vali
arguments in addition to the date and time eld
Run the following journalctl command to list all journal entries from today's records

Run the following journalctl command to list all journal entries ranging from 2019-02-1
20:30:00 to 2019-02-13 12:00:00

[root@host ~]# journalctl --since "2014-02-10 20:30:00" --until "2014-02-1

12:00:00
...output omitted
"

fi
.

fi
.

fi
fi
.

"

 NOT
You can use other, more sophisticated time speci cations with the --since and -
until options. For some examples, see the systemd.time(7) man page
*************
In addition to the visible content of the journal, there are elds attached to the log entries that ca
only be seen when verbose output is turned on. Any displayed extra eld can be used to lter th
output of a journal query. This is useful to reduce the output of complex searches for certain event
in the journal

The following list gives the common elds of the system journal that can be used to search for line
relevant to a particular process or event
• _COMM The name of the comman
• _EXE The path to the executable for the proces
• _PID The PID of the proces
• _UID The UID of the user running the proces
• _SYSTEMD_UNIT The systemd unit that started the proces
E

fi
.

fi
fi
s

fi
.

fi
e

More than one of the system journal elds can be combined to form a granular search query wit
the journalctl command. For example, the following journalctl command shows all journa
entries related to the sshd.service systemd unit from a process with PID 1182

[root@host ~]# journalctl _SYSTEMD_UNIT=sshd.service _PID=118

Apr 03 19:34:27 host.lab.example.com sshd[1182]: Accepted password for roo
from ::1 port 52778 ssh
Apr 03 19:34:28 host.lab.example.com sshd[1182]: pam_unix(sshd:session): sessio
opened for user root by (uid=0
...output omitted
…

fi
2

 PRESERVING THE SYSTEM JOURNA

OBJECTIVE
After completing this section, you should be able to con gure the system journal to preserve th
record of events when a server is rebooted

STORING THE SYSTEM JOURNAL PERMANENTL

By default, the system journals are kept in the /run/log/journal directory, which means th
journals are cleared when the system reboots. You can change the con guration settings of th
systemd-journald service in the /etc/systemd/journald.conf le to make the journal
persist across reboot
The Storage parameter in the /etc/systemd/journald.conf le de nes whether t
store system journals in a volatile manner or persistently across reboot. Set this parameter t
persistent, volatile, or auto as follows
• persistent: stores journals in the /var/log/journal directory which persists acros
reboots
If the /var/log/journal directory does not exist, the systemd-journald service creates it
• volatile: stores journals in the volatile /run/log/journal directory
As the /run le system is temporary and exists only in the runtime memory, data stored in it
including system journals, do not persist across reboot
• auto: rsyslog determines whether to use persistent or volatile storage. If the /var/log
journal directory exists, then rsyslog uses persistent storage, otherwise it uses volatil
storage
This is the default action if the Storage parameter is not set
The advantage of persistent system journals is that the historic data is available immediatel
at boot. However, even with a persistent journal, not all data is kept forever. The journal has
built-in log rotation mechanism that triggers monthly. In addition, by default, the journals are no
allowed to get larger than 10% of the le system it is on, or leave less than 15% of the le syste
free. These values can be tuned for both the runtime and persistent journals in /etc/systemd
journald.conf. The current limits on the size of the journal are logged when the systemdjournal
process starts. The following command output shows the journal entries that re ect th
current size limits

Con guring Persistent System Journal

To con gure the systemd-journald service to preserve system journals persistently acros
reboot, set Storage to persistent in the /etc/systemd/journald.conf le. Run the tex
editor of your choice as the superuser to edit the /etc/systemd/journald.conf le.
fi
fi
.

fi
:

fi
.

fi
s

fi
fi
.

fi
fi
L

fi
fi
o

fl
s

fi
e

 After editing the con guration le, restart the systemd-journald service to bring th
con guration changes into effect

[root@host ~]# systemctl restart systemd-journal

If the systemd-journald service successfully restarts, you can see that the /var/log
journal directory is created and contains one or more subdirectories. These subdirectories hav
hexadecimal characters in their long names and contain *.journal les. The *.journal le
are the binary les that store the structured and indexed journal entries

[root@host ~]# ls /var/log/journa

73ab164e278e48be9bf80e80714a8cd
[root@host ~]# ls /var/log/journal/73ab164e278e48be9bf80e80714a8cd
system.journal user-1000.journa

While the system journals persist across reboot, you get an extensive number of entries in th
output of the journalctl command that includes entries from the current system boot as wel
as the previous ones. To limit the output to a speci c system boot, use the -b option with th
journalctl command. The following journalctl command retrieves the entries limited to th
rst system boot

[root@host ~]# journalctl -b

...output omitted..

The following journalctl command retrieves the entries limited to the second system boot. Th
following argument is meaningful only if the system has been rebooted for more than twice

[root@host ~]# journalctl -b

The following journalctl command retrieves the entries limited to the current system boot

[root@host ~]# journalctl -

NOT
When debugging a system crash with a persistent journal, it is usually required t
limit the journal query to the reboot before the crash happened. The -b option ca
be accompanied by a negative number indicating how many prior system boots th
output should include. For example, journalctl -b -1 limits the output to onl
the previous boot
fi
fi
E

fi
.

fi
b

fi
5

fi
5

fi
.

fi
e

 MAINTAINING ACCURATE TIM

OBJECTIVE
After completing this section, you should be able to maintain accurate time synchronization usin
NTP and con gure the time zone to ensure correct time stamps for events recorded by the syste
journal and logs

SETTING LOCAL CLOCKS AND TIME ZONE

Correct synchronized system time is critical for log le analysis across multiple systems. Th
Network Time Protocol (NTP) is a standard way for machines to provide and obtain correct tim
information on the Internet. A machine may get accurate time information from public NT
services on the Internet, such as the NTP Pool Project. A high-quality hardware clock to serv
accurate time to local clients is another option
The timedatectl command shows an overview of the current time-related system settings
including current time, time zone, and NTP synchronization settings of the system.

A database of time zones is available and can be listed with the timedatectl list-timezones command

Time zone names are based on the public time zone database that IANA maintains. Time zones ar
named based on continent or ocean, then typically but not always the largest city within the tim
zone region. For example, most of the US Mountain time zone is America/Denver
Selecting the correct name can be non-intuitive in cases where localities inside the time zon
have different daylight saving time rules. For example, in the USA, much of the state of Arizon
(US Mountain time) does not have a daylight saving time adjustment at all and is in the time zon
America/Phoenix
37
The command tzselect is useful for identifying correct zoneinfo time zone names. It interactivel
prompts the user with questions about the system's location, and outputs the name of the correc
time zone. It does not make any change to the time zone setting of the system
The superuser can change the system setting to update the current time zone using th
fi
.

fi
E

 timedatectl set-timezone command. The following timedatectl command updates th

current time zone to America/Phoenix

NOT
Should you need to use the Coordinated Universal Time (UTC) on a particular server
set its time zone to UTC. The tzselect command does not include the name of th
UTC time zone. Use the timedatectl set-timezone UTC command to set th
system's current time zone to UTC
***********

Use the timedatectl set-time command to change the system's current time. The time i
speci ed in the "YYYY-MM-DD hh:mm:ss" format, where either date or time can be omitted. Th
following timedatectl command changes the time to 09:00:00

The timedatectl set-ntp command enables or disables NTP synchronization for automati
time adjustment. The option requires either a true or false argument to turn it on or off. Th
following timedatectl command turns on NTP synchronization

[root@host ~]# timedatectl set-ntp tru

**************
NOT
In Red Hat Enterprise Linux 8, the timedatectl set-ntp command will adjus
whether or not chronyd NTP service is operating. Other Linux distributions migh
use this setting to adjust a different NTP or SNTP service
Enabling or disabling NTP using other utilities in Red Hat Enterprise Linux, such as i
the graphical GNOME Settings application, also updates this setting
**************
E

fi
*

 CONFIGURING AND MONITORING CHRONY

The chronyd service keeps the usually-inaccurate local hardware clock (RTC) on track b
synchronizing it to the con gured NTP servers. If no network connectivity is available, chrony
calculates the RTC clock drift, which is recorded in the drift le speci ed in the /etc
chrony.conf con guration le
By default, the chronyd service uses servers from the NTP Pool Project for the tim
synchronization and does not need additional con guration. It may be useful to change the NT
servers when the machine in question is on an isolated network
The stratum of the NTP time source determines its quality. The stratum determines the numbe
of hops the machine is away from a high-performance reference clock. The reference clock is
stratum 0 time source. An NTP server directly attached to it is a stratum 1, while a machin
synchronizing time from the NTP server is a stratum 2 time source
The server and peer are the two categories of time sources that you can in the /etc
chrony.conf con guration le. The server is one stratum above the local NTP server, and th
peer is at the same stratum level. More than one server and more than one peer can be speci ed
one per line
The rst argument of the server line is the IP address or DNS name of the NTP server. Followin
the server IP address or name, a series of options for the server can be listed. It is recommended t
use the iburst option, because after the service starts, four measurements are taken in a shor
time period for a more accurate initial clock synchronization
The following server classroom.example.com iburst line in the /etc/chrony.conf l
causes the chronyd service to use the classroom.example.com NTP time source

# Use public servers from the pool.ntp.org project

...output omitted..
server classroom.example.com iburs
...output omitted..

After pointing chronyd to the local time source, classroom.example.com, you should restar
the service

[root@host ~]# systemctl restart chrony

The chronyc command acts as a client to the chronyd service. After setting up NT
synchronization, you should verify that the local system is seamlessly using the NTP server t
synchronize the system clock using the chrony sources command. For more verbose outpu
with additional explanations about the output, use the chronyc sources -v command.

The * character in the S (Source state) eld indicates that the classroom.example.com serve
has been used as a time source and is the NTP server the machine is currently synchronized to
fi
.

fi
fi
fi
fi
fi
.

fi
.

fi
fi
.

fi
fi
D

fi
a

 EMAIL SERVER

GOAL
To relay all email sent by the system through an SMTP gateway

OBJECTIVES
• Con gure a Red Hat Enterprise Linux server to transmit all email through an unauthenticated SMTP
gateway

SECTIONS
• Con guring a Send-only Email Con guration (and Practice

LAB
• Con guring Email Transmission

fi
fi
fi

fi
)

 CONFIGURING SEND-ONLY EMAIL SERVIC

OBJECTIVE
After completing this section, students should be able to
Con gure a Red Hat Enterprise Linux server to transmit all email through an unauthenticate
SMTP gateway

EMAIL ARCHITECTURE AND NULL CLIENT

In today's corporate environments, email is a common method of communication. End users ma
use dedicated mail clients such as Evolution and mutt to read and send email, or the organizatio
may have a web-based interface for its mail service
However, Linux servers also send email, usually for automatic purposes or to report errors to a
administrator. They generally use, directly or indirectly, a standard program called /usr/sbin
sendmail (provided in RHEL 7 by Post x) to send these messages
In practice, most servers are monitored and send out mails when incidents occur. This ofte
requires a con gured /usr/sbin/sendmail to send emails to notify the responsible syste
administrators by using the corporate SMTP server to transmit the messages
A null client is a client machine that runs a local mail server which forwards all emails to a
outbound mail relay for delivery. A null client does not accept local delivery for any messages, i
can only send them to the outbound mail relay. Users may run mail clients on the null client to rea
and send emails
This section will look at how to con gure a RHEL 7 server as a Post x null client, which will us
sendmail and the SMTP protocol to transmit mail messages to the outside world through a
existing outgoing mail server

TRANSMISSION OF AN EMAIL MESSAG

To send an email, in most cases the mail client communicates with an outgoing mail server, whic
will help relay that message to its nal destination. The mail client transmits messages to the mai
server using the Simple Mail Transfer Protocol (SMTP)
The outgoing mail relay may require no authentication from internal clients, in which case th
server listens on port 25/TCP. In that case, the relay will restrict which hosts can relay through I
address based restrictions or rewall rules

**************************
NOT
In cases where the outbound SMTP relay is reachable from the Internet, it i
normally con gured as a mail submission agent (MSA) for security and anti-spa
reasons. An MSA listens on port 587/TCP and requires authentication of the user'
mail client before accepting mail. This may be by username and password (as som
webmail services provide) or through other means
This course will only cover how to con gure a basic unauthenticated null client
****************************

The outgoing mail relay then uses DNS to look up the MX record identifying the mail server tha
accepts delivery for messages sent to the recipient's domain. The relay then uses SMTP on por
25/TCP to transmit the email to that server
The recipient's mail service may provide a POP3 or IMAP server, such as Dovecot or Cyrus, to allo
a dedicated mail client to download their messages. Frequently, the mail service provides a webbase
interface, allowing clients to use a web browser as a mail client
The following image illustrates how an email client retrieves incoming mail from an IMAP serve
and sends outgoing mail through an SMTP server. The mail client on serverX.example.co
fetches incoming mails from the IMAP server imapX.example.com. Outgoing mails are sent t
smtpX.example.com. An MX DNS record de nes smtpX.example.com as the responsible mai
server for the desktopX.example.com domain
fi
E

fi
fi
.

fi
fi
fi
fi
fi
.

fi
.

fi
E

 POSTFI
Post x is a powerful but easy-to-con gure mail server. It is the default mail server in Red Ha
Enterprise Linux 7. Post x is provided by the post x RPM package. It is a modular program mad
up of several cooperating programs. Its components are controlled by the master process
The main con guration le of the post x mail server is /etc/post x/main.cf

*****************
NOT
There are other con guration les present in the /etc/post x directory. One o
the important les is /etc/post x/master.cf, which controls what subservice
are started
******************
fi
E

fi
X

fi
*

fi
fi
fi
fi
fi
fi
fi
fi
fi
fi
.

 The con guration le /etc/post x/main.cf can be edited in two ways: by hand using a tex
editor such as vim, or it can be edited using the postconf utility. The postconf command allow
for querying by individual or all settings, modifying settings, querying defaults, or showing al
settings that differ from the built-in defaults
• Run the postconf command without any parameter to query all settings from the /etc
post x/main.cf con guration le
fi
fi
fi
fi
fi
fi
:

 [root@serverX ~]# postcon

2bounce_notice_recipient = postmaste
access_map_defer_code = 45
access_map_reject_code = 55
address_verify_cache_cleanup_interval = 12
address_verify_default_transport = $default_transpor
..
• Query a particular set of options by listing them after the postconf command, separated b
white space. Run the following to list the inet_interfaces and myorigin options with thei
corresponding values

[root@serverX ~]# postconf inet_interfaces myorigi

inet_interfaces = loopback-onl
myorigin = $myhostnam

*******************
NOT
If a value in /etc/post x/main.cf starts with a dollar sign ($), it is not a litera
value, but instead points to the value of a different setting. In the previous example
the myorigin setting will have the same value as the myhostname setting. Usin
this syntax can simplify maintenance, since the value only has to be updated in on
place
*********************

• Run the following to add new or change existing options in the /etc/post x/main.c
con guration le: postconf -e 'setting = value' If there was already a setting by tha
name in the con guration le, it will be updated to the new value; otherwise, it will be added t
the bottom of the con guration le
Run the following to change the myorigin setting to rewrite the domain part of the FROM: Emai
address to example.com

[root@serverX ~]# postconf -e 'myorigin = example.com

*********************
IMPORTAN
The post x service requires a reload or restart after the changes have bee
made to /etc/post x/main.cf

***********************

NOT
When troubleshooting email, a log of all mail-related operations is kept in th
systemd journal and /var/log/maillog, which includes information of any mai
server-related actions
The postqueue -p command displays a list of any outgoing mail messages tha
have been queued. To attempt to deliver all queued messages again immediately
run the postqueue -f command; otherwise, Post x will attempt to resend the
about once an hour until they are accepted or expire
************************

POSTFIX NULL CLIENT CONFIGURATIO

Remember, to act as a null client, Post x and the RHEL system must be con gured so that th
following things are true
• The sendmail command and programs that use it forward all emails to an existing outboun
mail relay for deliver
• The local Post x service does not accept local delivery for any email message
• Users may run mail clients on the null client to read and send emails
The following diagram illustrates how a null client setup works. For sending mails, the null client o
serverX delivers all messages to the corporate SMTP mail server smtpX.example.com
.

fi
E

fi
T

fi
fi
fi
fi
*

fi
*

fi
:

fi
.

fi
r

fi
n

'

fi
.

fi
l

fi
t

 NOT
A complete overview of all settings that are adjustable in /etc/post x/main.c
and their explanations can be found in the postconf(5) man page. To access thi
man page, use the man 5 postconf command. Omitting the 5 will display the ma
page for the postconf command rather than the con guration guide

CONFIGURE POSTFIX AS NULL CLIEN

In this example, please follow along with these steps while your instructor demonstrates how t
con gure Post x on serverX.example.com as a null client that uses the smtpX.example.co
mail server as a relay host. All mails passed on by the local null client have their sender addres
domain rewritten to desktopX.example.com
Adjust the con guration of the Post x mail server on your serverX system to act as a null clien
that forwards all messages to the corporate mail server on smtpX.example.com, which i
responsible for the desktopX.example.com domain

1. Adjust the relayhost directive to point to the corporate mail server. The host name of th
corporate mail server needs to be enclosed in square brackets to prevent an MX record looku
with the DNS server

[root@serverX ~]# postconf -e "relayhost=[smtpX.example.com]

2. Con gure the Post x mail server to only relay emails from the local system

1. Let the Post x mail server listen only on the loopback interface for emails to deliver

[root@serverX ~]# postconf -e "inet_interfaces=loopback-only

2. Change the con guration of the null client so that mails originating from th
127.0.0.0/8 IPv4 network and the [::1]/128 IPv6 network are forwarded to th
relay host by the local null client

[root@serverX ~]# postconf -e "mynetworks=127.0.0.0/8 [::1]/128

3. Con gure Post x so all outgoing mails have their sender domain rewritten to the compan
domain desktopX.example.com

[root@serverX ~]# postconf -e "myorigin=desktopX.example.com

4. Prohibit the Post x mail server from delivering any messages to local accounts
1. Con gure the null client to not act as an end point for any mail domain. Mail
where the recipient is a local email account are not accepted for local delivery. Th
mydestination option needs to be set to an empty value to achieve this
fi
E

fi
fi
fi
fi
fi
fi
fi
fi
fi
.

fi
.

fi
.

fi
"

"

"

"

fi
.

 [root@serverX ~]# postconf -e "mydestination=

2. Con gure the local null client to not sort any mails into mailboxes on the local system
Local email delivery is turned off

root@serverX ~]# postconf -e "local_transport=error: local delivery disabled

5. Restart the local post x null client

[root@serverX ~]# systemctl restart post

The following table shows a summary of the /etc/post x/main.cf con guration l
settings to con gure Post x on serverX.example.com as a null client that uses the

smtpX.example.com mail server as a relay host. All messages that are passed on by the loca
null client have their sender address domain rewritten to desktopX.example.com
[

fi
fi
fi
fi
.

fi
x

"

fi
"

fi
fi
e