Three different approaches can be used by services to write log information: Direct write, rsyslogd, journald.

Journald collects messages from the kernel, the entire boot procedure, and services generated by Systemd units. It can be queried using
the journalctl command.
Rsyslogd service writes the messages to different files in the /var/log directory.
Auditd service provides auditing, an in-depth trace of what specific services, processes, or users have been doing.
To get more information about what has been happening on a machine running RHEL, administrators have to take three approaches:
 Monitor the files in /var/log that are written by rsyslogd.
 Use the journalctl command to get more detailed information from the journal.
 Use the systemctl status <unit> command to get a short overview of the last significant events that have been logged by Systemd
units through journald. This command shows the status of services, as well as the last couple of lines that have been logged.
Ex.: [root@server1 ~]# systemctl status sshd -l

System Log Files Overview

/var/log/messages The most commonly used log file, it is the generic log file where most messages are written to.
/var/log/dmesg Contains kernel log messages.
/var/log/secure Contains authentication-related messages. Look here to see which authentication errors have occurred on a server.
/var/log/boot.log Look here for messages that are related to system startup.
/var/log/audit/audit.log Contains audit messages. SELinux writes to this file.
/var/log/maillog Look here for mail-related messages.
/var/log/samba Provides log files for the Samba service. Samba by default is not managed through rsyslog,
/var/log/sssd Contains messages that have been written by the sssd service, which plays an important role in the authentication process.
/var/log/cups Contains log messages that were generated by the print service CUPS.
/var/log/httpd/ Directory that contains log files that are written by the Apache web server. Apache writes messages directly.

The tail -f <logfile> command shows in real time which lines are added to the log file.
Type logger, followed by the message you want to write to the logs.
Using Live Log Monitoring and logger: From the root shell, type tail -f /var/log/messages
From other user shell, type logger hello. The message appears in the /var/log/messages file.
Press Ctrl-C to stop tracing the messages file.
Configure the rsyslogd service through the /etc/rsyslog.conf file. The /etc/rsyslog.conf file is the central location where rsyslogd is
configured. The rsyslog.conf file is used to specify what should be logged and where it should be logged.

#### MODULES ####: rsyslogd is modular. Modules are included to enhance the supported features in rsyslogd.
#### GLOBAL DIRECTIVES ####: To specify global parameters, location where auxiliary files are written or the default timestamp format.
#### RULES ####: It contains the rules that specify what information should be logged to which destination.

Rsyslogd Facilities
auth/authpriv (facility 4/facility 10) Messages related to authentication.
Cron (facility 9) Messages generated by the crond service.
daemon (facility 3) Generic facility that can be used for nonspecified daemons.
kern (facility 0) Kernel messages.
lpr Messages generated through the legacy lpd print system.
mail (facility 2) Email-related messages.
mark Special facility that can be used to write a marker periodically.
news Messages generated by the NNTP news system.
security Same as auth/authpriv. Should not be used anymore.
syslog Messages generated by the syslog system.
user (facility 1) Messages generated in user space.
uucp Messages generated by the legacy UUCP system.
local0-7 Messages generated by services that are configured by any of the local0 through local7 facilities.

The daemon facility is a generic facility that can be used by any daemon. In addition, the local0 through local7 facilities can be used.
If there are services that do not have their own rsyslogd facility that need to write log messages it can be configured to use any of the
local0 through local7 facilities. Configure the services to use these facilities as well specific to the service you are using. Then you need
to add a rule to the rsyslog.conf file to send messages that come in through that facility to a specific log file.

To determine which types of messages should be logged, different severities can be used in rsyslog.conf lines.
debug (7) Debug messages that will give as much information as possible about service operation.
info (6) Informational messages about normal service operation.
notice (5) Informational messages about items that might become an issue later.
warning /warn (4) Something is suboptimal, but there is no real error yet.
err /error (3) A noncritical error has occurred.
crit (2) A critical error has occurred.
alert (1) Used when the availability of the service is about to be discontinued.
emerg/panic (0) Message generated when the availability of the service is discontinued.
If you need to configure logging in a detailed way, where messages with different priorities are sent to different files, specify the priority
with an equals sign (=) in front of it, as in the following line, which will write all cron messages with only the debug priority to a specific file
with the name /var/log/cron.debug. The - in front of the line specifies to buffer writes so that information is logged in a more efficient way.
Ex.: cron.=debug -/var/log/cron.debug

The logrotate utility is started periodically through the crond service to take care of rotating log files kept in the file /etc/logrotate.

Type journalctl to view in less.

Type journalctl -b (or -k, -u, -g etc; for boot, kernel, system units or grep flags)
Type journalctl --no-pager. This shows the contents of the journal without using a pager.
Type journalctl -f. To view mode of journalctl, which allows you to see new messages scrolling by in real time. Press Ctrl-C to interrupt.
Type journalctl and press the Tab key twice. This shows specific options that can be used for filtering.
Type journalctl -n 20. The -n 20 option displays the last 20 lines of the journal (just like tail -n 20).
Type journalctl -p err. This command shows errors only.
Type journalctl --since yesterday to show all messages written since yesterday. Or use time parameter YYYY-MM-DD hh:mm:ss
Journalctl w/ different options, as well. To show all messages with a priority err written since yesterday journalctl -- since yesterday -p err.
Type journalctl -o verbose. This shows different options that are used when writing to the journal.
Type journalctl --dmesg. This shows kernel-related messages only.
Type journalctl -G to go to the end of the journal.

By default, the journal is stored in the file /run/log/journal, and all run files the journal is cleared when the system reboots. To
make the journal persistent between system restarts, you should make sure that a directory /var/log/journal exists.
Storing the journal permanently requires setting the Storage=auto parameter in /etc/systemd/journal.conf.

Making the journald Journal Permanent

1. Open a root shell and type mkdir /var/log/journal.
2. Set ownership. Type chown root:systemd-journal /var/log/journal, followed by chmod 2755 /var/log/journal.
3. Next, you can either reboot your system or use the killall -USR1 systemd-journald command.
4. The Systemd journal is now persistent across reboots. If you want to see the log messages since the last reboot, use journalctl -b

Common third-party agents

Splunk: monitoring and log aggregation.
ElasticStack: used to collect, correlate and visualize data
AlienVault: a Security Information and Event Management (SIEM) tool, used to detect and correlant secturity events.
Datagog: a monitoring platform for cloud appications, focusing on observability

Relevant directories and files

 /etc/rsyslog.d/50-default.conf
 /etc/rsyslog.conf
 /var/log/syslog
 /var/log/*.log
 /var/spool/rsyslog/
 /etc/logrotate
 /etc/systemd/journald.conf
 /var/log/journal
 /run/log/journal/

Relevant commands
 tail -f <logfile>
 logger
 journalctl
 systemctl status <unit>