System Logging

System logging is one of the objectives on the Red Hat Certified System Administrator (RHCSA) exam and I thought it would be fun to go over a few things that I learned while studying this section.

Most system logs are logged in text files in the /var/log directory. You can use less or tail to view the contents of the logs. In RHEL, the systemd-journald and rsyslog services handle the syslog messages.

systemd-journald

The systemd-journald service is the core of the operating system event logging. It collects event messages from many sources including the kernel, the boot process, and standard error from daemons as they start up. By default, this journal is stored on a file system that does not persist when the system reboots.

Journalctl is the utility for querying and displaying from from journald. Journald stores data in binary format instead of plain-text format. Using journalctl is the way you read the log messages processed by journald. journald is part of the systemd software bundle. Systemd was a project started in 2010 in an effort to surpass the efficiency of the init daemon by improving the framework for dependencies and allowing more concurrent processing during system booting, for example. Systemd is in charge of managing services – starting them and keeping them alive. In May 2011, Fedora became the first major Linux distribution to enable systemd by default. RHEL 7 was the first RHEL unit to support systemd.

rsyslog

The rsyslog service sorts and writes syslog messages to the log files that do not persist when the system reboots. Rsyslog can be used to “capture” systemd-journald logs and store them in log files that are persistent on reboot. It stores these log files in /var/log. The log messages are sorted to specific log files based on the type of program that sent each message (the facility), and the priority of each syslog message. Some common log files and the types of messages stored are:

• /var/log/messages Most syslog messages except for messages related to authentication and email processing, schedule job execution, and debugging.

• /var/log/secure Messages related to security and authentication.

• /var/log/maillog Messages related to the mail server.

• /var/log/cron Messages related to scheduled job execution.

• /var/log/boot.log Non-syslog console messages related to startup of the system.

Syslog Priorities

Code Priority Severity

0 emerg System is unusable

1 alert Action should be taken immediately to fix the issue

2 crit Critical issue

3 err Non-critical error issue

4 warning Warning

5 notice Normal but significant issue

6 info Informational event

7 debug Debugging-level message

The rsyslog service uses the facility and priority of log messages to decide how to handle them. The rules are configured in /etc/rsyslog.conf and any files added to the /etc/rsyslog.d directory that has a .conf extension.

Here are some sample rules from the rsyslog.conf file:

Viewing System Journal Entries

You can use the journalctl command to view all messages in the journal or search for specific messages based on a number of options:

Similar to tail-f, you can use the journal -f command to display the last 10 lines of the system journal and continue to view new journal entries as they are written to the journal. You can then use, Ctrl + C to exit out of the process when you are done:

You can run the journalctl command and just view logs based on priority. For example, journalctl -p crit will display all entries that are at a critical priority or higher:

When looking for specific events, you can limit the output that is displayed, to a specific time frame. For example, you can use the –since and –until option with date arguments or with the arguments yesterday, today, or tomorrow:

Or use the since option to view all entries in the last hour:

Or, you can get even more granular and search for just entries related to a service, such a sshd:

There are a lot of options when it comes to searching the journal entries. Be sure to check out the man page for journalctl and take a look at the example section to try out other specific field options to search on. Here is a snippet from the man page: