Introduction

In this article, we will explain some of the important aspects of logging in Linux. As the case with many aspects of technology, logging mechanisms are being transitioned, modified and improved. It is also noteworthy that systemd, the default system startup and service manager for RHEL 7/Centos 7 provides it’s own logging mechanism in the form of a binary log called journald. Both Syslog and rsyslog use a similar methodology for tracking where certain programs should store their logs and that is managed through facilities and priorities. Please note that we will be using the terms Syslog and rsyslog interchangeably in this article. We’ll be working with a Centos 6 system for all practical purposes which uses rsyslog as it’s logging system.

Understanding Syslog terminology: facilities, priorities, and actions
If every service running on the system writes every type of log it generates to a single log file then it would be tedious to parse that data. So, a sort of labeling system was devised wherein all programs that could generate log messages would send their corresponding logs messages to a system logger. The system logger would identify the received log message based on two attribute and these attributes are facilities and priorities. Given below is the list of available facilities and priorities that we may use with Syslog

facilities:

  • auth (security or authorization related messages)
  • user (user-level messages)
  • kern (kernel log messages)
  • cron (crond daemon log messages)
  • daemon (system daemon)
  • mail (mail system messages)
  • local1 (local use 1)
  • local2 (local use 2)

 

priorities:

  • emerg (panic messages indicating that the system is in an unusable condition)
  • alert (messages requiring immediate attention)
  • crit (critical messages like device errors)
  • err (error messages)
  • warn (warning messages)
  • notice (normal but significant messages)
  • info (informational messages)
  • debug (messages containing information useful when debugging programs)

An application developer will decide and pick a facility to log messages for their application under. There are no set conventions which ought to be followed in this regard and generally, the most appropriate facility is chosen depending on the purpose of the application. For example, ssh program generally logs it’s message under the auth facility since it relates to authorization. Individual messages being logged under a facility are then further categorized according to a priority. Messages under a facility belonging to a particular severity level and sorted and grouped under a priority. The emerg is the highest severity level and debug is the lowest severity level. We generally do not log messages under the debugging priority level unless a certain situation requires us to do so. Every log message that is sent to the Syslog server will be tagged with a facility, a priority and an action Once a log message has been tagged with a facility and priority, the Syslog server needs to determine what action needs to be taken for that message. The action could involve sending the log message to a file, print the log message on the terminal screen of a user or send the message to a remote server. The relationship of the actions to be taken according to the facility and priority tags is defined in the rsyslog configuration file.

Given below is the abridged version (comments and blank lines removed) of the /etc/rsyslog.conf file:

[root@linuxnix ~]# grep -Ev '^#|^$' /etc/rsyslog.conf
$ModLoad imuxsock # provides support for local system logging (e.g. via logger command)
$ModLoad imklog # provides kernel logging support (previously done by rklogd)
$ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat
$IncludeConfig /etc/rsyslog.d/*.conf
*.info;mail.none;authpriv.none;cron.none /var/log/messages
authpriv.* /var/log/secure
mail.* -/var/log/maillog
cron.* /var/log/cron
*.emerg *
uucp,news.crit /var/log/spooler
local7.* /var/log/boot.log
[root@linuxnix ~]#

The * here is a wildcard and implies inclusion of all available items in the given category. The – placed in front of some of the actions implies the use of memory caching. This means that the log messages are not written to the file one by one but instead are cached and then dumped to the log file. Interestingly if you tag something at the info level then rsyslog will send all messages that are more critical than the info level out to the appropriate destination defined by the action section for that message. Logs pertaining to the kernel are handled by a separate daemon called klogd which works in a similar manner as Syslog and rsyslog but is a separate binary.

 

Demonstration:

Although we don’t have an application available at the moment for which we could set the facility and priority tags and then set an action to route messages to. Nevertheless, we will show you a quick demo about modifying the rsyslog.conf file and redirect authorization logs to a file different than the default file specified in the actions section. Given below is the updated action for authorization facility-related messages for all priorities. I’m redirecting the log messages to /home/sahil/auth.log file.

[root@linuxnix ~]# grep auth.log /etc/rsyslog.conf
authpriv.* /home/sahil/auth.log
[root@linuxnix ~]#

To allow the modification to take effect we’ll need to restart the rsyslog service as shown below.

[root@linuxnix ~]# service rsyslog restart
Shutting down system logger: [ OK ]
Starting system logger: [ OK ]

After restarting the service, I initiated a couple of ssh connections to this system and then checked for logs in the file /home/sahil/auth.log. The file contained the following log messages:

[root@linuxnix ~]# cat /home/sahil/auth.log
Feb 20 21:56:59 linuxnix unix_chkpwd[5306]: password check failed for user (root)
Feb 20 21:56:59 linuxnix sshd[5302]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=linuxnix user=root
Feb 20 21:57:00 linuxnix sshd[5302]: Failed password for root from 192.168.188.134 port 51418 ssh2
Feb 20 21:57:03 linuxnix sshd[5302]: Accepted password for root from 192.168.188.134 port 51418 ssh2
Feb 20 21:57:03 linuxnix sshd[5302]: pam_unix(sshd:session): session opened for user root by (uid=0)
Feb 20 21:57:05 linuxnix sshd[5302]: Received disconnect from 192.168.188.134: 11: disconnected by user
Feb 20 21:57:05 linuxnix sshd[5302]: pam_unix(sshd:session): session closed for user root

Connection initiation, failure, and connection closure messages were logged to the file.

Conclusion

In this article, we explained what the Syslog/rsyslog log management system is and how it works. We also provided a quick demonstration of how the rsyslog configuration could be modified to suit custom requirements. We hope that you’ve found this article to be useful and we look forward to your feedback.

Post Views: 7,048
The following two tabs change content below.

Sahil Suri

He started his career in IT in 2011 as a system administrator. He has since worked with HP-UX, Solaris and Linux operating systems along with exposure to high availability and virtualization solutions. He has a keen interest in shell, Python and Perl scripting and is learning the ropes on AWS cloud, DevOps tools, and methodologies. He enjoys sharing the knowledge he's gained over the years with the rest of the community.