System Log Monitor is a problem daemon in node problem detector. It monitors specified system daemon log and detects problems following predefined rules.

The System Log Monitor matches problems according to a set of predefined rule list in the configuration files. ( config/kernel-monitor.json as an example). The rule list is extensible.

• System Log Monitor currently supports file-based logs, journald, and kmsg. Additional sources can be added by implementing a new log watcher.

To support new node conditions, you can extend the conditions field in the configuration file with new condition definition:

{
  "type": "NodeConditionType",
  "reason": "CamelCaseDefaultNodeConditionReason",
  "message": "arbitrary default node condition message"
}

To detect new problems, you can extend the rules field in the configuration file with new rule definition:

{
  "type": "temporary/permanent",
  "condition": "NodeConditionOfPermanentIssue",
  "reason": "CamelCaseShortReason",
  "message": "regexp matching the issue in the log"
}

Note that the pattern must match to the end of the line excluding the tailing newline character, and multi-line pattern is supported.

System log monitor supports different log management tools with different log watchers:

• filelog: Log watcher for arbitrary file based log.
• journald: Log watcher for journald.
• kmsg: Log watcher for the kernel ring buffer device, /dev/kmsg. Set plugin in the configuration file to specify log watcher.

Log watcher specific configurations are configured in pluginConfig.

• journald
  • source: The SYSLOG_IDENTIFIER of the log to watch.
• filelog:
  • timestamp: The regular expression used to match timestamp in the log line. Submatch is supported, but only the last result will be used as the actual timestamp.
  • message: The regular expression used to match message in the log line. Submatch is supported, but only the last result will be used as the actual message.
  • timestampFormat: The format of the timestamp. The format string is the time 2006-01-02T15:04:05Z07:00 in the expected format. (See golang timestamp format)
• kmsg: No configuration for now.

Log on different OS distros may locate in different path. The logPath field in the configuration file is the log path. You can always configure logPath to match your OS distro.

• filelog: logPath is the path of log file, e.g. /var/log/kern.log for kernel log.
• journald: logPath is the journal log directory, usually /var/log/journal.

System log monitor uses Log Watcher to support different log management tools. It is easy to implement a new log watcher.