Centralized Logging

with syslog-ng and

SEC
Leon Towns-von Stauber, Intelius
LinuxFest Northwest, April 2010
http://www.occam.com/
Contents
Introduction 3
Example Issues 10
syslog-ng 18
Simple Event Correlator 37

2
Introduction
This talk describes an infrastructure that provides:
– Aggregation of system logs from many UNIX hosts
and other network devices
– Automated analysis of logged events

3
Introduction
The benefits of centralized log aggregation and analysis
include:
– Log reduction and correlation reduce the workload
associated with viewing logs, making regular review
feasible
– Regular review of logs gives sysadmins a better feel
for the computing environment, allows them to spot
anomalies more readily
– Automated analysis and reporting provides early
warning of unusual and possibly problematic events
– Relaying log messages to a secure loghost makes
them immune to tampering by a local intruder, permits
later forensic analysis
4
Legal Notices
This presentation Copyright © 2008-2010 Leon Towns-von
Stauber. All rights reserved.
Trademark notices
– syslog-ng™ is a trademark of BalaBit IT Security. See
http://www.balabit.com/trademarks/.
– Other trademarks are the property of their respective
owners.

5
Introduction - Logging Environment
Loghost
– HP ProLiant DL360 G5
• Two quad-core 2.33-GHz 64-bit Intel Xeon CPUs
• 16 GB RAM
• Two Gigabit Ethernet interfaces (1 used)
• Two 146-GB disks, RAID 1 => 136-GB boot volume
• Fifteen 146-GB disks, RAID 5 => 1.9 TB for log data
– Red Hat Enterprise Linux 4.6
– syslog-ng 2.0.9, SEC 2.4.2
• This host placed in service May 2008, previous
server in November 2007
6
Introduction - Logging Environment
Clients
– About 320 Red Hat Enterprise Linux hosts
– Over 80 networking devices: F5 BIG-IP load
balancers, Juniper NetScreen firewalls and SSL-VPN
concentrators, Cisco, Juniper, and Nortel switches,
Cisco wireless controllers

7
Introduction - Logging Environment

Centralized Logging Environment 8

Introduction
General approach
– Send all logs from clients to loghost
– Run all logs through filters
• Suppress routine or unimportant things
• Use correlation to simplify complex logging events
– Send whatever makes it through the filters to admins
on a regular basis
• Realtime alerts for specific known events

9
Example Issues
Example Issue - Hardware problems
Loose fan
Mar 4 09:37:26 host1.intelius.com hpasmlited: WARNING: System Fans Not Redundant (Location Power Supply)
Mar 4 09:37:36 host1.intelius.com hpasmlited: NOTICE: System Fans Not Redundant (Location Power Supply) has
been repaired
Mar 4 09:55:50 host1.intelius.com hpasmlited: WARNING: System Fans Not Redundant (Location Power Supply)
Mar 4 09:56:00 host1.intelius.com hpasmlited: NOTICE: System Fans Not Redundant (Location Power Supply) has
been repaired

Broken fan
Apr 2 10:00:11 host2.intelius.com hpasmlited: CRITICAL: Fan Failure (Fan 2, Location CPU)
Apr 2 10:00:11 host2.intelius.com hpasmlited: WARNING: System Fans Not Redundant (Location CPU)
Apr 2 10:00:21 host2.intelius.com hpasmlited: NOTICE: Fan Failure (Fan 2, Location CPU) has been repaired
Apr 2 10:00:21 host2.intelius.com hpasmlited: NOTICE: System Fans Not Redundant (Location CPU) has been
repaired
Apr 2 10:39:13 host2.intelius.com hpasmlited: CRITICAL: Fan Failure (Fan 2, Location CPU)
Apr 2 10:39:13 host2.intelius.com hpasmlited: WARNING: System Fans Not Redundant (Location CPU)

Fan reseated or replaced

11
Example Issue - Orphaned crontabs
crond complaining about root.cfsaved
Jan 21 16:31:01 host5.intelius.com crond: (root.cfsaved) ORPHAN (no passwd entry)
Jan 21 16:31:01 host7.intelius.com crond: (root.cfsaved) ORPHAN (no passwd entry)
Jan 21 16:31:01 host3.intelius.com crond: (root.cfsaved) ORPHAN (no passwd entry)
Jan 21 16:31:01 host4.intelius.com crond: (root.cfsaved) ORPHAN (no passwd entry)
Jan 21 16:31:01 host1.intelius.com crond: (root.cfsaved) ORPHAN (no passwd entry)
Jan 21 16:31:01 host2.intelius.com crond: (root.cfsaved) ORPHAN (no passwd entry)

When Cfengine updated the root crontab, it saved a

backup as root.cfsaved
– crond complained since no user named root.cfsaved
exists
Set backup=false in Cfengine config that copies crontab

12
Example Issue - xinetd won’t start
Recurring messages
Jan 21 17:00:08 host4.intelius.com cfengine:host4: Executing shell command: /etc/
init.d/xinetd start;/sbin/chkconfig xinetd on
Jan 21 17:00:08 host4.intelius.com cfengine:host4: (Done with /etc/init.d/xinetd
start;/sbin/chkconfig xinetd on)

Problem in /etc/sysconfig/network
– Changed
• NETWORKING=YES
– to
• NETWORKING=yes
– Who knew that was case-sensitive?

13
Example Issue - DHCP misconfiguration
Errors from dhcpd
Apr 7 12:56:34 host1.intelius.com dhcpd: /etc/dhcpd/172.27.4.conf line 153: expecting numeric value.
Apr 7 12:56:34 host1.intelius.com dhcpd: hardware ethernet 00:b0:c7:82:3u:
Apr 7 12:56:34 host1.intelius.com dhcpd: ^
Apr 7 12:56:34 host1.intelius.com dhcpd: /etc/dhcpd/172.27.4.conf line 158: expecting numeric value.
Apr 7 12:56:34 host1.intelius.com dhcpd: /etc/dhcpd.conf line 17: /etc/dhcpd/172.27.4.conf: bad parse.
Apr 7 12:56:34 host1.intelius.com dhcpd: include "/etc/dhcpd/172.27.4.conf"
Apr 7 12:56:34 host1.intelius.com dhcpd: ^
Apr 7 12:56:34 host1.intelius.com dhcpd: Configuration file errors encountered -- exiting

3e mistakenly entered as non-hexadecimal 3u

14
Example Issue - NTP problems
Time not synced very well on some hosts, as indicated by
weekly cron jobs running off schedule
This rule suppresses logs associated with weekly syslogd
restart, if they’re within 10 secs of scheduled time of 04:02
type=suppress
desc=Syslogd restart after regular log rotation
ptype=regexp
pattern=04:02:0\d [\w.-]+ syslogd [\d.]+: restart\.

So when syslogd restarts show up, it’s worth investigating

Dec 2 04:02:13 host6 syslogd 1.4.1: restart.
Dec 2 04:02:10 host5 syslogd 1.4.1: restart.
Dec 2 04:01:43 host1 syslogd 1.4.1: restart.
Dec 9 04:02:10 host6 syslogd 1.4.1: restart.
Dec 9 04:01:41 host1 syslogd 1.4.1: restart.
Dec 16 04:01:39 host1 syslogd: restarted
Dec 16 04:02:12 host5 syslogd: restarted
Dec 23 04:01:37 host1 syslogd: restarted

15
Example Issue - NTP problems
Variety of fixes
– Resetting clock
– Starting ntpd
– Updating zoneinfo files
– Relinking /etc/localtime
– Replacing /etc/ntp.conf to use correct servers

16
Example Issue - DNS probes
Lots of DNS zone transfer attempts on our external
nameservers from a variety of sources
Dec 4 17:08:50 MULTIPLE-HOSTS named: PROBE from 12.108.127.137: zone transfer '125.94.64.in-addr.arpa'
denied
Dec 4 17:08:52 MULTIPLE-HOSTS named: PROBE from 208.117.131.116: zone transfer 'intelius.com' denied
Dec 4 17:08:52 MULTIPLE-HOSTS named: PROBE from 129.24.211.26: zone transfer 'intelius.com' denied
Dec 4 17:08:52 MULTIPLE-HOSTS named: PROBE from 142.150.238.13: zone transfer 'intelius.com' denied
Dec 4 17:08:53 MULTIPLE-HOSTS named: PROBE from 131.246.191.41: zone transfer 'intelius.com' denied

Traced to a PlanetLab project described here:

– http://wwwse.inf.tu-dresden.de/SEDNS/
SEDNS_home.htm
Contacted researchers, added our nameservers to exclusion
list

17
syslog-ng
syslog-ng - Intro
syslog-ng is a replacement for UNIX syslogd, started by
Balázs Scheider in 1998
– Now also offered in a commercial version by BalaBit
– http://www.balabit.com/network-security/syslog-ng/
– Central Logging for Unix
• http://sial.org/talks/central-logging/
This talk is based on version 2.0.9
– Current versions are 2.0.10, 3.0.6, and 3.1.1

19
syslog-ng - Client Setup
Clients continue to use stock syslogd
– They require only one configuration change
/etc/syslog.conf
– Send all logs to loghost
• *.debug @loghost
– Here’s the full config file used on our Linux hosts:
*.info;mail.none;authpriv.none;cron.none! /var/log/messages
authpriv.*! ! ! ! ! ! ! ! ! ! /var/log/secure
local7.*! ! ! ! ! ! ! ! ! ! ! /var/log/boot.log

*.emerg! ! ! ! ! ! ! ! ! ! ! *

*.debug! ! ! ! ! ! ! ! ! ! ! @loghost

20
syslog-ng - Server Setup
Dedicated account for use by syslog-ng
– syslog:x:514:514::/mnt0/syslog:/bin/false
– Locked password
– Group is used by those who need to view logs
After compiling, installed under /usr/local/
Created init script
Disabled syslogd

21
syslog-ng - Server Setup
All the log files are under /mnt0/syslog/
– The complete record for the day is all
– The working files used by SEC for regular updates are
net.tmp and unix.tmp
• These files go away when a regular update is sent
out
– syslog-ng-filtered logs are in byfac/ and byapp/
• Some handy symlinks are in bylnk/, to help
remember what the various local facilities
(local1, local2, etc.) are used for
– SEC-filtered logs are in sec/
– Rotated log files are in archive/ 22
 syslog-ng - Client Setup
-rw-r--r-- 1 syslog syslog 388844629 2008-04-18 16:38 all
drwxr-sr-x 6 syslog syslog 4096 2007-11-30 15:03 archive
drwxr-sr-x 2 syslog syslog 4096 2008-04-13 23:50 byapp
drwxr-sr-x 2 syslog syslog 4096 2008-04-13 23:52 byfac
drwxr-sr-x 2 syslog syslog 4096 2008-02-25 16:44 bylnk
-rw-r--r-- 1 root syslog 206 2008-04-18 16:09 net.tmp
drwxr-xr-x 2 syslog syslog 4096 2008-04-14 16:29 sec
-rw-r--r-- 1 root syslog 1448 2008-04-18 16:22 unix.tmp

Contents of /mnt0/syslog/ 23
 syslog-ng - Client Setup
byapp:
total 336
-rw-r--r-- 1 syslog syslog 299657 2008-04-24 09:38 emerg
-rw-r--r-- 1 syslog syslog 34988 2008-04-24 10:11 su

byfac:
total 1863780
-rw-r--r-- 1 syslog syslog 6060090 2008-04-24 10:25 auth
-rw-r--r-- 1 syslog syslog 254037293 2008-04-24 10:25 authpriv
-rw-r--r-- 1 syslog syslog 33953953 2008-04-24 10:25 cron
-rw-r--r-- 1 syslog syslog 80001833 2008-04-24 10:25 daemon
-rw-r--r-- 1 syslog syslog 2682950 2008-04-24 09:34 kern
-rw-r--r-- 1 syslog syslog 641499016 2008-04-24 10:25 local0
-rw-r--r-- 1 syslog syslog 28625 2008-04-23 23:50 local1
-rw-r--r-- 1 syslog syslog 948894 2008-04-24 10:23 local3
-rw-r--r-- 1 syslog syslog 990184 2008-04-24 10:25 local4
-rw-r--r-- 1 syslog syslog 228 2008-04-21 10:24 local5
-rw-r--r-- 1 syslog syslog 44242940 2008-04-24 10:25 local6
-rw-r--r-- 1 syslog syslog 139737 2008-04-24 09:54 local7
-rw-r--r-- 1 syslog syslog 840311651 2008-04-24 10:25 mail
-rw-r--r-- 1 syslog syslog 344949 2008-04-24 10:20 syslog
-rw-r--r-- 1 syslog syslog 1311198 2008-04-24 10:24 user

Contents of /mnt0/syslog/ 24
 syslog-ng - Client Setup
bylnk:
total 0
lrwxrwxrwx 1 root syslog 15 2007-12-14 12:04 boot -> ../byfac/local7
lrwxrwxrwx 1 root syslog 15 2007-11-30 14:21 cisco -> ../byfac/local7
lrwxrwxrwx 1 root syslog 15 2007-12-19 15:05 named -> ../byfac/local4
lrwxrwxrwx 1 root syslog 13 2007-12-03 13:58 netbackup -> ../byfac/user
lrwxrwxrwx 1 root syslog 15 2007-11-30 14:22 netscreen -> ../byfac/local0
lrwxrwxrwx 1 root syslog 15 2007-11-30 14:19 rsyncd -> ../byfac/local3
lrwxrwxrwx 1 root syslog 15 2007-11-30 14:18 sec -> ../byfac/local1
lrwxrwxrwx 1 root syslog 15 2008-02-25 16:44 slapd -> ../byfac/local4
lrwxrwxrwx 1 root syslog 15 2008-02-22 15:05 snmpd -> ../byfac/local5
lrwxrwxrwx 1 root syslog 17 2007-12-03 10:32 sudo -> ../byfac/authpriv

Contents of /mnt0/syslog/ 25
syslog-ng - Config File
Config file is /usr/local/etc/syslog-ng.conf
The config file has 5 kinds of statements
– General options
– Sources and destinations
– Filters
– Log statements, where you direct messages from
sources to destinations through filters
I use this configuration for rough filtering and message
routing, and to launch SEC processes for further, finer-
grained parsing
– Also, SEC can’t filter based on facility or severity
unless they’re included in the message text, so
syslog-ng is useful for that 26
syslog-ng - Config File
Options
options {
! group("syslog");
! perm(0644);
! create_dirs(yes);
! dir_group("syslog");
! dir_perm(0755);
! use_fqdn(yes);
! chain_hostnames(no);
! dns_cache_expire(21600);
! dns_cache_size(2000);
! log_fifo_size(200000);
};

– Setting group and permissions

– create_dirs(yes) - Create log dirs on the fly
– use_fqdn(yes) - Log messages with host’s FQDN
– chain_hostnames(no) - Record only the source
host of a message 27
syslog-ng - Config File
Options
options {
! group("syslog");
! perm(0644);
! create_dirs(yes);
! dir_group("syslog");
! dir_perm(0755);
! use_fqdn(yes);
! chain_hostnames(no);
! dns_cache_expire(21600);
! dns_cache_size(2000);
! log_fifo_size(200000);
};

– dns_cache_expire, dns_cache_size - Increase

retention time and size of DNS lookup cache (default
3600 secs and 1007)
– log_fifo_size - Increase size of message buffer
(default 100)
28
syslog-ng - Config File
Here’s the first log statement
log { source(s_all); destination(d_all); };

– Every log message received by syslog-ng goes to the

d_all destination (no filtering)
– Here’s the source definition
source s_all {
! internal();
! unix-stream("/dev/log");
! udp();
};

• Messages are generated internally by syslog-ng,

from the loghost itself, or from remote clients (via
UDP)

29
syslog-ng - Config File
Here’s the first log statement
log { source(s_all); destination(d_all); };

– Here’s the destination

destination d_all! { file("/mnt0/syslog/all"
! ! ! template("$R_DATE $HOST $MSG\n")
! ! ! template_escape(no));
! ! ! program("`/usr/local/bin/secStart main`"
! ! ! template("$R_DATE $HOST $MSG\n")
! ! ! template_escape(no)); };

• All messages are recorded in /mnt0/syslog/all

• Records messages with the time received, the
source hostname, and the message content,
consistent with standard syslog format

30
syslog-ng - Config File
Here’s the first log statement
log { source(s_all); destination(d_all); };

– Here’s the destination

destination d_all! { file("/mnt0/syslog/all"
! ! ! template("$R_DATE $HOST $MSG\n")
! ! ! template_escape(no));
! ! ! program("`/usr/local/bin/secStart main`"
! ! ! template("$R_DATE $HOST $MSG\n")
! ! ! template_escape(no)); };

• In addition to specifying the log file, when this

destination is set up the secStart script runs, used
to spawn an SEC process to handle the same set of
messages
–More on secStart later

31
syslog-ng - Config File
The second log statement
log { source(s_all); destination(d_fac); };

– Here’s the destination

destination d_fac!
{ file("/mnt0/syslog/byfac/$FACILITY"
! ! ! template("$R_DATE $HOST $MSG\n")
! ! ! template_escape(no)); };

• Messages are automatically sorted into separate log

files per syslog facility
–/mnt0/syslog/byfac/auth, /mnt0/syslog/byfac/
daemon, /mnt0/syslog/byfac/kern, /mnt0/syslog/
byfac/local0, etc.

32
syslog-ng - Config File
One more example, with a simple filter attached
log { source(s_all); filter(f_emerg); destination(d_emerg); };

– Destination
destination d_emerg! { file("/mnt0/syslog/byapp/emerg"
! ! ! template("$R_DATE $HOST $MSG\n")
! ! ! template_escape(no));
! ! ! program("`/usr/local/bin/secStart emerg`"
! ! ! template("$R_DATE $HOST $MSG\n")
! ! ! template_escape(no)); };

• Messages of emerg severity have a dedicated SEC

process to generate immediate email notifications
– Filter
filter f_emerg { level(emerg); };

• You can use keywords (such as level, program,

or host) and Boolean logic to construct filters 33
syslog-ng - Performance
Currently, this installation of syslog-ng handles 60-80 million
log messages per day (over 800 msgs/sec on average)
– Up from 10-20 million in 4/09, 4-5 million in 11/08, and
1.3 million in 6/08
– About 65 million from firewalls
– Nearly all the remaining logs are related to email, BIG-
IPs, SSH, cron, and Nagios agents (nrpe)
At current rate, syslog-ng takes about 25 MB of RAM, using
about 5% of CPU

34
syslog-ng - Performance
Loghost Syslog Metrics
24,000,000

21,000,000

18,000,000
11/11/08
McColo shutdown
Categorized Messages Per Day

15,000,000

12,000,000

9,000,000

6,000,000

3,000,000

weekly periodicity

0
11/07! 12/07! 1/08! 2/08! 3/08! 4/08! 5/08! 6/08! 7/08! 8/08! 9/08! 10/08! 11/08! 12/08! 1/09! 2/09! 3/09! 4/09

Firewall Firewall
BIG-IP BIG-IP
Email Email
nrpe nrpe
cron cron
sshd sshd
dhcpd dhcpd

Messages Per Day 35

syslog-ng - Performance
Loghost Syslog Metrics
90,000,000

80,000,000

70,000,000

60,000,000
Categorized Messages Per Day

50,000,000

40,000,000

11/11/08
McColo shutdown
30,000,000

20,000,000

10,000,000

weekly periodicity
0
11/07! 1/08! 3/08! 5/08! 7/08! 9/08! 11/08! 1/09! 3/09! 5/09! 7/09! 9/09! 11/09! 1/10! 3/10

Firewall Firewall
BIG-IP BIG-IP
Email Email
nrpe nrpe
cron cron
sshd sshd
dhcpd dhcpd

Messages Per Day 36

Simple Event
Correlator
SEC - Intro
Simple Event Correlator (SEC) is written by Risto Vaarandi,
and was first released in 2001
– http://www.estpak.ee/~risto/sec/
– Working with SEC
• http://sixshooter.v6.thrupoint.net/SEC-examples/
article.html
It’s essentially an 8000+-line Perl script used to
automatically process log messages of any kind
– Similar to Swatch, but much more sophisticated, and
with that sophistication comes greater complexity
This talk is based on version 2.4.2
– Current version is 2.5.3
38
SEC - Intro
Started by syslog-ng using secStart script
– Argument to secStart specifies SEC config to use
– Why do this instead of running independent sec
processes to monitor the log files themselves?
• Difficult to guarantee that messages wouldn’t be
missed, or parsed two or more times, when procs
restarted (during log rotation, config update, etc.)
• Know that every message received by syslog-ng is
parsed exactly once by the appropriate sec proc,
and that all procs stop and start in sync
• secStart makes syslog-ng.conf much cleaner

39
SEC - Intro
#!/bin/sh
#
# secStart - Print SEC command line with default options.

usage () {
! echo "usage:!$progname config

! 'config' is the name of an SEC config file in /usr/local/etc/

sec/." >&2
! exit 2
}

progname=`basename $0`

[ $# -eq 1 ] || usage

echo "/usr/local/sbin/sec -conf=/usr/local/etc/sec/$1 -pid=/var/

run/sec-$1.pid -dump=/mnt0/syslog/sec-$1.dump -debug=5 -
syslog=local1 -intevents -input=-"
secStart 40
SEC - Configuration
SEC config files are located in /usr/local/etc/sec/
– They could be located anywhere, as they’re specified
in the sec command line
– There’s a main config (currently over 5300 lines, 896
rules) and some small special-purpose configs (disk,
emerg, hitemp, and outbound, 15-80 lines apiece)

41
SEC - Configuration
An SEC configuration is composed of multi-line stanzas, or
rule definitions, with each line containing a key and value
Keys include:
– type - Type of rule (examples later)
– desc - Textual description of rule
– ptype - Type of pattern (typically regexp)
– pattern - String or Perl-style regular expression
used to match log message
– context - Apply rule only when named context in
effect
– action - What to do when rule is matched
– continue - After this rule, continue or stop (default) 42
SEC - Configuration
Rule types used in the examples
– suppress - Simple rule to toss messages that match
– single - If message matches, take immediate action
– singlewithsuppress - If message matches, take
immediate action, but then ignore similar messages
for a time given by value of window
– singlewiththreshold - Take action if the number
of matching messages within a given window reaches
a threshold
– pairwithwindow - Specify 2 patterns; when 1st
pattern matches, watch for 2nd pattern to appear
within window; if it does, execute action; if not,
execute different action 43
SEC - Configuration
For each message, rules are processed one at a time, in
order, until the message matches a rule without
continue=takenext, or end-of-file is reached
A few examples from the main config follow

44
SEC - Configuration Example: main
Lots of simple suppress rules like these:
type=suppress
ptype=regexp
pattern=last message repeated

type=suppress
desc=SEC logs
ptype=regexp
pattern=loghost\.intelius\.com sec\[\d+\]: SEC \(Simple Event Correlator\)

type=suppress
desc=xinetd startup/reconfig msgs
ptype=regexp
pattern=xinetd\[\d+\]: (Reading included configuration file:|removing|Exiting\.\.\.|
xinetd Version|Started working: \d+ available service|Starting reconfiguration|
Swapping defaults|readjusting service)

type=suppress
desc=CRON
ptype=regexp
pattern=(CROND|crond|crond\(pam_unix\)|\/USR\/SBIN\/CRON|\/usr\/sbin\/cron)\[\d+\]:
\(\w+\) (CMD|RELOAD|STARTUP)
45
SEC - Configuration Example: main
Correlation of boot logs
##############
# BOOT RULES #
##############

type=single
desc=Create boot context
ptype=regexp
pattern=(\w+\s+\d+\s+\d+:\d+:\d+) ([\w.-]+) kernel: Linux version
action=create BOOT_$2 180; create CFENGINE_BOOT_$2 900;\
create NTP_STOPSTART_$2 900; event 0 UNDUPED:$1 $2 starting up...
context=!BOOT_$2

This first rule sets up the host-specific boot context

– Also sets up contexts for Cfengine and NTP logs that
show up later than the rest
– Logs a message as UNDUPED:, rather than PARSED:,
to bypass multi-host correlation and see every bootup
46
SEC - Configuration Example: main
Correlation of boot logs
– Remaining rules are suppressions, which can turn this
Apr 24 14:22:50 host2.intelius.com syslogd 1.4.1: restart.
Apr 24 14:22:50 host2.intelius.com syslog: syslogd startup succeeded
Apr 24 14:22:50 host2.intelius.com kernel: klogd 1.4.1, log source = /proc/kmsg started.
Apr 24 14:22:50 host2.intelius.com kernel: Bootdata ok (command line is ro root=LABEL=/ console=tty0 console=ttyS1)
Apr 24 14:22:50 host2.intelius.com kernel: Linux version 2.6.9-67.ELsmp ([email protected]) (gcc version 3.4.6 20060404 (Red Hat 3.4.6-8)) #1 SMP Wed Nov 7 13:56:44 EST 2007
Apr 24 14:22:50 host2.intelius.com kernel: BIOS-provided physical RAM map: Apr 24 14:22:50 host2.intelius.com kernel: PCI: Ignoring BAR0-3 of IDE controller 0000:00:1f.1 Apr 24 14:22:50 host2.intelius.com kernel: ehci_hcd 0000:00:1d.7: new USB bus registered, assigned bus number 6
Apr 24 14:22:50 host2.intelius.com kernel: BIOS-e820: 0000000000000000 - 000000000009f400 (usable) Apr 24 14:22:50 host2.intelius.com kernel: PCI: Transparent bridge - 0000:00:1e.0 Apr 24 14:22:50 host2.intelius.com kernel: PCI: cache line size of 32 is not supported by device 0000:00:1d.7
Apr 24 14:22:50 host2.intelius.com kernel: BIOS-e820: 000000000009f400 - 00000000000a0000 (reserved) Apr 24 14:22:50 host2.intelius.com kernel: ACPI: PCI Interrupt Routing Table [\_SB_.PCI0._PRT] Apr 24 14:22:50 host2.intelius.com kernel: ehci_hcd 0000:00:1d.7: USB 2.0 enabled, EHCI 1.00, driver 2004-May-10
Apr 24 14:22:50 host2.intelius.com kernel: BIOS-e820: 00000000000f0000 - 0000000000100000 (reserved) Apr 24 14:22:50 host2.intelius.com kernel: ACPI: PCI Interrupt Routing Table [\_SB_.PCI0.IP2P._PRT] Apr 24 14:22:50 host2.intelius.com kernel: hub 6-0:1.0: USB hub found
Apr 24 14:22:50 host2.intelius.com kernel: BIOS-e820: 0000000000100000 - 00000000cfe56000 (usable) Apr 24 14:22:50 host2.intelius.com kernel: ACPI: PCI Interrupt Routing Table [\_SB_.PCI0.PT02._PRT] Apr 24 14:22:50 host2.intelius.com kernel: hub 6-0:1.0: 8 ports detected
Apr 24 14:22:50 host2.intelius.com kernel: BIOS-e820: 00000000cfe56000 - 00000000cfe5e000 (ACPI data) Apr 24 14:22:50 host2.intelius.com kernel: ACPI: PCI Interrupt Routing Table [\_SB_.PCI0.PT02.IPE4._PRT] Apr 24 14:22:50 host2.intelius.com kernel: md: Autodetecting RAID arrays.
Apr 24 14:22:50 host2.intelius.com kernel: BIOS-e820: 00000000cfe5e000 - 00000000cfe5f000 (usable) Apr 24 14:22:50 host2.intelius.com kernel: ACPI: PCI Interrupt Routing Table [\_SB_.PCI0.PT02.IPE4.IPE1._PRT] Apr 24 14:22:50 host2.intelius.com kernel: md: autorun ...
Apr 24 14:22:50 host2.intelius.com kernel: BIOS-e820: 00000000cfe5f000 - 00000000d0000000 (reserved) Apr 24 14:22:50 host2.intelius.com kernel: ACPI: PCI Interrupt Routing Table [\_SB_.PCI0.PT02.P2P2._PRT] Apr 24 14:22:50 host2.intelius.com kernel: md: ... autorun DONE.
Apr 24 14:22:50 host2.intelius.com kernel: BIOS-e820: 00000000e0000000 - 00000000f0000000 (reserved) Apr 24 14:22:50 host2.intelius.com kernel: ACPI: PCI Interrupt Routing Table [\_SB_.PCI0.PT03._PRT] Apr 24 14:22:50 host2.intelius.com kernel: ACPI: Power Button (FF) [PWRF]
Apr 24 14:22:50 host2.intelius.com kernel: BIOS-e820: 00000000fec00000 - 00000000fed00000 (reserved) Apr 24 14:22:50 host2.intelius.com kernel: ACPI: PCI Interrupt Routing Table [\_SB_.PCI0.PT04._PRT] Apr 24 14:22:50 host2.intelius.com kernel: usb 5-1: new full speed USB device using address 2
Apr 24 14:22:50 host2.intelius.com kernel: BIOS-e820: 00000000fee00000 - 00000000fee10000 (reserved) Apr 24 14:22:50 host2.intelius.com kernel: ACPI: PCI Interrupt Routing Table [\_SB_.PCI0.PT06.NB01._PRT] Apr 24 14:22:50 host2.intelius.com kernel: input: USB HID v1.01 Keyboard [HP Virtual Keyboard] on usb-0000:01:04.4-1
Apr 24 14:22:50 host2.intelius.com syslog: klogd startup succeeded Apr 24 14:22:50 host2.intelius.com kernel: ACPI: PCI Interrupt Routing Table [\_SB_.PCI0.PT07.NB02._PRT] Apr 24 14:22:50 host2.intelius.com kernel: input: USB HID v1.01 Mouse [HP Virtual Keyboard] on usb-0000:01:04.4-1
Apr 24 14:22:50 host2.intelius.com kernel: BIOS-e820: 00000000ffc00000 - 0000000100000000 (reserved) Apr 24 14:22:50 host2.intelius.com kernel: ACPI: PCI Interrupt Link [LNKA] (IRQs *5 7 10 11) Apr 24 14:22:50 host2.intelius.com kernel: usb 5-2: new full speed USB device using address 3
Apr 24 14:22:50 host2.intelius.com kernel: BIOS-e820: 0000000100000000 - 000000072ffff000 (usable) Apr 24 14:22:50 host2.intelius.com kernel: ACPI: PCI Interrupt Link [LNKB] (IRQs 5 *7 10 11) Apr 24 14:22:50 host2.intelius.com kernel: hub 5-2:1.0: USB hub found
Apr 24 14:22:50 host2.intelius.com kernel: ACPI: RSDP (v002 HP ) @ 0x00000000000f4f00 Apr 24 14:22:50 host2.intelius.com kernel: ACPI: PCI Interrupt Link [LNKC] (IRQs 5 7 *10 11) Apr 24 14:22:50 host2.intelius.com kernel: hub 5-2:1.0: 7 ports detected
Apr 24 14:22:50 host2.intelius.com kernel: ACPI: XSDT (v001 HP ProLiant 0x00000002 “ Apr 24 14:22:50 host2.intelius.com kernel: ACPI: PCI Interrupt Link [LNKD] (IRQs 5 7 *10 11) Apr 24 14:22:50 host2.intelius.com kernel: EXT3 FS on cciss/c0d0p1, internal journal
0x0000162e) @ 0x00000000cfe567c0 Apr 24 14:22:50 host2.intelius.com kernel: ACPI: PCI Interrupt Link [LNKE] (IRQs 5 7 10 11) *0, disabled. Apr 24 14:22:50 host2.intelius.com kernel: device-mapper: 4.5.5-ioctl (2006-12-01) initialised: [email protected]
Apr 24 14:22:50 host2.intelius.com kernel: ACPI: FADT (v003 HP ProLiant 0x00000002 “ Apr 24 14:22:50 host2.intelius.com kernel: ACPI: PCI Interrupt Link [LNKF] (IRQs *5 7 10 11) Apr 24 14:22:50 host2.intelius.com kernel: kjournald starting. Commit interval 5 seconds
0x0000162e) @ 0x00000000cfe56840 Apr 24 14:22:50 host2.intelius.com kernel: ACPI: PCI Interrupt Link [LNKG] (IRQs 5 7 *10 11) Apr 24 14:22:50 host2.intelius.com kernel: EXT3 FS on cciss/c0d0p2, internal journal
Apr 24 14:22:50 host2.intelius.com kernel: ACPI: SPCR (v001 HP SPCRRBSU 0x00000001 “ Apr 24 14:22:50 host2.intelius.com kernel: ACPI: PCI Interrupt Link [LNKH] (IRQs 5 *7 10 11) Apr 24 14:22:50 host2.intelius.com kernel: EXT3-fs: mounted filesystem with ordered data mode.
0x0000162e) @ 0x00000000cfe56140 Apr 24 14:22:50 host2.intelius.com kernel: usbcore: registered new driver usbfs Apr 24 14:22:50 host2.intelius.com kernel: kjournald starting. Commit interval 5 seconds
Apr 24 14:22:50 host2.intelius.com kernel: ACPI: MCFG (v001 HP ProLiant 0x00000001 0x00000000) @ 0x00000000cfe561c0 Apr 24 14:22:50 host2.intelius.com kernel: usbcore: registered new driver hub Apr 24 14:22:50 host2.intelius.com kernel: EXT3 FS on cciss/c0d0p6, internal journal
Apr 24 14:22:50 host2.intelius.com kernel: ACPI: HPET (v001 HP ProLiant 0x00000002 “ Apr 24 14:22:50 host2.intelius.com kernel: PCI: Using ACPI for IRQ routing Apr 24 14:22:50 host2.intelius.com kernel: EXT3-fs: mounted filesystem with ordered data mode.
0x0000162e) @ 0x00000000cfe56200 Apr 24 14:22:50 host2.intelius.com kernel: GSI 16 sharing vector 0xA9 and IRQ 16 Apr 24 14:22:50 host2.intelius.com kernel: kjournald starting. Commit interval 5 seconds
Apr 24 14:22:50 host2.intelius.com kernel: ACPI: SPMI (v005 HP ProLiant 0x00000001 “ Apr 24 14:22:50 host2.intelius.com kernel: ACPI: PCI Interrupt 0000:00:1d.0[A] -> GSI 16 (level, low) -> IRQ 169 Apr 24 14:22:50 host2.intelius.com kernel: EXT3 FS on cciss/c0d0p3, internal journal
0x0000162e) @ 0x00000000cfe56240 Apr 24 14:22:50 host2.intelius.com kernel: GSI 17 sharing vector 0xB1 and IRQ 17 Apr 24 14:22:50 host2.intelius.com kernel: EXT3-fs: mounted filesystem with ordered data mode.
Apr 24 14:22:50 host2.intelius.com kernel: ACPI: ERST (v001 HP ProLiant 0x00000001 “ Apr 24 14:22:50 host2.intelius.com kernel: ACPI: PCI Interrupt 0000:00:1d.1[B] -> GSI 17 (level, low) -> IRQ 177 Apr 24 14:22:50 host2.intelius.com kernel: Adding 4192924k swap on /dev/cciss/c0d0p5. Priority:-1 extents:1
0x0000162e) @ 0x00000000cfe56280 Apr 24 14:22:50 host2.intelius.com kernel: GSI 18 sharing vector 0xB9 and IRQ 18 Apr 24 14:22:50 host2.intelius.com kernel: IA-32 Microcode Update Driver: v1.14 <[email protected]>
Apr 24 14:22:50 host2.intelius.com kernel: ACPI: MADT (v001 HP ProLiant 0x00000002 0x00000000) @ 0x00000000cfe56480 Apr 24 14:22:50 host2.intelius.com kernel: ACPI: PCI Interrupt 0000:00:1d.2[C] -> GSI 18 (level, low) -> IRQ 185 Apr 24 14:22:50 host2.intelius.com kernel: microcode: No new microdata for cpu 2
Apr 24 14:22:50 host2.intelius.com kernel: ACPI: FFFF (v001 HP ProLiant 0x00000001 “ Apr 24 14:22:50 host2.intelius.com nfslock: rpc.statd startup succeeded Apr 24 14:22:50 host2.intelius.com kernel: microcode: No new microdata for cpu 4
0x0000162e) @ 0x00000000cfe56540 Apr 24 14:22:50 host2.intelius.com kernel: GSI 19 sharing vector 0xC1 and IRQ 19 Apr 24 14:22:50 host2.intelius.com kernel: microcode: No new microdata for cpu 6
Apr 24 14:22:50 host2.intelius.com kernel: ACPI: BERT (v001 HP ProLiant 0x00000001 “ Apr 24 14:22:50 host2.intelius.com kernel: ACPI: PCI Interrupt 0000:00:1d.3[D] -> GSI 19 (level, low) -> IRQ 193 Apr 24 14:22:50 host2.intelius.com kernel: microcode: No new microdata for cpu 7
0x0000162e) @ 0x00000000cfe566c0 Apr 24 14:22:50 host2.intelius.com kernel: ACPI: PCI Interrupt 0000:00:1d.7[A] -> GSI 16 (level, low) -> IRQ 169 Apr 24 14:22:50 host2.intelius.com kernel: microcode: No new microdata for cpu 1
Apr 24 14:22:50 host2.intelius.com kernel: ACPI: HEST (v001 HP ProLiant 0x00000001 “ Apr 24 14:22:50 host2.intelius.com kernel: ACPI: PCI Interrupt 0000:00:1f.1[A] -> GSI 17 (level, low) -> IRQ 177 Apr 24 14:22:50 host2.intelius.com kernel: microcode: No new microdata for cpu 3
0x0000162e) @ 0x00000000cfe56700 Apr 24 14:22:50 host2.intelius.com kernel: ACPI: PCI Interrupt 0000:0a:00.0[A] -> GSI 16 (level, low) -> IRQ 169 Apr 24 14:22:50 host2.intelius.com kernel: microcode: No new microdata for cpu 5
Apr 24 14:22:50 host2.intelius.com kernel: ACPI: DSDT (v001 HP DSDT 0x00000001 INTL 0x20030228) @ 0x0000000000000000 Apr 24 14:22:50 host2.intelius.com kernel: ACPI: PCI Interrupt 0000:0a:01.0[A] -> GSI 17 (level, low) -> IRQ 177 Apr 24 14:22:50 host2.intelius.com kernel: microcode: No new microdata for cpu 0
Apr 24 14:22:50 host2.intelius.com kernel: No NUMA configuration found Apr 24 14:22:50 host2.intelius.com kernel: ACPI: PCI Interrupt 0000:0a:02.0[A] -> GSI 18 (level, low) -> IRQ 185 Apr 24 14:22:50 host2.intelius.com kernel: IA-32 Microcode Update Driver v1.14 unregistered
Apr 24 14:22:50 host2.intelius.com kernel: Faking a node at 0000000000000000-000000072ffff000 Apr 24 14:22:50 host2.intelius.com kernel: ACPI: PCI Interrupt 0000:06:00.0[A] -> GSI 16 (level, low) -> IRQ 169 Apr 24 14:22:50 host2.intelius.com kernel: ip_tables: (C) 2000-2002 Netfilter core team
Apr 24 14:22:50 host2.intelius.com kernel: Bootmem setup node 0 0000000000000000-000000072ffff000 Apr 24 14:22:50 host2.intelius.com kernel: ACPI: PCI Interrupt 0000:03:00.0[A] -> GSI 18 (level, low) -> IRQ 185 Apr 24 14:22:50 host2.intelius.com kernel: ip_conntrack version 2.1 (8192 buckets, 65536 max) - 456 bytes per conntrack
Apr 24 14:22:50 host2.intelius.com kernel: On node 0 totalpages: 7536639 Apr 24 14:22:50 host2.intelius.com kernel: ACPI: PCI Interrupt 0000:05:00.0[A] -> GSI 19 (level, low) -> IRQ 193 Apr 24 14:22:50 host2.intelius.com kernel: bnx2: eth0: using MSI
Apr 24 14:22:50 host2.intelius.com kernel: DMA zone: 4096 pages, LIFO batch:1 Apr 24 14:22:50 host2.intelius.com kernel: GSI 20 sharing vector 0xC9 and IRQ 20 Apr 24 14:22:50 host2.intelius.com kernel: bnx2: eth0 NIC Link is Up, 1000 Mbps full duplex
Apr 24 14:22:50 host2.intelius.com kernel: Normal zone: 7532543 pages, LIFO batch:16 Apr 24 14:22:50 host2.intelius.com kernel: ACPI: PCI Interrupt 0000:01:03.0[A] -> GSI 23 (level, low) -> IRQ 201 Apr 24 14:22:50 host2.intelius.com rpcidmapd: rpc.idmapd startup succeeded
Apr 24 14:22:50 host2.intelius.com kernel: HighMem zone: 0 pages, LIFO batch:1 Apr 24 14:22:50 host2.intelius.com kernel: GSI 21 sharing vector 0xD1 and IRQ 21 Apr 24 14:22:50 host2.intelius.com netfs: Mounting NFS filesystems: succeeded
Apr 24 14:22:50 host2.intelius.com kernel: DMI 2.4 present. Apr 24 14:22:50 host2.intelius.com kernel: ACPI: PCI Interrupt 0000:01:04.0[A] -> GSI 21 (level, low) -> IRQ 209 Apr 24 14:22:50 host2.intelius.com netfs: Mounting other filesystems: succeeded
Apr 24 14:22:50 host2.intelius.com kernel: ServerWorks chipset detected. Disabling timer routing over 8254. Apr 24 14:22:50 host2.intelius.com kernel: GSI 22 sharing vector 0xD9 and IRQ 22 Apr 24 14:22:51 host2.intelius.com kernel: i2c /dev entries driver
Apr 24 14:22:50 host2.intelius.com kernel: ACPI: PM-Timer IO Port: 0x908 Apr 24 14:22:50 host2.intelius.com kernel: ACPI: PCI Interrupt 0000:01:04.2[B] -> GSI 22 (level, low) -> IRQ 217 Apr 24 14:22:51 host2.intelius.com rc: Starting lm_sensors: succeeded
Apr 24 14:22:50 host2.intelius.com kernel: ACPI: Local APIC address 0xfee00000 Apr 24 14:22:50 host2.intelius.com kernel: ACPI: PCI Interrupt 0000:01:04.4[B] -> GSI 22 (level, low) -> IRQ 217 Apr 24 14:22:51 host2.intelius.com acpid: acpid startup succeeded
Apr 24 14:22:50 host2.intelius.com kernel: ACPI: LAPIC (acpi_id[0x00] lapic_id[0x00] enabled) Apr 24 14:22:50 host2.intelius.com kernel: ACPI: PCI Interrupt 0000:01:04.6[A] -> GSI 21 (level, low) -> IRQ 209 Apr 24 14:22:51 host2.intelius.com snmpd: snmpd startup succeeded
Apr 24 14:22:50 host2.intelius.com kernel: Processor #0 6:15 APIC version 16 Apr 24 14:22:50 host2.intelius.com kernel: PCI-DMA: Using software bounce buffering for IO (SWIOTLB) Apr 24 14:22:51 host2.intelius.com sshd[3243]: Server listening on 0.0.0.0 port 22.
Apr 24 14:22:50 host2.intelius.com kernel: ACPI: LAPIC (acpi_id[0x04] lapic_id[0x04] enabled) Apr 24 14:22:50 host2.intelius.com kernel: IA32 emulation $Id: sys_ia32.c,v 1.32 2002/03/24 13:02:28 ak Exp $ Apr 24 14:22:51 host2.intelius.com sshd: succeeded
Apr 24 14:22:50 host2.intelius.com kernel: Processor #4 6:15 APIC version 16 Apr 24 14:22:50 host2.intelius.com kernel: audit: initializing netlink socket (disabled) Apr 24 14:22:51 host2.intelius.com xinetd[3258]: Reading included configuration file: /etc/xinetd.d/bpcd [file=/etc/xinetd.conf] [line=15]
Apr 24 14:22:50 host2.intelius.com kernel: ACPI: LAPIC (acpi_id[0x02] lapic_id[0x02] enabled) Apr 24 14:22:50 host2.intelius.com kernel: audit(1209046945.230:1): initialized Apr 24 14:22:51 host2.intelius.com xinetd: xinetd startup succeeded
Apr 24 14:22:50 host2.intelius.com kernel: Processor #2 6:15 APIC version 16 Apr 24 14:22:50 host2.intelius.com kernel: Total HugeTLB memory allocated, 0 Apr 24 14:22:51 host2.intelius.com xinetd[3258]: Reading included configuration file: /etc/xinetd.d/bpjava-msvc [file=/etc/xinetd.d/bpjava-msvc] [line=11]
Apr 24 14:22:50 host2.intelius.com kernel: ACPI: LAPIC (acpi_id[0x06] lapic_id[0x06] enabled) Apr 24 14:22:50 host2.intelius.com kernel: VFS: Disk quotas dquot_6.5.1 Apr 24 14:22:51 host2.intelius.com xinetd[3258]: Reading included configuration file: /etc/xinetd.d/chargen [file=/etc/xinetd.d/chargen] [line=12]
Apr 24 14:22:50 host2.intelius.com kernel: Processor #6 6:15 APIC version 16 Apr 24 14:22:50 host2.intelius.com kernel: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) Apr 24 14:22:51 host2.intelius.com xinetd[3258]: Reading included configuration file: /etc/xinetd.d/chargen-udp [file=/etc/xinetd.d/chargen-udp] [line=18]
Apr 24 14:22:50 host2.intelius.com kernel: ACPI: LAPIC (acpi_id[0x01] lapic_id[0x01] enabled) Apr 24 14:22:50 host2.intelius.com kernel: SELinux: Registering netfilter hooks Apr 24 14:22:51 host2.intelius.com xinetd[3258]: Reading included configuration file: /etc/xinetd.d/cups-lpd [file=/etc/xinetd.d/cups-lpd] [line=17]
Apr 24 14:22:50 host2.intelius.com kernel: Processor #1 6:15 APIC version 16 Apr 24 14:22:50 host2.intelius.com kernel: Initializing Cryptographic API Apr 24 14:22:51 host2.intelius.com ntpd[3270]: ntpd [email protected] Thu Oct 5 04:11:33 EDT 2006 (1)
Apr 24 14:22:50 host2.intelius.com kernel: ACPI: LAPIC (acpi_id[0x05] lapic_id[0x05] enabled) Apr 24 14:22:50 host2.intelius.com kernel: ksign: Installing public key data Apr 24 14:22:51 host2.intelius.com ntpd[3270]: precision = 1.000 usec
Apr 24 14:22:50 host2.intelius.com kernel: Processor #5 6:15 APIC version 16 Apr 24 14:22:50 host2.intelius.com kernel: Loading keyring Apr 24 14:22:51 host2.intelius.com ntpd[3270]: Listening on interface wildcard, 0.0.0.0#123
Apr 24 14:22:50 host2.intelius.com kernel: ACPI: LAPIC (acpi_id[0x03] lapic_id[0x03] enabled) Apr 24 14:22:50 host2.intelius.com kernel: - Added public key 666031A17F0A96D8 Apr 24 14:22:51 host2.intelius.com ntpd: ntpd startup succeeded
Apr 24 14:22:50 host2.intelius.com kernel: Processor #3 6:15 APIC version 16 Apr 24 14:22:50 host2.intelius.com kernel: - User ID: Red Hat, Inc. (Kernel Module GPG key) Apr 24 14:22:51 host2.intelius.com ntpd[3270]: Listening on interface lo, 127.0.0.1#123
Apr 24 14:22:50 host2.intelius.com kernel: ACPI: LAPIC (acpi_id[0x07] lapic_id[0x07] enabled) Apr 24 14:22:50 host2.intelius.com kernel: pci_hotplug: PCI Hot Plug PCI Core version: 0.5 Apr 24 14:22:51 host2.intelius.com ntpd[3270]: Listening on interface eth0, 10.192.165.6#123
Apr 24 14:22:50 host2.intelius.com kernel: Processor #7 6:15 APIC version 16 Apr 24 14:22:50 host2.intelius.com kernel: ACPI: Processor [CPU0] (supports C1) Apr 24 14:22:51 host2.intelius.com ntpd[3270]: kernel time sync status 0040
Apr 24 14:22:50 host2.intelius.com kernel: ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) Apr 24 14:22:50 host2.intelius.com kernel: ACPI: Processor [CPU1] (supports C1) Apr 24 14:22:51 host2.intelius.com xinetd[3258]: Reading included configuration file: /etc/xinetd.d/daytime [file=/etc/xinetd.d/daytime] [line=11]
Apr 24 14:22:50 host2.intelius.com kernel: Setting APIC routing to flat Apr 24 14:22:50 host2.intelius.com kernel: ACPI: Processor [CPU2] (supports C1) Apr 24 14:22:51 host2.intelius.com snmpd[3231]: NET-SNMP version 5.1.2
Apr 24 14:22:50 host2.intelius.com kernel: ACPI: IOAPIC (id[0x08] address[0xfec00000] gsi_base[0]) Apr 24 14:22:50 host2.intelius.com kernel: ACPI: Processor [CPU3] (supports C1) Apr 24 14:22:51 host2.intelius.com xinetd[3258]: Reading included configuration file: /etc/xinetd.d/daytime-udp [file=/etc/xinetd.d/daytime-udp] [line=15]
Apr 24 14:22:50 host2.intelius.com kernel: IOAPIC[0]: apic_id 8, version 32, address 0xfec00000, GSI 0-23 Apr 24 14:22:50 host2.intelius.com kernel: ACPI: Processor [CPU4] (supports C1) Apr 24 14:22:51 host2.intelius.com xinetd[3258]: Reading included configuration file: /etc/xinetd.d/echo [file=/etc/xinetd.d/echo] [line=15]
Apr 24 14:22:50 host2.intelius.com kernel: ACPI: IOAPIC (id[0x09] address[0xfec80000] gsi_base[24]) Apr 24 14:22:50 host2.intelius.com kernel: ACPI: Processor [CPU5] (supports C1) Apr 24 14:22:51 host2.intelius.com xinetd[3258]: Reading included configuration file: /etc/xinetd.d/echo-udp [file=/etc/xinetd.d/echo-udp] [line=14]
Apr 24 14:22:50 host2.intelius.com kernel: IOAPIC[1]: apic_id 9, version 32, address 0xfec80000, GSI 24-47 Apr 24 14:22:50 host2.intelius.com kernel: ACPI: Processor [CPU6] (supports C1) Apr 24 14:22:51 host2.intelius.com xinetd[3258]: Reading included configuration file: /etc/xinetd.d/eklogin [file=/etc/xinetd.d/eklogin] [line=15]
Apr 24 14:22:50 host2.intelius.com kernel: ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 high edge) Apr 24 14:22:50 host2.intelius.com kernel: ACPI: Processor [CPU7] (supports C1) Apr 24 14:22:51 host2.intelius.com xinetd[3258]: Reading included configuration file: /etc/xinetd.d/gssftp [file=/etc/xinetd.d/gssftp] [line=13]
Apr 24 14:22:50 host2.intelius.com kernel: ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) Apr 24 14:22:50 host2.intelius.com kernel: ACPI: Thermal Zone [THM0] (8 C) Apr 24 14:22:51 host2.intelius.com ntpd[3270]: frequency initialized 151.504 PPM from /var/lib/ntp/drift
Apr 24 14:22:50 host2.intelius.com kernel: ACPI: IRQ0 used by override. Apr 24 14:22:50 host2.intelius.com kernel: Real Time Clock Driver v1.12 Apr 24 14:22:51 host2.intelius.com xinetd[3258]: Reading included configuration file: /etc/xinetd.d/klogin [file=/etc/xinetd.d/klogin] [line=14]
Apr 24 14:22:50 host2.intelius.com kernel: ACPI: IRQ2 used by override. Apr 24 14:22:50 host2.intelius.com kernel: Linux agpgart interface v0.100 (c) Dave Jones Apr 24 14:22:51 host2.intelius.com xinetd[3258]: Reading included configuration file: /etc/xinetd.d/krb5-telnet [file=/etc/xinetd.d/krb5-telnet] [line=13]
Apr 24 14:22:50 host2.intelius.com kernel: ACPI: IRQ9 used by override. Apr 24 14:22:50 host2.intelius.com kernel: serio: i8042 AUX port at 0x60,0x64 irq 12 Apr 24 14:22:51 host2.intelius.com xinetd[3258]: Reading included configuration file: /etc/xinetd.d/kshell [file=/etc/xinetd.d/kshell] [line=13]
Apr 24 14:22:50 host2.intelius.com kernel: ACPI: HPET id: 0x8086a201 base: 0xfed00000 Apr 24 14:22:50 host2.intelius.com kernel: serio: i8042 KBD port at 0x60,0x64 irq 1 Apr 24 14:22:51 host2.intelius.com xinetd[3258]: Reading included configuration file: /etc/xinetd.d/nrpe [file=/etc/xinetd.d/nrpe] [line=13]
Apr 24 14:22:50 host2.intelius.com kernel: Using ACPI (MADT) for SMP configuration information Apr 24 14:22:50 host2.intelius.com kernel: Serial: 8250/16550 driver $Revision: 1.90 $ 68 ports, IRQ sharing enabled Apr 24 14:22:51 host2.intelius.com xinetd[3258]: Reading included configuration file: /etc/xinetd.d/rsync [file=/etc/xinetd.d/rsync] [line=17]
Apr 24 14:22:50 host2.intelius.com kernel: Allocating PCI resources starting at d1000000 (gap: d0000000:10000000) Apr 24 14:22:50 host2.intelius.com kernel: ttyS0 at I/O 0x3f8 (irq = 4) is a 16550A Apr 24 14:22:51 host2.intelius.com xinetd[3258]: Reading included configuration file: /etc/xinetd.d/time [file=/etc/xinetd.d/time] [line=13]
Apr 24 14:22:50 host2.intelius.com kernel: Checking aperture... Apr 24 14:22:50 host2.intelius.com kernel: ttyS1 at I/O 0x2f8 (irq = 3) is a 16550A Apr 24 14:22:51 host2.intelius.com xinetd[3258]: Reading included configuration file: /etc/xinetd.d/time-udp [file=/etc/xinetd.d/time-udp] [line=17]
Apr 24 14:22:50 host2.intelius.com kernel: Built 1 zonelists Apr 24 14:22:50 host2.intelius.com kernel: RAMDISK driver initialized: 16 RAM disks of 16384K size 1024 blocksize Apr 24 14:22:51 host2.intelius.com xinetd[3258]: Reading included configuration file: /etc/xinetd.d/vnetd [file=/etc/xinetd.d/vnetd] [line=19]
Apr 24 14:22:50 host2.intelius.com kernel: Kernel command line: ro root=LABEL=/ console=tty0 console=ttyS1 Apr 24 14:22:50 host2.intelius.com kernel: divert: not allocating divert_blk for non-ethernet device lo Apr 24 14:22:51 host2.intelius.com xinetd[3258]: Reading included configuration file: /etc/xinetd.d/vopied [file=/etc/xinetd.d/vopied] [line=13]
Apr 24 14:22:50 host2.intelius.com kernel: Initializing CPU#0 Apr 24 14:22:50 host2.intelius.com kernel: Uniform Multi-Platform E-IDE driver Revision: 7.00alpha2 Apr 24 14:22:51 host2.intelius.com xinetd[3258]: removing chargen
Apr 24 14:22:50 host2.intelius.com kernel: PID hash table entries: 4096 (order: 12, 131072 bytes) Apr 24 14:22:50 host2.intelius.com kernel: ide: Assuming 33MHz system bus speed for PIO modes; override with idebus=xx Apr 24 14:22:51 host2.intelius.com xinetd[3258]: removing chargen
Apr 24 14:22:50 host2.intelius.com kernel: time.c: Using 14.318180 MHz HPET timer. Apr 24 14:22:50 host2.intelius.com kernel: ESB2: IDE controller at PCI slot 0000:00:1f.1 Apr 24 14:22:51 host2.intelius.com xinetd[3258]: removing printer
Apr 24 14:22:50 host2.intelius.com kernel: time.c: Detected 2333.422 MHz processor. Apr 24 14:22:50 host2.intelius.com kernel: ACPI: PCI Interrupt 0000:00:1f.1[A] -> GSI 17 (level, low) -> IRQ 177 Apr 24 14:22:51 host2.intelius.com xinetd[3258]: removing daytime
Apr 24 14:22:50 host2.intelius.com kernel: Console: colour VGA+ 80x25 Apr 24 14:22:50 host2.intelius.com kernel: ESB2: chipset revision 9 Apr 24 14:22:51 host2.intelius.com xinetd[3258]: removing daytime
Apr 24 14:22:50 host2.intelius.com kernel: Dentry cache hash table entries: 4194304 (order: 13, 33554432 bytes) Apr 24 14:22:50 host2.intelius.com kernel: ESB2: not 100% native mode: will probe irqs later Apr 24 14:22:51 host2.intelius.com xinetd[3258]: removing echo
Apr 24 14:22:50 host2.intelius.com kernel: Inode-cache hash table entries: 2097152 (order: 12, 16777216 bytes) Apr 24 14:22:50 host2.intelius.com kernel: ide0: BM-DMA at 0x0500-0x0507, BIOS settings: hda:DMA, hdb:pio Apr 24 14:22:51 host2.intelius.com xinetd[3258]: removing echo
Apr 24 14:22:50 host2.intelius.com kernel: Placing software IO TLB between 0x4163000 - 0x8163000 Apr 24 14:22:50 host2.intelius.com kernel: ide1: BM-DMA at 0x0508-0x050f, BIOS settings: hdc:pio, hdd:pio Apr 24 14:22:51 host2.intelius.com xinetd[3258]: removing eklogin
Apr 24 14:22:50 host2.intelius.com kernel: Memory: 28823540k/30146556k available (2126k kernel code, 0k reserved, 1313k data, 212k init) Apr 24 14:22:50 host2.intelius.com kernel: Probing IDE interface ide0... Apr 24 14:22:51 host2.intelius.com xinetd[3258]: removing ftp
Apr 24 14:22:50 host2.intelius.com kernel: Calibrating delay using timer specific routine.. 4670.16 BogoMIPS (lpj=2335080) Apr 24 14:22:50 host2.intelius.com kernel: hda: DW-224E-V, ATAPI CD/DVD-ROM drive Apr 24 14:22:51 host2.intelius.com xinetd[3258]: removing klogin
Apr 24 14:22:50 host2.intelius.com kernel: Security Scaffold v1.0.0 initialized Apr 24 14:22:50 host2.intelius.com kernel: Using cfq io scheduler Apr 24 14:22:51 host2.intelius.com xinetd[3258]: removing telnet
Apr 24 14:22:50 host2.intelius.com kernel: SELinux: Initializing. Apr 24 14:22:50 host2.intelius.com kernel: ide0 at 0x1f0-0x1f7,0x3f6 on irq 14 Apr 24 14:22:51 host2.intelius.com xinetd[3258]: removing kshell
Apr 24 14:22:50 host2.intelius.com kernel: SELinux: Starting in permissive mode Apr 24 14:22:50 host2.intelius.com kernel: Probing IDE interface ide1... Apr 24 14:22:51 host2.intelius.com xinetd[3258]: removing rsync
Apr 24 14:22:50 host2.intelius.com irqbalance: irqbalance startup succeeded Apr 24 14:22:50 host2.intelius.com kernel: Probing IDE interface ide1... Apr 24 14:22:51 host2.intelius.com xinetd[3258]: removing time
Apr 24 14:22:50 host2.intelius.com kernel: There is already a security framework initialized, register_security failed. Apr 24 14:22:50 host2.intelius.com kernel: Probing IDE interface ide2... Apr 24 14:22:51 host2.intelius.com xinetd[3258]: removing time
Apr 24 14:22:50 host2.intelius.com kernel: selinux_register_security: Registering secondary module capability Apr 24 14:22:50 host2.intelius.com kernel: Probing IDE interface ide3... Apr 24 14:22:51 host2.intelius.com xinetd[3258]: xinetd Version 2.3.13 started with libwrap loadavg options compiled in.
Apr 24 14:22:50 host2.intelius.com kernel: Capability LSM initialized as secondary Apr 24 14:22:50 host2.intelius.com kernel: Probing IDE interface ide4... Apr 24 14:22:51 host2.intelius.com xinetd[3258]: Started working: 5 available services
Apr 24 14:22:50 host2.intelius.com kernel: Mount-cache hash table entries: 256 (order: 0, 4096 bytes) Apr 24 14:22:50 host2.intelius.com kernel: Probing IDE interface ide5... Apr 24 14:22:52 host2.intelius.com sendmail[3285]: alias database /etc/aliases rebuilt by root
Apr 24 14:22:50 host2.intelius.com kernel: CPU: L1 I cache: 32K, L1 D cache: 32K Apr 24 14:22:50 host2.intelius.com kernel: hda: ATAPI 24X DVD-ROM CD-R/RW drive, 1654kB Cache, UDMA(33) Apr 24 14:22:52 host2.intelius.com sendmail[3285]: /etc/aliases: 68 aliases, longest 84 bytes, 856 bytes total
Apr 24 14:22:50 host2.intelius.com kernel: CPU: L2 cache: 4096K Apr 24 14:22:50 host2.intelius.com kernel: Uniform CD-ROM driver Revision: 3.20 Apr 24 14:22:52 host2.intelius.com sendmail[3294]: starting daemon (8.13.1): SMTP+queueing@01:00:00
Apr 24 14:22:50 host2.intelius.com kernel: using mwait in idle threads. Apr 24 14:22:50 host2.intelius.com kernel: ide-floppy driver 0.99.newide Apr 24 14:22:52 host2.intelius.com sendmail: sendmail startup succeeded
Apr 24 14:22:50 host2.intelius.com kernel: CPU0: Physical Processor ID: 0 Apr 24 14:22:50 host2.intelius.com kernel: usbcore: registered new driver hiddev Apr 24 14:22:52 host2.intelius.com sm-msp-queue[3306]: starting daemon (8.13.1): queueing@01:00:00
Apr 24 14:22:50 host2.intelius.com kernel: CPU0: Processor Core ID: 0 Apr 24 14:22:50 host2.intelius.com kernel: usbcore: registered new driver usbhid Apr 24 14:22:52 host2.intelius.com sendmail: sm-client startup succeeded
Apr 24 14:22:50 host2.intelius.com kernel: CPU0: Initial APIC ID: 0 Apr 24 14:22:50 host2.intelius.com kernel: drivers/usb/input/hid-core.c: v2.0:USB HID core driver Apr 24 14:22:52 host2.intelius.com gpm[3340]: *** info [startup.c(95)]:
Apr 24 14:22:50 host2.intelius.com kernel: Using IO APIC NMI watchdog Apr 24 14:22:50 host2.intelius.com kernel: mice: PS/2 mouse device common for all mice Apr 24 14:22:52 host2.intelius.com gpm[3340]: Started gpm successfully. Entered daemon mode.
Apr 24 14:22:50 host2.intelius.com kernel: CPU: L1 I cache: 32K, L1 D cache: 32K Apr 24 14:22:50 host2.intelius.com kernel: input: AT Translated Set 2 keyboard on isa0060/serio0 Apr 24 14:22:52 host2.intelius.com gpm[3340]: *** info [mice.c(1766)]:
Apr 24 14:22:50 host2.intelius.com kernel: CPU: L2 cache: 4096K Apr 24 14:22:50 host2.intelius.com kernel: md: md driver 0.90.0 MAX_MD_DEVS=256, MD_SB_DISKS=27 Apr 24 14:22:52 host2.intelius.com gpm[3340]: imps2: Auto-detected intellimouse PS/2
Apr 24 14:22:50 host2.intelius.com kernel: CPU0: Physical Processor ID: 0 Apr 24 14:22:50 host2.intelius.com kernel: NET: Registered protocol family 2 Apr 24 14:22:52 host2.intelius.com rc.sysinit: -e
Apr 24 14:22:50 host2.intelius.com kernel: CPU0: Processor Core ID: 0 Apr 24 14:22:50 host2.intelius.com kernel: IP route cache hash table entries: 524288 (order: 10, 4194304 bytes) Apr 24 14:22:52 host2.intelius.com date: Thu Apr 24 14:22:33 PDT 2008
Apr 24 14:22:50 host2.intelius.com kernel: CPU0: Initial APIC ID: 0 Apr 24 14:22:50 host2.intelius.com kernel: TCP established hash table entries: 262144 (order: 10, 4194304 bytes) Apr 24 14:22:52 host2.intelius.com rc.sysinit: Setting clock (localtime): Thu Apr 24 14:22:33 PDT 2008 succeeded
Apr 24 14:22:50 host2.intelius.com kernel: CPU0: Intel(R) Xeon(R) CPU E5345 @ 2.33GHz stepping 0b Apr 24 14:22:50 host2.intelius.com kernel: TCP bind hash table entries: 262144 (order: 10, 4194304 bytes) Apr 24 14:22:52 host2.intelius.com start_udev: Starting udev: succeeded
Apr 24 14:22:50 host2.intelius.com kernel: per-CPU timeslice cutoff: 4095.42 usecs. Apr 24 14:22:50 host2.intelius.com kernel: TCP: Hash tables configured (established 262144 bind 262144) Apr 24 14:22:52 host2.intelius.com udevsend[1663]: starting udevd daemon
Apr 24 14:22:50 host2.intelius.com kernel: task migration cache decay timeout: 4 msecs. Apr 24 14:22:50 host2.intelius.com kernel: Initializing IPsec netlink socket Apr 24 14:22:52 host2.intelius.com rc.sysinit: -e
Apr 24 14:22:50 host2.intelius.com kernel: Booting processor 1/4 rip 6000 rsp 100081f5f58 Apr 24 14:22:50 host2.intelius.com kernel: NET: Registered protocol family 1 Apr 24 14:22:52 host2.intelius.com sysctl: net.ipv4.ip_forward = 0
Apr 24 14:22:50 host2.intelius.com kernel: Initializing CPU#1 Apr 24 14:22:50 host2.intelius.com kernel: NET: Registered protocol family 17 Apr 24 14:22:52 host2.intelius.com sysctl: net.ipv4.conf.default.rp_filter = 1
Apr 24 14:22:50 host2.intelius.com kernel: Calibrating delay using timer specific routine.. 4666.88 BogoMIPS (lpj=2333443) Apr 24 14:22:50 host2.intelius.com kernel: ACPI wakeup devices: Apr 24 14:22:52 host2.intelius.com sysctl: net.ipv4.conf.default.accept_source_route = 0
Apr 24 14:22:50 host2.intelius.com kernel: CPU: L1 I cache: 32K, L1 D cache: 32K Apr 24 14:22:50 host2.intelius.com kernel: Apr 24 14:22:52 host2.intelius.com sysctl: kernel.sysrq = 0
Apr 24 14:22:50 host2.intelius.com kernel: CPU: L2 cache: 4096K Apr 24 14:22:50 host2.intelius.com kernel: ACPI: (supports S0 S4 S5) Apr 24 14:22:52 host2.intelius.com sysctl: kernel.core_uses_pid = 1
Apr 24 14:22:50 host2.intelius.com kernel: CPU1: Physical Processor ID: 1 Apr 24 14:22:50 host2.intelius.com kernel: Freeing unused kernel memory: 212k freed Apr 24 14:22:52 host2.intelius.com sysctl: net.core.somaxconn = 512
Apr 24 14:22:50 host2.intelius.com kernel: CPU1: Processor Core ID: 4 Apr 24 14:22:50 host2.intelius.com kernel: SCSI subsystem initialized Apr 24 14:22:52 host2.intelius.com sysctl: net.core.optmem_max = 20480
Apr 24 14:22:50 host2.intelius.com kernel: CPU1: Initial APIC ID: 4 Apr 24 14:22:50 host2.intelius.com kernel: HP CISS Driver (v 2.6.16.RH1) Apr 24 14:22:52 host2.intelius.com sysctl: net.core.netdev_max_backlog = 1024
Apr 24 14:22:50 host2.intelius.com kernel: Intel(R) Xeon(R) CPU E5345 @ 2.33GHz stepping 0b Apr 24 14:22:50 host2.intelius.com kernel: cciss: Device 0x3230 has been found at bus 6 dev 0 func 0 Apr 24 14:22:52 host2.intelius.com sysctl: net.core.dev_weight = 64
Apr 24 14:22:50 host2.intelius.com kernel: Booting processor 2/2 rip 6000 rsp 10716d2df58 Apr 24 14:22:50 host2.intelius.com kernel: ACPI: PCI Interrupt 0000:06:00.0[A] -> GSI 16 (level, low) -> IRQ 169 Apr 24 14:22:52 host2.intelius.com gpm: gpm startup succeeded
Apr 24 14:22:50 host2.intelius.com kernel: Initializing CPU#2 Apr 24 14:22:50 host2.intelius.com kernel: MSI INIT SUCCESS Apr 24 14:22:52 host2.intelius.com sysctl: net.core.rmem_default = 262141
Apr 24 14:22:50 host2.intelius.com kernel: Calibrating delay using timer specific routine.. 4666.87 BogoMIPS (lpj=2333438) Apr 24 14:22:50 host2.intelius.com kernel: cciss: using DAC cycles Apr 24 14:22:52 host2.intelius.com sysctl: net.core.wmem_default = 262141
Apr 24 14:22:50 host2.intelius.com kernel: CPU: L1 I cache: 32K, L1 D cache: 32K Apr 24 14:22:50 host2.intelius.com kernel: blocks= 286677119 block_size= 512 Apr 24 14:22:52 host2.intelius.com sysctl: net.core.rmem_max = 262141
Apr 24 14:22:50 host2.intelius.com kernel: CPU: L2 cache: 4096K Apr 24 14:22:50 host2.intelius.com kernel: heads= 255, sectors= 32, cylinders= 35132 Apr 24 14:22:52 host2.intelius.com sysctl: net.core.wmem_max = 262141
Apr 24 14:22:50 host2.intelius.com kernel: CPU2: Physical Processor ID: 0 Apr 24 14:22:50 host2.intelius.com kernel: Apr 24 14:22:52 host2.intelius.com sysctl: fs.file-max = 209708
Apr 24 14:22:50 host2.intelius.com kernel: CPU2: Processor Core ID: 2 Apr 24 14:22:50 host2.intelius.com kernel: blocks= 286677119 block_size= 512 Apr 24 14:22:52 host2.intelius.com rc.sysinit: Configuring kernel parameters: succeeded
Apr 24 14:22:50 host2.intelius.com kernel: CPU2: Initial APIC ID: 2 Apr 24 14:22:50 host2.intelius.com kernel: heads= 255, sectors= 32, cylinders= 35132 Apr 24 14:22:52 host2.intelius.com rc.sysinit: Loading default keymap succeeded
Apr 24 14:22:50 host2.intelius.com kernel: Intel(R) Xeon(R) CPU E5345 @ 2.33GHz stepping 0b Apr 24 14:22:50 host2.intelius.com kernel: Apr 24 14:22:52 host2.intelius.com rc.sysinit: Setting hostname host2.intelius.com: succeeded
Apr 24 14:22:50 host2.intelius.com kernel: Booting processor 3/6 rip 6000 rsp 100cfe23f58 Apr 24 14:22:50 host2.intelius.com kernel: cciss/c0d0: p1 p2 p3 p4 < p5 p6 > Apr 24 14:22:52 host2.intelius.com fsck: [/sbin/fsck.ext3 (1) -- /] fsck.ext3 -a /dev/cciss/c0d0p1
Apr 24 14:22:50 host2.intelius.com kernel: Initializing CPU#3 Apr 24 14:22:50 host2.intelius.com kernel: libata version 2.00 loaded. Apr 24 14:22:52 host2.intelius.com fsck: /: clean, 97515/919296 files, 849731/1835008 blocks
Apr 24 14:22:50 host2.intelius.com kernel: Calibrating delay using timer specific routine.. 4666.87 BogoMIPS (lpj=2333438) Apr 24 14:22:50 host2.intelius.com kernel: EXT3-fs: INFO: recovery required on readonly filesystem. Apr 24 14:22:52 host2.intelius.com rc.sysinit: Checking root filesystem succeeded
Apr 24 14:22:50 host2.intelius.com kernel: CPU: L1 I cache: 32K, L1 D cache: 32K Apr 24 14:22:50 host2.intelius.com kernel: EXT3-fs: write access will be enabled during recovery. Apr 24 14:22:52 host2.intelius.com rc.sysinit: Remounting root filesystem in read-write mode: succeeded
Apr 24 14:22:50 host2.intelius.com kernel: CPU: L2 cache: 4096K Apr 24 14:22:50 host2.intelius.com kernel: kjournald starting. Commit interval 5 seconds Apr 24 14:22:52 host2.intelius.com lvm.static: No volume groups found
Apr 24 14:22:50 host2.intelius.com kernel: CPU3: Physical Processor ID: 1 Apr 24 14:22:50 host2.intelius.com kernel: EXT3-fs: recovery complete. Apr 24 14:22:52 host2.intelius.com rc.sysinit: Setting up Logical Volume Management: succeeded
Apr 24 14:22:50 host2.intelius.com kernel: CPU3: Processor Core ID: 6 Apr 24 14:22:50 host2.intelius.com kernel: EXT3-fs: mounted filesystem with ordered data mode. Apr 24 14:22:52 host2.intelius.com fsck: Checking all file systems.
Apr 24 14:22:50 host2.intelius.com kernel: CPU3: Initial APIC ID: 6 Apr 24 14:22:50 host2.intelius.com kernel: SELinux: Disabled at runtime. Apr 24 14:22:52 host2.intelius.com fsck: [/sbin/fsck.ext3 (1) -- /tmp] fsck.ext3 -a /dev/cciss/c0d0p2
Apr 24 14:22:50 host2.intelius.com kernel: Intel(R) Xeon(R) CPU E5345 @ 2.33GHz stepping 0b Apr 24 14:22:50 host2.intelius.com kernel: SELinux: Unregistering netfilter hooks Apr 24 14:22:52 host2.intelius.com fsck: /tmp: recovering journal
Apr 24 14:22:50 host2.intelius.com kernel: Booting processor 4/1 rip 6000 rsp 10037e99f58 Apr 24 14:22:50 host2.intelius.com kernel: inserting floppy driver for 2.6.9-67.ELsmp Apr 24 14:22:52 host2.intelius.com crond[3416]: (CRON) STARTUP (V5.0)
Apr 24 14:22:50 host2.intelius.com kernel: Initializing CPU#4 Apr 24 14:22:50 host2.intelius.com kernel: Floppy drive(s): fd0 is 1.44M Apr 24 14:22:52 host2.intelius.com crond: crond startup succeeded
Apr 24 14:22:50 host2.intelius.com portmap: portmap startup succeeded Apr 24 14:22:50 host2.intelius.com kernel: floppy0: no floppy controllers found Apr 24 14:22:52 host2.intelius.com fsck: /tmp: clean, 691/2626560 files, 115429/5242880 blocks
Apr 24 14:22:50 host2.intelius.com kernel: Calibrating delay using timer specific routine.. 4666.83 BogoMIPS (lpj=2333419) Apr 24 14:22:50 host2.intelius.com kernel: Broadcom NetXtreme II Gigabit Ethernet Driver bnx2 v1.5.11-rh (June 4, 2007) Apr 24 14:22:52 host2.intelius.com fsck: [/sbin/fsck.ext3 (1) -- /usr/local] fsck.ext3 -a /dev/cciss/c0d0p6
Apr 24 14:22:50 host2.intelius.com kernel: CPU: L1 I cache: 32K, L1 D cache: 32K Apr 24 14:22:50 host2.intelius.com kernel: ACPI: PCI Interrupt 0000:03:00.0[A] -> GSI 18 (level, low) -> IRQ 185 Apr 24 14:22:52 host2.intelius.com fsck: /usr/local: recovering journal
Apr 24 14:22:50 host2.intelius.com kernel: CPU: L2 cache: 4096K Apr 24 14:22:50 host2.intelius.com kernel: divert: allocating divert_blk for eth0 Apr 24 14:22:52 host2.intelius.com fsck: /usr/local: clean, 21736/12943360 files, 532008/25870666 blocks
Apr 24 14:22:50 host2.intelius.com kernel: CPU4: Physical Processor ID: 0 Apr 24 14:22:50 host2.intelius.com kernel: eth0: Broadcom NetXtreme II BCM5708 1000Base-T (B2) PCI-X 64-bit 133MHz found at mem f8000000, IRQ 185, node addr 001cc4aba452 Apr 24 14:22:52 host2.intelius.com fsck: [/sbin/fsck.ext3 (1) -- /var] fsck.ext3 -a /dev/cciss/c0d0p3
Apr 24 14:22:50 host2.intelius.com kernel: CPU4: Processor Core ID: 1 Apr 24 14:22:50 host2.intelius.com kernel: ACPI: PCI Interrupt 0000:05:00.0[A] -> GSI 19 (level, low) -> IRQ 193 Apr 24 14:22:52 host2.intelius.com fsck: /var: recovering journal
Apr 24 14:22:50 host2.intelius.com kernel: CPU4: Initial APIC ID: 1 Apr 24 14:22:50 host2.intelius.com kernel: divert: allocating divert_blk for eth1 Apr 24 14:22:52 host2.intelius.com fsck: /var: clean, 669/919296 files, 81825/1835008 blocks
Apr 24 14:22:50 host2.intelius.com kernel: Intel(R) Xeon(R) CPU E5345 @ 2.33GHz stepping 0b Apr 24 14:22:50 host2.intelius.com kernel: eth1: Broadcom NetXtreme II BCM5708 1000Base-T (B2) PCI-X 64-bit 133MHz found at mem fa000000, IRQ 193, node addr 001cc4aba432 Apr 24 14:22:52 host2.intelius.com rc.sysinit: Checking filesystems succeeded
Apr 24 14:22:50 host2.intelius.com kernel: Booting processor 5/5 rip 6000 rsp 10037e11f58 Apr 24 14:22:50 host2.intelius.com kernel: hw_random: RNG not detected Apr 24 14:22:52 host2.intelius.com rc.sysinit: Mounting local filesystems: succeeded
Apr 24 14:22:50 host2.intelius.com kernel: Initializing CPU#5 Apr 24 14:22:50 host2.intelius.com kernel: USB Universal Host Controller Interface driver v2.2 Apr 24 14:22:52 host2.intelius.com rc.sysinit: Enabling local filesystem quotas: succeeded
Apr 24 14:22:50 host2.intelius.com kernel: Calibrating delay using timer specific routine.. 4666.87 BogoMIPS (lpj=2333436) Apr 24 14:22:50 host2.intelius.com kernel: ACPI: PCI Interrupt 0000:00:1d.0[A] -> GSI 16 (level, low) -> IRQ 169 Apr 24 14:22:52 host2.intelius.com rc.sysinit: Enabling swap space: succeeded
Apr 24 14:22:50 host2.intelius.com kernel: CPU: L1 I cache: 32K, L1 D cache: 32K Apr 24 14:22:50 host2.intelius.com kernel: uhci_hcd 0000:00:1d.0: UHCI Host Controller Apr 24 14:22:52 host2.intelius.com init: Entering runlevel: 3
Apr 24 14:22:50 host2.intelius.com kernel: CPU: L2 cache: 4096K Apr 24 14:22:50 host2.intelius.com kernel: PCI: Setting latency timer of device 0000:00:1d.0 to 64 Apr 24 14:22:52 host2.intelius.com microcode_ctl: microcode_ctl startup succeeded
Apr 24 14:22:50 host2.intelius.com kernel: CPU5: Physical Processor ID: 1 Apr 24 14:22:50 host2.intelius.com kernel: uhci_hcd 0000:00:1d.0: irq 169, io base 0000000000001000 Apr 24 14:22:52 host2.intelius.com sysstat: Calling the system activity data collector (sadc):
Apr 24 14:22:50 host2.intelius.com kernel: CPU5: Processor Core ID: 5 Apr 24 14:22:50 host2.intelius.com kernel: uhci_hcd 0000:00:1d.0: new USB bus registered, assigned bus number 1 Apr 24 14:22:52 host2.intelius.com sysstat:
Apr 24 14:22:50 host2.intelius.com kernel: CPU5: Initial APIC ID: 5 Apr 24 14:22:50 host2.intelius.com kernel: hub 1-0:1.0: USB hub found Apr 24 14:22:52 host2.intelius.com rc: Starting sysstat: succeeded
Apr 24 14:22:50 host2.intelius.com kernel: Intel(R) Xeon(R) CPU E5345 @ 2.33GHz stepping 0b Apr 24 14:22:50 host2.intelius.com kernel: hub 1-0:1.0: 2 ports detected Apr 24 14:22:52 host2.intelius.com rc: Starting openibd: succeeded
Apr 24 14:22:50 host2.intelius.com kernel: Booting processor 6/3 rip 6000 rsp 10037e47f58 Apr 24 14:22:50 host2.intelius.com kernel: ACPI: PCI Interrupt 0000:00:1d.1[B] -> GSI 17 (level, low) -> IRQ 177 Apr 24 14:22:52 host2.intelius.com anacron[3425]: Anacron 2.3 started on 2008-04-24
Apr 24 14:22:50 host2.intelius.com kernel: Initializing CPU#6 Apr 24 14:22:50 host2.intelius.com kernel: uhci_hcd 0000:00:1d.1: UHCI Host Controller Apr 24 14:22:52 host2.intelius.com anacron: anacron startup succeeded
Apr 24 14:22:50 host2.intelius.com kernel: Calibrating delay using timer specific routine.. 4666.85 BogoMIPS (lpj=2333429) Apr 24 14:22:50 host2.intelius.com kernel: PCI: Setting latency timer of device 0000:00:1d.1 to 64 Apr 24 14:22:52 host2.intelius.com iptables: succeeded
Apr 24 14:22:50 host2.intelius.com kernel: CPU: L1 I cache: 32K, L1 D cache: 32K Apr 24 14:22:50 host2.intelius.com kernel: uhci_hcd 0000:00:1d.1: irq 177, io base 0000000000001020 Apr 24 14:22:52 host2.intelius.com sysctl: net.ipv4.ip_forward = 0
Apr 24 14:22:50 host2.intelius.com kernel: CPU: L2 cache: 4096K Apr 24 14:22:50 host2.intelius.com kernel: uhci_hcd 0000:00:1d.1: new USB bus registered, assigned bus number 2 Apr 24 14:22:52 host2.intelius.com sysctl: net.ipv4.conf.default.rp_filter = 1
Apr 24 14:22:50 host2.intelius.com kernel: CPU6: Physical Processor ID: 0 Apr 24 14:22:50 host2.intelius.com kernel: hub 2-0:1.0: USB hub found Apr 24 14:22:52 host2.intelius.com sysctl: net.ipv4.conf.default.accept_source_route = 0
Apr 24 14:22:50 host2.intelius.com kernel: CPU6: Processor Core ID: 3 Apr 24 14:22:50 host2.intelius.com kernel: hub 2-0:1.0: 2 ports detected Apr 24 14:22:52 host2.intelius.com sysctl: kernel.sysrq = 0
Apr 24 14:22:50 host2.intelius.com kernel: CPU6: Initial APIC ID: 3 Apr 24 14:22:50 host2.intelius.com kernel: ACPI: PCI Interrupt 0000:00:1d.2[C] -> GSI 18 (level, low) -> IRQ 185 Apr 24 14:22:52 host2.intelius.com sysctl: kernel.core_uses_pid = 1
Apr 24 14:22:50 host2.intelius.com kernel: Intel(R) Xeon(R) CPU E5345 @ 2.33GHz stepping 0b Apr 24 14:22:50 host2.intelius.com kernel: uhci_hcd 0000:00:1d.2: UHCI Host Controller Apr 24 14:22:52 host2.intelius.com sysctl: net.core.somaxconn = 512
Apr 24 14:22:50 host2.intelius.com kernel: Booting processor 7/7 rip 6000 rsp 10037e7df58 Apr 24 14:22:50 host2.intelius.com kernel: PCI: Setting latency timer of device 0000:00:1d.2 to 64 Apr 24 14:22:52 host2.intelius.com sysctl: net.core.optmem_max = 20480
Apr 24 14:22:50 host2.intelius.com kernel: Initializing CPU#7 Apr 24 14:22:50 host2.intelius.com kernel: uhci_hcd 0000:00:1d.2: irq 185, io base 0000000000001040 Apr 24 14:22:52 host2.intelius.com sysctl: net.core.netdev_max_backlog = 1024
Apr 24 14:22:50 host2.intelius.com kernel: Calibrating delay using timer specific routine.. 4666.83 BogoMIPS (lpj=2333417) Apr 24 14:22:50 host2.intelius.com kernel: uhci_hcd 0000:00:1d.2: new USB bus registered, assigned bus number 3 Apr 24 14:22:52 host2.intelius.com sysctl: net.core.dev_weight = 64
Apr 24 14:22:50 host2.intelius.com kernel: CPU: L1 I cache: 32K, L1 D cache: 32K Apr 24 14:22:50 host2.intelius.com kernel: hub 3-0:1.0: USB hub found Apr 24 14:22:52 host2.intelius.com sysctl: net.core.rmem_default = 262141
Apr 24 14:22:50 host2.intelius.com kernel: CPU: L2 cache: 4096K Apr 24 14:22:50 host2.intelius.com kernel: hub 3-0:1.0: 2 ports detected Apr 24 14:22:52 host2.intelius.com sysctl: net.core.wmem_default = 262141
Apr 24 14:22:50 host2.intelius.com kernel: CPU7: Physical Processor ID: 1 Apr 24 14:22:50 host2.intelius.com kernel: ACPI: PCI Interrupt 0000:00:1d.3[D] -> GSI 19 (level, low) -> IRQ 193 Apr 24 14:22:52 host2.intelius.com sysctl: net.core.rmem_max = 262141
Apr 24 14:22:50 host2.intelius.com kernel: CPU7: Processor Core ID: 7 Apr 24 14:22:50 host2.intelius.com kernel: uhci_hcd 0000:00:1d.3: UHCI Host Controller Apr 24 14:22:52 host2.intelius.com sysctl: net.core.wmem_max = 262141
Apr 24 14:22:50 host2.intelius.com kernel: CPU7: Initial APIC ID: 7 Apr 24 14:22:50 host2.intelius.com kernel: PCI: Setting latency timer of device 0000:00:1d.3 to 64 Apr 24 14:22:52 host2.intelius.com sysctl: fs.file-max = 209708
Apr 24 14:22:50 host2.intelius.com kernel: Intel(R) Xeon(R) CPU E5345 @ 2.33GHz stepping 0b Apr 24 14:22:50 host2.intelius.com kernel: uhci_hcd 0000:00:1d.3: irq 193, io base 0000000000001060 Apr 24 14:22:52 host2.intelius.com network: Setting network parameters: succeeded
Apr 24 14:22:50 host2.intelius.com rpc.statd[3107]: Version 1.0.6 Starting Apr 24 14:22:50 host2.intelius.com kernel: uhci_hcd 0000:00:1d.3: new USB bus registered, assigned bus number 4 Apr 24 14:22:52 host2.intelius.com network: Bringing up loopback interface: succeeded
Apr 24 14:22:50 host2.intelius.com kernel: Total of 8 processors activated (37338.20 BogoMIPS). Apr 24 14:22:50 host2.intelius.com kernel: hub 4-0:1.0: USB hub found Apr 24 14:22:52 host2.intelius.com network: Bringing up interface eth0: succeeded
Apr 24 14:22:50 host2.intelius.com kernel: activating NMI Watchdog ... done. Apr 24 14:22:50 host2.intelius.com kernel: hub 4-0:1.0: 2 ports detected Apr 24 14:22:52 host2.intelius.com atd: atd startup succeeded
Apr 24 14:22:50 host2.intelius.com kernel: testing NMI watchdog ... OK. Apr 24 14:22:50 host2.intelius.com kernel: ACPI: PCI Interrupt 0000:01:04.4[B] -> GSI 22 (level, low) -> IRQ 217 Apr 24 14:22:52 host2.intelius.com anacron[3425]: Normal exit (0 jobs run)
Apr 24 14:22:50 host2.intelius.com kernel: Using local APIC timer interrupts. Apr 24 14:22:50 host2.intelius.com kernel: uhci_hcd 0000:01:04.4: UHCI Host Controller Apr 24 14:22:52 host2.intelius.com cfenvd[3441]: cfenvd: starting
Apr 24 14:22:50 host2.intelius.com kernel: Detected 20.834 MHz APIC timer. Apr 24 14:22:50 host2.intelius.com kernel: uhci_hcd 0000:01:04.4: irq 217, io base 0000000000003800 Apr 24 14:22:52 host2.intelius.com cfenvd: cfenvd startup succeeded
Apr 24 14:22:50 host2.intelius.com kernel: checking TSC synchronization across 8 CPUs: passed. Apr 24 14:22:50 host2.intelius.com kernel: uhci_hcd 0000:01:04.4: new USB bus registered, assigned bus number 5 Apr 24 14:22:53 host2.intelius.com cfexecd[3453]: cfexecd starting Thu Apr 24 14:22:53 2008
Apr 24 14:22:50 host2.intelius.com kernel: Brought up 8 CPUs Apr 24 14:22:50 host2.intelius.com kernel: uhci_hcd 0000:01:04.4: port count misdetected? forcing to 2 ports Apr 24 14:22:53 host2.intelius.com cfexecd[3572]: Lock lock..host2.cfexecd.execd_12 expired (after 30430/0 minutes)
Apr 24 14:22:50 host2.intelius.com kernel: time.c: Using HPET/TSC based timekeeping. Apr 24 14:22:50 host2.intelius.com kernel: hub 5-0:1.0: USB hub found Apr 24 14:22:53 host2.intelius.com cfexecd: cfexecd startup succeeded
Apr 24 14:22:50 host2.intelius.com kernel: checking if image is initramfs... it is Apr 24 14:22:50 host2.intelius.com kernel: hub 5-0:1.0: 2 ports detected Apr 24 14:26:04 host2.intelius.com ntpd[3270]: synchronized to 140.142.1.8, stra
Apr 24 14:22:50 host2.intelius.com kernel: NET: Registered protocol family 16 Apr 24 14:22:50 host2.intelius.com kernel: ACPI: PCI Interrupt 0000:00:1d.7[A] -> GSI 16 (level, low) -> IRQ 169
Apr 24 14:22:50 host2.intelius.com kernel: PCI: Using configuration type 1 Apr 24 14:22:50 host2.intelius.com kernel: ehci_hcd 0000:00:1d.7: EHCI Host Controller
Apr 24 14:22:50 host2.intelius.com kernel: PCI: Using MMCONFIG at e0000000 Apr 24 14:22:50 host2.intelius.com kernel: PCI: Setting latency timer of device 0000:00:1d.7 to 64
Apr 24 14:22:50 host2.intelius.com kernel: mtrr: v2.0 (20020519) Apr 24 14:22:50 host2.intelius.com kernel: ehci_hcd 0000:00:1d.7: irq 169, pci mem ffffff000001a000
Apr 24 14:22:50 host2.intelius.com kernel: ACPI: Subsystem revision 20040816
Apr 24 14:22:50 host2.intelius.com kernel: ACPI: Interpreter enabled
Apr 24 14:22:50 host2.intelius.com kernel: ACPI: Using IOAPIC for interrupt routing
Apr 24 14:22:50 host2.intelius.com kernel: ACPI: PCI Root Bridge [PCI0] (00:00)
Apr 24 14:22:50 host2.intelius.com kernel: PCI: Probing PCI hardware (bus 00)

47
SEC - Configuration Example: main
Correlation of boot logs
– Into this
Apr 24 14:22:50 host2.intelius.com starting up...

:-)

48
SEC - Configuration Example: main
Example Sendmail correlation
– Sendmail events are split across multiple messages
type=single
desc=Save from address
ptype=regexp
pattern=\w+\s+\d+\s+\d+:\d+:\d+ ([\w.-]+) (sendmail|.+sm-mta)\[\d+\]: (\w+): from=
(\S+),
action=create MAIL_$1_$3 360; fill MAIL_$1_$3 $4

– This rule grabs the sender address from one log

message
• Creates a context named after host and message
ID, puts sender address into it

49
SEC - Configuration Example: main
Example Sendmail correlation
type=single
desc=Mail receiver unknown
ptype=regexp
pattern=(\w+\s+\d+\s+\d+:\d+:\d+) ([\w.-]+) (sendmail|.+sm-mta)\[\d+\]: (\w+): to=
(\S+),.+(relay=[^,]+),.+stat=(User unknown|Service unavailable)
action=copy MAIL_$2_$4 %from;\
event 0 PARSED:$1 $2 sendmail: User $5 unknown from %from ($6)
context=MAIL_$2_$4

– This is one of several possible followup rules,

depending on how the SMTP transaction goes
• Enabled when context from prior rule is in effect
• Copies the sender address from the context into a
variable, uses it to construct a single correlated log
message

50
SEC - Configuration Example: main
Example Sendmail correlation
– Along with some suppression rules, they turn this
Apr 25 04:59:46 host5.intelius.com sendmail[14779]: m3PBxkhp014779: Authentication-
Warning: host5.intelius.com: apache set sender to [email protected] using -f
Apr 25 04:59:46 host5.intelius.com sendmail[14779]: m3PBxkhp014779:
[email protected], size=738, class=0, nrcpts=1,
msgid=<[email protected]>, relay=apache@localhost
Apr 25 04:59:46 host5.intelius.com sendmail[14782]: m3PBxkHF014782:
from=<[email protected]>, size=1076, class=0, nrcpts=1,
msgid=<[email protected]>, proto=ESMTP, daemon=MTA,
relay=localhost.localdomain [127.0.0.1]
Apr 25 04:59:46 host5.intelius.com sendmail[14779]: m3PBxkhp014779:
[email protected], [email protected] (48/48), delay=00:00:00,
xdelay=00:00:00, mailer=relay, pri=30738, relay=[127.0.0.1] [127.0.0.1], dsn=2.0.0,
stat=Sent (m3PBxkHF014782 Message accepted for delivery)
Apr 25 04:59:53 host5.intelius.com sendmail[14784]: m3PBxkHF014782:
to=<[email protected]>, delay=00:00:07, xdelay=00:00:07, mailer=esmtp, pri=121076,
relay=mx2.yandex.ru. [213.180.223.88], dsn=5.1.1, stat=User unknown
Apr 25 04:59:53 host5.intelius.com sendmail[14784]: m3PBxkHF014782: m3PBxrHF014784:
DSN: User unknown

51
SEC - Configuration Example: main
Example Sendmail correlation
– Into this
Apr 25 04:59:53 host5.intelius.com sendmail: User <[email protected]> unknown from
<[email protected]> (relay=mx2.yandex.ru. [213.180.223.88])

52
SEC - Configuration Example: main
Example Sendmail correlation
– Remember that order of rules can make a difference
– For instance, these suppress rules appear after all
the correlations of mail message logs are complete
# Suppress this after reducing mail errors, otherwise we can miss second message
# of pair.
type=suppress
desc=Deferred email
ptype=regexp
pattern=(sendmail|.+sm-mta).+stat=Deferred

# Suppress this after reducing mail errors, otherwise we can miss second message
# of pair when there are multiple addressees and some are successful.
type=suppress
desc=Successful email
ptype=regexp
pattern=(sendmail|.+sm-mta).+msgid=

– If they appeared earlier, they could prevent

correlations from working
53
SEC - Configuration Example: main
Another Sendmail correlation: Load average
– When the load average on a host exceeds a
threshold, Sendmail stops processing connections
and logs the value of the load average
• That can be a lot of log messages
– This rule reduces logging volume by only reporting
load in factors of 10
# Replace last digit in load average with "0+", to cut down on number of msgs
type=single
desc=High load average
ptype=regexp
pattern=(\w+\s+\d+\s+\d+:\d+:\d+ [\w.-]+ sendmail).+rejecting connections on daemon
M[ST]A: (load average: \d+)\d
action=assign %loadavg $2; event 0 PARSED:$1: %{loadavg}0+

54
SEC - Configuration Example: main
Another Sendmail correlation: Load average
– Turns this
Apr 17 17:54:19 host3.intelius.com sendmail[2942]: rejecting connections on daemon MTA: load average: 48
Apr 17 17:54:34 host3.intelius.com sendmail[2942]: rejecting connections on daemon MTA: load average: 54
Apr 17 17:54:49 host3.intelius.com sendmail[2942]: rejecting connections on daemon MTA: load average: 58
Apr 17 17:55:04 host3.intelius.com sendmail[2942]: rejecting connections on daemon MTA: load average: 57
Apr 17 17:55:19 host3.intelius.com sendmail[2942]: rejecting connections on daemon MTA: load average: 57
Apr 17 17:55:34 host3.intelius.com sendmail[2942]: rejecting connections on daemon MTA: load average: 58
Apr 17 17:55:49 host3.intelius.com sendmail[2942]: rejecting connections on daemon MTA: load average: 54
Apr 17 17:56:04 host3.intelius.com sendmail[2942]: rejecting connections on daemon MTA: load average: 52
Apr 17 17:56:19 host3.intelius.com sendmail[2942]: rejecting connections on daemon MTA: load average: 51
Apr 17 17:56:34 host3.intelius.com sendmail[2942]: rejecting connections on daemon MTA: load average: 48

– Into this
Apr 17 17:54:19 host3.intelius.com sendmail: load average: 40+
Apr 17 17:54:34 host3.intelius.com sendmail: load average: 50+

55
SEC - Configuration Example: main
Detecting syslog-ng overflow
– syslog-ng logs statistics on messages it’s processed
• Via internal source, every 10 minutes by default
• Messages look like this
Apr 25 08:40:05 loghost.intelius.com syslog-ng[16671]: Log statistics;
dropped='program(`/usr/local/bin/secStart emerg`)=0', dropped='program(`/usr/local/
bin/secStart main`)=0', dropped='program(`/usr/local/bin/secStart nmi`)=0',
dropped='program(`/usr/local/bin/secStart outbound`)=0', processed='center(queued)
=2419827', processed='center(received)=1205703', processed='destination(d_emerg)
=3204', processed='destination(d_fac)=1205703', processed='destination(d_all)
=1205703', processed='destination(d_su)=13', processed='destination(d_nmi)=0',
processed='destination(d_outbound)=5204', processed='destination(d_int)=0',
processed='source(s_int)=0', processed='source(s_all)=1205703'

– Most of the time we don’t care to see these, but if

syslog-ng drops any messages, particularly to the
main SEC process, we want to know
56
SEC - Configuration Example: main
Detecting syslog-ng overflow
# Drop regular log stats reports unless messages get dropped. If that happens,
# send reduced message, but not too frequently, since this won't go away until
# syslog-ng is restarted.
type=suppress
ptype=regexp
pattern=loghost\.intelius\.com syslog-ng\[\d+\]: Log statistics\;.+dropped=\'program
\(\`\/usr\/local\/bin\/secStart main\`\)=0\'

type=singlewithsuppress
desc=Dropped $2 messages
ptype=regexp
pattern=(\w+\s+\d+\s+\d+:\d+:\d+ loghost\.intelius\.com syslog-ng)\[\d+\]: Log
statistics\;.+dropped=\'program\(\`\/usr\/local\/bin\/secStart main\`\)=(\d+)\'
action=event 0 PARSED:$1: dropped $2 messages
window=3600

57
SEC - Configuration Example: main
Email sent when overflow detected
type=single
desc=syslog-ng overwhelmed
continue=takenext
ptype=regexp
pattern=^UNDUPED:(\w+\s+\d+\s+\d+:\d+:\d+ log1\.tuk\.intelius\.com syslog-ng:
dropped \d+ messages)
action=pipe '$1' /bin/mail -s "SEC: syslog-ng message buffer overrun" %a

58
SEC - Configuration Example: main
Password expiration notification
– This rule sends email to a user when his or her
password is about to expire
# Window is set to a day, which basically means as long as SEC/syslog-ng go
# without restarting (and thus, resetting this correlation).
type=singlewithsuppress
desc=The user account "$2" on $1 $3. If you use this account, please log in and
change your password.
continue=takenext
ptype=regexp
pattern=^UNDUPED:\w+\s+\d+\s+\d+:\d+:\d+ ([\w.-]+) sshd: password for user (\w+)
(will expire in \d+ days)
action=pipe '%s' /usr/bin/mail -s "SEC: Your account on $1 $3" [email protected]
window=86400

59
SEC - Emailed Reports
sendLogs
– cron calls sendLogs to issue regular reports of
anomalous events
0 0,6-18 * * 1-5 /usr/local/bin/sendLogs
0 0,6,12,18 * * 0,6 /usr/local/bin/sendLogs
# Temporary holiday schedule
#0 0,6,12,18 * * * /usr/local/bin/sendLogs

– Hourly during work hours (6 AM - 6 PM weekdays),

every six hours otherwise

60
SEC - Emailed Reports
#!/bin/sh
#
# sendLogs - Email accumulated reduced logs to admins.
#

PATH=/bin:/usr/bin

LOG_DIR=/mnt0/syslog

for group in unix net drupal; do

! if [ -s ${LOG_DIR}/${group}.tmp ]; then
! ! logFile=$LOG_DIR/$group
! ! mv $logFile.tmp $logFile.$$
! ! cat $logFile.$$ | mail -s "SEC: Interesting $group logs
`date`" $group-log-report
! ! rm $logFile.$$
! fi
done
sendLogs 61
SEC - Emailed Reports
sendLogs
– Example email
Subject: SEC: Interesting unix logs Thu Apr 24 06:00:01 PDT 2008
Date: Thu, 24 Apr 2008 06:00:01 -0700
From: "loghost root" <[email protected]>
To: <[email protected]>

Apr 24 00:55:34 host1.intelius.com kernel: ide-cd: cmd 0x1e timed out

Apr 24 02:37:22 host2.intelius.com cmaeventd: Logical drive 2 of Array Controller in
slot 1: surface analysis consistency initialization completed.
Apr 24 02:46:18 loghost.intelius.com nrpe: Error: Could not complete SSL handshake.
5
Apr 24 02:50:15 host1.intelius.com kernel: ide-cd: cmd 0x1e timed out
Apr 24 03:17:46 loghost.intelius.com nrpe: Error: Could not complete SSL handshake.
5
Apr 24 04:05:57 host1.intelius.com kernel: ide-cd: cmd 0x1e timed out
Apr 24 04:39:29 host1.intelius.com kernel: ide-cd: cmd 0x1e timed out
Apr 24 04:45:01 host3.intelius.com sshd: Disallowed user root from 172.27.5.2

62
SEC - Conclusion
By far, the bulk of the setup work is creating the log filters
– The process is iterative
• Let logs through, figure out what you don’t care to
see, create filters to suppress or correlate
• Repeat until volume is bearable
• Learn your Perl-style regular expressions
Missing important log messages is bad
– But having so many to look at that you ignore them
can be just as bad

63
SEC - Conclusion
How effective is the log reduction and correlation at
highlighting anomalous events?
– Let’s look at how many messages make it to the
regular reports
Current volume is about 4.4 million messages per day (not
counting NetScreen traffic logs)
– Lately, an average of 300 messages per day (~13 per
hour) make it to the regular emailed reports
• Pretty stable over last year; down from ~26/hr 11/08,
~36/hr 5/08
• Reduced to about 0.007% of total
–99.993% of messages filtered or correlated
64
 SEC - Conclusion
Loghost Syslog Metrics
60,000

50,000

40,000
SEC-Processed Messages Per Day

10/3/08
SMTP RCPT floods
start
30,000
1/15/09-1/16/09
SMTP RCPT floods
are back

20,000
11/11/08
McColo shutdown

10,000

0
11/07! 1/08! 3/08! 5/08! 7/08! 9/08! 11/08! 1/09! 3/09! 5/09! 7/09! 9/09! 11/09! 1/10! 3/10

cron sshd dhcpd SEC unix SEC net SEC probe

SEC-Processed Messages Per Day 65

SEC - Conclusion
What is the drain on system resources imposed by SEC?
As stated earlier, current volume is ~4.4M msgs/day
– Each message is processed at least once by SEC,
often multiple times
– Many messages are held in memory due to contexts,
pairwithwindow rules, etc.
At current rate
– Smaller processes (disk, emerg, hitemp) each take
up about 8 MB of RAM and negligible CPU
– The main process uses ~14 MB RAM and 10% CPU

66
SEC - Conclusion
Juniper NetScreen firewall traffic logs are processed by a
specialized SEC config, bypassing the main config
– Only one singlewithsuppress rule that rewrites
the logs into a simpler format
– Current volume: 55-75 million msgs/day
– This SEC process uses nearly 200 MB and ~65% of a
CPU

67
Centralized Logging
with syslog-ng and
SEC
Leon Towns-von Stauber, Intelius
LinuxFest Northwest, April 2010
http://www.occam.com/