Advanced

 Linux  System  Administra3on  

Topic  9.  Logging  

Pablo  Abad  Fidalgo  

José Ángel Herrero Velasco
Departamento  de  Ingeniería  Informá2ca  y  Electrónica  

Este  tema  se  publica  bajo  Licencia:  

Crea2ve  Commons  BY-­‐NC-­‐SA  4.0  
Index  
•  Introduc,on.  

•  Event  gathering  system  (syslog).  

•  How  to  maintain  log  informa,on.  

•  How  to  use  log  informa,on.  

Introduc,on  
•  Kernel,  services,  apps  generate/send  events  constantly:  
–  Informa8on  about  normal  ac,vity.  
–  Informa8on  about  failures  and  other  anomalies.  
–  Failed  boo,ng  of  system  and  services.  
–  Access  informa8on  (security).  
•  Correct  management  of  this  informa8on  is  essen8al  to  
discover  and  solve  problems.  
•  The  events  from  all  services  have  a  common  manager:  
–  Event  collector  employed  by  kernel,  services  and  apps.  
–  In  UNIX,  a  service  named  “syslog”  (rsyslog,  syslog-­‐ng).  
–  Flexible,  easy,  safe  and  powerful.  
Index  
•  Introduc,on.  

•  Event  gathering  system  (syslog).  

•  How  to  maintain  log  informa,on.  

•  How  to  use  log  informa,on.  

Syslog   /var/log
cron

openlog
messages
syslogd
maillog

•  Syslog  structure:   cron shell script

–  syslogd:  logging  service.  The  

logger
boot.log

rest  of  services  and  apps  

communicate  with  syslogrd   /etc/rsyslogd.conf

to  send  messages  to  log  files.  

–  openlog:  libraries  to  use  this  service  from  another  service/app:  
•  Perl:  use  sys::syslog  (openlog(),  syslog().  
•  C:  openlog  lib.  
–  logger:  command  to  send  messages  to  the  log  file  from  a  shell.  
–  rsyslogd.conf:  configura8on  of  ac8ons  to  be  performed  according  to  
the  messages  sent  by  the  services.<  
 # Log all kernel messages to the console.

Syslog  
kern.* /dev/console

# Log anything (except mail) of level info or higher .

# dont log private authentication messages!
*.info;mail.none;authpriv.none /val/log/messages

•  rsyslogd.conf:   # Log cron stuff

cron.* /var/log/cron

–  One  line  per  ac8on,  with  the  

format:  facility.level        ac8on.  
–  Facility:  short  list  of  defined  
(by  the  kernel)  values:  
•  Kern,  user,  daemon  (other  service),  auth  (login,  su,  ssh…),  syslog,  mail,  lpr,  cron…  
–  No8fica8on  levels:  
•  emerg,  alert,  crit,  err,  warning,  no8ce,  info,  debug,  *  (all  levels).  
–  Ac8ons:  
•  file:  write  the  message  to  the  specified  file  (/var/log/messages,  /dev/console).  
•  @hostname/@IP:  send  the  message  to  the  syslogd  of  the  specified  host  (centraliza8on).  
•  user1,  user2:  send  the  message  to  users  user1  and  user2  if  logged  on.  
•  *:  send  the  message  to  every  user  logged  on.  
Syslog  
•  Special  Files  that  do  not  make  use  of  syslog:  
–  /var/log/wtmp:  contains,  in  binary  format,  user  loggings  and  
system  reboots:  
•  Employed  by  last  and  up8me.  
–  /var/log/lastlog:  contains  the  last  login  of  each  user.  
–  /var/log/dmesg:  boo8ng  process  events,  wriVen  by  kernel  and  init.  
Index  
•  Introduc,on.  

•  Event  gathering  system  (syslog).  

•  How  to  maintain  log  informa,on.  

•  How  to  use  log  informa,on.  

Maintaining  Log  informa,on  
•  Log  file:  basic  tool  for  control  and  repair.  
•  The  more  logged  informa8on-­‐>  The  more  disk  consumed:  
–  Can  exhaust  disk  quota.  
–  Hard  to  find  informa8on  in  a  file  with  millions  of  lines.  
•  Log  rota8on:  
–  Mechanism  consis8ng  of  periodically  wri8ng  to  a  new  log  file,  crea8ng  
a  new  empty  one  and  dele8ng  the  oldest  ones.  
–  Manual  Rota8on:  Example  script  performing  it.  
#!/bin/sh
cd /var/log/
mv messages.2 messages.3
syslogd mv messages.1 messages.2
mv messages messages.1
cat /dev/null > messages
chmod 600 messages
#Reiniciar syslog
service restart rsyslog
Maintaining  Log  informa,on  
•  Automa8c  Rota8on:  logrotate:  
–  Unsupervised  organiza8on  of  log  rota8on.  Avoids  disk  overflow  and  
organizes  log  files  according  to  their  crea8on  dates.  
–  Configura8on  through  the  file:  
 /etc/logrotate.conf:  
•  Applied  by  default  to  every   # rotate log files weekly, monthly
service.   weekly
# keep 4 weeks worth of backlogs
–  Par8culariza8on  for  a  service:   rotate 4
# send errors to root
/etc/logrotate.d/:   errors root
# create new(empty)log files after rotating old ones
•  Overwrites  the  op8ons  in   create

logrotate.conf.   # compressed log files

compress
# DEB packages drop log rotation info into this dir
/var/log/dpkg.log { include /etc/logrotate.d
monthly #no packages own lastlog or wtmp, rotate them here
rotate 12 /var/log/wtmp cd /var/log/{
compress monthly
notifempty create 0664 root utmp
create 0664 root adm rotate 1
} }
Index  
•  Introduc,on.  

•  Event  gathering  system  (syslog).  

•  How  to  maintain  log  informa,on.  

•  How  to  use  log  informa,on.  

Using  Log  informa,on  
•  How  to  use  the  informa8on  of  a  log:  
–  Debugging:  increase  available  informa8on  when  something  goes  wrong:  
•  E.g.  ac8vate  “verbose”  mode  for  services  (example,  in  /etc/init.d/ssh  sshd  –d).  
•  Deac8vate  when  moving  back  to  produc8on!!  
–  Monitoring:  
•  Problem:  huge  amount  of  informa8on  (not  everything  is  useful).  
•  Start  being  generous,  reduce/remove  unnecessary  informa8on  gradually.  
•  Make  use  of  specialized  tools  to  look  for  relevant  messages:  
–  Swatch:  bp://bp.stanford.edu/general/security-­‐tools/swatch/.  
–  LogWatch:  highly  recommended,  available  in  debian  repository.  
Using  log  info  
################### Logwatch 7.3.1 (09/15/06) ####################
Processing Initiated: Tue Dec 2 15:56:56 2008
Date Range Processed: yesterday ( 2008-Dec-01 )Period is day.
Detail Level of Output: 5

•  logwatch  -­‐-­‐print   Type of Output: unformatted

Logfiles for Host: debian
##################################################################

--------------- courier mail services Begin------------------

Courier restarted itself 4 Times
Courier was started by hand (or init) 2 Times
Courier was stopped by hand (or init) 2 Times
[ root si ~ ] vi /etc/cron.daily/00logwatch
Failed delivery attempts: 6 Times
#!/bin/bash
because 550 User unknown. - 6 Times
#Check if removed-but-not-purged From - 2 Times
To [email protected] - 2 Times
test -x /usr/share/logwatch/scripts/logwatch.pl
From #@[] - 2 Times
|| exit 0 To [email protected] - 2 Times
From [email protected] - 2 Times
#execute To [email protected] - 2 Times
/usr/sbin/logwatch --mailto root
--------------------- httpd Begin ------------------------
172.09 MB transferred in 220781 responses (1xx 0, 2xx 3444, 3xx 96, 4xx
217227, 5xx 14)
1316 Images (26.44 MB),
6985 Documents (55.30 MB),
6 Archives (0.83 MB),
2 Sound files (0.00 MB),
27286 Windows executable files (7.90 MB),
102944 Content pages (38.00 MB),
-------------------- Disk Space Begin ------------------------
Filesystem Size Used Avail Use% Mounted on
/dev/xvda2 9.9G 1.8G 7.6G 19% /
/dev/xvda3 504M 30M 450M 7% /boot
/dev/xvda4 2.0G 182M 1.9G 9% /files
/dev/xvda5 20G 4.0G 15G 21% /var/www
/dev/xvda6 2.0G 695M 1.2G 37% /var/cache/openafs
/dev/xvdb1 917G 390G 481G 45% /data
AFS 8.6G 0 8.6G 0% /afs
---------------------- Disk Space End -------------------------

###################### Logwatch End #########################