Big Brother

A Web-based System & Network

Monitor
Sean MacGuire, Director
The MacLawran Group Inc, Montreal
[email protected]

Sean MacGuire / Big Brother / SANS98 1

Notes:

I’m Sean MacGuire, Director of the MacLawran Group in Montreal.

I’ve been working with Unix since 1983 as everything from a programmer.,
Systems Administrator, Manager of Technical Support, and I’m currently
consulting as a Technical Project Manager.

I’m grateful to be here. I’m surprised with how popular Big Brother has
become, and really impressed with how excellent the Big Brother community
is.

I’m here to tell you about it.

2-1
 Overview

u What is Big Brother?

u Why was it created?
u Design Considerations
u Components and Structure
u Configuration and Security
u Benefits and where to get BB
u The future of Big Brother
u Conclusion and Questions

Sean MacGuire / Big Brother / SANS98 2

Notes:

These talks are tough. I don’t want to bore those people who already know
about Big Brother, but don’t want to do a disservice to those who’ve never
heard of it either.

Hopefully you’ll leave here with an appreciation for where Big Brother comes
from, what it is, how it was designed, how it works, why it works, and why it
is so wonderful.

The irony of course, is that in the time it takes to present this talk, you could
go the web site, download it, compile it, and have it up and running on your
network. Not only that, but this Powerpoint presentation is larger than the
entire source code to BB.

Ultimately, Big Brother is interesting because it was built by a Sys Admin for
Systems Administration. That changed the whole focus of the tool; it’s really
a holistic approach to monitoring - that something broken in one place will
naturally show up somewhere else and get noticed...

2-2
 What is Big Brother?

A Web-based System and

Network Monitor

u Web-based status display

u Matrix of machines and areas
u Green is Good / Red is Bad
u Very lightweight and small
u Written in Bourne shell & C
u Simple client-server design
u Ability to set thresholds and
notify administrators

Sean MacGuire / Big Brother / SANS98 3

Notes:

Big Brother is the same stuff that every Sys Admin has been writing since
entropy first attacked Unix machines, and things started to fall apart.

Watching over machines isn’t Rocket Science. It’s like the description of
being a pilot: Dull and Boring with occasional moments of sheer terror.

The general task of an admin is to make sure that the machines stay up and
running, that everybody else, users and managers, don’t bother the admin.
Generally, my goal has been to keep my phone from ringing.

So what does BB do differently - what makes it so special and wonderful that

you should run out and download it? Two things. It includes a really simple
set of client server programs that allow you to send data in a relatively secure
fashion, and it displays the information you send on a really pretty web page
for everyone to see.

That results in two things. Everyone knows how well the network is doing,
and the phone doesn’t ring as often for stupid questions.

2-3
 The Need for Big Brother

u Nothing simple was available for monitoring

u Commercial and Custom solutions were examined
u Very complex, some requiring months to configure
u They seemed Incredibly expensive
u Very heavy load on the servers
u None were web-based
u Complex, counterintuitive interfaces
u X-window or console based
u You had to be physically present to see things

Sean MacGuire / Big Brother / SANS98 4

Notes:

I was told that we had to get something to monitor this network that was
supposed to be available 24/7. There was a copy of HP OpenView around,
that I tried to install, but it had expired three months earlier.

I had seen quite a few “performance monitors”, and they had three things in
common. First, they almost always listed themselves as taking up half the
system resources, second, they were almost always a lot of money, and third,
most insisted on tweaking the kernel.

And getting the information meant going to a specific location, X-terminal or a

console in the machine room... maybe they paged you...

In addition to being the acting admin, I was also working on the Netscape
NSAPI and hacking their Proxy Server (to make it do unnatural things with
Oracle), and it surprised me that there weren’t Web-based tools out there to do
simple monitoring. The Web allows the creation of incredibly simple GUI
interfaces, really quickly and easily, not to mention the instant publishing of
information.

2-4
 The Real Reason for BB

u I defined the monitoring requirements

u I got mad at one of the vendors
u He had contempt for technical personnel (including his own)
u He had contempt for the company he was dealing with
u A little while later, he presented his quote
u $313,000 to monitor 20 servers
u Lots of additional hardware
u 90 man days of consulting to set this up
u They were going to buy this crap!
u Unless I wrote something that did the job

Sean MacGuire / Big Brother / SANS98 5

Notes:

OK... now the real reason BB came about.

Management brought in an “expert” from an integration company. I defined

the tasks - what had to be monitored, and how, etc. This expert, however, held
technical people in contempt, shmoozed with the bosses, and abused his own
staff in public. I didn’t like this guy.

Then when he came back with his quote - $313,000 to monitor fewer than 20
servers, all from a central console, with redundant servers, not to mention at
least 90 man-days of consulting at well over $100 an hour. He was very
pleased with himself.

I freaked. Unfortunately where I was working at the time, they happily spent
money provided the correct number of technobabble buzzwords were uttered
within the specified amount of time, out came the checkbook. The only way
they wouldn’t do this was if the problem was solved another way, and fast.
The infrastructure of Big Brother was built over that weekend.

2-5
 Design considerations

u Big Brother is not Rocket Science!

u I needed something to monitor and notify
u I needed constant access to this information
u It couldn’t be a drain on network resources
u It had to be as safe and secure as possible
u It had to be simple, portable and extensible
u I needed it yesterday

Sean MacGuire / Big Brother / SANS98 6

Notes:

I don’t understand why software insists on being so incredibly complicated.

Well, I do, sort of. If it’s really complicated, you can sell it for lots of money,
charge lots of money to work with it, and become an expert in it and make lots
of money.

Big Brother had to do what I did as an administrator. In fact, it had to do it the

same way I did it. So the health of any system was determined more by feel
and a couple of simple tests than anything else. The network worked if the
thing at the other side answered within a reasonable amount of time. Programs
worked if they produced the output that was expected, and as long as they
were running. And it’s always a good idea to keep an eye on the messages
file since that’s where a lot of the cries for help end up.

It needed some simple thresholds to watch, a way to send this data to a central
location, a method of testing the network, and finally a visual display that
would allow me to know what was going on from across the room, or via a
pager if things got bad. It also needed to be as redundant as possible.

2-6
 What BB doesn’t use

u SMNP u perl
SNMP is a network Perl is a great
protocol and is not language, but isn’t
well suited to shipped with every
collecting system Unix machine.
information.
I knew sh better than
It was scary and perl at the time BB
strange to me. was written

Sean MacGuire / Big Brother / SANS98 7

Notes:

Hard-core network people have asked why BB doesn’t use SNMP for
monitoring. The official reason is that it’s very tough to do the kind of testing
BB does on-the-fly using SNMP traps.

The real reason for not using SNMP is that at the time BB was created, I had
no idea how SNMP worked. Meanwhile great products like MRTG do the
SNMP thing very elegantly, and people like my friend Robert-Andre Croteau
have integrated BB and MRTG together very nicely.

So why wasn’t perl used when writing BB? A couple of reasons. The first is
that my shell was better than my perl. More importantly, though, /bin/sh is on
every machine. I wouldn’t have to port perl to every machine that BB was
running on. Finally, the BB scripts are really simple. Issue a command, check
the result, and send it somewhere. Bourne Shell is just fine for that.

Same with the C programs... write the client and server in C, that’s what C is
for. (yes I know you can write it in perl!) On a Sun box, the client was 4K, the
server was 7K. The load was imperceptible. That was the idea.

2-7
 Setting up Big Brother

u Download and unpack the source code

u Configure, Compile, and Install
u Set BBHOME in runbb.sh
u Set PAGER & thresholds in etc/bbdef.sh
u Define hosts and daemons in etc/bb-hosts
u Define BBDISPLAY, BBPAGER &
BBNET
u Link www on BBDISPLAY Document Root
Sean MacGuire / Big Brother / SANS98 8

Notes:

The source code lives on the Big Brother Internet Site. Don’t worry if you
can’t remember the address: http://www.iti.qc.ca/iti/users/sean/bb-dnld You
can always get back there by clicking my face on any BB site you see.

You’ll need a C compiler for Unix versions, kermit and a modem for paging
and A little time and patience.

Set BBHOME in runbb.sh. runbb.sh is the main program that runs everything
else based on the information you put in the etc/bb-hosts file. You can tell Big
Brother who to page, and under what circumstances by editing parameters in
etc/bbdef.sh. Most of the defaults are sensible.

BBDISPLAY is the machine running a Web server where the BB output will
be generated. BBNET is the machine that will test the network, and
BBPAGER is the machine you’ve configured to do the paging.

In order for the web pages created by Big Brother to be visible, you might
have to link the www directory underneath the Document Root directory of the
machine defined as BBDISPLAY your Web Server is running on.

2-8
 Directory Structure

bb
BBHOME
runbb .s h

doc src b in etc www w eb tm p

README S ourc e C ode P ro gra m s C onfig S tuff W e b pa ges P ro gra m s T em p
bb c onfig & M ak efiles an d S c ripts bb -ho s ts bb .htm l m k bb.s h

g ifs log s no tes

G IF files log file data note files
from bbd by m achine

Sean MacGuire / Big Brother / SANS98 9

Notes:

The directory structure is very straightforward and pretty obvious.

BBHOME is the directory where Big Brother is been installed.

etc contains system-dependent configuration information
web contains the scripts to create the web pages
www contains all the information visible from the Web pages

This structure allows you to isolate the web pages which are created by a
process that has absolutely nothing to do with the web server. The only
suggestion is that the pages be readable by persons accessing these files via the
web. This was done for security.

The logs directory contains the actual data as reported to Big Brother from the
network monitor and the local clients. Files in this directory are owned by
whomever the Big Brother Daemon is running under. These files are named
“machine.area”; i.e. a disk report for a machine called coffee would be called
coffee.disk, and would have a corresponding row and column on the Big
Brother output.

2-9
 Architectural Overview
BBDISPLAY BBDISPLAY
running every 5 min
bbd mkbb
bb-local
cpu, disk,
msgs, procs bb
via
bb
machine.area
port log files
web
bb-network 1984 pages
daemons
ftp, http, pop3
smtp BBPAGER Notify Admin
running by pager
bbd or e-mail

Sean MacGuire / Big Brother / SANS98 10

Notes:

runbb.sh runs on every machine that runs Big Brother. It decides what to do
based on the information you’ve defined in the etc/bb-hosts file.

If the machine is defined as BBDISPLAY, then a copy of bbd will be running

to receive data from clients. mkbb.sh will also run every 5 minutes to create
the Big Brother Web page, www/bb.html, based on the data in the logs
directory.

If the machine is defined as BBPAGER, then it too will be running bbd, but
only to receive and process pager requests. This machine notifies whomever
has been defined in the PAGER variable in etc/bbdef.sh.

If the machine is defined as BBNET, then it will run bin/bb-network.sh to test

the network. This data will be sent to bbd running on the BBDISPLAY
machine. If a required service is down, this information will also be sent to the
bbd running on the machine defined as BBPAGER.

bin/bb-local.sh runs on every machine collecting local data and sending it to

BBDISPLAY. Pager requests are sent directly to BBPAGER if need be.

2 - 10
 Big Brother Components

u Testing shell scripts do the real work

u bb-network.sh polls daemons on the whole network
u bb-local.sh tests each local system and reports
u Simple Client/Server programs use port 1984
u bb, the client program, sends pre-formatted data
u bbd, the Daemon, receives and processes bb data
u Web Display and Paging
u for Web-page creation mkbb.sh & friends
u for Paging bb-page.sh
Sean MacGuire / Big Brother / SANS98 11

Notes:

The scripts are not complex. bb-network.sh tests the entire network for
connectivity by default, and for the specific services listed in the etc/bb-hosts
file. bb-local.sh does the same for each machine.

Since Big Brother uses port 1984 (what would you expect it to use?). Make
sure it’s available and not blocked by any firewalls. Otherwise it won’t work,
eh?

web/mkbb.sh runs on the BBDISPLAY machine every 5 minutes creating the

Web page www/bb.html is created.

The BBPAGER machine receives requests to notify admins via bbd, and calls
bin/bb-page.sh to do the actual paging. If the PAGER variable contains a
numeric string, BB assumes it’s a pager number, and handles numeric paging
using kermit. If it appears to contain an e-mail address, then mail is sent. The
PAGER variable may contain multiple numbers and addresses, and the bb-
page.sh script has already been modified to support other paging methods
(including sendpage).

2 - 11
 What Big Brother tests

runbb.sh calls these scripts based on bb-hosts

u bb-local.sh runs on the u bb-network.sh tests all the
local machine to daemons for each host
u Available disk space listed in bb-hosts
u 5 minute CPU load u http, pop3, smtp
u notices and warnings u ftp, nntp
u processes running u connectivity via ping

uAll scripts run every 5 minutes - they

usend data back to the display server BBDISPLAY
usend data to pager BBPAGER if required
Sean MacGuire / Big Brother / SANS98 12

Notes:

bb-local.sh runs on every system Big Brother is installed on, checking that the
local machine is sane, that the disk hasn’t exploded, the CPU isn’t too
overloaded, or that important processes haven’t dropped dead.

bb-network.sh uses the program bbnet to test all the daemons listed for each
machine in the etc/bb-hosts file in addition to pinging each of them.

This structure results in a certain amount of built-in redundancy. If a machine

is down, bb-network.sh will catch it and report. If the bb-network machine
itself is down, the BBDISPLAY machine notices that the log files haven’t been
updated and changes the screen color.

The Big Brother Web pages will always have a background color
corresponding to the most severe condition on the network at that time.
Remember you can click on any dot on the Big Brother Web page to get more
details about the results of any particular test.

2 - 12
 bb-hosts controls BB

u Similar in format to /etc/hosts

u Keywords control everything - here they are:
u BBDISPLAY defines the machine to send the results to
u BBNET is the machine responsible for network testing
u BBPAGER is the machine that will handle pager requests
u Daemons as defined in /etc/services
u pop3, smtp, ftp, telnet, http, nntp
u group is used for display groupings
u summary sends summary info to another BBDISPLAY
u dialup doesn’t make BB upset if it can’t be reached
Sean MacGuire / Big Brother / SANS98 13

Notes:

The etc/bb-hosts file controls the execution of Big Brother. This file should be
the same on all machines running BB. If Big Brother is having trouble, this is
the first place to look.

BBDISPLAY and BBNET must be defined. BBPAGER should be defined,

but will only be used if the PAGER variable in etc/bbdef.sh contains a number
or an e-mail address.

BBDISPLAY is the Web Server where the Big Brother Display will live and
where the BB pages will be created.

BBNET is the machine that will do the network testing. This can be the same
as BBDISPLAY, and often is.

2 - 13
 Sample bb-hosts file

#
# BIG BROTHER HOSTS FILE COMMENTS WILL INTERFERE WITH PROCESSING!!
#
group <H3><I>The Big Brother Display Server</I></H3>
204.19.116.1 iti-s01.iti.qc.ca # BBPAGER BBNET BBDISPLAY ftp smtp pop3
group <H3><I>Local Server Group</I></H3>
204.19.116.2 iti-s02.iti.qc.ca # ftp smtp pop3 http:/iti-s02/
204.19.117.1 ns.iti.qc.ca # ftp smtp
group <H3><I>Test Modem Banks</I></H3>
dialup modem-bank-1 204.19.50.1 16
dialup modem-bank-2 204.19.50.17 16
summary canada.bc 204.19.116.1 http://www.iti.qc.ca/iti/users/sean/bb/
summary america.ny 204.19.116.1 http://www.iti.qc.ca/iti/users/sean/bb/
summary europe.uk 204.19.116.1 http://www.iti.qc.ca/iti/users/sean/bb/

Sean MacGuire / Big Brother / SANS98 14

Notes:

Comments confuse the scripts since they blindly search for keywords.
Therefore, do not comment out a line you don’t want in the file. Remove it
completely!

Groups are in effect until the next group line is reached. This will give the
display a pleasant table structure. HTML codes are permitted only on the
group lines.

Make sure that daemons are listed precisely as they appear in the /etc/services
file. A common error are misspellings of pop3 as pop-3 or even just pop.
Since we look for precisely these daemons, spelling counts.

dialup lines are used to test banks of modems for connectivity. It’s nothing
special, it just pings banks of IP addresses to see which are active.

summary lines allow you to send the cumulative results of a BBDISPLAY to

another machine defined as a BBDISPLAY. In this way you can stack BBs.

2 - 14
 Big Brother Protocol

u bb sends data to bbd over port 1984

u scripts call bb with 2 arguments, the address and the
data line, containing the action to take
u bb IP-ADDR “action machine.area color date data”
u bbd listening on port 1984 gets this data; the action is
either status, or page
u status writes a file in the log directory with the name
machine.area containing the rest of the line sent
u page calls the bb-page.sh script to page the admin with the
error message to send to the pager
Sean MacGuire / Big Brother / SANS98 15

Notes:

The protocol is pretty trivial. Make sure port 1984 is available and not
blocked. Otherwise it won’t work, eh?

Most of the work is handled locally by the scripts; the severity levels and data
pre-formatted. All bbd has to do is take very simple actions, it either writes a
log file with status information, or calls the pager.

Because the status information is pre-formatted and encoded by color in the

data sent to bbd, different machines can have different thresholds for red, and
yellow. Also since the status is in the file, page creation is trivial and just
involves creating a matrix of machines and areas, and putting the correct
colors in the boxes. The colors of course, are the first word in the status files.

Extending BB to test for other functions is easy; just have bb send the new
data to bbd with a new function name. So to add a function called bobo, do
the test and have bb send this data to “machine.bobo” and the display will be
updated automatically the next time the mkbb.sh script runs.

2 - 15
 Big Brother Security

u etc/security allows u All commands are

you to define which executed using their
hosts and networks full pathname to avoid
can connect Trojan horses
u Big Brother doesn’t u Big Brother has it’s
have to run as root own tmp directory
u Big Brother daemons u You have the source
can only write in the code!
BB logs directory

Sean MacGuire / Big Brother / SANS98 16

Notes:

The etc/security file just contains lines with IP or network addresses of clients
permitted to connect to bbd running on that machine. If the file exists, then
only those hosts and networks listed will be allowed to connect. All others
will be silently dropped. For example:
204.101.110.101 Allow client to connect
204.101.112.0 Allow subnet to connect

If Big Brother is not running as root, it might have trouble reading log files on
certain machines depending on permissions.

bbd checks to see if the bb client is trying to do funny things with the
pathnames or is attempting to overflow buffers.

All BB commands are stored in environment variables, and are executed using
their full pathnames to avoid possible Trojan horses.

The best security is that you have the source code!

2 - 16
 Big Brother Benefits

u System information is available to anyone

who needs it, anytime from anywhere
u Systems Administrators
u Help Desk personnel
u Even Management can understand it
u Simple, portable, and highly configurable
u Pager support allows admins to rest easy.

Sean MacGuire / Big Brother / SANS98 17

Notes:

Probably the greatest benefit is the ability to publish system status information
in a format that is understandable by all. No more calls about whether or not a
service is available.

There’s enough information on the Web Pages to finally let managers know
how well their complex network and even more complex administrators are
doing. Green screens are good.

Even when there is trouble, management and the help desks can be confident
that the admin has already been notified by Big Brother and that they are
working on the problem.

Since installing BB, I’ve never been caught by surprise by a user. It also
allowed me to go for coffee in peace knowing I’ll get paged if need be. Being
proactive, that’s what they call it.

Conservatively I figure it saves me about an hour per system per month.

2 - 17
 Big Brother Statistics

u Initial Release: 31 October 1996

u 16 releases since the first version .90 Beta
u Article in Sys Admin - March 1997
u Paul Sittler’s Linux Journal Article
u Over 24,000 downloads
u From over 90 countries
u Over 500 Brothers on the mailing list

Sean MacGuire / Big Brother / SANS98 18

Notes:

The primary mode of people discovering Big Brother is the personal

recommendation of another Sys Admin. Once they discover BB, Brothers take
BB with them from job to job. That’s the finest compliment I could receive.

The articles in Sys Admin, and Paul Sittler’s article in Linux Journal certainly
haven’t hurt getting the word out, either. And I’m here ‘cause Shawn Welsh is
running BB @home.

Remember that Big Brother is only a base to work from. It doesn’t do

everything, it’s not supposed to. Monitor and Notify, but don’t touch
anything!

I think my favorite place using BB is the Kingdom of Tonga.

Best of all is the community that has spontaneously appeared around BB.
These Brothers are wise and good.

2 - 18
 Getting Big Brother

u Big Brother lives exclusively at

http://www.iti.qc.ca/iti/users/sean/bb-dnld
u Works with all Unix machines
u Should compile, configure and install in
under an hour
u It’s free as long as you’re not charging for it
or using the source in other applications
u Support via the BB mailing list & archives

Sean MacGuire / Big Brother / SANS98 19

Notes:

The best way to understand Big Brother is to download it and try it out. It’s
pretty simple and does a lot of what admins need to do.

The code has been ported to just about every Unix box around, from the very
old, to Crays... there’s even an OpenVMS port out there somewhere.

The install is simple. You may have to adjust some paths in etc/bbdef.sh in
case there is some screaming, but that should be the extent of the mods
required.

Commercial use is restricted. You can’t charge others for the services BB
provides or include it in a product for sale without first obtaining a
Commercial License.

And if you have any questions, hit the BB mailing list, or the fine archives run
by Nick Silberstein. They live at http://www.fusioni.com/~bb/

2 - 19
 Future Directions

u Support for Windows NT/95

u Windows NT client available now
u Windows NT server available soon
u Windows 95 available soon
u Logging and historical system info
u Enhanced Paging matrix
u By area affected (if http crashes contact...)
u By time of day, night, weekend, holiday
Sean MacGuire / Big Brother / SANS98 20

Notes:

Changes to Big Brother happen relatively slowly. It seems any time I touch
the source code I’m as likely to break something as to fix something.

Windows NT is looking more and more like a likely target for Big Brother.
Robert-Andre Croteau has already done an excellent client, the bbd server
ports out of the box, and all that really needs to be done are the Web-page
creation programs.

Logging and elegant and useful historical system info have been on the “to do”
list for a long time. Maybe this year.

Enhancing the paging functions is another important advance. The ability to

page by function, time of day, holiday, or notify multiple admins would be a
useful addition to the product.

Maybe Sun will license Big Brother to replace Sun Net Manager :)

2 - 20
 In Conclusion

u Big Brother is simple

u Shell scripts and C programs
u Polls and Collects data from your network
u Displays this information on a Web page
u Big Brother isn’t
u too big for a human being to understand
u costly, complicated or cumbersome
u Any Questions?
Sean MacGuire / Big Brother / SANS98 21

Notes:

Big Brother is a combination of monitoring methods. Unlike SNMP where

information is just collected and devices polled, Big Brother is designed in
such a way that each local system broadcasts it's own information to a central
location. Simultaneously, Big Brother also polls all networked systems from a
central location. This creates a highly efficient and redundant method for
proactive network monitoring.

The entire network status is displayed on an incredibly intuitive web page.

Red is bad, and green is good. Click on the dot and get more information.
You get paged if things get really bad.

Since BB is so lightweight, free, and simple, there’s no reason not to install it

on your network. Even if only to make sure that the expensive monitoring
system is up and running, or as a panacea to upper management.

Thanks for listening, and at this point I’d be happy to answer any questions
you might have about BB.

2 - 21