IS 7212: Advanced Networking and

System Administration

By Dr. Samuel Asferaw

2020/2021
• Introduction
• Managing Workstations / Desktops/, Managing Servers,
Managing Services
• Booting
• Processes
• Disks, Partitions, Volumes, File systems, Files
• Printing
• Accounts
• Service Monitoring
• People, Help Desk, Debagging
• Directories
• Data Centers
• Namespaces
• DNS
• Networking
• Network Architecture
2018/2019 IS 7212: Advanced Networking and Sys 2
tem Administration by Dr. Samuel Asfer
 Introduction

2018/2019 IS 7212: Advanced Networking and Sys 3

tem Administration by Dr. Samuel Asfer
 Topics
1. What is system administration?
2. What do sysadmins do?
3. Administration Challenges
4. Principles and First Steps
5. Organizations and Certifications
6. Maturity and Complexity
7. SAGE Code of Ethics
8. X-Windows Operating Systems

2018/2019 IS 7212: Advanced Networking and Sys 4

tem Administration by Dr. Samuel Asfer
 What is a system?
System: An organized collection of computers interacting
with a group of users looking for services to accomplish
work.

Servers PCs
run on

Use
Network

Services Users
help to accomplish work

2018/2019 IS 7212: Advanced Networking and Sys 5

tem Administration by Dr. Samuel Asfer
 System State
System policy: specification of a system’s
configuration and its acceptable usage.
System state S(t): the current configuration (files,
kernel, memory or CPU usage) of a system.
Ideal states S*(t): states of the system that
match the system policy. Over time, the
system state shifts away from the ideal state.
System administration: modifying the system to
bring it closer to S*(t).

2018/2019 IS 7212: Advanced Networking and Sys 6

tem Administration by Dr. Samuel Asfer
 What do sysadmins do?
Small org: sysadmin can be entire IT staff
– Phone support
– Order and install software and hardware
– Fix anything that breaks from phones to servers
– Develop software
Large org: sysadmin is one of many IT staff
– Specialists instead of “jack of all trades”
– Database admin, Network admin, Fileserver
admin, Help desk worker, Programmers,
Logistics

2018/2019 IS 7212: Advanced Networking and Sys 7

tem Administration by Dr. Samuel Asfer
 What do sysadmins do? …
1. Add and remove users
2. Add and remove hardware
3. Perform backups
4. Install new software systems
5. Troubleshooting
6. System monitoring
7. Auditing security
8. Help users
9. Communicate
All are common activities for sysadmin.
2018/2019 IS 7212: Advanced Networking and Sys 8
tem Administration by Dr. Samuel Asfer
 User Account Management
Creating user accounts
– Consistency requires automation
– Startup (dot) files
Removing user accounts
– Consistency requires automation
– Many accounts across different systems
Namespace management
– Usernames and UIDs
– Multiple namespaces or SSI?

2018/2019 IS 7212: Advanced Networking and Sys 9

tem Administration by Dr. Samuel Asfer
Hardware Management
Adding and removing hardware
– Configuration, cabling, etc.
Purchase
– Evaluate and purchase servers + other hardware
Capacity planning
– How many servers? How much bandwidth, storage?
Data Center management
– Power, environment (cooling, fire alarm)
Virtualization
– When can virtual servers be used vs. physical?
2018/2019 IS 7212: Advanced Networking and Sys 10
tem Administration by Dr. Samuel Asfer
 Data Backups
Backup strategy and policies
– Scheduling: when and how often?
– Capacity planning
– Location: on-site vs. off-site.
Monitoring backups
– Checking logs
– Verifying media
Performing restores when requested

2018/2019 IS 7212: Advanced Networking and Sys 11

tem Administration by Dr. Samuel Asfer
 Software Installation
Evaluation of software
Automated consistent OS installs
– Desktop vs. server OS image needs.
Installation of software
– Purchase, find, or
build custom software.
Managing software installations
– Distributing software to multiple hosts.
– Managing multiple versions of a software pkg.
Patching and updating software

2018/2019 IS 7212: Advanced Networking and Sys 12

tem Administration by Dr. Samuel Asfer
 System Monitoring
Automatically monitor systems for
– Problems (disk full, error logs, security)
– Performance (CPU, mem, disk, network)
Security
– Passwords
– Break-ins
Capacity
– Disk, RAM, CPU, network
System logs
– Examination
– Periodic rotation and truncation
– Archival storage (at least
summaries)
Provides data for capacity planning
– Determine need for resources
– Establish case to bring to management
2018/2019 IS 7212: Advanced Networking and Sys 13
tem Administration by Dr. Samuel Asfer
 Troubleshooting
Problem identification, diagnosis, and resolution
– By user notification
– By log files or monitoring programs
Finding the root cause of problems
– Provide temporary solution if necessary
– Solve the root problem to permanently eliminate
• Often quite difficult!
Often requires
– Broad and thorough
system knowledge
– Outside experts
– Luck
2018/2019 IS 7212: Advanced Networking and Sys 14
tem Administration by Dr. Samuel Asfer
 User Assistance

• Time intensive!
• Techniques
– Help desks
– Trouble-ticket systems
• Software availability and usage
• Software configuration settings
• Hardware usage, maintenance, and troubleshooting
• User documentation and training
– Policies and procedures
• Schedule and communicate downtimes
2018/2019 IS 7212: Advanced Networking and Sys 15
tem Administration by Dr. Samuel Asfer
 Security Concerns
• System logging and audit facilities
– Evaluation and implementation
– Monitoring and analysis
– Traps, auditing and monitoring programs
• Unexpected or unauthorized user
detection
• Monitoring of security advisories
– Security holes and weaknesses
– Live exploits
2018/2019 IS 7212: Advanced Networking and Sys 16
tem Administration by Dr. Samuel Asfer
 Communicate
Customers
– Keep customer appraised of process
• When you’ve started working on a request
• When you make progress, need feedback
• When you’ve finished
– Communicate system status
• Uptime, scheduled downtimes, failures
– Meet regularly with customer managers
Managers
– Meet regularly with your manager
– Write weekly status reports

2018/2019 IS 7212: Advanced Networking and Sys 17

tem Administration by Dr. Samuel Asfer
 Specialized Skills
Heterogeneous Environments
Integrating multiple-OSes, hardware types, or network
protocols, distributed sites.
Databases
SQL RDMS
Networking
Complex routing, high speed networks, voice.
Security
Firewalls, authentication, biometrics, cryptography.
Storage
Network Attached Storage (NAS), Storage Area
Networks (SANs), cloud storage.
Virtualization and Cloud Computing
VMware, cloud architectures.
2018/2019 IS 7212: Advanced Networking and Sys 18
tem Administration by Dr. Samuel Asfer
 Practice of sysadmin
• Common practice may not be good practice
– There are three reasons for common practice:
• Someone did it and others followed blindly;
• Believe it is good after careful thought;
• An arbitrary choice had to be made;
– Think for yourself
– Pay attention to experts but don’t automatically
believe anyone
– Every choice needs a reason

2018/2019 IS 7212: Advanced Networking and Sys 19

tem Administration by Dr. Samuel Asfer
 Practice of sysadmin ...
• Good practices
– Look for answers in manuals, newsgroups, and
archive of mailing lists
• Usually “google” helps for most common problems
– Use controlled trial and error for diagnosis
– Listen to people who tell us there is a problem. It might
be true
– Write down problems and solutions in a log book, and
write down experiences
– Take responsibilities for our actions
– Remember to tidy things up regularly
– After learning something new, ask yourself “How
does this apply to my work?”
2018/2019 IS 7212: Advanced Networking and Sys 20
tem Administration by Dr. Samuel Asfer
 System administrator (sysadmin)
• Successful System Administrators
– No stereotypes, “White/black Cats Theory”
– Normally if one can keep the users happy she is
successful (which is very difficult)
– Life-long learning skills. One’s proud knowledge and
expertise today may be obsolete tomorrow (Be
prepared to be jobless if you do not keep learning)
• Some myths for new sysadmin
– There exists a right answer for every problem
– Things should always work in the way we expect
– Every problem should have a happy end

2018/2019 IS 7212: Advanced Networking and Sys 21

tem Administration by Dr. Samuel Asfer
Challenges of System Administration

• Not only just installing system/software, also

about planning and designing an efficient
community of computers
• Design a logical and efficient network
• Easy upgrade for a large number of computers
• Decide what and where services are installed
• Plan and implement security
• Provide a comfortable environment for users
• Develop ways of fixing problems and errors
• Keep track of new technology and software
2018/2019 IS 7212: Advanced Networking and Sys 22
tem Administration by Dr. Samuel Asfer
Challenges of System Administration ...
• Need
– Broad knowledge of hardware and software
– To balance conflicting requirements
• Short-term vs. long-term needs
• End-user vs. organizational requirements
• Service provider vs. police model
– To work well and efficiently under pressure
– 24x7 availability
– Flexibility, tolerance, and patience
– Good communication skills
• People think of sysadmins only when things
don't work!
2018/2019 IS 7212: Advanced Networking and Sys 23
tem Administration by Dr. Samuel Asfer
 Principles of SA
Simplicity
– Choose the simplest solution that solves the entire problem
– Work towards a predictable system
Clarity
– Choose a straightforward solution that’s easy to change,
maintain, debug, and explain to other SAs
Generality
– Choose reusable solutions that scale up; use open protocols
Automation
– Use software to replace human effort
Communication
– Be sure that you’re solving the right problems and that people
know what you’re doing
Basics First
– Solve basic infrastructure problems before advanced ones

2018/2019 IS 7212: Advanced Networking and Sys 24

tem Administration by Dr. Samuel Asfer
 Qualities of a Successful Sysadmin

Customer oriented
– Ability to deal with interrupts, time pressure
– Communication skills
– Service provider, not system police
Technical knowledge
– Hardware, network, and software knowledge
– Debugging and troubleshooting skills
Time management
– Automate everything possible
– Ability to prioritize tasks: urgency and importance

2018/2019 IS 7212: Advanced Networking and Sys 25

tem Administration by Dr. Samuel Asfer
 First Steps to Better SA
Use a request system
– Customers know what you’re doing
– You know what you’re doing
Manage quick requests right
– Handle emergencies quickly
– Use request system to avoid interruptions
Policies
– How do people get help?
– What is the scope of responsibility for SA team?
– What is our definition of emergency?
Start every host in a known state.
2018/2019 IS 7212: Advanced Networking and Sys 26
tem Administration by Dr. Samuel Asfer
 Types of Sites
Small
2-10 computers, 1 OS, 2-20 users.
Small staff size requires outsourcing to obtain most
specialized skills.
Midsized
11-100 computers, 1-3 OSes, 21-100 users.
Large
100+ computers, multiples OSes, 100+ users
Outsources to reduce costs, some specializations.

2018/2019 IS 7212: Advanced Networking and Sys 27

tem Administration by Dr. Samuel Asfer
 Organizations

USENIX: Advanced Computing Systems

Association
LISA: Large Installation System
Administration
SAGE: System Administration Guild
LOPSA: League of Professional System
Administrators

2018/2019 IS 7212: Advanced Networking and Sys 28

tem Administration by Dr. Samuel Asfer
 Certifications

• CCNA, CCNP, CCIE (Cisco)

• cSAGE (SAGE)
• MCSA (Microsoft)
• RHCE (Red Hat)
• SCSA (Sun)
• VCP (VMware)

2018/2019 IS 7212: Advanced Networking and Sys 29

tem Administration by Dr. Samuel Asfer
 SAGE Job Descriptions
Novice
OS familiarity, help desk skills
Junior
Can use OS system administration tools
Intermediate
Understanding of distributed computing, common
servers, automate small tasks, independent action
Senior
Understanding of scaling issues, including capacity
planning, solve problems by addressing root cause,
higher level programming abilities, write proposals for
purchasing, data center planning, etc.
2018/2019 IS 7212: Advanced Networking and Sys 30
tem Administration by Dr. Samuel Asfer
 SA Maturity Model (SAMM)
1. Ad Hoc
Ad-hoc non-repeatable solutions, firefighting.
2. Repeatable
Some repeatable processes.
3. Defined
Documented standard processes
4. Managed
Process effectiveness measured, adapted.
5. Optimized
2018/2019 IS 7212: Advanced Networking and Sys 31
tem Administration by Dr. Samuel Asfer
 Maturity and Complexity

Scalable but Low downtime,

time lost in high efficiency
process
Maturity

Constant firefighting,
Works, but hard high downtime
to scale up

Complexity: increasing numbers of systems and/or services

2018/2019 IS 7212: Advanced Networking and Sys 32

tem Administration by Dr. Samuel Asfer
 Tool Maturity Levels
1. Ad Hoc
OS GUI, CLI, or web administration interfaces.
2. Repeatable
Version control (RCS, SVN, GIT), request tracker
3. Defined
Automatic monitoring (Nagios, monit, god)
4. Managed
Configuration management (AutomateIt, cfengine)
5. Optimized

2018/2019 IS 7212: Advanced Networking and Sys 33

tem Administration by Dr. Samuel Asfer
 SAGE Code of Ethics
• Professionalism
• Personal Integrity
• Privacy
• Laws and Policies
• Communication
• System Integrity
• Education
• Social Responsibility
http://www.sage.org/ethics/

2018/2019 IS 7212: Advanced Networking and Sys 34

tem Administration by Dr. Samuel Asfer
 X-Windows
• Mac Operating System
• Unix Operating System
• Linux Operating System
• etc

2018/2019 IS 7212: Advanced Networking and Sys 35

tem Administration by Dr. Samuel Asfer
 Which OS to learn to admin?

2018/2019 IS 7212: Advanced Networking and Sys 36

tem Administration by Dr. Samuel Asfer
 Brief History of UNIX
• Originated as a research project in 1969 at AT&T Bell
Labs
– Made available to universities (free) in 1976
• Berkeley UNIX started in 1977 when UCB licensed code
from AT&T.
– Berkeley Software Distribution started in 1977 with 1BSD, and
ended in 1993 with 4.4BSD
– Licensing costs from AT&T increased, so Berkeley attempted to
remove AT&T code, but ran out of funds before completion.
– Final release of AT&T-free code called 4.4BSD-Lite.
• Most current BSD distributions (FreeBSD, NetBSD, OpenBSD) are
derived from 4.4BSD-Lite.
• Most commercial versions of UNIX (Solaris, HP-UX) are
derived from the AT&T code
2018/2019 IS 7212: Advanced Networking and Sys 37
tem Administration by Dr. Samuel Asfer
 What is Linux?
much is courtesy of www.kernel.org

• Linux is a clone of the operating system Unix, written by a loosely-

knit team of hackers across the Net.
• Like any modern full-fledged Unix, Linux includes:
– true multitasking
– proper memory management and virtual memory
– shared libraries and shared copy-on-write executables
– TCP/IP networking
• Linux really refers to the kernel
– Most commands and apps are really separate programs, not specific to
Linux, and often are part of the Free Software Foundation's GNU project.
• Linux was first developed for 32-bit x86-based PCs (386 or
higher). These days it also runs on dozens of other processors.
2018/2019 IS 7212: Advanced Networking and Sys 38
tem Administration by Dr. Samuel Asfer
 Brief History of Linux
• Created as a personal project (and still
controlled) by Linus Torvalds, a Finnish
graduate student, in 1991
• Conceived as an offshoot of Minix (a model
OS)
– Not derived from AT&T or BSD UNIX code
• Red Hat (one of many Linux vendors) founded
in 1993
• Kernel v1.0 released 1994
• Most recent (Jan 2016) kernel release is 4.4?
2018/2019 IS 7212: Advanced Networking and Sys 39
tem Administration by Dr. Samuel Asfer
 Where to get answers?
• Linux/UNIX documentation can be found
in many places
– Manual pages (man pages, using man
command)
– Texinfo documents (read with info command)
– HOWTOs – focused descriptions of a topic
– Distribution-specific documentation
– Your favorite Web search engine
• Will typically find online versions of the above

2018/2019 IS 7212: Advanced Networking and Sys 40

tem Administration by Dr. Samuel Asfer
 Where to get answers?

2018/2019 IS 7212: Advanced Networking and Sys 41

tem Administration by Dr. Samuel Asfer
• Introduction
• Managing Workstations / Desktops/, Managing Servers,
Managing Services
• Booting
• Processes
• Disks, Partitions, Volumes, File systems, Files
• Printing
• Accounts
• Service Monitoring
• People, Help Desk, Debagging
• Directories
• Data Centers
• Namespaces
• DNS
• Networking
• Network Architecture
2018/2019 IS 7212: Advanced Networking and Sys 42
tem Administration by Dr. Samuel Asfer
 Managing Workstations/Desktops,
Servers, Services

• Managing (lots of) desktops

– Loading, updating, configuring

• – Managing servers
– Important to lots of people

• – Managing services
– The reason for most servers

2018/2019 IS 7212: Advanced Networking and Sys 43

tem Administration by Dr. Samuel Asfer
Initial OS and App Installation
• Automation solves many problems
– Saves time/money; reduces mistakes; ensures
uniformity
– Examples: Solaris JumpStart, Red Hat Kickstart,
AutoYaST, Preseed
– Cloning (ghosting, disk imaging) sometimes an option
• Full automation much better than partial
– Eliminate prompts in installation scripts
– Can include automatically notifying people when
complete
• Partial automation better than none
– Needs to be well-documented for consistency
2018/2019 IS 7212: Advanced Networking and Sys 44
tem Administration by Dr. Samuel Asfer
 Managing (lots of) Desktops
• Three main sysadmin tasks for managing
workstations
– i) Initial loading of system software and applications
– ii) Updating system software and applications
– iii) Configuring network parameters
• Need to get all three right
– Initial load must be consistent across machines
– Updates must be quick
– Network configuration best managed centrally
• Solution is automation (for supported platforms)

2018/2019 IS 7212: Advanced Networking and Sys 45

tem Administration by Dr. Samuel Asfer
 Lots of Desktops

You really don't want to install, configure, and

update lots of machines individually. Why?
2018/2019 IS 7212: Advanced Networking and Sys 46
tem Administration by Dr. Samuel Asfer
 Machine Life-Cycle

Source: Evard, 1997

2018/2019 IS 7212: Advanced Networking and Sys 47
tem Administration by Dr. Samuel Asfer
 Machine Life-Cycle …
• There are five states and many transitions
– Need to plan for them
• – Computer is only usable in the configured state
– Want to maximize useful time
– Minimize useless time
– Setup and recovery should be fast and efficient -->
– automation (manual processes are slow and error-prone)
– Slow (minimize) entropy
– Restrict root privileges Why?
– Control where changes can and are made (e.g., 3rd party
apps)
• Rebuilding and retiring may require moving data and
apps Why ?
2018/2019 IS 7212: Advanced Networking and Sys 48
tem Administration by Dr. Samuel Asfer
 States of Machines
New
A new machine
Clean
OS installed, but not yet configured for operating
environment.
Configured
Configured correctly for the operating environment.
Unknown
Misconfigured, broken, newly discovered, etc.
Off
Retired
2018/2019 IS 7212: Advanced Networking and Sys 49
tem Administration by Dr. Samuel Asfer
 State Transitions
Build
Set up hardware and install OS.
Initialize
Configure for operating environment; often part
of build.
Update
Install new software.
Patch old software.
Change configurations.

2018/2019 IS 7212: Advanced Networking and Sys 50

tem Administration by Dr. Samuel Asfer
 Use your own Installation
• Don't trust the vendor's pre-installed OS
– Adding apps to a truly clean installation can
be easier
– Their install image can change over time
– You'll need to re-install eventually
• You want to be certain that you have everything
(drivers, software, etc.) to re-install
– You may not want or need their special
applications and add-ons

2018/2019 IS 7212: Advanced Networking and Sys 51

tem Administration by Dr. Samuel Asfer
 Updating System and Apps
• Over time, people find
– New bugs
– New security holes
– New applications
• Updates can (and should) be automated
too
• Example automation systems include Linux
package updaters like pup/yum and apt

2018/2019 IS 7212: Advanced Networking and Sys 52

tem Administration by Dr. Samuel Asfer
 Differences for Updates
• Updates are performed on functioning machines
• The machine is already deployed
– Can't flood network
– May not have physical access
• Users of host will expect it to work after update
– Must be extremely careful! Gradual deployment.
• Host may not be in known state
• Host may have live users (requiring downtime)
• Host may be disconnected periodically
• Host may dual-boot (long periods between
updates)
2018/2019 IS 7212: Advanced Networking and Sys 53
tem Administration by Dr. Samuel Asfer
 Network Configuration
• Network config different from install
– Values vary by location, rather than OS+apps
• Typical solution is to use DHCP (Dynamic Host
Configuration Protocol). Because:
– Eliminate time and manual error
• By sysadmin or user (assigning himself an IP address
and/or hostname)
– More secure (only authorized systems get access)
– Can assign a particular IP to an individual host
– Centralized control makes updates and changes
easier (e.g., new DNS (Domain Name System)
server)
2018/2019 IS 7212: Advanced Networking and Sys 54
tem Administration by Dr. Samuel Asfer
 Managing Servers

2018/2019 IS 7212: Advanced Networking and Sys 55

tem Administration by Dr. Samuel Asfer
 Managing Servers
● Different from desktop?
Yes!
● May serve tens, hundreds
or many thousands of users
● Requires reliability
● Requires tighter security
● Often expected to last
longer
● Extra cost is amortized
across users, life span

2018/2019 IS 7212: Advanced Networking and Sys 56

tem Administration by Dr. Samuel Asfer
 Managing Servers …
• Servers typically have
– Different OS configurations than
desktops
– Deployment within the data center
– Maintenance contracts
– Disk backup systems
– Better remote access

2018/2019 IS 7212: Advanced Networking and Sys 57

tem Administration by Dr. Samuel Asfer
 Server Hardware
• Buy server hardware for servers with
– More internal space
– More CPU performance
– High performance I/O (both disk and network)
– More upgrade options
– Rack mountable/optimized
• Use vendors known for reliability
– Your time is valuable

2018/2019 IS 7212: Advanced Networking and Sys 58

tem Administration by Dr. Samuel Asfer
Do Servers Really Cost More?
• Typical vendor has three product lines
• – Home
– Absolute cheapest purchase price
– Components change often
• – Business
– Longer life, reduced TCO (Total Cost of Ownership)
– Fewer component changes
• – Server
– Lowest cost per performance metric
– Easier to service components and design
2018/2019 IS 7212: Advanced Networking and Sys 59
tem Administration by Dr. Samuel Asfer
 Maintenance contracts, spare parts
• All machines eventually break!
• Vendors have variety of service contracts
– On-site with 4-hour, 12-hour, or next-day response
– Customer-purchased spare parts get replaced when used
• How to select maintenance contract? Determine needs.
– Non-critical hosts: next-day or two-day response time is likely reasonable,
or perhaps no contract
– Large groups of similar hosts: use spares approach
– Controlled model: only use a small set of distinct technologies so that few
spare part kits needed
– Critical host: stock failure-prone and interchangeable parts (power supplies,
hard drives); get same-day contract for remainder
– Large variety of models from same vendor: sufficiently large sites may opt
for a contract with an on-site technician

2018/2019 IS 7212: Advanced Networking and Sys 60

tem Administration by Dr. Samuel Asfer
 Data Backups
• Servers are often unique with critical data
that must be backed up
• Clients are often not backed up. Why?
(most data is on server)
• Consider separate administrative network
– Might want to keep bandwidth-hungry backup
jobs off of production network
– Provides alternate access during network
problems
– Requires additional cabling, switches
2018/2019 IS 7212: Advanced Networking and Sys 61
tem Administration by Dr. Samuel Asfer
 Servers in the Data Center
• Servers should be located in data centers
• Data centers provide for servers:
– Proper power (enough power, conditioned, UPS,
or generator)
– Fire protection/suppression
– Networking
– Sufficient air conditioning (climate controlled)
– Physical security

2018/2019 IS 7212: Advanced Networking and Sys 62

tem Administration by Dr. Samuel Asfer
 Remote Administration
• Data centers are expensive, and thus often
overcrowded, cold, noisy, and may be distant from
admin office
• Servers should not require physical presence at a
console
• Typical solution is a console server
– Eliminate need for keyboard and screen
– Can see booting, can send special keystrokes
– Access to console server can be remote (e.g., ssh
(secure shell), rdesktop (remote desktop protocol))
• Power cycling provided by remote-access power-strips
• Media insertion and hardware servicing are still
problems
2018/2019 IS 7212: Advanced Networking and Sys 63
tem Administration by Dr. Samuel Asfer
 Mirrored Root Disks
• Disk drives fail!
• Often useful to consider RAID (originally Redundant Array
of Inexpensive Disks, but now Redundant Array of
Independent Disks) for data integrity, redundancy,
performance, etc
• The main system disk is often the most difficult to replace
• Software RAID often comes with the OS for “free”;
hardware RAID is getting cheaper
• Two approaches for mirrored root disks:
– Two disks; copy from the working disk to the clone at regular
intervals (e.g., once a night)
– Use hardware or software RAID to keep both in sync
• RAID disks still need to be backed up
•
2018/2019 Why? IS 7212: Advanced Networking and Sys 64
tem Administration by Dr. Samuel Asfer
 Redundant Power Supplies
• Power supplies 2nd most
failure-prone part
• Ideally, servers should have
redundant power supplies
– Means the server will still operate if one
power supply fails
– Should have separate power cords
– Should draw power from different sources
(e.g., separate UPSes)
2018/2019 IS 7212: Advanced Networking and Sys 65
tem Administration by Dr. Samuel Asfer
Router with dual power supplies

• This is a Cisco 4506 switch

that serves as one of the
backbone switches.
• Fiber (or copper if nearby)
travels from this switch to
each router on campus.
• It has redundant power
supplies, one connected
to a UPS and one
connected directly to
commercial power.

2018/2019 IS 7212: Advanced Networking and Sys 66

tem Administration by Dr. Samuel Asfer
 Hot-swap Components
• Redundant components
should be hotswappable
– New components can be added
without downtime
– Failed components can be
replaced without outage
• Hot-swap components
increases cost
– But consider cost of downtime
• Always check
– Does OS fully support hot-
swapping components?
– What parts are not hot-
swappable?
– How long/severe is the service
interruption?
2018/2019 IS 7212: Advanced Networking and Sys 67
tem Administration by Dr. Samuel Asfer
 Managing Services
• Services distinguish a structured computing environment
from a bunch of standalone computers
• Larger groups are typically linked by shared services that
ease communication and optimize resources
• Typical environments have many services
– DNS, email, authentication, networking, printing
– Remote access, license servers, DHCP, software repositories,
backup services, Internet access, file service
• Providing a service means
– Not just putting together hardware and software
– Making service reliable
– Scaling the service
– Monitoring, maintaining, and supporting the service
2018/2019 IS 7212: Advanced Networking and Sys 68
tem Administration by Dr. Samuel Asfer
 Designing a solid service
• Get customer requirements
– Reason for service
• How service will be used
• Features needed vs. desired
• Level of reliability required
• Justifies budget level
– Define a service level agreement (SLA)
• Enumerates services
• Defines level of support provided
• Response time commitments for various kinds of problems
– Estimate satisfaction from demos or small usability
trials
2018/2019 IS 7212: Advanced Networking and Sys 69
tem Administration by Dr. Samuel Asfer
 Designing a solid service …
• Get operational requirements
– What other services does it depend on?
• Only services/systems built to same standards or higher
• Integration with existing authentication or directory services?
– How will the service be administered?
– Will the service scale for growth in usage or data?
– How is it upgraded? Will it require touching each desktop?
– Consider high-availability or redundant hardware
– Consider network impact and performance for remote users
• Revisit budget after considering operational concerns

2018/2019 IS 7212: Advanced Networking and Sys 70

tem Administration by Dr. Samuel Asfer
 Designing a solid service …
• Consider an open architecture
– E.g., open protocols and open file formats
– Proprietary protocols and formats can be changed, may
cause dependent systems/vendors to become incompatible
– Beware of vendors who “embrace and extend” so that
claims can be made for standards support, while not
providing customer interoperability
– Open protocols allow different parties to select client vs.
server portions separately
– Open protocols change slowly, typically in upward
compatible ways, giving maximum product choice
– No need for protocol gateways (another system/service)

2018/2019 IS 7212: Advanced Networking and Sys 71

tem Administration by Dr. Samuel Asfer
 Designing a solid service …
• Favor simplicity
– Simple systems are more reliable, easier to
maintain, and less expensive
– Typically a features vs. reliability trade-off
• Take advantage of vendor relationships
– Provide recommendations for standard services
– Let multiple vendors compete for your business
– Understand where the product is going

2018/2019 IS 7212: Advanced Networking and Sys 72

tem Administration by Dr. Samuel Asfer
 Designing a solid service …
• Machine independence
– Clients should access service using generic name
• e.g., www, calendar, pop, imap, etc.
– Moving services to different machines becomes invisible to
users
– Consider (right from the start) what it will take to move the
service to a new machine
• Supportive environment
– Data center provides power, AC (Air Conditioner), security,
networking
– Only rely on systems/services also found in data center
(within protected environment)
• e.g., don't depend on a service from a PC in the closet (common
building or room)
2018/2019 IS 7212: Advanced Networking and Sys 73
tem Administration by Dr. Samuel Asfer
 Designing a solid service …
• Reliability
– Build on reliable hardware
– Exploit redundancy when available
• Plug redundant power supply into different UPS on different circuit
– Components of service should be tightly coupled
– Why?
• Reduce single points of failure
– e.g., all on same power circuit, network switch, etc.
• Includes dependent services
– e.g., authentication, authorization, DNS, etc.
– Make service as simple as possible
– Independent services on separate machines, when
possible
2018/2019 IS 7212: Advanced Networking and Sys 74
tem Administration by Dr. Samuel Asfer
 Designing a solid service …
• Restrict access
– Customers should not need
physical access to servers
• Fewer people -> more stable,
more resources, more secure
– Eliminate any unnecessary
services on server (security)
– Centralization and standards
• Building a service = centralizing management of service
• May be desirable to standardize the service and
centralize within the organization as well
– Makes support easier, reducing training costs
– Eliminates redundant resources
2018/2019 IS 7212: Advanced Networking and Sys 75
tem Administration by Dr. Samuel Asfer
 Designing a solid service …
• Performance
– If a service is deployed, but slow, it is unsuccessful
– Need to build in the ability to scale
• Can't afford to build servers for service every year
• Need to understand how the service can be split across
multiple machines if needed
– Estimate capacity required for production (and get
room for growth)
– First impression of user base is very difficult to correct
– When choosing hardware, consider whether service is
likely
• Disk I/O, memory, or network limited

2018/2019 IS 7212: Advanced Networking and Sys 76

tem Administration by Dr. Samuel Asfer
 Designing a solid service
• Monitoring
– Helpdesk or front-line support must
be automatically alerted to problems
– Customers that notice major
problems before sysadmins are
getting poor service
– Need to monitor for capacity
planning as well
• Service roll-out
– First impressions
• Have all documentation available
• Helpdesk fully trained
• Use slow roll-out
2018/2019 IS 7212: Advanced Networking and Sys 77
tem Administration by Dr. Samuel Asfer
• Introduction
• Managing Workstations / Desktops/, Managing Servers,
Managing Services
• Booting
• Processes
• Disks, Partitions, Volumes, File systems, Files
• Printing
• Accounts
• Service Monitoring
• People, Help Desk, Debagging
• Directories
• Data Centers
• Namespaces
• DNS
• Networking
• Network Architecture
2018/2019 IS 7212: Advanced Networking and Sys 78
tem Administration by Dr. Samuel Asfer
 Booting
• System startup and shutdown
– Bootstrapping
– Booting PCs
– Boot loaders
– Booting into single user mode/network mode
– Startup scripts
– Rebooting and shutting down

2018/2019 IS 7212: Advanced Networking and Sys 79

tem Administration by Dr. Samuel Asfer
 Bootstrapping
i.e., starting the computer
• System is particularly vulnerable to
errors
• Steps in boot process
– Execution of boot code in ROM
– Loading and initialization of kernel
– Device detection and configuration
– Creation of spontaneous system processes
– Operator intervention (manual boot only)
– Execution of system startup scripts

2018/2019 IS 7212: Advanced Networking and Sys 80

tem Administration by Dr. Samuel Asfer
Booting PCs
• PC starts by executing
code in ROM (the BIOS)
– Usually BIOS has a
configuration mode with
special keypress during
boot
– Tries to load first 512B of
the boot disk – the MBR (Master Boot Record)
– MBR contains program to specify which partition
from which to load the secondary boot program
(the “boot loader”)
2018/2019 IS 7212: Advanced Networking and Sys 81
tem Administration by Dr. Samuel Asfer
 Boot loaders
• Load and start the kernel
– Could be one of many
kernels or OSes!
– MBR set to load the
master boot loader
– Each disk partition can
have its own second stage loader
– LILO is an older Linux boot loader
– GRUB is the modern Linux boot loader
• Supports most OSes, not just Linux

2018/2019 IS 7212: Advanced Networking and Sys 82

tem Administration by Dr. Samuel Asfer
 Example of multi-boot laptop
GRUB
#
# grub.conf generated by anaconda
# Note that you do not have to rerun grub after making changes
# to this file
# NOTICE: You have a /boot partition. This means that
# all kernel and initrd paths are relative to /boot/, eg.
# root (hd0,2)
# kernel /vmlinuz-version ro root=/dev/hda6
# initrd /initrd-version.img
# boot=/dev/hda
default=1
timeout=10
splashimage=(hd0,2)/grub/splash.xpm.gz
title Red Hat Linux (2.4.20-8)
root (hd0,2)
kernel /vmlinuz-2.4.20-8 ro root=LABEL=/
initrd /initrd-2.4.20-8.img
title Microsoft XP
rootnoverify (hd0,1)
chainloader +1
2018/2019 IS 7212: Advanced Networking and Sys 83
tem Administration by Dr. Samuel Asfer
 Rebooting and shutting down
• Not needed as often as in consumer OSes
– Needed for
• Adding or removing hardware
• Change to boot configuration
– Including new kernel
– Ways to reboot or shutdown:
• Use the shutdown command
• Use the halt and reboot commands
• Use poweroff to tell system to turn off
• Use hardware reset switch or turn off power (last
resort!)
2018/2019 IS 7212: Advanced Networking and Sys 84
tem Administration by Dr. Samuel Asfer
 Turning off power
• Turning off power can cause data loss and
leave filesystem in an inconsistent state
– Linux (and other modern OS) filesystems
buffer changes in memory, and only
sporadically write them back to disk
• Makes disk I/O faster, but more sensitive to loss
– Uninterrupted power is important

2018/2019 IS 7212: Advanced Networking and Sys 85

tem Administration by Dr. Samuel Asfer
 Shutdown
• shutdown command is safest, considerate, and most
thorough to halt, reboot, or change to single user
mode
• shutdown can wait before bringing down system
– sends warning messages (like wall) to logged-in users
• – should explain why, and when it is coming back
• Can specify whether to halt, or reboot:
• shutdown -r +15 “Rebooting to fix NFS”

2018/2019 IS 7212: Advanced Networking and Sys 86

tem Administration by Dr. Samuel Asfer
 Halt; reboot
• halt
– called by shutdown -h
– logs the shutdown
– kills non-essential processes
– executes sync
• waits for filesystem to finish writes
• halts the kernel
• reboot
– called by shutdown -r
– similar to halt, but tells kernel to reboot system
2018/2019 IS 7212: Advanced Networking and Sys 87
tem Administration by Dr. Samuel Asfer
 telinint; poweroff
• telinit
– Directs init to go to a specific run level
• telinit 1 – takes system to single-user mode
• poweroff
– Identical to halt, but adds request to
power management system to turn off
system's power

2018/2019 IS 7212: Advanced Networking and Sys 88

tem Administration by Dr. Samuel Asfer
 Key Points
Boot Steps
1. BIOS
2. Bootloader (GRUB)
3. Kernel
4. init
init is PID 1, parent of all processes, started by kernel
– Uses rc command to run all scripts in /etc.rcN.d dir where N is
runlevel
– RHEL: use chkconfig to configure, /etc/sysconfig for options
Run levels
1 = Single user mode (use to fix problems)
3 = multiuser + network (servers)
5 = multiuser + network + GUI (workstations)

2018/2019 IS 7212: Advanced Networking and System 89

Administration by Dr. Samuel Asferaw
• Introduction
• Managing Workstations / Desktops/, Managing Servers,
Managing Services
• Booting
• Processes
• Disks, Partitions, Volumes, File systems, Files
• Printing
• Accounts
• Service Monitoring
• People, Help Desk, Debagging
• Directories
• Data Centers
• Namespaces
• DNS
• Networking
• Network Architecture
2018/2019 IS 7212: Advanced Networking and Sys 90
tem Administration by Dr. Samuel Asfer
 Processes
• Controlling Processes
– Components of a process
– Life cycle of a process
– Signals
– Send signals using kill and killall
– Process states
– Influence scheduling priority with nice and renice
– Monitoring processes with ps and top
– Runaway processes
– Periodic processes
2018/2019 IS 7212: Advanced Networking and Sys 91
tem Administration by Dr. Samuel Asfer
 Components of a process
• A process is the instantiation of a program
• From the kernel's perspective, a process is:
– An address space (the set of memory pages with
code, libraries, and data)
– Set of data structures (within the kernel)
– The process's address space map
• Current status
• Execution priority
• Resources used
• Signal mask (which signals are blocked)
• The owner
• Which instructions are currently being executed
2018/2019 IS 7212: Advanced Networking and Sys 92
tem Administration by Dr. Samuel Asfer
 Process attributes
• Process ID – PID
– Unique identifier, wraps around
• Parent PID – PPID
– When a process is cloned, there is a parent and a child
• Real and effective user ID – UID and EUID
– EUID is used to determine what permissions the process has
– Also records original EUID (saved UID)
• Can be re-accessed later in program (even after changing EUID)
• Real and effective group ID – GID and EGID
• Niceness
– The CPU time available depends on its scheduling priority
– Users can make their processes 'nicer' to the rest of the
system
2018/2019 IS 7212: Advanced Networking and Sys 93
tem Administration by Dr. Samuel Asfer
 Process life cycle
• An existing process calls fork(2)
– Parent is told PID of child
– Child process is told 0
• Child can use exec (or similar) to start a new
program
• When ready to die, process calls _exit(2) with exit
code
• Parent must wait(2) to collect status of dead
children
– Resource usage, why killed
• 2018/2019
Orphans are re-mapped to init
IS 7212: Advanced Networking and Sys 94
tem Administration by Dr. Samuel Asfer
 Signals
• Signals are process-level interrupt requests
• Uses
– Inter-process communication
– Terminal driver can kill, interrupt or suspend
processes (Ctrl-C, Ctrl-Z)
– Can be sent by admin (with kill) for various purposes
– Can be sent by kernel when process breaks a rule
• e.g., division by zero
– Can be sent by kernel for i/o available, death of child

2018/2019 IS 7212: Advanced Networking and Sys 95

tem Administration by Dr. Samuel Asfer
 Handling signals
• Process can designate a signal handler for a
particular signal
• If no handler, kernel takes some default action
• When handler is finished catching signal,
execution continues where the signal was
received
• Process can request that particular signals be
ignored, or blocked
• If signal is received while blocked, one instance
of that signal is buffered until it is unblocked
2018/2019 IS 7212: Advanced Networking and Sys 96
tem Administration by Dr. Samuel Asfer
 Important signals
# Name Description Default Catch? Block? Dump?
1 HUP Hangup Terminate Yes Yes No
Reset request; clean up process on terminal (modem hangup)
*csh processes ignore HUP; bash users need nohup command
2 INT Interrupt Terminate Yes Yes No
Control-C, can catch and clean up before quiting.
3 QUIT Quit Terminate Yes Yes Yes
Similar to TERM, but generates a core dump
9 KILL Kill Terminate No No No
Never received by process; OS terminates process.
* BUS Bus error Terminate Yes Yes Yes
Error signal. Typically a memory alignment problem.
11 SEGV Segmentation Fault Terminate Yes Yes Yes
Error signal. Typically a memory access to protected space.
2018/2019 IS 7212: Advanced Networking and Sys 97
tem Administration by Dr. Samuel Asfer
 Sending signals
# kill [-signal] pid
# kill sends TERM signal by default
# kill -9 pid === kill -KILL pid
“Guarantees” that the process will die
# kill -USR1 910 3044
# sudo killall -USR1 httpd
killall removes need for pid

2018/2019 IS 7212: Advanced Networking and Sys 98

tem Administration by Dr. Samuel Asfer
 Process States
• Process exist in one of four states:
– Runnable – can be executed
– Sleeping – waiting for some resources
• Gets no CPU time until resource is available
– Zombie – trying to die (parent hasn't
waited)
– Stopped – process is suspended (i.e., not
permitted to run)
• Like sleeping, but can't wake until CONT
received
2018/2019 IS 7212: Advanced Networking and Sys 99
tem Administration by Dr. Samuel Asfer
 Monitoring processes: ps
• /bin/ps primary tool
• Shows
– PID, UID, priority, control terminal
– Memory usage, CPU time, status
• Multiple variations of ps
– ps -aux (BSD, Linux)
– ps -Af (Solaris)
2018/2019 IS 7212: Advanced Networking and Sys 100
tem Administration by Dr. Samuel Asfer
 Monitoring processes: top
• /usr/bin/top is optional in some OSes
• Shows top-n CPU-using processes
– Plus other stats, like memory usage and
availability, system load
– Can renice within top
– Automatically refreshes screen every 5
seconds
– Can focus on a particular user

2018/2019 IS 7212: Advanced Networking and Sys 101

tem Administration by Dr. Samuel Asfer
 Runaway processes
• What can you do about processes using an
unusual amount of resources (memory, CPU,
disk space)?
– Identify resource hogs using top and/or ps
– Contact owner and ask about resource usage
– Suspend using STOP signal (might break job)
• Contact owner, restart or kill later
– Renice CPU hog

2018/2019 IS 7212: Advanced Networking and Sys 102

tem Administration by Dr. Samuel Asfer
 Creating periodic processes
• Automation, as you've heard, is key to
efficiency
• Instead of manually performing tasks daily,
weekly, or monthly, you can schedule
them
– cron
– anacron
• Includes tasks like:
– monitoring, log rotation, backups, file
distribution
2018/2019 IS 7212: Advanced Networking and Sys 103
tem Administration by Dr. Samuel Asfer
 cron
• cron daemon performs tasks at scheduled times
• crontab files are examined by cron for schedule
• /etc/crontab, /etc/cron.d/*, /var/spool/cron/*
• cron wakes up each minute and checks to see if anything
needs to be executed
• cron is susceptible to changes in time
– doesn't compensate for when machine is down, or time
changes (clock adjustments or daylight savings time) that are
sufficiently large (3 hours, at least for some implementations)
• anacron works daily
– records when task last performed, and will catch up with
missing time

2018/2019 IS 7212: Advanced Networking and Sys 104

tem Administration by Dr. Samuel Asfer
• Introduction
• Managing Workstations / Desktops/, Managing Servers,
Managing Services
• Booting
• Processes
• Disks, Partitions, Volumes, File systems, Files
• Printing
• Accounts
• Service Monitoring
• People, Help Desk, Debagging
• Directories
• Data Centers
• Namespaces
• DNS
• Networking
• Network Architecture
2018/2019 IS 7212: Advanced Networking and Sys 105
tem Administration by Dr. Samuel Asfer
• Disks
• Partitions
• Volumes
• Filesystems
• Directories

2018/2019 IS 7212: Advanced Networking and Sys 106

tem Administration by Dr. Samuel Asfer
 SCSI:
Small Computer Systems Interface
• Many versions
• SCSI-1 (1986) 8-bits, 5MB/s
• SCSI-2 (1990) added command
queuing, …
• Fast SCSI-2 8-bits, 10MB/s
• Fast/wide SCSI-2 16-bits, 20MB/s
• Ultra SCSI 8 bits, 20MB/s
• Wide Ultra SCSI 16bits, 40MB/s
• Wide Ultra2 SCSI 16bits, 80MB/s
• Wide Ultra3 SCSI 16bits, 160MB/s
• Ultra-320, Ultra-640 SCSI
2018/2019 IS 7212: Advanced Networking and Sys 107
tem Administration by Dr. Samuel Asfer
 Disk interfaces
• Relatively few
– SCSI (pronounced “scuzzy”)
• Common, widely supported
– IDE a.k.a. ATA or PATA, and SATA
• Inexpensive, simple
– Fibre Channel
• High bandwidth, lots of simultaneous devices
• Supports up to 16Gbit
– Universal Serial Bus (USB)
• Typically used for slow devices (e.g., CD-ROMs,
portable, removable drives)
2018/2019 IS 7212: Advanced Networking and Sys 108
tem Administration by Dr. Samuel Asfer
 Adding a disk to Linux
STEP-BY-STEP
• Install new hardware
– verify that hardware is recognized by BIOS or controller
• Boot, make certain device files already exist in /dev
e.g., /dev/sdc
• Use fdisk/parted (or similar) to partition the drive
– Verify the system type on each partition
• Use mke2fs (-t ext4) on each regular partition
– To create (an ext4) filesystem
• Use mkswap to initialize swap partitions
• Add entries to /etc/fstab
• Then, reboot to verify everything
2018/2019 IS 7212: Advanced Networking and Sys 109
tem Administration by Dr. Samuel Asfer
 Disk partitions
• Drives are divided into one or more partitions that
are treated independently
– Partitions make backups easier, confine damage
• Typically have at least two or three partitions:
– root partition (one)
• everything needed to bring system up in single-user mode
(often copied onto another disk for emergencies)
– swap partition (at least one)
• stores virtual memory when physical memory is insufficient
– user partition(s)
• home directories, data files, etc.
– boot partition - boot loader, kernel, etc.
2018/2019 IS 7212: Advanced Networking and Sys 110
tem Administration by Dr. Samuel Asfer
 Logical Volumes
• Partitions are static, and sometimes you want to
change them
• LVM (Linux Logical Volume Manager) lets you
combine partitions and drives to present an
aggregate volume as a regular block device (just
like a disk or partition)
– Use and allocate storage more efficiently
– Move logical volumes among different physical devices
– Grow and shrink logical volume sizes on the fly
– Take “snapshots” of whole filesystems
– Replace on-line drives without interrupting service
– Similar systems are available for other OSes
2018/2019 IS 7212: Advanced Networking and Sys 111
tem Administration by Dr. Samuel Asfer
 Filesystems
• Linux filesystems are created in partitions or volumes
– ext2fs (2nd Extended File System) is old
– ext3fs (3rd Extended File System) is common
• Augments ext2fs to incorporate journaling
– Journals contain filesystem updates
– Journal log can reconstruct consistent filesystem
– Journal speeds filesystem consistency checks
– ext4fs (Fourth Extended File System) is modern
• Speeds large directories
• Compatible with ext2 and ext3
– Other filesystems also supported
• ReiserFS, IBM's JFS, SGI's XFS
– Can read foreign filesystems (e.g., FAT, NTFS, ISO 9660)

2018/2019 IS 7212: Advanced Networking and Sys 112

tem Administration by Dr. Samuel Asfer
fsck: check and repair filesystems
• During power failure, superblock, inodes, and data
blocks may not get written to disk
• fsck can fix minor damage (ext3/4 systems quickly)
– unreferenced inodes
– inexplicably large link counts
– unused data blocks not recorded in block maps
– data blocks listed as free that are also used in a file
– incorrect summary info in superblock
• More complex damage will make fsck ask human
– Places unfixable files in lost+found directory
– You should re-run fsck until no errors are found
2018/2019 IS 7212: Advanced Networking and Sys 113
tem Administration by Dr. Samuel Asfer
 The Filesystem
• A filesystem incorporates:
– A way of naming and organizing things
(namespace)
– A security model for protecting, hiding, and
sharing objects
– An implementation to tie the model to the
hardware
• Linux abstract kernel interface supports
many different filesystems
– from disk, network, memory
2018/2019 IS 7212: Advanced Networking and Sys 114
tem Administration by Dr. Samuel Asfer
 Filesystem hierarchy
http://www.pathname.com/fhs/
• /bin : Essential user command binaries (for use by all users)
• /boot : Static files of the boot loader (e.g., kernel)
• /dev : Device files (terminals, disks, modems, etc.)
• /etc : Host-specific system configuration
• /home : User home directories (optional)
• /lib : Essential shared libraries and kernel modules
• /media : Filesystems on removable media
• /opt : Add-on application software packages
• /proc : Kernel and process information virtual filesystem
• /root : Home directory for the root user (optional)
• /sbin : Static system binaries for repairing, booting, & recovering OS
• /tmp : Temporary files (that disappear at reboot)
• /usr : (more next slide)
• /var : (more next slide)
2018/2019 IS 7212: Advanced Networking and Sys 115
tem Administration by Dr. Samuel Asfer
 /usr, /var
• /usr
• /usr/bin : Most commands and executables
• /usr/include : Header files for C programs
• /usr/lib : Libraries and support files for standard programs
• /usr/local : Local software (stuff you install)
• /usr/man : Manual pages
• /usr/sbin : Less essential sysadmin commands
• /usr/share : Content that is common to multiple systems (RO)
• /usr/src : Source code for (nonlocal) software packages
• /var
• /var/adm : Various logs, system setup records
• /var/log : System log files
• /var/spool : Spooling directories for printers, mail, dns
• /var/tmp : More temporary space (preserved between reboots)
2018/2019 IS 7212: Advanced Networking and Sys 116
tem Administration by Dr. Samuel Asfer
 Directories
• Created with mkdir, deleted with rmdir (if
empty) or rm -r
• Contains named references (links) to other
files
• Special entries “.” and “..” refer to self and
parent directories respectively
• Filenames are stored within parent directory
• More than one directory entry can refer to the
same file (hard links)
– Can be created with ln, removed with rm
2018/2019 IS 7212: Advanced Networking and Sys 117
tem Administration by Dr. Samuel Asfer
• Introduction
• Managing Workstations / Desktops/, Managing Servers,
Managing Services
• Booting
• Processes
• Disks, Partitions, Volumes, File systems, Files
• Printing
• Accounts
• Service Monitoring
• People, Help Desk, Debagging
• Directories
• Data Centers
• Namespaces
• DNS
• Networking
• Network Architecture
2018/2019 IS 7212: Advanced Networking and Sys 118
tem Administration by Dr. Samuel Asfer
 Printing

• Printing and print services

• Printing policies and architecture
• Printing terms
• Types of printers
• LPD, LPRng, CUPS
• Adding a printer
• Common printing software

2018/2019 IS 7212: Advanced Networking and Sys 119

tem Administration by Dr. Samuel Asfer
 Print services
• People depend on print services
– for contracts
– for proofreading
– for quizzes
– for reading long material
that is less pleasant to
read on-screen
• Print is a utility
– It should always work

2018/2019 IS 7212: Advanced Networking and Sys 120

tem Administration by Dr. Samuel Asfer
 Where should printers be located?

• Some want a printer on their

own desk
– Very convenient but expensive
• Some want to be able to print
to any printer, no matter where it is
– Flexible, able to borrow specialty printers as needed
• Finance people want to centralize everything
– A single high-speed printer, single high-quality printer, and one
color printer per building (most cost-effective)
• Others want to charge every expense
– Regardless of how much is out there, those who use it, pay for
it
2018/2019 IS 7212: Advanced Networking and Sys 121
tem Administration by Dr. Samuel Asfer
 Real world
• People need to be able to print to any printer
they have permission to use
• Centralized printing services can save
money
– Ten people who might otherwise buy slow, low-
quality personal printers for $50-150, without
support contracts, can buy a single high-quality,
fast shared printer with long-term maintenance
• Plus the sysadmin only has to support one printer
driver/printer rather than 10

2018/2019 IS 7212: Advanced Networking and Sys 122

tem Administration by Dr. Samuel Asfer
 Print architecture
• How centralized will printing be?
– How many people will share a printer for
general printing?
– Who qualifies for a personal printer?
– How will they be networked?
– How will they be maintained?
– How will they be paid for?

2018/2019 IS 7212: Advanced Networking and Sys 123

tem Administration by Dr. Samuel Asfer
 Print architecture (cont.)
• Who orders supplies and resupplies the printers?
– Are the printers re-supplied when they are out (and
users complain), or does someone visit them
regularly?
• What kinds of printing technologies will be
supported?
– Postscript/PCL/PDF
– Duplex printing
– Laser vs. InkJet
– LPD over IP vs. SMB, USB or parallel, etc.
• How will the printers be named?
2018/2019 IS 7212: Advanced Networking and Sys 124
tem Administration by Dr. Samuel Asfer
 Print system architecture
• Peer-to-peer
– All hosts spool jobs directly to the destination printer
– Simplest, but all clients must know current printer IP/name
– Limited by printer spool memory
• Central funnel
– Hosts send print jobs to a central server which
distributes
– Can convert formats
– Can provide access control
– Can collect per-page billing
– Can intelligently select printers
– Single place for printer drivers
2018/2019 IS 7212: Advanced Networking and Sys 125
tem Administration by Dr. Samuel Asfer
 Printing terms
• spooler
– Daemon that receives • RIP
print jobs, stores, – Raster Image Processor
prioritizes, and sends – Accepts PDL input,
them sequentially to be generates bitmap
printed appropriate for a
• PDL particular device
– Page Description • filters
Language, usually device – Modify print jobs on their
and resolution way to a printer
independent
• PostScript
– PostScript, PCL, PDF
– Most common PDL –
• bitmap
also a full programming
– JPEG, TIFF, GIF, PNG language
2018/2019 IS 7212: Advanced Networking and Sys 126
tem Administration by Dr. Samuel Asfer
 Types of Printers
• Classified by connection interface
– Serial and parallel printers
• USB faster and the default today for personal printers
– Network printers
• Contain network interfaces (e.g., ethernet or wifi)
• Accept jobs via one or more printing protocols
– including via LPD, CIFS, IPP, HP JetDirect

• Classified by type of data

– PostScript is well-supported under Linux/UNIX
– Non-postscript printers require special software to
convert to unique PDL (vendor supplied, or
ghostscript)
2018/2019 IS 7212: Advanced Networking and Sys 127
tem Administration by Dr. Samuel Asfer
 Print Server Packages
(LPD, LPRng, CUPS)
• LPD is the old standard
– Not found on current distributions
• LPRng
– Designed for backwards compatibility with
Berkeley and System V printing systems
– Was common ages ago (default for Red Hat 7.3),
but is now replaced by...
• CUPS – Common UNIX Printing System
– Standard on modern distributions (our focus)
– Now owned and maintained by Apple
2018/2019 IS 7212: Advanced Networking and Sys 128
tem Administration by Dr. Samuel Asfer
 CUPS
• Common UNIX Printing System
– Latest printing system
• Also supports secure printing (SSL, etc.)
• Implements IPP: Internet Printing Protocol
(HTTP-based)
• Supports load-balancing across a class of
printers
• Supports automatic network configuration
• Standard in most Linux distributions
2018/2019 IS 7212: Advanced Networking and Sys 129
tem Administration by Dr. Samuel Asfer
 Adding a printer in CUPS
• From command line:
• lpadmin -p fezmo -E -v socket://192.168.0.12 -m
laserjet.ppd
• lpadmin -p groucho -E -v parallel:/dev/lp0 -m
pxlcolor.ppd
• From browser:
• http://localhost:631/admin
– Even works on Macs!
• From Red Hat/CentOS
– Command line: system-config-printer
– GUI: System->Administration->Printing
2018/2019 IS 7212: Advanced Networking and Sys 130
tem Administration by Dr. Samuel Asfer
• Introduction
• Managing Workstations / Desktops/, Managing Servers,
Managing Services
• Booting
• Processes
• Disks, Partitions, Volumes, File systems, Files
• Printing
• Accounts
• Service Monitoring
• People, Help Desk, Debagging
• Directories
• Data Centers
• Namespaces
• DNS
• Networking
• Network Architecture
2018/2019 IS 7212: Advanced Networking and Sys 131
tem Administration by Dr. Samuel Asfer
 Accounts
• User accounts • Root powers
– The /etc/passwd file – Ownership of files and
– The /etc/shadow file processes
– The /etc/group file – The superuser
– Adding users – Choosing a root
– password
Removing users
– Becoming root
– Disabling logins
– Other pseudo-users
– Account management
utilities

2018/2019 IS 7212: Advanced Networking and Sys 132

tem Administration by Dr. Samuel Asfer
 The /etc/passwd file
• /etc/passwd lists all recognized users
and contains:
– login name
– encrypted password (unless /etc/shadow used)
– UID number
– default GID number
– full name, office, extension,
– home phone (optional)
– home directory
– login shell
2018/2019 IS 7212: Advanced Networking and Sys 133
tem Administration by Dr. Samuel Asfer
 Login name
• Syntax
– usernames must be unique
• <= 32 chars
(old systems/NIS: limit 8 chars)
– any characters except newlines and colons
• Recommendations
– use lower case (even though case sensitive)
– choose easy to remember
– avoid nicknames
2018/2019 IS 7212: Advanced Networking and Sys 134
tem Administration by Dr. Samuel Asfer
 Passwords

2018/2019 IS 7212: Advanced Networking and Sys 135

tem Administration by Dr. Samuel Asfer
 UID number
• In Linux, UIDs are unsigned 32-bit integers
• Root is (almost always) UID 0
• Avoid recycling UIDs, Why ?
– Old files, backups are identified by UID
• Preserve unique UIDs across org
– helpful for consistency across network
filesystems

2018/2019 IS 7212: Advanced Networking and Sys 136

tem Administration by Dr. Samuel Asfer
 Adding users
• For small installations, adding users is simple
– Have user sign and date user agreement
– Create user account with useradd
– Set password with passwd
– Change defaults with usermod

2018/2019 IS 7212: Advanced Networking and Sys 137

tem Administration by Dr. Samuel Asfer
 Steps to add a user (1)
• Edit the /etc/passwd and /etc/shadow files to
define account
– Use vipw to lock and edit with $EDITOR
• Set an initial password
# passwd user
• Create, chown, and chmod the user's home
directory
# mkdir /home/staff/tyler
# chown tyler.staff /home/staff/tyler
# chmod 700 /home/staff/tyler
2018/2019 IS 7212: Advanced Networking and Sys 138
tem Administration by Dr. Samuel Asfer
 Steps to add a user (2)
• Copy default startup files to the user's home
directory
– bash
• .bashrc, .bash_profile
– csh/tcsh
• .login, .cshrc, .logout
– X-windows
• .Xdefaults, .Xclients, .xsession
– Need to create and store default files!

2018/2019 IS 7212: Advanced Networking and Sys 139

tem Administration by Dr. Samuel Asfer
 Steps to add a user (3)
• Copy files to new directory
# cp /etc/skel/.[a-zA-Z]* ~tyler
# chmod 644 ~tyler/.[a-zA-Z]*
# chown tyler ~tyler/.[a-zA-Z]*
# chgrp staff ~tyler/.[a-zA-Z]*

2018/2019 IS 7212: Advanced Networking and Sys 140

tem Administration by Dr. Samuel Asfer
 Steps to add a user (4)
• Edit /etc/group file
– Add to relevant groups
• Might set disk quotas with edquota
• Verify new login
– log in as new user
– execute pwd and ls -la
• Notify new user of account and
initial password
• Record account status and
contact information
2018/2019 IS 7212: Advanced Networking and Sys 141
tem Administration by Dr. Samuel Asfer
 Removing users
• Generally with userdel
– Set disk quota to zero
– Remove user from local databases or phone lists
– Remove from aliases file (or add forwarding)
– Remove crontab file and any pending at jobs
– Kill any running processes
– Remove temporary files in /var/tmp or /tmp
– Remove from passwd, and group files
– Remove home directory (backup first) and mail
spool
2018/2019 IS 7212: Advanced Networking and Sys 142
tem Administration by Dr. Samuel Asfer
 Disabling logins

• Sometimes you need to

temporarily disable a login
• Can't just put a star in front of encrypted pw
– Might still be able to log in via network w/out pw
• Current practice
– Replace shell with program explaining status and
instructions on how to fix

2018/2019 IS 7212: Advanced Networking and Sys 143

tem Administration by Dr. Samuel Asfer
 Account management utilities
• Basic utilities
– useradd – adds to passwd and shadow files
– usermod – changes existing passwd entry
– userdel – remove user, opt. delete home dir
– groupadd, groupmod, groupdel operate on
/etc/group
– Common to write custom adduser and rmuser
scripts

2018/2019 IS 7212: Advanced Networking and Sys 144

tem Administration by Dr. Samuel Asfer
The superuser
• The root account has UID of 0
– Can change the name and create other users
with same UID; neither recommended
• The superuser (any process with effective
UID 0) can perform any valid operation on
any file or process.
• All other users are “normal”

2018/2019 IS 7212: Advanced Networking and Sys 145

tem Administration by Dr. Samuel Asfer
 Restricted operations
• Superuser privileges are required for:
– Changing the root directory of a process with chroot
– Creating device files
– Setting the system clock
– Raising resource usage limits and process priorities
– Setting the system's hostname
– Configuring the network interfaces
– Opening privileged network ports (<= 1024)
– Shutting down the system
– Changing process UID and GID (only one way)
• Example: login
2018/2019 IS 7212: Advanced Networking and Sys 146
tem Administration by Dr. Samuel Asfer
 Choosing a root password
• Any password? Not if you
want it to be difficult to crack.
• Should be
– At least eight characters
(more may not be helpful)
– Not easily guessed or found
by trial and error
– Memorable (so you don't need to write it down)
– A seemingly random sequence of letters, digits, & punctuation
– Shocking nonsense!
– Memorable, unguessable, unique, undisclosed

2018/2019 IS 7212: Advanced Networking and Sys 147

tem Administration by Dr. Samuel Asfer
 Changing the root password
• Should be performed
– At least every three months
– Every time someone who
might know the password
leaves the site
– Whenever you think security
might be compromised
– On a day when you will
remember the new pw!
2018/2019 IS 7212: Advanced Networking and Sys 148
tem Administration by Dr. Samuel Asfer
 Being root
• Responsibilities!
– Do not give out root password
– Do not create new accounts with UID 0
– Use root account for admin work only
– Change root password often
– Do not leave root shell unattended
– Be extra careful!
– Perhaps more, depending on policies at
location
2018/2019 IS 7212: Advanced Networking and Sys 149
tem Administration by Dr. Samuel Asfer
 su
• su: substitute user identity (switch users)
– Without args, su prompts for root password
and then
starts root shell
– Logs who became root
– Use “su –” to execute new user's shell
• Otherwise new PATH is not established
– Good idea to use full pathname to su
• Linux: /bin/su
• Solaris: /sbin/su
2018/2019 IS 7212: Advanced Networking and Sys 150
tem Administration by Dr. Samuel Asfer
 sudo
• sudo: a limited su
– When you want to provide limited root-
privileges
– sudo <program to be executed>
• Checks /etc/sudoers for authorization
• Asks for user's password
• Logs command executed, person, time, and
directory
• Executes command
• Additional sudo commands can be executed
without password for another five minutes
2018/2019 IS 7212: Advanced Networking and Sys 151
tem Administration by Dr. Samuel Asfer
 sudo advantages
• Accountability – commands are logged
• Operators can do chores without root privileges
• Real root password can be known to very few people
• sudo is faster to use than su or logging in as root
• Privileges can be revoked without changing root pw
• A complete list of users with root is maintained
• Less chance of a root shell being
left unattended
• A single file can control access for
an entire network
2018/2019 IS 7212: Advanced Networking and Sys 152
tem Administration by Dr. Samuel Asfer
 Service Monitoring

• Reading Assignment

2018/2019 IS 7212: Advanced Networking and Sys 153

tem Administration by Dr. Samuel Asfer
2018/2019 IS 7212: Advanced Networking and Sys 154
tem Administration by Dr. Samuel Asfer
• Introduction
• Managing Workstations / Desktops/, Managing Servers,
Managing Services
• Booting
• Processes
• Disks, Partitions, Volumes, File systems, Files
• Printing
• Accounts
• Service Monitoring
• People, Help Desk, Debagging
• Directories
• Data Centers
• Namespaces
• DNS
• Networking
• Network Architecture
2018/2019 IS 7212: Advanced Networking and Sys 155
tem Administration by Dr. Samuel Asfer
 People
Topics
1. Organizations
2. Perception and Visibility
3. Time Management

2018/2019 IS 7212: Advanced Networking and Sys 156

tem Administration by Dr. Samuel Asfer
 Sizing
• Questions
– How many people do you need?
– Which people do you need?
• Categories
– Customer/desktop support
– Customer server support
– Infrastructure support

2018/2019 IS 7212: Advanced Networking and Sys 157

tem Administration by Dr. Samuel Asfer
 Scaling the Organization
• Small Company
– 1-2 SAs, no formal helpdesk
– Need helpdesk software at around 20 employees
– helpdesk software: keep track of user requests and deal with
other customer-care-related issues
• For asset management or IT service management
• Medium Company
– Need formal helpdesk by 1000 employees.
– SAs specialize by OS or task (network, security)
• Large Company
– Well-staffed helpdesk with 2nd tier support team.
– Highly specialized SAs.
– Formal communication paths.
2018/2019 IS 7212: Advanced Networking and Sys 158
tem Administration by Dr. Samuel Asfer
 Infrastructure Teams
• Infrastructure must be consistent and
interoperable across different sites.
– Routing
– Authentication
– Email
• Infrastructure teams may be centralized or
distributed across sites.

2018/2019 IS 7212: Advanced Networking and Sys 159

tem Administration by Dr. Samuel Asfer
 Customer Service
• Centralized Support
– Good automation improves support.
– Better communication w/ SAs than customers.
• De-centralized Support
– Better communication with customer base.
– Handling requests may dominate infrastructure needs.
• Centralize help desk contact.
– Have single phone number to call.
– Have single e-mail address / request tracking.
– Each large site needs its own physical help desk.

2018/2019 IS 7212: Advanced Networking and Sys 160

tem Administration by Dr. Samuel Asfer
 Outsourcing
• When should you outsource?
– Small organization.
– Basic computing needs.
• When should you not outsource?
– Large organization.
– Need for high availability systems.
– Need for high levels of security.
– Internet site generates revenue.

2018/2019 IS 7212: Advanced Networking and Sys 161

tem Administration by Dr. Samuel Asfer
 Consultants and Contractors
• Consultants
– Bring in outside expertise your SAs don’t have.
– Useful for introducing new technologies.
– Must work with in-house SAs to ensure
maintainability of new services.
• Contractors
– Perform same tasks as current in-house SAs.
– Use contractors to allow SAs to work on new
development projects.
2018/2019 IS 7212: Advanced Networking and Sys 162
tem Administration by Dr. Samuel Asfer
 Perception and Visibility

• Perception: A qualitative measure of how

people see you.

• Visibility: A quantitative measure of

much people see you.

2018/2019 IS 7212: Advanced Networking and Sys 163

tem Administration by Dr. Samuel Asfer
 Perception is Reality
• If customers don’t know you exist, you
don’t exist.
• If customers can’t see evidence of your
activities, they’ll assume you’re doing
nothing.
• If it takes a long time to resolve
customer requests, customers will
assume you’re lazy or incompetent.

2018/2019 IS 7212: Advanced Networking and Sys 164

tem Administration by Dr. Samuel Asfer
 First Impressions
• Ensure customer is set up for first day.
– PC is installed and configured.
– Account is created.
– Customer knows where to get support.
• Meeting customers
– Be on time.
– Be polite, friendly.
– Listen.

2018/2019 IS 7212: Advanced Networking and Sys 165

tem Administration by Dr. Samuel Asfer
 Attitude
• Who do you support?
• Align your priorities with their expectations
– Satisfy “small” requests quickly.
– Prioritize larger requests.
• Customers aren’t always right
– Some requests have unexpected impacts.
– Others are too expensive, time-consuming.

2018/2019 IS 7212: Advanced Networking and Sys 166

tem Administration by Dr. Samuel Asfer
 Be an Advocate
• “System clerk”
– Do requests when received.
– Do infrastructure work when asked by mngr.
• Advocate
– Automate simple tasks.
– Advocate customer needs to manager.
– Involved in planning new projects.

2018/2019 IS 7212: Advanced Networking and Sys 167

tem Administration by Dr. Samuel Asfer
 Visibility Paradox
• SAs aren’t noticed when everything works.
• 100% uptime takes tremendous effort.
• SAs are noticed when something breaks.
• SA becomes a hero by fixing down server,
etc.

2018/2019 IS 7212: Advanced Networking and Sys 168

tem Administration by Dr. Samuel Asfer
 Managing your Visibility
• System Status Web Page
• Make it your customers home page.
• Include useful content so they keep using it.
• Announce down services so customers know you’re
on top of the situation.
• Management meetings
• Meet with customer managers to inform them of the
status of projects relevant to their groups.
• Learn their needs and priorities.
• Newsletters
• Regular updates, etc.

2018/2019 IS 7212: Advanced Networking and Sys 169

tem Administration by Dr. Samuel Asfer
 Time Management
• Why is it hard?
• SA is interrupt-driven work.
• Convert interrupts into requests.
• Follow through
• Resolve all requests in a timely fashion.
• Make it to all meetings on time.

2018/2019 IS 7212: Advanced Networking and Sys 170

tem Administration by Dr. Samuel Asfer
 Principles of Time Managment
1. One “database” for time management
information (use one organizer).
2. Conserve your brain power for what's important
3. Develop routines and stick with them (reuse
code libraries; don't reinvent the wheel).
4. Develop habits and mantras (replace runtime
calculations with precomputed decisions).
5. Maintain focus during “project time.”
6. Manage your social life with the same tools you
use for your work life.

2018/2019 IS 7212: Advanced Networking and Sys 171

tem Administration by Dr. Samuel Asfer
 One Database
• Use one organizer
• Digital (PDA , Smart Phone )
• Analog (notebook)
• Take it with you everywhere.

2018/2019 IS 7212: Advanced Networking and Sys 172

tem Administration by Dr. Samuel Asfer
 Conserve Brain Power
• Write it down
• Use request tracking system for requests.
• Use organizer for other to-do items.
• Use wiki or notes file for instructions.
• Memory is fallible.
• You will forget.
• You will waste time worrying about what you
forgot.

2018/2019 IS 7212: Advanced Networking and Sys 173

tem Administration by Dr. Samuel Asfer
 Develop Routines

• Do your daily planning every morning.

• Do backups on a regular schedule.
• Re-use code from previous scripts.

2018/2019 IS 7212: Advanced Networking and Sys 174

tem Administration by Dr. Samuel Asfer
 Habits
• Trust your processes.
• Don’t skip planning on busy days.
• Don’t put off essential tasks on hectic
days.

2018/2019 IS 7212: Advanced Networking and Sys 175

tem Administration by Dr. Samuel Asfer
 Maintain Focus
• Ensure you’re not interrupted when
working on projects.
• Deal with interrupts appropriately
• Resolve critical issues as needed.
• File requests for other issues.

2018/2019 IS 7212: Advanced Networking and Sys 176

tem Administration by Dr. Samuel Asfer
 One System
• Manage your social life in your organizer
too.
• Avoids conflicts between work/social events.
• Ensures that you do something other than
work.

2018/2019 IS 7212: Advanced Networking and Sys 177

tem Administration by Dr. Samuel Asfer
 Daily Planning
• Start the day by planning
• Review to-do list.
• Prioritize tasks.
• Schedule tasks.
• Should only take about 5 minutes.

2018/2019 IS 7212: Advanced Networking and Sys 178

tem Administration by Dr. Samuel Asfer
 Touch All Paper Once
• Process each mail message completely.
• Throw it away.
• Resolve the problem, then throw it away.
• Respond to it, then throw it away.
• File it.
• Don’t put it in a pile to read later.

2018/2019 IS 7212: Advanced Networking and Sys 179

tem Administration by Dr. Samuel Asfer
 Help Disk
Topics
1. Help Desk Management
2. Help Desk Software
3. Workflow
4. Trend Analysis

2018/2019 IS 7212: Advanced Networking and Sys 180

tem Administration by Dr. Samuel Asfer
 Types of Help Desks
• Physical
• Walk-up counter
• Virtual
• Phone help
• E-mail or web-based request system
• Unofficial
• Visit sysadmin A in morning, B in afternoon.

2018/2019 IS 7212: Advanced Networking and Sys 181

tem Administration by Dr. Samuel Asfer
 Help Desk Size
• Expressed as Customer:SA ratio.
• 50:1 for a R&D organization (Research and
Development organization)
• 100,000:1 for an e-commerce site

• Conflicting interests
• Management wants a larger ratio.
• Customers want a smaller ratio.
• If ratio too large, customers do their own SA.
2018/2019 IS 7212: Advanced Networking and Sys 182
tem Administration by Dr. Samuel Asfer
 Scope of Coverage
• What is supported?
• Hardware, OS, applications, networking.
• How are unsupported platforms handled?
• Who is supported?
• Department, location.
• Where are the customers?
• Customers on the road, customers at home.
• When is the help desk open?
• What do you do about support when it’s not?

2018/2019 IS 7212: Advanced Networking and Sys 183

tem Administration by Dr. Samuel Asfer
 Out-of-scope Issues
• Advocate or refer?
• Advocate issue to another service provider.
• Refer customer to other service provider.
• Time-limited support.
• Offer to help for short fixed time period.
• If can’t fix it within period, then customer is on
their own.

2018/2019 IS 7212: Advanced Networking and Sys 184

tem Administration by Dr. Samuel Asfer
 Escalation Process
• First line support
• Handles 80+% of calls.
• May have time limit before escalation.
• May be working from a script.
• Second line support
• Subject matter experts.
• May sit with first line support periodically.
• Indicates a problem if too many calls
received.
• Manager
2018/2019 IS 7212: Advanced Networking and Sys 185
tem Administration by Dr. Samuel Asfer
 Help Desk Software
• Functionality
• Create, update, and resolve requests.
• Sort and search requests.
• Generate request metrics.
• Knowledge base (customers, PCs).
• Often called
• Request tracker
• Trouble ticket tracking system
• Issue tracking system
2018/2019 IS 7212: Advanced Networking and Sys 186
tem Administration by Dr. Samuel Asfer
 Why Help Desk Software?
• You have to track help requests.
• E-mail doesn’t work
• How do you know someone has replied?
• How do you know what they said?
• How do you know two people haven’t replied?
• What happens when you’re out of town?

2018/2019 IS 7212: Advanced Networking and Sys 187

tem Administration by Dr. Samuel Asfer
 Ticket (Request)
• Ticket also known as (a.k.a) Request
• A report of a customer problem.
• Contains important data on problem:
• Customer who’s having the problem.
• Problem description.
• Problem classification.
• Urgency and importance.
• Notes added by SAs working on problem.
• Status: resolved or not.
2018/2019 IS 7212: Advanced Networking and Sys 188
tem Administration by Dr. Samuel Asfer
 Queues
• Tickets are organized into queues.
• Queues defined by help desk admin.
– Type of problem (net, email, web, print, etc.)
– Location
– Whatever classification makes sense
• Queue features
– Only specified SAs have access to a queue.
– Queues can auto-reply to certain questions.

2018/2019 IS 7212: Advanced Networking and Sys 189

tem Administration by Dr. Samuel Asfer
 Help Desk Software

Open Ticket Request System RT: Request Tracker

Ticket
Liberum Help Desk

Remedy AR System

2018/2019 IS 7212: Advanced Networking and Sys 190

tem Administration by Dr. Samuel Asfer
 OTRS

2018/2019 IS 7212: Advanced Networking and Sys 191

tem Administration by Dr. Samuel Asfer
 Workflow in Help Desk
1. Greeting
2. Problem Identification
a. Classification
b. Problem Statement
c. Problem Reproduction
3. Planning and Execution
a. Solution Proposals
b. Solution Selection
c. Execution
4. Verification
2018/2019 IS 7212: Advanced Networking and Sys 192
tem Administration by Dr. Samuel Asfer
 Greeting
Soliciting issues from customers.

Greeters
1. Front line help desk support (phone, in person)
2. E-mail
3. Web
4. Network monitoring system

2018/2019 IS 7212: Advanced Networking and Sys 193

tem Administration by Dr. Samuel Asfer
 Problem Identification
• Who classifies problem?
• First-line support.
• Customer (phone or web interface).
• Customer should be told of classification.
• Important that customer is involved in
process.
• Ensures customer knows something is being
done about the problem.
• Provides opportunity for customer feedback.
2018/2019 IS 7212: Advanced Networking and Sys 194
tem Administration by Dr. Samuel Asfer
 Problem Statement
• Describes the problem in full detail.
• Usually responsibility of first-line support.
• Often requires more customer interaction.
• “Help, I can’t print!”
• To which printer?
• From which machine?
• Using what application?

2018/2019 IS 7212: Advanced Networking and Sys 195

tem Administration by Dr. Samuel Asfer
 Reproducing the Problem
• If you can’t reproduce it, you can’t fix it.
• May require access to customer PC.
• Can be difficult over phone/email.
• Record method used to reproduce
problem.

2018/2019 IS 7212: Advanced Networking and Sys 196

tem Administration by Dr. Samuel Asfer
 Solution Selection
• Solutions vary in effectiveness and cost.
• Desk visits more expensive than e-mail.
• Temporary solution vs. permanent solution?

• Experienced customers may be useful

participants in selecting a solution.
• Inexperienced customers will get scared.

2018/2019 IS 7212: Advanced Networking and Sys 197

tem Administration by Dr. Samuel Asfer
 Execution
• Execution is often done by SA.
• Customer may have to execute solution.
• Remote customers and no remote ctl
software.
• Dialog with customer has to be adjusted
based on customer knowledge level.

2018/2019 IS 7212: Advanced Networking and Sys 198

tem Administration by Dr. Samuel Asfer
 Verification
• Verify that the problem is solved.
• Use the same technique you used to
reproduce the problem.
• Customer is final verification.
• Request should not be closed until customer
verifies that the problem is fixed.

2018/2019 IS 7212: Advanced Networking and Sys 199

tem Administration by Dr. Samuel Asfer
 Trend Analysis
1. Does a customer report the same issue
repeatedly?
2. Are there many questions under a particular
classification?
3. Are many customers reporting the same issue?
4. Are there any classes of requests that can
become self-service?
5. Who are your most frequent customers?
6. What are your most frequent time-consuming
requests?
2018/2019 IS 7212: Advanced Networking and Sys 200
tem Administration by Dr. Samuel Asfer
Debugging
 Debugging
Debugging
1. Learn the customer’s problem.
2. Find the problem’s cause and fix it.
3. Have the right tools.
Fix things once
4. Fix something once rather than over and
over.
5. Avoid the temporary fix trap.
6. Measure twice, cut once, and other advice.
2018/2019 IS 7212: Advanced Networking and Sys 202
tem Administration by Dr. Samuel Asfer
 Learn the Customer’s Problem
• Understand at a high level what customer is
attempting to do and what part is failing.
• Customer problem reports vary:
– My mail program is broken.
– I can’t reach the mail server.
– My mailbox disappeared!
• Actual problem might be:
– Network problem.
– Power failure.
– DNS problem.

2018/2019 IS 7212: Advanced Networking and Sys 203

tem Administration by Dr. Samuel Asfer
 Find the Problem and Fix it
• Approach debugging systematically
• Form a hypothesis.
• Test hypothesis.
• Record results.
• Modify hypothesis based on results.
• Problem is often in the last change.
• Last config change, last new hardware, etc.
• Avoid random changes and workarounds.
• Rebooting is not always a solution!

2018/2019 IS 7212: Advanced Networking and Sys 204

tem Administration by Dr. Samuel Asfer
 Find the Problem and Fix it ..
• Find the Problem and Fix it through
either:
• Process of Elimination

• Successive Refinement

2018/2019 IS 7212: Advanced Networking and Sys 205

tem Administration by Dr. Samuel Asfer
 Process of Elimination
• Process of Elimination
• Remove parts of system one by one until
problem disappears.
• Problem must have been in last component.
• Examples
• Remove DIMMs (Dual in-line Memory
Module) one by one to identify a bad memory
unit.
• Remove driver or application one by one to
identify the source of the conflict.
2018/2019 IS 7212: Advanced Networking and Sys 206
tem Administration by Dr. Samuel Asfer
 Successive Refinement
• Successive Refinement
• Add a component at a time, verifying that it
works correctly at each step along the way.
• Examine output at each step along the way.
• Examples
• traceroute: tests network connectivity one
hop at a time until it encounters problem or
reaches destnation
• pipeline: develop a piped set of commands by
adding one command a time to the pipeline
2018/2019 IS 7212: Advanced Networking and Sys 207
tem Administration by Dr. Samuel Asfer
 Have the Right Tools
• Tools to let you see inside devices/systems.
• network: sniffer, ping, traceroute, telnet/nc
• network services: netstat, rpcinfo
• operating system: log files
• process: system call tracer, e.g. strace
• performance: top, ps, vmstat, iostat
• Know how tools draw their conclusions.
• Tools can make mistakes or mislead you.

2018/2019 IS 7212: Advanced Networking and Sys 208

tem Administration by Dr. Samuel Asfer
 Fix Things Once
• Fixing something once is faster than fixing
it over and over again. Why?
• Corollaries
– Fix the problem permanently.
– Don’t reinvent the wheel.
– Fix the problem for all hosts at the same time.

2018/2019 IS 7212: Advanced Networking and Sys 209

tem Administration by Dr. Samuel Asfer
 Avoid Temporary Fix Trap
Quick fixes aren’t.
– A few minutes of your time every day adds up
over a month or year.
– Temporary fixes accumulate until you spend
your entire day doing one quick fix after another.
Temporary fixes may be required
– Lack of resources (hardware/software) or time.
– Must always be followed by permanent fixes.
– Add permanent fix to your calendar or request
system to ensure that it happens.

2018/2019 IS 7212: Advanced Networking and Sys 210

tem Administration by Dr. Samuel Asfer
 Learning from Carpenters
Measure twice, cut once.
– Double-check your work before making
changes.
– ex: Reread configuration file before restarting
server.
Copy exact
– Develop correct solution and test it.
– Copy solution exactly to other hosts or sites.

2018/2019 IS 7212: Advanced Networking and Sys 211

tem Administration by Dr. Samuel Asfer
 Automate
• Automation can fix problems permanently
• Log rotation script will ensure you don’t have to
manually delete logs to avoid full disks.
• Tape jukebox will ensure that you don’t forget to
manually swap backup tapes.
• Avoid using automation for quick fixes
• Automation can perform a temporary fix without
needing human intervention, e.g. kill runaways.
• Problem may grow over time without your awareness,
and automation can’t fix buggy software.

2018/2019 IS 7212: Advanced Networking and Sys 212

tem Administration by Dr. Samuel Asfer
 Debugging with High Level
Languages
Same goals as low-level debugging
– Examine and set values in memory
– Execute portions of program
– Stop execution when (and where) desired

Want debugging tools to operate on

high-level language constructs
– Examine and set variables, not memory locations
– Trace and set breakpoints on
statements and function calls, not instructions
– ...but also want access to low-level tools when needed
2018/2019 IS 7212: Advanced Networking and Sys 213
tem Administration by Dr. Samuel Asfer
 Types of Errors
Syntactic Errors
– Input code is not legal
– Caught by compiler (or other translation mechanism)

Semantic Errors
– Legal code, but not what programmer intended
– Not caught by compiler because syntax is correct

Algorithmic Errors
– Problem with the logic of the program
– Program does what programmer intended,
but it doesn't solve the right problem
2018/2019 IS 7212: Advanced Networking and Sys 214
tem Administration by Dr. Samuel Asfer
 Syntactic Errors
Common errors:
– missing semicolon or brace
– mis-spelled type in declaration
One mistake can cause an avalanche of errors
– because compiler can't recover and gets confused

missing semicolon
main () {
int i
int j;
for (i = 0; i <= 10; i++) {
j = i * 7;
printf("%d x 7 = %d\n", i, j);
}
}

2018/2019 IS 7212: Advanced Networking and Sys 215

tem Administration by Dr. Samuel Asfer
 Semantic Errors
Common Errors
– Missing braces to group statements together
– Confusing assignment with equality
– Wrong assumptions about operator precedence, associativity
– Wrong limits on for-loop counter
– Uninitialized variables
missing braces,
so printf not part of if
main () {
int I;
int j;
for (i = 0; i <= 10; i++)
j = i * 7;
printf("%d x 7 = %d\n", i, j);
}

2018/2019 IS 7212: Advanced Networking and Sys 216

tem Administration by Dr. Samuel Asfer
 Algorithmic Errors
Design is wrong,
so program does not solve the correct problem

Difficult to find
– Program does what we intended
– Problem might not show up until many runs of program
Maybe difficult to fix
– Have to redesign, may have large impact on program code

Classic example: Y2K bug

– only allow 2 digits for year, assuming 19__

2018/2019 IS 7212: Advanced Networking and Sys 217

tem Administration by Dr. Samuel Asfer
 Debugging Techniques
Ad-Hoc
– Insert printf statements to track control flow and values
– Code explicitly checks for values out of expected range, etc.
– Advantage:
• No special debugging tools needed
– Disadvantages:
• Requires intimate knowledge of code and expected values
• Frequent re-compile and execute cycles
• Inserted code can be buggy

Source-Level Debugger
– Examine and set variable values
– Tracing, breakpoints, single-stepping on source-code statements

2018/2019 IS 7212: Advanced Networking and Sys 218

tem Administration by Dr. Samuel Asfer
 Source-Level Debugger

main window
of Cygwin
version

2018/2019 IS 7212: Advanced Networking and Sys 219

tem Administration by Dr. Samuel Asfer
Source-Level Debugging Techniques

Breakpoints
– Stop when a particular statement is reached
– Stop at entry or exit of a function
Single-Stepping
– Execute one statement at a time
– Step "into" or step "over" function calls
• Step into: next statement is first inside function call
• Step over: execute function without stopping
• Step out: finish executing current function
and stop on exit

2018/2019 IS 7212: Advanced Networking and Sys 220

tem Administration by Dr. Samuel Asfer
Source-Level Debugging Techniques

Displaying Values
– Show value consistent with declared type of
variable
– Dereference pointers (variables that hold
addresses)
• See Chapter 17
– Inspect parts of a data structure
• See Chapters 19 and 17

2018/2019 IS 7212: Advanced Networking and Sys 221

tem Administration by Dr. Samuel Asfer
 Key Points
• Learn the customer’s problem.
• Systematically identify the cause and fix it.
– Process of elimination.
– Successive refinement.
• Fix the problem permanently.
– Don’t reinvent the wheel.
– Test your solution.
– Use fix on all of your hosts.
• Use automation wisely.
2018/2019 IS 7212: Advanced Networking and Sys 222
tem Administration by Dr. Samuel Asfer
• Introduction
• Managing Workstations / Desktops/, Managing Servers,
Managing Services
• Booting
• Processes
• Disks, Partitions, Volumes, File systems, Files
• Printing
• Accounts
• Service Monitoring
• People, Help Desk, Debagging
• Directories
• Data Centers
• Namespaces
• DNS
• Networking
• Network Architecture
2018/2019 IS 7212: Advanced Networking and Sys 223
tem Administration by Dr. Samuel Asfer
 Directories
(Reading Assignment)
Topics:
1. Directories
2. LDAP Structure
3. LDIF
4. Distinguished Names
5. Replication
6. OpenLDAP Configuration

2018/2019 IS 7212: Advanced Networking and Sys 224

tem Administration by Dr. Samuel Asfer
 What is a Directory?
• Directory: A collection of
information that is primarily
searched and read, rarely
modified.

• Directory Service: Provides

access to directory information.

• Directory Server: Application

that provides a directory service.
2018/2019 IS 7212: Advanced Networking and Sys 225
tem Administration by Dr. Samuel Asfer
 Directories vs. Databases
Directories are optimized for reading.
– Databases balanced for read and write.
Directories are tree-structured.
– Databases typically have relational
structure.
Directories are usually replicated.
– Databases can be replicated too.
Both are extensible data storage systems.
Both have advanced search capabilities.
2018/2019 IS 7212: Advanced Networking and Sys 226
tem Administration by Dr. Samuel Asfer
 System Administration Directories

Types of directory data

– Accounts
– Mail aliases and lists (address book)
– Cryptographic keys
– IP addresses
– Hostnames
– Printers
Common directory services
– DNS, LDAP, NIS
2018/2019 IS 7212: Advanced Networking and Sys 227
tem Administration by Dr. Samuel Asfer
 Advantages of Directories
Make administration easier.
– Change data only once: people, accounts, hosts.
Unify access to network resources.
– Single sign on.
– Single place for users to search (address book)
Improve data management
– Improve consistency (one location vs many)
– Secure data through only one server.

2018/2019 IS 7212: Advanced Networking and Sys 228

tem Administration by Dr. Samuel Asfer
NIS: Network Information Service
• Originally called Sun Yellow Pages
– Clients run ypbind.
– Servers run ypserv.
– Data stored under /var/yp on server.
• Server shares NIS maps with clients
– Each UNIX file may provide multiple NIS maps.
– NIS maps map keys like UID, username to data.
– passwd: passwd.byname, passwd.byuid
• Slave servers replicate master server content.
• Easy to use, but insecure, difficult to extend.

2018/2019 IS 7212: Advanced Networking and Sys 229

tem Administration by Dr. Samuel Asfer
 LDAP
Lightweight Directory Access Protocol
– Lightweight compared to X.500 directories.
– Directory, not a database, service.
– Access Protocol, not a directory itself.

2018/2019 IS 7212: Advanced Networking and Sys 230

tem Administration by Dr. Samuel Asfer
 LDAP Clients and Servers
• LDAP Clients
• Standalone directory browsers.
• Embedded clients (mail clients, logins, etc.)
• Cfg /etc/nsswitch.conf on UNIX to use LDAP.
• Common LDAP servers

2018/2019 IS 7212: Advanced Networking and Sys 231

tem Administration by Dr. Samuel Asfer
 LDAP Structure
An LDAP directory is made of entries.
– Entries may be employee records, hosts, etc.
Each entries consists of attributes.
– Attributes can be names, phone numbers, etc.
– objectClass attribute identifies entry type.
Each attribute is a type / value pair.
– Type is a label for the information stored (name)
– Value is value for the attribute in this entry.
– Attributes can be multi-valued.

2018/2019 IS 7212: Advanced Networking and Sys 232

tem Administration by Dr. Samuel Asfer
• Introduction
• Managing Workstations / Desktops/, Managing Servers,
Managing Services
• Booting
• Processes
• Disks, Partitions, Volumes, File systems, Files
• Printing
• Accounts
• Service Monitoring
• People, Help Desk, Debagging
• Directories
• Data Centers
• Namespaces
• DNS
• Networking
• Network Architecture
2018/2019 IS 7212: Advanced Networking and Sys 233
tem Administration by Dr. Samuel Asfer
 Data Centers
• What is a datacenter?
– Page 129, “ a data center is a place where
you keep machines that are a shared
resource”
– Other terms that basically mean the same
thing
• Server room
• Machine room
• Server closet
– Many things that make a data center more
than just another room where computers run
2018/2019 IS 7212: Advanced Networking and Sys 234
tem Administration by Dr. Samuel Asfer
 Data Centers …
• To get the benefits on a good server room, you
don’t need to “build-it-yourself”
• $100 - $400 per square foot to build
• Rent space from a hosting company
– This is known as a “co-location Facility”
• Rent ‘services’ from a hosting company such as
RackSpace.com or peer1.com where they provide
you the CPU, memory, disk (I/O) and networking.
• Here, you don’t care about the physical server,
just the service running on it. If the service is up
and secure, what matters.
2018/2019 IS 7212: Advanced Networking and Sys 235
tem Administration by Dr. Samuel Asfer
Major Components to a Data Center
What makes a data center more than just a closet with
a bunch of servers?
1. Location
2. Access
3. Security
4. Power, Cooling, Humidity
5. Fire Suppression
6. Racks
7. Cable Management
8. Communications
9. Console Access
10. Workbench
11. Tools, parts, etc

2018/2019 IS 7212: Advanced Networking and Sys 236

tem Administration by Dr. Samuel Asfer
 Location
• Where is your data center going to be located? How
many do you need?
• Large multi-national corporations may have many with
one acting as a primary and others as a backup
• May run data centers concurrently as to balance load
and provide immediate fail-over
• Smaller companies or educational institutions may have
many, one for each college, or a few for the entire
university
• May be placed ‘strategically’ around the university or
around the area to minimize expense.
• Even the best data centers with redundant power,
cooling, etc. can fall victim to a contactor with a backhoe
or excavating equipment.
2018/2019 IS 7212: Advanced Networking and Sys 237
tem Administration by Dr. Samuel Asfer
 Location …
• If your area is susceptible to
flooding, don’t put your data center
in the basement.
• E.g. “One company I’ve read about
has two data centers, one in
Florida and one in Colorado. They
change primary data centers every
6 months.”
• Why?
• Florida is susceptible to hurricanes
and Colorado is susceptible to
huge snow storms.
• Also a great way of testing their
disaster recovery environment.

2018/2019 IS 7212: Advanced Networking and Sys 238

tem Administration by Dr. Samuel Asfer
 Access
• What type of access is
required?
– Wheelchair, ramps, loading
docks to unload
equipment?
• Some equipment is wider
than the average sized
door. Need double-doors.
• Restrict access to people
who don’t need it.
2018/2019 IS 7212: Advanced Networking and Sys 239
tem Administration by Dr. Samuel Asfer
 Security
• What type of security do you require?
– Numeric key pads – bad idea. Anyone
can share the code. No way of knowing
who came in.
– Keys, - better, at least you know who you
gave the key to originally
– Card swipes – even better, logs entry
information and controls access
– Proximity detectors – better still, same
advantages as card swipes but more
convenient
– Biometrics – almost there. Thumb print
reader or voice recognition.
– Two Factor, -best, something you have
and something you know. A numeric
keypad that requires both a static or non-
changing code and a one-time-password
security token.
2018/2019 IS 7212: Advanced Networking and Sys 240
tem Administration by Dr. Samuel Asfer
 Cooling
• For every watt of power
used in the data center, you
need to plan for the same
amount of power to cool
your equipment.
• Direct your cool air where
you need it
• New types of cooling
concepts, “cool the servers,
not the entire room”
• Rear door heat exchangers.
Cools only the servers and
not surrounding room air.
Most efficient.
2018/2019 IS 7212: Advanced Networking and Sys 241
tem Administration by Dr. Samuel Asfer
 Large Data Center Air Conditioner and
Rear Door Heat Exchanger

2018/2019 IS 7212: Advanced Networking and Sys 242

tem Administration by Dr. Samuel Asfer
 Hot and Cold isles and Humidity
• Cold air goes in front of rack,
heated by server, exited out back.
• Arrange your data center so the
backs of the racks are towards each
other. This forms hot and cold isles.
• Easier to collect the hot air. Heat
rises.
• Keep server room humidity between
45% - 55%. Too low, static, too
high, condensation.

2018/2019 IS 7212: Advanced Networking and Sys 243

tem Administration by Dr. Samuel Asfer
 Temperature
• Temperature: 64-80F
– Ambient temperature (in room) is usually 40+
degrees lower than inside of computer
– When chips reach ~120F, they may not work
correctly; at ~160F, they break (some CPUs
can operate up to ~200F)

2018/2019 IS 7212: Advanced Networking and Sys 244

tem Administration by Dr. Samuel Asfer
 Power
• Multiple sources of electricity.
– Multiple feeds from multiple
substations.
– Generator, not only for emergencies
but also for normal operations.
– What type of power does it produce,
do you need? 110V, 220V, 480V?
– Discussion is driven by what type of
power your servers require. Not all
servers require 110V “standard line
power”.
– Can you use DC power? 240 Volts?
More efficient than 110V. Save up to
4% in electric usage.
– Be “Green” in your data center

2018/2019 IS 7212: Advanced Networking and Sys 245

tem Administration by Dr. Samuel Asfer
 Power …
• Do you have a Uninterruptable
power supply (UPS)? How big?
• How long does it need to last, 10
minutes, 1 hour, 4 hours?
• Varies depending if you have a
generator or not.
• Do you have automatic transfer
switches, (ATM)?
– Switches that ‘sense’ if line power or
in Syracuse, National Grid power if
present, and if not, automatically
starts generator and transfers load.
When line power returns, shuts off
generator and returns load to line
power.
2018/2019 IS 7212: Advanced Networking and Sys 246
tem Administration by Dr. Samuel Asfer
 Power Distribution Unit (PDU)
• Looks like a power strip, much more.
• Can be horizontally or virtually mounted
• Monitor, record, and control each outlet remotely via
network.
• Should have two per server, one connected to line power
and one connected to UPS / Generator power

2018/2019 IS 7212: Advanced Networking and Sys 247

tem Administration by Dr. Samuel Asfer
 Monitor your Power

2018/2019 IS 7212: Advanced Networking and Sys 248

tem Administration by Dr. Samuel Asfer
 Fire Suppression
• Require Fire suppression
methods, required by law
/code
• Conventional (Water and
Sprinklers) = Bad
• Many other methods
– CO2, good for servers, bad
for people
– Conventional extinguishers
– Consult local fire authorities
2018/2019 IS 7212: Advanced Networking and Sys 249
tem Administration by Dr. Samuel Asfer
 Racks
• The ‘spine’ of the data center
• Dictate other components
– Wiring, cooling, power, etc
– Not inexpensive, up to $10,000 per rack with
required products (power, cooling, networking, etc)
• Standard 19” wide, width of standard mountable
equipment. 19” between posts.
• Some telecomm equipment is wider. Special
order racks available or equipment comes with
these racks already.
• May purchase racks wider and deeper than your
needs as to leave room for networking, power,
cooling, etc.
• 2 or 4 posts
– 4 is better but more expensive.
• Rack equipment in U’s or rack units,
– 1.75 inches per “U”
– Can use square nuts or screws to mount equipment

2018/2019 IS 7212: Advanced Networking and Sys 250

tem Administration by Dr. Samuel Asfer
 Rack Cooling
• Racks are typically open at the bottom to
allow cool air to flow in.
• Heat rises exits out the back
• Not a location to run wires. Run cabling
overhead with above-the-racks wire trays
• If keep doors and sides on if using raised
floor, off it using standard air-conditioning
• Monitor temperature inside racks
• We use device called “Weather Goose”

2018/2019 IS 7212: Advanced Networking and Sys 251

tem Administration by Dr. Samuel Asfer
Racks Determine Cable Management
• A good cabling job is a pleasure
to work with.
• Don’t ‘build-your-own’ cables.
Purchase cat6 or category 6
cables.
• Purchase special cable
management products.
• Easily identify which cable
connects to what.
• Color code or label your cables
• Separate power from network
cables. Power down one side,
network cables down the other
2018/2019 IS 7212: Advanced Networking and Sys 252
tem Administration by Dr. Samuel Asfer
 Cable Management Images

2018/2019 IS 7212: Advanced Networking and Sys 253

tem Administration by Dr. Samuel Asfer
 Our Guidelines

2018/2019 IS 7212: Advanced Networking and Sys 254

tem Administration by Dr. Samuel Asfer
 Communications
• Put a telephone in your server room
incase you need to call someone in for
assistance or be speaking with a vendor
while standing in front of the server.
• “Bridge” the telephone line into a
infrequently used circuit to save money.
• Don’t rely on cell phones. Can be difficult
to hear plus more interference in server
room.

2018/2019 IS 7212: Advanced Networking and Sys 255

tem Administration by Dr. Samuel Asfer
 Workbench, Tools, Parts
• Have a place where
your staff can test out
or ‘burn in’ a server
before putting it into
production.
• Place to troubleshoot
failed servers
• Have extra patch
cables, nuts, bolts,
“spare parts” on hand.

2018/2019 IS 7212: Advanced Networking and Sys 256

tem Administration by Dr. Samuel Asfer
 Data Centre Space

• Rarely do you get to see a data center like this (empty).

• Note the grates in the floor for cool air. The bulky refrigerator-sized units
along the walls are probably air conditioners.
2018/2019 IS 7212: Advanced Networking and Sys 257
tem Administration by Dr. Samuel Asfer
 Summary
• Data center is much more than a standard
room or closet.
• Many things make a server room unique.
• $100 / $400 or more per square foot to
create a server room.
• Look for alternatives, outsource
• If you are going to build it, do it right the
first time.
2018/2019 IS 7212: Advanced Networking and Sys 258
tem Administration by Dr. Samuel Asfer
Facebook data centre in Sweden.
2018/2019 IS 7212: Advanced Networking and Sys 259
tem Administration by Dr. Samuel Asfer
Facebook Oregon Data Center (2012)
2018/2019 IS 7212: Advanced Networking and Sys 260
tem Administration by Dr. Samuel Asfer
 Buy a pre-built data center

2018/2019 IS 7212: Advanced Networking and Sys 261

tem Administration by Dr. Samuel Asfer
 Easy transport by truck or ship

2018/2019 IS 7212: Advanced Networking and Sys 262

tem Administration by Dr. Samuel Asfer
• Introduction
• Managing Workstations / Desktops/, Managing Servers,
Managing Services
• Booting
• Processes
• Disks, Partitions, Volumes, File systems, Files
• Printing
• Accounts
• Service Monitoring
• People, Help Desk, Debagging
• Directories
• Data Centers
• Namespaces
• DNS
• Networking
• Network Architecture
2018/2019 IS 7212: Advanced Networking and Sys 263
tem Administration by Dr. Samuel Asfer
 Namespaces
• Namespaces – the lists and directories
in your environment
− files in filesystem
− account names in use
− printers available
− names of hosts
− ethernet addresses
− service-name/port-number lists
− home directory location maps
2018/2019 IS 7212: Advanced Networking and Sys 264
tem Administration by Dr. Samuel Asfer
 Namespaces …
• Some namespaces are flat
– there are no duplicate names
• Some namespaces are hierarchical
– duplicate items within different branches of a
tree
• Need policies to govern namespaces
– Ideally, written policies
• Can become training for new SAs
• Needed to enforce adherence to policy

2018/2019 IS 7212: Advanced Networking and Sys 265

tem Administration by Dr. Samuel Asfer
 Namespace policies
• Naming policy
– What names are permitted/not permitted?
• Technology – specific syntax
• Organizational – not offensive
• Standards compliance (fulfilment)
– How are names selected?
– How are collisions resolved?
– How do you merge namespaces?
• Technological and political concerns

2018/2019 IS 7212: Advanced Networking and Sys 266

tem Administration by Dr. Samuel Asfer
 Namespace policies (2)
• Naming policy
– How are names selected?
• Formulaic
– e.g., hostname: pc-0418; user-id: xyz210
• Thematic
– e.g., using planet names for servers; coffee for printers
• Functional
– e.g., specific-purpose accounts: admin, secretary, guest;
hostnames dns1, web3; disk partitions /finance, /devel
• Descriptive
– e.g., location, object type (pl122-ps)
• No method
– Everyone picks their own, first-come first-serve
– Once you choose one scheme, difficult to change; choose
well!
2018/2019 IS 7212: Advanced Networking and Sys 267
tem Administration by Dr. Samuel Asfer
 Namespace policies (3)
• Comments on naming
– Some schemes are easier to use than others
• easier to remember/figure out, to type, etc.
• Some names imply interesting targets
– secureserver, sourcecodedb, accounting, etc.
– avoid exceptions to formulaic names
• Formulaic names suggest problems when
incomplete
– server1, server2, server4, server7
• Sometimes helpful when desktop matches user's
name
– Assuming user wants to be easily identified
2018/2019 IS 7212: Advanced Networking and Sys 268
tem Administration by Dr. Samuel Asfer
 Namespace policies (4)
• Protection policy
– What kind of protection does the namespace
require?
• password list
• UIDs
• login IDs, e-mail addresses
– Who can add/delete/change an entry?
• Need backups or change management to roll back
a change

2018/2019 IS 7212: Advanced Networking and Sys 269

tem Administration by Dr. Samuel Asfer
 Namespace policies (5)
• Longevity policy
– When are entries removed?
• after IP address not used for months
• contractor ID each year
• student accounts a year after graduation
• employee accounts the day they leave
– Functional names might be exceptions
• [email protected]
• [email protected]

2018/2019 IS 7212: Advanced Networking and Sys 270

tem Administration by Dr. Samuel Asfer
 Namespace policies (6)
• Scope policy
– Where is the namespace to be used?
• How widely (geographically) shall it be used?
– Global authentication is possible with RADIUS
– NIS often provides a different space per cluster
• How many services will use it? (thickness)
– ID might serve for login, email, name on modem pools
– Across different authentication services
» ActiveDirectory, NIS, RADIUS (even with different pw)
• What happens when a user must span namespaces?
– Different IDs? Confusing, lead to collisions
• Single flat namespace is appealing; not always
needed
2018/2019 IS 7212: Advanced Networking and Sys 271
tem Administration by Dr. Samuel Asfer
 Namespace policies (7)
• Consistency policy
– Where the same name is used in multiple
namespaces, which attributes are also retained?
• E.g., UNIX name, requires same (real) person, same
UID, but not same password for email, login
• Reuse policy
– How soon after deletion can the name be
reused?
• Sometimes want immediate re-use (new printer)
• Sometimes long periods (prevent confusion and old
email from being sent to new user)
2018/2019 IS 7212: Advanced Networking and Sys 272
tem Administration by Dr. Samuel Asfer
 Namespace Management
• Namespace change procedures
– Need procedures for additions, changes, and
deletions
– Likely restricted to subgroup of admins
– Documentation can provide for enforcement,
training and step-by-step instruction
• Namespace management
– Should be centralized
• Maintain, backup, and distribute from one source
• Difficult to enforce uniqueness when distributed
– CentralizationIS provides
2018/2019
consistency
7212: Advanced Networking and Sys 273
tem Administration by Dr. Samuel Asfer
• Introduction
• Managing Workstations / Desktops/, Managing Servers,
Managing Services
• Booting
• Processes
• Disks, Partitions, Volumes, File systems, Files
• Printing
• Accounts
• Service Monitoring
• People, Help Desk, Debagging
• Directories
• Data Centers
• Namespaces
• DNS
• Networking
• Network Architecture
2018/2019 IS 7212: Advanced Networking and Sys 274
tem Administration by Dr. Samuel Asfer
 DNS
• DNS – The Domain Name System
– What does DNS do?
– The DNS namespace
– BIND software
– How DNS works?
– DNS database
– Testing and debugging (tools)

2018/2019 IS 7212: Advanced Networking and Sys 275

tem Administration by Dr. Samuel Asfer
 What does DNS do?
• Provides hostname – IP lookup services
• DNS defines
– A hierarchical namespace for hosts and IP
addresses
– A distributed database of hostname and address info
– A “resolver” – library routines that query this
database
– Improved routing for email
– A mechanism for finding services on a network
– A protocol for exchanging naming information
• DNS is essential for any org using the
Internet
2018/2019 IS 7212: Advanced Networking and Sys 276
tem Administration by Dr. Samuel Asfer
2018/2019 IS 7212: Advanced Networking and Sys 277
tem Administration by Dr. Samuel Asfer
 What uses DNS?
• Any application that operates over the Internet
• Such as
– email
• Spam filters
– WWW
– FTP
– IRC, IM
– Windows update
– telnet, ssh

2018/2019 IS 7212: Advanced Networking and Sys 278

tem Administration by Dr. Samuel Asfer
 The DNS Namespace

• A tree of “domains”
• Root is “.” (dot), followed
by top-level (root-level)
domains
• Two branches of tree
– One maps hostnames to IP addresses
– Other maps IP address back to hostnames
• Two types of top-level domain names used today
• gTLDs: generic top-level domains
• ccTLDs: country code top-level domains
2018/2019 IS 7212: Advanced Networking and Sys 279
tem Administration by Dr. Samuel Asfer
 Generic top-level domains
Domain Purpose Domai Purpose
n
com Companies aero Air transport industry
edu Educational institutions biz Businesses
gov (US) government coop Cooperatives
agencies
mil (US) military agencies info Unrestricted
net Network providers jobs Human resources folks
org Nonproft organizations museum Museums
int International name Individuals
organizations
arpa IP address lookup pro Professionals (attorneys,
etc.)
2018/2019 IS 7212: Advanced Networking and Sys 280
tem Administration by Dr. Samuel Asfer
 Generic top-level domains ...
• But today there are an abundance of top-
level domains
– .black, .blue, .airforce, .agency, .audio, etc.

• See http://www.iana.org/domains/root/db/

2018/2019 IS 7212: Advanced Networking and Sys 281

tem Administration by Dr. Samuel Asfer
 Common country codes
Code Country Code Country
au Australia hk Hong Kong
br Brazil jp Japan
ca Canada mx Mexico
cc Cocos Islands nu Niue
ch Switzerland se Sweden
de Germany tm Turkmenistan
et Ethiopia tv Tuvalu
fr France us United States
See http://www.iana.org/domains/root/db/
2018/2019 IS 7212: Advanced Networking and Sys 282
tem Administration by Dr. Samuel Asfer
 Domain name management
• Network Solutions (now VeriSign) used to
manage .com, .org, .net, and .edu directly
• VeriSign now manages infrastructure for
.com, .net, .tv, .name
– Dozens of others manage country codes and other top-
level domains
• Organizations can now register with many
different registrars (even when VeriSign manages
the underlying database)
• Domain holders must have two name servers
authoritative for the domain
2018/2019 IS 7212: Advanced Networking and Sys 283
tem Administration by Dr. Samuel Asfer
Selecting a domain name
• Most good (short) names in .com and other
old gTLDs are already in use
• Domain names are up to 63 characters per
segment (but a 12 character length limit is
recommended), and up to 255 chars overall
• Identify two authoritative name servers
• Select a registrar, and pay ~$1-$35/year for
registration

2018/2019 IS 7212: Advanced Networking and Sys 284

tem Administration by Dr. Samuel Asfer
 BIND software
• Berkeley Internet Name Domain system
– By far, the most popular nameserver
[Measurement Factory 2010 study]
• Three components
– a daemon called named that answers queries
– library routines that resolve host queries by
contacting DNS servers
– command-line utilities (nslookup, dig, host)

2018/2019 IS 7212: Advanced Networking and Sys 285

tem Administration by Dr. Samuel Asfer
 How DNS works?
• A client calls gethostbyname(), which is part of
the resolver library
• The resolver library sends a lookup request to the
first nameserver that it knows about (from
/etc/resolv.conf)
• If the nameserver knows the answer, it sends it
back to the client
• If the nameserver doesn't know, it either
– asks the next server, or
– returns a failure, and suggests that the client contact
the next server
2018/2019 IS 7212: Advanced Networking and Sys 286
tem Administration by Dr. Samuel Asfer
 Resolving process

2018/2019 IS 7212: Advanced Networking and Sys 287

tem Administration by Dr. Samuel Asfer
Delegation
• Impractical for high-level
servers to know about all
hosts (or even subdomains)
below
• Servers delegate specific
zones to other servers
• Names and addresses of
authoritative servers for
the relevant zone are
returned in referrals
2018/2019 IS 7212: Advanced Networking and Sys 288
tem Administration by Dr. Samuel Asfer
 What servers know ?
• All servers know about the 13 root servers
– hardcoded (rarely changes!), or in hint file
– a.root-servers.net ... m.root-servers.net
• Each root server knows about servers for
every top-level domain (.com, .net, .uk, etc.)
• Each top-level domain knows the servers for
each second-level domain within the toplevel
domain
• Authoritative servers know about their hosts
2018/2019 IS 7212: Advanced Networking and Sys 289
tem Administration by Dr. Samuel Asfer
 Example resolution

2018/2019 IS 7212: Advanced Networking and Sys 290

tem Administration by Dr. Samuel Asfer
 Types of Name Servers
• Recursive vs. nonrecursive servers
– Servers that allow recursive queries will do all
the work
– Nonrecursive servers will only return referrals
or answers
• Authoritative vs. caching-only servers
– Authoritative servers have the original data
– Caching servers retain data previously seen
for future use
2018/2019 IS 7212: Advanced Networking and Sys 291
tem Administration by Dr. Samuel Asfer
 Caching reduces DNS load

2018/2019 IS 7212: Advanced Networking and Sys 292

tem Administration by Dr. Samuel Asfer
 IP-to-hostname resolution
• IP resolution works essentially the same as hostname
resolution
• – Query for
15.16.192.152
– Rendered as
query for
152.192.16.
15.in-addr.arpa
– Each layer can
delegate to the
next
2018/2019 IS 7212: Advanced Networking and Sys 293
tem Administration by Dr. Samuel Asfer
 BIND client configuration
• Each host has /etc/resolv.conf which lists
DNS servers
– Can be set manually, or via DHCP
• Servers must be recursive, and should
have a cache
• Servers are contacted in order, only after
timing out previous attempt

2018/2019 IS 7212: Advanced Networking and Sys 294

tem Administration by Dr. Samuel Asfer
 BIND server issues
• named is typically started at boot time
• Configured using /etc/named.conf
• Can decide between
– caching vs. authoritative
– slave vs. master (per zone)
– answering recursive or only iterative queries
• Lots more options
– Who can access, what port, etc.

2018/2019 IS 7212: Advanced Networking and Sys 295

tem Administration by Dr. Samuel Asfer
 DNS on Linux
• Linux uses /etc/nsswitch.conf to determine
what sources to use for name lookups
# /etc/nsswitch.conf
# passwd: files nisplus
group: files nisplus
hosts: files dns
• Configuration is in /etc/named.conf
• Other files in /var/named

2018/2019 IS 7212: Advanced Networking and Sys 296

tem Administration by Dr. Samuel Asfer
 Dynamic updates to DNS
• DNS was originally designed for an
environment in which hostnames (and
other DNS info) changed slowly, if at all
• DHCP breaks this assumption
• Recent versions of BIND allow DHCP to
notify BIND of address assignments

2018/2019 IS 7212: Advanced Networking and Sys 297

tem Administration by Dr. Samuel Asfer
 DNS Database
(Reading Assignment)
• Exactly what data is stored?
• Resource records
– Specify nameservers
– Name to address translation
– Address to name translation
– Host aliases
– Mail routing
– Free text, location, etc.
• Format
– [name] [ttl] [class] type data
2018/2019 IS 7212: Advanced Networking and Sys 298
tem Administration by Dr. Samuel Asfer
 Resource record: name
[name] [ttl] [class] type data
• name is host or domain for the record
• Absolute names must end with a dot
• Relative names do not – the current
domain is added (sometimes causing
mistakes!)
• www.cse.lehigh.edu.cse.lehigh.edu

2018/2019 IS 7212: Advanced Networking and Sys 299

tem Administration by Dr. Samuel Asfer
 Resource record: ttl
[name] [ttl] [class] type data
• The time to live (ttl) field specifies in seconds
the time that the data item may still be
cached
• Increasing the ttl (say to a week) decreases
traffic and DNS load substantially
• Setting a value too low can hurt web site
• performance
• Typical values are in days or weeks
2018/2019 IS 7212: Advanced Networking and Sys 300
tem Administration by Dr. Samuel Asfer
 Resource record: class
[name] [ttl] [class] type data
• Three values of class are supported
• IN: Internet
• default (and only one modern systems care about)
• CH: ChaosNet
• obsolete protocol used by obsolete machines
• HS: Hesiod
• database service built on top of BIND (from MIT)

2018/2019 IS 7212: Advanced Networking and Sys 301

tem Administration by Dr. Samuel Asfer
 Resource record: type
[name] [ttl] [class] type data
• Many DNS record types
– Zone
• SOA: Start of authority (define a zone)
• NS: Name server
– Basic
• A: IPv4 address (name to address translation)
• AAAA: IPv6 address (name to address translation)
• PTR: address-to-name translation
• MX: Mail exchanger
– Other
• CNAME: Canonical name (implements aliases)
2018/2019 IS 7212: Advanced Networking and Sys 302
tem Administration by Dr. Samuel Asfer
 Data /record/
• refresh = how often slave servers must check master
• retry = when the slave will try again after failure
• expire = how long data can be considered valid without
master
• minimum = TTL for cached negative answers
• Ex:
cs.colorado.edu 86400 IN SOA ns.cs.colorado.edu.
hostmaster.cs.colorado.edu.
(2001111300 ; serial number
7200 ; refresh (2 hours)
1800 ; retry (30 minutes)
604800 ; expire (1 week)
7200 ) ; minimum (2 hours)
2018/2019 IS 7212: Advanced Networking and Sys 303
tem Administration by Dr. Samuel Asfer
 Testing and debugging (tools)
• named supports lots of logging options
• typical BIND tools:
– nslookup (old, possibly deprecated)
– host
– dig
• whois – find domain and network registration
info

2018/2019 IS 7212: Advanced Networking and Sys 304

tem Administration by Dr. Samuel Asfer
 Other Issues
• Many aspects of DNS haven't been covered in
lecture
– Lots of details!
– DNS database
– Security issues
– IPv6
– Internationalization – now supported!
• DNS is generally case-insensitive
• VeriSign Site Finder product
– See http://cyber.law.harvard.edu/tlds/sitefinder/
2018/2019 IS 7212: Advanced Networking and Sys 305
tem Administration by Dr. Samuel Asfer
• Introduction
• Managing Workstations / Desktops/, Managing Servers,
Managing Services
• Booting
• Processes
• Disks, Partitions, Volumes, File systems, Files
• Printing
• Accounts
• Service Monitoring
• People, Help Desk, Debagging
• Directories
• Data Centers
• Namespaces
• DNS
• Networking
• Network Architecture
2018/2019 IS 7212: Advanced Networking and Sys 306
tem Administration by Dr. Samuel Asfer
 Networking (TCP/IP)
• Reading Assignment

2018/2019 IS 7212: Advanced Networking and Sys 307

tem Administration by Dr. Samuel Asfer
• Introduction
• Managing Workstations / Desktops/, Managing Servers,
Managing Services
• Booting
• Processes
• Disks, Partitions, Volumes, File systems, Files
• Printing
• Accounts
• Service Monitoring
• People, Help Desk, Debagging
• Directories
• Data Centers
• Namespaces
• DNS
• Networking
• Network Architecture
2018/2019 IS 7212: Advanced Networking and Sys 308
tem Administration by Dr. Samuel Asfer
 Network Architecture
• Network Architecture
– Hardware
– Routing
– Getting connected
– Centralization/decentralization
– Network topology
– Network debugging tools

2018/2019 IS 7212: Advanced Networking and Sys 309

tem Administration by Dr. Samuel Asfer
 Networking Hardware
• Ethernet is the core of most networks
– 10 Mbit 10Base2, 10BaseT
– 100 Mbit 100BaseTX
– 1 Gbit 1000BaseT
– 10 Gbit 10GBase-T
• Many competing LAN technologies
– ATM, Token Ring, FDDI
• Wireless
– 802.11b/a/g/n/ac
2018/2019 IS 7212: Advanced Networking and Sys 310
tem Administration by Dr. Samuel Asfer
 Connecting Ethernets
• Hub/repeater (physical layer)
– Retimes and reconstitutes Ethernet frames to all ports
– Single collision domain
• Switch (link layer)
– Learns locations of MAC addresses, selectively
forwards frames
– Receives, buffers, and retransmits packets
• Separate collision domains
• Router (IP/network layer)
– Connects separate ethernet networks
– Can connect different LAN/WAN technologies
2018/2019 IS 7212: Advanced Networking and Sys 311
tem Administration by Dr. Samuel Asfer
 Routing
• Given a packet, on which of multiple
network interfaces should it be sent?
• UNIX kernel keeps routing table (netstat –
rn)

2018/2019 IS 7212: Advanced Networking and Sys 312

tem Administration by Dr. Samuel Asfer
 Routing ...
• Routing is static for most systems
– Established when network card configured
• Additional static routes can be added using
the route command
• Dynamic routing can be managed using
quagga and xorp (ULSAH if interested)
– routed and gated are obsolete

2018/2019 IS 7212: Advanced Networking and Sys 313

tem Administration by Dr. Samuel Asfer
 Connecting to the Internet
• How can you connect your network
to the Internet?
– Dialup
– Wireless (cellular, satellite, point-to-point)
– xDSL
– Cable
– ISDN
– Frame relay
– T1, T3, OC3, etc.
2018/2019 IS 7212: Advanced Networking and Sys 314
tem Administration by Dr. Samuel Asfer
 Remote access
• Connecting users to the organization
– Check email, access data when traveling
– Work from home
– Remote facilities (e.g., stationed at customer site)
but need access to organization network regularly
• Different needs, different support requirements
– Worry about authentication, security, performance,
costs
– Consider centralization of authentication
– Consider outsourcing changing technologies
2018/2019 IS 7212: Advanced Networking and Sys 315
tem Administration by Dr. Samuel Asfer
 Network Topology
• Network architecture should
– Be clean and simple
– Provide for growth (new LAN segments, new
remote offices)
– Ensure reliability through redundancy
• Need to consider both physical and logical
topologies
• Typical forms: Bus, Star, Ring, Mesh

2018/2019 IS 7212: Advanced Networking and Sys 316

tem Administration by Dr. Samuel Asfer
Columbia University Network
February 23, 1999

2018/2019 IS 7212: Advanced Networking and Sys 317

tem Administration by Dr. Samuel Asfer
2018/2019 IS 7212: Advanced Networking and Sys 318
tem Administration by Dr. Samuel Asfer
 Network debugging
• Questions to ask
– Do you have physical connectivity and a link light?
– Is your interface configured properly?
– Is DNS configured properly?
– Do your ARP tables show other hosts?
– Can you ping the local host address (127.0.0.1)?
– Can you ping other local hosts by IP address?
– Can you ping other local hosts by hostname?
– Can you ping hosts on another network?
– Do high-level commands like telnet and ssh work?
2018/2019 IS 7212: Advanced Networking and Sys 319
tem Administration by Dr. Samuel Asfer
 Network tools
(Reading Assignment)
• ping
• traceroute
• netstat
• arp
• tcpdump/wireshark

2018/2019 IS 7212: Advanced Networking and Sys 320

tem Administration by Dr. Samuel Asfer
 ping
• If ping works, networking between hosts is
likely to be working
• It does not test availability of services
• Ex.
# ping www.dbu.edu

2018/2019 IS 7212: Advanced Networking and Sys 321

tem Administration by Dr. Samuel Asfer
 traceroute
• Finds the sequence of gateways traveled
• Works by increasing the TTL (Time to
Live) of the packet sent
• traceroute -n skips DNS
• EX.
% traceroute www.dbu.edu

2018/2019 IS 7212: Advanced Networking and Sys 322

tem Administration by Dr. Samuel Asfer
 netstat
• Tons o' network statistics
• Can also show
– interface configurations, routing tables,
counter values
• Ex.
# netstat | more

2018/2019 IS 7212: Advanced Networking and Sys 323

tem Administration by Dr. Samuel Asfer
 Packet sniffers
• Show you what is really on the network
• Examples: tcpdump, wireshark
• Ex.
# tcpdump

2018/2019 IS 7212: Advanced Networking and Sys 324

tem Administration by Dr. Samuel Asfer
 Thank you !

2018/2019 IS 7212: Advanced Networking and Sys 325

tem Administration by Dr. Samuel Asfer