The Linux Sysadmin Guide
The Linux Sysadmin Guide
Version 0.6.2
Lars Wirzenius
liw@iki.
Joanna Oja
viu@iki.
The Linux System Administrators Guide: Version 0.6.2 by Lars Wirzenius by Joanna Oja
Copyright 19931998 Lars Wirzenius. Trademarks are owned by their owners. Permission is granted to make and distribute verbatim copies of this manual provided the copyright notice and this permission notice are preserved on all copies. Permission is granted to process the document source code through TeX or other formatters and print the results, and distribute the printed document, provided the printed document carries copying permission notice identical to this one, including the references to where the source code can be found and the ofcial home page. Permission is granted to copy and distribute modied versions of this manual under the conditions for verbatim copying, provided that the entire resulting derived work is distributed under the terms of a permission notice identical to this one. Permission is granted to copy and distribute translations of this manual into another language, under the above conditions for modied versions. The author would appreciate a notication of modications, translations, and printed versions. Thank you.
Table of Contents
Dedication...................................................................................................................................................7 Source and pre-formatted versions available ..........................................................................................8 1. Introduction............................................................................................................................................9 The Linux Documentation Project ...................................................................................................11 2. Overview of a Linux System ...............................................................................................................12 Various parts of an operating system ...............................................................................................12 Important parts of the kernel ............................................................................................................12 Major services in a UNIX system ....................................................................................................14 init ..........................................................................................................................................14 Logins from terminals ............................................................................................................14 Syslog .....................................................................................................................................15 Periodic command execution: cron and at .............................................................................15 Graphical user interface..........................................................................................................15 Networking .............................................................................................................................15 Network logins .......................................................................................................................16 Network le systems ..............................................................................................................16 Mail.........................................................................................................................................16 Printing ...................................................................................................................................17 The lesystem layout..............................................................................................................17 3. Overview of the Directory Tree ..........................................................................................................19 Background ......................................................................................................................................19 The root lesystem...........................................................................................................................21 The /etc directory ..........................................................................................................................23 The /dev directory ..........................................................................................................................25 The /usr lesystem.........................................................................................................................26 The /var lesystem.........................................................................................................................27 The /proc lesystem ......................................................................................................................28 4. Using Disks and Other Storage Media...............................................................................................32 Two kinds of devices........................................................................................................................32 Hard disks ........................................................................................................................................33 Floppies ............................................................................................................................................36 CD-ROMs .......................................................................................................................................37 Tapes ................................................................................................................................................38 Formatting ........................................................................................................................................38 Partitions ..........................................................................................................................................41 The MBR, boot sectors and partition table.............................................................................41
Extended and logical partitions ..............................................................................................42 Partition types .........................................................................................................................43 Partitioning a hard disk...........................................................................................................44 Device les and partitions ......................................................................................................45 Filesystems.......................................................................................................................................45 What are lesystems? .............................................................................................................45 Filesystems galore ..................................................................................................................46 Which lesystem should be used?..........................................................................................49 Creating a lesystem ..............................................................................................................49 Mounting and unmounting .....................................................................................................51 Checking lesystem integrity with fsck .................................................................................55 Checking for disk errors with badblocks ...............................................................................55 Fighting fragmentation ...........................................................................................................56 Other tools for all lesystems.................................................................................................56 Other tools for the ext2 lesystem..........................................................................................57 Disks without lesystems.................................................................................................................58 Allocating disk space .......................................................................................................................59 Partitioning schemes...............................................................................................................59 Space requirements.................................................................................................................60 Examples of hard disk allocation ...........................................................................................60 Adding more disk space for Linux .........................................................................................61 Tips for saving disk space.......................................................................................................61 5. Memory Management .........................................................................................................................63 What is virtual memory? ..................................................................................................................63 Creating a swap space ......................................................................................................................63 Using a swap space ..........................................................................................................................64 Sharing swap spaces with other operating systems..........................................................................66 Allocating swap space......................................................................................................................66 The buffer cache...............................................................................................................................67 6. Boots And Shutdowns..........................................................................................................................70 An overview of boots and shutdowns ..............................................................................................70 The boot process in closer look........................................................................................................71 More about shutdowns .....................................................................................................................73 Rebooting .........................................................................................................................................74 Single user mode ..............................................................................................................................75 Emergency boot oppies..................................................................................................................75 7. init..........................................................................................................................................................76 init comes rst .................................................................................................................................76 Conguring init to start getty: the /etc/inittab le ..................................................................76
Run levels.........................................................................................................................................78 Special conguration in /etc/inittab .........................................................................................79 Booting in single user mode.............................................................................................................79 8. Logging In And Out.............................................................................................................................81 Logins via terminals.........................................................................................................................81 Logins via the network.....................................................................................................................83 What login does ...............................................................................................................................84 X and xdm ........................................................................................................................................84 Access control ..................................................................................................................................84 Shell startup......................................................................................................................................85 9. Managing user accounts......................................................................................................................86 Whats an account? ..........................................................................................................................86 Creating a user .................................................................................................................................86 /etc/passwd and other informative les .............................................................................86 Picking numeric user and group ids .......................................................................................87 Initial environment: /etc/skel ............................................................................................87 Creating a user by hand ..........................................................................................................88 Changing user properties .................................................................................................................89 Removing a user...............................................................................................................................89 Disabling a user temporarily ............................................................................................................90 10. Backups...............................................................................................................................................92 On the importance of being backed up ............................................................................................92 Selecting the backup medium ..........................................................................................................93 Selecting the backup tool .................................................................................................................93 Simple backups ................................................................................................................................94 Making backups with tar .......................................................................................................94 Restoring les with tar ...........................................................................................................96 Multilevel backups ...........................................................................................................................97 What to back up ...............................................................................................................................99 Compressed backups ........................................................................................................................99 11. Keeping Time ...................................................................................................................................101 Time zones .....................................................................................................................................101 The hardware and software clocks .................................................................................................102 Showing and setting time ...............................................................................................................102 When the clock is wrong................................................................................................................103 Glossary (DRAFT).................................................................................................................................105
List of Tables
4-1. Partition types (from the Linux fdisk program). ................................................................................43 7-1. Run level numbers ..............................................................................................................................78 10-1. Efcient backup scheme using many backup levels.........................................................................98
List of Figures
2-1. Some of the more important parts of the Linux kernel ......................................................................12 3-1. Parts of a Unix directory tree. Dashed lines indicate partition limits.................................................19 4-1. A schematic picture of a hard disk. ....................................................................................................34 4-2. A sample hard disk partitioning. ........................................................................................................42 4-3. Three separate lesystems..................................................................................................................51 4-4. /home and /usr have been mounted. ................................................................................................51 4-5. Sample output from dumpe2fs ..........................................................................................................57 8-1. Logins via terminals: the interaction of init, getty, login, and the shell............................................81 10-1. A sample multilevel backup schedule. .............................................................................................97
Dedication
This place is dedicated to a future dedication.
Chapter 1. Introduction
In the beginning, the le was without form, and void; and emptiness was upon the face of the bits. And the Fingers of the Author moved upon the face of the keyboard. And the Author said, Let there be words, and there were words.
This manual, the Linux System Administrators Guide, describes the system administration aspects of using Linux. It is intended for people who know next to nothing about system administration (as in what is it?), but who have already mastered at least the basics of normal usage. This manual also doesnt tell you how to install Linux; that is described in the Installation and Getting Started document. See below for more information about Linux manuals. System administration is all the things that one has to do to keep a computer system in a useable shape. It includes things like backing up les (and restoring them if necessary), installing new programs, creating accounts for users (and deleting them when no longer needed), making certain that the lesystem is not corrupted, and so on. If a computer were, say, a house, system administration would be called maintenance, and would include cleaning, xing broken windows, and other such things. System administration is not called maintenance, because that would be too simple. 1 The structure of this manual is such that many of the chapters should be usable independently, so that if you need information about, say, backups, you can read just that chapter. This hopefully makes the book easier to use as a reference manual, and makes it possible to read just a small part when needed, instead of having to read everything. However, this manual is rst and foremost a tutorial, and a reference manual only as a lucky coincidence. This manual is not intended to be used completely by itself. Plenty of the rest of the Linux documentation is also important for system administrators. After all, a system administrator is just a user with special privileges and duties. A very important resource are the manual pages, which should always be consulted when a command is not familiar. While this manual is targeted at Linux, a general principle has been that it should be useful with other UNIX based operating systems as well. Unfortunately, since there is so much variance between different versions of UNIX in general, and in system administration in particular, there is little hope to cover all variants. Even covering all possibilities for Linux is difcult, due to the nature of its development. There is no one ofcial Linux distribution, so different people have different setups, and many people have a setup they have built up themselves. This book is not targeted at any one distribution, even though I use the Debian GNU/Linux system almost exclusively. When possible, I have tried to point out differences, and explain several alternatives. I have tried to describe how things work, rather than just listing ve easy steps for each task. This means that there is much information here that is not necessary for everyone, but those parts are marked
Chapter 1. Introduction
as such and can be skipped if you use a precongured system. Reading everything will, naturally, increase your understanding of the system and should make using and administering it more pleasant. Like all other Linux related development, the work was done on a volunteer basis: I did it because I thought it might be fun and because I felt it should be done. However, like all volunteer work, there is a limit to how much effort I have been able to spend, and also on how much knowledge and experience I have. This means that the manual is not necessarily as good as it would be if a wizard had been paid handsomely to write it and had spent a few years to perfect it. I think, of course, that it is pretty nice, but be warned. One particular point where I have cut corners is that I have not covered very thoroughly many things that are already well documented in other freely available manuals. This applies especially to program specic documentation, such as all the details of using mkfs. I only describe the purpose of the program, and as much of its usage as is necessary for the purposes of this manual. For further information, I refer the gentle reader to these other manuals. Usually, all of the referred to documentation is part of the full Linux documentation set. Lars has tried to make this manual as good as possible and I would like, as a current maintainer, to keep up the good work. I would really like to hear from you if you have any ideas on how to make it better. Bad language, factual errors, ideas for new areas to cover, rewritten sections, information about how various UNIX versions do things, I am interested in all of it. My contact information is available via the World Wide Web at http://www.iki./viu/ (http://www.iki./viu/). Many people have helped me with this book, directly or indirectly. I would like to especially thank Matt Welsh for inspiration and LDP leadership, Andy Oram for getting me to work again with much-valued feedback, Olaf Kirch for showing me that it can be done, and Adam Richter at Yggdrasil and others for showing me that other people can nd it interesting as well. Stephen Tweedie, H. Peter Anvin, Remy Card, Theodore Tso, and Stephen Tweedie have let me borrow their work (and thus make the book look thicker and much more impressive): a comparison between the xia and ext2 lesystems, the device list and a description of the ext2 lesystem. These arent part of the book any more. I am most grateful for this, and very apologetic for the earlier versions that sometimes lacked proper attribution. In addition, I would like to thank Mark Komarinski for sending his material in 1993 and the many system administration columns in Linux Journal. They are quite informative and inspirational. Many useful comments have been sent by a large number of people. My miniature black hole of an archive doesnt let me nd all their names, but some of them are, in alphabetical order: Paul Caprioli, Ales Cepek, Marie-France Declerfayt, Dave Dobson, Olaf Flebbe, Helmut Geyer, Larry Greeneld and his father, Stephen Harris, Jyrki Havia, Jim Haynes, York Lam, Timothy Andrew Lister, Jim Lynch, Michael J. Micek, Jacob Navia, Dan Poirier, Daniel Quinlan, Jouni K Seppnen, Philippe Steindl, G.B. Stotte. My apologies to anyone I have forgotten. META need to add typographical conventsions and LDP blurb here.
10
Chapter 1. Introduction
Notes
1. There are some people who do call it that, but thats just because they have never read this manual, poor things.
11
This chapter gives an overview of a Linux system. First, the major services provided by the operating system are described. Then, the programs that implement these services are described with a considerable lack of detail. The purpose of this chapter is to give an understanding of the system as a whole, so that each part is described in detail elsewhere.
12
Figure 2-1 shows some of them. Figure 2-1. Some of the more important parts of the Linux kernel
Memory manager
Process manager
Floppy disk
Probably the most important parts of the kernel (nothing else works without them) are memory management and process management. Memory management takes care of assigning memory areas and swap space areas to processes, parts of the kernel, and for the buffer cache. Process management creates processes, and implements multitasking by switching the active process on the processor. At the lowest level, the kernel contains a hardware device driver for each kind of hardware it supports. Since the world is full of different kinds of hardware, the number of hardware device drivers is large.
13
There are often many otherwise similar pieces of hardware that differ in how they are controlled by software. The similarities make it possible to have general classes of drivers that support similar operations; each member of the class has the same interface to the rest of the kernel but differs in what it needs to do to implement them. For example, all disk drivers look alike to the rest of the kernel, i.e., they all have operations like initialize the drive, read sector N, and write sector N. Some software services provided by the kernel itself have similar properties, and can therefore be abstracted into classes. For example, the various network protocols have been abstracted into one programming interface, the BSD socket library. Another example is the virtual lesystem (VFS) layer that abstracts the lesystem operations away from their implementation. Each lesystem type provides an implementation of each lesystem operation. When some entity tries to use a lesystem, the request goes via the VFS, which routes the request to the proper lesystem driver.
init
The single most important service in a UNIX system is provided by init. init is started as the rst process of every UNIX system, as the last thing the kernel does when it boots. When init starts, it continues the boot process by doing various startup chores (checking and mounting lesystems, starting daemons, etc). The exact list of things that init does depends on which avor it is; there are several to choose from. init usually provides the concept of single user mode, in which no one can log in and root uses a shell at the console; the usual mode is called multiuser mode. Some avors generalize this as run levels; single and multiuser modes are considered to be two run levels, and there can be additional ones as well, for example, to run X on the console. In normal operation, init makes sure getty is working (to allow users to log in), and to adopt orphan processes (processes whose parent has died; in UNIX all processes must be in a single tree, so orphans must be adopted). When the system is shut down, it is init that is in charge of killing all other processes, unmounting all lesystems and stopping the processor, along with anything else it has been congured to do.
14
Logins from terminals (via serial lines) and the console (when not running X) are provided by the getty program. init starts a separate instance of getty for each terminal for which logins are to be allowed. getty reads the username and runs the login program, which reads the password. If the username and password are correct, login runs the shell. When the shell terminates, i.e., the user logs out, or when login terminated because the username and password didnt match, init notices this and starts a new instance of getty. The kernel has no notion of logins, this is all handled by the system programs.
Syslog
The kernel and many system programs produce error, warning, and other messages. It is often important that these messages can be viewed later, even much later, so they should be written to a le. The program doing this is syslog. It can be congured to sort the messages to different les according to writer or degree of importance. For example, kernel messages are often directed to a separate le from the others, since kernel messages are often more important and need to be read regularly to spot problems.
15
graphical user interface can be implemented. The three most popular user interface styles implemented over X are Athena, Motif, and Open Look.
Networking
Networking is the act of connecting two or more computers so that they can communicate with each other. The actual methods of connecting and communicating are slightly complicated, but the end result is very useful. UNIX operating systems have many networking features. Most basic services (lesystems, printing, backups, etc) can be done over the network. This can make system administration easier, since it allows centralized administration, while still reaping in the benets of microcomputing and distributed computing, such as lower costs and better fault tolerance. However, this book merely glances at networking; see the Linux Network Administrators Guide for more information, including a basic description of how networks operate.
Network logins
Network logins work a little differently than normal logins. There is a separate physical serial line for each terminal via which it is possible to log in. For each person logging in via the network, there is a separate virtual network connection, and there can be any number of these. 2 It is therefore not possible to run a separate getty for each possible virtual connection. There are also several different ways to log in via a network, telnet and rlogin being the major ones in TCP/IP networks. Network logins have, instead of a herd of gettys, a single daemon per way of logging in (telnet and rlogin have separate daemons) that listens for all incoming login attempts. When it notices one, it starts a new instance of itself to handle that single attempt; the original instance continues to listen for other attempts. The new instance works similarly to getty.
Network le systems
One of the more useful things that can be done with networking services is sharing les via a network le system. The one usually used is called the Network File System, or NFS, developed by Sun. With a network le system any le operations done by a program on one machine are sent over the network to another computer. This fools the program to think that all the les on the other computer are actually on the computer the program is running on. This makes information sharing extremely simple, since it requires no modications to programs.
16
Mail
Electronic mail is usually the most important method for communicating via computer. An electronic letter is stored in a le using a special format, and special mail programs are used to send and read the letters. Each user has an incoming mailbox (a le in the special format), where all new mail is stored. When someone sends mail, the mail program locates the receivers mailbox and appends the letter to the mailbox le. If the receivers mailbox is in another machine, the letter is sent to the other machine, which delivers it to the mailbox as it best sees t. The mail system consists of many programs. The delivery of mail to local or remote mailboxes is done by one program (the mail transfer agent or MTA, e.g., sendmail or smail), while the programs users use are many and varied (mail user agent or MUA, e.g., pine or elm). The mailboxes are usually stored in /var/spool/mail .
Printing
Only one person can use a printer at one time, but it is uneconomical not to share printers between users. The printer is therefore managed by software that implements a print queue: all print jobs are put into a queue and whenever the printer is done with one job, the next one is sent to it automatically. This relieves the users from organizing the print queue and ghting over control of the printer. 3 The print queue software also spools the printouts on disk, i.e., the text is kept in a le while the job is in the queue. This allows an application program to spit out the print jobs quickly to the print queue software; the application does not have to wait until the job is actually printed to continue. This is really convenient, since it allows one to print out one version, and not have to wait for it to be printed before one can make a completely revised new version.
17
Notes
1. In fact, it is often mistakenly considered to be the operating system itself, but it is not. An operating system provides many more services than a plain kernel. 2. Well, at least there can be many. Network bandwidth still being a scarce resource, there is still some practical upper limit to the number of concurrent logins via one network connection. 3. Instead, they form a new queue at the printer, waiting for their printouts, since no one ever seems to be able to get the queue software to know exactly when anyones printout is really nished. This is a great boost to intra-ofce social relations.
18
This chapter describes the important parts of a standard Linux directory tree, based on the FSSTND lesystem standard. It outlines the normal way of breaking the directory tree into separate lesystems with different purposes and gives the motivation behind this particular split. Some alternative ways of splitting are also described.
Background
This chapter is loosely based on the Linux lesystem standard, FSSTND, version 1.2 (see the bibliography), which attempts to set a standard for how the directory tree in a Linux system is organized. Such a standard has the advantage that it will be easier to write or port software for Linux, and to administer Linux machines, since everything will be in their usual places. There is no authority behind the standard that forces anyone to comply with it, but it has got the support of most, if not all, Linux distributions. It is not a good idea to break with the FSSTND without very compelling reasons. The FSSTND attempts to follow Unix tradition and current trends, making Linux systems familiar to those with experience with other Unix systems, and vice versa. This chapter is not as detailed as the FSSTND. A system administrator should also read the FSSTND for a complete understanding. This chapter does not explain all les in detail. The intention is not to describe every le, but to give an overview of the system from a lesystem point of view. Further information on each le is available elsewhere in this manual or the manual pages. The full directory tree is intended to be breakable into smaller parts, each on its own disk or partition, to accomodate to disk size limits and to ease backup and other system administration. The major parts are the root, /usr, /var, and /home lesystems (see Figure 3-1). Each part has a different purpose. The directory tree has been designed so that it works well in a network of Linux machines which may share some parts of the lesystems over a read-only device (e.g., a CD-ROM), or over the network with NFS.
19
Figure 3-1. Parts of a Unix directory tree. Dashed lines indicate partition limits.
home
dev etc
var
The roles of the different parts of the directory tree are described below.
The root lesystem is specic for each machine (it is generally stored on a local disk, although it could be a ramdisk or network drive as well) and contains the les that are necessary for booting the system up, and to bring it up to such a state that the other lesystems may be mounted. The contents of the root lesystem will therefore be sufcient for the single user state. It will also contain tools for xing a broken system, and for recovering lost les from backups. The /usr lesystem contains all commands, libraries, manual pages, and other unchanging les needed during normal operation. No les in /usr should be specic for any given machine, nor should they be modied during normal use. This allows the les to be shared over the network, which can be cost-effective since it saves disk space (there can easily be hundreds of megabytes in /usr), and can make administration easier (only the master /usr needs to be changed when updating an application, not each machine separately). Even if the lesystem is on a local disk, it could be mounted read-only, to lessen the chance of lesystem corruption during a crash. The /var lesystem contains les that change, such as spool directories (for mail, news, printers, etc), log les, formatted manual pages, and temporary les. Traditionally everything in /var has been somewhere below /usr, but that made it impossible to mount /usr read-only. The /home lesystem contains the users home directories, i.e., all the real data on the system. Separating home directories to their own directory tree or lesystem makes backups easier; the other parts often do not have to be backed up, or at least not as often (they seldom change). A big /home might have to be broken on several lesystems, which requires adding an extra naming level below
20
Although the different parts have been called lesystems above, there is no requirement that they actually be on separate lesystems. They could easily be kept in a single one if the system is a small single-user system and the user wants to keep things simple. The directory tree might also be divided into lesystems differently, depending on how large the disks are, and how space is allocated for various purposes. The important part, though, is that all the standard names work; even if, say, /var and /usr are actually on the same partition, the names /usr/lib/libc.a and /var/log/messages must work, for example by moving les below /var into /usr/var, and making /var a symlink to /usr/var. The Unix lesystem structure groups les according to purpose, i.e., all commands are in one place, all data les in another, documentation in a third, and so on. An alternative would be to group les les according to the program they belong to, i.e., all Emacs les would be in one directory, all TeX in another, and so on. The problem with the latter approach is that it makes it difcult to share les (the program directory often contains both static and shareable and changing and non-shareable les), and sometimes to even nd the les (e.g., manual pages in a huge number of places, and making the manual page programs nd all of them is a maintenance nightmare).
/bin
Commands needed during bootup that might be used by normal users (probably after bootup).
/sbin
Like /bin, but the commands are not intended for normal users, although they may use them if necessary and allowed.
/etc
21
/root
/lib
/lib/modules
Loadable kernel modules, especially those that are needed to boot the system when recovering from disasters (e.g., network and lesystem drivers).
/dev
Device les.
/tmp
Temporary les. Programs running after bootup should use /var/tmp, not /tmp, since the former is probably on a disk with more space.
/boot
Files used by the bootstrap loader, e.g., LILO. Kernel images are often kept here instead of in the root directory. If there are many kernel images, the directory can easily grow rather big, and it might be better to keep it in a separate lesystem. Another reason would be to make sure the kernel images are within the rst 1024 cylinders of an IDE disk.
/mnt
Mount point for temporary mounts by the system administrator. Programs arent supposed to mount on /mnt automatically. /mnt might be divided into subdirectories (e.g., /mnt/dosa might be the oppy drive using an MS-DOS lesystem, and /mnt/exta might be the same with an ext2 lesystem).
22
Scripts or directories of scripts to run at startup or when changing the run level. See the chapter on init for further information.
/etc/passwd
The user database, with elds giving the username, real name, home directory, encrypted password, and other information about each user. The format is documented in the passwd manual page.
/etc/fdprm
Floppy disk parameter table. Describes what different oppy disk formats look like. Used by setfdprm. See the setfdprm manual page for more information.
/etc/fstab
Lists the lesystems mounted automatically at startup by the mount -a command (in /etc/rc or equivalent startup le). Under Linux, also contains information about swap areas used automatically by swapon -a. See the section called Mounting and unmounting in Chapter 4 and the mount manual page for more information.
/etc/group
23
Similar to /etc/passwd, but describes groups instead of users. See the group manual page for more information.
/etc/inittab
/etc/issue
Output by getty before the login prompt. Usually contains a short description or welcoming message to the system. The contents are up to the system administrator.
/etc/magic
The conguration le for le. Contains the descriptions of various le formats based on which le guesses the type of the le. See the magic and le manual pages for more information.
/etc/motd
The message of the day, automatically output after a successful login. Contents are up to the system administrator. Often used for getting information to every user, such as warnings about planned downtimes.
/etc/mtab
List of currently mounted lesystems. Initially set up by the bootup scripts, and updated automatically by the mount command. Used when a list of mounted lesystems is needed, e.g., by the df command.
/etc/shadow
Shadow password le on systems with shadow password software installed. Shadow passwords move the encrypted password from /etc/passwd into /etc/shadow; the latter is not readable by anyone except root. This makes it harder to crack passwords.
/etc/login.defs
24
/etc/printcap
Files executed at login or startup time by the Bourne or C shells. These allow the system administrator to set global defaults for all users. See the manual pages for the respective shells.
/etc/securetty
Identies secure terminals, i.e., the terminals from which root is allowed to log in. Typically only the virtual consoles are listed, so that it becomes impossible (or at least harder) to gain superuser privileges by breaking into a system over a modem or a network.
/etc/shells
Lists trusted shells. The chsh command allows users to change their login shell only to shells listed in this le. ftpd, the server process that provides FTP services for a machine, will check that the users shell is listed in /etc/shells and will not let people log in unles the shell is listed there.
/etc/termcap
The terminal capability database. Describes by what escape sequences various terminals can be controlled. Programs are written so that instead of directly outputting an escape sequence that only works on a particular brand of terminal, they look up the correct sequence to do whatever it is they want to do in /etc/termcap. As a result most programs work with most kinds of terminals. See the termcap, curs_termcap, and terminfo manual pages for more information.
25
The /dev directory contains the special device les for all the devices. The device les are named using special conventions; these are described in the Device list (see XXX). The device les are created during installation, and later with the /dev/MAKEDEV script. The /dev/MAKEDEV.local is a script written by the system administrator that creates local-only device les or links (i.e., those that are not part of the standard MAKEDEV, such as device les for some non-standard device driver).
/usr/X11R6
The X Window System, all les. To simplify the development and installation of X, the X les have not been integrated into the rest of the system. There is a directory tree below /usr/X11R6 similar to that below /usr itself.
/usr/X386
/usr/bin
/usr/sbin
System administration commands that are not needed on the root lesystem, e.g., most server programs.
26
Manual pages, GNU Info documents, and miscellaneous other documentation les, respectively.
/usr/include
Header les for the C programming language. This should actually be below /usr/lib for consistency, but the tradition is overwhelmingly in support for this name.
/usr/lib
Unchanging data les for programs and subsystems, including some site-wide conguration les. The name lib comes from library; originally libraries of programming subroutines were stored in /usr/lib.
/usr/local
/var/catman
A cache for man pages that are formatted on demand. The source for manual pages is usually stored in /usr/man/man*; some manual pages might come with a pre-formatted version, which is stored in /usr/man/cat*. Other manual pages need to be formatted when they are rst viewed; the formatted version is then stored in /var/man so that the next person to view the same page wont have to wait for it to be formatted. (/var/catman is often cleaned in the same way temporary directories are cleaned.)
/var/lib
27
/var/local
Variable data for programs that are installed in /usr/local (i.e., programs that have been installed by the system administrator). Note that even locally installed programs should use the other /var directories if they are appropriate, e.g., /var/lock.
/var/lock
Lock les. Many programs follow a convention to create a lock le in /var/lock to indicate that they are using a particular device or le. Other programs will notice the lock le and wont attempt to use the device or le.
/var/log
Log les from various programs, especially login (/var/log/wtmp, which logs all logins and logouts into the system) and syslog (/var/log/messages , where all kernel and system program message are usually stored). Files in /var/log can often grow indenitely, and may require cleaning at regular intervals.
/var/run
Files that contain information about the system that is valid until the system is next booted. For example, /var/run/utmp contains information about people currently logged in.
/var/spool
Directories for mail, news, printer queues, and other queued work. Each different spool has its own subdirectory below /var/spool, e.g., the mailboxes of the users are in /var/spool/mail .
/var/tmp
Temporary les that are large or that need to exist for a longer time than what is allowed for /tmp. (Although the system administrator might not allow very old les in /var/tmp either.)
28
/proc/1
A directory with information about process number 1. Each process has a directory below /proc with the name being its process identication number.
/proc/cpuinfo
Information about the processor, such as its type, make, model, and perfomance.
/proc/devices
/proc/dma
/proc/filesystems
/proc/interrupts
Shows which interrupts are in use, and how many of each there have been.
/proc/ioports
29
/proc/kcore
An image of the physical memory of the system. This is exactly the same size as your physical memory, but does not really take up that much memory; it is generated on the y as programs access it. (Remember: unless you copy it elsewhere, nothing under /proc takes up any disk space at all.)
/proc/kmsg
/proc/ksyms
/proc/loadavg
The load average of the system; three meaningless indicators of how much work the system has to do at the moment.
/proc/meminfo
/proc/modules
/proc/net
/proc/self
A symbolic link to the process directory of the program that is looking at /proc. When two processes look at /proc, they get different links. This is mainly a convenience to make it easier for programs to get at their process directory.
30
/proc/stat
Various statistics about the system, such as the number of page faults since the system was booted.
/proc/uptime
/proc/version
Note that while the above les tend to be easily readable text les, they can sometimes be formatted in a way that is not easily digestable. There are many commands that do little more than read the above les and format them for easier understanding. For example, the free program reads /proc/meminfo and converts the amounts given in bytes to kilobytes (and adds a little more information, as well).
31
When you install or upgrade your system, you need to do a fair amount of work on your disks. You have to make lesystems on your disks so that les can be stored on them and reserve space for the different parts of your system. This chapter explains all these initial activities. Usually, once you get your system set up, you wont have to go through the work again, except for using oppies. Youll need to come back to this chapter if you add a new disk or want to ne-tune your disk usage. The basic tasks in administering disks are:
Format your disk. This does various things to prepare it for use, such as checking for bad sectors. (Formatting is nowadays not necessary for most hard disks.) Partition a hard disk, if you want to use it for several activities that arent supposed to interfere with one another. One reason for partitioning is to store different operating systems on the same disk. Another reason is to keep user les separate from system les, which simplies back-ups and helps protect the system les from corruption. Make a lesystem (of a suitable type) on each disk or partition. The disk means nothing to Linux until you make a lesystem; then les can be created and accessed on it. Mount different lesystems to form a single tree structure, either automatically, or manually as needed. (Manually mounted lesystems usually need to be unmounted manually as well.)
Chapter 5 contains information about virtual memory and disk caching, of which you also need to be aware when using disks.
32
(and no special application programming methodology, such as catching interrupts or polling a serial port) are necessary to access devices; for example, to send a le to the printer, one could just say
$ cat filename $
/dev/lp1
and the contents of the le are printed (the le must, of course, be in a form that the printer understands). However, since it is not a good idea to have several people cat their les to the printer at the same time, one usually uses a special program to send the les to be printed (usually lpr). This program makes sure that only one le is being printed at a time, and will automatically send les to the printer as soon as it nishes with the previous le. Something similar is needed for most devices. In fact, one seldom needs to worry about device les at all. Since devices show up as les in the lesystem (in the /dev directory), it is easy to see just what device les exist, using ls or another suitable command. In the output of ls -l, the rst column contains the type of the le and its permissions. For example, inspecting a serial device gives on my system
$ ls -l /dev/cua0 crw-rw-rw1 root $
uucp
5,
64 Nov 30
1993 /dev/cua0
The rst character in the rst column, i.e., c in crw-rw-rw- above, tells an informed user the type of the le, in this case a character device. For ordinary les, the rst character is -, for directories it is d, and for block devices b; see the ls man page for further information. Note that usually all device les exist even though the device itself might be not be installed. So just because you have a le /dev/sda, it doesnt mean that you really do have an SCSI hard disk. Having all the device les makes the installation programs simpler, and makes it easier to add new hardware (there is no need to nd out the correct parameters for and create the device les for the new device).
Hard disks
This subsection introduces terminology related to hard disks. If you already know the terms and concepts, you can skip this subsection. See Figure 4-1 for a schematic picture of the important parts in a hard disk. A hard disk consists of one or more circular platters, 1 of which either or both surfaces are coated with a magnetic substance used for recording the data. For each surface, there is a read-write head that examines or alters the recorded data. The platters rotate on a common axis; a typical rotation speed is 3600 rotations per minute, although high-performance hard disks have higher speeds. The heads move along the radius of the platters; this movement combined with the rotation of the platters allows the head to access all parts of the surfaces.
33
The processor (CPU) and the actual disk communicate through a disk controller. This relieves the rest of the computer from knowing how to use the drive, since the controllers for different types of disks can be made to use the same interface towards the rest of the computer. Therefore, the computer can say just hey disk, gimme what I want, instead of a long and complex series of electric signals to move the head to the proper location and waiting for the correct position to come under the head and doing all the other unpleasant stuff necessary. (In reality, the interface to the controller is still complex, but much less so than it would otherwise be.) The controller can also do some other stuff, such as caching, or automatic bad sector replacement. The above is usually all one needs to understand about the hardware. There is also a bunch of other stuff, such as the motor that rotates the platters and moves the heads, and the electronics that control the operation of the mechanical parts, but that is mostly not relevant for understanding the working principle of a hard disk. The surfaces are usually divided into concentric rings, called tracks, and these in turn are divided into sectors. This division is used to specify locations on the hard disk and to allocate disk space to les. To nd a given place on the hard disk, one might say surface 3, track 5, sector 7. Usually the number of sectors is the same for all tracks, but some hard disks put more sectors in outer tracks (all sectors are of the same physical size, so more of them t in the longer outer tracks). Typically, a sector will hold 512 bytes of data. The disk itself cant handle smaller amounts of data than one sector.
34
From above
Read/write head
Platter
Surfaces
Cylinder
35
Each surface is divided into tracks (and sectors) in the same way. This means that when the head for one surface is on a track, the heads for the other surfaces are also on the corresponding tracks. All the corresponding tracks taken together are called a cylinder . It takes time to move the heads from one track (cylinder) to another, so by placing the data that is often accessed together (say, a le) so that it is within one cylinder, it is not necessary to move the heads to read all of it. This improves performance. It is not always possible to place les like this; les that are stored in several places on the disk are called fragmented . The number of surfaces (or heads, which is the same thing), cylinders, and sectors vary a lot; the specication of the number of each is called the geometry of a hard disk. The geometry is usually stored in a special, battery-powered memory location called the CMOS RAM , from where the operating system can fetch it during bootup or driver initialization. Unfortunately, the BIOS 2 has a design limitation, which makes it impossible to specify a track number that is larger than 1024 in the CMOS RAM, which is too little for a large hard disk. To overcome this, the hard disk controller lies about the geometry, and translates the addresses given by the computer into something that ts reality. For example, a hard disk might have 8 heads, 2048 tracks, and 35 sectors per track. 3 Its controller could lie to the computer and claim that it has 16 heads, 1024 tracks, and 35 sectors per track, thus not exceeding the limit on tracks, and translates the address that the computer gives it by halving the head number, and doubling the track number. The math can be more complicated in reality, because the numbers are not as nice as here (but again, the details are not relevant for understanding the principle). This translation distorts the operating systems view of how the disk is organized, thus making it impractical to use the all-data-on-one-cylinder trick to boost performance. The translation is only a problem for IDE disks. SCSI disks use a sequential sector number (i.e., the controller translates a sequential sector number to a head, cylinder, and sector triplet), and a completely different method for the CPU to talk with the controller, so they are insulated from the problem. Note, however, that the computer might not know the real geometry of an SCSI disk either. Since Linux often will not know the real geometry of a disk, its lesystems dont even try to keep les within a single cylinder. Instead, it tries to assign sequentially numbered sectors to les, which almost always gives similar performance. The issue is further complicated by on-controller caches, and automatic prefetches done by the controller. Each hard disk is represented by a separate device le. There can (usually) be only two or four IDE hard disks. These are known as /dev/hda, /dev/hdb, /dev/hdc, and /dev/hdd, respectively. SCSI hard disks are known as /dev/sda, /dev/sdb, and so on. Similar naming conventions exist for other hard disk types; see XXX (device list) for more information. Note that the device les for the hard disks give access to the entire disk, with no regard to partitions (which will be discussed below), and its easy to mess up the partitions or the data in them if you arent careful. The disks device les are usually used only to get access to the master boot record (which will also be discussed below).
36
Floppies
A oppy disk consists of a exible membrane covered on one or both sides with similar magnetic substance as a hard disk. The oppy disk itself doesnt have a read-write head, that is included in the drive. A oppy corresponds to one platter in a hard disk, but is removable and one drive can be used to access different oppies, whereas the hard disk is one indivisible unit. Like a hard disk, a oppy is divided into tracks and sectors (and the two corresponding tracks on either side of a oppy form a cylinder), but there are many fewer of them than on a hard disk. A oppy drive can usually use several different types of disks; for example, a 3.5 inch drive can use both 720 kB and 1.44 MB disks. Since the drive has to operate a bit differently and the operating system must know how big the disk is, there are many device les for oppy drives, one per combination of drive and disk type. Therefore, /dev/fd0H1440 is the rst oppy drive (fd0), which must be a 3.5 inch drive, using a 3.5 inch, high density disk (H) of size 1440 kB (1440), i.e., a normal 3.5 inch HD oppy. For more information on the naming conventions for the oppy devices, see XXX (device list). The names for oppy drives are complex, however, and Linux therefore has a special oppy device type that automatically detects the type of the disk in the drive. It works by trying to read the rst sector of a newly inserted oppy using different oppy types until it nds the correct one. This naturally requires that the oppy is formatted rst. The automatic devices are called /dev/fd0, /dev/fd1, and so on. The parameters the automatic device uses to access a disk can also be set using the program setfdprm. This can be useful if you need to use disks that do not follow any usual oppy sizes, e.g., if they have an unusual number of sectors, or if the autodetecting for some reason fails and the proper device le is missing. Linux can handle many nonstandard oppy disk formats in addition to all the standard ones. Some of these require using special formatting programs. Well skip these disk types for now, but in the mean time you can examine the /etc/fdprm le. It species the settings that setfdprm recognizes. The operating system must know when a disk has been changed in a oppy drive, for example, in order to avoid using cached data from the previous disk. Unfortunately, the signal line that is used for this is sometimes broken, and worse, this wont always be noticeable when using the drive from within MS-DOS. If you are experiencing weird problems using oppies, this might be the reason. The only way to correct it is to repair the oppy drive.
CD-ROMs
A CD-ROM drive uses an optically read, plastic coated disk. The information is recorded on the surface of the disk 4 in small holes aligned along a spiral from the center to the edge. The drive directs a laser beam along the spiral to read the disk. When the laser hits a hole, the laser is reected in one way; when
37
it hits smooth surface, it is reected in another way. This makes it easy to code bits, and therefore information. The rest is easy, mere mechanics. CD-ROM drives are slow compared to hard disks. Whereas a typical hard disk will have an average seek time less than 15 milliseconds, a fast CD-ROM drive can use tenths of a second for seeks. The actual data transfer rate is fairly high at hundreds of kilobytes per second. The slowness means that CD-ROM drives are not as pleasant to use instead of hard disks (some Linux distributions provide live lesystems on CD-ROMs, making it unnecessary to copy the les to the hard disk, making installation easier and saving a lot of hard disk space), although it is still possible. For installing new software, CD-ROMs are very good, since it maximum speed is not essential during installation. There are several ways to arrange data on a CD-ROM. The most popular one is specied by the international standard ISO 9660. This standard species a very minimal lesystem, which is even more crude than the one MS-DOS uses. On the other hand, it is so minimal that every operating system should be able to map it to its native system. For normal UNIX use, the ISO 9660 lesystem is not usable, so an extension to the standard has been developed, called the Rock Ridge extension. Rock Ridge allows longer lenames, symbolic links, and a lot of other goodies, making a CD-ROM look more or less like any contemporary UNIX lesystem. Even better, a Rock Ridge lesystem is still a valid ISO 9660 lesystem, making it usable by non-UNIX systems as well. Linux supports both ISO 9660 and the Rock Ridge extensions; the extensions are recognized and used automatically. The lesystem is only half the battle, however. Most CD-ROMs contain data that requires a special program to access, and most of these programs do not run under Linux (except, possibly, under dosemu, the Linux MS-DOS emulator). A CD-ROM drive is accessed via the corresponding device le. There are several ways to connect a CD-ROM drive to the computer: via SCSI, via a sound card, or via EIDE. The hardware hacking needed to do this is outside the scope of this book, but the type of connection decides the device le. See XXX (device-list) for enlightment.
Tapes
A tape drive uses a tape, similar 5 to cassettes used for music. A tape is serial in nature, which means that in order to get to any given part of it, you rst have to go through all the parts in between. A disk can be accessed randomly, i.e., you can jump directly to any place on the disk. The serial access of tapes makes them slow. On the other hand, tapes are relatively cheap to make, since they do not need to be fast. They can also easily be made quite long, and can therefore contain a large amount of data. This makes tapes very suitable for things like archiving and backups, which do not require large speeds, but benet from low
38
Formatting
Formatting is the process of writing marks on the magnetic media that are used to mark tracks and sectors. Before a disk is formatted, its magnetic surface is a complete mess of magnetic signals. When it is formatted, some order is brought into the chaos by essentially drawing lines where the tracks go, and where they are divided into sectors. The actual details are not quite exactly like this, but that is irrelevant. What is important is that a disk cannot be used unless it has been formatted. The terminology is a bit confusing here: in MS-DOS, the word formatting is used to cover also the process of creating a lesystem (which will be discussed below). There, the two processes are often combined, especially for oppies. When the distinction needs to be made, the real formatting is called low-level formatting, while making the lesystem is called high-level formatting. In UNIX circles, the two are called formatting and making a lesystem, so thats what is used in this book as well. For IDE and some SCSI disks the formatting is actually done at the factory and doesnt need to be repeated; hence most people rarely need to worry about it. In fact, formatting a hard disk can cause it to work less well, for example because a disk might need to be formatted in some very special way to allow automatic bad sector replacement to work. Disks that need to be or can be formatted often require a special program anyway, because the interface to the formatting logic inside the drive is different from drive to drive. The formatting program is often either on the controller BIOS, or is supplied as an MS-DOS program; neither of these can easily be used from within Linux. During formatting one might encounter bad spots on the disk, called bad blocks or bad sectors. These are sometimes handled by the drive itself, but even then, if more of them develop, something needs to be done to avoid using those parts of the disk. The logic to do this is built into the lesystem; how to add the information into the lesystem is described below. Alternatively, one might create a small partition that covers just the bad part of the disk; this approach might be a good idea if the bad spot is very large, since lesystems can sometimes have trouble with very large bad areas. Floppies are formatted with fdformat. The oppy device le to use is given as the parameter. For example, the following command would format a high density, 3.5 inch oppy in the rst oppy drive:
$ fdformat /dev/fd0H1440 Double-sided, 80 tracks, 18 sec/track. Total capacity 1440 kB. Formatting ... done Verifying ... done $
39
Note that if you want to use an autodetecting device (e.g., /dev/fd0), you must set the parameters of the device with setfdprm rst. To achieve the same effect as above, one would have to do the following:
$ setfdprm /dev/fd0 1440/1440 $ fdformat /dev/fd0 Double-sided, 80 tracks, 18 sec/track. Total capacity 1440 kB. Formatting ... done Verifying ... done $
It is usually more convenient to choose the correct device le that matches the type of the oppy. Note that it is unwise to format oppies to contain more information than what they are designed for. fdformat will also validate the oppy, i.e., check it for bad blocks. It will try a bad block several times (you can usually hear this, the drive noise changes dramatically). If the oppy is only marginally bad (due to dirt on the read/write head, some errors are false signals), fdformat wont complain, but a real error will abort the validation process. The kernel will print log messages for each I/O error it nds; these will go to the console or, if syslog is being used, to the le /usr/log/messages . fdformat itself wont tell where the error is (one usually doesnt care, oppies are cheap enough that a bad one is automatically thrown away).
$ fdformat /dev/fd0H1440 Double-sided, 80 tracks, 18 sec/track. Total capacity 1440 kB. Formatting ... done Verifying ... read: Unknown error $
The badblocks command can be used to search any disk or partition for bad blocks (including a oppy). It does not format the disk, so it can be used to check even existing lesystems. The example below checks a 3.5 inch oppy with two bad blocks.
$ badblocks /dev/fd0H1440 1440 718 719 $
badblocks outputs the block numbers of the bad blocks it nds. Most lesystems can avoid such bad blocks. They maintain a list of known bad blocks, which is initialized when the lesystem is made, and can be modied later. The initial search for bad blocks can be done by the mkfs command (which initializes the lesystem), but later checks should be done with badblocks and the new blocks should be added with fsck. Well describe mkfs and fsck later. Many modern disks automatically notice bad blocks, and attempt to x them by using a special, reserved good block instead. This is invisible to the operating system. This feature should be documented in the
40
disks manual, if youre curious if it is happening. Even such disks can fail, if the number of bad blocks grows too large, although chances are that by then the disk will be so rotten as to be unusable.
Partitions
A hard disk can be divided into several partitions. Each partition functions as if it were a separate hard disk. The idea is that if you have one hard disk, and want to have, say, two operating systems on it, you can divide the disk into two partitions. Each operating system uses its partition as it wishes and doesnt touch the other ones. This way the two operating systems can co-exist peacefully on the same hard disk. Without partitions one would have to buy a hard disk for each operating system. Floppies are not partitioned. There is no technical reason against this, but since theyre so small, partitions would be useful only very rarely. CD-ROMs are usually also not partitioned, since its easier to use them as one big disk, and there is seldom a need to have several operating systems on one.
41
Id 82 83 83 5 83 83
System Linux swap Linux native Linux native Extended Linux native Linux native
42
MBR Boot sector Data area of partition Boot sector Unused boot sector Data area Unused boot sector Data area Unused disk space Boot sector Data area Primary partition Extended partition Primary partition
Partition types
The partition tables (the one in the MBR, and the ones for extended partitions) contain one byte per partition that identies the type of that partition. This attempts to identify the operating system that uses the partition, or what it uses it for. The purpose is to make it possible to avoid having two operating systems accidentally using the same partition. However, in reality, operating systems do not really care about the partition type byte; e.g., Linux doesnt care at all what it is. Worse, some of them use it incorrectly; e.g., at least some versions of DR-DOS ignore the most signicant bit of the byte, while others dont. There is no standardization agency to specify what each byte value means, but some commonly accepted ones are included in in Table 4-1. The same list is available in the Linux fdisk program. Table 4-1. Partition types (from the Linux fdisk program). 0 Empty 40 Venix 80286 94 Amoeba BBT
43
1 2 3 4
DOS 12-bit FAT 51 XENIX root XENIX usr DOS 16-bitf 32M Extended
Novell? Microport GNU HURD Novell PC/IX Old MINIX Linux/MINIX Linux swap Linux native Amoeba
a5 b7 b8 c7 db e1 e3 f2 ff
BSD/386 BSDI fs BSDI swap Syrinx CP/M DOS access DOS R/O DOS secondary BBT
52 63 64 75 80 81 82 83 93
5 6 7 8 9 a
44
result in any problems, but it is ugly, and some versions of fdisk will warn about it. Changing a partitions size usually requires rst backing up everything you want to save from that partition (preferably the whole disk, just in case), deleting the partition, creating new partition, then restoring everything to the new partition. If the partition is growing, you may need to adjust the sizes (and backup and restore) of the adjoining partitions as well. Since changing partition sizes is painful, it is preferable to get the partitions right the rst time, or have an effective and easy to use backup system. If youre installing from a media that does not require much human intervention (say, from CD-ROM, as opposed to oppies), it is often easy to play with different conguration at rst. Since you dont already have data to back up, it is not so painful to modify partition sizes several times. There is a program for MS-DOS, called ps, which resizes an MS-DOS partition without requiring the backup and restore, but for other lesystems it is still necessary.
Filesystems
What are lesystems?
A lesystem is the methods and data structures that an operating system uses to keep track of les on a disk or partition; that is, the way the les are organized on the disk. The word is also used to refer to a partition or disk that is used to store the les or the type of the lesystem. Thus, one might say I have two lesystems meaning one has two partitions on which one stores les, or that one is using the extended lesystem, meaning the type of the lesystem. The difference between a disk or partition and the lesystem it contains is important. A few programs (including, reasonably enough, programs that create lesystems) operate directly on the raw sectors of a disk or partition; if there is an existing le system there it will be destroyed or seriously corrupted. Most
45
programs operate on a lesystem, and therefore wont work on a partition that doesnt contain one (or that contains one of the wrong type). Before a partition or disk can be used as a lesystem, it needs to be initialized, and the bookkeeping data structures need to be written to the disk. This process is called making a lesystem. Most UNIX lesystem types have a similar general structure, although the exact details vary quite a bit. The central concepts are superblock, inode, data block, directory block, and indirection block. The superblock contains information about the lesystem as a whole, such as its size (the exact information here depends on the lesystem). An inode contains all information about a le, except its name. The name is stored in the directory, together with the number of the inode. A directory entry consists of a lename and the number of the inode which represents the le. The inode contains the numbers of several data blocks, which are used to store the data in the le. There is space only for a few data block numbers in the inode, however, and if more are needed, more space for pointers to the data blocks is allocated dynamically. These dynamically allocated blocks are indirect blocks; the name indicates that in order to nd the data block, one has to nd its number in the indirect block rst. UNIX lesystems usually allow one to create a hole in a le (this is done with lseek; check the manual page), which means that the lesystem just pretends that at a particular place in the le there is just zero bytes, but no actual disk sectors are reserved for that place in the le (this means that the le will use a bit less disk space). This happens especially often for small binaries, Linux shared libraries, some databases, and a few other special cases. (Holes are implemented by storing a special value as the address of the data block in the indirect block or inode. This special address means that no data block is allocated for that part of the le, ergo, there is a hole in the le.) Holes are moderately useful. On the authors system, a simple measurement showed a potential for about 4 MB of savings through holes of about 200 MB total used disk space. That system, however, contains relatively few programs and no database les.
Filesystems galore
Linux supports several types of lesystems. As of this writing the most important ones are:
minix The oldest, presumed to be the most reliable, but quite limited in features (some time stamps are missing, at most 30 character lenames) and restricted in capabilities (at most 64 MB per lesystem).
xia
46
A modied version of the minix lesystem that lifts the limits on the lenames and lesystem sizes, but does not otherwise introduce new features. It is not very popular, but is reported to work very well.
ext2 The most featureful of the native Linux lesystems, currently also the most popular one. It is designed to be easily upwards compatible, so that new versions of the lesystem code do not require re-making the existing lesystems.
ext An older version of ext2 that wasnt upwards compatible. It is hardly ever used in new installations any more, and most people have converted to ext2.
In addition, support for several foreign lesystem exists, to make it easier to exchange les with other operating systems. These foreign lesystems work just like native ones, except that they may be lacking in some usual UNIX features, or have curious limitations, or other oddities.
msdos Compatibility with MS-DOS (and OS/2 and Windows NT) FAT lesystems.
usmdos Extends the msdos lesystem driver under Linux to get long lenames, owners, permissions, links, and device les. This allows a normal msdos lesystem to be used as if it were a Linux one, thus removing the need for a separate partition for Linux.
iso9660 The standard CD-ROM lesystem; the popular Rock Ridge extension to the CD-ROM standard that allows longer le names is supported automatically.
nfs
47
A networked lesystem that allows sharing a lesystem between many computers to allow easy access to the les from all of them.
The choice of lesystem to use depends on the situation. If compatibility or other reasons make one of the non-native lesystems necessary, then that one must be used. If one can choose freely, then it is probably wisest to use ext2, since it has all the features but does not suffer from lack of performance. There is also the proc lesystem, usually accessible as the /proc directory, which is not really a lesystem at all, even though it looks like one. The proc lesystem makes it easy to access certain kernel data structures, such as the process list (hence the name). It makes these data structures look like a lesystem, and that lesystem can be manipulated with all the usual le tools. For example, to get a listing of all processes one might use the command
$ ls -l /proc total 0 dr-xr-xr-x 4 root dr-xr-xr-x 4 liw dr-xr-xr-x 4 liw dr-xr-xr-x 4 liw dr-xr-xr-x 4 root dr-xr-xr-x 4 liw -r-r-r1 root -r-r-r1 root -r-r-r1 root -r-r-r1 root -r----1 root -r-r-r1 root -r-r-r1 root -r-r-r1 root -r-r-r1 root -r-r-r1 root dr-xr-xr-x 2 root dr-xr-xr-x 4 root
root users users users users users root root root root root root root root root root root root
0 0 0 0 8654848 0 0 0 0 0
0 Jan 0 Jan 0 Jan 0 Jan 0 Jan 0 Jan Jan 31 Jan 31 Jan 31 Jan 31 Jan 31 Jan 31 Jan 31 Jan 31 Jan 31 Jan 31 0 Jan 0 Jan
31 20:37 1 31 20:37 63 31 20:37 94 31 20:37 95 31 20:37 98 31 20:37 99 20:37 devices 20:37 dma 20:37 filesystems 20:37 interrupts 20:37 kcore 11:50 kmsg 20:37 ksyms 11:51 loadavg 20:37 meminfo 20:37 modules 31 20:37 net 31 20:37 self
48
-r-r-r-r-r-r-r-r-r$
(There will be a few extra les that dont correspond to processes, though. The above example has been shortened.) Note that even though it is called a lesystem, no part of the proc lesystem touches any disk. It exists only in the kernels imagination. Whenever anyone tries to look at any part of the proc lesystem, the kernel makes it look as if the part existed somewhere, even though it doesnt. So, even though there is a multi-megabyte /proc/kcore le, it doesnt take any disk space.
Creating a lesystem
Filesystems are created, i.e., initialized, with the mkfs command. There is actually a separate program for each lesystem type. mkfs is just a front end that runs the appropriate program depending on the desired lesystem type. The type is selected with the -t fstype option. The programs called by mkfs have slightly different command line interfaces. The common and most important options are summarized below; see the manual pages for more.
-t fstype
-c
Search for bad blocks and initialize the bad block list accordingly.
-l lename
49
Read the initial bad block list from the name le.
To create an ext2 lesystem on a oppy, one would give the following commands:
$ fdformat -n /dev/fd0H1440 Double-sided, 80 tracks, 18 sec/track. Total capacity 1440 kB. Formatting ... done $ badblocks /dev/fd0H1440 1440 $>$ bad-blocks $ mkfs -t ext2 -l bad-blocks /dev/fd0H1440 mke2fs 0.5a, 5-Apr-94 for EXT2 FS 0.5, 94/03/10 360 inodes, 1440 blocks 72 blocks (5.00%) reserved for the super user First data block=1 Block size=1024 (log=0) Fragment size=1024 (log=0) 1 block group 8192 blocks per group, 8192 fragments per group 360 inodes per group Writing inode tables: done Writing superblocks and filesystem accounting information: done $
First, the oppy was formatted (the -n option prevents validation, i.e., bad block checking). Then bad blocks were searched with badblocks, with the output redirected to a le, bad-blocks. Finally, the lesystem was created, with the bad block list initialized by whatever badblocks found. The -c option could have been used with mkfs instead of badblocks and a separate le. The example below does that.
$ mkfs -t ext2 -c /dev/fd0H1440 mke2fs 0.5a, 5-Apr-94 for EXT2 FS 0.5, 94/03/10 360 inodes, 1440 blocks 72 blocks (5.00%) reserved for the super user First data block=1 Block size=1024 (log=0) Fragment size=1024 (log=0) 1 block group 8192 blocks per group, 8192 fragments per group 360 inodes per group Checking for bad blocks (read-only test): done Writing inode tables: done Writing superblocks and filesystem accounting information: done
50
The -c option is more convenient than a separate use of badblocks, but badblocks is necessary for checking after the lesystem has been created. The process to prepare lesystems on hard disks or partitions is the same as for oppies, except that the formatting isnt needed.
bin
dev
home
etc lib
usr
abc
liw
ftp
bin
etc
lib
51
bin
dev
home
etc lib
usr
abc
liw
ftp
bin
etc
lib
The mount command takes two arguments. The rst one is the device le corresponding to the disk or partition containing the lesystem. The second one is the directory below which it will be mounted. After these commands the contents of the two lesystems look just like the contents of the /home and /usr directories, respectively. One would then say that /dev/hda2 is mounted on /home, and similarly for /usr. To look at either lesystem, one would look at the contents of the directory on which it has been mounted, just as if it were any other directory. Note the difference between the device le, /dev/hda2, and the mounted-on directory, /home. The device le gives access to the raw contents of the disk, the mounted-on directory gives access to the les on the disk. The mounted-on directory is called the mount point. Linux supports many lesystem types. mount tries to guess the type of the lesystem. You can also use the -t fstype option to specify the type directly; this is sometimes necessary, since the heuristics mount uses do not always work. For example, to mount an MS-DOS oppy, you could use the following command:
$ mount -t msdos /dev/fd0 /floppy $
The mounted-on directory need not be empty, although it must exist. Any les in it, however, will be inaccessible by name while the lesystem is mounted. (Any les that have already been opened will still be accessible. Files that have hard links from other directories can be accessed using those names.) There is no harm done with this, and it can even be useful. For instance, some people like to have /tmp and
52
/var/tmp synonymous, and make /tmp be a symbolic link to /var/tmp. When the system is booted, before the /var lesystem is mounted, a /var/tmp directory residing on the root lesystem is used instead. When /var is mounted, it will make the /var/tmp directory on the root lesystem inaccessible. If /var/tmp didnt exist on the root lesystem, it would be impossible to use temporary les before mounting /var.
If you dont intend to write anything to the lesystem, use the -r switch for mount to do a readonly mount. This will make the kernel stop any attempts at writing to the lesystem, and will also stop the kernel from updating le access times in the inodes. Read-only mounts are necessary for unwritable media, e.g., CD-ROMs. The alert reader has already noticed a slight logistical problem. How is the rst lesystem (called the root lesystem, because it contains the root directory) mounted, since it obviously cant be mounted on another lesystem? Well, the answer is that it is done by magic. 7 The root lesystem is magically mounted at boot time, and one can rely on it to always be mounted. If the root lesystem cant be mounted, the system does not boot. The name of the lesystem that is magically mounted as root is either compiled into the kernel, or set using LILO or rdev. The root lesystem is usually rst mounted readonly. The startup scripts will then run fsck to verify its validity, and if there are no problems, they will re-mount it so that writes will also be allowed. fsck must not be run on a mounted lesystem, since any changes to the lesystem while fsck is running will cause trouble. Since the root lesystem is mounted readonly while it is being checked, fsck can x any problems without worry, since the remount operation will ush any metadata that the lesystem keeps in memory. On many systems there are other lesystems that should also be mounted automatically at boot time. These are specied in the /etc/fstab le; see the fstab man page for details on the format. The details of exactly when the extra lesystems are mounted depend on many factors, and can be congured by each administrator if need be; see Chapter 6. When a lesystem no longer needs to be mounted, it can be unmounted with umount. 8 umount takes one argument: either the device le or the mount point. For example, to unmount the directories of the previous example, one could use the commands
$ umount /dev/hda2 $ umount /usr $
See the man page for further instructions on how to use the command. It is imperative that you always unmount a mounted oppy. Dont just pop the oppy out of the drive! Because of disk caching, the data is not necessarily written to the oppy until you unmount it, so removing the oppy from the drive too early might cause the contents to become garbled. If you only read from the oppy, this is not very likely, but if you write, even accidentally, the result may be catastrophic.
53
Mounting and unmounting requires super user privileges, i.e., only root can do it. The reason for this is that if any user can mount a oppy on any directory, then it is rather easy to create a oppy with, say, a Trojan horse disguised as /bin/sh, or any other often used program. However, it is often necessary to allow users to use oppies, and there are several ways to do this:
Give the users the root password. This is obviously bad security, but is the easiest solution. It works well if there is no need for security anyway, which is the case on many non-networked, personal systems. Use a program such as sudo to allow users to use mount. This is still bad security, but doesnt directly give super user privileges to everyone. 9 Make the users use mtools, a package for manipulating MS-DOS lesystems, without mounting them. This works well if MS-DOS oppies are all that is needed, but is rather awkward otherwise. List the oppy devices and their allowable mount points together with the suitable options in /etc/fstab.
The last alternative can be implemented by adding a line like the following to the /etc/fstab le:
/dev/fd0 /floppy msdos user,noauto 0 0
The columns are: device le to mount, directory to mount on, lesystem type, options, backup frequency (used by dump), and fsck pass number (to specify the order in which lesystems should be checked upon boot; 0 means no check). The noauto option stops this mount to be done automatically when the system is started (i.e., it stops mount -a from mounting it). The user option allows any user to mount the lesystem, and, because of security reasons, disallows execution of programs (normal or setuid) and interpretation of device les from the mounted lesystem. After this, any user can mount a oppy with an msdos lesystem with the following command:
$ mount /floppy $
The oppy can (and needs to, of course) be unmounted with the corresponding umount command. If you want to provide access to several types of oppies, you need to give several mount points. The settings can be different for each mount point. For example, to give access to both MS-DOS and ext2 oppies, you could have the following to lines in /etc/fstab:
/dev/fd0 /dev/fd0 /dosfloppy /ext2floppy msdos ext2 user,noauto user,noauto 0 0 0 0
For MS-DOS lesystems (not just oppies), you probably want to restrict access to it by using the uid, gid, and umask lesystem options, described in detail on the mount manual page. If you arent careful,
54
mounting an MS-DOS lesystem gives everyone at least read access to the les in it, which is not a good idea.
55
/dev/fd0H1440: ***** FILE SYSTEM WAS MODIFIED ***** /dev/fd0H1440: 11/360 files, 63/1440 blocks $
If badblocks reports a block that was already used, e2fsck will try to move the block to another place. If the block was really bad, not just marginal, the contents of the le may be corrupted.
Fighting fragmentation
When a le is written to disk, it cant always be written in consecutive blocks. A le that is not stored in consecutive blocks is fragmented . It takes longer to read a fragmented le, since the disks read-write head will have to move more. It is desireable to avoid fragmentation, although it is less of a problem in a system with a good buffer cache with read-ahead. The ext2 lesystem attempts to keep fragmentation at a minimum, by keeping all blocks in a le close together, even if they cant be stored in consecutive sectors. Ext2 effectively always allocates the free block that is nearest to other blocks in a le. For ext2, it is therefore seldom necessary to worry about fragmentation. There is a program for defragmenting an ext2 lesystem, see XXX (ext2-defrag) in the bibliography. There are many MS-DOS defragmentation programs that move blocks around in the lesystem to remove fragmentation. For other lesystems, defragmentation must be done by backing up the lesystem, re-creating it, and restoring the les from backups. Backing up a lesystem before defragmening is a good idea for all lesystems, since many things can go wrong during the defragmentation.
56
A maximal mount count. e2fsck enforces a check when lesystem has been mounted too many times, even if the clean ag is set. For a system that is used for developing or testing the system, it might be a good idea to reduce this limit. A maximal time between checks. e2fsck can also enforce a maximal time between two checks, even if the clean ag is set, and the lesystem hasnt been mounted very often. This can be disabled, however. Number of blocks reserved for root. Ext2 reserves some blocks for root so that if the lesystem lls up, it is still possible to do system administration without having to delete anything. The reserved amount is by default 5 percent, which on most disks isnt enough to be wasteful. However, for oppies there is no point in reserving any blocks.
See the tune2fs manual page for more information. dumpe2fs shows information about an ext2 lesystem, mostly from the superblock. Figure 4-5 shows a sample output. Some of the information in the output is technical and requires understanding of how the lesystem works (see appendix XXX ext2fspaper), but much of it is readily understandable even for layadmins. Figure 4-5. Sample output from dumpe2fs dumpe2fs 0.5b, 11-Mar-95 for EXT2 FS 0.5a, 94/10/23 Filesystem magic number: 0xEF53 Filesystem state: clean Errors behavior: Continue
57
Inode count: 360 Block count: 1440 Reserved block count: 72 Free blocks: 1133 Free inodes: 326 First block: 1 Block size: 1024 Fragment size: 1024 Blocks per group: 8192 Fragments per group: 8192 Inodes per group: 360 Last mount time: Tue Aug 8 01:52:52 1995 Last write time: Tue Aug 8 01:53:28 1995 Mount count: 3 Maximum mount count: 20 Last checked: Tue Aug 8 01:06:31 1995 Check interval: 0 Reserved blocks uid: 0 (user root) Reserved blocks gid: 0 (group root) Group 0: Block bitmap at 3, Inode bitmap at 4, Inode table at 5 1133 free blocks, 326 free inodes, 2 directories Free blocks: 307-1439 Free inodes: 35-360 debugfs is a lesystem debugger. It allows direct access to the lesystem data structures stored on disk and can thus be used to repair a disk that is so broken that fsck cant x it automatically. It has also been known to be used to recover deleted les. However, debugfs very much requires that you understand what youre doing; a failure to understand can destroy all your data. dump and restore can be used to back up an ext2 lesystem. They are ext2 specic versions of the traditional UNIX backup tools. See Chapter 10 for more information on backups.
58
written directly on the raw disk, without a lesystem. Linux boot oppies dont contain a lesystem, only the raw kernel. Avoiding a lesystem has the advantage of making more of the disk usable, since a lesystem always has some bookkeeping overhead. It also makes the disks more easily compatible with other systems: for example, the tar le format is the same on all systems, while lesystems are different on most systems. You will quickly get used to disks without lesystems if you need them. Bootable Linux oppies also do not necessarily have a lesystem, although that is also possible. One reason to use raw disks is to make image copies of them. For instance, if the disk contains a partially damaged lesystem, it is a good idea to make an exact copy of it before trying to x it, since then you can start again if your xing breaks things even more. One way to do this is to use dd:
$ dd if=/dev/fd0H1440 of=floppy-image 2880+0 records in 2880+0 records out $ dd if=floppy-image of=/dev/fd0H1440 2880+0 records in 2880+0 records out $
The rst dd makes an exact image of the oppy to the le floppy-image, the second one writes the image to the oppy. (The user has presumably switched the oppy before the second command. Otherwise the command pair is of doubtful usefulness.)
59
by using NFS), thereby reducing the total disk space required by several tens or hundreds of megabytes times the number of machines. The problem with having many partitions is that it splits the total amount of free disk space into many small pieces. Nowadays, when disks and (hopefully) operating systems are more reliable, many people prefer to have just one partition that holds all their les. On the other hand, it can be less painful to back up (and restore) a small partition. For a small hard disk (assuming you dont do kernel development), the best way to go is probably to have just one partition. For large hard disks, it is probably better to have a few large partitions, just in case something does go wrong. (Note that small and large are used in a relative sense here; your needs for disk space decide what the threshold is.) If you have several disks, you might wish to have the root lesystem (including /usr) on one, and the users home directories on another. It is a good idea to be prepared to experiment a bit with different partitioning schemes (over time, not just while rst installing the system). This is a bit of work, since it essentially requires you to install the system from scratch several times, but it is the only way to be sure you do it right.
Space requirements
The Linux distribution you install will give some indication of how much disk space you need for various congurations. Programs installed separately may also do the same. This will help you plan your disk space usage, but you should prepare for the future and reserve some extra space for things you will notice later that you need. The amount you need for user les depends on what your users wish to do. Most people seem to need as much space for their les as possible, but the amount they will live happily with varies a lot. Some people do only light text processing and will survive nicely with a few megabytes, others do heavy image processing and will need gigabytes. By the way, when comparing le sizes given in kilobytes or megabytes and disk space given in megabytes, it can be important to know that the two units can be different. Some disk manufacturers like to pretend that a kilobyte is 1000 bytes and a megabyte is 1000 kilobytes, while all the rest of the computing world uses 1024 for both factors. Therefore, my 345 MB hard disk was really a 330 MB hard disk. 10 Swap space allocation is discussed in the section called Allocating swap space in Chapter 5.
60
I used to have a 109 MB hard disk. Now I am using a 330 MB hard disk. Ill explain how and why I partitioned these disks. The 109 MB disk I partitioned in a lot of ways, when my needs and the operating systems I used changed; Ill explain two typical scenarios. First, I used to run MS-DOS together with Linux. For that, I needed about 20 MB of hard disk, or just enough to have MS-DOS, a C compiler, an editor, a few other utilities, the program I was working on, and enough free disk space to not feel claustrophobic. For Linux, I had a 10 MB swap partition, and the rest, or 79 MB, was a single partition with all the les I had under Linux. I experimented with having separate root, /usr, and /home partitions, but there was never enough free disk space in one piece to do much interesting. When I didnt need MS-DOS anymore, I repartitioned the disk so that I had a 12 MB swap partition, and again had the rest as a single lesystem. The 330 MB disk is partitioned into several partitions, like this: 5 MB 10 MB 180 MB 120 MB 15 MB root lesystem swap partition
/usr filesystem /home filesystem
scratch partition
The scratch partition is for playing around with things that require their own partition, e.g., trying different Linux distributions, or comparing speeds of lesystems. When not needed for anything else, it is used as swap space (I like to have a lot of open windows).
61
example, some on-line documentation might be unnecessary, as might some of the Elisp les for GNU Emacs, some of the fonts for X11, or some of the libraries for programming. If you cannot uninstall packages, you might look into compression. Compression programs such as gzip or zip will compress (and uncompress) individual les or groups of les. The gzexe system will compress and uncompress programs invisibly to the user (unused programs are compressed, then uncompressed as they are used). The experimental DouBle system will compress all les in a lesystem, invisibly to the programs that use them. (If you are familiar with products such as Stacker for MS-DOS, the principle is the same.)
Notes
1. The platters are made of a hard substance, e.g., aluminium, which gives the hard disk its name. 2. The BIOS is some built-in software stored on ROM chips. It takes care, among other things, of the initial stages of booting. 3. The numbers are completely imaginary. 4. That is, the surface inside the disk, on the metal disk inside the plastic coating. 5. But completely different, of course. 6. Illogical? 7. For more information, see the kernel source or the Kernel Hackers Guide. 8. It should of course be unmount, but the n mysteriously disappeared in the 70s, and hasnt been seen since. Please return it to Bell Labs, NJ, if you nd it. 9. It requires several seconds of hard thinking on the users behalf. 10. Sic transit discus mundi.
62
This section describes the Linux memory management features, i.e., virtual memory and the disk buffer cache. The purpose and workings and the things the system administrator needs to take into consideration are described.
63
A swap le is an ordinary le; it is in no way special to the kernel. The only thing that matters to the kernel is that it has no holes, and that it is prepared for use with mkswap. It must reside on a local disk, however; it cant reside in a lesystem that has been mounted over NFS due to implementation reasons. The bit about holes is important. The swap le reserves the disk space so that the kernel can quickly swap out a page without having to go through all the things that are necessary when allocating a disk sector to a le. The kernel merely uses any sectors that have already been allocated to the le. Because a hole in a le means that there are no disk sectors allocated (for that place in the le), it is not good for the kernel to try to use them. One good way to create the swap le without holes is through the following command:
$ dd if=/dev/zero of=/extra-swap bs=1024 count=1024 1024+0 records in 1024+0 records out $
where /extra-swap is the name of the swap le and the size of is given after the count=. It is best for the size to be a multiple of 4, because the kernel writes out memory pages, which are 4 kilobytes in size. If the size is not a multiple of 4, the last couple of kilobytes may be unused. A swap partition is also not special in any way. You create it just like any other partition; the only difference is that it is used as a raw partition, that is, it will not contain any lesystem at all. It is a good idea to mark swap partitions as type 82 (Linux swap); this will the make partition listings clearer, even though it is not strictly necessary to the kernel. After you have created a swap le or a swap partition, you need to write a signature to its beginning; this contains some administrative information and is used by the kernel. The command to do this is mkswap, used like this:
$ mkswap /extra-swap 1024 Setting up swapspace, size = 1044480 bytes $
Note that the swap space is still not in use yet: it exists, but the kernel does not use it to provide virtual memory. You should be very careful when using mkswap, since it does not check that the le or partition isnt used for anything else. You can easily overwrite important les and partitions with mkswap! Fortunately, you should only need to use mkswap when you install your system. The Linux memory manager limits the size of each swap space to about 127 MB (for various technical reasons, the actual limit is (4096-10) * 8 * 4096 = 133890048$ bytes, or 127.6875 megabytes). You can, however, use up to 8 swap spaces simultaneously, for a total of almost 1 GB. 2
64
Swap spaces can be used automatically by listing them in the /etc/fstab le.
/dev/hda8 /swapfile none none swap swap sw sw 0 0 0 0
The startup scripts will run the command swapon -a, which will start swapping on all the swap spaces listed in /etc/fstab. Therefore, the swapon command is usually used only when extra swap is needed. You can monitor the use of swap spaces with free. It will tell the total amount of swap space used.
$ free total Mem: 15152 -/+ buffers: Swap: 32452 $ used 14896 12368 6684 free 256 2784 25768 shared 12404 buffers 2528
The rst line of output (Mem:) shows the physical memory. The total column does not show the physical memory used by the kernel, which is usually about a megabyte. The used column shows the amount of memory used (the second line does not count buffers). The free column shows completely unused memory. The shared column shows the amount of memory shared by several processes; the more, the merrier. The buffers column shows the current size of the disk buffer cache. That last line (Swap:) shows similar information for the swap spaces. If this line is all zeroes, your swap space is not activated. The same information is available via top, or using the proc lesystem in le /proc/meminfo. It is currently difcult to get information on the use of a specic swap space. A swap space can be removed from use with swapoff. It is usually not necessary to do it, except for temporary swap spaces. Any pages in use in the swap space are swapped in rst; if there is not sufcient physical memory to hold them, they will then be swapped out (to some other swap space). If there is not enough virtual memory to hold all of the pages Linux will start to thrash; after a long while it should recover, but meanwhile the system is unusable. You should check (e.g., with free) that there is enough free memory before removing a swap space from use.
65
All the swap spaces that are used automatically with swapon -a can be removed from use with swapoff -a; it looks at the le /etc/fstab to nd what to remove. Any manually used swap spaces will remain in use. Sometimes a lot of swap space can be in use even though there is a lot of free physical memory. This can happen for instance if at one point there is need to swap, but later a big process that occupied much of the physical memory terminates and frees the memory. The swapped-out data is not automatically swapped in until it is needed, so the physical memory may remain free for a long time. There is no need to worry about this, but it can be comforting to know what is happening.
Estimate your total memory needs. This is the largest amount of memory youll probably need at a time, that is the sum of the memory requirements of all the programs you want to run at the same time. This can be done by running at the same time all the programs you are likely to ever be running at the same time. For instance, if you want to run X, you should allocate about 8 MB for it, gcc wants several megabytes (some les need an unusually large amount, up to tens of megabytes, but usually about four should do), and so on. The kernel will use about a megabyte by itself, and the usual shells and other small utilities perhaps a few hundred kilobytes (say a megabyte together). There is no need to try to be exact, rough estimates are ne, but you might want to be on the pessimistic side. Remember that if there are going to be several people using the system at the same time, they are all going to consume memory. However, if two people run the same program at the same time, the total memory consumption is usually not double, since code pages and shared libraries exist only once. The free and ps commands are useful for estimating the memory needs.
66
Add some security to the estimate in step 1. This is because estimates of program sizes will probably be wrong, because youll probably forget some programs you want to run, and to make certain that you have some extra space just in case. A couple of megabytes should be ne. (It is better to allocate too much than too little swap space, but theres no need to over-do it and allocate the whole disk, since unused swap space is wasted space; see later about adding more swap.) Also, since it is nicer to deal with even numbers, you can round the value up to the next full megabyte. Based on the computations above, you know how much memory youll be needing in total. So, in order to allocate swap space, you just need to subtract the size of your physical memory from the total memory needed, and you know how much swap space you need. (On some versions of UNIX, you need to allocate space for an image of the physical memory as well, so the amount computed in step 2 is what you need and you shouldnt do the subtraction.) If your calculated swap space is very much larger than your physical memory (more than a couple times larger), you should probably invest in more physical memory, otherwise performance will be too low.
Its a good idea to have at least some swap space, even if your calculations indicate that you need none. Linux uses swap space somewhat aggressively, so that as much physical memory as possible can be kept free. Linux will swap out memory pages that have not been used, even if the memory is not yet needed for anything. This avoids waiting for swapping when it is needed: the swapping can be done earlier, when the disk is otherwise idle. Swap space can be divided among several disks. This can sometimes improve performance, depending on the relative speeds of the disks and the access patterns of the disks. You might want to experiment with a few schemes, but be aware that doing the experiments properly is quite difcult. You should not believe claims that any one scheme is superior to any other, since it wont always be true.
67
Disk buffering works for writes as well. On the one hand, data that is written is often soon read again (e.g., a source code le is saved to a le, then read by the compiler), so putting data that is written in the cache is a good idea. On the other hand, by only putting the data into the cache, not writing it to disk at once, the program that writes runs quicker. The writes can then be done in the background, without slowing down the other programs. Most operating systems have buffer caches (although they might be called something else), but not all of them work according to the above principles. Some are write-through: the data is written to disk at once (it is kept in the cache as well, of course). The cache is called write-back if the writes are done at a later time. Write-back is more efcient than write-through, but also a bit more prone to errors: if the machine crashes, or the power is cut at a bad moment, or the oppy is removed from the disk drive before the data in the cache waiting to be written gets written, the changes in the cache are usually lost. This might even mean that the lesystem (if there is one) is not in full working order, perhaps because the unwritten data held important changes to the bookkeeping information. Because of this, you should never turn off the power without using a proper shutdown procedure (see Chapter 6), or remove a oppy from the disk drive until it has been unmounted (if it was mounted) or after whatever program is using it has signaled that it is nished and the oppy drive light doesnt shine anymore. The sync command ushes the buffer, i.e., forces all unwritten data to be written to disk, and can be used when one wants to be sure that everything is safely written. In traditional UNIX systems, there is a program called update running in the background which does a sync every 30 seconds, so it is usually not necessary to use sync. Linux has an additional daemon, bdush, which does a more imperfect sync more frequently to avoid the sudden freeze due to heavy disk I/O that sync sometimes causes. Under Linux, bdush is started by update. There is usually no reason to worry about it, but if bdush happens to die for some reason, the kernel will warn about this, and you should start it by hand (/sbin/update). The cache does not actually buffer les, but blocks, which are the smallest units of disk I/O (under Linux, they are usually 1 kB). This way, also directories, super blocks, other lesystem bookkeeping data, and non-lesystem disks are cached. The effectiveness of a cache is primarily decided by its size. A small cache is next to useless: it will hold so little data that all cached data is ushed from the cache before it is reused. The critical size depends on how much data is read and written, and how often the same data is accessed. The only way to know is to experiment. If the cache is of a xed size, it is not very good to have it too big, either, because that might make the free memory too small and cause swapping (which is also slow). To make the most efcient use of real memory, Linux automatically uses all free RAM for buffer cache, but also automatically makes the cache smaller when programs need more memory. Under Linux, you do not need to do anything to make use of the cache, it happens completely automatically. Except for following the proper procedures for shutdown and removing oppies, you do
68
Notes
1. Thus quite needlessly annoying a number of computer scientists something horrible. 2. A gigabyte here, a gigabyte there, pretty soon we start talking about real memory. 3. Except a RAM disk, for obvious reasons.
69
This section explains what goes on when a Linux system is brought up and taken down, and how it should be done properly. If proper procedures are not followed, les might be corrupted or lost.
70
71
oppy, then, when the system is otherwise installed and working well, will install LILO and start booting from the hard disk. After the Linux kernel has been read into the memory, by whatever means, and is started for real, roughly the following things happen:
The Linux kernel is installed compressed, so it will rst uncompress itself. The beginning of the kernel image contains a small program that does this. If you have a super-VGA card that Linux recognizes and that has some special text modes (such as 100 columns by 40 rows), Linux asks you which mode you want to use. During the kernel compilation, it is possible to preset a video mode, so that this is never asked. This can also be done with LILO or rdev. After this, the kernel checks what other hardware there is (hard disks, oppies, network adapters, etc), and congures some of its device drivers appropriately; while it does this, it outputs messages about its ndings. For example, when I boot, I it looks like this:
LILO boot: Loading linux. Console: colour EGA+ 80x25, 8 virtual consoles Serial driver version 3.94 with no serial options enabled tty00 at 0x03f8 (irq = 4) is a 16450 tty01 at 0x02f8 (irq = 3) is a 16450 lp_init: lp1 exists (0), using polling driver Memory: 7332k/8192k available (300k kernel code, 384k reserved, 176k data) Floppy drive(s): fd0 is 1.44M, fd1 is 1.2M Loopback device init Warning WD8013 board not found at i/o = 280. Math coprocessor using irq13 error reporting. Partition check: hda: hda1 hda2 hda3 VFS: Mounted root (ext filesystem). Linux version 0.99.pl9-1 (root@haven) 05/01/93 14:12:20
The exact texts are different on different systems, depending on the hardware, the version of Linux being used, and how it has been congured.
Then the kernel will try to mount the root lesystem. The place is congurable at compilation time, or any time with rdev or LILO. The lesystem type is detected automatically. If the mounting of the root lesystem fails, for example because you didnt remember to include the corresponding lesystem driver in the kernel, the kernel panics and halts the system (there isnt much it can do, anyway). The root lesystem is usually mounted read-only (this can be set in the same way as the place). This makes it possible to check the lesystem while it is mounted; it is not a good idea to check a lesystem that is mounted read-write.
72
After this, the kernel starts the program init (located in /sbin/init) in the background (this will always become process number 1). init does various startup chores. The exact things it does depends on how it is congured; see Chapter 7 for more information (not yet written). It will at least start some essential background daemons. init then switches to multi-user mode, and starts a getty for virtual consoles and serial lines. getty is the program which lets people log in via virtual consoles and serial terminals. init may also start some other programs, depending on how it is congured. After this, the boot is complete, and the system is up and running normally.
System should
This will warn everybody that the system will shut down in ten minutes, and that theyd better get lost or lose data. The warning is printed to every terminal on which someone is logged in, including all xterms:
73
2 01:03:25 1995...
We will install a new disk. System should be back on-line in three hours. The system is going DOWN for system halt in 10 minutes !!
The warning is automatically repeated a few times before the boot, with shorter and shorter intervals as the time runs out. When the real shutting down starts after any delays, all lesystems (except the root one) are unmounted, user processes (if anybody is still logged in) are killed, daemons are shut down, all lesystem are unmounted, and generally everything settles down. When that is done, init prints out a message that you can power down the machine. Then, and only then, should you move your ngers towards the power switch. Sometimes, although rarely on any good system, it is impossible to shut down properly. For instance, if the kernel panics and crashes and burns and generally misbehaves, it might be completely impossible to give any new commands, hence shutting down properly is somewhat difcult, and just about everything you can do is hope that nothing has been too severely damaged and turn off the power. If the troubles are a bit less severe (say, somebody hit your keyboard with an axe), and the kernel and the update program still run normally, it is probably a good idea to wait a couple of minutes to give update a chance to ush the buffer cache, and only cut the power after that. Some people like to shut down using the command sync 3 three times, waiting for the disk I/O to stop, then turn off the power. If there are no running programs, this is about equivalent to using shutdown. However, it does not unmount any lesystems and this can lead to problems with the ext2fs clean lesystem ag. The triple-sync method is not recommended . (In case youre wondering: the reason for three syncs is that in the early days of UNIX, when the commands were typed separately, that usually gave sufcient time for most disk I/O to be nished.)
Rebooting
Rebooting means booting the system again. This can be accomplished by rst shutting it down completely, turning power off, and then turning it back on. A simpler way is to ask shutdown to reboot the system, instead of merely halting it. This is accomplished by using the -r option to shutdown, for example, by giving the command shutdown -r now. Most Linux systems run shutdown -r now when ctrl-alt-del is pressed on the keyboard. This reboots the system. The action on ctrl-alt-del is congurable, however, and it might be better to allow for some delay before the reboot on a multiuser machine. Systems that are physically accessible to anyone might even be congured to do nothing when ctrl-alt-del is pressed.
74
Notes
1. On early computers, it wasnt enough to merely turn on the computer, you had to manually load the operating system as well. These new-fangled thing-a-ma-jigs do it all by themselves. 2. This is called the power on self test, or POST for short. 3. sync ushes the buffer cache.
75
Chapter 7. init
Uuno on numero yksi (Slogan for a series of Finnish movies.)
This chapter describes the init process, which is the rst user level process started by the kernel. init has many important duties, such as starting getty (so that users can log in), implementing run levels, and taking care of orphaned processes. This chapter explains how init is congured and how you can make use of the different run levels.
76
Chapter 7. init
The elds are described below. In addition, /etc/inittab can contain empty lines, and lines that begin with a number sign (#); these are both ignored.
id This identies the line in the le. For getty lines, it species the terminal it runs on (the characters after /dev/tty in the device le name). For other lines, it doesnt matter (except for length restrictions), but it should be unique.
runlevels The run levels the line should be considered for. The run levels are given as single digits, without delimiters. (Run levels are described in the next section.)
action What action should be taken by the line, e.g., respawn to run the command in the next eld again, when it exits, or once to run it just once.
To start a getty on the rst virtual terminal (/dev/tty1), in all the normal multi-user run levels (2-5), one would write the following line:
1:2345:respawn:/sbin/getty 9600 tty1
77
Chapter 7. init
The rst eld says that this is the line for /dev/tty1. The second eld says that it applies to run levels 2, 3, 4, and 5. The third eld means that the command should be run again, after it exits (so that one can log in, log out, and then log in again). The last eld is the command that runs getty on the rst virtual terminal. 3 If you wanted to add terminals or dial-in modem lines to a system, youd add more lines to
/etc/inittab, one for each terminal or dial-in line. For more details, see the manual pages init, inittab, and getty.
If a command fails when it starts, and init is congured to restart it, it will use a lot of system resources: init starts it, it fails, init starts it, it fails, init starts it, it fails, and so on, ad innitum. To prevent this, init will keep track of how often it restarts a command, and if the frequency grows to high, it will delay for ve minutes before restarting again.
Run levels
A run level is a state of init and the whole system that denes what system services are operating. Run levels are identied by numbers, see Table 7-1. There is no consensus of how to use the user dened run levels (2 through 5). Some system administrators use run levels to dene which subsystems are working, e.g., whether X is running, whether the network is operational, and so on. Others have all subsystems always running or start and stop them individually, without changing run levels, since run levels are too coarse for controlling their systems. You need to decide for yourself, but it might be easiest to follow the way your Linux distribution does things. Table 7-1. Run level numbers 0 1 2-5 6 Halt the system. Single-user mode (for special administration). Normal operation (user dened). Reboot.
The rst eld is an arbitrary label, the second one means that this applies for run level 2. The third eld means that init should run the command in the fourth eld once, when the run level is entered, and that init should wait for it to complete. The /etc/init.d/rc command runs whatever commands are necessary to start and stop services to enter run level 2.
78
Chapter 7. init
The command in the fourth eld does all the hard work of setting up a run level. It starts services that arent already running, and stops services that shouldnt be running in the new run level any more. Exactly what the command is, and how run levels are congured, depends on the Linux distribution. When init starts, it looks for a line in /etc/inittab that species the default run level:
id:2:initdefault:
You can ask init to go to a non-default run level at startup by giving the kernel a command line argument of single or emergency. Kernel command line arguments can be given via LILO, for example. This allows you to choose the single user mode (run level 1). While the system is running, the telinit command can change the run level. When the run level is changed, init runs the relevant command from /etc/inittab.
powerwait
Allows init to shut the system down, when the power fails. This assumes the use of a UPS, and software that watches the UPS and informs init that the power is off.
ctrlaltdel
Allows init to reboot the system, when the user presses ctrl-alt-del on the console keyboard. Note that the system administrator can congure the reaction to ctrl-alt-del to be something else instead, e.g., to be ignored, if the system is in a public location. (Or to start nethack.)
sysinit
Command to be run when the system is booted. This command usually cleans up /tmp, for example.
The list above is not exhaustive. See your inittab manual page for all possibilities, and for details on how to use the above ones.
79
Chapter 7. init
Notes
1. init itself is not allowed to die. You cant kill init even with SIGKILL. 2. Using the command kill -HUP 1 as root, for example 3. Different versions of getty are run differently. Consult your manual page, and make sure it is the correct manual page. 4. It probably shouldnt be used for playing nethack.
80
This section describes what happens when a user logs in or out. The various interactions of background processes, log les, conguration les, and so on are described in some detail.
81
Figure 8-1. Logins via terminals: the interaction of init, getty, login, and the shell.
START
login: exit
no
sh: exit
82
Note that the only new process is the one created by init (using the fork system call); getty and login only replace the program running in the process (using the exec system call). A separate program, for noticing the user, is needed for serial lines, since it can be (and traditionally was) complicated to notice when a terminal becomes active. getty also adapts to the speed and other settings of the connection, which is important especially for dial-in connections, where these parameters may change from call to call. There are several versions of getty and init in use, all with their good and bad points. It is a good idea to learn about the versions on your system, and also about the other versions (you could use the Linux Software Map to search them). If you dont have dial-ins, you probably dont have to worry about getty, but init is still important.
83
provide the service. This single listener is called inetd; see the Linux Network Administrators Guide for more information.
X and xdm
XXX X implements logins via xdm; also: xterm -ls
Access control
84
The user database is traditionally contained in the /etc/passwd le. Some systems use shadow passwords, and have moved the passwords to /etc/shadow. Sites with many computers that share the accounts use NIS or some other method to store the user database; they might also automatically copy the database from one central location to all other computers. The user database contains not only the passwords, but also some additional information about the users, such as their real names, home directories, and login shells. This other information needs to be public, so that anyone can read it. Therefore the password is stored encrypted. This does have the drawback that anyone with access to the encrypted password can use various cryptographical methods to guess it, without trying to actually log into the computer. Shadow passwords try to avoid this by moving the password into another le, which only root can read (the password is still stored encrypted). However, installing shadow passwords later onto a system that did not support them can be difcult. With or without passwords, it is important to make sure that all passwords in a system are good, i.e., not easily guessable. The crack program can be used to crack passwords; any password it can nd is by denition not a good one. While crack can be run by intruders, it can also be run by the system adminstrator to avoid bad passwords. Good passwords can also be enforced by the passwd program; this is in fact more effective in CPU cycles, since cracking passwords requires quite a lot of computation. The user group database is kept in /etc/group; for systems with shadow passwords, there can be a /etc/shadow.group . root usually cant login via most terminals or the network, only via terminals listed in the /etc/securetty le. This makes it necessary to get physical access to one of these terminals. It is, however, possible to log in via any terminal as any other user, and use the su command to become root.
Shell startup
When an interactive login shell starts, it automatically executes one or more pre-dened les. Different shells execute different les; see the documentation of each shell for further information. Most shells rst run some global le, for example, the Bourne shell (/bin/sh) and its derivatives execute /etc/profile; in addition, they execute .profile in the users home directory. /etc/profile allows the system administrator to have set up a common user environment, especially by setting the PATH to include local command directories in addition to the normal ones. On the other hand, .profile allows the user to customize the environment to his own tastes by overriding, if necessary, the default environment.
Notes
1. Good Linux distributions do this out of the box.
85
This chapter explains how to create new user accounts, how to modify the properties of those accounts, and how to remove the accounts. Different Linux systems have different tools for doing this.
Whats an account?
When a computer is used by many people it is usually necessary to differentiate between the users, for example, so that their private les can be kept private. This is important even if the computer can only be used by a single person at a time, as with most microcomputers. 1 Thus, each user is given a unique username, and that name is used to log in. Theres more to a user than just a name, however. An account is all the les, resources, and information belonging to one user. The term hints at banks, and in a commercial system each account usually has some money attached to it, and that money vanishes at different speeds depending on how much the user stresses the system. For example, disk space might have a price per megabyte and day, and processing time might have a price per second.
Creating a user
The Linux kernel itself treats users are mere numbers. Each user is identied by a unique integer, the user id or uid , because numbers are faster and easier for a computer to process than textual names. A separate database outside the kernel assigns a textual name, the username, to each user id. The database contains additional information as well. To create a user, you need to add information about the user to the user database, and create a home directory for him. It may also be necessary to educate the user, and set up a suitable initial environment for him. Most Linux distributions come with a program for creating accounts. There are several such programs available. Two command line alternatives are adduser and useradd; there may be a GUI tool as well. Whatever the program, the result is that there is little if any manual work to be done. Even if the details are many and intricate, these programs make everything seem trivial. However, the section called Creating a user by hand describes how to do it by hand.
86
Username. Password, in an encrypted form. Numeric user id. Numeric group id. Full name or other description of account. Home directory. Login shell (program to run at login).
The format is explained in more detail on the passwd manual page. Any user on the system may read the password le, so that they can, for example, learn the name of another user. This means that the password (the second eld) is also available to everyone. The password le encrypts the password, so in theory there is no problem. However, the encryption is breakable, especially if the password is weak (e.g., it is short or it can be found in a dictionary). Therefore it is not a good idea to have the password in the password le. Many Linux systems have shadow passwords. This is an alternative way of storing the password: the encrypted password is stored in a separate le, /etc/shadow, which only root can read. The /etc/passwd le only contains a special marker in the second eld. Any program that needs to verify a user is setuid, and can therefore access the shadow password le. Normal programs, which only use the other elds in the password le, cant get at the password. 2
87
Edit /etc/passwd with vipw and add a new line for the new account. Be careful with the syntax. Do not edit directly with an editor! vipw locks the le, so that other commands wont try to update it at the same time. You should make the password eld be *, so that it is impossible to log in. Similarly, edit /etc/group with vigr, if you need to create a new group as well. Create the home directory of the user with mkdir. Copy the les from /etc/skel to the new home directory. Fix ownerships and permissions with chown and chmod. The -R option is most useful. The correct permissions vary a little from one site to another, but usually the following commands do the right thing:
cd /home/newusername chown -R username.group . chmod -R go=u,go-w . chmod go= .
After you set the password in the last step, the account will work. You shouldnt set it until everything else has been done, otherwise the user may inadvertently log in while youre still copying the les. It is sometimes necessary to create dummy accounts 3 that are not used by people. For example, to set up an anonymous FTP server (so that anyone can download les from it, without having to get an account
88
rst), you need to create an account called ftp. In such cases, it is usually not necessary to set the password (last step above). Indeed, it is better not to, so that no-one can use the account, unless they rst become root, since root can become any user.
The super-user may use these commands to change the properties of any account. Normal users can only change the properties of their own account. It may sometimes be necessary to disable these commands (with chmod) for normal users, for example in an environment with many novice users. Other tasks need to be done by hand. For example, to change the username, you need to edit /etc/passwd directly (with vipw, remember). Likewise, to add or remove the user to more groups, you need to edit /etc/group (with vigr). Such tasks tend to be rare, however, and should be done with caution: for example, if you change the username, e-mail will no longer reach the user, unless you also create a mail alias. 4
Removing a user
89
To remove a user, you rst remove all his les, mailboxes, mail aliases, print jobs, cron and at jobs, and all other references to the user. Then you remove the relevant lines from /etc/passwd and /etc/group (remember to remove the username from all groups its been added to). It may be a good idea to rst disable the account (see below), before you start removing stuff, to prevent the user from using the account while it is being removed. Remember that users may have les outside their home directory. The nd command can nd them:
find / -user username
However, note that the above command will take a long time, if you have large disks. If you mount network disks, you need to be careful so that you wont trash the network or the server. Some Linux distributions come with special commands to do this; look for deluser or userdel. However, it is easy to do it by hand as well, and the commands might not do everything.
The rst two characters (#!) tell the kernel that the rest of the line is a command that needs to be run to interpret this le. The tail command in this case outputs everything except the rst line to the standard output. If user billg is suspected of a security breach, the system administrator would do something like this:
# chsh -s /usr/local/lib/no-login/security billg # su - tester
90
Please call 555-1234 and wait for the men in black to arrive.
#
The purpose of the su is to test that the change worked, of course. Tail scripts should be kept in a separate directory, so that their names dont interfere with normal user commands.
Notes
1. It might be quite embarrassing if my sister could read my love letters. 2. Yes, this means that the password le has all the information about a user except his password. The wonder of development. 3. Surreal users? 4. The users name might change due to marriage, for example, and he might want to have his username reect his new name. 5. But they can be so fun, if youre a BOFH.
91
This chapter explains about why, how, and when to make backups, and how to restore things from backups.
92
93
dump is different in that it reads the lesystem directly and not via the lesystem. It is also written specically for backups; tar and cpio are really for archiving les, although they work for backups as well. Reading the lesystem directly has some advantages. It makes it possible to back les up without affecting their time stamps; for tar and cpio, you would have to mount the lesystem read-only rst. Directly reading the lesystem is also more effective, if everything needs to be backed up, since it can be done with much less disk head movement. The major disadvantage is that it makes the backup program specic to one lesystem type; the Linux dump program understands the ext2 lesystem only. dump also directly supports backup levels (which well be discussing below); with tar and cpio this has to be implemented with other tools. A comparison of the third party backup tools is beyond the scope of this book. The Linux Software Map lists many of the freeware ones.
Simple backups
A simple backup scheme is to back up everything once, then back up everything that has been modied since the previous backup. The rst backup is called a full backup, the subsequent ones are incremental backups. A full backup is often more laborius than incremental ones, since there is more data to write to the tape and a full backup might not t onto one tape (or oppy). Restoring from incremental backups can be many times more work than from a full one. Restoration can be optimized so that you always back up everything since the previous full backup; this way, backups are a bit more work, but there should never be a need to restore more than a full backup and an incremental backup. If you want to make backups every day and have six tapes, you could use tape 1 for the rst full backup (say, on a Friday), and tapes 2 to 5 for the incremental backups (Monday through Thursday). Then you make a new full backup on tape 6 (second Friday), and start doing incremental ones with tapes 2 to 5 again. You dont want to overwrite tape 1 until youve got a new full backup, lest something happens while youre making the full backup. After youve made a full backup to tape 6, you want to keep tape 1 somewhere else, so that when your other backup tapes are destroyed in the re, you still have at least something left. When you need to make the next full backup, you fetch tape 1 and leave tape 6 in its place. If you have more than six tapes, you can use the extra ones for full backups. Each time you make a full backup, you use the oldest tape. This way you can have full backups from several previous weeks, which is good if you want to nd an old, now deleted le, or an old version of a le.
94
The example above uses the GNU version of tar and its long option names. The traditional version of tar only understands single character options. The GNU version can also handle backups that dont t on one tape or oppy, and also very long paths; not all traditional versions can do these things. (Linux only uses GNU tar.) If your backup doesnt t on one tape, you need to use the -multi-volume (-M) option:
# tar -cMf /dev/fd0H1440 /usr/src tar: Removing leading / from absolute path names in the archive Prepare volume #2 for /dev/fd0H1440 and hit return: #
Note that you should format the oppies before you begin the backup, or else use another window or virtual terminal and do it when tar asks for a new oppy. After youve made a backup, you should check that it is OK, using the -compare (-d) option:
# tar -compare -verbose -f /dev/ftape usr/src/ usr/src/linux usr/src/linux-1.2.10-includes/ .... #
Failing to check a backup means that you will not notice that your backups arent working until after youve lost the original data. An incremental backup can be done with tar using the -newer (-N) option:
# tar -create -newer 8 Sep 1995 -file /dev/ftape /usr/src -verbose tar: Removing leading / from absolute path names in the archive usr/src/ usr/src/linux-1.2.10-includes/ usr/src/linux-1.2.10-includes/include/ usr/src/linux-1.2.10-includes/include/linux/ usr/src/linux-1.2.10-includes/include/linux/modules/ usr/src/linux-1.2.10-includes/include/asm-generic/ usr/src/linux-1.2.10-includes/include/asm-i386/ usr/src/linux-1.2.10-includes/include/asm-mips/ usr/src/linux-1.2.10-includes/include/asm-alpha/
95
Unfortunately, tar cant notice when a les inode information has changed, for example, that its permission bits have been changed, or when its name has been changed. This can be worked around using nd and comparing current lesystem state with lists of les that have been previously backed up. Scripts and programs for doing this can be found on Linux ftp sites.
You also extract only specic les or directories (which includes all their les and subdirectories) by naming on the command line:
# tar xpvf /dev/fd0H1440 usr/src/linux-1.2.10-includes/include/linux/hdreg.h usr/src/linux-1.2.10-includes/include/linux/hdreg.h #
Use the -list (-t) option, if you just want to see what les are on a backup volume:
# tar -list -file /dev/fd0H1440 usr/src/ usr/src/linux usr/src/linux-1.2.10-includes/ usr/src/linux-1.2.10-includes/include/ usr/src/linux-1.2.10-includes/include/linux/ usr/src/linux-1.2.10-includes/include/linux/hdreg.h usr/src/linux-1.2.10-includes/include/linux/kernel.h ...
96
Note that tar always reads the backup volume sequentially, so for large volumes it is rather slow. It is not possible, however, to use random access database techniques when using a tape drive or some other sequential medium. tar doesnt handle deleted les properly. If you need to restore a lesystem from a full and an incremental backup, and you have deleted a le between the two backups, it will exist again after you have done the restore. This can be a big problem, if the le has sensitive data that should no longer be available.
Multilevel backups
The simple backup method outlined in the previous section is often quite adequate for personal use or small sites. For more heavy duty use, multilevel backups are more appropriate. The simple method has two backup levels: full and incremental backups. This can be generalized to any number of levels. A full backup would be level 0, and the different levels of incremental backups levels 1, 2, 3, etc. At each incremental backup level you back up everything that has changed since the previous backup at the same or a previous level. The purpose for doing this is that it allows a longer backup history cheaply. In the example in the previous section, the backup history went back to the previous full backup. This could be extended by having more tapes, but only a week per new tape, which might be too expensive. A longer backup history is useful, since deleted or corrupted les are often not noticed for a long time. Even a version of a le that is not very up to date is better than no le at all. With multiple levels the backup history can be extended more cheaply. For example, if we buy ten tapes, we could use tapes 1 and 2 for monthly backups (rst Friday each month), tapes 3 to 6 for weekly backups (other Fridays; note that there can be ve Fridays in one month, so we need four more tapes), and tapes 7 to 10 for daily backups (Monday to Thursday). With only four more tapes, weve been able to extend the backup history from two weeks (after all daily tapes have been used) to two months. It is true that we cant restore every version of each le during those two months, but what we can restore is often good enough. Figure 10-1 shows which backup level is used each day, and which backups can be restored from at the end of the month.
97
Backup levels can also be used to keep lesystem restoration time to a minimum. If you have many incremental backups with monotonously growing level numbers, you need to restore all of them if you need to rebuild the whole lesystem. Instead you can use level numbers that arent monotonous, and keep down the number of backups to restore. To minimize the number of tapes needed to restore, you could use a smaller level for each incremental tape. However, then the time to make the backups increases (each backup copies everything since the previous full backup). A better scheme is suggested by the dump manual page and described by the table XX (efcient-backup-levels). Use the following succession of backup levels: 3, 2, 5, 4, 7, 6, 9, 8, 9, etc. This keeps both the backup and restore times low. The most you have to backup is two days worth of work. The number of tapes for a restore depends on how long you keep between full backups, but it is less than in the simple schemes. Table 10-1. Efcient backup scheme using many backup levels Tape 1 2 3 4 5 6 7 8 9 10 11 Level 0 3 2 5 4 7 6 9 8 9 9 Backup (days) n/a 1 2 1 2 1 2 1 2 1 1 Restore tapes 1 1, 2 1, 3 1, 2, 4 1, 2, 5 1, 2, 5, 6 1, 2, 5, 7 1, 2, 5, 7, 8 1, 2, 5, 7, 9 1, 2, 5, 7, 9, 10 1, 2, 5, 7, 9, 10, 11
98
Tape ...
Level 9
Backup (days) 1
A fancy scheme can reduce the amount of labor needed, but it does mean there are more things to keep track of. You must decide if it is worth it. dump has built-in support for backup levels. For tar and cpio it must be implemented with shell scripts.
What to back up
You want to back up as much as possible. The major exception is software that can be easily reinstalled, 4 but even they may have conguration les that it is important to back up, lest you need to do all the work to congure them all over again. Another major exception is the /proc lesystem; since that only contains data that the kernel always generates automatically, it is never a good idea to back it up. Expecially the /proc/kcore le is unnecessary, since it is just an image of your current physical memory; its pretty large as well. Gray areas include the news spool, log les, and many other things in /var. You must decide what you consider important. The obvious things to back up are user les (/home) and system conguration les (/etc, but possibly other things scattered all over the lesystem).
Compressed backups
Backups take a lot of space, which can cost quite a lot of money. To reduce the space needed, the backups can be compressed. There are several ways of doing this. Some programs have support for for compression built in; for example, the -gzip (-z) option for GNU tar pipes the whole backup through the gzip compression program, before writing it to the backup medium. Unfortunately, compressed backups can cause trouble. Due to the nature of how compression works, if a single bit is wrong, all the rest of the compressed data will be unusable. Some backup programs have some built in error correction, but no method can handle a large number of errors. This means that if the backup is compressed the way GNU tar does it, with the whole output compressed as a unit, a single error makes all the rest of the backup lost. Backups must be reliable, and this method of compression is not a good idea. An alternative way is to compress each le separately. This still means that the one le is lost, but all other les are unharmed. The lost le would have been corrupted anyway, so this situation is not much worse than not using compression at all. The ao program (a variant of cpio) can do this.
99
Compression takes some time, which may make the backup program unable to write data fast enough for a tape drive. 5 This can be avoided by buffering the output (either internally, if the backup program if smart enough, or by using another program), but even that might not work well enough. This should only be a problem on slow computers.
Notes
1. The fth reason is something else. 2. Dont laugh. This has happened to several people. 3. Been there, done that... 4. You get to decide whats easy. Some people consider installing from dozens of oppies easy. 5. If a tape drive doesnt data fast enough, it has to stop; this makes backups even slower, and can be bad for the tape and the drive.
100
This chapter explains how a Linux system keeps time, and what you need to do to avoid causing trouble. Usually, you dont need to do anything about time, but it is good to understand it.
Time zones
Time measurement is based on mostly regular natural phenomena, such as alternating light and dark periods caused by the rotation of the planet. The total time taken by two successive periods is constant, but the lengths of the light and dark period vary. One simple constant is noon. Noon is the time of the day when the Sun is at its highest position. Since the Earth is round, 1 noon happens at different times in different places. This leads to the concept of local time. Humans measure time in many units, most of which are tied to natural phenomena like noon. As long as you stay in the same place, it doesnt matter that local times differ. As soon as you need to communicate with distant places, youll notice the need for a common time. In modern times, most of the places in the world communicate with most other places in the world, so a global standard for measuring time has been dened. This time is called universal time (UT or UTC, formerly known as Greenwich Mean Time or GMT, since it used to be local time in Greenwich, England). When people with different local times need to communicate, they can express times in universal time, so that there is no confusion about when things should happen. Each local time is called a time zone. While geography would allow all places that have noon at the same time have the same time zone, politics makes it difcult. For various reasons, many countries use daylight savings time, that is, they move their clocks to have more natural light while they work, and then move the clocks back during winter. Other countries do not do this. Those that do, do not agree when the clocks should be moved, and they change the rules from year to year. This makes time zone conversions denitely non-trivial. Time zones are best named by the location or by telling the difference between local and universal time. In the US and some other countries, the local time zones have a name and a three letter abbreviation. The abbreviations are not unique, however, and should not be used unless the country is also named. It is better to talk about the local time in, say, Helsinki, than about East European time, since not all countries in Eastern Europe follow the same rules. Linux has a time zone package that knows about all existing time zones, and that can easily be updated when the rules change. All the system administrator needs to do is to select the appropriate time zone. Also, each user can set his own time zone; this is important since many people work with computers in
101
different countries over the Internet. When the rules for daylight savings time change in your local time zone, make sure youll upgrade at least that part of your Linux system. Other than setting the system time zone and upgrading the time zone data les, there is little need to bother about time.
102
That time is Sunday, 14th of July, 1996, at about ten before ten at the evening, in the time zone called EET DST (which might be East European Daylight Savings Time). date can also show the univeral time:
$ date -u
See the date manual page for more details; the syntax is a bit arcane. Only root can set the time. While each user can have his own time zone, the clock is the same for everyone. date only shows or sets the software clock. The clock commands syncronizes the hardware and software clocks. It is used when the system boots, to read the hardware clock and set the software clock. If you need to set both clocks, you rst set the software clock with date, and then the hardware clock with clock -w. The -u option to clock tells it that the hardware clock is in universal time. You must use the -u option correctly. If you dont, your computer will be quite confused about what the time is. The clocks should be changed with care. Many parts of a Unix system require the clocks to work correctly. For example, the cron daemon runs commands periodically. If you change the clock, it can be confused of whether it needs to run the commands or not. On one early Unix system, someone set the clock twenty years into the future, and cron wanted to run all the periodic commands for twenty years all at once. Current versions of cron can handle this correctly, but you should still be careful. Big jumps or backward jumps are more dangeours than smaller or forward ones.
103
If you need to adjust the hardware clock, it is usually simplest to reboot, go into the BIOS setup screen, and do it from there. This avoids all trouble that changing system time might cause. If doing it via BIOS is not an option, set the new time with date and clock (in that order), but be prepared to reboot, if some part of the system starts acting funny. A networked computer (even if just over the modem) can check its own clock automatically, by comparing it to some other computers time. If the other computer is known to keep very accurate time, then both computers will keep accurate time. This can be done by using the rdate and netdate commands. Both check the time of a remote computer (netdate can handle several remote computers), and set the local computers time to that. By running one these commands regularly, your computer will keep as accurate time as the remote computer. XXX say something intelligent about NTP
Notes
1. According to recent research. 2. Beware of the time command, which does not show the current time.
104
Glossary (DRAFT)
The Librarian of the Unseen University had unilaterally decided to aid comprehension by producing an Orang-utan/Human Dictionary. Hed been working on it for three months. It wasnt easy. Hed got as far as Oook. (Terry Pratchett, Men At Arms)
This is a short list of word denitions for concepts relating to Linux and system administration. ambition The act of writing funny sentences in the hope of getting them into the Linux cookie le.
application program Software that does something useful. The results of using an application program is what the computer was bought for. See also system program, operating system.
daemon A process lurking in the background, usually unnoticed, until something triggers it into action. For example, the update daemon wakes up every thirty seconds or so to ush the buffer cache, and the sendmail daemon awakes whenever someone sends mail.
le system The methods and data structures that an operating system uses to keep track of les on a disk or partition; the way the les are organized on the disk. Also used about a partition or disk that is used to store the les or the type of the lesystem.
glossary A list of words and explanations of what they do. Not to be confused with a dictionary, which is also a list of words and explanations.
kernel
105
Glossary (DRAFT)
Part of an operating system that implements the interaction with hardware and the sharing of resources. See also system program.
operating system Software that shares a computer systems resources (processor, memory, disk space, network bandwidth, and so on) between users and the application programs they run. Controls access to the system to provide security. See also kernel, system program, application program.
system call The services provided by the kernel to application programs, and the way in which they are invoked. See section 2 of the manual pages.
system program Programs that implement high level functionality of an operating system, i.e., things that arent directly dependent on the hardware. May sometimes require special privileges to run (e.g., for delivering electronic mail), but often just commonly thought of as part of the system (e.g., a compiler). See also application program, kernel, operating system.
106