Linux Redirection & Pipes Features:: Linux Basic and Administration Commands
Linux Redirection & Pipes Features:: Linux Basic and Administration Commands
Features:
1. Ability to control input and output
Input redirection '<':
1. cat < 123.txt
Note: Use input redirection when program does NOT default to file as input
Pipes '|':
Features: Connects the output stream of one command to the input stream of a subsequent command
1. cat 123.txt | sort
2. cat 456.txt 123.txt | sort
3. cat 456.txt 123.txt | sort | grep 3
Command Chaining
Features:
1. Permits the execution of multiple commands in sequence
2. Also permits execution based on the success or failure of a previous command
1. cat 123.txt ; ls -l - this runs first command, then second command without regards for exit
status of the first command
2. cat 123.txt && ls -l - this runs second command, if first command is successful
3. cat 1234.txt && ls -l
Features:
1. Compression utilities (gzip, bzip2, zip)
2. File rollers (the ability to represent many files as one)
Gzip:
Includes:
1. gzip - compresses/decompresses files
2. gunzip - decompresses gzip files
Example:
1. compress '1million.txt' file using gzip
a. gzip -c 1million.txt > 1million.txt.gz
Bzip2:
1. bzip2 -c 1million.txt > 1million.txt.bz2
GREP
Features:
1. The ability to parse lines based on text and/or RegExes
2. Post-processor
3. Searches case-sensitively, by default
4. Searches for the text anywhere on the line
Note: Anchors are RegEx characters (meta-characters). They're used to match at the beginning and end
of lines
Note: Most, if not all, Linux programs log linearly, which means one line after another, from the earliest
to the current
Features:
1. Field/Column processor
2. Supports egrep-compatible (POSIX) RegExes
3. Can return full lines like grep
4. Awk runs 3 steps:
a. BEGIN - optional
b. Body, where the main action(s) take place
c. END - optional
5. Multiple body actions can be executed by separating them using semicolons. e.g. '{ print $1; print $2 }'
6. Awk, auto-loops through input stream, regardless of the source of the stream. e.g. STDIN, Pipe, File
Usage:
1. awk '/optional_match/ { action }' file_name | Pipe
2. awk '{ print $1 }' grep1.txt
Note: Use single quotes with awk, to avoid shell interpolation of awk's variables
4. awk '/linux/ { print } ' grep1.txt - this will print ALL lines containing 'linux'
5. awk '{ if ($2 ~ /Linux/) print}' grep1.txt
6. awk '{ if ($2 ~ /8/) print }' /var/log/messages - this will print the entire line for log items for the 8th
7. awk '{ print $3 }' /var/log/messages | awk -F: '{ print $1}'
Features:
1. Facilitates automated text editing
2. Supports RegExes (POSIX)
3. Like Awk, supports scripting using '-F' option
4. Supports input via: STDIN, pipe, file
Usage:
1. sed [options] 'instruction[s]' file[s]
2. sed -n '1p' grep1.txt - prints the first line of the file
3. sed -n '1,5p' grep1.txt - prints the first 5 lines of the file
4. sed -n '$p' grep1.txt - prints the last line of the file
5. sed -n '1,3!p' grep1.txt - prints ALL but lines 1-3
6. sed -n '/linux/p' grep1.txt - prints lines with 'linux'
7. sed -e '/^$/d' grep1.txt - deletes blank lines from the document
8. sed -e '/^$/d' grep1.txt > sed1.txt - deletes blank lines from the document 'grep1.txt' and creates
'sed1.txt'
9. sed -ne 's/search/replace/p' sed1.txt
10. sed -ne 's/linux/unix/p' sed1.txt
11. sed -i.bak -e 's/3/4' sed1.txt - this backs up the original file and creates a new 'sed1.txt' with the
modifications indicated in the command
Note: Generally, to create new files, use output redirection, instead of allowing sed to write to STDOUT
Perl
Features:
1. Parses text
2. Executes programs
3. CGI - Web forms, etc.
4. Supports RegExes (Perl and POSIX)
5. etc.
Example:
1. Print 'Hello World' to STDOUT
a. perl -c helloworld.pl - checks the syntax of the script
b. perl helloworld.pl - executes the script
c. chmod +x helloworld.pl && ./helloworld.pl
System Utilities
Features:
1. Process listing
2. Free/available memory
3. Disk utilization
1. ps - process status/listing
a. ps -ef or ps -aux
2. top - combines, ps, uptime, free and updates regularly
6. vmstat - reports on: processes, memory, paging, block I/O, traps, CPU activity
a. vmstat
b. vmstat -p /dev/hda1 - returns partitions stats for /dev/hda1 (/boot)
8. ls -ltr /proc
a. cat /proc/cpuinfo
User/Group Management
Features:
1. The ability to control users and groups
Primary tools:
1. useradd - used to add users and modify group membership
2. system-config-users
Example:
1. Create a user named 'student1' using 'useradd'
username:shadow_reference:uid:gid:Description(GECOS):$HOME:$SHELL
Note: /etc/passwd is a world-readable file
Note: /etc/shadow now stores passwords in encrypted form
Note: /etc/shadow is NOT world-readable
Fields in /etc/shadow:
student1:$1$XSFMv2ru$lfTACjN.XxaxbHA0EkB4U0:13891:0:99999:7:::
1. username:
2. encrypted_password:
3. Days_since_Unix_epoch_password_was_changed (01/01/1970)
4. Days before password may be changed
5. Days after which the password MUST be changed
6. Days before password is to expire that user is warned
7. Days after password expires, that account is disabled
8. Days since Unix epoch, that account is disabled
9. Reserved field (currently unused)
Groups:
1. groupadd - adds new group
2. groups - lists groups on the system: /etc/group
/etc/group - maintains group membership information
Example: Create a 'sales' group and add 'linuxusr' and 'student1' as members
1. groupadd sales
2. usermod -G sales linuxusr
3. usermod -G sales student1
Features:
1. The ability to restrict/control access to files
Example:
1. Manipulate file permissions using 'chmod'
a. chmod -x regextest.pl
Example:
Update 'regextest.pl' so that owner and group owner may modify the file
a. chmod 660 regextest.pl
SETUID:
Features:
1. ability to execute file as owner
chmod 4760 regextest.pl - this will ensure that the perl script always executes as the user 'linuxusr'
-rwsrw---- 1 linuxusr sales 787 Jan 13 16:08 regextest.pl
's' in the execute position means that the program will execute as that user
SETGID:
Features:
1. Ability to enforce permissions to a directory structure
mkdir /sales
chmod 2775 /sales
chgrp:
Permits updating of group permissions
Sticky Bit:
Features:
1. Ability to ensure that users cannot delete others' files in a directory
chmod 3777 /sales - ensures that /sales will not lose files from incorrect users
Example:
1. Set '/sales' using sticky bit and test
a. chmod 3777 /sales && ls -ld /sales OR chmod 777 /sales && chmod +t /sales
Symlinks
Features:
Soft Links:
1. ln -s source_file target
a. ln -s ./regextest.pl lastscript.pl
Note: With soft links, if you change the name or location of the source file, you will break ALL of the
symlinks (soft)
Hard Links:
Features:
1. The ability to reference the same inode/hard drive location from multiple places within the same file
system
a. ln source target
ln regextest.pl ./testhardregextest.pl - creates a hard link
Quotas
Features:
1. Limits disk usage (blocks or inodes)
2. Tied to file systems (set on a per file system basis)
3. Can be configured for users and groups
5. Check quotas
a. quota username
quota student1
6. Report on usage
a. repquota -a - this reports on usage
Note: The blocks are measured in 1K increments. i.e. 20000 blocks is roughly 20MB
Features:
1. Ability to provision extra storage on-the-fly
Steps:
1. Identify available storage
a. 'fdisk -l' - returns connected storage
Note: use 'partprobe partition (/dev/sdb1)' to force a write to a hard drive's partition table on a running
system
Note: 'fdisk' creates raw partitions
Steps:
1. Identify current swap space
a. swapon -s - enumerates partitions and/or files, which constitute swap storage
b. free -m
5. update /etc/fstab
a. /dev/sdb2 swap swap defaults 0 0
Example:
1. Improve system performance by distributing swapping to /dev/sdb2
a. swapon /dev/sdb2
b. swapoff /dev/sda6
c. disable /dev/sda6 via /etc/fstab
Features:
1. The ability to provision swap space based on a file, similar to pagefile.sys in Windows NT, etc., if you
have no available disk space to partition.
2. Doesn't waste partitions
Example:
1. Create 512MB swap file
a. dd if=/dev/zero of=/home1/swapfile1 bs=1024 count=524288
b. mkswap /home1/swapfile1 - overlays swap file system
c. swapon /home1/swapfile1 - makes swap space available to the kernel
2. Ensure that when the system reboots, the swap file is made available to the kernel
a. nano /etc/fstab - /home1/swapfile1 swap swap defaults 0 0
Features:
1. Ability to create volume sets and stripe sets
2. LVM masks the underlying physical technology (ATA,ATAPI,IDE,SCSI,SATA,PATA,etc.)
3. LVM represents storage using a hierarchy:
a. Volume groups
a1. Physical volumes (/dev/sda2, /dev/sdb2, etc.)
b. Logical Volumes
b1. File systems
3. LVM physical volumes can be of various sizes
4. Ability to resize volumes on the fly
Note: Volume groups join: physical volumes (PVs) and Logical Volumes (LVs)
Note: Be certain to update: /etc/fstab so that volumes are mounted when the system reboots
Note: You may resize file systems online if the following are met:
1. 2.6x kernel series
2. MUST be formatted with ext3
Note: Check disk utilization prior to shrinking to reduce the risk of losing data
RAID
Features:
1. The ability to increase availability and reliability of data
Example:
1. Create a RAID-1 Device (/dev/md0..n)
a. fdisk /dev/sdb - to create usable raw partitions
b. partprobe /dev/sdb - to force a kernel update of the partition layout of the disk: /dev/sdb
b. mdadm --create /dev/md0 --level=1 --raid-devices=2 /dev/sdb5 /dev/sdb6
c. cat /proc/mdstat - lists active RAID (md) information
d. mke2fs -j /dev/md0 - overlays a file system on the RAID device
e. mount /dev/md0 /raid1
f. update: /etc/fstab
Note: use 'mdadm --query /dev/md0' to get information about a RAID device
Note: You may create RAID volumes/devices on a single or on multiple disks
Ideally, your RAID volumes should span multiple physical disks to improve:
a. reliability
b. performance
c. availability
RPM
Features:
1. Provides package management
a. Query
b. Install
c. Uninstall
d. Upgrade
e. Verify
2. Auto-verifies packages using GPG, MD5, SHA1SUMs
3. Automatically reports on unresolved dependencies
'rpm'
Query:
1. rpm -qa - dumps all installed packages
2. rpm -qa | wc -l - this dumps all packages and provides a count
3. rpm -qa | grep -i nano
4. rpm -qi nano - dumps info. about the 'nano' package as it's recorded in the local RPM database
5. rpm -qf /usr/bin/nano - dumps package membership info. for the 'nano' file
6. rpm -qpi http://192.168.1.101/RH5/i386/Server/dhcp-3.0.5-7.el5.i386.rpm - dumps info. about the
uninstalled 'dhcp' package, which resides on the repository
7. rpm -ql package_name - returns all included files