Linux

User mgmt & File Permission

May 2020
 Content

● Linux user management

● User group management
● File permissions
 Types of users
There are 3 types of users in linux.
● Root- It is the main user account in linux system. It automatically
created during installation and has highest privileges in system. It can
do any administrative work and can access any services.

● Regular – This is a normal user account created by root user . Root

can create as many user with limited permissions.

● Service- This account is created by installation package when they

are installed.These account are used by services to run processes and
execute functions.
 User Mgmt
● Print user informantion

-$ who am I
-$ Who mom likes
-$ who

• Output
 User Mgmt
• Parameters of who:
Parameter Description

-a Print all information

-d Print dead process
-m Same as am i,mom likes
-q their user names
-u Get the list of users logged in
-r Get the current run level

• Output
 Create User
● To add new user
○ $ sudo adduser veer
● To log in new user account
○ $ su -l veer
● To exit user use exit command or ctrl+D
 Create user group
● How to see user group name？

○ Method 1: Use groups Command

○ $ groups Aney

○ Method 2 ：View /etc/group File

○ $ cat /etc/group | sort

• Output
 Add user to root group
● Check user is root user or not.
○ $ su -l jack
○ $ sudo ls
○ $ exit

● Add user to the root group

○ $ sudo groups jack
○ $ sudo usermod -G sudo jack
○ $ sudo groups jack
 Delete a user

● Delete a user
$ sudo deluser jack --remove-home

• Output
 File permissions

Every UNIX file has the following attributes:

• Owner Permission : determines what actions the owner(user) of the file

determine

• Group Permission : determines what actions a user ,who is member of

the group that a file belongs to can perform on the file

• Other Permission : determines what actions the other users can

perform on the file .
 File Access mode

• Read permission (r)

• Write permission (w)

• Execute permission (x)

 File Access mode
Each file has three groups of permissions (user, group, others)
corresponding to a "rwx" triplet.
 File permissions
To view the file permissions of files in system: $ ls –l
 File permissions

To view the file permissions and details of the file: $ls –l

To view the file size: $ls –lh

To display all files except the . (current directory) and .. (parent

directory): $ ls –A

To view the full properties of a directory: $ ls -dl <directory>

 Change File permissions

To change the owner of the file:

$ touch test
$ chown new_user test

To change the group of the file: $ chgrp new_group test

 Change File permissions

To Change the file permission:

Method 1：Binary number representation

$chmod 700 filename

Method 2：Addition and subtraction assignments

To achieve the same result as by method 1:

$chmod go-rwx filename

 Change File permissions
Method 1：Binary number representation
$chmod 754 filename
 Change File permissions
Method 1：Binary number representation
 Change File permissions

Method 2：Addition and subtraction assignments

To achieve the same result as by method 1:

$chmod u=rwx ,g-w,o-rwx filename

 Assignment

Introduction:

Our Team has one server and each team member has an account
to be used for daily routine job. Today we have two new
employees joining us and we need you to help them create their
accounts.
 Assignment
Please finish these tasks:

Account 1
Username：Tom
Home directory：/home/Tom
User Tom uses zsh by default.
User Tom belongs to Rapidcode user group as well as dev user group.

Account 2
Username：Jerry
Home directory：/home/Jerry
User Jerry uses bash by default.
User Jerry belongs to Rapidcode user group as well as test user group
 Linux
Directory, File operation and Environment variables
May 2020
 Directory structure

• In Microsoft Windows, files are stored in folders on different data drives like C: D:
E:

• But, in Linux, files are ordered in a tree structure starting with the root
directory.
 Tree like Directory structure

Linux has a tree-like directory structure to build the entire system.

Directory structure in Linux
Listing the ‘/ ‘ Directory structure
 Tree structure in Linux
You can also execute the following command to view the entire directory structure:
$tree
 Installing Tree

If "command not found“ pops up, you should install it first:

$ sudo apt-get update

$ sudo apt-get install tree

 Understanding Path

What "Path" is:

The path is the route to enter a specific directory or want to fetch a file, you can
use path to find the directory or the file.

Command to switch directories:

$ cd path (cd: change directory)
 Absolute Path
A pathname is Absolute, if it is described in relation to root, thus it always start with
a/

$ /etc/passwd
$ users/Rapidcode/notes

Use pwd command to get the current path (absolute path).

$ pwd
 Relative Path
Uses either the current or parent directory as reference and specifies the path
relative to it.

In Linux, . represents the current directory and .. represents the upper directory,
also called parent directory.

$ cd ..
To go to your home directory : $ cd ~
 Directory commands

To Create a New Directory:

$ mkdir dir_name

To Create a parent directory if it doesn’t exist :

$ mkdir –p dname
$ mkdir -p father/son/grandson

To remove a directory:
$rmdir dir_name
$rmdir –f dir_name
 Directory commands

To copy a directory: $ cp source_dir dest_dir

This will show the following error:

Then use this : $ cp -r rapidcode company

 File commands
To Create a New File: $ touch fname

To remove a file : $rm fname , $rm –f fname

To Edit files :$ vi fname

To start the Linux internal vim tutorial, Enter the following command to start it:
$ vimtutor

To copy a File: $ cp source_file dest_file

$ cp test father/son/grandson

To move a File: $ mv source_file dest_file

$ mv test father/son/grandson
 File commands

To view file content: $cat fname

To view file content in reverse order: $tac fname

To Use head and tail to View a File: $ head –n fname, $tail –n fname
 File permissions
To view the file permissions of files in system: $ ls –l
 Listing Files

To view the file permissions and details of the file: $ls –l

To view the file size: $ls –lh

To display all files except the . (current directory) and .. (parent

directory): $ ls –A

To view the full properties of a directory: $ ls -dl <directory>

 Understanding the Variable
To create a variable in the shell and read the value of that variable

Use the command declare to create a variable named tmp: $ declare tmp

In fact, you don't need to declare a variable at first, you can create it when you want to
use it.

Use = to assign the variable tmp to engineer

$ tmp=engineer

Use echo and $ to read the value of a variable（$ is used to refer to the value of a
variable, don't forget to enter it.）:
$ echo $tmp
 Types of Variable

Three variable types:

∙ Private custom variables of current shell process, such as tmp we've created
above which is only valid in the current shell;
∙ Built-in variable of shell;
∙ Environment variables derived from custom variables.
 Environment Variable

Environment variables or ENVs basically define behavior of the environment.

They can affect the processes ongoing or the programs that are executed in the
environment.

An environment variable in Linux can have global or local scope.

There are three commands related to the environment variables set, env, export. These
three commands are very similar. They are used to print environment variable
information. The range of variables involved varies though.
 Environment Variable
Command Description
It can display all the variables of the
current shell, including its built-in
set environment variables, user-defined
variables and exported environment
variables.
It can display the environment variables
associated with the current user and
env
allow the command to run in the
specified environment.
It displays variables that are exported
from the shell as environment variables
export
and it can also export custom variables
as environment variables.
 Files associated to Environment
Variable
Linux has two important files /etc/bashrc and /etc/profile. They store shell
variables and environment variables.

Note that there is a hidden file in every user directory .profile.

You can use ls -a to view this file:

.profile is valid only for the current user. Environment variables written
in /etc/profile are valid for all users.
 Environment Variable: PATH
Find the path and order for a command:
PATH saves the search paths of commands.

View the contents of the PATH environment variable:

$ echo $PATH
 Environment Variable: PATH

To add a path to the PATH Environment variables:

: is a delimiter in PATH.

We can add a path like this:

$PATH=$PATH:/home/rapidcode/mybin (Be sure to use an absolute path.)

It only works in the current shell.

 Environment variable

Modify the path as follows:

$ path=$PATH
$ echo $path
$ path=${path:/home/rapidcode/mybin}
$ path=${path:/mybin}

To Delete an Environment Variable:

$ unset temp
 Configuration File

In every user's home directory, there is a configuration script that will run
automatically to initialize the environment . zsh configuration file is .zshrc and Bash
configuration file is .bashrc.

We can use the following command to add content to .bashrc:

$ echo PATH=$PATH:/home/rapidcode/mybin >> .bashrc

 Modifying the variable
To modify and delete Existing variable:

Setting method Description

From front to back, delete the shortest string

${name#match string}
that matches the string

From front to back, delete the longest string

${name##match string}
that matches the string

From back to front, delete the shortest string

${name%match string}
that matches the string

From back to front, delete the longest string

${name%%match string}
that matches the string

Replace the first string that matches the old

${nameold stringnew string}
string with the new string

Replace all strings that match the old string

${nameold stringnew string}
with the new string
 Assignment

1. Create a directory, named homework.

2. In the directory homework, create files named 1.txt to 10.txt.
3. Delete the files with names in the range 1.txt to 5.txt.
4. Where are the log files for Linux saved?
 Linux
Find Operation , File Packaging and executing Help commands
May 2020
 find
• find
$ sudo find etc -name interfaces

Time-related parameters:

Parameter Description
Last visit time (Last access
-atime
time)
The last time when the
-ctime contents of the file were
modified
The last time when the
-mtime file attributes were
modified
 find
Take -mtime for example：
∙ -mtime n：The file that was modified on the day n days ago
∙ -mtime +n：List the files that were modified n days ago (not including the nth
day)
∙ -mtime –n：List the files that have been modified within n days (including the
day)
∙ -newer file ：List all the files that are more recent than the existing file
named file

List all the files that have changed within 24 hours:

$ find ~ -mtime 0
List all the files that are more recent than the Code folder in the user directory:
$ find ~ -newer /home/rapidcode/Code
 File formats

File formats commonly used by compression package:

The most common file formats on Windows are * .zip, * .rar, * .7z.

On Linux, in addition to the above three formats, there

are *.gz, *.xz, *.bz2, *.tar, *.tar.gz, *.tar.xz, *.tar.bz2.
 File formats:extensions

Extension Description
*.zip The files are packaged and compressed using zip programs
*.rar The files are packaged and compressed using rar programs
*.7z The files are packaged and compressed using 7z programs
*.tar The files without compression are packaged using tar programs
*.gz The compressed files are packaged using gzip programs
*.xz The files are packaged and compressed using xz programs
*.bz2 The files are packaged and compressed using bzip2 programs
The files are packaged using tar programs and compressed using
*.tar.gz
gzip programs
The files are packaged using tar programs and compressed using
*.tar.xz
xz programs
The files are packaged using tar programs and compressed using
*tar.bz2
bzip2 programs
The files are packaged using tar programs and compressed using
*.tar.7z
7z programs
 Using zip to package programs

• Use zip to package a folder:

$ cp -r /lib .
$ zip -r -q -o rapidcode.zip /home/rapidcode/lib
$ du -h rapidcode.zip
$ file rapidcode.zip

-r : the recursive package contains the entire contents of the subdirectories.

-q : a quiet mode meaning no messages are output to the screen.
-o : indicates that the output file is required,

After packaging and compressing the directory, use du to view the size of the
compressed file (zip file).
Using zip to package programs
 Using zip to compress programs

• Set the compression level to 9 and 1 (9 maximum, 1 minimum):

$ zip -r -9 -q -o rapidcode_9.zip /home/rapidcode/lib -x ~/*.zip
$ zip -r -1 -q -o rapidcode_1.zip /home/rapidcode/lib -x ~/*.zip

1 means the fastest compression but bulky.

9 stands for the smallest size but takes the longest time.
-x is to avoid the previous package-generated zip file being packaged into this current
zip file.

You can only use absolute path, otherwise it will not work.
 Using zip to compress programs

Use du to view the default compression level, the minimum and the highest
compression levels, and also the sizes of uncompressed files:

$ du -h -d 0 *.zip ~ | sort
h， --human-readable
d， --max-depth（Display an entry for all files and directories depth.)
 Creating an encrypted zip package

Use the -e parameter to create an encrypted file:

$ zip -r -e -o rapidcode_encryption.zip /home/rapidcode/lib

Warning: For zip, there are some compatibility issues between Windows and
Linux/UNIX, such as line break(a non-printable character signifying end of line),
which is CR+LF (Carriage-Return + Line-Feed) in Windows and LF in Linux. If you
open a text content on a Windows platform, which was edited on a Linux platform,
you can't see line breaks.

$ zip -r -l -o rapidcode.zip /home/rapidcode/lib

For the above problem, you need to add -l parameter to convert LF to CR+LF.
 Using unzip to decompress the zip file

Extract files in rapidcode.zip to the current directory:

$ unzip rapidcode.zip

Use quiet mode to extract the file to a directory specified using -d parameter:
$ unzip -q rapidcode.zip -d ziptest

If you only intend to see the contents of the compressed package, you can use
the -l parameter. The files in the package will not be extracted:
$ unzip -l rapidcode.zip

Using the -O parameter, we can specify the encoding type:

$ unzip -O GBK ww.zip
Using unzip to decompress the zip file
 Using rar to create .*rar files

rar is a compressed file format on Windows. On Linux, you can use rar and unrar to
create and extract *.rar files.

• Install rar and unrar :

$ sudo apt-get update
$ sudo apt-get install rar unrar

• Add files to archive:

$ rm *.zip
$ rar a rapidcode.rar /home/rapidcode/lib

The above command uses the a parameter to add a directory /home/rapidcode/lib

to an archive.

Warning：There is no - before parameters of rar command.

 Using rar to extract .*rar files

• Remove a file from an archive:

$ rar d rapidcode.rar .zshrc

• View an archive but not decompress it:

$ rar l rapidcode.rar

• Use unrar to extract an rar archive:

Extract files to the current path-

$ unrar x rapidcode.rar

Extract files to a specified path-

$ mkdir tmp
$ unrar e rapidcode.rar /tmp/

rar command has a lot of parameters. We are only going through some
basic operations.
 Creating tar archive

First of all, we start by learning some of the basic uses of tar.

• Create a tar archive (called a tarball) without compression:

$ tar -cf rapidcode.tar home/rapidcode/lib

-c means the creation of a tar archive.

-f is used to specify a file name.

Pay attention that the file name must follow the -f parameter.
 Extracting and view tar archive
• Extracting a tar archive (tarball)

-x parameter to extract to the current directory.

-C parameter along with -x parameter to extract to a specified path:
$ mkdir tardir
$ tar -xf rapidcode.tar -C tardir

• Viewing an archive

-t parameter to view the contents of a tar archive but not decompress it:

$ tar -tf rapidcode.tar

 Create a *.tar.gz archive
• Adding the -z parameter, we can use gzip to compress files:
$ tar -czf rapidcode.tar.gz home/rapidcode/lib

• Extracting a *.tar.gz archive

Use -x parameter for extraction:

$ tar -xzf rapidcode.tar.gz
We only need to change one parameter if we want to use a different compression
tool:

File format Parameter

*.tar.gz -z

*.tar.xz -J

*.tar.bz2 -j
 Built in commands and external commands:

Built-in commands are actually part of the shell program, which contains
some Linux system commands. These commands are written in the
builtins of the bash source and are recognized by the shell program and
run in the shell program itself.

External commands are more practical programs in Linux. Since the functions of
these programs are usually strong, these contain a large amount of code.
External commands are installed outside Bash, usually placed in /bin, /usr/bin,
/sbin and /usr/sbin.
 Built in commands and external commands:

To determine if the command is built-in or external: $ type cmd

For example, the results of these two command executions are different:
$ type exit

$ type service
 Help commands
• help
• man
• info

• help: help command is only applicable to shell built-in commands

$ help ls , $ ls --help
 Help commands
• man: man can get more information than help. Moreover, man applies to all
commands (built-in commands and external commands).
$ man ls
 Help commands

In The above output, top left corner shows LS (1). LS indicates the name of the
manual and (1) indicates that the manual is in the first chapter. What is this
section or chapter? There are so many chapters in the man manual：

Chapter Number Description

1 Standard commands
2 System calls
3 Library functions
4 Special devices
5 File formats
6 Games and toys
7 Miscellaneous
8 Administrative commands
9 Others
 Help commands

info: is a GNU project from the Free Software Foundation. It can have a more complete
display of GNU information. So you can get more information about commands.
$ info cat
 Assignment

Create a file named test, then use zip and tar to package and compress it
respectively. Last step, you need to extract the archive to /home/rapidcode.
 Linux
Execution Sequence control and pipeline
May 2020
 Executing commands sequentially

If you need to use apt-get to install a software package and run the installed software
(or command tool) immediately but your host has not updated the software list, then
you may have to perform the following series of operations:

$ sudo apt-get update

$ sudo apt-get install some-tool
$ some-tool

To enter all the commands at once and let the commands execute by themselves
in order , use ; to complete as follows:

$ sudo apt-get update;sudo apt-get install s

 Executing commands optionally
Problem with the above operation, if the previous command is unsuccessful and the
subsequent command is dependent on the result of the previous command, it will
take a long time and eventually we might end up with a wrong result.

So we need to be able to selectively execute the commands

$ which cowsay>/dev/null && cowsay -f head-in ohch~

&& is used to implement selective execution of commands. It means that if the

preceding command execution returns a value of 0 as its result, the subsequent
command is executed.

To install the cowsay :

$ sudo apt-get install cowsay
$ cowsay hello everyone
$ cowsay –f elephant hello everyone
 Installing Cowsay
Rapidcode
Rapidcode
 Actions using cowsay
Rapidcode

Rapidcode
 Cowsay performing ohch
Rapidcod
e

Rapidcod
e
 What is pipeline

A pipeline is a sequence of processes chained together by their standard

streams, so that the output of each of the processes (stdout) feeds directly as
input (stdin) to the next one.

Pipes are divided into anonymous pipes and named pipes.

• When we use some filtering procedures, we use anonymous pipes. At the

command line, the pipe is represented by |

• A named pipe appears as a file, and generally has processes attached for IPC
(interprocess communication).
 Executing ‘ls’ on etc
Let's try the pipeline to view /etc directory. We use ls to view the directory listing:

$ ls -al /etc
Abhi
 Implementing Pipe
Use a pipeline here: $ ls -al /etc | less
 Using cut command

To print some fields for each row - $ cut /etc/passwd -d ':' -f 1,6s

Printing the first N characters of every line in /etc/passwd

Print the first five characters (including the fifth one):
$ cut /etc/passwd -c -5

Print the characters after the first five characters (including the fifth one):
$ cut /etc/passwd -c 5-

Print the fifth character:

$ cut /etc/passwd -c 5

Printing second to fifth characters (including both of them):

$ cut /etc/passwd -c 2-5
Performing cat on passwd file
Performing cut on passwd file
 Using grep command
grep : Global regular expression print
It is used for searching for the matching string in text or stdin

It combines regular expressions to achieve very complex but very efficient

match-and-search functions.

The general form of grep is：

$ grep [options]... The_expression_used_to_match [file]...

Search all the text files containing “originally" in the /home directory and
display the line number:
$ grep -rnI “originally" ~

The -r parameter indicates recursively searching the file in all subdirectories.

-I Indicates that binary files are ignored.

View the strings in the environment variables ending with “code":

$ export | grep ".*code$“

$ means the end of a line.

Implementing grep Utility
 wc : word count
wc- word count
It’s a simple and counting compact tool
General form : $ wc [options] filenames

$ wc /etc/passwd
Count the number of folders in /etc:
$ ls -dl /etc/*/ | wc -l
 wc : word count

The following are the options and usage provided by the command.

wc -l : Prints the number of lines in a file.

wc -w : prints the number of words in a file.
wc -c : Displays the count of bytes in a file.
wc -m : prints the count of characters from a file.
wc -L : prints only the length of the longest line in a file.
 Using sort command

This command can sort the input in a certain way.

It supports sorting by dictionary, number, month, by random and in reverse
order.
It can also sort by specific fields and so on.

Sort by default or dictionary order:

$ cat /etc/passwd | sort

Sort in reverse order:

$ cat /etc/passwd | sort -r

Sort by specific fields:

$ cat /etc/passwd | sort -t':' -k 3

-t is used to specify a delimiter. Here we use ":" as a delimiter; -k n is used to

specify what field to sort by.
Using sort command
 Assignment

1. Match all lines that start with ‘hello’. E.g.: “hello there”
2. Count all lines that end with ‘done’. E.g.: “well done”
3. Match all lines that contain any of the letters ‘a’, ‘b’, ‘c’, ‘d’ or ‘e’.
4. Match all lines that do not contain a vowel
 Linux
Simple text processing
May 2020
 Text processing commands

• tr
• col
• join
• paste
 Using tr command
The command tr can be used to delete word(s) in a piece of text or to convert
it.

Format：
$ tr [option]...SET1 [SET2]

The Use of tr：

Option Description
Delete characters in string1 from
-d
the input.
Remove the characters specified
-s by set1 that are consecutive and
repeated in the input text.
 Using tr command
Delete all of 'o', 'l' and 'h' characters in "hello rapidcode":
$ echo 'hello rapidcode' | tr -d ‘olh’

Delete duplicated char 'l' in 'hello':

$ echo 'hello' | tr -s 'l'

Show the input text in uppercase or lowercase. The following commands show in
uppercase:
$ cat /etc/passwd | tr '[:lower:]' '[:upper:]'
or
$ cat /etc/passwd | tr '[a-z]' '[A-Z]'

For more uses of tr, use tr --help or man tr to fetch further information.
Executing tr command
 Using col command
col: col can replace the Tab character with a number of space characters or reverse
the operation.

Format:
$ col [option]

Use Cases for col：

Option Description

Convert Tab to multiple

-x
spaces.

Convert Spaces to Tab

-h
(Default).
 Using col command

Example：

To view invisible characters in /etc/protocols: We can see a lot of ^I in the file.

As a matter of fact, this is Tab transformed into visible characters:
$ cat -A /etc/protocols

Use col -x to convert tabs in /etc/protocols to spaces, then use cat to see the file
content; and we will find that ^I is gone:
$ cat /etc/protocols | col -x | cat -A
Performing cat on protocol file
Executing col command
 Using join command
join: The join utility performs an "equality join'' on the specified files and writes the
result to the standard output.

The "join field'' is the field in each file by which the files are compared.

Format：
$ join [option]... file1 file2

Parameters (Options) of join：

Option Description
-t Specify the delimiter. Default is a space.
-i Ignore case differences.
Indicate which field to use for the first file to
-1 field
compare. Default is the first field.
Indicate which field to use for the second file to
-2 field
compare. Default is the first field.
 joining two files
Create two files
$ echo '1 hello' > file1
$ echo '1 rapidcode' > file2
$ join file1 file2

Merge /etc/passwd with /etc/shadow, specifying ':' as a delimiter

$ sudo join -t':' /etc/passwd /etc/shadow
$ sudo join -t':' -1 4 /etc/passwd -2 3 /etc/group
Executing join command
 Using paste command
paste is similar to join. It simply merges multiple files together without comparison.

Format：
$ paste [option] file...

Parameters (Options) of paste：

Option Description
Specify the merged
-d delimiter, which
defaults to tab.
Concatenate all of the
lines of each of the
-s
input files in
command-line order.
 Executing paste command
Example：
$ echo hello > file1
$ echo rapidcode > file2
$ echo rapidcode.io > file3
$ paste -d ':' file1 file2 file3
$ paste -s file1 file2 file3
 Assignment

In the "File Packing and Compression" section, we mentioned that some special
characters in text files on Windows/DOS and Linux/UNIX are not the same, such as
line break:
It is CR+LF (\r\n) on Windows, and LF (\n) on Linux/UNIX.
We can use the cat -A text to see the invisible special characters contained in the
text. On Linux, \n is shown as $, and \r\n is shown as ^M$ on Windows.

We want you to use the previously learned commands to complete the conversion
of the DOS text format to the UNIX text format.
 Linux
Data Streaming & File descriptor
May 2020
 Data Stream

A Linux stream is data traveling in a Linux shell from one process to another
through a pipe, or from one file to another as a redirect.

Streams can travel through several Linux stream-pipe connections of incremental

commands to accomplish administrative tasks.

A Linux shell, such as Bash, receives input and sends output as sequences or
streams of characters
 Standard streams
Linux shells use three standard I/O streams, each of which is associated with a
well-known file descriptor:

1. stdin is the standard input stream, which provides input to commands. It has
file descriptor 0

2. stdout is the standard output stream, which displays output from commands. It
has file descriptor 1.

3. stderr is the standard error stream, which displays error output from
commands. It has file descriptor 2.
Understanding Standard streams
 Understanding File descriptor

When a process makes a successful request to open a file, the kernel returns a file
descriptor which points to an entry in the kernel's global file table.
The file table entry contains information such as the inode of the file, byte offset, and
the access restrictions for that data stream (read-only, write-only, etc.).
 Data stream redirection
In the Linux command line interface, operators like pipe (|) and redirect (< and >)
(>>) control input and output streams.

The original output to the standard output is redirected to a file. Since the standard
output (/dev/stdout) is a file, there is no problem when we redirect the output to
another file.

$ echo 'hello rapidcode' > redirect

$ echo 'www.rapidcode.io' >> redirect
$ cat redirect

You can also try echo 'hello rapidcode' < redirect.

The difference between < and >, >> is the direction of redirect. >, >> mean the
direction is left to right, and < means the direction is right to left.
Redirecting stdout
 stderr redirection

Both the standard output and the standard error are directed to the screen
display. So, the output of the command we often see usually includes the standard
output and standard error results, such as the following:

Use the cat command to read two files at the same time, one of which exists and the
other does not exist:
$ cat test.c hello.c

Then, we redirect the output to a file.

$ cat test.c hello.c > check
Performing redirection on stderr
 Using file descriptors to hide errors
Use file descriptors to hide some errors or warnings

Redirect the standard error to the standard output, and then redirect the
standard output to the file:
$ cat Documents/test.c hello.c >somefile 2>&1

Or use the special redirection symbol "&" provided by bash to redirect

standard output and standard error to the file at the same time:
$ cat Documents/test.c hello.c &>somefile

Note that you should add & before the output redirect file descriptor, otherwise the
shell will be redirected to a file named 1.
Redirecting stdout & stderr
 Using tee command
Sometimes, in addition to redirecting the output to a file, we also need to print the
information at the terminal. Then you can use the tee command:
$ echo 'hello rapidcode' | tee hello
 Permanent redirection
Redirect operations seen above were temporary, which means that the redirection
was only valid for the current command. We can use exec to implement a
permanent redirect:

# Open a sub-shell
$ zsh

# Use exec to redirect the standard output to a file

$ exec 1>somefile

# The output of the command you execute later will be redirected to the file
until you exit the current shell, or cancel the exec's redirect (later we will tell
you how to operate)
$ ls
$ exit
$cat somefile
Performing permanent redirection
 Creating a file descriptor
There are 9 file descriptors in the shell.

We can also use the file descriptors 3 to 8, but they are not open by default.

You can use the following command to view the file descriptors that are
open in the current shell process:
$ cd /dev/fd/;ls –Al

exec can also create a new file descriptor:

$ bash
$ exec 3>somefile
$ cd /dev/fd/;ls -Al;cd -

# Note that there should be no space between > and &. If there is a space,
result will be wrong.
$ echo "this is test" >&3
$ cat somefile
$ exit
Showing file descriptor
 Closing a File descriptor
Use the following operation to turn 3 file descriptor off:
$ exec 3>&-
$ cd /dev/fd;ls -Al;cd -
 Assignment

Understand the following code. This code does not work properly.
Please use what you learned in this section to analyze the underlying reason
and try to solve this problem.

while read filename; do

rm -iv $filename
done <<(ls)
 Linux
Regular Expression | grep| sed |awk
May 2020
 Regular Expression

• A regular expression, regex or regexp is, a sequence of characters that

defines a search pattern.
Usually, this pattern is then used by string searching algorithms for "find" or
"find and replace" operations on strings.

• Many programming languages support the use of regular expressions

for string manipulation. For example, in Perl, it builds a powerful regular
expression engine.

• In terms of form and function, regular expressions are similar to

wildcards.
However, there is a big difference between them, especially in the meanings of
some special matching characters. Hopefully you will not confuse these two.
 Understanding Regular expression

Example:

Suppose that we have a text file, including two strings "rapidcode" and
"coderapid".
rapid*

If it is a regular expression, it will only match rapid. If * is a wildcard, both strings

(rapidcode coderapid) will be matched at the same time.

Why?

Because in regular expression, * indicates that the preceding

sub-expression is matched (the character before it) zero or multiple times,
such as it can match "rapid", "rapids", "rapidcode", "coderapid" , and as a
wildcard, it indicates matching any wildcard followed by any number of
arbitrary characters. So it can match "rapidcode", and "coderapid".
 Basic syntax of regular expression
| separates alternate possibilities.
For example, "boy|girl" can match "boy" or "girl".

How to limit the number of matches？

• + can match the preceding pattern element one or multiple times.
For example, "goo+gle", can match "gooogle", "goooogle" and so on;

• ? can match the preceding pattern element zero or one time.

For example, "colou?r" can match "color" or "colour";

• * can match the preceding pattern element zero or more times.

For example, "0*42" can match "42", "042", "0042", "00042" and so on.

Range and Priority:

( ) can be used to define the scope and priority of the pattern string, which can be
simply understood as a string within the parentheses as a whole. For example,
"gr(a|e)y" is equivalent to "gray|grey". "(grand)?father" can match "father" and
"grandfather".
 Basic syntax of regular expression
Syntax:Regular expressions have many different styles. Here are some of the
commonly used rules for regular expression matching for Perl and Python
programming languages:

∙ \: *Mark the next character as a special character or a literal character. *For example, "n"
matches the character "n". "\n" matches a line break. "\"" matches """ and "\(" matches "(".
∙ ^: Match the start position of the input string.
∙ $: Match the end position of the input string.
∙ {n}: n in {n} is a non-negative integer that means matching n times. For example, "o{2}" cannot
match "o" in "Bob", but it can match two "o" in "food".
∙ {n,}: n in {n,} is a non-negative integer that means matching at least n times. For example,
"o{2,}" cannot match "o" in "Bob", but it can match all "o" in "fooooood".
∙ {n,m}: m and n are non-negative integers, where n <= m. It can match at least n times and match
utmost m times. For example, "o{1,3}" will match the first three "o" in "fooooood". "O{0,1}" is
equivalent to "o?". Please note that there cannot be spaces between commas and two numbers.
∙ *: Match the previous subexpression zero or more times. For example, "zo*" can match "z",
"zo" and "zoo". * is equivalent to {0,}.
 The Priority of Operators
The priority is decreasing from top to bottom, from left to right:
∙ \ : Escape character
∙ (), (?:), (?=), []
∙ *, +, ?, {n}, {n,}, {n,m} : Restrictions
∙ ^, $ : Location point
∙ | : The choice (also known as alternation or set union) operator matches either
the expression before the operator or the expression after the operator.
 Introduction to ’grep’
grep is used to print the matching pattern string in the output text, which uses a
regular expression as a condition for pattern matching. grep supports three regular
expression engines respectively, with three parameters to specify the chosen
engine:
Parameter Description
POSIX extended
-E regular expression,
ERE

POSIX basic regular

-G
expression, BRE

Perl regular
-P
expression, PCRE

In most cases, you will only use ERE and

BRE.
 grep parameters
Parameter Description

The offset in bytes of a matched pattern is displayed in front

-b
of the respective matched line.
Only a count of selected lines is written to the standard
-c
output.
-i Ignores case.
Displays the line number where the matching text is
-n
located.
Selected lines are those not matching any of the specified
-v
patterns.
-r Recursively search subdirectories listed.
-A n Print n lines of trailing context after each match.
-B n Print n lines of leading context before each match.
Mark up the matching text with the expression stored in
--color=auto GREP_COLOR environment variable. The possible values
of when can be never, always or auto.
 grep:Using POSIX Basic Regular Expression

• Position match
Find the line that starts with "rapidcode" in /etc/group:
$ grep 'rapidcode' /etc/group
$ grep '^rapidcode' /etc/group

• Restriction match
Match all strings that begin with 'z' and end with 'o':
$ echo 'zero\nzo\nzoo' | grep 'z.*o’

Match strings beginning with 'z', ending with 'o' and with an arbitrary character in
the middle:
$ echo 'zero\nzo\nzoo' | grep 'z.o’

Match strings that begin with 'z' and end with any number of 'o':
$ echo 'zero\nzo\nzoo' | grep 'zo*’

\n is the line break.

 grep:Using POSIX Basic Regular Expression

• Choice match
By default, grep is case-sensitive. The command below will match all
lowercase letters:
$ echo '1234\nabcd' | grep '[a-z]’

Match all the numbers:

$ echo '1234\nabcd' | grep '[0-9]’

Match all the numbers:

$ echo '1234\nabcd' | grep '[[:digit:]]’

Exclude characters:
$ echo 'geek\ngood' | grep '[^o]’

Note that when ^ is placed in class (square brackets covering a pattern), it

means to exclude the characters, otherwise ^ means the first line.
 grep: special symbol and instruction
The following contains the complete list of special symbols and instructions:

Special
Description
Symbol
[:alnum:] Upper and lower case letters and digits (0-9, A-Z, a-z)
[:alpha:] Any English uppercase and lowercase letters (A-Z, a-z)
[:blank:] Blank key and [Tab]
Control buttons on the top of the keyboard, including
[:cntrl:]
CR, LF, Tab, Del and so on
[:digit:] Numeral digits (0-9)
All the keys except for blank key (for example, Space)
[:graph:]
and [Tab]
[:lower:] Lowercase letters (a-z)
[:print:] Characters that can be printed out
[:punct:] punctuation symbols (" ' ? ! ; : # $ ...)
[:upper:] uppercase letters ( A-Z)
[:space:] Symbols include blank keys, [Tab], CR and so on
[:xdigit:] Hexadecimal digits, including 0-9, A-F, a-f
 grep:Using POSIX 'Extended Regular Expression'

Using Extended Regular Expression with grep requires adding the -E parameter,
or using egrep.
• Restriction match
Match only "zo":
$ echo 'zero\nzo\nzoo' | grep -E 'zo{1}’

Matches all words beginning with "zo":

$ echo 'zero\nzo\nzoo' | grep -E 'zo{1,}’
• Choice match
Match "www.rapidcode.co.in" and "www.google.com":
$ echo 'www.rapidcode.co.in\nwww.baidu.com\nwww.google.com' | grep -E
'www\.(rapidcode.co.in|google.com)’

Or match the content that does not contain "baidu":

$ echo 'www.rapidcode.co.in\nwww.baidu.com\nwww.google.com' | grep -Ev
'www\.baidu\.com’

Since . has a special meaning, so we need to use \. to escape it.

 Introduction to sed
sed is short for "stream editor for filtering and transforming text", meaning
that the stream editor for filtering and converting text.

Format:

sed [Parameters]... [Command] [File]...

# For example：
$ sed -i '1s/sad/happy/' test
#Replace the "sad" in the first line of the test with "happy"
Commonly used parameters of sed

Parameter Description
By default, each line of input is echoed to the
standard output after all of the commands have
-n
been applied to it. The -n option suppresses this
behavior.
Append the editing commands specified by the
-e
command argument to the list of commands.
Specify to execute the commands in
-f filename
the filename file.
Use extended regular expressions, which default to
-r
standard regular expressions.

Directly modify the contents of the input file instead

-i
of printing to standard output.
 The execution command sed
Format：

[n1],[n2]command
[n1]~[step]command
# Some of these commands can be added to the scope of the role, such as:
$ sed -i 's/sad/happy/g' test # g represents the global scope
$ sed -i 's/sad/happy/4' test # 4 represents the 4th matching string

[n1],[n2] means all lines from n1 to n2. [n1]~[step] means all lines from n1
and the step size is step. command means the execution command. Here
are some commonly used execution commands:
 The execution command sed

Command Description

S Replace the specified character in a line

C Change the selected line to the new text

A Insert the text below the current line (a=append)

I Insert the text above the selected line (i=insert)

P Print the row of the template block

D Delete the selected row

 Operation experience of sed
First let's choose a text file for practice:
$ cp /etc/passwd ~
Then print the specified line:
# Print 2-5 lines
$ nl passwd | sed -n '2,5p'
# Print odd lines
$ nl passwd | sed -n '1~2p'

Replace the specified character in a line:

# Replace "rapidcode" in the input text globally with "hehe" and print the
replacement line. Note that the last "p" command cannot be omitted here.
$ sed -n 's/rapidcode/hehe/gp' passwd

Change the selected line to the new text:

$ nl passwd | grep "rapidcode"
# change line 21
$ sed -n '21c\www.rapidcode.co.in' passwd
Executing sed
Executing sed
 Introduction to ‘awk’

• AWK is an excellent text processing tool, one of the most powerful data
processing engines available in Linux and Unix environments.

• Its name comes from its founders, Alfred Aho, Peter Jay Weinberger (Peter
Weberger) and Brian Wilson Kernighan (Brian Colin), first letters of their last
names

• The three creators have officially defined AWK programming language as

"style scanning and processing language".

• It allows you to create short programs that read input files, sort data,
process data, perform calculations on input and generate reports, as well as
countless other functions.

• Most simply, AWK is a programming language tool for handling text.

 Versions of awk
nawk: In the mid-1980s, the awk language was updated and replaced with an
enhanced version called nawk (new awk). The old awk interpreter still exists in many
systems, but it is usually installed as an oawk (old awk) command.
And the nawk interpreter is installed as the main awk command, which can also use
the nawk command. Like gawk, it is also open source and can be obtained free of
charge.

gawk is the open source implementation of the awk interpreter for GNU Project.
Although the early GAWK release was an older AWK alternative, it was continually
updated to include NAWK's features.

mawk is also an interpreter for the awk programming language. The mawk
follows the AWK language defined by POSIX 1003.2 (draft 11.3) and contains some
features that are not mentioned in the AWK manual.
 Basic concept of awk

All operations are based on the pattern-action statements, as in the following

form:
$ pattern {action}

You can see that as with many programming languages all the actions are in {}.
pattern is usually a "relational" or "regular expression" that represents the text
used to match the input and action is the action that will be executed after the
match has been made. In a complete awk operation, you may have only one of
them. If there is no pattern , the default is to match all the input text. If there is
no action, the default is to print the matching content to the screen.
 Basic format of awk

awk [-F fs] [-v var=value] [-f prog-file | 'program text'] [file...]

The parameter -F is used to pre-specify the field delimiter.

-v is used to specify variables for the awk program in advance.
-f is used to specify the program file to be executed by the awk command.
 Operation experience of awk

Create a new text document using vim:

$ vim test

The text should include the following content：

I like linux
www.rapidcode.co.in

• Use awk to print the text:

$ awk '{
> print
> }' test
# Or just write a line
$ awk '{print}' test

In this operation, we have omitted the pattern. So, awk will match the
entire contents of the input text by default.
 Operation experience of awk

• Each field of the first row of test is shown as a single line:

$ awk '{
> if(NR==1){
> print $1 "\n" $2 "\n" $3
> } else {
> print}
> }' test

# or
$ awk '{
> if(NR==1){
> OFS="\n"
> print $1, $2, $3
> } else {
> print}
> }' test
 Operation experience of awk

• Change the separator of the second line to a space:

$ awk -F'.' '{
> if(NR==2){
> print $1 "\t" $2 "\t" $3
> }}' test

# or
$ awk '
> BEGIN{
> FS="."
> OFS="\t"
> }{
> if(NR==2){
> print $1, $2, $3
> }}' test
Performing awk
 Some built-in variable of awk
Name Description

If there are multiple files, only the first one is valid. If the input is from the
FILENAME
standard input, it is NULL.

$0 The contents of the current record.

$N N represents the field number. The maximum value is the value of the NF variable.

FS Field separator, represented by a regular expression, defaults to blank.

RS Input record separator (default is newline).

NF Number of fields in the current record.

NR Ordinal number of the current record.

FNR Ordinal number of the current record in the current file.

OFS Output field separator (default is blank).

ORS Output record separator (default is newline).

 Assignment

∙ Exercise 1: use regular expressions for more exercises.

∙ Exercise 2: Refer to sed Reference to master the basic principles of text
processing with sed; understand the concept of pattern space and hold
space.
∙ Exercise 3: Output some text in reverse order, and exchange odd and even
lines based on pattern space and hold space.
 Thank You

Document for internal circulation only. All right reserved. 2020