Data sets: fields and field separator
Generally, a Dataset is organized in fields.
A field is a unit of information and can contain either numeric or non-numeric data arranged in
rows or columns.
A field can be a group of columns or rows. Usually, fields are arranged in columns. In this
course, we refer to fields as a group of columns.
In the following example, the data file has 5 fields separated by a space character. In this
example, the fields are:
1. First name
2. Last name
3. Birth year
4. Career
5. Area of focus
Jane Bolden 1932 author economics
John Talbot 1945 poet english
In this other example, the field separator is a comma, and there are 5 fields
Jane,Bolden,1932,author,economics
John,Talbot,1945,poet,english
Video Lecture4-intro-and-wc command-4min
Introduction - Text processing
wc: get info on a data file
grep: extract lines matching a pattern
cut: extract columns or fields
sort: sort lines of text files
cat and paste: used to concatenate files together
sed: transform text
Before we start:
• Download the zip file data-temp.zip from Canvas -> Files -> zip files
• Put the data-temp.zip into your home directory
• Unzip the file data-temp.zip and a directory called data-temp will be created
• Open a terminal and cd to data-temp directory
• List the content of data directory. You will find data files temp.dat, temp-clean.dat
and temp-clean1.dat
You can also unzip by typing at the terminal
unzip data-temp.zip
1
wc command
wc (options) filename
Try:
wc temp.dat
Use man page to find out which info the wc command provides
man wc
Which option will display specifically the number of lines?
Explore the file: display the first 10 lines, and the last 10 lines of the file
Slides 2-4 video Lecture4-grep-5min
2
grep command
grep searches and prints lines that match one or more patterns
grep (options) pattern(s) filename
grep Data temp.dat
grep Climate temp.dat
grep AZ temp.dat
3
Some useful options of the grep command: -c, -e, -i, and –v
-c option: count the number of lines matching patterns
grep –c pattern filename
multiple -e options: search multiple patterns
grep –e pattern1 –e pattern2 filename
….pattern1…
….pattern2…
-v option: search lines excluding a pattern
grep –v pattern filename
Ignore case
grep –i pattern filename
Slide 4 video Lecture4-sort-4min
4
sort command to sort lines
sort (options) filename
Useful options of the sort command:
-u to sort and remove duplicates
-knumber to sort lines based on a certain field number
-n to sort numerically (if you do not specify this, it will sort alphabetically)
-tsep to specify field separator (only if it is different than whitespaces)
If the field separator is one or more than one whitespaces (spaces, tabs) you do not need to
use option –t.
Try These
sort temp-clean.dat #sort alphabetically based on 1st field
sort –n temp-clean.dat #sort numerically based based on 1st field
sort –k2 temp-clean.dat #sort alphabetically based on 2nd field
sort -nk3 temp-clean.dat #sort numerically based on 3rd field
sort –t: -k3 temp-clean1.dat #specify that field separator is : and
sort alphabetically based on 3rd field
Slide 5 video Lecture4-cut-4min
5
cut command to extract columns or fields
While grep extracts lines matching a pattern, cut extracts columns or fields
cut option(s) filename
• option –c n to cut specific columns by specifying column number n
Try these
cut –c 1 temp-clean.dat
cut –c 1-3 temp-clean.dat
cut -c 1-3,7-9 temp-clean.dat
• option –d"sep" –f n to cut specific field(s) by specifying field separator sep and field
number n. The field separator is also called a delimiter.
Try these
#field separator of temp-clean.dat is a blank character
#field separator of temp-clean1.dat is a :
cut -d":" –f 1 temp-clean1.dat
cut -d" " –f 1 temp-clean.dat
cut -d" " –f 1,3 temp-clean.dat
Slide 6 video Lecture4-cat and paste-2min
6
cat and paste
cat and paste commands can be used to concatenate files together
Try this:
Make a file called file1 and write in it Hello
Make another file called file2 and in it write Unix
Now try these and look at the output
cat file1 file2 #vertically concatenate
paste file2 file1 #horizontally concatenate
Slide 7 video Lecture4-sed-3min
7
The sed command
sed (stream editor) is a Unix utility that parses and transforms text
sed 's/word1/word2/' filename
sed 's/word1/word2/g' filename
Make a file called sed-example and write in it:
I love bla. I said I love bla
Then run sed:
sed 's/bla/tea/' sed-example
sed 's/b/Co/' sed-example
sed 's/bla/tea/g' sed-example
sed 's/b/Co/g' sed-example
If you add g at the end, it will replace all occurrences
Slides 8-9 video Lecture4-printf-type7min
8
printf – format and print text
Example
var1=mass
var2=18.547
echo $var1 $var2 Kg
printf "%-8s %.2f Kg\n" $var1 $var2
mass 18.55 Kg
printf to format text: define the type 9
printf formats and prints argument(s) under control of the format(s)
printf "format" argument
printf "%[width].[precision]type" argument
width and precision are optional modifiers
%type
%s string
%i or %d integer
%f float or real number
%e scientific notation or exponential
\n new line
Try these
printf "%f\n" 1.6547
printf "%e\n" 1250000
printf "%d\n" 2
printf "%s\n" Two
Slide 10 video Lecture4-printf-precision-3min
10
printf to format text: specify precision
printf "%[width].[precision]type" argument
the precision modifier is a positive integer number which follows
the dot %.precisiontype
%.precisions for string type specifies the number of string
characters to output
%.precisionf for float type specifies the number of decimal
places to output
%.precisione for scientific notation type specifies the number of
decimal places to output
Try these
printf "%.2f\n" 1.6547 #format float with 2 decimal places
printf "%.3f\n" 1.6547
printf "%.0f\n" 1.6547
#sometimes 1 is included for width: printf "%1.2f\n" 1.6547
printf "%.3e\n" 1250000 #format in scientific notation with 3
decimal places
printf "%.1s\n" Two #output 1 character
printf "%.2d\n" 2
Slide 11 video Lecture4-printf-width-4min
11
printf to format text: specify width
The width modifier specifies the number of characters to print.
% width.precisiontype format
The width is an integer number, which precedes the dot.
%width.precisiontype
If the width is larger than the number of characters of the
output, it will add whitespace characters to the left.
Negative integer - whitespace characters are added to the right
%-width.precisiontype
Try these
printf "%.2f\n" 1.6547
printf "%5.2f\n" 1.6547
printf "%6.2f\n" 1.6547
printf "%3s\n" Two %6.2f
printf "%4s\n" Two
printf "%5s\n" Two
printf "%-5s\n" Two
printf "%-6.2f Kg\n" 1.6547
Slide 12 video Lecture4-printf-multiple-arguments-4min
12
printf – define type and precision of multiple arguments and include text
printf "format1 format2" argument1 argument2
printf "%w.ptype1 %w.ptype2" argument1 argument2
printf "%-10.4s %.2f\n" Temperature 1.6547
The text you include within the " " will be printed to the screen, including the spaces
characters
printf "Mass %.2f .. in %s\n" 65.4747 Kg
Mass 65.47 .. in Kg
Play with the numbers
printf "%s %.2f Kg\n" mass 18.547
printf "%6s %7.2f Kg\n" mass 18.547
printf "%-6.2s %7.2f Kg\n" mass 18.547
