2023/4/21 10:42 For print

Lecture 2: Shell Tools and Scripting (2020)

Time Subtitle
0s Okay, welcome back.
2s Today we're gonna cover a couple separate
5s two main topics related to the shell.
7s First, we're gonna do some kind of shell scripting,
mainly related to bash,
11s which is the shell that most of you will start
14s in Mac, or like in most Linux systems, that's the default
shell.
18s And it's also kind of backward compatible through other
shells like zsh, it's pretty nice.
22s And then we're gonna cover some other shell tools that
are really convenient,
26s so you avoid doing really repetitive tasks,
29s like looking for some piece of code
31s or for some elusive file.
33s And there are already really nice built-in commands
36s that will really help you to do those things.
40s So yesterday we already kind of introduced
43s you to the shell and some of it's quirks,
46s and like how you start executing commands,
48s redirecting them.
50s Today, we're going to kind of cover more about
52s the syntax of the variables, the control flow,
56s functions of the shell.
57s So for example, once you drop into a shell, say you
want to
1:02 define a variable, which is one of the first things you
1:06 learn to do in a programming language.
1:09 Here you could do something like foo equals bar.
1:12 And now we can access the value of foo by doing
"$foo".
1:18 And that's bar, perfect.
1:21 One quirk that you need to be aware of is that
1:24 spaces are really critical when you're dealing with bash.
1:27 Mainly because spaces are reserved, and that will be for
separating arguments.
1:33 So, for example, something like foo equals bar
1:36 won't work, and the shell is gonna tell you why it's not
working.
about:blank 1/24
2023/4/21 10:42 For print

1:42 It's because the foo command is not working, like foo is
non-existent.
1:46 And here what is actually happening, we're not
assigning foo to bar,
1:47 what is happening is we're calling the foo program
1:52 with the first argument "=" and the second argument
"bar".
1:57 And in general, whenever you are having some issues,
like some files with spaces
2:03 you will need to be careful about that.
2:06 You need to be careful about quoting strings.
2:10 So, going into that, how you do strings in bash. There
are two ways that you can define a string:
2:16 You can define strings using double quotes and you can
define strings using single,
2:24 sorry,
2:26 using single quotes.
2:29 However, for literal strings they are equivalent,
2:32 but for the rest they are not equivalent.
2:35 So, for example, if we do value is $foo,
2:43 the $foo has been expanded like a string, substituted to
the
2:48 value of the foo variable in the shell.
2:50 Whereas if we do this with a simple quote, we are just
getting the $foo as it is
2:58 and single quotes won't be replacing. Again,
3:02 it's really easy to write a script, assume that this is kind
of like Python, that you might be
3:07 more familiar with, and not realize all that.
3:10 And this is the way you will assign variables.
3:14 Then bash also has control flow techniques that we'll
see later,
3:17 like for loops, while loops, and one main thing is you can
define functions.
3:24 We can access a function I have defined here.
3:28 Here we have the MCD function, that has been defined,
and the thing is
3:34 so far, we have just kind of seen how to execute several
commands by piping
3:38 into them, kind of saw that briefly yesterday.
3:40 But a lot of times you want to do first one thing and then
another thing.
3:44 And that's kind of like the
3:47 sequential execution that we get here.

about:blank 2/24
2023/4/21 10:42 For print

3:50 Here, for example, we're calling the MCD function.

3:56 We, first,
3:57 are calling the makedir command, which is creating this
directory.
4:02 Here, $1 is like a special variable.
4:05 This is the way that bash works,
4:07 whereas in other scripting languages there will be like
argv,
4:12 the first item of the array argv will contain the argument.
4:16 In bash it's $1. And in general, a lot
4:19 of things in bash will be dollar something
4:21 and will be reserved, we will be seeing more examples
later.
4:26 And once we have created the folder, we CD into that
folder,
4:30 which is kind of a fairly common pattern that you will
see.
4:34 We will actually type this directly into our shell, and it will
work and
4:39 it will define this function. But sometimes it's nicer to
write things in a file.
4:45 What we can do is we can source this. And that will
4:50 execute this script in our shell and load it.
4:53 So now it looks like nothing happened, but now the
MCD function has
4:59 been defined in our shell. So we can now for example
do
5:03 MCD test, and now we move from the tools directory to
the test
5:09 directory. We both created the folder and we moved into
it.
5:15 What else. So a result is...
5:18 We can access the first argument with $1.
5:22 There's a lot more reserved commands,
5:26 for example $0 will be the name of the script,
5:30 $2 through $9 will be the second through the ninth
arguments
5:35 that the bash script takes. Some of these reserved
5:38 keywords can be directly used in the shell, so for
example
5:43 $? will get you the error code from the previous
command,
5:50 which I'll also explain briefly.
5:53 But for example, $_ will get you the last argument of the

about:blank 3/24
2023/4/21 10:42 For print

5:58 previous command. So another way we could have

done this is
6:03 we could have said like "mkdir test"
6:07 and instead of rewriting test, we can access that last
argument
6:12 as part of the (previous command), using $_
6:18 like, that will be replaced with test and now we go into
test.
6:25 There are a lot of them, you should familiarize with
them.
6:27 Another one I often use is called "bang bang" ("!!"), you
will run into this
6:32 whenever you, for example, are trying to create
something and you don't have
6:37 enough permissions. Then, you can do "sudo !!"
6:41 and then that will replace the command in
6:43 there and now you can just try doing
6:46 that. And now it will prompt you for a password,
6:48 because you have sudo permissions.
6:53 Before, I mentioned the, kind of the error command.
6:57 Yesterday we saw that, in general, there are
6:59 different ways a process can communicate
7:02 with other processes or commands.
7:05 We mentioned the standard input, which also was like
7:09 getting stuff through the standard input,
7:11 putting stuff into the standard output.
7:13 There are a couple more interesting things, there's also
like a
7:16 standard error, a stream where you write errors
7:19 that happen with your program and you don't want to
pollute the standard output.
7:23 There's also the error code, which is like a general
7:27 thing in a lot of programming languages,
7:29 some way of reporting how the entire run of something
went.
7:34 So if we do
7:36 something like echo hello and we
7:41 query for the value, it's zero. And it's zero
7:43 because everything went okay and there
7:45 weren't any issues. And a zero exit code is
7:49 the same as you will get in a language
7:50 like C, like 0 means everything went fine, there were no
errors.

about:blank 4/24
2023/4/21 10:42 For print

7:54 However, sometimes things won't work.

7:57 Sometimes, like if we try to grep for foobar in our MCD
script,
8:04 and now we check for that value, it's 1. And that's
8:08 because we tried to search for the foobar
8:10 string in the MCD script and it wasn't there.
8:13 So grep doesn't print anything, but
8:17 let us know that things didn't work by
8:19 giving us a 1 error code.
8:22 There are some interesting commands like
8:24 "true", for example, will always have a zero
8:29 error code, and false will always have a one error code.
8:35 Then there are like
8:37 these logical operators that you can use
8:40 to do some sort of conditionals. For example, one way...
8:43 you also have IF's and ELSE's, that we will see later, but
you can do
8:47 something like "false", and echo "Oops fail".
8:51 So here we have two commands connected by this OR
operator.
8:56 What bash is gonna do here, it's gonna execute the first
one
9:00 and if the first one didn't work, then it's
9:04 gonna execute the second one. So here we get it,
9:07 because it's gonna try to do a logical OR. If the first one
didn't have
9:12 a zero error code, it's gonna try to do the second one.
Similarly, if we
9:15 instead of use "false", we use something like "true",
9:19 since true will have a zero error code, then the
9:22 second one will be short-circuited and
9:24 it won't be printed.
9:32 Similarly, we have an AND operator which will only
9:36 execute the second part if the first one
9:39 ran without errors.
9:41 And the same thing will happen.
9:44 If the first one fails, then the second part of this thing
won't be executed.
9:50 Kind of not exactly related to that, but another thing that
you will see is
10:00 that no matter what you execute, then you can
concatenate
10:04 commands using a semicolon in the same line,
about:blank 5/24
2023/4/21 10:42 For print

10:07 and that will always print.

10:10 Beyond that, what we haven't seen, for example, is how
10:13 you go about getting the output of a command into a
variable.
10:19 And the way we can do that is doing something like this.
10:24 What we're doing here is we're getting the output of the
PWD command,
10:29 which is just printing the present working directory
10:32 where we are right now.
10:33 And then we're storing that into the foo variable.
10:37 So we do that and then we ask for foo, we view our
string.
10:42 More generally, we can do this thing called command
substitution
10:50 by putting it into any string.
10:51 And since we're using double quotes instead of single
quotes
10:55 that thing will be expanded and
10:57 it will tell us that we are in this working folder.
11:02 Another interesting thing is, right now, what this is
expanding to is a string
11:09 instead of
11:11 It's just expanding as a string.
11:13 Another nifty and lesser known tool is called process
substitution,
11:17 which is kind of similar. What it will do...
11:24 it will, here for example, the "<(", some command and
another parenthesis,
11:30 what that will do is: that will execute, that will get the
output to
11:34 kind of like a temporary file and it will give the file handle
to the command.
11:39 So here what we're doing is we're getting...
11:42 we're LS'ing the directory, putting it into a temporary file,
11:45 doing the same thing for the parent folder and then
11:48 we're concatenating both files. And this
11:51 will, may be really handy, because some commands
instead of expecting
11:55 the input coming from the stdin, they are expecting
things to
11:59 come from some file that is giving some of the
arguments.
12:04 So we get both things concatenated.

about:blank 6/24
2023/4/21 10:42 For print

12:12 I think so far there's been a lot of information, let's see a

simple,
12:17 an example script where we see a few of these things.
12:23 So for example here we have a string and we
12:27 have this $date. So $date is a program.
12:30 Again there's a lot of programs in UNIX you will kind of
slowly
12:34 familiarize with a lot of them.
12:36 Date just prints what the current date is and you can
specify different formats.
12:43 Then, we have these $0 here. $0 is the name
12:48 of the script that we're running.
12:50 Then we have $#, that's the number of arguments that
we are giving
12:56 to the command, and then $$ is the process ID of this
command that is running.
13:01 Again, there's a lot of these dollar things, they're not
intuitive
13:06 because they don't have like a mnemonic
13:07 way of remembering, maybe, $#. But
13:10 it can be... you will just be
13:12 seeing them and getting familiar with them.
13:14 Here we have this $@, and that will expand to all the
arguments.
13:19 So, instead of having to assume that,
13:21 maybe say, we have three arguments and writing $1,
$2, $3,
13:25 if we don't know how many arguments we can put all
those arguments there.
13:29 And that has been given to a for loop. And the for loop
13:33 will, in time, get the file variable
13:39 and it will be giving each one of the arguments.
13:43 So what we're doing is, for every one of the arguments
we're giving.
13:47 Then, in the next line we're running the
13:51 grep command which is just search for a substring in
some file and we're
13:56 searching for the string foobar in the file.
14:01 Here, we have put the variable that the file took, to
expand.
14:06 And yesterday we saw that if we care about the output
of a program, we can
14:11 redirect it to somewhere, to save it or to connect it to
some other file.

about:blank 7/24
2023/4/21 10:42 For print

14:15 But sometimes you want the opposite.

14:18 Sometimes, here for example, we care...
14:21 we're gonna care about the error code. About this script,
we're gonna care whether the
14:25 grep ran successfully or it didn't.
14:28 So we can actually discard entirely what the output...
14:33 like both the standard output and the standard error of
the grep command.
14:37 And what we're doing is we're
14:39 redirecting the output to /dev/null which
14:43 is kind of like a special device in UNIX
14:46 systems where you can like write and
14:49 it will be discarded. Like you can
14:51 write no matter how much you want,
14:52 there, and it will be discarded. And here's the ">" symbol
14:57 that we saw yesterday for redirecting output. Here you
have a "2>"
15:02 and, as some of you might have
15:04 guessed by now, this is for redirecting the
15:06 standard error, because those those two
15:08 streams are separate, and you kind of have to
15:11 tell bash what to do with each one of them.
15:14 So here, we run, we check if the file has
15:17 foobar, and if the file has foobar then it's
15:20 going to have a zero code. If it
15:22 doesn't have foobar, it's gonna have a
15:24 nonzero error code. So that's exactly what we
15:26 check. In this if part of the command we
15:31 say "get me the error code". Again, this $?
15:34 And then we have a comparison operator
15:37 which is "-ne", for "non equal". And some
15:41 other programming languages will have "==", "!=", these
15:47 symbols. In bash there's
15:51 like a reserved set of comparisons and
15:53 it's mainly because there's a lot of
15:54 things you might want to test for when
15:57 you're in the shell. Here for example
15:59 we're just checking for two values, two integer values,
being the same. Or for
16:03 example here, the "-F" check will let
16:08 us know if a file exists, which is

about:blank 8/24
2023/4/21 10:42 For print

16:10 something that you will run into very,

16:12 very commonly. I'm going back to the
16:17 example. Then, what happens when we
16:24 if the file did not have foobar, like there was a
16:28 nonzero error code, then we print
16:31 "this file doesn't have any foobar,
16:33 we're going to add one". And what we do is
16:36 we echo this "# foobar", hoping this
16:40 is a comment to the file and then we're
16:43 using the operator ">>" to append at the end of
16:47 the file. Here since the file has
16:50 been fed through the script, and we don't know it
beforehand, we have to substitute
16:54 the variable of the filename. We can actually run this.
We already have
17:03 correct permissions in this script and
17:05 we can give a few examples. We have a few files in this
folder, "mcd" is the
17:10 one we saw at the beginning for the MCD
17:12 function, some other "script" function and
17:15 we can even feed the own script to itself to check if it
has foobar in it.
17:21 And we run it and first we can see that there's different
17:26 variables that we saw, that have been
17:29 successfully expanded. We have the date, that has
17:33 been replaced to the current time, then
17:36 we're running this program, with three
17:39 arguments, this randomized PID, and then
17:44 it's telling us MCD doesn't have any
17:46 foobar, so we are adding a new one,
17:48 and this script file doesn't
17:50 have one. So now for example let's look at MCD
17:52 and it has the comment that we were looking for.
17:59 One other thing to know when you're executing scripts is
that
18:05 here we have like three completely
18:07 different arguments but very commonly
18:10 you will be giving arguments that
18:12 can be more succinctly given in some way.
18:16 So for example here if we wanted to
18:20 refer to all the ".sh" scripts we
18:25 could just do something like "ls *.sh"
about:blank 9/24
2023/4/21 10:42 For print

18:31 and this is a way of filename expansion that most shells

have
18:36 that's called "globbing". Here, as you
18:38 might expect, this is gonna say
18:39 anything that has any kind of sort of
18:42 characters and ends up with "sh".
18:45 Unsurprisingly, we get "example.sh" and "mcd.sh". We
also have these
18:52 "project1" and "project2", and if there
18:54 were like a... we can do a "project42", for example
19:00 And now if we just want to refer to the projects that have
19:04 a single character, but not two characters
19:07 afterwards, like any other characters,
19:08 we can use the question mark. So "?" will expand to
only a single one.
19:13 And we get, LS'ing, first
19:17 "project1" and then "project2".
19:21 In general, globbing can be very powerful. You can also
combine it.
19:31 A common pattern is to use what is called curly braces.
19:35 So let's say we have an image, that we have in this
folder
19:39 and we want to convert this image from PNG to JPG
19:43 or we could maybe copy it, or...
19:46 it's a really common pattern, to have two or more
arguments that are
19:49 fairly similar and you want to do something with them as
arguments to some command.
19:55 You could do it this way, or more succinctly, you can just
do
20:01 "image.{png,jpg}"
20:09 And here, I'm getting some color feedback, but what this
will do, is
20:13 it'll expand into the line above.
20:17 Actually, I can ask zsh to do that for me. And that what's
happening here.
20:23 This is really powerful. So for example
20:26 you can do something like... we could do...
20:29 "touch" on a bunch of foo's, and all of this will be
expanded.
20:35 You can also do it at several levels and you will do the
Cartesian...
20:41 if we have something like this, we have one group here,
"{1,2}"
about:blank 10/24
2023/4/21 10:42 For print

20:49 and then here there's "{1,2,3}", and this is going to do

20:53 the Cartesian product of these
20:54 two expansions and it will expand into all these things,
20:59 that we can quickly "touch".
21:03 You can also combine the asterisk glob with the curly
braces glob.
21:10 You can even use kind of ranges. Like, we can do
"mkdir"
21:16 and we create the "foo" and the "bar" directories, and
then we
21:21 can do something along these lines. This
21:25 is going to expand to "fooa", "foob"...
21:28 like all these combinations, through "j", and
21:31 then the same for "bar". I haven't
21:35 really tested it... but yeah, we're getting all these
combinations that we
21:38 can "touch". And now, if we touch something
21:41 that is different between these two [directories], we
21:47 can again showcase the process substitution that we
saw
21:55 earlier. Say we want to check what files are different
between these
21:59 two folders. For us it's obvious, we just saw it, it's X and
Y,
22:03 but we can ask the shell to do this "diff" for us between
the
22:07 output of one LS and the other LS.
22:10 Unsurprisingly we're getting: X is
22:12 only in the first folder and Y is
22:14 only in the second folder. What is more
22:20 is, right now, we have only seen bash scripts. If you like
other
22:26 scripts, like for some tasks bash is probably not the
best,
22:30 it can be tricky. You can actually write scripts that
22:33 interact with the shell implemented in a lot
22:35 of different languages. So for example, let's see here a
22:39 Python script that has a magic line at the
22:43 beginning that I'm not explaining for now.
22:45 Then we have "import sys",
22:48 it's kind of like... Python is not, by default, trying to
interact
22:53 with the shell so you will have to import

about:blank 11/24
2023/4/21 10:42 For print

22:56 some library. And then we're doing a

22:58 really silly thing of just iterating
23:01 over "sys.argv[1:]".
23:06 "sys.argv" is kind of similar to what in bash we're getting
as $0, $1, &c.
23:12 Like the vector of the arguments, we're printing it in the
reversed order.
23:16 And the magic line at the beginning is
23:21 called a shebang and is the way that the
23:23 shell will know how to run this program.
23:26 You can always do something like
23:30 "python script.py", and then "a b c" and that
23:34 will work, always, like that. But
23:36 what if we want to make this to be
23:39 executable from the shell? The way the
23:41 shell knows that it has to use python as the
23:44 interpreter to run this file is using
23:48 that first line. And that first line is
23:52 giving it the path to where that thing lives.
23:58 However, you might not know.
23:59 Like, different machines will have probably
24:01 different places where they put python
24:04 and you might not want to assume where
24:06 python is installed, or any other interpreter.
24:08 So one thing that you can do is use the
24:16 "env" command.
24:18 You can also give arguments in the shebang, so
24:21 what we're doing here is specifying
24:23 run the "env" command, that is for pretty much every
system, there are some exceptions, but like for
24:29 pretty much every system it's is in
24:31 "usr/bin", where a lot of binaries live,
24:33 and then we're calling it with the
24:36 argument "python". And then that will make
24:38 use of the path environment variable
24:42 that we saw in the first lecture. It's
24:43 gonna search in that path for the Python
24:45 binary and then it's gonna use that to
24:48 interpret this file. And that will make
24:50 this more portable so it can be run in

about:blank 12/24
2023/4/21 10:42 For print

24:52 my machine, and your machine and some other

machine.
25:08 Another thing is that the bash is not
25:12 really like modern, it was
25:14 developed a while ago. And sometimes
25:16 it can be tricky to debug. By
25:18 default, and the ways it will fail
25:21 sometimes are intuitive like the way we
25:24 saw before of like foo command not
25:26 existing, sometimes it's not. So there's
25:28 like a really nifty tool that we have
25:31 linked in the lecture notes, which is called
25:34 "shellcheck", that will kind of give you
25:37 both warnings and syntactic errors
25:40 and other things that you might not have quoted
properly,
25:43 or you might have misplaced spaces in
25:46 your files. So for example for extremely simple "mcd.sh"
25:50 file we're getting a couple
25:51 of errors saying hey, surprisingly,
25:54 we're missing a shebang, like this
25:56 might not interpret it correctly if you're
25:59 it at a different system. Also, this
26:02 CD is taking a command and it might not
26:05 expand properly so instead of using CD
26:08 you might want to use something like CD
26:11 and then an OR and then an "exit". We go
26:14 back to what we explained earlier, what
26:16 this will do is like if the
26:18 CD doesn't end correctly, you cannot CD
26:21 into the folder because either you
26:23 don't have permissions, it doesn't exist...
26:25 That will give a nonzero error
26:28 command, so you will execute exit
26:32 and that will stop the script
26:33 instead of continue executing as if
26:35 you were in a place that you are
26:37 actually not in. And actually I haven't tested, but I
26:42 think we can check for "example.sh"
26:47 and here we're getting that we should be
26:50 checking the exit code in a different way, because it's
about:blank 13/24
2023/4/21 10:42 For print

26:55 probably not the best way, doing it this

26:57 way. One last remark I want to make
27:01 is that when you're writing bash scripts
27:05 or functions for that matter,
27:07 there's kind of a difference between
27:09 writing bash scripts in isolation like a
27:12 thing that you're gonna run, and a thing
27:14 that you're gonna load into your shell.
27:16 We will see some of this in the command
27:19 line environment lecture, where we will kind of
27:23 be tooling with the bashrc and the sshrc. But in general,
if you make
27:29 changes to for example where you are,
27:31 like if you CD into a bash script and you
27:34 just execute that bash script, it won't CD
27:36 into the shell are right now. But if you
27:39 have loaded the code directly into
27:42 your shell, for example you load...
27:45 you source the function and then you execute
27:48 the function then you will get those
27:50 side effects. And the same goes for
27:52 defining variables into the shell.
27:57 Now I'm going to talk about some tools that I think are
nifty when
28:03 working with the shell. The first was
28:07 also briefly introduced yesterday.
28:09 How do you know what flags, or like
28:13 what exact commands are. Like how I am
28:15 supposed to know that LS minus L will list the files in a
list format, or that
28:21 if I do "move - i", it's gonna like prom me
28:25 for stuff. For that what you have is the "man"
28:28 command. And the man command will kind of
28:30 have like a lot of information of how
28:33 will you go about... so for example here it
28:35 will explain for the "-i" flag, there are
28:40 all these options you can do. That's
28:43 actually pretty useful and it will work
28:45 not only for really simple commands that come
packaged with your OS

about:blank 14/24
2023/4/21 10:42 For print

28:51 but will also work with some tools that you install from
the internet
28:55 for example, if the person that did the
28:58 installation made it so that the man
29:01 package were also installed. So for example
29:03 a tool that we're gonna cover in a bit
29:06 which is called "ripgrep" and is called with RG, this didn't
29:12 come with my system but it has installed
29:14 its own man page and I have it here and
29:17 I can access it. For some commands the
29:21 man page is useful but sometimes it can be
29:25 tricky to decipher because it's more
29:28 kind of a documentation and a
29:30 description of all the things the tool
29:32 can do. Sometimes it will have
29:35 examples but sometimes not, and sometimes
29:37 the tool can do a lot of things so a
29:41 couple of good tools that I use commonly
29:45 are "convert" or "ffmpeg", which deal with images and
video respectively and
29:50 the man pages are like enormous. So there's
29:52 one neat tool called "tldr" that
29:54 you can install and you will have like
29:58 some nice kind of explanatory examples
30:02 of how you want to use this command. And you
30:05 can always Google for this, but I find
30:07 myself saving going into the
30:10 browser, looking about some examples and
30:12 coming back, whereas "tldr" are
30:14 community contributed and
30:16 they're fairly useful. Then,
30:19 the one for "ffmpeg" has a lot of
30:23 useful examples that are more nicely
30:24 formatted (if you don't have a huge
30:26 font size for recording). Or even
30:30 simple commands like "tar", that have a lot
30:33 of options that you are combining. So for
30:35 example, here you can be combining 2, 3...
30:37 different flags and it can not be
30:41 obvious, when you want to combine
30:43 different ones. That's how you
about:blank 15/24
2023/4/21 10:42 For print

30:48 would go about finding more about these tools. On the

topic of finding, let's try
30:54 learning how to find files. You can
30:58 always go "ls", and like you can go like
31:03 "ls project1", and
31:05 keep LS'ing all the way through. But
31:08 maybe, if we already know that we want
31:11 to look for all the folders called
31:15 "src", then there's probably a better command
31:19 for doing that. And that's "find".
31:21 Find is the tool that, pretty much comes with every UNIX
system. And find,
31:26 we're gonna give it... here we're saying we want to call
find in the
31:35 current folder, remember that "." stands
31:37 for the current folder, and we want the
31:40 name to be "src" and we want the type to be a directory.
And by typing that it's
31:46 gonna recursively go through the current
31:49 directory and look for all these files,
31:52 or folders in this case, that match this pattern. Find has
a lot of useful
31:58 flags. So for example, you can even test
32:01 for the path to be in a way. Here we're
32:05 saying we want some number of folders,
32:08 we don't really care how many folders,
32:09 and then we care about all the Python
32:13 scripts, all the things with the extension ".py", that are
within a
32:17 test folder. And we're also checking, just in
32:19 cases really but we're checking just
32:21 that it's also a type F, which stands for
32:24 file. We're getting all these files.
32:28 You can also use different flags for things
32:32 that are not the path or the name.
32:34 You could check things that have been
32:38 modified ("-mtime" is for the modification time), things
that have been
32:42 modified in the last day, which is gonna
32:44 be pretty much everything. So this is gonna print
32:46 a lot of the files we created and files
32:49 that were already there. You can even
about:blank 16/24
2023/4/21 10:42 For print

32:51 use other things like size, the owner,

32:54 permissions, you name it. What is even more
32:59 powerful is, "find" can find stuff
33:01 but it also can do stuff when you
33:04 find those files. So we could look for all
33:10 the files that have a TMP
33:14 extension, which is a temporary extension, and
33:18 then, we can tell "find" that for every one of those files,
33:22 just execute the "rm" command for them. And
33:26 that will just be calling "rm" with all
33:29 these files. So let's first execute it
33:32 without, and then we execute it with it.
33:35 Again, as with the command line
33:38 philosophy, it looks like nothing
33:41 happened. But since we have a zero error code,
something
33:48 happened - just that everything went
33:49 correct and everything is fine. And now,
33:51 if we look for these files, they aren't there anymore.
33:57 Another nice thing about the shell in general is that
there are
34:02 these tools, but people will keep
34:05 finding new ways, so alternative
34:08 ways of writing these tools. It's nice to know about it. So,
for
34:12 example find if you just want to match the things that
end in "tmp"
34:20 it can be sometimes weird to do this thing, it has a long
command.
34:24 There's things like "fd",
34:27 for example, that is a shorter command that by default
will use regex
34:32 and will ignore your gitfiles, so you
34:34 don't even search for them. It
34:38 will color-code, it will have better Unicode support... It's
nice to
34:42 know about some of these tools. But, again,
34:45 the main idea is that if you are aware that these tools
exist, you can
34:52 save yourself a lot of time from doing
34:53 kind of menial and repetitive tasks.
34:57 Another command to bear in mind is like

about:blank 17/24
2023/4/21 10:42 For print

35:00 "find". Some of you may be

35:01 wondering, "find" is probably just
35:04 actually going through a directory
35:06 structure and looking for things but
35:09 what if I'm doing a lot of "finds" a day?
35:11 Wouldn't it be better, doing kind of
35:12 a database approach and build an index first, and then
use that index
35:18 and update it in some way. Well, actually
35:21 most Unix systems already do it and
35:23 this is through the "locate" command and
35:28 the way that the locate will
35:31 be used... it will just look for paths in
35:35 your file system that have the substring
35:38 that you want. I actually don't know if it will work... Okay,
it worked. Let me try to
35:44 do something like "missing-semester".
35:51 You're gonna take a while but
35:53 it found all these files that are somewhere
35:56 in my file system and since it has
35:57 built an index already on them, it's much
36:01 faster. And then, to keep it updated,
36:05 using the "updatedb" command that is running through
cron,
36:13 to update this database. Finding files, again, is
36:18 really useful. Sometimes you're actually concerned
about, not the files themselves,
36:23 but the content of the files. For that
36:26 you can use the grep command that we
36:31 have seen so far. So you could do
36:33 something like grep foobar in MCD, it's there.
36:37 What if you want to, again, recursively search through
the current
36:43 structure and look for more files, right?
36:45 We don't want to do this manually.
36:48 We could use "find", and the "-exec", but
36:51 actually "grep" has the "-R" flag that will go through the
entire
36:58 directory, here. And it's telling us
37:03 that oh we have the foobar line in example.sh
37:06 at these three places and in
37:09 this other two places in foobar. This can be
about:blank 18/24
2023/4/21 10:42 For print

37:14 really convenient. Mainly, the

37:16 use case for this is you know you have
37:18 written some code in some programming
37:21 language, and you know it's somewhere in
37:23 your file system but you actually don't
37:26 know. But you can actually quickly search.
37:28 So for example, I can quickly search
37:35 for all the Python files that I have in my
37:40 scratch folder where I used the request library.
37:45 And if I run this, it's giving me
37:47 through all these files, exactly in
37:50 what line it has been found. And here
37:53 instead of using grep, which is fine,
37:56 you could also do this, I'm using "ripgrep",
37:58 which is kind of the same idea but again trying to bring
some more
38:05 niceties like color coding or file
38:09 processing and other things. It think it has, also, unicode
support. It's also pretty
38:16 fast so you are not paying like a trade-off on this being
slower and
38:22 there's a lot of useful flags. You
38:25 can say, oh, I actually want to get some
38:27 context around those results.
38:33 So I want to get like five lines of context around
38:36 that, so you can see where that import lives and see
code around it.
38:42 Here in the import it's not really useful
38:44 but like if you're looking for where you
38:45 use the function, for example, it will
38:49 be very handy. We can also do things like
38:54 we can search, for example here,.
38:59 A more advanced use, we can say,
39:04 "-u" is for don't ignore hidden files, sometimes
39:12 you want to be ignoring hidden files, except if you want
to
39:16 search config files, that are by default hidden. Then,
instead of printing
39:23 the matches, we're asking to do something that would
be kind of hard, I think,
39:28 to do with grep, out of my head, which is
39:31 "I want you to print all the files that

about:blank 19/24
2023/4/21 10:42 For print

39:34 don't match the pattern I'm giving you", which

39:37 may be a weird thing to ask here but
39:40 then we keep going... And this pattern here
39:42 is a small regex which is saying
39:45 at the beginning of the line I have a
39:48 "#" and a "!", and that's a shebang.
39:51 Like that, we're searching here for all
39:53 the files that don't have a shebang
39:56 and then we're giving it, here,
39:59 a "-t sh" to only look for "sh"
40:02 files, because maybe all your Python or text files are
fine
40:07 without a shebang. And here it's telling us
40:10 "oh, MCD is obviously missing a shebang"
40:14 We can even... It has like some
40:16 nice flags, so for example if we
40:19 include the "stats" flag
40:28 it will get all these results but it will also tell us
information about all
40:34 the things that it searched. For example,
40:35 the number of matches that it found, the lines, the file
searched,
40:40 the bytes that it printed, &c.
40:44 Similar as with "fd", sometimes it's not as useful
40:48 using one specific tool or another and
40:50 in fact, as ripgrep, there are several other tools. Like
"ack",
40:55 is the original grep alternative that was
40:57 written. Then the silver searcher,
41:00 "ag", was another one... and they're all
41:04 pretty much interchangeable so
41:05 maybe you're at a system that has one and
41:07 not the other, just knowing that you can
41:09 use these things with these tools can be
41:12 fairly useful. Lastly, I want to cover
41:15 how you go about, not finding files or code, but how you
go about
41:19 finding commands that you already
41:22 some time figured out. The first, obvious way is just
using the up arrow,
41:30 and slowly going through all your history, looking for
these matches.

about:blank 20/24
2023/4/21 10:42 For print

41:34 This is actually not very efficient, as

41:36 you probably guessed. So the bash has ways to do this
more easily.
41:42 There is the "history" command, that will
41:44 print your history. Here I'm in zsh and it only prints some
of my history, but
41:49 if I say, I want you to print everything from the beginning
of time, it will print
41:54 everything from the beginning of whatever this history is.
41:58 And since this is a lot of results,
42:00 maybe we care about the ones where we
42:02 use the "convert" command to go from some type of file
to some other type of file.
42:08 Some image, sorry. Then, we're getting all
42:12 these results here, about all the ones
42:15 that match this substring.
42:21 Even more, pretty much all shells by default will
42:24 link "Ctrl+R", the keybinding,
42:27 to do backward search. Here we
42:29 have backward search, where we can
42:31 type "convert" and it's finding the
42:34 command that we just typed. And if we just
42:36 keep hitting "Ctrl+R", it will
42:38 kind of go through these matches and
42:41 it will let re-execute it
42:44 in place. Another thing that you can do,
42:49 related to that, is you can use this
42:51 really nifty tool called "fzf", which is
42:53 like a fuzzy finder, like it will...
42:57 It will let you do kind of
42:58 like an interactive grep. We could do
43:02 for example this, where we can cat our
43:06 example.sh command, that will print
43:10 print to the standard output, and then we
43:11 can pipe it through fzf. It's just getting
43:14 all the lines and then we can interactively look for the
43:18 string that we care about. And the nice
43:21 thing about fzf is that, if you enable the default bindings,
it will bind to
43:26 your "Ctrl+R" shell execution and now
43:33 you can quickly and dynamically like

about:blank 21/24
2023/4/21 10:42 For print

43:36 look for all the times you try to convert a favicon in your
history.
43:42 And it's also like fuzzy matching, whereas like by default
in grep
43:46 or these things you have to write a regex or some
43:49 expression that will match within here.
43:52 Here I'm just typing "convert" and "favicon" and
43:54 it's just trying to do the best scan,
43:57 doing the match in the lines it has.
44:01 Lastly, a tool that probably you have already seen, that
I've been using
44:06 for not retyping these extremely long
44:08 commands is this "history substring search", where
44:13 as I type in my shell,
44:15 and both F fail to mention but both face
44:19 which I think was originally introduced, this concept, and
then
44:22 zsh has a really nice implementation)
44:25 what it'll let you do is
44:26 as you type the command, it will dynamically search
back in your
44:31 history to the same command that has a common prefix,
44:34 and then, if you...
44:39 it will change as the match list stops
44:42 working and then as you do the
44:44 right arrow you can select that command and then re-
execute it.
45:05 We've seen a bunch of stuff... I think I have
45:09 a few minutes left so I'm going to cover a couple of tools
to do
45:16 really quick directory listing and directory navigation.
45:20 So you can always use the "-R" to recursively list some
directory structure,
45:30 but that can be suboptimal, I cannot really make sense
of this easily.
45:36 There's tool called "tree" that will be the much more
friendly form of
45:44 printing all the stuff, it will also color code based on...
45:47 here for example "foo" is blue because it's a directory
and
45:50 this is red because it has execute permissions.
45:55 But we can go even further than that. There's really nice
tools

about:blank 22/24
2023/4/21 10:42 For print

46:00 like a recent one called "broot" that will do the same
thing but here
46:04 for example instead of doing this thing of listing
46:07 every single file, for example in bar
46:09 we have these "a" through "j" files,
46:11 it will say "oh there are more, unlisted here".
46:15 I can actually start typing and it will again
46:18 again facily match to the files that are there
46:21 and I can quickly select them and navigate through
them.
46:24 So, again, it's good to know that
46:28 these things exist so you don't lose a large amount of
time
46:34 going for these files.
46:37 There are also, I think I have it installed
46:40 also something more similar to what you would expect
your OS to have,
46:44 like Nautilus or one of the Mac finders that have like an
46:49 interactive input where you can just use your navigation
arrows and quickly explore.
46:59 It might be overkill but you'll be surprised how quickly
you can
47:03 make sense of some directory structure by just
navigating through it.
47:07 And pretty much all of these tools will let you edit, copy
files...
47:12 if you just look for the options for them.
47:17 The last addendum is kind of going places.
47:20 We have "cd", and "cd" is nice, it will get you
47:26 to a lot of places. But it's pretty handy if
47:30 you can like quickly go places,
47:33 either you have been to recently or that
47:36 you go frequently. And you can do this in
47:40 many ways there's probably... you can start
47:42 thinking, oh I can make bookmarks, I can
47:44 make... I can make aliases in the shell,
47:46 that we will cover at some point,
47:49 symlinks... But at this point,
47:53 programmers have like built all these
47:54 tools, so programmers have already figured
47:56 out a really nice way of doing this.
47:59 One way of doing this is using what is

about:blank 23/24
2023/4/21 10:42 For print

48:01 called "auto jump", which I think is not loaded here...

48:14 Okay, don't worry. I will cover it in the command line
environment.
48:21 I think it's because I disabled the "Ctrl+R" and that also
48:25 affected other parts of the script. I think at this point if
anyone has
48:31 any questions that are related to this, I'll be more than
happy to answer
48:35 them, if anything was left unclear.
48:37 Otherwise, a there's a bunch of exercises that we wrote,
kind of
48:42 touching on these topics and we encourage you to try
them and
48:46 come to office hours, where we can help
48:48 you figure out how to do them, or some bash quirks that
are not clear.

about:blank 24/24