set -e, -u, -o, -x pipefail explanation · GitHub

gist.github.com/mohanpedala/1e2ff5661761d3abd0385e8223e16425

set -e, -u, -o, -x pipefail explanation

Table of Contents

set -e, -u, -x, -o pipefail

The set lines

These lines deliberately cause your script to fail. Wait, what?
Believe me, this is a good thing.

With these settings, certain common errors will cause the script to
immediately fail, explicitly and loudly. Otherwise, you can get hidden
bugs that are discovered only when they blow up in production.

set -euxo pipefail is short for:

set -e
set -u
set -o pipefail
set -x

set -e

The set -e option instructs bash to immediately exit if any command [1]
has a non-zero exit status. You wouldn't want to set this for your
command-line shell, but in a script it's massively helpful. In all widely used
general-purpose programming languages, an unhandled runtime error

1/15
 whether that's a thrown exception in Java, or a segmentation fault in C, or
a syntax error in Python - immediately halts execution of the program;
subsequent lines are not executed.

By default, bash does not do this. This default behavior is exactly

what you want if you are using bash on the command line
you don't want a typo to log you out! But in a script, you really want
the opposite.
If one line in a script fails, but the last line succeeds, the whole
script has a successful exit code. That makes it very easy to miss
the error.
Again, what you want when using bash as your command-line shell
and using it in scripts are at odds here. Being intolerant of errors is
a lot better in scripts, and that's what set -e gives you.

set -x

Enables a mode of the shell where all executed commands are printed to
the terminal. In your case it's clearly used for debugging, which is a
typical use case for set -x : printing every command as it is executed may
help you to visualize the control flow of the script if it is not functioning as
expected.

set -u

Affects variables. When set, a reference to any variable you haven't

previously defined - with the exceptions of ∗and@ - is an error, and
causes the program to immediately exit. Languages like Python, C, Java
and more all behave the same way, for all sorts of good reasons. One is
so typos don't create new variables without you realizing it. For example:

#!/bin/bash
firstName="Aaron"
fullName="$firstname Maxwell"
echo "$fullName"

Take a moment and look. Do you see the error? The right-hand side of
the third line says "firstname", all lowercase, instead of the camel-cased
"firstName". Without the -u option, this will be a silent error. But with the -u
option, the script exits on that line with an exit code of 1, printing the
message "firstname: unbound variable" to stderr.

This is what you want: have it fail explicitly and immediately, rather than
create subtle bugs that may be discovered too late.

2/15
set -o pipefail

This setting prevents errors in a pipeline from being masked. If any

command in a pipeline fails, that return code will be used as the return
code of the whole pipeline. By default, the pipeline's return code is that of
the last command even if it succeeds. Imagine finding a sorted list of
matching lines in a file:

$ grep some-string /non/existent/file | sort

grep: /non/existent/file: No such file or directory
% echo $?
0

Here, grep has an exit code of 2, writes an error message to stderr, and
an empty string to stdout.

This empty string is then passed through sort, which happily accepts it as
valid input, and returns a status code of 0.

This is fine for a command line, but bad for a shell script: you almost
certainly want the script to exit right then with a nonzero exit code... like
this:

$ set -o pipefail
$ grep some-string /non/existent/file | sort
grep: /non/existent/file: No such file or directory
$ echo $?
2

Setting IFS

3/15
The IFS variable - which stands for Internal Field Separator - controls
what Bash calls word splitting. When set to a string, each character in the
string is considered by Bash to separate words. This governs how bash
will iterate through a sequence. For example, this script:

#!/bin/bash
IFS=$' '
items="a b c"
for x in $items; do
echo "$x"
done

IFS=$'\n'
for y in $items; do
echo "$y"
done
... will print out this:

a
b
c
a b c

In the first for loop, IFS is set to ″.(The'...' syntax creates a string, with
backslash-escaped characters replaced with special characters - like "\t"
for tab and "\n" for newline.) Within the for loops, x and y are set to
whatever bash considers a "word" in the original sequence.

For the first loop, IFS is a space, meaning that words are separated by a
space character.

For the second loop, "words" are separated by a newline, which means
bash considers the whole value of "items" as a single word. If IFS is more
than one character, splitting will be done on any of those characters.

4/15
 Got all that? The next question is, why are we setting IFS to a string
consisting of a tab character and a newline? Because it gives us better
behavior when iterating over a loop. By "better", I mean "much less likely
to cause surprising and confusing bugs". This is apparent in working with
bash arrays:

#!/bin/bash
names=(
"Aaron Maxwell"
"Wayne Gretzky"
"David Beckham"
)

echo "With default IFS value..."

for name in ${names[@]}; do
echo "$name"
done

echo ""
echo "With strict-mode IFS value..."
IFS=$'\n\t'
for name in ${names[@]}; do
echo "$name"
done

## Output
With default IFS value...
Aaron
Maxwell
Wayne
Gretzky
David
Beckham

With strict-mode IFS value...

Aaron Maxwell
Wayne Gretzky
David Beckham

Or consider a script that takes filenames as command line arguments:

```
for arg in $@; do
echo "doing something with file: $arg"
done
```

5/15
 If you invoke this as myscript.sh notes todo-list 'My Resume.doc', then
with the default IFS value, the third argument will be mis-parsed as two
separate files - named "My" and "Resume.doc". When actually it's a file
that has a space in it, named "My Resume.doc".

Which behavior is more generally useful? The second, of course - where

we have the ability to not split on spaces. If we have an array of strings
that in general contain spaces, we normally want to iterate through them
item by item, and not split an individual item into several.

Setting IFS to $'\n\t' means that word splitting will happen only on
newlines and tab characters. This very often produces useful splitting
behavior.

By default, bash sets this to $' \n\t' - space, newline, tab - which is too
eager.

Original Reference
Maxwell, A. Unofficial-bash-strict-mode. Retrieved 2018, from
http://redsymbol.net/articles/unofficial-bash-strict-mode/

berkant commented May 14, 2021 • edited

set -euo pipefail does not include -x, which you said shorthand for the following lines of
individual commands that wrongly includes -x as well. Change to -euxo pipefail?

mohanpedala commented May 16, 2021 • edited

set -euo pipefail does not include -x, which you said shorthand for the following lines
of individual commands that wrongly includes -x as well. Change to -euxo pipefail?

thank you added it..

6/15
awolad commented Jul 1, 2021

I'm getting the error from Github actions.

./scripts/deploy.sh: 2: set: Illegal option -o pipefail

berkant commented Jul 1, 2021 via email • edited

@awolad use bash instead of dash (sh). check your shebang line.

awolad commented Jul 1, 2021 • edited

@berkant Previously I was executing the script using sh ./scripts/deploy.sh Now it's
working using bash ./scripts/deploy.sh Thank you so much :)

cpauya commented Aug 18, 2021

Thanks very much! The -u and -o pipefail are new to me.

🙏😺
Just a trivial one: maybe change title/heading from set -e, -u, -o, -x pipefail to set -e, -u, -x, -o
pipefail to be more consistent/correct?

leiless commented Aug 24, 2021

Thanks very much! The -u and -o pipefail are new to me.

Just a trivial one: maybe change title/heading from set -e, -u, -o, -x pipefail to set -e, -u,
-x, -o pipefail to be more consistent/correct?

7/15
Upvote!

mohanpedala commented Aug 24, 2021

Thanks very much! The -u and -o pipefail are new to me.

Just a trivial one: maybe change title/heading from set -e, -u, -o, -x pipefail to set -
e, -u, -x, -o pipefail to be more consistent/correct?

Upvote!

I have chnaged it thank you

thingsiplay commented Sep 15, 2021 • edited

Previously I was executing the script using sh ./scripts/deploy.sh Now it's working
using bash ./scripts/deploy.sh Thank you so much :)

I am a bit late, but just found this wonderful tutorial. And by accident I just found in the man
bash that using bash as sh would put it into strict POSIX mode, which is similar to dash and
does not support any of the bash features. So if you have bash set as your sh interpreter and
use the sh command, then it goes into this strict mode. That is why it did not work. This is the
line from the manual in question:

If bash is invoked with the name sh, it tries to mimic the startup behavior of historical
versions of sh as closely as possible, while conforming to the POSIX standard as well.

bkahlert commented Oct 20, 2021

Nice that this gist receives so much positive feedback. I just ask myself if all that gratitude
shouldn't be better directed to Aaron Maxwell, the author of the original article to be found at
http://redsymbol.net/articles/unofficial-bash-strict-mode/.
This gist is close to a 1:1 copy.
I recommend to original article for reading as it also describes what to do in cases were the
"strict mode" causes problems.

cvega21 commented Oct 22, 2021

8/15
thank you Aaron Maxwell for this, and thank you mohan for compiling it here! very clear
explanations and super useful knowledge.

kayomarz commented Oct 26, 2021

Very useful gist but as @bkahlert pointed out its almost a copy of the original author
@redsymbol who deserves a mention.

Nice that this gist receives so much positive feedback. I just ask myself if all that gratitude
shouldn't be better directed to Aaron Maxwell, the author of the original article to be
found at http://redsymbol.net/articles/unofficial-bash-strict-mode/. This gist is close to a
1:1 copy. I recommend to original article for reading as it also describes what to do in
cases were the "strict mode" causes problems.

mohanpedala commented Oct 26, 2021 • edited

Nice that this gist receives so much positive feedback. I just ask myself if all that gratitude
shouldn't be better directed to Aaron Maxwell, the author of the original article to be
found at http://redsymbol.net/articles/unofficial-bash-strict-mode/. This gist is close to a
1:1 copy. I recommend to original article for reading as it also describes what to do in
cases were the "strict mode" causes problems.

Hi @bkahlert , Original reference has been added at the bottom of the gist when I created it
("Full Reference Click Here"). Please read the full gist before pointing out. and for your
reference I keep my daily notes and findings as a gist.
I like to see most of the content in markdown so I have created the content I need in the
markdown version. I really appreciate Aaron Maxwell article that is the reason why I did-not
change the names or content in the gist.

mohanpedala commented Oct 26, 2021 • edited

9/15
 Very useful gist but as @bkahlert pointed out its almost a copy of the original author
@redsymbol who deserves a mention.

Nice that this gist receives so much positive feedback. I just ask myself if all that
gratitude shouldn't be better directed to Aaron Maxwell, the author of the original
article to be found at http://redsymbol.net/articles/unofficial-bash-strict-mode/. This
gist is close to a 1:1 copy. I recommend to original article for reading as it also
describes what to do in cases were the "strict mode" causes problems.

Hi @kayomarz Original reference has been added at the bottom of the gist when I created it
("Full Reference Click Here"). Please read the full gist before pointing out. and for your
reference I keep my daily notes and findings as a gist.
I like to see most of the content in markdown so I have created the content I need in the
markdown version. I really appreciate Aaron Maxwell article that is the reason why I did-not
change the names or content in the gist.

bkahlert commented Oct 29, 2021

Very useful gist but as @bkahlert pointed out its almost a copy of the original
author @redsymbol who deserves a mention.

Nice that this gist receives so much positive feedback. I just ask myself if all
that gratitude shouldn't be better directed to Aaron Maxwell, the author of the
original article to be found at http://redsymbol.net/articles/unofficial-bash-
strict-mode/. This gist is close to a 1:1 copy. I recommend to original article
for reading as it also describes what to do in cases were the "strict mode"
causes problems.

Hi @kayomarz Original reference has been added at the bottom of the gist when I
created it ("Full Reference Click Here"). Please read the full gist before pointing out.
and for your reference I keep my daily notes and findings as a gist. I like to see most of
the content in markdown so I have created the content I need in the markdown version. I
really appreciate Aaron Maxwell article that is the reason why I did-not change the names
or content in the gist.

Hi @mohanpedala, I did read your gist completely and well noticed your "Full Reference Click
Here" link. That's what made me comment. I consider such a generic reference to a 1:1 copied
article inappropriate. Especially in the light of its perceived value. You might want to read
Citation styles guide: Choosing a style and citing correctly.

10/15
mohanpedala commented Oct 29, 2021 • edited

Very useful gist but as @bkahlert pointed out its almost a copy of the original
author @redsymbol who deserves a mention.

Nice that this gist receives so much positive feedback. I just ask myself
if all that gratitude shouldn't be better directed to Aaron Maxwell, the
author of the original article to be found at
http://redsymbol.net/articles/unofficial-bash-strict-mode/. This gist is
close to a 1:1 copy. I recommend to original article for reading as it also
describes what to do in cases were the "strict mode" causes problems.

Hi @kayomarz Original reference has been added at the bottom of the gist when I
created it ("Full Reference Click Here"). Please read the full gist before pointing
out. and for your reference I keep my daily notes and findings as a gist. I like to see
most of the content in markdown so I have created the content I need in the
markdown version. I really appreciate Aaron Maxwell article that is the reason why I
did-not change the names or content in the gist.

Hi @mohanpedala, I did read your gist completely and well noticed your "Full Reference
Click Here" link. That's what made me comment. I consider such a generic reference to a
1:1 copied article inappropriate. Especially in the light of its perceived value. You might
want to read Citation styles guide: Choosing a style and citing correctly.

Thank you @bkahlert I hope this looks good 😊

bkahlert commented Nov 11, 2021

Wonderful

Gboom345 commented Mar 14, 2022

very good explanation

bolshakoff commented Mar 30, 2022

Nice.

11/15
fazlearefin commented Apr 1, 2022

@mohanpedala Consider using -E as well as mentioned in

https://vaneyckt.io/posts/safer_bash_scripts_with_set_euxo_pipefail/ ?

So it becomes

set -Eeuxo pipefail

patrikjuvonen commented Apr 6, 2022 • edited

https://blog.cloudflare.com/pipefail-how-a-missing-shell-option-slowed-cloudflare-down/

Interesting real-life scenario post about how missing the pipefail option caused a major
cascading failure in Cloudflare's production.

RimElbaddazi commented May 3, 2022

i didn't get : set -o pipefail

icntrs commented Jun 16, 2022 • edited

Little tinny thing :
$ grep some-string /non/existent/file | sort
grep: /non/existent/file: No such file or directory
% echo $? --> $ echo $?
0

very useful info, thanks :)

LukeSavefrogs commented Oct 6, 2022 • edited

This has already been said by others (like @jhult), but i'll leave my 2 cents here...

12/15
I suggest you not to use -e without knowing its side effects.

The problem
I had a script which used the arithmetic expansion to increase a counter through the
((counter++)) syntax. It kept exiting without ever reaching the end, and it seemed it failed
always at the same point. So i put set -x inside the function that was the cause and
discovered that the culprit was this line: ((variable++)).

Well... Turns out that arithmetic expansion returns the number, so when it returns 1 (error
return code) it caused the entire script to stop:

$ i=-1;

$ ((i++)) && echo Succeed || echo Fail; echo $i;

Succeed
0

$ ((i++)) && echo Succeed || echo Fail; echo $i;

Fail
1

$ ((i++)) && echo Succeed || echo Fail; echo $i;

Succeed
2

If i use the ((++i)) construct, instead:

$ i=-1;

$ ((++i)) && echo Succeed || echo Fail; echo $i;

Fail
0

$ ((++i)) && echo Succeed || echo Fail; echo $i;

Succeed
1

$ ((++i)) && echo Succeed || echo Fail; echo $i;

Succeed
2

The same goes for ((i+=1)):

13/15
$ i=-1;

$ ((i+=1)) && echo Succeed || echo Fail; echo $i;

Fail
0

$ ((i+=1)) && echo Succeed || echo Fail; echo $i;

Succeed
1

Sure, there are other ways to do arithmetic operations that don't trigger the error, but this is
something to keep in mind..

Considerations
You need to be extra careful when using set -e, since it can lead to all kind of unexpected
results if you use constructs you don't fully know.
It is good to use while testing, since it is a more "destructive" way to test your scripts, but i
wouldn't use it in production.
Instead of relying on set -e to exit on errors, learn to catch errors by checking the return code
and exiting from the script where and when you want it to

frankmalin commented Dec 1, 2022

@LukeSavefrogs thanks, I had encountered the failure on my loop, and got past the issue w/
|| true and never understood the root cause (which when explained makes logical sense).

LukeSavefrogs commented Dec 1, 2022 • edited

@LukeSavefrogs thanks, I had encountered the failure on my loop, and got past the
issue w/ || true and never understood the root cause (which when explained makes
logical sense).

Happy it was useful to someone else too 😃

You're right, i forgot to add that you can "bypass" this behaviour when using set -e by adding
|| true after the arithmetial operation:

((i+=1)) || true

Thanks for pointing that out 👍

14/15
ernstki commented Aug 15, 2023 • edited

There are valid use cases for mucking with IFS. But I have mixed feelings about the argument
made for setting IFS=$'\n\t', in the original blog post1 that served as raw material for this
Gist.

For one thing, it breaks the normal (and very useful) behavior of "${array[*]}", which yields
all elements of array as a single string, separated by the first character of IFS; by default, a
space. You could be forgiven for messing with IFS if you weren't aware of this behavior, and
maybe the blog post's author wasn't at that time.

As for preserving whitespace within array elements, that use case is better accommodated by
simply double-quoting your variables when they're expanded:

array=("the rain" "in Spain" lies mainly "on the plain")

# "${array[@]}" yields each array element individually double-quoted

for var in "${array[@]}"; do
touch "$var.txt" # or "${var}.txt"
done

Result:

$ ls -1 *.txt
in Spain.txt
lies.txt
mainly.txt
on the plain.txt
the rain.txt

Double-quoting variable expansions is a Bash scripting "best practice" anyway—at least until
you're proficient enough to understand the few times when it's not required.

The downside is it requires a bit more discipline on the part of you, the programmer. The
upside is you will write more robust code, and won't be scratching your head when you happen
upon a filename that contains a literal tab or newline, which would otherwise crash your script.

Sep 17, 2023

15/15