Writing Safer Bash

April 8th, 2019

Bash scripts are a really convenient way to write simple utilities. Unfortunately many bash scripts in the wild are littered with bugs. Writing reliable bash can be hard. I’ve been reviewing and fixing a lot of bash while working on cleaning up the Kubernetes project’s scripts and wanted to collect some tips for writing more reliable scripts.

# Use ShellCheck

ShellCheck is an excellent open source linter for shell capable of detecting many errors.

Many editors have shellcheck extensions, personally I like the VSCode extension.

Running your scripts through ShellCheck is a great way to catch many common bugs. We opted to turn off a few of the lints when testing Kubernetes pull requests, but generally these are useful and worth fixing.

# Enable Error Handling Options

Add the following snippet to the beginning of your scripts:

1# enable common error handling options
2set -o errexit
3set -o nounset
4set -o pipefail

• set -o errexit causes the script to exit immediately if a pipeline sets a non-zero status (exit code). This provides a basic mechanism to ensure that scripts fail unless you intended to ignore a failure. You can still intentionally ignore errors like: some_failing_command || true

• set -o nounset causes unset variables to be treat as errors and cause an exit. This will help prevent using variables that were never assigned.

• set -o pipefail causes pipelines to retain / set hte last non-zero status. Normally a pipeline will set the exit code of the last command instead. Without setting this, set -o errexit may have surprising behavior like:

1# if foo_fails exits non-zero but bar_does_not_fail exits 0, this will not
2# trigger errexit unless pipefail is also set
3foo_fails | bar_does_not_fail

For more on these, read the bash manual page for the set builtin.

# Avoid Common Gotchas

Bash has a few quirks that tend to be surprising to developers used to other languages. These are a few common ones to be aware of.

# export and local always succeed

Though ShellCheck will capture this one (see SC2155 for more details), it seems to be common enough and misunderstood enough that I’m calling it out here.

Do this:

1FOO=$(compute_foo)
2export FOO

Not this:

1export FOO=$(compute_foo)

Many people seem to be surprised to learn that export always sets the status to zero for the above line (I know I was!), meaning that even if compute_foo fails with a non-zero status, the line will succeed and errexit will not be triggered.

This also applies to local and readonly, which should be written like:

1local foo
2foo=$(compute_foo)

and like:

1foo=$(compute_foo)
2readonly foo

# local is not so local

Unlike local variables in many other popular programming languages, local variables in bash are visible to called functions. In other words, if function bar declares local foo and then calls a function echo_foo, echo_foo will be able to read the value of foo, whereas bar’s caller will not be able to read foo.

Keep this behavior in mind when using local. For stronger isolation, consider executing child calls in a subshell like (echo_foo).

 1# `local` behavior example
 2
 3echo_foo(){
 4    echo "${foo}"
 5}
 6
 7bar() {
 8    local foo="foo"
 9    echo_foo
10}
11
12# prints `foo`
13# locals are available to child functions
14bar
15
16# prints nothing (or errors with nounset)
17# locals are not available to the caller
18echo "${foo}" 
19
20# prints nothing (or errors with nounset)
21# locals are not visible to child processes
22(echo_foo)

# Quoting is Hard

Quoting and parameter expansion ("${FOO[@]}") behavior can be surprising.

Don’t make assumptions about how strings and expansion work in bash, read the manuals instead. Incorrect quoting and parameter expansion seem to be common sources of bugs.

For this specifically I recommend reading:

• The bash shell expansion and quoting docs
• ShellCheck’s entry on double quoting and word splitting: SC2086

# macOS ships old bash

Apple still ships an old 3.X version of Bash in macOS. Meanwhile Bash 5.0 released in January.

Several useful features (mapfile, associative arrays, …) are not present or are more poorly behaved in Bash 3. If you intend to support macOS, be prepared to be painfully aware of Mac’s default shell.

For some of our scripts we simply check for and require a more recent bash version, which may be installed with Homebrew. This is easy to check with $BASH_VERSINFO:

 1# Check bash version
 2if ((${BASH_VERSINFO[0]}<4)); then
 3  echo
 4  echo "ERROR: Bash v4+ required."
 5  # Extra help for OSX
 6  if [[ "$(uname -s)" == "Darwin" ]]; then
 7    echo
 8    echo "Ensure you are up to date on the following packages:"
 9    echo "$ brew install md5sha1sum bash jq"
10  fi
11  echo
12  exit 9
13fi

(Excerpt from Kubernetes hack/update-vendor-licenses.sh)

It is also worth noting that while most Linux distros ship GNU coreutils or something reasonably compatible, macOS ships many *BSD derived utilities. sed differences alone have led to many scripts not being portable. Again we simply require GNU sed in some cases and ask the user to install this with Homebrew.

# Recommended Reading

This post only touched on a few common problems, to write safer Bash I recommend reading these general references to understand it better:

• The GNU Bash Reference Manual (not surprisingly, the official reference is helpful)
• The Linux Documentation Project’s Advanced Bash-Scripting Guide.
• The Bash Hackers Wiki

You may also want to read Google’s Shell Style Guide.

# Final Note

Consider writing non-trivial utilities in another lanuage (or don’t! bash is great!) :^)

Bash is a fun, useful, and easy to distribute. It is also difficult to write complex, reliable programs in. The Google Shell Style Guide has some fairly reasonable recommendations about when to use Bash.

While you could probably implement your entire app in bash, it might not be the best idea ¹ :bash_fire: