BenTheElder

Writing Safer Bash

:bash_fire::bash:

April 8th, 2019


Bash scripts are a really convenient way to write simple utilities. Unfortunately many bash scripts in the wild are littered with bugs. :bash_fire: Writing reliable bash can be hard. I’ve been reviewing and fixing a lot of bash while working on cleaning up the Kubernetes project’s scripts and wanted to collect some tips for writing more reliable scripts.

BASH The Bourne-Again Shell

Use ShellCheck 🔗︎

ShellCheck is an excellent open source linter for shell capable of detecting many errors.

Many editors have shellcheck extensions, personally I like the VSCode extension.

Running your scripts through ShellCheck is a great way to catch many common bugs. We opted to turn off a few of the lints when testing Kubernetes pull requests, but generally these are useful and worth fixing.

Enable Error Handling Options 🔗︎

Add the following snippet to the beginning of your scripts:

1# enable common error handling options
2set -o errexit
3set -o nounset
4set -o pipefail

For more on these, read the bash manual page for the set builtin.

Avoid Common Gotchas 🔗︎

Bash has a few quirks that tend to be surprising to developers used to other languages. These are a few common ones to be aware of.

export and local always succeed 🔗︎

Though ShellCheck will capture this one (see SC2155 for more details), it seems to be common enough and misunderstood enough that I’m calling it out here.

Do this:

1FOO=$(compute_foo)
2export FOO

Not this:

1export FOO=$(compute_foo)

Many people seem to be surprised to learn that export always sets the status to zero for the above line (I know I was!), meaning that even if compute_foo fails with a non-zero status, the line will succeed and errexit will not be triggered.

This also applies to local and readonly, which should be written like:

1local foo
2foo=$(compute_foo)

and like:

1foo=$(compute_foo)
2readonly foo

local is not so local 🔗︎

Unlike local variables in many other popular programming languages, local variables in bash are visible to called functions. In other words, if function bar declares local foo and then calls a function echo_foo, echo_foo will be able to read the value of foo, whereas bar’s caller will not be able to read foo.

Keep this behavior in mind when using local. For stronger isolation, consider executing child calls in a subshell like (echo_foo).

 1# `local` behavior example
 2
 3echo_foo(){
 4    echo "${foo}"
 5}
 6
 7bar() {
 8    local foo="foo"
 9    echo_foo
10}
11
12# prints `foo`
13# locals are available to child functions
14bar
15
16# prints nothing (or errors with nounset)
17# locals are not available to the caller
18echo "${foo}" 
19
20# prints nothing (or errors with nounset)
21# locals are not visible to child processes
22(echo_foo)

Quoting is Hard 🔗︎

Quoting and parameter expansion ("${FOO[@]}") behavior can be surprising.

Don’t make assumptions about how strings and expansion work in bash, read the manuals instead. Incorrect quoting and parameter expansion seem to be common sources of bugs.

For this specifically I recommend reading:

macOS ships old bash 🔗︎

Apple still ships an old 3.X version of Bash in macOS. Meanwhile Bash 5.0 released in January.

Several useful features (mapfile, associative arrays, …) are not present or are more poorly behaved in Bash 3. If you intend to support macOS, be prepared to be painfully aware of Mac’s default shell.

For some of our scripts we simply check for and require a more recent bash version, which may be installed with Homebrew. This is easy to check with $BASH_VERSINFO:

 1# Check bash version
 2if ((${BASH_VERSINFO[0]}<4)); then
 3  echo
 4  echo "ERROR: Bash v4+ required."
 5  # Extra help for OSX
 6  if [[ "$(uname -s)" == "Darwin" ]]; then
 7    echo
 8    echo "Ensure you are up to date on the following packages:"
 9    echo "$ brew install md5sha1sum bash jq"
10  fi
11  echo
12  exit 9
13fi

(From: https://github.com/kubernetes/kubernetes/blob/66329fcea955522f2eeb5ec13c90d3f5ae7928d5/hack/update-vendor-licenses.sh#L134-L146)

It is also worth noting that while most Linux distros ship GNU coreutils or something reasonably compatible, macOS ships many *BSD derived utilities. sed differences alone have led to many scripts not being portable. Again we simply require GNU sed in some cases and ask the user to install this with Homebrew.

This post only touched on a few common problems, to write safer Bash I recommend reading these general references to understand it better:

You may also want to read Google’s Shell Style Guide.

Final Note 🔗︎

Consider writing non-trivial utilities in another lanuage (or don’t! bash is great!) :^)

Bash is a fun, useful, and easy to distribute. It is also difficult to write complex, reliable programs in. The Google Shell Style Guide has some fairly reasonable recommendations about when to use Bash.

While you could probably implement your entire app in bash, it might not be the best idea 1 :bash_fire:

wargames ending WOPR on writing safe bash

1. I’ll probably do it someday anyhow, just for fun. I actually like bash a lot. It’s just tricky to avoid trivial bugs.