Skip to content
Cloud, DevOps & Infrastructure

Shell Scripting interview questions and answers

79 questions, 55 with code. About 6 h 8 min to read them all.

By TestLab · Updated

Plan for an interview
79 questions

Track what you know and what to revise.

Sign in to track progress

You can open a few full answers without signing in. Sign in to read all

1

What is a shell? How do Bash, sh, zsh, and fish differ?

A shell is a command interpreter that reads commands, expands variables and patterns, starts programs, manages input/output, and returns exit codes. Bash is a widely used GNU shell for Linux automation, while sh, zsh, and fish differ in compatibility, features, and intended use.

Read full answer

Applies toGNU Bash 5.2, POSIX sh (implementation-dependent), Z shell 5.9, fish 4.x

How a Shell Works

When you run a command, the shell generally:

  1. Reads a command line or script.

  2. Performs expansions, including variable, command, arithmetic, and pathname expansion.

  3. Splits the result into words when appropriate.

  4. Applies redirections and pipelines.

  5. Executes a shell function, builtin, or external program.

  6. Waits for completion and exposes an exit status.

External commands usually run in a child process. For example, grep is not part of the shell; the shell starts grep and waits for it. Builtins such as cd, printf, export, and exit are executed by the shell itself, which is why cd changes the current shell’s directory but cannot change the directory of its parent process.

A successful command normally returns 0; a failure returns a nonzero status. These statuses are essential in CI/CD jobs, deployment scripts, health checks, and orchestration systems.

Shell Comparison

Shell Main characteristics Typical use
sh POSIX-oriented, small, portable interface; the actual implementation varies System administration scripts where POSIX portability matters
Bash GNU shell with arrays, [[ ]], brace expansion, process substitution, advanced job control, and pipefail Linux automation, CI/CD, deployment scripts, and interactive shells
zsh Powerful interactive shell with extensive completion, themes, history, recursive globbing, and many configurable options Interactive development and customization
fish User-friendly shell with syntax emphasizing variables, pipelines, and interactive features; intentionally not Bash-compatible Interactive use and fish-specific automation

sh is not necessarily an older version of Bash. On Ubuntu, /bin/sh commonly points to dash; on some other distributions it may point to Bash, BusyBox ash, or another implementation. A script that works under one sh may fail under another if it depends on implementation-specific behavior.

Bash is largely Bourne-compatible, but it is not a strict POSIX shell. zsh offers excellent interactive features but is not a drop-in Bash replacement. fish has a deliberately different syntax, such as set -x NAME value instead of export NAME=value, and does not support Bash’s set -euo pipefail convention.

Bash Scripting Capabilities

Bash provides the features commonly needed in production automation:

  • Variables: name=value, expansion with $name or ${name}, and environment variables with export.

  • Quoting: "$name" preserves one argument; "${array[@]}" safely expands every array element.

  • Conditionals: if, case, [[ ... ]], and command exit-status tests.

  • Loops: for, while, and until, including reading files with while IFS= read -r.

  • Functions: Reusable blocks with positional parameters such as $1, $2, and $#.

  • Arguments: "$@" passes each argument separately; "$*" joins them into one string.

  • Error handling: set -Eeuo pipefail enables stricter error behavior, unset-variable detection, and pipeline failure detection.

  • Text processing: grep selects matching lines, sed transforms text, and awk performs field-based or aggregate processing.

Be careful with set -e: it does not make every possible failure fatal. For example, a command used as an if condition is intentionally allowed to return nonzero. Use explicit checks for operations where partial failure is unacceptable.

Production and Portability Pitfalls

  • Quote every expansion: Prefer "$variable" and "${array[@]}"; unquoted values can cause word splitting and glob expansion.

  • Avoid eval: It is difficult to audit and can enable command injection.

  • Use mktemp, trap, and atomic renames when updating temporary files or deploying artifacts, so interruptions do not leave partial state.

  • Make scripts idempotent: Retrying a deployment should not duplicate resources or corrupt existing state.

  • Log deliberately: Send meaningful progress and failure information to standard error, and avoid exposing secrets.

  • Monitor failures: A nonzero exit status should fail a CI job or alert an orchestrator; silent failures are more dangerous than crashes.

  • Test under the target interpreter: A Bash script may work on a developer laptop but fail on a minimal container where /bin/sh is dash.

  • Prefer portable tools for broad deployment: POSIX utilities and a /bin/sh script are safer across heterogeneous systems; Bash is preferable when arrays, robust conditionals, and pipefail are required.

2

What is a shebang line and why use #!/usr/bin/env bash?

A shebang line is the first line of an executable script that tells a Unix-like operating system which interpreter to run. #!/usr/bin/env bash delegates the search for bash to /usr/bin/env, which uses the caller’s PATH; this helps when Bash is installed outside a fixed location, but it also makes interpreter selection depend on PATH.

Read full answer

Applies toall Bash versions on Unix-like systems; exact shebang execution semantics are platform-specific

What a Shebang Line Does

  • The shebang, also called a hashbang, must begin at the first byte of the file for direct execution to recognize it.

  • Its conventional form is #! followed by an interpreter pathname and, on some systems, an optional argument.

  • When a script is executed directly, the operating system starts the selected interpreter with the script pathname. Once Bash starts, it treats the shebang as a comment.

  • The shebang is not required when explicitly invoking an interpreter, such as bash script.sh; it matters when running ./script.sh.

  • Without a shebang, a direct execve normally fails with ENOEXEC. An invoking shell may then interpret the file as a shell script, but that fallback is shell behavior rather than kernel behavior.

  • Direct execution also requires suitable filesystem permissions, normally including an execute bit for the script.

How #!/usr/bin/env bash Works

  • On Linux, the kernel recognizes #! and starts the named interpreter. Linux supports one optional interpreter argument.

  • With #!/usr/bin/env bash, the kernel starts /usr/bin/env with bash and the script pathname.

  • env then searches PATH for an executable named bash and executes it with the script pathname.

  • If /usr/bin/env does not exist, or no executable bash is found in PATH, execution fails before the script body starts.

  • Other Unix-like systems have broadly similar behavior, but details such as supported arguments and error handling can differ.

Why #!/usr/bin/env bash Is Often Used

  • Interpreter discovery: Bash may be installed at /bin/bash, /usr/bin/bash, or another location, so searching PATH can avoid assuming one fixed path.

  • Version selection: In development environments, PATH can deliberately select a preferred Bash version.

  • Environment consistency: The same script can work across systems with different installation layouts, provided /usr/bin/env exists and the required interpreter is available through PATH.

  • Limitations: The form still hard-codes /usr/bin/env, and an unsafe or unexpected PATH can select the wrong executable. For example, stock macOS provides Bash 3.2 unless a newer version is installed separately.

Hard-Coded Path vs. env Lookup

Aspect #!/bin/bash #!/usr/bin/env bash
Interpreter discovery Uses exactly /bin/bash Uses /usr/bin/env, then searches PATH for bash
Determinism Higher when the target path is standardized Depends on the invocation environment
Security Does not use PATH to select Bash Vulnerable to manipulated PATH
Portability Requires Bash at that path Requires /usr/bin/env and Bash on PATH
Startup One interpreter startup Adds the env lookup and execution step
Best fit Controlled production hosts with a known package-managed path Development or scripts running across known environments

Production Best Practices and Portability

  • Prefer determinism on managed hosts: Install Bash through the platform package manager and use its known absolute path when the execution environment is controlled.

  • Control PATH before launch: If using env, sanitize or set PATH in the launcher before executing the script. Changing PATH inside the script is too late to affect interpreter selection.

  • Use POSIX sh when required: For broad Unix portability, write POSIX-compliant code and use #!/bin/sh; Bash syntax such as arrays, [[ ... ]], and process substitution will not be portable.

  • Preserve file metadata: Keep Unix line endings, the shebang on line 1, and the required execute permission.

  • Test on target platforms: Run the script in CI for every supported OS and architecture, and use bash -n plus ShellCheck for Bash scripts.

  • Manage upgrades: When upgrading Bash or changing the standard system path, retest scripts and automation that rely on interpreter-specific behavior.

3

How do you make a Bash script executable and run it?

Set the script’s executable permission with chmod, then run it using its path, such as ./script.sh. The file must also contain a valid shebang so the operating system knows which interpreter should execute it.

Read full answer

Applies toBash 5.x on Linux (Ubuntu 22.04/24.04, RHEL 9)

Permissions and Commands

Create or edit the script with a shebang as its first line:

#!/usr/bin/env bash

Make it executable and run it:

chmod 755 deploy.sh
./deploy.sh production

Possible output:

Deploying to production

Common permission modes are:

Mode Meaning Typical use
700 Owner can read, write, and execute; group and others have no access Private administrative script
750 Owner has full access; group can read and execute; others have no access Team-managed operational script
755 Owner has full access; group and others can read and execute Public or generally accessible script
744 Owner has full access; group and others can read only, so only the owner (or root) can run it directly Script readable by the team but runnable only by its owner

To grant only the owner execute permission, use:

chmod u+x deploy.sh

Verify the permissions with:

ls -l deploy.sh

An executable script normally appears like this:

-rwxr-xr-x 1 sre sre 2048 Mar 10 09:30 deploy.sh

The permission triplet after the file-type character is owner/group/other: the third character of the owner triplet (x here) is the owner’s execute bit.

How Execution Works

  1. The shell executes ./deploy.sh and verifies that the file has an applicable execute bit.

  2. The kernel reads the #!/usr/bin/env bash shebang and starts /usr/bin/env with the argument bash.

  3. Because that argument is not an absolute path, env searches PATH for bash.

  4. Bash opens the script and starts executing its commands.

A path containing /, such as ./deploy.sh, is resolved directly. A command without a slash, such as deploy.sh, is searched through PATH:

export PATH="$HOME/bin:$PATH"
mkdir -p "$HOME/bin"
mv deploy.sh "$HOME/bin/deploy.sh"
deploy.sh production

If the script is not executable, it can still be run explicitly by the interpreter:

bash deploy.sh production

This bypasses the execute-bit requirement, although the script must still be readable.

Code Example

#!/usr/bin/env bash
set -euo pipefail

if (( $# != 1 )); then
    printf 'Usage: %s <environment>\n' "$(basename "$0")" >&2
    exit 64
fi

readonly environment="$1"

printf 'Deploying to %s\n' "$environment"

Make and run it:

chmod 755 deploy.sh
./deploy.sh production

Check it with ShellCheck before deployment:

shellcheck deploy.sh

The script uses:

  • Quoted expansions: "$1" and "$(basename "$0")" prevent word splitting and pathname expansion.

  • Strict error handling: set -euo pipefail stops on unexpected command failures, rejects unset variables, and detects failures in pipeline components.

  • Explicit exit status: Exit code 64 reports invalid command-line usage.

  • Script arguments: "$1" is the first argument, "$@" represents all arguments, and "$#" is the argument count.

Production and Portability Considerations

  • Use /usr/bin/env bash for portability across Bash installation locations, but ensure PATH is trusted. For privileged automation, a fixed interpreter path such as #!/bin/bash can reduce PATH-manipulation risk.

  • Bash features such as [[ ... ]], arrays, (( ... )), and local are not portable to POSIX sh. Use #!/bin/sh and POSIX syntax when the script must run on minimal systems.

  • Do not rely on set -e as a complete error-handling strategy. Check critical operations explicitly, propagate meaningful exit codes, and use trap for cleanup when appropriate.

  • Keep deployment scripts idempotent so retries do not create duplicate resources. For production automation, add locking, validation, dry-run support, structured logging, and rollback procedures.

  • Avoid embedding credentials in scripts. Prefer a secret manager, protected environment file, or workload identity mechanism.

  • Preserve the executable bit through source control by committing the file’s executable mode; otherwise, configure the deployment pipeline to apply chmod explicitly.

4

How are variables declared and accessed in Bash?

In Bash, variables are named, case-sensitive storage slots for strings: assign them with name=value and access them as $name or ${name}. They are untyped, should normally be expanded inside double quotes, and are passed to child processes only when exported or supplied as a command-prefix assignment.

Read full answer

Applies toBash 5.x on Linux

Declaring and Assigning Variables

Plain assignment creates a shell variable; no separate declaration is required. Do not put spaces around the first =.

  • declare -r name=value: create a read-only variable.

  • declare -x name=value: create an exported variable.

  • declare -a items=("api" "worker"): create an indexed array.

  • declare -A settings=([region]="eu-west-1"): create an associative array.

Variable names begin with a letter or underscore and contain only letters, digits, and underscores. Values are strings, although arithmetic contexts evaluate them as numeric expressions.

Accessing Variables and Scope

  • Reading: Prefix a name with $; use ${name} when adjacent text could obscure the variable boundary.

  • Global and local scope: Ordinary assignments are global. Use local name=value inside a function. Bash uses dynamic scoping: a callee can read and modify a caller’s local variable unless it declares its own local variable with that name.

  • Child processes: export NAME=value marks a variable for the environment of subsequently executed commands. NAME=value command passes it only to that command without exporting it in the current shell.

  • Arguments: $1 is the first positional argument, $# is their count, and "$@" expands each argument as a separate quoted word.

Defaults, Error Handling, and Control Flow

Parameter expansions provide defaults and required-value checks:

Expansion Behaviour
${name:-default} Use default when name is unset or empty
${name-default} Use default only when name is unset
${name:=default} Assign default when name is unset or empty
${name:?message} Report an error when name is unset or empty

set -euo pipefail is a useful automation baseline:

  • -e requests that the shell exit when a command, pipeline, or list returns non-zero, subject to Bash’s documented exceptions.

  • -u treats expansion of an unset variable as an error.

  • pipefail makes a pipeline return non-zero when any component fails.

Do not rely on set -e alone; explicitly check expected failures. Bash conditionals include if/elif/else and case, while loops include for, while, and until. A function can use return N, and a script can use exit N; $? contains the previous command’s status and should be captured immediately.

Text Processing and Portability

Use field- and pattern-aware tools rather than fragile shell splitting:

  • grep -q '^api_' "$file" tests for matching lines; grep returns 0 for a match, 1 for no match, and 2 for an error.

  • sed -E 's/^DEBUG=true$/DEBUG=false/' "$file" performs a stream transformation.

  • awk -F: '$1 == "environment" { print $2 }' "$file" extracts a field from structured records.

Bash-specific features include arrays, [[ ... ]], local, process substitution, and set -o pipefail. A POSIX sh script should use #!/bin/sh, [ ... ], and set -eu, but cannot rely on Bash arrays, local, or pipefail.

Code Example

#!/usr/bin/env bash
set -euo pipefail

# Fail early when required arguments are missing.
if (( $# < 2 )); then
    printf 'Usage: %s <release> <config-file>...\n' "${0##*/}" >&2
    exit 64
fi

readonly release="$1"
shift
readonly app_name="${APP_NAME:-payments-api}"
readonly backup_root="${BACKUP_ROOT:-/var/backups/payments}"

# Restrict permissions for newly created backups.
umask 077

timestamp="$(date -u +%Y%m%dT%H%M%SZ)"
readonly timestamp
readonly backup_dir="${backup_root}/${app_name}/${timestamp}"

mkdir -p -- "$backup_dir"

backup_one() {
    local source_file="$1"
    local destination="$2"

    if [[ ! -f "$source_file" || ! -r "$source_file" ]]; then
        printf 'Config is not a readable file: %s\n' "$source_file" >&2
        return 1
    fi

    cp -- "$source_file" "$destination"
}

# Copy each input into the timestamped backup directory.
index=0
for config_file in "$@"; do
    index=$((index + 1))
    destination="${backup_dir}/${index}-${config_file##*/}"
    backup_one "$config_file" "$destination"
done

printf 'Backed up %d config file(s) for %s release %s to %s\n' \
    "$index" "$app_name" "$release" "$backup_dir"

Example invocation:

APP_NAME=checkout-api BACKUP_ROOT=/tmp/payments-backups \
./backup.sh v2025.03.1 /tmp/payments.json

Example output:

Backed up 1 config file(s) for checkout-api release v2025.03.1 to /tmp/payments-backups/checkout-api/20250308T101530Z

Best Practices and Common Pitfalls

  • Always quote expansions such as "$variable" and "${array[@]}".

  • Use printf instead of echo for predictable formatting and safe handling of user-controlled values.

  • Validate arguments, paths, URLs, and numeric values before using them in production operations.

  • Avoid eval; it can execute attacker-controlled input and obscure failure handling.

  • Do not store secrets directly in scripts or ordinary environment variables. Use a secret manager and inject credentials at runtime.

  • Use read-only variables for values that must not change, but do not treat read-only as a security boundary.

  • In production, add retention, alerting, disk-capacity checks, cleanup, concurrency protection, and recovery testing. A timestamped backup directory alone does not guarantee retention, consistency, or recoverability.

5

Bash quoting: single vs double vs unquoted values?

In Bash, quoting controls whether special characters are expanded and whether whitespace, wildcard characters, and command substitutions are treated as literal text or shell syntax. Single quotes produce literal values, double quotes permit expansions while preventing word splitting and pathname expansion, and unquoted values can be split and globbed.

Read full answer

Applies toBash 5.x

Core Difference

Form Expands variables? Preserves spaces? Allows globbing? Typical use
'text' No Yes No Literal strings
"text" Yes Yes No Variables and command substitutions
text Yes, when applicable No Yes Deliberate shell syntax or splitting
  • Single quotes: Everything between the quotes is literal, including $, backticks, *, and spaces. A literal single quote can be embedded with '\''.

  • Double quotes: Parameter, command, and arithmetic expansions are evaluated, but word splitting and pathname expansion are not performed on the resulting text. The special parameter "$@" is an exception: it expands to separate positional arguments.

  • Unquoted text: Expansion results may be split into multiple arguments using IFS and may be interpreted as glob patterns.

  • Ordinary words: An unquoted literal such as production behaves like 'production'; differences appear when the value contains whitespace, expansions, or special characters.

Expansion and Word Splitting

Bash performs expansions before executing a command. Unquoted expansion results can undergo word splitting and then pathname expansion, while quoted results remain single fields. The exception is "$@", which deliberately preserves the positional-parameter boundaries.

For example, with name="Ada Lovelace", printf '<%s>\n' $name passes Ada and Lovelace as separate arguments, producing <Ada> and <Lovelace>. Using printf '<%s>\n' "$name" passes one argument and produces <Ada Lovelace>.

Inside double quotes, backslash is special only when followed by $, a backtick, ", another backslash, or a newline. Characters such as * remain literal, preventing accidental filename matching.

Security and Production Risks

Unquoted values can cause argument-count, whitespace, and glob-related failures in commands such as kubectl, aws, or deployment scripts. Quoting preserves boundaries, but it does not stop an argument beginning with - from being interpreted as an option. Validate option-like values and use -- before user-controlled arguments when the command supports it.

Command substitution used as an argument should also be quoted: tool "$(printf 'healthy')" passes the output as one argument. In an assignment such as message=$(printf 'healthy'), Bash does not split the command-substitution result. However, quoting the assignment does not preserve it later; expanding "$message" is what prevents splitting at the point of use. Command substitution also removes trailing newline characters.

Never use eval or construct a command string from untrusted input such as HTTP responses, CI variables, cloud metadata, or user-provided parameters. These values can inject shell syntax. Prefer direct argument passing, arrays, explicit validation, and allowlists.

Best Practices and Portability

  • Quote every variable expansion: use "$variable", not $variable.

  • Use single quotes for literals that should not be expanded.

  • Use "${array[@]}" to pass every array element as a separate, preserved argument.

  • Use "$@" to forward positional arguments safely.

  • Use [[ ... ]] for Bash-specific conditionals. For an optional variable under set -u, use [[ -z "${value:-}" ]]; require mandatory variables earlier with an explicit validation or ${variable:?message} check.

  • Use set -euo pipefail, while remembering that set -e does not report every failed command used in a conditional context.

  • Run ShellCheck in CI to detect missing quotes and portability problems.

  • POSIX sh supports single and double quotes but lacks Bash arrays, [[ ... ]], and $'...' quoting. POSIX also does not require set -o pipefail, and common /bin/sh implementations such as dash do not support it. Keep deployment scripts POSIX-compatible when they must run under /bin/sh.

6

Why should variables be quoted in Bash?

Quoting variables in Bash preserves argument boundaries and prevents unintended word splitting and pathname expansion. Unquoted expansions can create extra arguments, disappear when empty, or unexpectedly expand into filenames.

Read full answer

Applies toBash 5.x; the core behavior also applies to POSIX-compliant shells

How Bash Processes Variables

During command processing, Bash recognizes quoted regions before performing expansions. For an unquoted variable expansion, the resulting text can be:

  • Split into multiple words according to the current IFS.

  • Subjected to pathname expansion, so wildcard characters can select matching files.

  • Removed entirely when the variable is set to an empty string and the expansion is the entire word.

A double-quoted expansion remains one word. For example, a value of alpha beta passed as printf '<%s>\n' "$value" is one argument containing a space, whereas printf '<%s>\n' $value normally produces two arguments.

Quoting also prevents expanded content from becoming shell syntax. If an ordinary variable contains a semicolon, redirection operator, or similar character, that character remains data rather than being reinterpreted as shell syntax. Command substitutions written directly inside double quotes are still executed, however, because those substitutions are part of the command before variable expansion occurs.

Common Failure Modes

  • Extra arguments: A path containing spaces becomes several arguments if expanded without quotes.

  • Unexpected globbing: A value containing *.log can expand into every matching file in the current directory.

  • Missing empty values: "$empty" preserves one empty argument when empty is set but empty; $empty normally removes the word.

  • Incorrect loops: Iterating over an unquoted space-separated string can split complete items into fragments.

  • Option interpretation: Quoting keeps a value as one argument, but it does not stop a value beginning with - from being interpreted as an option or operator. Commands differ in how they handle such arguments.

  • IFS-dependent splitting: Unquoted values are split only on characters currently present in IFS. By default, these are space, tab, and newline, not arbitrary Unicode whitespace.

set -euo pipefail does not disable word splitting or pathname expansion. set -u detects references to unset variables, but it does not make unquoted expansions safe.

Choosing the Correct Quote

Form Behavior Typical use
"$variable" Expands the value while preserving one argument Default choice for strings
'$variable' Treats the text literally, including $variable Literal strings and regex fragments
"${array[@]}" Preserves each array element as a separate argument Iterating over Bash arrays
"${array[*]}" Joins elements using the first character of IFS Deliberate serialization
"$@" Preserves each positional parameter as a separate argument Forwarding script arguments

Double quotes permit parameter expansion, command substitution, and specific escape sequences. Single quotes suppress these forms of interpretation. Bash array forms are not portable to POSIX sh.

Security and Operational Considerations

Quoting is an argument-handling control, not a complete security boundary. Do not pass untrusted input to eval, bash -c, or another shell interpreter because those interfaces parse their input as shell code.

When a command supports an end-of-options marker such as --, use it before user-controlled arguments when appropriate. Also validate expected values and prefer APIs that accept structured data instead of constructing shell command strings. Quoting prevents expansion accidents; it does not replace validation, authorization, or safe command design.

For production automation, use quoting consistently rather than relying on assumptions about current directory, IFS, locale, or the values supplied by an upstream job. ShellCheck and tests containing spaces, empty strings, glob characters, and leading hyphens can catch regressions before deployment.

Bash Versus POSIX sh

The basic "$variable" quoting rules are the same in POSIX sh. Bash-specific features such as arrays, [[ ... ]], process substitution, and local are not specified by POSIX and should not be used in portable scripts. Likewise, set -o pipefail is a Bash extension rather than a POSIX option.

A portable script should use POSIX parameter expansion, positional parameters, and loop constructs. The script shebang determines which shell interprets it, so #!/bin/sh should not contain Bash-only behavior.

Best Practices

  • Default to double quotes for string expansions: "$variable".

  • Quote command substitutions: "$(command)".

  • Use "$@" when forwarding positional arguments.

  • Use "${array[@]}" when iterating over Bash arrays.

  • Quote values passed to commands such as printf, test, and external utilities.

  • Use read -r when input should preserve literal backslashes.

  • Remember that Bash assignments do not undergo word splitting or pathname expansion, but quote the right-hand side for clarity and quote the stored value wherever it is later expanded.

  • Run ShellCheck in CI and treat its findings as errors.

7

How do environment variables and export work in Bash?

An environment variable is a string entry in a process’s startup environment, while export gives a Bash shell variable the export attribute so that Bash includes it in the environment of child processes. Each child receives its own environment copy, so changes made by a child do not modify its parent’s variables.

Read full answer

Applies toGNU Bash 5.x on Linux

Environment Variables and Shell Variables

Bash distinguishes variables by how they are used:

  • A shell variable is a named value stored by the current Bash shell. It is visible to shell functions and subshells but is not automatically supplied to newly started external commands.

  • An exported variable is a shell variable with the export attribute. Bash includes it in the environment constructed for child processes.

  • When Bash starts, it imports inherited environment variables as shell variables and retains their export attributes.

Common environment variables include PATH, HOME, LANG, TZ, and application-specific values such as DATABASE_URL. Environment values are strings; Bash does not preserve numeric or Boolean types for child processes. Arrays are shell features and cannot be exported as arrays to external programs.

How export Works

export NAME marks an existing shell variable for export without changing its value. export NAME=value both assigns a value and marks it for export. Exporting does not start a process or send the value anywhere by itself; it affects commands Bash starts afterward.

Related operations include:

  • export -p: Prints variables that have the export attribute.

  • export -n NAME: Removes the export attribute without deleting the shell variable.

  • unset NAME: Removes the shell variable and therefore removes it from the environment inherited by future children.

  • readonly NAME: Prevents later modification of a variable but does not export it; export and readonly are separate attributes.

  • VAR=value command: For an external command, supplies VAR in that command’s environment without permanently changing the current shell variable. Functions and special built-ins have different assignment-persistence semantics.

  • env VAR=value command: Starts an external command with the current environment plus the supplied assignment.

Inheritance and Process Boundaries

When Bash starts an external program, it constructs the program’s environment and passes it during process creation. The child receives a copy, so modifying or unsetting an environment variable in the child does not affect the parent.

This inheritance continues through scripts and descendant processes. For example, if Bash exports DEPLOY_ENV=staging, a Bash script started by that shell imports DEPLOY_ENV, and commands started by the script inherit it unless the script deliberately changes or removes the export attribute.

Changing an exported variable in the parent affects only commands started afterward; it cannot alter the environment of an already-running process.

Production and Security Considerations

  • Validate configuration early: Use an explicit check or ${VAR:?message} so scripts fail clearly when required configuration is missing or empty.

  • Quote expansions: Write "$VAR" to prevent word splitting and pathname expansion.

  • Export only necessary values: Every child inherits exported variables, increasing exposure through diagnostics, logs, env output, or same-user process inspection.

  • Avoid exporting secrets by default: Prefer workload identity or restricted secret delivery where practical. Redact tracing and diagnostic output.

  • Avoid accidental bulk export: set -a automatically exports subsequently assigned variables and can expose temporary or internal values.

  • Use readonly carefully: It helps protect validated configuration from accidental changes but is not a substitute for validation.

  • Handle failures explicitly: Production Bash scripts commonly use set -euo pipefail. set -e has intentional exceptions, including commands in if conditions and non-final commands in && or || lists, so critical operations should still be checked explicitly.

  • Protect logs: set -x can print expanded values, including credentials, so it should not be enabled around secrets.

Portability and Common Pitfalls

export is supported by POSIX shells, but export -n, Bash arrays, [[ ... ]], and set -o pipefail are Bash-specific or otherwise not portable. For example, /bin/sh on Ubuntu commonly points to dash, which does not support pipefail.

Exporting a variable does not make it global across unrelated shells, preserve its original type, or encrypt its value. It only controls whether Bash includes that variable in the environment of commands it starts.

8

What are Bash positional parameters like `$1`, `$@`, and `$#`?

Bash positional parameters are the arguments supplied to a script or function. $1 through $9 represent individual arguments, $# contains their count, and "$@" expands all arguments while preserving their boundaries.

Read full answer

Applies toBash 5.x on Linux and Unix systems

How They Work

When Bash starts a script, it assigns the script’s command-line arguments to positional parameters. For example, invoking ./deploy.sh production api-service --verbose provides:

  • $0: the script name used for invocation, such as ./deploy.sh.

  • $1: production.

  • $2: api-service.

  • $3: --verbose.

  • $#: 3, the number of positional parameters.

Positional parameters above nine require braces: ${10} refers to the tenth argument, while $10 would be interpreted as $1 followed by the character 0.

Each function has its own positional parameters. Bash restores the caller’s parameters when the function returns. The complete positional-parameter set can also be replaced with set --, but syntax such as 1=value is invalid.

Important Parameter Expansions

Expression Meaning
$0 Name used to invoke the script; inside a function, it normally remains the script or shell name
$1–$9 Individual positional arguments
${10}, ${11} Positional arguments with two or more digits
$# Number of positional parameters
"$@" All arguments, each preserved as a separate word
"$*" All arguments joined into one word using the first character of IFS
$? Exit status of the previous command; not a positional parameter
$$ Process ID of the current shell; not a positional parameter
$! Process ID of the most recently started background job; not a positional parameter

The distinction between "$@" and "$*" matters when arguments contain spaces. Given arguments such as web server and api server, "$@" preserves them as two arguments, while "$*" combines them into the single string web server api server. Unquoted $@ and $* are unsafe for user-supplied values because they undergo word splitting and pathname expansion.

Functions and Argument Forwarding

The shift command removes parameters from the beginning of the positional-parameter list and renumbers those that remain. A plain shift removes one argument, while shift 2 removes two. After either operation, $# and "$@" reflect the updated list. The argument count must be checked before shifting to avoid an error.

When forwarding arguments to another command or function, use "$@". This preserves each argument as a distinct value and avoids word splitting, pathname expansion, and accidental execution of shell metacharacters.

Production Practices and Common Pitfalls

  • Quote expansions: Use "$@", "$*", "$1", and "$#" where appropriate. Quoting prevents values containing spaces or wildcard characters from being altered.

  • Validate argument counts: Check $# before accessing required arguments. Under set -u, expanding an unset positional parameter can terminate the script.

  • Use shift carefully: Confirm that enough arguments exist before shifting, and account for the updated numbering afterward.

  • Prefer "$@" for forwarding: It preserves argument boundaries and safely passes empty or space-containing values.

  • Use getopts for short options: Bash’s getopts is suitable for options such as -v or -n value. Long options generally require a separate case parser.

  • Avoid eval: Never use eval to process arguments because attacker-controlled input can become executable shell code.

  • Plan for portability: "$@", $#, and shift are supported by POSIX shells, but arrays, brace expansion such as {1..5}, [[ ... ]], and process substitution are Bash-specific.

9

How do positional parameters $@ and $* differ in Bash?

In Bash, quoted expansion is the decisive distinction: "$@" preserves each positional parameter as a separate argument, while "$*" joins the positional parameters into one argument. This distinction matters when values contain whitespace, empty strings, or wildcard characters.

Read full answer

Applies toBash 5.x, including Bash 5.2

Expansion Rules

Positional parameters are the arguments supplied to a script or function.

Expression Expansion behavior
"$@" Produces one field per positional parameter and preserves their exact boundaries. With no positional parameters, it produces zero fields.
"$*" Joins the positional parameters into one field using the first character of IFS. With no positional parameters, it produces one empty field.
$@ Starts with separate positional parameters, but because the expansion is unquoted, word splitting and pathname expansion may alter their boundaries and contents.
$* Joins the positional parameters first, then word splitting and pathname expansion may alter the resulting text.

The quoted forms "$@" and "$*" are not subject to word splitting or pathname expansion.

IFS and Empty Arguments

The default IFS begins with a space, so "$*" normally uses a space as its separator. If IFS is changed, its first character becomes the separator; for example, an IFS beginning with a colon makes "$*" join arguments with colons. Setting IFS to the empty string removes the separator.

These behaviors must be distinguished from an explicitly supplied empty argument. If a script receives one empty-string argument, "$@" produces one empty field and "$*" also produces one empty field.

Changing IFS affects other parsing operations in the shell, so production scripts should avoid changing it globally merely to format one message.

Forwarding Arguments

Use "$@" when forwarding arguments to a function or local command because each value remains a distinct argument. Use "$*" only when flattening the values into one human-readable string is intentional.

Avoid reconstructing shell command strings from arguments or passing them through eval; that can introduce quoting errors, word-splitting vulnerabilities, and command-injection risks.

SSH requires an additional caveat: ssh concatenates the remote command arguments into a string, and the remote shell parses that string again. Consequently, "$@" does not guarantee preservation of remote argument boundaries. Untrusted values should not be embedded in a remote shell command without a transport or API designed for structured arguments.

Production Guidance

  • Prefer "$@" for forwarding; reserve "$*" for deliberate joining.

  • Do not rely on unquoted $@ or $*, especially in automation involving spaces, tabs, newlines, or glob characters.

  • set -euo pipefail does not make unsafe expansions safe; retain explicit status checks for failures that are expected or recoverable.

  • Under set -u, "$@" and "$*" remain safe when no positional parameters exist, while directly expanding an unset parameter such as $1 fails.

  • Run ShellCheck in CI to detect quoting defects and unused assignments.

  • Avoid exposing secrets directly in command-line arguments where possible, because arguments may be visible in process listings or operational logs.

10

What do Bash special parameters like $?, $$, and $! mean?

Special parameters are built-in Bash variables that expose process, argument, and execution-status information. They are expanded by the shell, so their values often change depending on the most recently executed command or background job.

Read full answer

Applies toBash 5.x on Linux

Core Special Parameters

Parameter Meaning
$? Exit status of the previously executed command: 0 generally means success; a non-zero value means failure.
$$ Process ID of the current shell. In a Bash subshell, $$ normally retains the parent shell’s PID.
$! Process ID of the most recently started background job. It may be empty if no background job exists.
$0 Name of the script as invoked, such as ./deploy.sh, or the shell name when running interactively.
$1–$9 Positional arguments. Use ${10} for arguments numbered ten or higher.
$# Number of positional arguments passed to the script or function.
$@ All positional arguments, preserving individual argument boundaries. Prefer "$@".
$* All positional arguments joined using the first character of IFS. Use "$*" when one combined string is intended.
$- Options currently active in the shell.
$_ Last argument of the previous command, with some context-dependent behavior; avoid using it for critical logic.

Bash also provides ${PIPESTATUS[@]}, containing the individual exit statuses of commands in the most recently executed pipeline. This differs from $?, which contains the pipeline’s overall status.

Status Handling and Error Handling

The shell updates $? after a command completes. Capture it before running another command; for example, after false, $? expands to 1, and immediately assigning that value to a variable preserves it.

Within an if condition, the else branch can capture the failed command’s status before performing other work. This is preferable to checking $? after an intervening command because that command may overwrite it.

set -euo pipefail provides common production safeguards:

  • -e: Exit when a command fails, except where failure is explicitly handled in contexts such as if, while, &&, or ||.

  • -u: Treat references to unset variables as errors. Use ${variable:-default} for optional values.

  • -o pipefail: Return failure when any component of a pipeline fails.

set -e is not a substitute for explicit error handling. Important operations should use if checks, functions should return meaningful statuses, and cleanup should normally be registered with trap.

Functions have their own positional parameters. Calling function_name "$@" preserves argument boundaries, while return "$status" communicates failure to the caller.

Production Practices

  • Quote expansions: Use "$variable", "$@", and "$1". Unquoted values can undergo word splitting and pathname expansion.

  • Handle statuses explicitly: grep returns 0 for a match, 1 for no match, and 2 for an error; these cases should be treated differently.

  • Wait for jobs: Use wait "$background_pid" rather than assuming a background operation completed.

  • Do not use $$ for locking: Multiple subshells can share it. Use flock or an atomic filesystem operation instead.

  • Prefer BASHPID inside subshells: $BASHPID identifies the current Bash process, whereas $$ may identify its parent shell.

  • Use ${BASH_SOURCE[0]} when sourcing: $0 can still refer to the calling script.

  • Process files safely: Prefer commands such as grep -F -- "$needle" "$file" for literal matching and always place user-controlled data in variables rather than constructing shell code.

Bash and POSIX Compatibility

$?, $$, $!, $0, $#, $@, $*, and ordinary positional arguments are defined by POSIX and are broadly supported by /bin/sh. Features not specified by POSIX—including ${PIPESTATUS[@]}, ${BASH_SOURCE[0]}, arrays, [[ ... ]], pipefail, and local—are not guaranteed across all /bin/sh implementations.

For POSIX scripts, use set -eu, avoid Bash-only syntax, and check pipeline components individually when pipeline failure matters.

11

How do exit status codes work in Bash scripts?

In Bash, every command produces an integer exit status: 0 conventionally indicates success, while a non-zero value indicates failure or an error. A script normally returns the status of its last command unless an explicit exit statement specifies the result.

Read full answer

Applies toBash 5.0–5.3 on Linux and POSIX sh where noted

Exit Status Fundamentals

The special parameter $? contains the exit status of the most recently completed foreground command:

true
printf '%s\n' "$?"  # 0

false
printf '%s\n' "$?"  # 1

Read $? immediately, because running another command replaces it.

Common conventions include:

Situation Typical status
Successful command 0
General failure 1
Incorrect command-line usage 64
Command not found 127
Command found but not executable 126
Terminated by SIGINT 130
Terminated by SIGTERM 143
Terminated by SIGKILL 137

These values are conventions rather than universal guarantees. For signals, Bash normally reports 128 + signal_number, unless a signal trap handles the event.

How Exit Statuses Propagate

  • Script completion: Without an explicit exit, the script returns the status of its last command.

  • exit n: Terminates the script with status n.

  • exit without an argument: Uses the status of the previous command.

  • return n: Returns status n from a function. Without an argument, it returns the previous command’s status.

  • Pipeline: Returns the status of the last command by default. With set -o pipefail, it returns the rightmost non-zero status, or zero if every command succeeds.

  • Negation: ! command reverses the status, so it returns zero when command fails.

  • Background process: $! contains the process ID, not its exit status. Use wait "$pid" to obtain the status.

  • Subshell: A subshell has its own environment, but exits with the status of its last command or an explicit exit.

For example, false | true normally returns 0, while the same pipeline with pipefail returns 1.

Error Handling with set -euo pipefail

The common Bash preamble is:

set -euo pipefail
  • -e: Exits when an unhandled command returns non-zero.

  • -u: Treats references to unset variables as errors.

  • pipefail: Makes pipelines fail when any component fails.

However, set -e has important exceptions. Bash does not exit for a failing command used as:

  • The condition of if, while, or until

  • Part of an && or || list, except the final command

  • A command prefixed by !

A function called from an if condition also runs with -e effectively suppressed throughout its body. Therefore, explicitly checking important commands is safer than depending only on set -e.

Code Example

#!/usr/bin/env bash
set -euo pipefail

usage() {
    printf 'Usage: %s <url>\n' "$0" >&2
    exit 64
}

check_url() {
    local url=$1
    local output=$2
    local status=0

    if curl --fail --silent --show-error --max-time 10 \
        "$url" >"$output"; then
        return 0
    else
        status=$?
    fi

    return "$status"
}

main() {
    if (( $# != 1 )); then
        usage
    fi

    local url=$1
    local output
    output=$(mktemp)

    # Remove the temporary file on normal or explicit-exit paths.
    trap 'rm -f -- "$output"' EXIT

    if check_url "$url" "$output"; then
        printf 'Health check succeeded.\n'
        exit 0
    else
        local status=$?
        printf 'Health check failed (exit %d).\n' "$status" >&2
        exit "$status"
    fi
}

main "$@"

The script captures the failure status in an else block because using if ! command discards the original status. For an HTTP 500 response, for example, curl --fail normally returns 22, which the script propagates.

Production Best Practices and Portability

  • Use explicit exits: End scripts with exit 0 on success and meaningful non-zero statuses on failure.

  • Handle expected failures: Check network calls, file operations, and validation commands instead of relying exclusively on set -e.

  • Clean up resources: Use an EXIT trap for temporary files, but do not rely on it for SIGKILL, which cannot be trapped.

  • Handle termination signals: In long-running services, trap SIGTERM and SIGINT to stop gracefully and report the final status.

  • Make failures observable: CI systems and service managers use the exit status to decide whether a deployment, job, or service succeeded.

  • Avoid ambiguous statuses: Reserve documented codes for categories such as configuration errors, dependency failures, and timeouts.

  • Remember portability: Exit statuses and $? are POSIX concepts, but pipefail is not required by POSIX sh. Scripts requiring Bash should retain #!/usr/bin/env bash rather than using #!/bin/sh.

12

How do you write if, elif, and else conditionals in Bash?

Conditional statements in Bash execute command lists and branch on their exit status: zero indicates success, while a non-zero status indicates failure. An if chain can contain optional elif branches and an optional else catch-all, and every chain must end with fi.

Read full answer

Applies toBash 5.x; POSIX `sh` portability notes

How It Works

  • if: Executes its body when the preceding command list returns status zero.

  • then and fi: then introduces the first body and fi closes the chain. then may follow the condition on the same line or the next line.

  • elif: Tests another condition only if the previous conditions failed.

  • else: Runs when no preceding condition succeeded.

  • && and ||: Chain commands based on status; they are not substitutes for a multi-branch conditional.

The Three Test Contexts

Context Typical use Important behavior POSIX sh
[ "$x" = "y" ] Portable string and file tests Quote expansions to preserve argument boundaries Supported
[[ "$x" =~ regex ]] Bash patterns, regular expressions and arrays No word splitting or pathname expansion; leave the regex right-hand side unquoted for regex matching Bash-only
(( count > 10 )) Integer arithmetic Variable names normally omit $ Bash-only

[ ] versus [[ ]]: [ is a command, so its arguments must be quoted where necessary. [[ ]] is a Bash keyword with safer expansion rules and additionally supports =~, internal &&/||, and array-element tests.

Common Tests

  • String: [ -z "$s" ] for empty and [ -n "$s" ] for non-empty.

  • Equality: [ "$a" = "$b" ] or [[ "$a" == "$b" ]].

  • Numeric: [ "$n" -gt 10 ] or (( n > 10 )); non-numeric input produces an error.

  • File: [ -f "$path" ], [ -d "$path" ], [ -e "$path" ], and [ -s "$path" ].

  • Glob: [[ "$filename" == *.log ]]; use case for broader multi-way dispatch.

Failure Modes and Pitfalls

  • Unquoted expansions: [ $value = "text" ] can change the argument count. An empty expansion can produce a misleading result, while a multi-word expansion can cause “too many arguments.”

  • Regular expressions: In Bash 5.x, [[ $text =~ ^[0-9]+$ ]] treats the right-hand side as a regular expression. Quoting part of that expression makes the quoted text match literally.

  • Invalid integers: Numeric tests such as [ "$x" -gt 5 ] return an error for non-numeric input; validate the value before comparing it.

  • Missing fi: This causes a parse error, so the script does not execute.

  • set -e behavior: The if and elif condition lists are exempt because their statuses are tested. Branch-body commands are normally checked, but commands in tested &&, ||, !, if, or while contexts are also exempt.

Code Example

This complete Bash 5 script demonstrates if, elif, and else while validating unsafe input:

#!/usr/bin/env bash
# Validate deployment arguments and report service health.
set -euo pipefail

usage() {
  printf 'Usage: %s <service-name> <replica-count>\n' "${0##*/}" >&2
  exit 64
}

[[ $# -eq 2 ]] || usage

service=$1
replicas=$2

# Restrict the name so it cannot contain path separators or newlines.
if [[ ! $service =~ ^[A-Za-z0-9][A-Za-z0-9._-]*$ ]]; then
  printf 'Error: invalid service name: %s\n' "$service" >&2
  exit 64
fi

if [[ ! $replicas =~ ^[0-9]+$ ]]; then
  printf 'Error: replica-count must be a non-negative integer\n' >&2
  exit 64
fi

# Force base 10 so values such as 08 are not interpreted as octal.
replicas=$((10#$replicas))

if (( replicas < 1 )); then
  status="CRITICAL"
elif (( replicas < 3 )); then
  status="DEGRADED"
elif (( replicas < 6 )); then
  status="WARNING"
else
  status="HEALTHY"
fi

printf 'service=%s replicas=%s status=%s\n' \
  "$service" "$replicas" "$status"

config_dir="${CONFIG_DIR:-/etc}"
config="$config_dir/$service.conf"

if [ -f "$config" ]; then
  printf 'config found: %s\n' "$config"
elif [ -d "$config_dir" ]; then
  printf 'no config for %s, but %s exists\n' "$service" "$config_dir"
else
  printf 'configuration directory does not exist: %s\n' "$config_dir" >&2
  exit 69
fi

With api 2 as the arguments and no /etc/api.conf, the output is:

service=api replicas=2 status=DEGRADED
no config for api, but /etc exists

Best Practices for Production Scripts

  • Start Bash scripts with #!/usr/bin/env bash and normally use set -euo pipefail; pipefail is not POSIX.

  • Quote expansions where word splitting or pathname expansion can occur. Bash’s [[ ]] and arithmetic contexts have different, safer rules.

  • Prefer [[ ]] in Bash and [ ] in portable POSIX sh scripts.

  • Use case for multi-way dispatch rather than a long elif chain.

  • Treat exit codes as an API, send diagnostics to stderr, and keep machine-readable output on stdout.

  • For production automation, add explicit timeouts, retries with backoff, audit logging, dry-run support, monitoring and alerting rather than relying solely on set -e.

13

What do [ ], [[ ]], and (( )) do differently in Bash?

In Bash, [ ... ] is the test command, [[ ... ]] is a conditional expression, and (( ... )) evaluates an arithmetic expression. They differ in supported operators, expansion behavior, exit-status semantics, and portability.

Read full answer

Applies toBash 5.x; all POSIX-compliant `sh` implementations for `[ ... ]` only

Core Differences

Construct Purpose Example Portable to POSIX sh
[ ... ] Tests strings, integers, and filesystem properties [ -f "$file" ] Yes
[[ ... ]] Bash conditional expression with pattern and regular-expression matching [[ "$name" == api-* ]] No
(( ... )) Evaluates an arithmetic expression (( count > 0 )) No

[ is a command name. Bash provides it as a builtin, while POSIX systems may also provide it as /usr/bin/[. Its arguments undergo normal shell expansion, so variable expansions should be quoted.

[[ and ]] are Bash reserved words handled directly by the parser. Words inside them do not undergo normal word splitting or pathname expansion.

(( and )) use Bash’s arithmetic evaluator. Arithmetic variables are referenced by name rather than with the usual $ prefix.

Supported Operators

  • String tests with [ ]: use -z, -n, =, and !=.

  • String tests with [[ ]]: use -z, -n, ==, !=, and pattern matching.

  • Regular expressions with [[ ]]: use =~, which matches against a POSIX extended regular expression.

  • Integer tests with [ ]: use -eq, -ne, -lt, -le, -gt, and -ge.

  • Arithmetic with (( )): use C-style operators such as +, %, &&, ||, and comparisons.

Unlike [ "$name" = api-* ], the right side of [[ "$name" == api-* ]] is interpreted as a glob pattern rather than a literal string.

Expansion and Security

Normal command expansion applies to [ ... ]. For example, [ $status = ready ] is unsafe when status is empty or contains spaces. Quote every expansion, as in [ "$status" = "ready" ].

Inside [[ ... ]], splitting and glob expansion do not occur, although quoting remains useful for clarity. Leave the right-hand regular expression unquoted so its metacharacters remain active, as in [[ "$email" =~ ^[a-z]+@[a-z]+\.example\.com$ ]].

Bash recursively evaluates arithmetic variable values as arithmetic expressions. Do not place untrusted input directly into (( ... )); validate that it is an expected integer first. Bash also interprets leading-zero integers as octal, so a value such as 08 can cause an arithmetic error.

Exit Status

All three constructs can be used for conditional execution through if, while, &&, and ||.

  • [ ... ] and [[ ... ]] return 0 for true, 1 for false, and an error status for invalid operations.

  • (( ... )) returns 0 when the arithmetic result is nonzero and 1 when the result is zero.

That distinction matters with set -e: a standalone (( count == 0 )) can terminate a script because the expression returned status 1. Inside an if condition, the status is handled normally and does not trigger errexit.

Production and Portability

Use [ ... ] in scripts deployed through /bin/sh, POSIX containers, or portable CI runners. Use [[ ... ]] when Bash-specific pattern matching, regular expressions, or safer expansion semantics are useful. Use (( ... )) for counters, thresholds, retries, and resource-limit calculations.

For operational scripts, report failures on stderr, return meaningful exit codes, and avoid relying on set -e as a substitute for explicit error handling. A failure inside a conditional list may be intentional and therefore ignored by errexit.

Best Practices and Pitfalls

  • Start Bash scripts with #!/usr/bin/env bash and use set -euo pipefail where appropriate.

  • Quote variables in [ ... ]; use "${value:-}" when an unset variable should be treated as empty under set -u.

  • Prefer [[ ... ]] over [ ... ] in Bash scripts when it provides clearer pattern or regular-expression behavior.

  • Use (( ... )) for arithmetic, not general string evaluation.

  • Never use eval as a substitute for structured tests or parsing.

  • Run ShellCheck in CI and test failure paths because malformed values, unset variables, and nonzero arithmetic results are common production failure modes.

14

How does Bash compare strings and numeric values?

Bash provides separate syntax for string comparison and numeric comparison. Use [[ ... ]] for string operations and the arithmetic command (( ... )) for integer comparisons.

Read full answer

Applies toBash 5.x on Linux

How It Works

Bash shell variables hold strings; the operator and evaluation context determine whether a value is treated as text or an integer.

  • String equality: [[ "$status" == "ready" ]]

  • String inequality: [[ "$status" != "ready" ]]

  • Lexical ordering: [[ "$name" < "$other_name" ]]

  • Numeric ordering: (( version < limit ))

  • POSIX-compatible numeric comparison: [ "$version" -lt "$limit" ]

Inside [[ ... ]], == and != perform string matching. The right-hand operand can act as a glob pattern unless it is quoted. Thus, [[ "$environment" == "prod" ]] is an exact literal comparison, while [[ "$environment" == prod-* ]] matches names beginning with prod-.

The < and > operators perform lexical ordering rather than numeric ordering. Their results can depend on the locale’s collation rules, which matters when scripts must produce deterministic comparisons.

The numeric operators -eq, -ne, -lt, -le, -gt, and -ge compare integers. The arithmetic command (( ... )) is usually clearer for compound expressions such as (( version >= minimum_version && version < maximum_version )). Within that command, ==, !=, <, <=, >, and >= are arithmetic operators, not string operators.

Bash arithmetic supports signed integers, not floating-point numbers.

Choosing an Operator

Requirement Bash form Notes
Exact string equality [[ "$a" == "$b" ]] Quote both operands for literal comparison
Glob-style string match [[ "$name" == prod-* ]] Unquoted right side is a pattern
Lexical string ordering [[ "$a" < "$b" ]] Uses locale collation
Integer comparison (( a < b )) Evaluates an arithmetic expression
POSIX numeric comparison [ "$a" -lt "$b" ] Works in Bash and POSIX sh
POSIX string equality [ "$a" = "$b" ] Use =, not Bash-specific ==

Failure Modes and Security

  • Validate numeric operands: Numeric comparisons require valid integer expressions. Inside [[ ... ]], -lt evaluates its operands as arithmetic expressions: a value such as 1.5 causes an error (status 1 with a diagnostic), while a word such as abc is treated as a variable name and silently becomes 0.

  • Do not evaluate untrusted values as arithmetic expressions: Bash recursively evaluates referenced variable values as arithmetic expressions. Validate or normalize input before using it.

  • Quote patterns when literal matching is intended: [[ "$value" == "literal" ]] prevents the right-hand side from being interpreted as a glob.

  • Avoid leading-zero surprises: Ordinary arithmetic interprets leading-zero values as octal, so 08 is invalid. The 10# base prefix, such as 10#$value, forces decimal interpretation after validation.

  • Watch exit statuses: [[ ... ]] returns 0 for true and 1 for false. An arithmetic command returns 1 when its expression evaluates to zero and can return a greater status for evaluation errors.

  • Understand set -e exceptions: A failed [[ ... ]] or (( ... )) used as an if, while, or until condition does not terminate the script. In an && or || list, errexit is ignored for commands except the final one, so a failing final command can terminate the script. A standalone failed comparison can also terminate it.

Portability and Best Practices

  • Prefer [[ ... ]] in Bash scripts. It does not perform word splitting or pathname expansion on its operands.

  • For portable POSIX sh, use [ ... ], single = for string equality, and escape ordering operators as in [ "$a" < "$b" ] (accepted by Bash and dash, though not specified by POSIX).

  • Set a deterministic locale when comparison order matters, such as export LC_ALL=C for bytewise ordering. This is not equivalent to natural Unicode linguistic sorting.

  • Pass function inputs as quoted positional parameters such as "$1" and "$@", validate them at the function boundary, and use comparison statuses normally in conditionals and loops.

  • Use grep, sed, and awk to extract or transform text, then validate the resulting values before comparing them numerically.

  • Production Bash scripts commonly use set -euo pipefail, but this does not replace explicit error handling or validation.

  • In CI/CD and orchestration systems, emit errors to stderr and return meaningful nonzero exit codes so failed comparisons and invalid inputs can trigger alerts or remediation.

15

How do Bash for, while, and until loops work?

Bash loops repeat one or more commands according to a collection, condition, or success check. A for loop is usually best for known collections, while for condition-based or input-driven processing, and until for repeating an operation until it succeeds.

Read full answer

Applies toBash 5.x on Linux

Choosing a Loop

Loop Typical use Continues when
for Iterate over files, array elements, or positional parameters Items remain in the collection
Arithmetic for Repeat a fixed or calculated number of times Its arithmetic condition remains true
while Process streams or repeat while a condition is true The condition returns success
until Retry an operation until it succeeds The condition returns failure

A list-based for loop expands a collection before executing its body. It can iterate over shell words, pathname expansions, command substitutions, or Bash array elements. An arithmetic for loop uses initialization, condition, and update expressions; it is a Bash extension rather than POSIX sh syntax.

How Each Loop Works

A while loop evaluates its condition before every iteration. If the condition succeeds, the body runs; if it fails, the loop ends. A while loop therefore performs zero iterations when its initial condition is false.

An until loop has the inverse condition behavior. Its body runs while the condition fails and stops when the condition succeeds. It also performs zero iterations when the initial condition succeeds. This makes until useful for retries, readiness checks, and waiting for an asynchronous operation to complete.

A list-based for loop consumes the remaining elements of its expanded collection. Quoting an array expansion such as "${items[@]}" preserves each element as one argument and prevents unwanted word splitting and pathname expansion.

Control Flow and Exit Status

All three loop forms support break and continue. break exits the current loop, while continue skips to its next iteration. Numeric variants such as break 2 and continue 2 affect two enclosing loop levels.

A loop normally returns the exit status of the last command executed in its body, or zero if its body never executes. Consequently, callers that check a loop’s status should not assume that reaching the end of the loop means success; the final body command must succeed deliberately.

With set -euo pipefail, set -e normally terminates a script after an unchecked command fails. Commands used directly as while or until conditions are exempt from that behavior, which allows an expected failure to control the loop. Commands in the loop body are generally subject to errexit. Bash also suppresses errexit throughout a function or compound command when that function or command is itself invoked in an exempt context, so production scripts should test critical operations explicitly rather than relying only on set -e.

set -u rejects references to unset variables, while set -o pipefail makes a pipeline fail when any component fails, rather than considering only the final component’s status. A grep exit status of 1, meaning no match, can therefore terminate a script unexpectedly; place such a command in an if condition when no match is an expected outcome.

Production Practices and Portability

  • Bound retries: An until loop can run forever if success never occurs. Add a maximum attempt count, timeout, or deadline.

  • Quote expansions: Use quoted parameter and array expansions to prevent word splitting and pathname expansion.

  • Preserve input: Use read -r when backslashes should remain literal. A while read loop should also handle a final unterminated line when processing files or streams.

  • Keep iterations observable: Send progress and diagnostic messages to stderr, and ensure the final exit status reflects the operation’s outcome.

  • Handle cleanup: Use trap to remove temporary files, release locks, and perform other required cleanup on failure.

  • Avoid Bash-only syntax in POSIX scripts: Arrays, arithmetic for loops, and several expansion features require Bash. POSIX sh supports list-based for, while, and until, but not Bash arithmetic-loop syntax.

16

How do I use a safe Bash read loop for file lines?

Use a while IFS= read -r loop to process a file one line at a time without losing whitespace or interpreting backslashes. Adding || [[ -n "$line" ]] also processes a final line that does not end with a newline.

Read full answer

Applies toBash 5.x on Linux

How the Read Loop Works

  • read: Reads one line from standard input.

  • IFS=: Prevents leading and trailing whitespace from being removed.

  • -r: Treats backslashes literally instead of interpreting escape sequences.

  • || [[ -n "$line" ]]: Handles a nonempty final line when the file does not end with a newline.

  • Redirection: < "$file" connects the loop directly to the file, so the loop runs in the current shell.

The pattern preserves indentation and literal backslashes such as C:\temp\file. By contrast, for line in $(cat "$file") splits on whitespace, removes blank lines, and may perform pathname expansion.

Why Not Pipe into the Loop?

With Bash’s default lastpipe setting disabled, each pipeline component runs in a subshell, so assignments made inside the loop do not affect the parent shell. set -o pipefail can report a failing cat, but it does not preserve variables changed inside the loop.

Direct redirection is preferable because it keeps the loop in the current shell and makes failures from opening or reading the file explicit. Process substitution can also keep the loop in the current shell, but direct redirection is simpler and avoids obscuring failures in the producing command.

Edge Cases and Portability

Situation Behavior or recommendation
Empty file The loop body does not execute.
Blank lines Blank lines are preserved and passed to the loop.

| Final line lacks a newline | The || [[ -n "$line" ]] condition processes it. | | Windows CRLF files | read retains the \r; remove it only when CRLF is expected. | | NUL bytes | Bash variables cannot represent NUL bytes reliably; use a binary-aware tool instead. | | Very large files | The read loop uses constant memory and is preferable to mapfile, which stores the input in memory. | | POSIX sh | Use read -r with [ -n "$line" ]; [[ ... ]], pipefail, and mapfile are Bash extensions. |

For a known CRLF file, remove the trailing carriage return inside the loop with line=${line%$'\r'}. Do this only when the input format is known, because it modifies otherwise valid content.

Code Example

The core pattern is:

while IFS= read -r line || [[ -n "$line" ]]; do
  # Process "$line"
done < "$file"

A complete, ShellCheck-compatible script is:

#!/usr/bin/env bash
set -euo pipefail

if (( $# != 1 )); then
  printf 'Usage: %s FILE\n' "$0" >&2
  exit 2
fi

file=$1

if [[ $file == "-" ]]; then
  input_file=/dev/stdin
else
  if [[ ! -f $file || ! -r $file ]]; then
    printf 'Not a readable regular file: %s\n' "$file" >&2
    exit 1
  fi
  input_file=$file
fi

line_number=0
line=''

while IFS= read -r line || [[ -n $line ]]; do
  line_number=$((line_number + 1))
  printf '%d:%s\n' "$line_number" "$line"
done < "$input_file"

Save it as process-lines.sh, then run it without requiring executable permissions:

printf 'alpha\nbeta\n' | bash ./process-lines.sh -

Avoid this pattern because the loop runs in a subshell:

count=0
cat "$file" | while IFS= read -r line; do
  count=$((count + 1))
done

Use redirection instead so that count is retained:

count=0
while IFS= read -r line || [[ -n $line ]]; do
  count=$((count + 1))
done < "$file"

Production Best Practices

  • Always quote filenames: < "$file" safely handles spaces, glob characters, and filenames beginning with -.

  • Use set -euo pipefail for strict error handling, but handle expected failures explicitly inside the loop.

  • Avoid eval, unquoted expansions, and for line in $(cat "$file").

  • If the file is rewritten in place while being read, the loop may observe a mixture of old and new content. For configuration and state files, write a new file and atomically rename it into place.

  • For log rotation, use a tool designed for that behavior, such as tail -F, rather than repeatedly reopening a fixed path.

  • Run the script through ShellCheck and test empty files, blank lines, missing final newlines, and filenames containing spaces.

17

How do you loop over files with spaces in Bash?

Use NUL-delimited iteration: have find emit each pathname terminated by \0, then read each value with IFS= read -r -d ''. This keeps filenames containing spaces, tabs, newlines, quotes, and backslashes intact as single values.

Read full answer

Applies toBash 5 on Linux (Ubuntu 22.04/24.04, RHEL 9)

How It Works

Filenames are strings, so spaces inside them are valid. Problems usually occur when a filename is incorrectly separated into multiple arguments through:

  • Unquoted variable expansion, such as for file in $files.

  • Newline-based input, such as find ... | while read file.

  • Command substitution, such as for file in $(find ...).

The robust pattern uses find -print0 and read -d '':

  • -print0: Separates paths with the NUL byte.

  • -d '': Reads until the NUL byte because Bash represents an empty delimiter as \0.

  • -r: Prevents backslashes from being interpreted.

  • IFS=: Prevents leading or trailing whitespace from being removed.

Approach Comparison

Approach Handles spaces Handles newlines Scope Portability
for file in "$dir"/*.txt Yes Yes One directory, fixed pattern POSIX-compatible
find -print0 with read -d '' Yes Yes Recursive and flexible Bash-specific
find -exec command {} + Yes Yes Recursive POSIX-compatible
find | while IFS= read -r Yes, with IFS= No Recursive Bash-specific

Code Example

This script safely processes every regular file below the supplied directory:

#!/usr/bin/env bash
set -euo pipefail

target_dir=${1:-.}

if [[ ! -d "$target_dir" ]]; then
    printf 'Not a directory: %s\n' "$target_dir" >&2
    exit 1
fi

find "$target_dir" -type f -print0 |
    while IFS= read -r -d '' file; do
        # The complete pathname is preserved in one variable.
        printf 'Processing: <%s>\n' "$file"
    done

Example output:

Processing: <quarterly report.txt>
Processing: <Q1 final (approved).txt>

For a single directory and a known extension, a Bash glob is simpler:

#!/usr/bin/env bash
set -euo pipefail

target_dir=${1:-.}
shopt -s nullglob

for file in "$target_dir"/*.txt; do
    printf 'Processing: <%s>\n' "$file"
done

nullglob prevents an unmatched pattern from being processed as the literal string *.txt. The quoted directory prefix ensures its spaces are not split.

A portable alternative for simple per-file operations is:

find "$target_dir" -type f -exec printf 'Processing: <%s>\n' {} +

Production Best Practices and Pitfalls

  • Avoid $(find ...): It loses the original argument boundaries and introduces command-substitution parsing problems.

  • Avoid parsing ls: Its output is intended for human reading and can be ambiguous or paginated.

  • Handle traversal errors: find reports permission and I/O failures to stderr. With set -o pipefail, its nonzero status causes the pipeline to fail.

  • Decide on symlink behavior: By default, find -type f does not process symlinks to regular files. Use the appropriate -H, -L, or -P behavior deliberately.

  • Use explicit status handling around destructive operations. In production, record processed counts and failures, and send structured logs rather than printing every filename.

  • Remember Bash differences: read -d '', nullglob, arrays, and pipefail are not supported by POSIX sh. Prefer find -exec when portability to systems such as Alpine’s default ash matters.

18

How do you use case statements in Bash?

A case statement in Bash matches a single value against multiple glob patterns and runs the commands attached to the first match, making it the right tool for branching on string values, command-line flags, or menu choices.

Read full answer

Applies toBash 5.x (also largely compatible with POSIX sh, with noted differences)

How It Works

The case keyword evaluates one word, then tests it against each pattern in order. The first matching branch runs, and execution continues after esac. Patterns are glob expressions, not regular expressions, so *, ?, and [...] work. Use | to list alternative patterns in one branch; extended globs such as @(a|b) or +(pattern) require shopt -s extglob.

case "$1" in
  start)
    echo "Starting service"
    ;;
  stop|restart)
    echo "Stopping or restarting"
    ;;
  *)
    echo "Unknown command: $1" >&2
    exit 1
    ;;
esac

Each branch ends with ;;. Use ;& to fall through to the next branch (Bash 4+) or ;;& to test subsequent patterns too. The *) catch-all should always be present so unexpected input fails loudly instead of silently doing nothing.

Pattern Matching Details

  • Patterns are tested branch by branch; the first matching branch wins.

  • Quote variables inside the test word ("$1") but do not quote patterns, or they become literal strings.

  • Case is sensitive by default. Set shopt -s nocasematch for case-insensitive matching.

  • Enable extended globs with shopt -s extglob to use @(a|b), +(pattern), etc.

Common Pitfalls

  • Forgetting ;; causes a syntax error.

  • Using regex syntax (^start$) silently never matches because case uses globs.

  • Leaving out the catch-all branch hides invalid input.

  • Comparing numbers with case works but (( )) or [[ ]] is clearer for arithmetic.

Code Example

#!/usr/bin/env bash
set -euo pipefail

action="${1:-}"

case "$action" in
  start)
    systemctl start myapp
    ;;
  stop)
    systemctl stop myapp
    ;;
  status)
    systemctl status myapp --no-pager
    ;;
  *)
    printf 'Usage: %s {start|stop|status}\n' "$0" >&2
    exit 2
    ;;
esac

Best Practices

  • Always include a *) default branch that prints usage and exits non-zero.

  • Quote the tested variable; never quote the patterns.

  • Prefer case over long if/elif chains when branching on a single string.

  • Keep branches short; delegate complex logic to functions.

  • Run scripts through ShellCheck to catch unquoted variables and missing defaults.

  • For POSIX sh portability, avoid ;&, ;;&, and shopt; stick to basic globs and ;;.

Comparison: case vs if/elif

Aspect case if/elif
Best for Single value, multiple string patterns Complex boolean conditions
Pattern type Glob Any test expression
Readability High for menus/flags High for ranges/logic
Fall-through ;& / ;;& (Bash 4+) Not supported

Production Notes

In production scripts, case statements commonly dispatch subcommands (start, stop, reload), parse environment names (prod, staging), or route log levels. Pair them with set -euo pipefail and explicit exit codes so failures surface in CI/CD pipelines and systemd units rather than being swallowed.

19

How are Bash functions defined and invoked?

In Bash, a function groups reusable commands under a name. Functions execute in the current shell context unless invoked in a subshell, and they communicate success or failure through an exit status.

Read full answer

Applies toBash 5.x

Defining and Calling Functions

Define a function with name() { ...; } and invoke it as a command using name or name "argument". Do not include parentheses at the call site.

Bash also accepts function name { ...; }. Unlike the name() form, the function keyword is not portable to POSIX sh.

Arguments and Variable Scope

  • $1, $2, …: Positional arguments passed to the function.

  • $@: Expands to all positional parameters as separate arguments.

  • $#: Number of positional parameters.

  • shift: Removes the first positional parameter and renumbers the others.

  • local: Creates or shadows a variable within the function, reducing accidental global-state changes.

  • Dynamic scoping: A local variable is visible to functions called by the current function and takes precedence over a global variable of the same name.

When forwarding every argument, use "$@":

run_command() {
    local command_name="$1"
    shift

    "$command_name" "$@"
}

This preserves argument boundaries and prevents unintended word splitting or pathname expansion.

Return Codes and Error Handling

A function returns the exit status of its last executed command unless return specifies another status.

  • return N: Ends the function with status N.

  • exit N: Ends the current shell or subshell, not merely the function.

  • Exit statuses range from 0 through 255; 0 conventionally means success.

With set -euo pipefail, set -e exits on an unhandled simple-command failure subject to documented exceptions, set -u rejects unset variables, and pipefail makes a pipeline fail when any component fails. When a function is invoked as an if or && condition, set -e is ignored for commands in the function body, so intermediate failures may not stop it. Validate inputs explicitly and return meaningful statuses.

Code Example

#!/usr/bin/env bash
set -euo pipefail

WORK_DIR=""

log() {
    local message="${1:-}"
    printf '[%s] %s\n' \
        "$(date -u +%Y-%m-%dT%H:%M:%SZ)" \
        "$message"
}

cleanup() {
    if [[ -n "$WORK_DIR" ]]; then
        rm -rf -- "$WORK_DIR"
    fi
}

greet() {
    local person="${1:-world}"
    printf 'Hello, %s!\n' "$person"
}

run_command() {
    local command_name="$1"
    shift

    "$command_name" "$@"
}

validate_port() {
    local port="${1:-}"

    if [[ ! "$port" =~ ^[0-9]+$ ]] ||
       (( port < 1 || port > 65535 )); then
        printf 'Invalid port: %s\n' "$port" >&2
        return 1
    fi

    return 0
}

validate_config() {
    local file="$1"

    if [[ ! -f "$file" ]]; then
        log "Configuration file not found: $file" >&2
        return 2
    fi

    if [[ ! -s "$file" ]]; then
        log "Configuration file is empty: $file" >&2
        return 3
    fi

    if [[ ! -r "$file" ]]; then
        log "Configuration file is not readable: $file" >&2
        return 4
    fi

    log "Configuration validated: $file"
}

main() {
    local config_file

    WORK_DIR="$(mktemp -d)"
    trap cleanup EXIT

    config_file="$WORK_DIR/app.conf"
    printf 'healthy\n' > "$config_file"

    greet "SRE"
    run_command printf '<%s>\n' 'argument with spaces' '*'
    validate_port 8080
    validate_config "$config_file"
}

main

Representative output:

Hello, SRE!
<argument with spaces>
<*>
[2026-03-10T14:25:31Z] Configuration validated: /tmp/tmp.K8s3dP/app.conf

Production and Portability Considerations

  • Validate dependencies: Check required commands, files, permissions, and connectivity before making changes.

  • Use bounded retries: Retry only transient failures, with a limit and delay; do not retry permanent validation errors indefinitely.

  • Add timeouts: Wrap network operations with timeout or the tool’s native timeout option.

  • Protect secrets: Avoid exposing credentials directly in command arguments, and remember that set -x can write secrets to logs.

  • Separate output streams: Send diagnostics and failures to stderr; reserve standard output for requested data or successful status output.

  • Design idempotently: Automation should safely handle repeated deployment, backup, or cleanup operations.

  • Preserve diagnostics: During incidents, retain logs and meaningful exit statuses so automation platforms can identify the failed stage.

Bash also supports arrays, process substitution, local, and export -f. POSIX sh has a smaller standardized feature set; name() definitions are portable, but local, arrays, process substitution, and exported functions should not be assumed to work everywhere.

Best Practices and Common Pitfalls

  • Quote expansions: use "$var", "$@", and "${array[@]}".

  • Prefer local for temporary variables to prevent global-state corruption.

  • Use return inside functions; reserve exit for top-level script termination.

  • Do not rely on set -e as a substitute for explicit error handling.

  • Avoid echo for variable output; use printf.

  • Check command and file existence before use.

  • Keep functions focused so failures have clear, testable statuses.

  • Run scripts through ShellCheck and test both success and failure paths before deployment.

20

How do Bash functions return values?

Bash functions can return a value through standard output and an exit status through return. In production scripts, use standard output for data and exit status for success or failure.

Read full answer

Applies toBash 5.x

Returning Data with Standard Output

A function commonly prints its result with printf. The caller captures that output using command substitution:

result=$(calculate_sum 10 20)

The result is assigned to a variable because command substitution executes the function in a subshell. Variable assignments made inside the function are therefore never visible to the caller, not even through a nameref or global variable. To return values through variables, call the function directly rather than inside $(...).

Prefer printf over echo: it handles formatting predictably and avoids implementation differences between shells.

Returning an Exit Status

Use return to stop a function and provide a status code:

  • return 0 indicates success.

  • A nonzero value indicates failure.

  • return without an argument returns the status of the most recently executed command.

  • Status codes are represented as integers in the range 0 through 255.

A caller can test the status directly:

if validate_config "$config_file"; then
    deploy "$config_file"
else
    printf 'Configuration validation failed\n' >&2
    exit 1
fi

When a function is called as a simple command, its exit status becomes the status of that command. This matters when set -e is enabled because a failing function can terminate the script.

Combining Data and Status

A robust function should print the requested value to standard output and reserve standard error for diagnostics. It should also return nonzero when the operation fails.

#!/usr/bin/env bash
set -euo pipefail

divide_numbers() {
    local numerator=$1
    local denominator=$2

    if (( denominator == 0 )); then
        printf 'Error: denominator must not be zero\n' >&2
        return 1
    fi

    printf '%s\n' "$((numerator / denominator))"
}

if result=$(divide_numbers 20 5); then
    printf 'Quotient: %s\n' "$result"
else
    printf 'Division failed\n' >&2
    exit 1
fi

Example successful output:

Quotient: 4

The if result=$(...) pattern is important with set -e: because the function call is part of a conditional, its expected nonzero status does not immediately terminate the script.

Key Behavior and Failure Modes

  • Command substitution captures stdout only: diagnostics written to standard error remain visible to the user or monitoring system.

  • Trailing newlines are removed: command substitution strips trailing newline characters from captured output.

  • Bash variables cannot store NUL bytes: use files or a structured format such as JSON when handling arbitrary binary data.

  • Function-local state should use local: prevents accidental modification of global variables.

  • Unquoted expansions are unsafe: always quote variable expansions such as "$result" and "$config_file".

  • A function has one exit status and one stdout stream: to return multiple values, use a nameref (declare -n) or a global variable with a directly called function (not inside command substitution), or print structured output such as JSON.

  • Use printf '%q' when constructing shell commands: it provides shell-safe quoting for values such as filenames with spaces.

Production Practices

  • Return 0 only when the function completed its requested operation successfully.

  • Return distinct nonzero statuses when callers need to distinguish failures, such as 1 for invalid input and 2 for an unavailable dependency.

  • Send errors to standard error with >&2 so command substitution does not mistake diagnostic text for a successful result.

  • Avoid returning formatted, multi-field text that callers must parse with awk or sed. For structured data, print JSON and parse it with a suitable tool.

  • Remember that return exits the current function. It does not by itself stop the parent script unless the function is invoked in a context that causes the script to exit.

Bash return is broadly supported, but the surrounding scripting features are not fully POSIX. A script using local, arrays, [[ ... ]], or process substitution should declare Bash explicitly with #!/usr/bin/env bash; a strict POSIX sh script should avoid those features.

21

How do local and global variables differ in Bash functions?

In Bash, variables assigned inside a function are global by default unless explicitly declared with local. A local variable exists only for the duration of the function call and temporarily shadows a global variable with the same name.

Read full answer

Applies toBash 5.x

Scope Fundamentals

Bash uses dynamic scoping for function-local variables. This differs from languages such as JavaScript or Python that generally use lexical scoping.

Behavior Local variable Global variable
Declaration local name=value Assignment outside a function, or declare -g inside one
Lifetime Until the function returns Until the shell exits or it is unset
Name conflict Shadows a global variable during the call Remains accessible after the function returns
Child processes Not exported by default Exported only when marked with export
Common use Temporary state, arguments, intermediate results Configuration, constants, shared counters

A function executes in the current shell, so assigning to a global variable inside it normally persists after the function returns. By contrast, a local variable is destroyed when the function returns, and any previous global value with the same name is restored.

Functions also receive their own positional parameters. After the function returns, the caller’s positional parameters are restored.

Dynamic Scoping

A local variable is visible to every function called by its declaring function. For example, if configure declares local region="eu-west-1" and then calls deploy, deploy can read region. If deploy assigns to region without declaring it local, it changes the nearest dynamically scoped variable, which in this case is configure’s local variable.

This behavior is convenient for small scripts but can make larger systems harder to reason about. Explicit parameters and return values make dependencies clearer.

Use declare -g when a global assignment inside a function is intentional, such as declare -g request_count=$((request_count + 1)).

Production Considerations

  • Isolate mutable state: Declare working variables with local to prevent accidental changes to deployment configuration, counters, paths, or credentials.

  • Return results explicitly: Write results to standard output and communicate failures through the exit status. Use return from a function rather than exit, which terminates the entire script.

  • Treat configuration as immutable: Mark constants with readonly, such as readonly APP_NAME="payments".

  • Export deliberately: export makes a global variable available to child processes. Local variables are not exported by default.

  • Check failures explicitly: set -euo pipefail catches many errors but does not replace explicit error handling for expected operational failures.

  • Avoid masking command status: Declare and assign separately because local output="$(command)" can hide the command’s failure status. Prefer declaring local output, then assigning with if ! output="$(fetch_metadata)"; then ...; fi.

Bash and POSIX Portability

local is a Bash feature and is not part of POSIX sh. Some shells, such as ksh, provide similar behavior, while dash does not. Portable scripts should use distinct variable names, positional parameters, or a subshell instead of relying on local.

For production Bash automation, prefer #!/usr/bin/env bash, set -euo pipefail, quoted expansions, printf, local function state, and ShellCheck-clean code.

22

How do Bash arrays store and expand elements?

Bash arrays are indexed or associative collections whose elements are strings. Indexed arrays use integer subscripts and iterate in subscript order, while associative arrays use non-empty string keys and have no defined iteration order. Quoted "${array[@]}" expansion preserves each element’s boundaries, spaces, and empty values.

Read full answer

Applies toBash 5.x

Array Types and Storage

Type Declaration Key Typical Use
Indexed array declare -a items=(...) Integer subscript Ordered lists, command results
Associative array declare -A config=(...) Non-empty string Maps, lookup tables

Bash stores indexed arrays as sparse collections and associative arrays using hash-table-like structures. Array elements are strings, even when they look numeric. An index can be skipped:

declare -a items=("first" "third")
items[5]="sixth"

"${#items[@]}" reports the number of assigned elements, not the highest index plus one.

Accessing and Modifying Elements

  • Read an element: "${items[0]}"

  • Read the highest-numbered occupied indexed element: "${items[-1]}"

  • Append: items+=("another-item")

  • Assign by index: items[2]="third-item"

  • Delete an element: unset 'items[2]'

  • Delete the entire array: unset items

  • List assigned indexes: "${!items[@]}"

  • List associative keys: "${!config[@]}"

For a dense indexed array, "${items[${#items[@]} - 1]}" is equivalent to "${items[-1]}". That count-based expression is not reliable for sparse arrays.

Associative array values are accessed by key:

declare -A limits=(
    [cpu]="500m"
    [memory]="1Gi"
)

printf '%s\n' "${limits[cpu]}"

Associative keys have no guaranteed iteration order, so sort them before producing deterministic output.

Expansion and Quoting

Quoting is critical because unquoted expansions undergo word splitting and pathname expansion:

declare -a values=("alpha" "beta gamma")
printf '<%s>\n' "${values[@]}"
  • "${values[@]}" expands each element as a separate argument while preserving spaces and empty elements.

  • "${values[*]}" joins elements using the first character of IFS; an empty IFS produces no separator.

  • ${#values[@]} gives the element count.

  • "${values[@]:1:2}" selects two elements beginning at index 1.

  • "${!values[@]}" returns assigned indexes.

Under set -u, accessing a missing associative key can terminate the script. Use a default when absence is expected:

declare -A limits=()
replicas="${limits[replicas]:-1}"

Passing Arrays to Functions

For an indexed array, Bash does not pass it as one function argument. Pass its elements individually and reconstruct a local indexed array from "$@":

process_items() {
    local -a selected=("$@")
    local item

    for item in "${selected[@]}"; do
        printf 'Processing %s\n' "$item"
    done
}

For an associative array, preserve the key/value relationship; "${map[@]}" passes values only. Pass explicit keys and values or use a Bash nameref (local -n) carefully.

Use local -a for indexed arrays and local -A for associative arrays. Without local, assignments can modify a global variable.

Code Example

#!/usr/bin/env bash
set -euo pipefail

declare -a services=("api" "worker")
services+=("scheduler")

declare -A endpoints=(
    [api]="https://api.example.com"
    [worker]="https://worker.example.com"
    [scheduler]="https://scheduler.example.com"
)

show_service() {
    local -a selected=("$@")
    local service

    for service in "${selected[@]}"; do
        printf '%s -> %s\n' "$service" "${endpoints[$service]}"
    done
}

printf 'count=%d\n' "${#services[@]}"
printf 'second=%s\n' "${services[1]}"

printf 'slice='
printf '<%s> ' "${services[@]:1:2}"
printf '\n'

show_service "${services[@]}"

Example output:

count=3
second=worker
slice=<worker> <scheduler>
api -> https://api.example.com
worker -> https://worker.example.com
scheduler -> https://scheduler.example.com

Production Best Practices

  • Use set -euo pipefail: set -e exits after a failed simple command except in exempt contexts such as conditionals, &&/|| lists, and loop conditions; set -u errors on unset variables; pipefail makes a pipeline return nonzero when any component fails. Still check failures whose meaning is important.

  • Always quote expansions: "${array[@]}" prevents word splitting and unintended pathname expansion.

  • Handle empty input explicitly: Check "${#array[@]}" before accessing an element. Bash 5.x permits expansion of empty arrays under set -u, but missing indexed or associative elements still require care.

  • Do not use eval: It can reinterpret array contents as shell syntax and introduce injection vulnerabilities.

  • Protect external commands: command -- "${array[@]}" only ends option parsing for command itself. Use program -- "${array[@]}" when the target program documents --, or validate and reject values beginning with -.

  • Avoid arrays for structured data: Bash variables cannot contain NUL bytes, and arrays cannot be exported as environment variables. Use jq for JSON or a dedicated serialization format when exchanging structured data.

  • Remember portability limits: POSIX sh has no arrays. Portable scripts commonly use positional parameters with set -- or an external tool such as awk.

  • Prefer mapfile for line-oriented input: mapfile -t lines < file reads each newline-delimited record into a separate element without invoking read in a loop.

23

How do Bash associative arrays store and retrieve key-value pairs?

Associative arrays in Bash are hash-table-style arrays that map string keys to string values. They are declared with declare -A, support key-based lookup, and do not preserve insertion order.

Read full answer

Applies toBash 5.x

How associative arrays work

  • Declaration: declare -A map creates an associative array.

  • Key-value storage: map["key"]="value" associates a string key with a value.

  • Lookup: map["key"] retrieves the value.

  • Key expansion: "${!map[@]}" expands to all keys, while "${map[@]}" expands to all values.

  • Internal implementation: Bash uses a hash-table-style index, making key lookup efficient for large collections, but it does not expose a deterministic key order.

Unlike an indexed array, an associative array does not use numeric positions. It is appropriate for configuration data, metadata, routing tables, and mappings between names and values.

Declaration and lookup rules

declare -A settings

settings["content type"]="application/json"
settings["retry count"]="3"

printf '%s\n' "${settings["content type"]}"
printf '%s\n' "${settings["retry count"]}"

Important details include:

  • Keys are strings. An empty string is a valid key, although empty keys are usually best avoided for clarity.

  • Values are stored as strings, even when they look numeric.

  • Quote keys and values, especially when they contain spaces, glob characters, or shell metacharacters.

  • declare -A map=([key]="value") is a common inline initialization form.

  • Inside functions, local -A map creates a function-local associative array.

A missing key requires careful handling. With set -u, directly expanding an unset key can terminate the script:

set -u

if [[ -z "${settings["timeout"]+present}" ]]; then
    printf 'The timeout setting is missing\n' >&2
else
    printf '%s\n' "${settings["timeout"]}"
fi

The +present test distinguishes an unset key without triggering an unbound-variable error. ${map[key]-default} supplies a default only when the key is unset, whereas ${map[key]:-default} also uses the default when the value is empty. In Bash 4.2 and later, [[ -v map[key] ]] is also a clear way to test whether a key exists.

Iteration and production use

Iterate over keys and retrieve each corresponding value:

declare -A endpoints=(
    [us-east-1]="https://api-us-east-1.example.com"
    [eu-west-1]="https://api-eu-west-1.example.com"
)

for region in "${!endpoints[@]}"; do
    printf '%s -> %s\n' "$region" "${endpoints[$region]}"
done

The iteration order is unspecified, so sort keys explicitly when output order matters:

mapfile -t sorted_regions < <(
    printf '%s\n' "${!endpoints[@]}" | sort
)

for region in "${sorted_regions[@]}"; do
    printf '%s -> %s\n' "$region" "${endpoints[$region]}"
done

For text-processing pipelines, keep the associative array in Bash and use tools such as grep, sed, and awk for transformation. For example, awk -F '\t' can parse tab-separated records, while grep -F -- performs fixed-string matching without interpreting the pattern as a regular expression. Avoid eval, and avoid parsing ls output; use find, printf, and null-delimited input when filenames may contain whitespace or newlines.

Code Example

This Bash 5 script demonstrates declaration, arguments, conditionals, functions, loops, exit codes, and safe error handling:

#!/usr/bin/env bash
set -euo pipefail

declare -A endpoints=(
    [us-east-1]="https://api-us-east-1.example.com"
    [eu-west-1]="https://api-eu-west-1.example.com"
)

list_regions() {
    local region

    for region in "${!endpoints[@]}"; do
        printf '%s\n' "$region"
    done
}

lookup_endpoint() {
    local region="$1"

    if [[ ! "$region" =~ ^[a-z]{2}-[a-z]+-[0-9]+$ ]]; then
        printf 'Invalid region: %s\n' "$region" >&2
        return 2
    fi

    if [[ -z "${endpoints[$region]+present}" ]]; then
        printf 'Unknown region: %s\n' "$region" >&2
        return 1
    fi

    printf 'Region %s uses %s\n' "$region" "${endpoints[$region]}"
}

case "${1:-}" in
    --list)
        list_regions
        ;;
    "")
        printf 'Usage: %s <region> | --list\n' "$0" >&2
        exit 64
        ;;
    *)
        lookup_endpoint "$1"
        ;;
esac

Example lookup output:

Region us-east-1 uses https://api-us-east-1.example.com

The --list output may list regions in either order.

Best practices, portability, and failure modes

  • Quote every expansion: "${value}", "${array[$key]}", and "${!array[@]}" prevent word splitting and pathname expansion.

  • Use set -euo pipefail: set -e improves failure handling, set -u catches unset variables, and pipefail detects failures earlier in pipelines. Still, explicitly check critical commands because set -e has conditional exceptions.

  • Validate external input: Treat command-line arguments and file contents as untrusted. Validate keys before using them as associative-array subscripts.

  • Return meaningful exit codes: For example, 64 for usage errors, 2 for invalid input, and 1 for a missing resource.

  • Do not rely on ordering: Sort keys when deterministic output is required.

  • Use Bash explicitly: The script requires Bash because POSIX sh does not provide associative arrays, [[ ... ]], local, or Bash arrays. For portable /bin/sh automation, use awk, sed, cut, or another external tool instead.

  • Remember process scope: Associative arrays exist only in the current shell process; they are not a persistent database or a replacement for structured configuration formats such as JSON or YAML.

24

How does Bash evaluate integer arithmetic expressions?

Arithmetic expansion is Bash’s built-in mechanism for integer calculations. $((expression)) expands to an integer-valued word, while ((expression)) is an arithmetic command whose exit status reflects whether the result is zero.

Read full answer

Applies toBash 5.0-5.3 on 64-bit Linux

How Evaluation Works

Bash supports two related constructs:

  • $((expression)) performs arithmetic expansion and substitutes the result into the surrounding word.

  • ((expression)) performs arithmetic evaluation as a command. Its exit status is 0 when the result is nonzero and 1 when the result is zero.

  • Arithmetic expressions normally reference variables without the $ prefix. For example, with x=3 and y=x, $((x + y)) expands to 6.

  • A referenced variable’s value is itself evaluated as an arithmetic expression. Thus, if y='3 + 4', $((y)) expands to 7.

  • Shell expansions embedded directly in the expression occur before final arithmetic evaluation. Parentheses should be used when precedence would otherwise be unclear.

  • Without set -u, an unset arithmetic variable normally has the value 0; with set -u, referencing it is an error.

Operators and Numeric Behavior

Syntax Behavior
$((expression)) Expands the expression to an integer string
((expression)) Evaluates the expression and sets its exit status
+, -, * Addition, subtraction, and multiplication
/, % Integer division and remainder
** Exponentiation
<<, >> Bit shifting
&, ‘ ,^,~‘ Bitwise operations
==, !=, <, > Comparisons returning 1 or 0

| &&, ||, ?: | Short-circuit logical and conditional operators |

Integer division truncates toward zero, and the remainder has the same sign as the dividend. Bash uses fixed-width signed integers—64-bit on the stated platform—so values outside the representable range can overflow or wrap.

Hexadecimal and octal constants are supported. For example, $((0x10 + 010)) evaluates to 24, because 0x10 is 16 and 010 is octal 8. The 10#number form forces base-10 interpretation, so 10#08 is 8 rather than an invalid octal value.

Under the Hood and Portability

Bash evaluates arithmetic internally; $((...)) does not start an external process such as bc or awk. This makes it inexpensive for deployment checks, health probes, and resource calculations.

POSIX sh supports the $((...)) container, but not every Bash operator or extension. In particular, ** and base notation such as 10#... are not required by POSIX. Other Bash-specific constructs include the ((...)) command, [[ ... ]], local, arrays, and set -o pipefail.

Use $((...)) rather than the older external expr command. Use bc, awk, or Python for floating-point or arbitrary-precision calculations.

Code Example

This complete Bash script validates decimal syntax, handles a zero divisor explicitly, and demonstrates arithmetic expansion:

#!/usr/bin/env bash
set -euo pipefail

is_uint() {
    [[ $1 =~ ^[0-9]+$ ]]
}

calculate() {
    local left="$1"
    local right="$2"

    if ! is_uint "$left" || ! is_uint "$right"; then
        printf 'error: operands must be non-negative decimal integers\n' >&2
        return 2
    fi

    # Force base 10 so values such as 08 are not interpreted as octal.
    left=$((10#$left))
    right=$((10#$right))

    if (( right == 0 )); then
        printf 'error: division by zero\n' >&2
        return 3
    fi

    local sum
    local product
    local quotient
    local remainder

    sum=$((left + right))
    product=$((left * right))
    quotient=$((left / right))
    remainder=$((left % right))

    printf 'sum=%d\n' "$sum"
    printf 'product=%d\n' "$product"
    printf 'quotient=%d\n' "$quotient"
    printf 'remainder=%d\n' "$remainder"
}

main() {
    if (( $# != 2 )); then
        printf 'usage: %s LEFT RIGHT\n' "${0##*/}" >&2
        return 2
    fi

    calculate "$@"
}

main "$@"

Running ./arithmetic.sh 10 3 produces:

sum=13
product=30
quotient=3
remainder=1

Failure Modes and Security

  • Division by zero: This is an arithmetic evaluation error, not merely an arithmetic command returning status 1. Bash prints a diagnostic to stderr and terminates a noninteractive shell even without set -e.

  • set -e pitfall: A standalone ((count > limit)) terminates a script under errexit when the comparison is false. Put such tests in an if condition or append an explicit || handler.

  • Unset variables: set -u exposes accidental use of unset arithmetic variables. Initialize operands explicitly and use ${value:-default} where appropriate.

  • Invalid expressions: Syntax errors are fatal to a noninteractive Bash shell because they are evaluation errors rather than ordinary command failures.

  • Untrusted input: Do not place unvalidated input directly into arithmetic expressions or use eval. Validate the expected decimal format and acceptable range first.

  • Leading zeros: Arithmetic interprets leading-zero values as octal. Use 10#$value after validating that the value contains only decimal digits.

Production Best Practices

  • Use set -euo pipefail in automation scripts, but do not treat it as a substitute for explicit error handling.

  • Quote variable expansions and pass arguments as separate parameters.

  • Validate both syntax and business limits; even representable operands can overflow when multiplied or added.

  • Prefer integer scaling for percentages, such as $((used_bytes * 100 / total_bytes)), while checking for intermediate multiplication overflow.

  • Send diagnostics to stderr so monitoring systems can distinguish failures from normal output.

  • Return distinct nonzero statuses for invalid input and invalid operational state, and never silently continue after an arithmetic evaluation failure.

25

How does Bash read capture user input from a terminal?

In Bash, read is a shell builtin that reads one line of input into one or more variables. For reliable interactive input, use IFS= read -r to preserve whitespace and treat backslashes literally.

Read full answer

Applies toBash 5.x on Linux and Unix-like systems

How read Works

The basic form is:

IFS= read -r variable
  • IFS= prevents leading, trailing, and repeated whitespace from being stripped or split.

  • -r prevents backslashes from escaping characters; without it, read interprets backslash sequences.

  • If variable names are supplied, Bash splits the line according to IFS.

  • If no variable name is supplied, the line is stored in REPLY.

  • Multiple variables receive successive fields, with the final variable receiving the remaining fields.

For example:

IFS= read -r username
IFS=, read -r environment region

The second command expects input such as production,eu-west-1. Bash-style comma-separated values are not the same as CSV: fields containing commas or quotes require a proper CSV parser.

Useful Options

Option Behavior
-r Reads input without interpreting backslash escapes.
-p prompt Displays prompt before reading input; this is a Bash extension.
-s Suppresses terminal echo, making it suitable for passwords and tokens.
-t timeout Stops waiting after the specified number of seconds.
-n nchars Returns after reading nchars, rather than waiting for a newline.
-N nchars Reads exactly nchars, ignoring the input delimiter.
-a array Stores the fields in an indexed array.
-d delimiter Uses delimiter instead of newline to terminate the input.

A successful read returns exit status 0. It returns a non-zero status for EOF, timeout, an invalid file descriptor, or another read failure. A timeout produces a status greater than 128.

Reading Files and Standard Input

read is commonly used to process files or redirected input:

while IFS= read -r line; do
  printf '%s\n' "$line"
done < input.txt

The || [[ -n $line ]] variant also processes a final line that does not end with a newline:

line=''
while IFS= read -r line || [[ -n $line ]]; do
  printf '%s\n' "$line"
done < input.txt

For log or configuration processing, read only retrieves each line. Use grep to select matching lines, sed to substitute text, and awk for field- or record-oriented processing. Prefer jq or yq for structured JSON or YAML instead of parsing them with Bash variables.

Code Example

This interactive script uses safe quoting, validation, a function, explicit error handling, and distinct exit codes:

#!/usr/bin/env bash
set -euo pipefail

read_service() {
  local service=''

  if ! IFS= read -r -p 'Service name: ' service; then
    printf '\nNo input received.\n' >&2
    return 1
  fi

  if [[ ! $service =~ ^[a-z][a-z0-9-]{0,62}$ ]]; then
    printf 'Invalid service name: %s\n' "$service" >&2
    return 2
  fi

  printf 'Validated service: %s\n' "$service"
}

if read_service; then
  exit 0
else
  status=$?
  exit "$status"
fi

Example output:

Service name: api
Validated service: api

This script processes a file supplied as its first argument and checks that the argument is a readable regular file:

#!/usr/bin/env bash
set -euo pipefail

if (( $# != 1 )); then
  printf 'Usage: %s FILE\n' "$0" >&2
  exit 64
fi

if [[ ! -f $1 || ! -r $1 ]]; then
  printf 'Not a readable regular file: %s\n' "$1" >&2
  exit 66
fi

line=''
while IFS= read -r line || [[ -n $line ]]; do
  if [[ -z $line || $line == \#* ]]; then
    continue
  fi

  printf '<%s>\n' "$line"
done < "$1"

Production and Portability Guidance

  • Do not rely on a bare read with set -e. EOF or timeout is a normal outcome in pipelines. Put read in an if, while, or until condition, or handle its failure explicitly.

  • Avoid interactive input in CI/CD and services. Accept configuration through validated arguments, environment variables, files, or standard input so automation cannot hang waiting for a terminal.

  • Use timeouts where appropriate. read -t prevents an unattended process from waiting indefinitely.

  • Protect secrets. read -s prevents terminal echo, but do not log the resulting value. Avoid passing secrets as command-line arguments because they may appear in process listings.

  • Do not execute input. read -r preserves backslashes, but input is still data, not code; never pass it to eval or execute it as shell code.

  • Bash is not POSIX sh. read -r is broadly available, but read -p, arrays, [[ ... ]], and other Bash extensions are not portable. In a POSIX sh script, use printf for prompts and avoid Bash-only options.

26

How does getopts parse Bash command-line options?

Bash’s getopts builtin parses short command-line options such as -v, -o output, and -h. It processes options from the script’s positional parameters, stores option arguments in OPTARG, and leaves non-option arguments for the script to handle.

Read full answer

Applies toBash 5.x (Linux and other Unix-like systems)

How It Works

getopts is invoked with three arguments:

getopts optstring option_variable
  • Optstring: Defines the supported options.

  • Option variable: Receives the current option letter, such as h or v.

  • OPTARG: Contains the argument for the current option, when applicable.

  • OPTIND: Tracks the next positional parameter to examine. getopts updates it automatically.

For example, this optstring defines:

:hvo:
  • h — no argument

  • v — no argument

  • o — requires an argument

  • The leading : — causes a missing option argument to be reported as : rather than ?

A while getopts ... loop continues until all options have been processed or an invalid option is encountered.

Code Example

#!/usr/bin/env bash
set -euo pipefail

usage() {
    printf 'Usage: %s [-h] [-v] [-o OUTPUT] [--] [TARGET...]\n' \
        "${0##*/}"
}

output=""
verbose=false

while getopts ":hvo:" option; do
    case "$option" in
        h)
            usage
            exit 0
            ;;
        v)
            verbose=true
            ;;
        o)
            output=$OPTARG
            ;;
        :)
            printf 'Error: option -%s requires an argument\n' "$OPTARG" >&2
            usage >&2
            exit 2
            ;;
        \?)
            printf 'Error: unknown option -%s\n' "$OPTARG" >&2
            usage >&2
            exit 2
            ;;
    esac
done

# Remove the options that getopts processed.
shift "$((OPTIND - 1))"

if (($# == 0)); then
    usage >&2
    exit 2
fi

if [[ "$verbose" == true ]]; then
    printf 'Deploying to %d target(s)\n' "$#"
fi

for target in "$@"; do
    printf 'output=%s target=%s\n' "$output" "$target"
done

Example usage and output:

$ ./deploy.sh -v -o /tmp/release.tar -- web-01 web-02
Deploying to 2 target(s)
output=/tmp/release.tar target=web-01
output=/tmp/release.tar target=web-02

Option and Argument Rules

  • getopts supports short options, not GNU-style long options such as --output.

  • Options normally appear before positional arguments. Parsing stops at the first non-option argument.

  • -- explicitly ends option processing, allowing values beginning with - to be treated as operands.

  • An option requiring an argument obtains it from OPTARG.

  • An unknown option produces ?; with the leading : in the optstring, a missing argument produces :.

  • The loop itself must use while getopts ...; do not manually increment OPTIND.

Error Handling and Security

  • Use set -euo pipefail in production scripts, but still handle getopts error cases explicitly. A while condition does not normally trigger errexit when getopts simply finishes parsing.

  • Quote every expansion, especially "$OPTARG", "$@", and "$#".

  • Avoid eval; it can reinterpret user-controlled input as shell code.

  • Do not pass passwords or tokens directly as command-line arguments when avoidable, because other users may inspect the process list. Use an environment variable, protected file, or secret manager instead.

  • Return conventional exit codes: 0 for success, 2 for invalid usage, and application-specific nonzero codes for operational failures.

Bash and POSIX Differences

getopts is available in POSIX sh, so the basic parsing pattern is portable. However, this example also uses Bash features such as [[ ... ]] and pipefail, so it is not POSIX sh code. For portable automation, use POSIX conditionals, avoid pipefail, and rely only on the POSIX getopts behavior.

27

What is Bash command substitution, and how does it differ from backticks?

Command substitution runs a command and replaces the substitution expression with its standard output. Bash’s $(...) is the preferred form because its quoting and nesting rules are clearer; backticks are a legacy form that remains supported but has more complicated escaping rules.

Read full answer

Applies toGNU Bash 5.x and POSIX.1-2024 `sh`

How It Works

Command substitution captures a command’s standard output:

  • $(command) executes command.

  • Its standard output becomes text used by the surrounding command or assignment.

  • Standard error is not captured unless it is explicitly redirected.

  • Trailing newline characters are removed from the captured output.

  • Bash variables cannot represent NUL bytes, so NUL bytes in captured output are not preserved.

For example, timestamp="$(date -u +'%Y-%m-%dT%H:%M:%SZ')" stores a UTC timestamp in timestamp.

$(...) Versus Backticks

Both forms perform command substitution, but $(...) is preferred.

Behavior $(...) Backticks
Readability Clear delimiters Visually noisy
Nested substitution Supports direct nesting Nested backticks require escaping
Quoting Balanced quotes are handled predictably Has special legacy backslash rules
ShellCheck Does not trigger SC2006 SC2006 recommends replacing them with $(...)
Recommendation Standard modern choice Use only for legacy compatibility

POSIX sh supports both forms, but $(...) is easier to read, quote, and maintain. For example, $(printf '%s' "$(hostname)") nests a substitution without the fragile escaping required for nested legacy backticks.

Quoting and Expansion

Always quote a substitution when its output should remain one argument, such as files="$(find /var/log -maxdepth 1 -type f)". Quoting preserves whitespace and prevents word splitting and pathname expansion.

Without quotes, word splitting and pathname expansion can occur when the substitution is used as an unquoted command argument. This may produce unexpected arguments or expand filenames containing characters such as *, ?, or [.

Assignment is a common exception: value="$(command)" does not split the captured output into multiple words or apply pathname expansion to it.

Failure Handling and Exit Codes

Under set -euo pipefail, a simple assignment whose only operation is command substitution receives the substitution’s exit status. Embedding it in another command can hide that failure. For example, in printf '%s\n' "$(some-tool)", the successful status of printf masks a failure by some-tool.

A common Bash pitfall is local output="$(some-tool)": local can mask the substitution’s exit status. Inside a function, declare and assign separately when failure must be detected: use local output, followed by output="$(some-tool)".

The set -o pipefail option is a Bash feature, not a POSIX sh option.

Production Best Practices

  • Quote substitutions: Use "$(command)" when the output must remain one argument; use arrays or mapfile when capturing multiple records.

  • Do not use eval: Never build a command string from untrusted input. Pass values as separately quoted arguments instead.

  • Handle stderr explicitly: For example, output="$(tool 2>>"$error_log")" captures output while preserving diagnostics in a log.

  • Check exit status: Use an assignment-only form or test the command explicitly when failures matter.

  • Avoid unnecessary sudo: Perform privileged operations only where required, and validate paths and input.

  • Preserve diagnostics: Capturing standard output without handling standard error can make incident diagnosis harder.

  • Prefer $(...) consistently: It avoids the quoting problems associated with legacy backticks and does not trigger ShellCheck SC2006.

28

What is process substitution in Bash?

Process substitution is a Bash feature that lets a command’s input or output be treated as a file path, using <(command) to provide readable input and >(command) to provide writable output.

Read full answer

Applies toBash 5.x (also available in ksh and zsh; not POSIX sh)

How It Works

  • <(list) runs list and connects its standard output to a file-like path, such as /dev/fd/63 or a named pipe.

  • >(list) runs list and connects its standard input to a writable file-like path.

  • The shell passes that generated path as an argument to another command.

  • The substituted command runs asynchronously, so the main command can consume or produce data while it runs.

Common Uses

  • Compare transformed outputs without creating temporary files, such as diff <(sort a.txt) <(sort b.txt).

  • Feed multiple generated inputs to a command that expects filenames.

  • Avoid the subshell problem when reading command output with while read ...; do ...; done < <(find ...).

  • Send a command’s output to several consumers with tee >(grep ERROR > errors.log).

Pitfalls and Failure Modes

  • The exit status of the substituted process is not part of the main command’s status, so set -e and pipefail do not catch its failure.

  • With >(...), the main command may finish before the consumer has processed all data, causing truncated or lost output.

  • Large FIFO-based transfers can block if the reader and writer are not coordinated.

  • Do not quote the process substitution construct itself; quote variables and paths inside it.

Portability

  • Process substitution is a Bash, ksh, and zsh feature. POSIX sh does not support it.

  • In portable scripts, use temporary files, named pipes created with mkfifo, or standard pipelines instead.

Best Practices

  • Use process substitution to avoid temporary files and unwanted subshells.

  • Do not rely on it for critical error propagation; check important subprocesses separately.

  • Prefer while read ... done < <(command) over command | while read ... when variables must persist.

  • Run scripts through ShellCheck to catch quoting and portability issues.

29

How do pipes and redirection work in Bash scripts?

Pipes connect the standard output of one command to the standard input of another, while redirection changes where a command’s file descriptors read or write. Together, they are the core mechanisms for composing commands and controlling data flow in a shell script.

Read full answer

Applies toBash 5.x (Linux, macOS with Homebrew Bash)

How Pipes and Redirection Work

A pipe is an anonymous, kernel-managed byte stream. Bash starts the commands as separate processes, connects the left command’s standard output to the right command’s standard input, and waits for the pipeline to finish. Linux pipes typically have a 64 KiB default capacity, allowing producer and consumer to run concurrently.

  • cmd1 | cmd2 connects stdout of cmd1 to stdin of cmd2.

  • cmd1 |& cmd2 also connects stderr of cmd1 into the pipe.

  • |& is Bash-specific; POSIX sh uses cmd1 2>&1 | cmd2.

Redirections are processed from left to right before the command runs.

Operator Meaning
> file Truncate the file and write stdout to it
>> file Append stdout to the file
< file Read stdin from the file
2> file Redirect stderr to the file
&> file Redirect stdout and stderr to the file
2>&1 Duplicate stderr onto stdout’s current target
1>&2 Send stdout to stderr
> /dev/null 2>&1 Discard stdout and stderr
<> file Open the file for reading and writing on fd 0

Order matters: cmd > file 2>&1 redirects stderr to the file because stdout already points there. In cmd 2>&1 > file, stderr still points to the terminal when it is duplicated.

File Descriptors and Process Substitution

Bash can open custom file descriptors with exec:

  • exec 3> /var/log/app.log opens fd 3 for writing.

  • echo "msg" >&3 writes to it.

  • exec 3>&- closes it.

This keeps stdout available for machine-readable data while diagnostics go to a log.

Process substitution exposes command streams as file-like paths:

  • diff <(sort a.txt) <(sort b.txt) compares two sorted streams.

  • tee >(gzip > out.gz) copies stdin to stdout and to a compressor.

The exit status of a process-substitution command is not normally propagated through the command consuming its path.

Exit Status, pipefail, and SIGPIPE

By default, a pipeline’s status is the status of its last command, so an upstream failure can be hidden. set -o pipefail makes the pipeline return the rightmost nonzero stage status when any stage fails.

set -e normally exits when such a pipeline returns nonzero, except when it is used as a condition or in another Bash context that exempts failures from errexit.

PIPESTATUS is a Bash array containing each stage’s status. Capture it immediately because another command can replace it:

cmd1 | cmd2
statuses=("${PIPESTATUS[@]}")

If a downstream command exits early, such as head, the producer can receive SIGPIPE and exit with status 141. This is expected in some pipelines and should not automatically be ignored.

Code Example

#!/usr/bin/env bash
set -euo pipefail

log_file="$(mktemp)"
archive_file="$(mktemp)"
trap 'rm -f "$log_file" "$archive_file"' EXIT

# Keep stdout available for data; send diagnostics to fd 3.
exec 3>"$log_file"

# tar's stdout enters the pipe; gzip writes compressed data to archive_file.
tar -czf - /etc/hosts 2>&3 | gzip -c >"$archive_file"

# Process substitution exposes each stream as a file-like path.
if diff -q \
    <(printf 'b\na\n' | sort) \
    <(printf 'a\nb\n') >/dev/null; then
  printf 'streams match\n' >&3
fi

# Capture stage statuses immediately. grep intentionally finds no match.
set +e
printf 'error\n' | grep -i fatal | sort | uniq -c
statuses=("${PIPESTATUS[@]}")
set -e
printf 'grep=%d sort=%d uniq=%d\n' \
  "${statuses[0]}" "${statuses[1]}" "${statuses[2]}" >&3

exec 3>&-

Common Pitfalls

  • Subshell scope: Every stage of a pipeline normally runs in a subshell, so variable assignments do not propagate. Bash’s lastpipe option can change this for the final stage when job control is disabled.

  • Process-substitution failures: diff <(false) other can succeed because diff does not receive false‘s status.

  • Truncation and permissions: > truncates the target before the command runs. Redirection-created files use the current umask.

  • Concurrent writes: PIPE_BUF applies to pipes, not regular files. >> opens files with O_APPEND, but separate writes can still interleave; use flock for coordinated records or read-modify-write operations.

Best Practices

  • Always quote filenames in redirections: >"$file".

  • Use set -o pipefail in production Bash scripts and handle expected failures explicitly.

  • Use |&, &>, process substitution, arrays, PIPESTATUS, lastpipe, and pipefail only in Bash-specific scripts; POSIX sh requires constructs such as 2>&1 |.

  • Prefer structured output such as --json or --format over parsing human-readable text.

  • Add timeout around potentially hanging stages and use pv when throughput visibility is useful.

  • Close custom descriptors and avoid leaking them into long-lived child processes.

  • For backup pipelines, verify the resulting artifact, monitor freshness and failures, and test restores rather than trusting pipeline status alone.

30

How do you redirect both stdout and stderr to a file in Bash?

To send both stdout and stderr to the same file in Bash, use command > file 2>&1 (portable) or command &> file (Bash shorthand). The order of redirections matters: 2>&1 must come after the stdout redirection so stderr is pointed at the already-opened file descriptor.

Read full answer

Applies toBash 5.x (portable `> file 2>&1` also works in POSIX sh)

How It Works

  • File descriptor 1 is stdout, 2 is stderr.

  • > file opens file on fd 1 (truncating it).

  • 2>&1 makes fd 2 a duplicate of fd 1, so both streams write to the same open file description.

  • &> file is Bash syntax sugar for the same thing; &>> file appends.

  • > file 2>&1 is the POSIX-portable form and is what you should use in scripts that may run under sh.

Common Variants

Goal Command Notes
Overwrite file with both streams cmd > out.log 2>&1 Portable; truncates
Append both streams cmd >> out.log 2>&1 Portable
Bash shorthand (overwrite) cmd &> out.log Bash-only
Bash shorthand (append) cmd &>> out.log Bash-only
Discard both cmd > /dev/null 2>&1 Classic idiom
Split streams to two files cmd > out.log 2> err.log Separate capture

Order Matters (Pitfall)

cmd 2>&1 > file does not do what you want: stderr is first duped to the current stdout (the terminal), then stdout is redirected to the file. stderr still goes to the terminal. Always put 2>&1 after > file.

Production Considerations

  • Atomicity and buffering: stdout is typically block-buffered when not a TTY, stderr is unbuffered. Interleaving in the file may not reflect true execution order. Use stdbuf -oL or line-buffered logging when ordering matters for incident analysis.

  • Log rotation: redirecting to a plain file bypasses logrotate/journald. For long-running services, prefer writing to journald or a logging daemon; if you must redirect, use a wrapper like rotatelogs or reopen on SIGHUP.

  • Permissions: the shell opens the file as the invoking user before execing the command. Ensure the directory is writable and not world-writable (sticky bit issues). Avoid redirecting to paths under /tmp without mktemp.

  • set -euo pipefail: redirection failures (e.g., permission denied) cause the shell to exit under set -e, which is usually what you want. Note that cmd | tee pipelines need pipefail to surface cmd‘s exit code.

  • Capturing exit codes: cmd > out.log 2>&1; rc=$? preserves the status. With set -e, a failing command aborts before you can inspect $? unless guarded with if cmd > out.log 2>&1; then ....

Code Example

#!/usr/bin/env bash
set -euo pipefail

log=/var/log/myjob.log
# Append both streams; mkdir -p the dir first in real scripts.
myjob --config /etc/myjob.conf >> "$log" 2>&1

# Capture output while still showing it on the terminal:
myjob --config /etc/myjob.conf 2>&1 | tee -a "$log"

Best Practices

  • Prefer > file 2>&1 over &> in any script with a #!/bin/sh shebang or that may be sourced by dash/ash.

  • Quote the filename: > "$log" not > $log.

  • Use exec >file 2>&1 at the top of a script to redirect all subsequent output of the script itself.

  • For systemd units, prefer StandardOutput=journal/StandardError=journal rather than shell redirection, so logs integrate with journalctl.

  • When you need both a file and the terminal, use tee with 2>&1 placed before the pipe.

31

What are here documents and here strings in Bash?

A here document and a here string are Bash redirections that supply text to a command’s standard input. A here document uses a delimited block of lines, while a here string supplies an expanded word and automatically appends a newline.

Read full answer

Applies toBash 5.x

How They Feed Standard Input

Both forms let a command receive data without the caller first creating a separate file:

  • Here document: Uses << followed by a delimiter. Bash reads subsequent lines until it encounters the delimiter by itself.

  • Here string: Uses <<< followed by a word. Bash expands that word and supplies the result to the command.

  • Standard input: The receiving command can consume the entire input or process it incrementally. For example, cat displays everything, while read consumes one line at a time.

Bash may implement these redirections using pipes or temporary files, depending on the implementation and circumstances. Large payloads therefore should not be assumed to remain exclusively in memory.

Syntax and Expansion

Feature Here document Here string
Syntax command <<EOF command <<<"$text"
Input shape Zero or more lines One expanded word or string
Expansion Performed unless the delimiter is quoted Performed on the supplied word
Trailing newline Preserves the body’s line structure Appends a newline
POSIX support Supported Not supported

A here-document delimiter must appear by itself on its own line. Its two common forms are:

  • Unquoted delimiter, such as <<EOF: Parameter, command, arithmetic, and backslash expansions are performed in the body.

  • Quoted delimiter, such as <<'EOF': No expansions are performed in the body. This is safer for fixed templates containing $, backticks, or shell-like syntax.

The <<- variant removes leading tab characters from both the body and the delimiter line. It does not remove leading spaces, which makes it useful for indented scripts.

For variable data, <<<"$message" is the conventional form and makes the intent to preserve the exact value explicit. Bash does not perform word splitting or pathname expansion on a here-string word. Quoting the expression itself also prevents expansion: <<<'$HOME' passes the literal text $HOME, whereas <<<"$HOME" expands the variable.

If the supplied value has no trailing newline, the here string causes the command to receive one. If the value already ends with a newline, the here string appends another.

Bash Versus POSIX sh

POSIX sh supports here-documents but not the <<< here-string syntax. A portable equivalent is printf '%s\n' "$text" | command.

The set -o pipefail option is Bash-specific rather than POSIX. Scripts intended for multiple shells should either detect the available shell explicitly or avoid Bash-only features.

Production Best Practices and Pitfalls

  • Handle errors explicitly: set -euo pipefail catches many unset-variable, failed-command, and failed-pipeline conditions, but it is not a substitute for checking expected failures. Put commands such as grep inside if statements when a nonzero result is valid.

  • Preserve input faithfully: Use IFS= read -r when whitespace and backslashes must remain unchanged.

  • Avoid accidental expansion: Prefer a quoted here-document for fixed content. For dynamic values, printf with quoted arguments is often easier to validate than embedding values in an unquoted template.

  • Do not use eval: Here-documents and here strings do not require evaluating their contents as shell code.

  • Watch trailing newlines: Command substitution strips trailing newlines from its output, while here-document bodies and here strings have different newline behavior.

  • Generate configuration safely: For production configuration updates, write to a temporary file with restrictive permissions, validate it, and atomically rename it into place. Keep a backup and a tested rollback procedure.

  • Protect secrets: Standard input avoids placing the value directly in the target command’s arguments, but secrets can still leak through scripts, shell history, logs, or process inspection. Use an appropriate secret-management mechanism rather than embedding credentials in templates.

32

What does set -euo pipefail do in Bash?

Strict mode combines Bash’s errexit, nounset, and pipefail options. set -euo pipefail is parsed as enabling -e, enabling -u, and setting -o pipefail.

Read full answer

Applies toBash 5.x

Option Semantics

Option Full name Behavior
-e errexit Exits when a command returns a nonzero status, subject to Bash’s documented exceptions.
-u nounset Treats expansion of an unset variable as an error. An empty variable is still valid.
-o pipefail pipefail Returns the status of the rightmost command in a pipeline that failed. If every command succeeds, the pipeline returns 0.

These are shell options. Use set +e, set +u, or set +o pipefail to disable them.

Failure and Exit Behavior

With errexit enabled, a failing simple command, pipeline, or list normally terminates the script. Bash suppresses errexit in contexts where failure is intentionally tested, including:

  • The condition of if, while, or until.

  • Commands on the left side of && or ||.

  • Commands prefixed with !.

For example, command -v aws || exit 127 safely handles a missing command because command -v is part of a tested || list.

Without pipefail, a pipeline normally returns the status of its final command. For example:

false | true

returns 0 because true succeeds. With pipefail, it returns 1 because false failed. This is important in deployments where an earlier stage—such as curl—fails but a later stage—such as tee—still succeeds.

errexit does not replace explicit error handling. Expected failures should be checked deliberately, especially when distinguishing “not found” from an operational error.

Bash Versus POSIX sh

  • pipefail is a Bash feature and is not required by POSIX sh.

  • [[ ... ]], arrays, and local are also Bash-specific.

  • POSIX sh supports portable -e and -u, but portable scripts should not rely on Bash-only behavior.

  • Use #!/usr/bin/env bash when Bash features such as pipefail are required.

Code Example

#!/usr/bin/env bash
set -euo pipefail

usage() {
  printf 'Usage: %s LOG_FILE\n' "$0" >&2
  exit 64
}

[[ $# -eq 1 ]] || usage
readonly log_file=$1

if [[ ! -r "$log_file" ]]; then
  printf 'Cannot read log file: %s\n' "$log_file" >&2
  exit 1
fi

for tool in grep awk; do
  if ! command -v "$tool" >/dev/null 2>&1; then
    printf 'Required tool not found: %s\n' "$tool" >&2
    exit 127
  fi
done

error_count="$(
  grep -E 'ERROR|FATAL' "$log_file" |
    awk 'END { print NR + 0 }'
)" || {
  status=$?

  # grep returns 1 when no lines match, which is not an operational failure.
  if [[ $status -eq 1 ]]; then
    error_count=0
  else
    exit "$status"
  fi
}

printf 'Found %d ERROR/FATAL records in %s\n' "$error_count" "$log_file"

For a log containing one ERROR and one FATAL record, the output is:

Found 2 ERROR/FATAL records in service.log

The script also demonstrates:

  • A function for consistent usage errors.

  • Quoted arguments and variable expansions.

  • Explicit handling of an expected grep exit status.

  • Distinct exit codes for usage, input, and missing-tool failures.

Production Best Practices

  • Quote expansions: Use "$variable" and "$@" to prevent word splitting and glob expansion.

  • Handle expected failures explicitly: Do not rely on errexit for business logic or recoverable conditions.

  • Preserve exit status: Cleanup traps and wrappers should not accidentally turn a failed deployment into a successful one.

  • Use idempotent cleanup: Remove temporary files with mktemp, register cleanup with trap, and design cleanup so it is safe to repeat during incidents.

  • Do not rely on errexit alone for rollback: A failed command may have already changed remote or cloud resources. Use a transactional deployment strategy and compensating actions.

  • Monitor meaningful status codes: Distinguish configuration errors, unavailable dependencies, failed health checks, and command failures so alerts and runbooks can respond appropriately.

  • Test strict mode locally: Run scripts through bash -n and ShellCheck, and test paths where commands fail, pipelines return partial results, and required environment variables are missing.

33

What are the pitfalls of set -e in Bash?

set -e (errexit) is useful, but it is not a complete error-handling strategy: it exits only in specific contexts and can hide failures in pipelines, functions, and command substitutions.

Read full answer

Applies toBash 5.x

How set -e Actually Behaves

set -e makes the shell exit when a simple command, pipeline, list, or compound command returns nonzero. However, it is deliberately disabled in several contexts, including commands used as if/while/until tests, commands in &&/|| lists except the final one, and commands whose status is inverted with !. This means a script can continue after failures that occur inside those constructs.

Common Pitfalls

  • Pipelines mask failures: without set -o pipefail, only the last command’s status matters, so false | true returns 0.

  • local hides command-substitution failures: local x=$(false) often returns 0 even though the substitution failed; assign first, then declare local.

  • Arithmetic can exit unexpectedly: (( count=0 )) returns 1, which can trigger errexit even though no real error occurred.

  • Expected nonzero statuses: commands such as grep return 1 when no match is found, so they need || true or an explicit conditional.

  • Functions called in conditionals lose errexit: if a function is invoked as an if test, errexit is disabled inside that function.

  • Background jobs are not checked automatically: a failing background job does not immediately exit the script; check it with wait.

Comparison of Risky vs Safer Patterns

Risky pattern Why it is risky Safer pattern
false | true Pipeline appears successful set -o pipefail plus explicit status handling
local x=$(cmd) local may return 0 x=$(cmd); local x or validate x
(( n=0 )) Arithmetic status is 1 n=0 or (( n=0 )) || true
grep pattern file No match returns 1 grep pattern file || true when absence is valid

Production Guidance

Use set -euo pipefail as a baseline, but still add explicit checks for commands whose failure is expected or meaningful. Combine it with an ERR trap for diagnostics, validate inputs, and log failures with context. In production, errexit should be treated as a safety net, not as the primary error-handling design.

34

How do Bash traps handle errors and cleanup?

Traps let Bash run handlers when errors, signals, or shell termination occur. Use set -euo pipefail for strict failure handling, an ERR trap for diagnostics, and an EXIT trap for cleanup that preserves the original exit status.

Read full answer

Applies toBash 5.x on Linux

Error Handling Model

  • set -e: Causes the shell to exit when an eligible command fails.

  • set -u: Treats an unset variable as an error.

  • set -o pipefail: Makes a pipeline fail if any component fails, rather than only considering the final command.

  • set -E: Makes the ERR trap inherit into functions, command substitutions, and subshells.

  • ERR trap: Runs when a command failure meets the same conditions as errexit.

  • EXIT trap: Runs when the shell exits normally, exits explicitly, or exits because of an enabled failure condition.

errexit has exceptions. For example, a failing command used in an if, while, &&, ||, or ! condition does not normally terminate the shell. Important status checks should therefore be written explicitly.

Trap Lifecycle

A trap uses the form:

trap 'command_or_function' SIGNAL

Common signals include:

Signal Typical meaning Conventional status
HUP Terminal disconnected 129
INT Interrupt, normally Ctrl+C 130
TERM Graceful termination request 143
EXIT Shell is exiting Preserves previous status

Signal handlers should usually call exit with an appropriate status. This allows the EXIT trap to perform cleanup. SIGKILL cannot be trapped, so cleanup cannot be guaranteed after kill -9, a kernel panic, or power loss.

Code Example

This script creates a private temporary directory, reports the failing command, removes the directory, and preserves the original failure status.

#!/usr/bin/env bash
set -Eeuo pipefail

work_dir=""

cleanup() {
    local exit_code="$1"

    # Prevent recursive EXIT handling and ignore termination signals during cleanup.
    trap - EXIT HUP INT TERM

    if [[ -n "${work_dir}" && -d "${work_dir}" ]]; then
        if ! rm -rf -- "${work_dir}"; then
            printf 'Cleanup failed for %s\n' "${work_dir}" >&2
        else
            printf 'Cleanup: removed %s\n' "${work_dir}" >&2
        fi
    fi

    # Return the status that caused the shell to exit.
    exit "${exit_code}"
}

report_error() {
    local exit_code="$1"
    local failed_command="$2"

    printf 'Command failed with status %d: %s\n' \
        "${exit_code}" "${failed_command}" >&2

    exit "${exit_code}"
}

# "$?" is expanded when the trap executes.
trap 'cleanup "$?"' EXIT
trap 'report_error "$?" "$BASH_COMMAND"' ERR

trap 'exit 129' HUP
trap 'exit 130' INT
trap 'exit 143' TERM

work_dir="$(mktemp -d)"
readonly work_dir

printf 'Working in %s\n' "${work_dir}"
printf 'Simulated failure\n' >&2
false

Example output:

Working in /tmp/tmp.K8m2s7D91
Simulated failure
Command failed with status 1: false
Cleanup: removed /tmp/tmp.K8m2s7D91

The script exits with status 1. Passing $? into the cleanup function preserves the original failure instead of accidentally returning the status of a cleanup command.

Common Pitfalls

  • Assuming trap catches every error: It does not. Some failures are explicitly tested, and SIGKILL cannot be handled.

  • Losing the original status: A cleanup command can overwrite $?. Capture the status first and return it explicitly.

  • Deleting an unsafe path: Never run rm -rf against an unchecked variable. Validate the path and use -- to prevent option-like arguments.

  • Running cleanup repeatedly: Make cleanup idempotent and disable the EXIT trap inside the handler to prevent recursion.

  • Ignoring cleanup failures: Log cleanup errors, but preserve the primary exit status so incidents retain the original cause.

  • Using Bash-only syntax with sh: POSIX sh uses trap cleanup 0 rather than EXIT, and does not support pipefail, [[ ... ]], or Bash arrays.

Production Practices

  • Keep EXIT cleanup local, fast, and bounded; avoid lengthy network calls inside signal handlers.

  • Use mktemp -d for isolated temporary storage. Its restrictive permissions also reduce the risk of exposing credentials.

  • For remote resources, use idempotent cleanup APIs with timeouts and retries. A process killed with SIGKILL can still leave orphaned cloud resources.

  • Under systemd, place required teardown in ExecStop or ExecStopPost rather than relying exclusively on the application’s shell trap.

  • In containers, ensure the entrypoint process receives and forwards SIGTERM; a shell that does not forward signals can prevent graceful termination.

  • Emit cleanup failures to standard error and monitor them. Repeated cleanup errors often indicate permissions, storage, or orchestration problems.

35

How do you trace Bash execution with set -x?

set -x enables Bash xtrace, which prints each command after variable expansion and immediately before execution. It is useful for locating failures, verifying control flow, and confirming the actual arguments passed to commands.

Read full answer

Applies toBash 5.x on Linux

How Bash Xtrace Works

With xtrace enabled, Bash writes diagnostic output to standard error by default. A trace entry typically contains the command and its expanded arguments:

+ printf 'Checking target: %s\n' /srv/app

For each command, Bash performs this sequence:

  1. Expands variables, command substitutions, and other shell expansions.

  2. Writes the expanded command to the trace stream.

  3. Executes the command.

  4. Continues or exits according to options such as set -e and set -o pipefail.

Xtrace does not change a command’s exit status. However, when combined with set -e, the script may stop at the first failing command. set -o pipefail makes a pipeline fail when any component fails, including an earlier command hidden by a successful final command.

Xtrace also does not show branches that were skipped or explain business-level errors. Use it alongside application logs, exit-code checks, and monitoring.

Enabling Xtrace

Use bash -x to trace a script from startup:

bash -x ./deploy-check.sh /srv/app

Enable tracing inside the script with set -x:

set -Eeuo pipefail
set -x

Disable it temporarily with set +x:

set -x
diagnostic_command
set +x

For a production script, tracing can be conditional:

if [[ "${DEBUG:-0}" == "1" ]]; then
  set -x
fi

The PS4 variable controls the trace prefix. A useful prefix includes the script name and line number:

PS4='+ ${BASH_SOURCE##*/}:${LINENO}: '

BASH_XTRACEFD can redirect traces to an open file descriptor instead of standard error:

exec 9>/tmp/application.trace
BASH_XTRACEFD=9
set -x

The descriptor must remain open while tracing is active. Disable tracing before closing it:

set +x
exec 9>&-

Reading Trace Output

Trace entries reveal the expanded command, which is often the fastest way to diagnose quoting and path problems. For example:

+ deploy-check.sh:14: [[ -d /srv/app ]]
+ deploy-check.sh:17: find /srv/app -type f
+ deploy-check.sh:17: wc -l

Look for:

  • Incorrect paths: An empty variable may become + rm -rf / rather than the intended relative path.

  • Missing quoting: + grep $pattern file can produce different behavior from + grep "$pattern" file.

  • Unexpected arguments: Command substitutions or default values may have expanded differently than expected.

  • Pipeline failures: With pipefail, an early failure can cause the whole pipeline to stop the script.

  • Undefined variables: Under set -u, an unset variable expansion terminates the script.

Code Example

This Bash 5 script optionally writes detailed traces to a dedicated file:

#!/usr/bin/env bash
set -Eeuo pipefail

script_name="${0##*/}"
target="${1:?Usage: $script_name TARGET}"
trace_file="${TRACE_FILE:-/tmp/${script_name}.trace}"

if [[ "${DEBUG:-0}" == "1" ]]; then
  PS4='+ ${BASH_SOURCE##*/}:${LINENO}: '
  exec 9>"$trace_file"
  BASH_XTRACEFD=9
  set -x
fi

printf 'Checking target: %s\n' "$target"
[[ -d "$target" ]]

file_count=$(find "$target" -type f | wc -l)
printf 'Found %s files under %s\n' "$file_count" "$target"

if [[ "${DEBUG:-0}" == "1" ]]; then
  set +x
  exec 9>&-
fi

Run it and inspect the trace:

DEBUG=1 TRACE_FILE=/tmp/deploy-check.trace \
  bash ./deploy-check.sh /srv/app

tail -f /tmp/deploy-check.trace

Run shellcheck ./deploy-check.sh before using the script in CI/CD or production.

Production Practices

  • Protect secrets: Xtrace can expose passwords, tokens, customer data, and signed URLs. Avoid tracing secret-bearing commands, and remember that disabling xtrace does not remove secrets from process arguments.

  • Bound the output: Long-running production traces can generate large volumes and fill disk space. Redirect them to a controlled temporary file with an expiration policy.

  • Reproduce safely: Prefer staging or a read-only diagnostic command. Do not repeatedly execute a destructive script merely to obtain a trace.

  • Use strict mode correctly: set -Eeuo pipefail provides early failure detection, but set -e has context-specific behavior. Explicitly validate important commands and handle expected failures.

  • Check portability: set -x is supported by POSIX shells, but PS4, BASH_XTRACEFD, [[ ... ]], and Bash-specific error-trap behavior are not. Use the required #!/usr/bin/env bash shebang when relying on Bash features.

36

What is ShellCheck and why use it?

ShellCheck is a static analysis tool for shell scripts that parses Bash, POSIX sh, dash, and ksh, then reports syntax errors, quoting bugs, portability issues, and common pitfalls before the script ever runs.

Read full answer

Applies toShellCheck 0.10.x analyzing Bash 5.x scripts (also POSIX sh, dash, ksh).

What ShellCheck Is

  • A linter written in Haskell that understands shell grammar rather than just matching text, so it tracks variable scope, quoting context, and command substitutions.

  • Runs locally from the CLI, inside editors such as VS Code and Vim, and in CI pipelines.

  • Emits diagnostics with stable codes (SC1xxx–SC3xxx) that link to detailed wiki explanations.

Why Use It

  • Catches bugs that only surface at runtime, such as unquoted variables causing word splitting or globbing.

  • Enforces portability when a script targets #!/bin/sh instead of Bash.

  • Blocks defective scripts from reaching production by failing CI on warnings or errors.

  • Teaches better shell idioms through its explanations, which is valuable for teams with mixed scripting experience.

What It Catches

Category Example issue ShellCheck code
Quoting rm $file splits on spaces SC2086
Unused variables count=0 never read SC2034
Portability [[ ]] used in a sh script SC3010
Deprecated syntax Backticks instead of $() SC2006
Logic == inside single-bracket [ ] SC3014
Masked failures Ignoring exit status of a command SC2181

Production Usage

  • Run it in CI with a severity threshold, e.g. shellcheck -S warning scripts/*.sh, and fail the build on findings.

  • Use inline directives such as # shellcheck disable=SC2086 only with a comment explaining why, never blanket-disable a rule file-wide.

  • Pair it with set -euo pipefail, trap cleanup handlers, and actual execution tests; ShellCheck finds static issues but cannot prove runtime behavior.

  • Keep scripts ShellCheck-clean so reviewers focus on logic rather than style.

Best Practices and Pitfalls

  • Treat warnings as errors in CI rather than letting them accumulate into noise.

  • Fix the root cause (usually missing quotes) instead of suppressing the diagnostic.

  • Re-run ShellCheck after refactoring, since moving code between functions changes quoting and scoping analysis.

  • Remember that ShellCheck cannot detect issues requiring runtime data, such as missing files or wrong permissions, so execution tests remain necessary.

37

How do you get substring, length, and replace in Bash?

Bash performs these operations with parameter expansion, so substring extraction, length calculation, and replacement work directly on string variables without starting external commands.

Read full answer

Applies toBash 5.x

Core Parameter Expansions

Operation Syntax Behavior
Length ${#var} Returns the number of characters according to the current locale. In the C locale, each byte counts as one character.
Substring ${var:offset:length} Returns length characters beginning at zero-based offset. ${var:offset} returns the remainder.
First replacement ${var/pattern/replacement} Replaces the first glob-pattern match.
All replacements ${var//pattern/replacement} Replaces every glob-pattern match.
Beginning match ${var/#pattern/replacement} Replaces only a match at the beginning.
Ending match ${var/%pattern/replacement} Replaces only a match at the end.

Patterns are shell glob patterns, not regular expressions. An empty replacement deletes the matched text. A negative substring offset counts from the end and must be separated from the colon, such as ${var: -3}, to avoid ambiguity with default-value syntax.

Case modifiers are also available: ${var^} uppercases the first character, ${var^^} uppercases all characters, while ${var,} and ${var,,} lowercase the first character or all characters. These modifiers require Bash 4 or later.

Scripting and Error Handling

Bash variables are untyped and created by assignment. Quote assignments and expansions:

name="production server"
printf '%s\n' "$name"

if and [[ ... ]] handle conditional tests, for and while implement loops, and functions group reusable logic. Positional arguments are available as $1, $2, and "$@".

Every command returns an exit status: zero indicates success and nonzero indicates failure. set -euo pipefail enables early termination on command failures, unset-variable checks, and pipeline-failure detection. It improves scripts but does not replace explicit checks for expected failures.

Use grep for selecting matching text, sed for stream transformations, and awk for record or field processing. These tools are appropriate for files and structured input; parameter expansion is more efficient for edits to one in-memory string.

Bash and POSIX sh

${#var} is POSIX, but Bash substring slicing and pattern replacement are extensions. In strict POSIX sh, use alternatives such as expr substr "$var" 2 5, cut -c2-6, or sed. Debian and Ubuntu normally provide dash as /bin/sh, so these Bash-only expansions fail in a #!/bin/sh script.

Code Example

#!/usr/bin/env bash
set -euo pipefail

s="${1:-production-server-42}"

echo "length: ${#s}"                      # 20
echo "substring: ${s:11:7}"               # server-
echo "tail: ${s: -2}"                    # 42
echo "first replace: ${s/server/node}"    # production-node-42
echo "all replace: ${s//-/_}"             # production_server_42
echo "prefix replace: ${s/#production/staging}" # staging-server-42
echo "delete: ${s/-42/}"                  # production-server
echo "upper: ${s^^}"                      # PRODUCTION-SERVER-42

slugify() {
    local input="$1"
    echo "${input//-/_}"
}

slugify "$s"

if [[ "$s" == *-42 ]]; then
    echo "has numeric suffix"
fi

IFS='-' read -r -a parts <<< "$s"
for part in "${parts[@]}"; do
    echo "part: $part"
done

if grep -q -- 'server' <<< "$s"; then
    echo "contains server"
else
    echo "server not found" >&2
    exit 1
fi

Running it without arguments produces:

length: 20
substring: server-
tail: 42
first replace: production-node-42
all replace: production_server_42
prefix replace: staging-server-42
delete: production-server
upper: PRODUCTION-SERVER-42
production_server_42
has numeric suffix
part: production
part: server
part: 42
contains server

Production Best Practices

  • Quote every expansion. Unquoted results can undergo word splitting and pathname expansion.

  • Use set -euo pipefail in operational scripts, but handle expected failures explicitly with if, case, or &&/||.

  • Prefer parameter expansion for simple single-string edits; it avoids subprocess overhead in loops.

  • Since Bash 5.2, the patsub_replacement option is enabled by default. An unquoted & in a replacement is then replaced by the matched text. Use shopt -u patsub_replacement when literal ampersands must not receive that substitution.

  • Validate scripts with bash -n script.sh and shellcheck script.sh before deployment.

38

What is parameter expansion in Bash?

Parameter expansion is Bash’s mechanism for reading, transforming, and substituting variable values using the ${...} syntax, including forms like ${var:-default} that supply fallbacks when a variable is unset or empty.

Read full answer

Applies toBash 5.x (forms marked POSIX also work in dash/POSIX sh)

How It Works

When Bash parses a word containing ${...}, it expands the parameter before executing the command. The colon (:) in forms like :-, :=, :+, :? tests for both “unset” and “null” (empty string); without the colon, only “unset” is tested. Because expansion happens during word parsing, quoting matters: an unquoted expansion is subject to word splitting and globbing, while "${var}" preserves the value as a single word.

Common Forms

Form Meaning POSIX?
${var} Value of var Yes
${var:-default} default if var unset or null Yes
${var-default} default only if var unset Yes
${var:=default} Assign default to var if unset/null, then expand Yes
${var:+replacement} replacement if var set and non-null Yes
${var:?message} Print error and exit if unset/null Yes
${#var} Length of value Yes
${var:offset:length} Substring No
${var#pat} / ${var##pat} Remove shortest/longest prefix matching pat Yes
${var%pat} / ${var%%pat} Remove shortest/longest suffix matching pat Yes
${var/old/new} / ${var//old/new} Replace first/all matches No
${var^^} / ${var,,} Uppercase/lowercase entire value No (Bash 4+)
${!var} Indirect expansion (value of variable named by var) No

Production Usage

In scripts, ${var:?} is the idiomatic way to enforce required inputs: : "${DB_HOST:?DB_HOST is required}" fails fast with a clear message instead of running with an empty value. ${var:-} is commonly used to normalize optional flags, e.g. debug=${DEBUG:-0}. Prefix/suffix removal (${file%.*}, ${path##*/}) avoids spawning basename/dirname in tight loops and is faster and safer than parsing with external tools.

Best Practices and Pitfalls

  • Always quote expansions: "${var}", not $var, to prevent word splitting and globbing.

  • Prefer ${var:-} over ${var-} when an empty string should also trigger the default.

  • Use ${var:?} for required values; it exits non-zero under set -e and prints to stderr.

  • Avoid ${var:=...} on read-only or positional parameters; it errors on $1 etc.

  • Remember set -u treats unset variables as errors, so ${var:-} is the safe idiom for optional values.

  • POSIX scripts cannot use substring, substitution, case conversion, or indirect expansion; stick to the POSIX subset or require Bash explicitly with #!/usr/bin/env bash.

39

How does grep use regular expressions in Bash scripts?

Extended regular expressions (ERE) let grep select lines containing specific text patterns. In Bash, use grep -E for ERE syntax, quote the pattern to prevent shell expansion, and handle grep’s exit status explicitly when automation must distinguish “no match” from an actual error.

Read full answer

Applies toBash 5.x, GNU grep 3.x, and POSIX ERE for portable scripts

Choosing a Regular Expression Mode

Option Pattern language Example Use case
grep -G Basic regular expression (BRE) ^(ERROR): Default in GNU grep; portable BRE grouping
grep -E Extended regular expression (ERE) ^(ERROR|FATAL): Portable grouping, alternation and repetition
grep -F Fixed string grep -F 'connection refused' Literal text without regex interpretation
grep -P Perl-compatible regex (PCRE) grep -P 'ERROR(?=()' GNU extension for lookarounds and other PCRE features

ERE is usually the most readable choice for Bash scripts. It supports alternation (|), grouping (()), repetition (*, +, ?, {m,n}) and anchors (^, $). Prefer portable classes such as [[:digit:]], [[:space:]] and [[:alpha:]].

POSIX BRE has no portable alternation operator; GNU’s \| extension should not be used in strictly portable scripts. GNU grep -P is also non-POSIX and may not be available in every build, so use it only when the target systems are known to support it.

How Matching Works

grep is an external utility, not a Bash builtin. The shell first performs expansions, quoting and word splitting; grep then compiles the pattern and scans its input line by line. A match prints the entire line unless another output mode is selected.

Useful options include:

  • -n: Prefix output with line numbers.

  • -i: Perform case-insensitive matching.

  • -c: Count matching lines.

  • -v: Select nonmatching lines.

  • -r: Search files recursively.

  • -I: Skip binary files.

  • -q: Suppress output and return only a status.

ERE matching is line-oriented. ^ and $ anchor to the beginning and end of a line, not to the complete file. Scripts should rely only on documented matching behavior rather than GNU grep’s internal choice of matching engine.

Exit Status and Error Handling

grep returns:

  • 0: At least one line matched.

  • 1: No lines matched.

  • 2: An error occurred, such as an unreadable file or invalid pattern.

This distinction matters with set -euo pipefail. A grep exit status of 1 can fail an entire pipeline even though it only means that nothing matched. Also, grep -q may stop reading early, causing an upstream producer to receive SIGPIPE; pipefail can report that as a pipeline failure.

Code Example

#!/usr/bin/env bash
set -euo pipefail

if (( $# < 2 )); then
  printf 'Usage: %s REGEX LOG_FILE[...]\n' "${0##*/}" >&2
  exit 64
fi

pattern=$1
shift

search_log() {
  local file=$1
  local regex=$2
  local status=0

  grep -nE -e "$regex" -- "$file" || status=$?
  return "$status"
}

overall_status=0

for log_file in "$@"; do
  if [[ ! -r $log_file ]]; then
    printf 'Cannot read log file: %s\n' "$log_file" >&2
    overall_status=2
    continue
  fi

  grep_status=0
  search_log "$log_file" "$pattern" || grep_status=$?

  case "$grep_status" in
    0)
      ;;
    1)
      printf 'No matching entries in %s\n' "$log_file" >&2
      if (( overall_status == 0 )); then
        overall_status=1
      fi
      ;;
    *)
      printf 'grep failed for %s with status %d\n' \
        "$log_file" "$grep_status" >&2
      overall_status=$grep_status
      ;;
  esac
done

exit "$overall_status"

Example invocation:

./find-errors.sh '^[[:space:]]*(ERROR|FATAL)[[:space:]]*:' app.log

Possible output:

42:ERROR: database connection timed out
87:FATAL: worker process exited unexpectedly

The invocation quotes the regex so the shell does not interpret its metacharacters. Inside the script, grep -e "$pattern" also permits a pattern beginning with -, while -- protects the filename from being interpreted as an option.

Production Best Practices

  • Use grep -F for literal text; it avoids regex semantics and often avoids regex-processing overhead.

  • Use sed -E for regex-based substitutions and awk for field, record and transformation logic; grep primarily selects lines.

  • Set LC_ALL=C when byte-oriented matching must remain stable across locales.

  • Never pass user-supplied patterns to eval; grep treats the value as regex data, not as shell commands.

  • Restrict untrusted pattern length and complexity, especially for PCRE features involving backtracking.

  • Decide explicitly whether “no match” should produce success or failure in CI, monitoring and incident-response scripts.

  • This script uses non-POSIX features such as [[, ((, local and pipefail; replace them and explicitly capture pipeline statuses when targeting POSIX sh.

40

How do you find and replace text in files using sed?

sed is a stream editor that transforms text line by line; its s (substitute) command is the standard way to find and replace text in files or pipelines. By default it prints the result to stdout, so it never modifies a file unless you explicitly request in-place editing.

Read full answer

Applies toGNU sed 4.x on Linux; BSD sed (macOS) differs in `-i` syntax.

How It Works

  • Basic form: sed 's/pattern/replacement/flags' file

  • The s command takes a delimiter (commonly /), a regex pattern, a replacement, and optional flags.

  • Without -i, output goes to stdout and the original file is untouched — safe for previewing.

  • Addresses (line numbers or ranges) can restrict where substitution happens: sed '2,5s/foo/bar/' file.

  • sed exits 0 even when no match is found, so do not rely on its exit status to detect matches; use grep -q first if needed.

Common Patterns

  • Global replace on every line: sed 's/foo/bar/g'

  • Case-insensitive (GNU sed): sed 's/foo/bar/gi'

  • Alternate delimiter for paths/URLs: sed 's|/var/log|/var/log/archive|g'

  • Capture groups with extended regex: sed -E 's/^([A-Za-z]+): (.*)$/\2 (\1)/'

  • Replace only on lines matching a condition: sed '/ERROR/s/foo/bar/'

  • Delete matching lines instead: sed '/pattern/d'

In-Place Editing

  • GNU sed: sed -i 's/foo/bar/g' file.txt

  • With backup: sed -i.bak 's/foo/bar/g' file.txt

  • GNU sed writes a temp file and renames it into place, preserving mode but changing ownership to the user running sed. In production, running as root or the file owner avoids surprising ownership changes.

  • BSD/macOS requires a backup argument: sed -i '' 's/foo/bar/g' file.txt (empty string means no backup).

  • For critical files, prefer writing to a temp file and mv atomically, or keep a .bak.

Portability: GNU vs BSD sed

Feature GNU sed BSD sed (macOS)
In-place, no backup sed -i 's/.../.../' sed -i '' 's/.../.../'
Extended regex -E (also -r) -E
Case-insensitive flag I not supported
n in replacement supported limited

Use -E for extended regex on both; avoid GNU-only flags in portable scripts.

Code Example

#!/usr/bin/env bash
set -euo pipefail

# Preview substitution (no file change)
sed 's/localhost/127.0.0.1/g' config.ini

# In-place global replace with backup (GNU sed)
sed -i.bak 's/old_value/new_value/g' config.ini

# Replace a path using an alternate delimiter
sed -i 's|/opt/app/v1|/opt/app/v2|g' deploy.conf

# Capture groups: "key: value" -> "value (key)"
sed -E 's/^([A-Za-z_]+): (.*)$/\2 (\1)/' env.txt

Sample input env.txt:

HOST: localhost
PORT: 8080

Sample output:

localhost (HOST)
8080 (PORT)

Best Practices and Pitfalls

  • Always quote the script to prevent shell expansion of $, *, or backticks.

  • Escape &, </code>, and the delimiter in the replacement; <code>& means “the whole match.”

  • Test without -i first, then add -i once output looks correct.

  • sed is line-oriented; it cannot easily handle multi-line or binary data — use perl or awk for those.

  • Under set -o pipefail, a failing upstream command in cmd | sed ... still surfaces the error.

41

How can awk extract and transform tabular text fields?

awk is a stream-oriented text-processing tool that splits each record into fields and lets you filter, select, aggregate, or reformat those fields using patterns and actions. It is well suited to logs, inventories, metrics, and other line-oriented operational data.

Read full answer

Applies toGNU awk 5.x and Bash 5.x on Linux

Processing Model

awk processes a file one record at a time:

  1. It reads a record, normally one line.

  2. It splits the record into fields using FS.

  3. It evaluates each pattern-action rule.

  4. It executes the action for matching records.

Important built-in variables include:

  • $0: The complete current record.

  • $1, $2, …: Individual fields.

  • NF: Number of fields in the current record.

  • NR: Current record number across all input files.

  • FNR: Current record number within the current file.

  • FS: Input field separator.

  • OFS: Output field separator used by print.

  • ORS: Output record separator, normally a newline.

With the default FS, runs of spaces and tabs separate fields. Set FS="\t" for tab-separated data or use -F ',' for simple comma-separated data.

Common Column Operations

Goal awk expression
Print one column awk -F 't' '{ print $3 }' data.tsv
Filter by a column awk -F 't' '$3 == "ERROR" { print }' data.tsv
Select multiple columns awk -F 't' '{ print $2, $4 }' data.tsv
Aggregate a numeric column awk -F 't' 'NR > 1 { total += $4 } END { print total + 0 }' data.tsv
Format output awk -F 't' '{ printf "%s: %.2fn", $2, $4 }' data.tsv
Process multiple files independently awk -F 't' 'BEGINFILE { total = 0 } { total += $4 } ENDFILE { print FILENAME, total }' data.tsv

The BEGINFILE and ENDFILE features in the final example are GNU awk extensions.

Code Example

The following Bash script reads tab-separated service data, selects the service and duration columns, calculates totals, and rejects malformed records.

#!/usr/bin/env bash
set -euo pipefail
export LC_ALL=C

input_file=${1:?"Usage: $0 INPUT.tsv"}

if [[ ! -f "$input_file" || ! -r "$input_file" ]]; then
    printf 'Input is not a readable file: %s\n' "$input_file" >&2
    exit 1
fi

# Prefix relative paths so filenames cannot be interpreted as options.
case "$input_file" in
    /*) ;;
    *) input_file="./$input_file" ;;
esac

awk '
BEGIN {
    FS = "\t"
    OFS = "\t"
}

NR == 1 {
    expected = NF
    if (expected != 4) {
        printf "Header has %d fields; expected 4\n", expected > "/dev/stderr"
        bad = 1
    }
    next
}

{
    if (NF != expected) {
        printf "Record %d has %d fields; expected %d\n", NR, NF, expected > "/dev/stderr"
        bad = 1
        next
    }

    # Remove a Windows CRLF line ending from the final field.
    sub(/\r$/, "", $4)

    if ($4 !~ /^[0-9]+$/) {
        printf "Record %d has an invalid duration: %s\n", NR, $4 > "/dev/stderr"
        bad = 1
        next
    }

    printf "%s\t%s\n", $2, $4
    total += $4

    if ($3 == "ERROR") {
        errors++
    }
}

END {
    if (bad) {
        exit 2
    }

    printf "TOTAL_DURATION_MS\t%.0f\n", total
    printf "ERROR_ROWS\t%d\n", errors
}
' "$input_file"

For this input:

timestamp	service	status	duration_ms
2025-01-10T10:00:00Z	api	OK	120
2025-01-10T10:01:00Z	worker	ERROR	85
2025-01-10T10:02:00Z	api	OK	430

The output is:

api	120
worker	85
api	430
TOTAL_DURATION_MS	635
ERROR_ROWS	1

Failure Modes and Security

  • Incorrect separator: Verify the actual delimiter before processing. Using -F ',' is not safe for RFC 4180 CSV containing quoted commas; use a proper CSV parser such as Python’s csv module or Miller for that format.

  • Malformed records: Check NF and validate numeric fields. Exit nonzero rather than silently converting invalid values to zero.

  • Whitespace in values: Default whitespace-based FS cannot distinguish a tab from spaces or preserve empty fields reliably.

  • Memory growth: awk streams records, but associative arrays consume memory for every unique key.

  • Untrusted input: Quote filenames and never concatenate untrusted values directly into the awk program text.

  • Partial artifacts: Write to a temporary file and publish it only after awk exits successfully. This prevents consumers from reading incomplete output.

Production and Portability

For scheduled jobs and CI/CD pipelines, log malformed-record counts, monitor nonzero exit statuses, and preserve the raw input for forensic replay under the required retention policy. Pin Bash and GNU awk versions in container images and regression-test scripts after upgrades.

The script uses Bash-specific [[ ... ]] and set -o pipefail. In POSIX sh, replace [[ ... ]] with [ ... ] and use set -eu; POSIX sh does not support pipefail. The awk logic is mostly POSIX-compatible, except for /dev/stderr, which is not a POSIX file path.

42

How do sort, uniq, and wc transform and count text lines?

sort, uniq, and wc form a text-processing pipeline: sort orders lines, uniq collapses adjacent duplicates or counts them, and wc counts lines, words, or bytes. Global deduplication and accurate frequency counts require sorting the input before passing it to uniq.

Read full answer

Applies toGNU coreutils 9.x on Linux; Bash 5.x or POSIX sh as the invoking shell

How the Tools Work

Tool Primary behavior Important detail
sort Orders lines lexicographically by default Duplicate lines remain unless -u is used
uniq Collapses or reports adjacent duplicate lines It does not find duplicates separated by other lines
wc Counts lines, words, and bytes -l counts newline characters, not necessarily logical records

A typical global deduplication pipeline is sort data.txt | uniq. The command sort -u data.txt provides the same unique result more directly. To count occurrences, use sort data.txt | uniq -c | sort -k1,1nr: the first sort makes equal lines adjacent, uniq -c counts each run, and the final sort orders the results by descending numeric frequency.

Common Command Patterns

  • Sort numerically: sort -n values.txt

  • Sort in reverse order: sort -r values.txt

  • Sort by a numeric field in simple, unquoted CSV: sort -t ',' -k2,2nr records.csv. Use a CSV-aware parser for quoted fields, embedded commas, or other RFC-style CSV features.

  • Deduplicate globally: sort -u data.txt

  • Count each unique line: sort data.txt | uniq -c

  • Show only repeated lines: sort data.txt | uniq -d

  • Show lines occurring once: sort data.txt | uniq -u

  • Count lines, words, bytes, or characters: wc -l data.txt, wc -w data.txt, wc -c data.txt, or wc -m data.txt

  • Count without printing the filename: wc -l < data.txt

  • Check whether input is sorted: sort -c data.txt

For locale-independent byte-value ordering, set LC_ALL=C. For locale-aware ordering, use an installed UTF-8 locale such as en_US.UTF-8.

Operational and Security Considerations

  • Pipeline failures: In Bash, set -o pipefail makes a pipeline fail if sort, uniq, or wc fails, rather than reporting only the final command’s status.

  • Temporary storage: For large inputs, GNU sort can use temporary files. Ensure TMPDIR points to storage with sufficient space and appropriate permissions.

  • Missing final newline: wc -l counts newline characters, so a final unterminated record is not included. Use awk 'END { print NR }' when logical-record counting is required.

  • Line endings: CRLF files retain \r as part of each record, so CRLF and LF versions of the same text do not compare as identical. Normalize line endings deliberately.

  • Large files: uniq and wc are streaming operations, while sort may consume substantial I/O and temporary disk space.

  • Automation: Check command and pipeline exit statuses, and emit logs or metrics before using results to trigger deployments, cleanup, or incident workflows.

  • Safe arguments: Quote filenames and use -- before user-controlled filenames to prevent them from being interpreted as options.

Bash and POSIX sh Portability

The utilities and their basic options are available in POSIX environments. Bash 5.x additionally provides shell features such as [[ ... ]], arithmetic evaluation with (( ... )), and set -o pipefail. POSIX sh instead uses test, arithmetic expansion, and set -eu, but has no direct equivalent of pipefail; detecting an intermediate pipeline failure requires another mechanism.

43

How do you use cut, tr, and paste in Bash?

cut, tr, and paste are text-streaming utilities: cut extracts fields or characters, tr transforms or removes individual characters, and paste combines lines from multiple files or streams. They are useful for lightweight shell automation but are not CSV- or JSON-aware parsers.

Read full answer

Applies toGNU coreutils 9.x on Linux and Bash 5.x

Command Roles

Command Primary purpose Common options
cut Select fields or character positions -d, -f, -c, -s
tr Translate, delete, or squeeze characters -d, -s, -c
paste Merge corresponding lines from multiple inputs -d, -s

These commands read from files, accept - for standard input, and normally write to standard output, making them easy to compose in pipelines.

Using cut

cut selects columns separated by a delimiter. Without -d, it uses tabs.

cut -d',' -f2,3 data.csv
cut -d':' -f1 access.log
cut -f2 report.tsv
  • -d,: Set the field delimiter to a comma.

  • -f2,3: Select fields 2 and 3.

  • -f2-4: Select a field range.

  • -c1-8: Select character positions 1 through 8.

  • -s: Suppress lines that do not contain the specified delimiter.

For example:

printf 'id,name,region\n10,alice,eu-west\n' | cut -d',' -f2,3

Output:

name,region
alice,eu-west

cut does not understand quoted CSV fields. A value such as "Berlin, Germany" may be split incorrectly, so use a proper CSV parser for production data.

Using tr

tr operates on individual characters rather than complete words or lines.

printf 'prod-api\n' | tr '[:lower:]' '[:upper:]'
printf 'text\r\n' | tr -d '\r'
printf 'a    b     c\n' | tr -s ' ' ' '
  • Translation: tr SET1 SET2 maps characters in SET1 to corresponding characters in SET2.

  • Deletion: tr -d SET removes every character in SET.

  • Squeezing: tr -s SET replaces repeated characters with one occurrence.

  • Complement: tr -c SET acts on characters outside SET.

If SET2 is shorter than SET1, the last character of SET2 is repeated; extra characters in SET2 are ignored. tr cannot replace substrings directly; use sed, parameter expansion, or a structured parser for that.

Using paste

paste reads corresponding lines from two or more inputs and writes them horizontally.

paste users.txt ports.txt
paste -d',' users.txt ports.txt

The default output delimiter is a tab. With -d, delimiters are selected from the supplied list and reused cyclically.

To serialize all lines from one stream into a single comma-separated line:

printf 'alice\nbob\n' | paste -sd, -

Output:

alice,bob

paste -s serializes each input file into a separate output line. It does not parse nested data or align records by a key.

Code Example

This Bash 5 script demonstrates all three commands and passes ShellCheck:

#!/usr/bin/env bash
set -euo pipefail

workdir="$(mktemp -d)"
trap 'rm -rf "$workdir"' EXIT

cat > "$workdir/users.csv" <<'EOF'
alice,api
bob,worker
EOF

cat > "$workdir/ports.txt" <<'EOF'
8080
9090
EOF

printf '%s\n' '--- cut: select fields ---'
cut -d',' -f1,2 "$workdir/users.csv"

printf '%s\n' '--- tr: convert to uppercase ---'
printf 'prod-api\n' | tr '[:lower:]' '[:upper:]'

printf '%s\n' '--- tr: remove carriage returns ---'
printf 'linux\r\n' | tr -d '\r'

printf '%s\n' '--- paste: combine corresponding lines ---'
paste -d',' "$workdir/users.csv" "$workdir/ports.txt"

Expected output:

--- cut: select fields ---
alice,api
bob,worker
--- tr: convert to uppercase ---
PROD-API
--- tr: remove carriage returns ---
linux
--- paste: combine corresponding lines ---
alice,api,8080
bob,worker,9090

Production Practices

  • Use set -euo pipefail to terminate on an unset variable reference, an unhandled command failure, or a failed pipeline stage. It is not a universal error handler: a command whose status is tested in an if or while condition, or used in an && or || list, may not trigger -e.

  • Define reusable operations as functions. Pass arguments through $1, $2, and so on, declare them with local, and quote every expansion.

  • Use if for expected failures and for loops for repeated work. When handling an expected failure, capture $? immediately in the else branch if the specific status is needed.

  • Quote file names and variables, especially when paths may contain spaces.

  • Use LC_ALL=C when byte-oriented behavior must remain consistent across hosts.

  • Create temporary files under a restrictive mktemp directory and clean them with a trap.

  • Do not put credentials directly in command arguments because they may appear in process listings or shell history.

  • Use awk, jq, or a dedicated CSV parser instead of these utilities for quoted CSV, JSON, XML, or key-based joins.

  • For long-running log processing, stream records through pipelines and monitor failures rather than loading entire files into memory.

  • The cut --complement option is a GNU coreutils extension and is unavailable in POSIX-only cut implementations, regardless of whether Bash or POSIX sh invokes it.

44

How does xargs work compared to a shell loop?

xargs reads items from standard input and passes them as arguments to another command, batching them efficiently and optionally running commands in parallel. A shell loop processes one item at a time and is better when each item needs conditional logic, variable handling, or shell builtins.

Read full answer

Applies toBash 5.x and GNU xargs (findutils) on Linux

How xargs Works

  • xargs collects input, usually separated by whitespace or newlines, and invokes the target command with as many arguments as the system allows.

  • Use find ... -print0 | xargs -0 so filenames containing spaces, quotes, or newlines are handled safely.

  • Use -r (--no-run-if-empty) to avoid running the command when there is no input.

  • Use -n to limit arguments per invocation and -P to run multiple commands in parallel.

  • Use -I {} only when each item must be placed in a specific position; it usually runs one command per item and loses batching.

How a Loop Differs

  • A while read loop gives full control over each item, including conditionals, retries, logging, and shell variables.

  • Loops are safer for complex per-item logic but invoke the command once per item unless you deliberately batch inside the loop.

  • Loops can use shell functions and builtins directly; xargs generally needs an external command or sh -c.

  • xargs automatically respects ARG_MAX; loops avoid argument-list limits by invoking commands one at a time.

Comparison

Aspect xargs Shell loop
Invocation count Batches many items Usually one per item
Parallelism Built-in with -P Manual with background jobs
Complex logic Limited Full shell logic
Safe filenames Requires -0 Requires read -d ''
Exit status Aggregated across invocations Controlled per item

Failure Modes and Pitfalls

  • Default xargs input splitting breaks on spaces and newlines; always pair find -print0 with xargs -0.

  • Without -r, an empty input can cause commands such as rm to run with no file arguments.

  • xargs -I {} is convenient but can be much slower because it disables batching.

  • In production, prefer xargs -P for CPU-bound or I/O-bound batch work, but cap parallelism to avoid overwhelming disks, APIs, or databases.

  • For destructive operations, log the generated command first with xargs -t or use -- before filenames.

Best Practices

  • Use find -print0 | xargs -0 for any filename processing.

  • Add -r to prevent empty-input command execution.

  • Use -n and -P deliberately, and test with xargs -t before destructive runs.

  • Use a loop when per-item decisions, retries, or shell functions are needed.

  • Prefer while IFS= read -r -d '' over parsing find output in a for loop.

45

How do run commands in parallel with Bash?

Parallel execution in Bash means starting independent commands in the background and then synchronizing with wait. Bash provides concurrent process execution; actual simultaneous CPU use depends on the number of available cores and the workload.

Read full answer

Applies toBash 5.x on Linux; GNU coreutils 9.x for `xargs -P`

How Bash Runs Background Commands

Append & to start a command asynchronously:

command_one &
command_two &
wait

When Bash encounters &, it:

  • Starts the command in a separate process or subshell.

  • Stores its process ID in $!.

  • Continues executing the parent script without waiting.

  • Adds the process to the shell’s job table.

For example:

curl --fail https://api.example.com/health &
pid=$!

# Perform other work while the request is running.

wait "$pid"

The value of $! must be captured immediately after launching the background command because later background commands overwrite it.

Waiting and Exit Statuses

wait "$pid" waits for one specific background process and returns that process’s exit status. This is important when the script must detect failures:

if wait "$pid"; then
  printf 'Job completed successfully\n'
else
  status=$?
  printf 'Job failed with status %d\n' "$status" >&2
fi

By contrast, wait without arguments waits for all background jobs but normally returns zero, so it can hide individual job failures. Under set -e, a failing wait can also terminate the script unless it is used in a conditional such as if wait "$pid"; then.

Bash also provides:

  • wait -n: Waits for the next background job to finish (Bash 4.3+).

  • wait -n -p finished_pid: Bash 5.1+ also records the finished job’s process ID.

  • kill "$pid": Requests termination of a specific background process.

Bash and POSIX sh Differences

POSIX sh supports background commands with & and wait, but it does not provide Bash arrays or wait -n. POSIX does not standardise local (dash, ash and ksh implement it as an extension), and job control is only active in interactive shells — a script normally runs without it. Because the example below relies on arrays and per-PID status handling, it should run with #!/usr/bin/env bash rather than #!/bin/sh.

Code Example

Parallel health checks with per-job exit status — the script records every process ID and reports individual failures:

#!/usr/bin/env bash
set -euo pipefail

check_service() {
  local name=$1
  local url=$2
  local attempt

  for attempt in 1 2 3; do
    if curl --fail --silent --show-error \
      --connect-timeout 5 --max-time 10 "$url" >/dev/null; then
      printf '%s: healthy\n' "$name"
      return 0
    fi

    sleep $((attempt * 2))
  done

  printf '%s: failed after retries\n' "$name" >&2
  return 1
}

declare -a pids=()
declare -a names=()

for name in api worker db; do
  check_service "$name" "https://${name}.example.com/health" &
  pids+=("$!")
  names+=("$name")
done

failed=0

for index in "${!pids[@]}"; do
  if wait "${pids[$index]}"; then
    printf '%s: completed successfully\n' "${names[$index]}"
  else
    status=$?
    printf '%s: failed with exit status %d\n' \
      "${names[$index]}" "$status" >&2
    failed=1
  fi
done

if (( failed != 0 )); then
  exit 1
fi

Output ordering is nondeterministic because the jobs finish independently.

For a large workload, bound the concurrency with GNU xargs -P (at most four curl processes at a time):

urls=(
  "https://api.example.com/health"
  "https://worker.example.com/health"
  "https://db.example.com/health"
)

printf '%s\0' "${urls[@]}" |
  xargs -0 -r -n 1 -P 4 \
    curl --fail --silent --show-error --max-time 10

xargs exits with 123 if an invocation exits with status 1-125, but it does not offer the same convenient per-item status collection as storing Bash PIDs.

Production Best Practices

  • Launching an unbounded number of jobs can overload CPUs, APIs, databases or network connections; cap concurrency (xargs -P, a semaphore in coproc, or a job pool).

  • Apply explicit timeouts, bounded retries and exponential backoff to every remote call.

  • Write each job’s output to a separate file to prevent interleaved or corrupted logs.

  • Make retried operations idempotent, especially during deployments or incident recovery.

  • Use trap handlers to terminate child processes when the parent receives a termination signal.

  • Avoid eval; validate inputs and quote every variable expansion.

  • Monitor duration, failure rate and resource saturation rather than treating parallel completion alone as success.

46

How do you run a Bash command in the background and wait?

Append & to start a background process, save $! if you need its process ID, and use the wait builtin to block until it finishes. wait "$pid" waits for a specific child, while wait without arguments waits for all active child processes.

Read full answer

Applies toBash 5.x on Linux

How Bash Tracks Background Jobs

When Bash executes a command containing &, it starts the command asynchronously and immediately continues with the next statement:

long_running_command &
pid=$!
  • &: Places the command in the background.

  • $!: Contains the process ID of the most recently started background job.

  • wait "$pid": Blocks the current shell until that child process exits.

  • $?: Contains the exit status returned by wait.

For example:

sleep 5 &
pid=$!
echo "Waiting for PID $pid"
wait "$pid"
echo "The job exited with status $?"

Waiting for Multiple Jobs

Bash provides several forms of wait:

Form Behavior
wait "$pid" Waits for the specified child process.
wait "$pid1" "$pid2" Waits for both processes; the returned status is from the last process waited for.
wait Waits for all active child processes and returns zero after they complete.
wait -n Waits for the next available background job to finish.

When every job must succeed, check each status separately. A multi-PID wait can hide an earlier failure if the final job succeeds.

job_one &
pid_one=$!

job_two &
pid_two=$!

failed=0
wait "$pid_one" || failed=$?
wait "$pid_two" || failed=$?

if (( failed != 0 )); then
  echo "A background job failed" >&2
  exit "$failed"
fi

Error Handling with set -euo pipefail

set -euo pipefail is a common foundation for production Bash scripts:

  • -e: Exits when an unhandled command fails, although commands used as conditions are exempt.

  • -u: Treats use of an unset variable as an error.

  • pipefail: Makes a pipeline fail when any component fails.

A background command does not immediately trigger errexit in the parent shell because the parent continues executing. The failure appears when wait returns the child’s nonzero status.

Use wait inside an if condition when you need to handle that failure. Calling wait "$pid" as a standalone command under set -e can terminate the script before custom cleanup or reporting runs.

Code Example

This script starts a job, redirects its output, waits for it, and preserves the job’s exit status:

#!/usr/bin/env bash
set -euo pipefail

log_dir="$(mktemp -d)"
trap 'rm -rf "$log_dir"' EXIT

sleep 2 >"$log_dir/job.log" 2>&1 &
pid=$!

printf 'Started background job with PID %s\n' "$pid"

if wait "$pid"; then
  printf 'Job completed successfully\n'
else
  status=$?
  printf 'Job failed with exit status %d\n' "$status" >&2
  exit "$status"
fi

Expected output, with a variable PID:

Started background job with PID 12345
Job completed successfully

Production Best Practices

  • Redirect output: Background output can interleave with the parent script. Write it to a log file or another managed destination.

  • Quote expansions: Use "$pid" and "$log_dir"; unquoted values can cause word splitting or pathname expansion.

  • Avoid eval: Pass command arguments directly rather than constructing shell code from untrusted input.

  • Use mktemp: Create temporary directories securely instead of predicting filenames in /tmp.

  • Add operational ownership: wait provides no progress reporting, health checks, or timeout. For long-running production work, use systemd, a Kubernetes Job, or a CI runner.

  • Understand shell lifetime: wait works only for children of the current Bash process. Exiting the shell can terminate or detach from jobs depending on their signal handling and environment.

  • Monitor failures: A successful wait means only that the process exited; inspect logs, metrics, and exit codes to determine whether the job actually completed its intended work.

47

How do you schedule a Bash script with cron?

A cron job is scheduled by adding an entry to a user’s crontab. For a Bash script, use its absolute path, configure explicit error handling and logging, and account for cron’s restricted environment.

Read full answer

Applies toDebian `cron` on Ubuntu 22.04/24.04, Cronie on RHEL 9, and Bash 5.x

How Cron Schedules Jobs

A crontab entry contains five time fields followed by the command: minute hour day-of-month month day-of-week command.

Field Values Example
Minute 0-59 15
Hour 0-23 2
Day of month 1-31 *
Month 1-12 or JAN-DEC *
Day of week 0-7 or SUN-SAT 1-5

Examples:

*/15 * * * * /usr/local/bin/check-health
30 2 * * * /usr/local/bin/nightly-report
0 3 * * 1-5 /usr/local/bin/backup

Common shorthand schedules include @hourly, @daily, @weekly, @monthly, and @reboot. If both day-of-month and day-of-week are restricted, standard cron behavior is to run the job when either field matches.

Use crontab -e to edit the current user’s schedule and crontab -l to inspect it. Schedule scripts as a dedicated non-login service account rather than as root.

Cron Execution Environment

Cron starts jobs with a minimal environment and normally executes the command through /bin/sh.

  • Use an absolute path for the script and important external commands.

  • Do not depend on the current working directory.

  • Set or override PATH inside the script when necessary.

  • Redirect both standard output and standard error to a log file.

  • Treat a non-zero exit status as a failed job and monitor it externally; cron does not provide production alerting by itself.

On Vixie-style implementations, including Debian cron and Cronie, % is special in a crontab command. Escape it as \% when it must be passed literally.

Production Behavior

The cron daemon’s service name differs by distribution:

Distribution Service Commands
Ubuntu/Debian cron sudo systemctl enable --now cron
RHEL-family crond sudo systemctl enable --now crond

Production considerations include:

  • Time zone: Schedules use the daemon host’s local time by default. Debian cron and Cronie support CRON_TZ for a per-crontab override; daylight-saving changes can cause local times to be skipped or repeated.

  • Missed execution: Ordinary crontab jobs missed while the daemon or host was stopped are not replayed.

  • Overlapping runs: Use flock when a job can exceed its interval.

  • Logging: Configure logrotate and monitor logs for failures or missing completion heartbeats.

  • Backups: Archive every user crontab together with the system schedules, for example sudo tar -czf "crontabs-$(date +%F).tar.gz" /var/spool/cron /etc/crontab /etc/cron.d (Ubuntu keeps user crontabs under /var/spool/cron/crontabs, RHEL under /var/spool/cron). Listing one crontab with crontab -l does not create a backup.

  • Upgrades: Re-test scripts and schedules after operating-system upgrades.

Code Example

Create a dedicated account, state directory, and Bash script:

sudo useradd --system \
  --home-dir /var/lib/report-job \
  --create-home \
  --shell /usr/sbin/nologin \
  report-job

sudo install -d -m 0750 -o report-job -g report-job /var/lib/report-job

sudo tee /usr/local/bin/report-job >/dev/null <<'EOF'
#!/usr/bin/env bash
set -euo pipefail

export PATH=/usr/local/bin:/usr/bin:/bin
readonly STATE_DIR="/var/lib/report-job"
readonly LOCK_FILE="${STATE_DIR}/report-job.lock"

# Prevent overlapping executions of this job.
exec 9>"${LOCK_FILE}"
if ! /usr/bin/flock --nonblock 9; then
  printf '%s report-job skipped: another instance is running\n' \
    "$(/usr/bin/date --iso-8601=seconds)"
  exit 0
fi

printf '%s report-job started\n' "$(/usr/bin/date --iso-8601=seconds)"

# Run the actual reporting or maintenance work here.

printf '%s report-job completed\n' "$(/usr/bin/date --iso-8601=seconds)"
EOF

sudo chmod 0755 /usr/local/bin/report-job

sudo touch /var/log/report-job.log
sudo chown report-job:report-job /var/log/report-job.log
sudo chmod 0640 /var/log/report-job.log

Edit that account’s crontab:

sudo crontab -u report-job -e

Add:

17 2 * * * /usr/local/bin/report-job >> /var/log/report-job.log 2>&1

This runs the script every day at 02:17 using the configured cron time zone.

Bash Scripting and Portability

  • Variables and quoting: Use readonly for constants and quote expansions such as "${LOCK_FILE}" to prevent word splitting and pathname expansion.

  • Conditionals, functions, and arguments: Use if or case for validation, organize larger jobs into functions, pass arguments with main "$@", and declare function-local variables with local.

  • Loops and text processing: Read records with while IFS= read -r line; use grep for selection, sed for substitutions, and awk for field processing or aggregation. Quote patterns and protect filenames from option-like values.

  • Exit codes: set -euo pipefail catches common failures, but critical operations still need explicit checks because set -e has conditional and function-related edge cases. Run ShellCheck in CI.

  • Portability: Keep the Bash shebang: set -o pipefail, arrays, [[ ... ]], and process substitution are not portable to a /bin/sh script such as dash.

48

How do you write an idempotent Bash script for production?

An idempotent script converges to the same desired state when repeated; a second run should make no additional changes to the managed resources. It may still validate state, acquire locks, and report status. Production scripts should combine state comparison, concurrency control, atomic updates, explicit error handling, and least-privilege execution.

Read full answer

Applies toBash 5.x on Ubuntu 22.04/24.04; Bash 4.4 on RHEL 9

How It Works

An idempotent automation script generally follows this pattern:

  1. Validate inputs, permissions, and required dependencies.

  2. Acquire a lock so concurrent executions cannot modify the same state.

  3. Compare the current state with the desired state.

  4. Apply only missing or incorrect configuration.

  5. Return a meaningful exit code and record what changed.

Idempotency does not mean the script performs no work or produces no output. It means repeated executions converge on the same managed state without accumulating additional changes.

Code Example

The following script installs a regular configuration file at a fixed path. It replaces a symlink, or a regular file whose content or mode differs; a matching regular file is left untouched.

#!/usr/bin/env bash
set -Eeuo pipefail

readonly APP_DIR="/opt/myapp"
readonly APP_FILE="${APP_DIR}/app.conf"
readonly LOCK_FILE="/var/lock/myapp-deploy.lock"

tmp_file=""

cleanup() {
  if [[ -n "$tmp_file" ]]; then
    rm -f -- "$tmp_file"
  fi
}

trap 'printf "Deployment failed at line %s\n" "$LINENO" >&2' ERR
trap cleanup EXIT

main() {
  if (( EUID != 0 )); then
    printf "This script must run as root.\n" >&2
    exit 1
  fi

  # Prevent two deployments from modifying the application concurrently.
  exec 9>"$LOCK_FILE"
  if ! flock -n 9; then
    printf "Another deployment is already running.\n" >&2
    exit 1
  fi

  install -d -m 0755 "$APP_DIR"

  # Create the candidate file on the same filesystem for an atomic rename.
  tmp_file="$(mktemp "${APP_DIR}/.app.conf.XXXXXX")"
  cat >"$tmp_file" <<'EOF'
listen 8080
workers 4
EOF
  chmod 0644 "$tmp_file"

  if [[ ! -f "$APP_FILE" || -L "$APP_FILE" ]] ||
     ! cmp -s -- "$tmp_file" "$APP_FILE" ||
     [[ "$(stat -c '%a' "$APP_FILE")" != "644" ]]; then
    # -T treats the destination as a file; GNU mv replaces it atomically.
    mv -fT -- "$tmp_file" "$APP_FILE"
    tmp_file=""
    printf "Updated %s\n" "$APP_FILE"
  else
    printf "Unchanged %s\n" "$APP_FILE"
  fi
}

main

Example output:

$ sudo ./deploy.sh
Updated /opt/myapp/app.conf
$ sudo ./deploy.sh
Unchanged /opt/myapp/app.conf

The script requires Bash, GNU coreutils, GNU diffutils for cmp, and flock from util-linux. Run it through ShellCheck before deployment.

Operational Design

  • Desired-state comparison: Inspect existing resources before modifying them. For cloud APIs, prefer idempotent PUT operations over repeated POST requests; use stable resource identifiers or idempotency keys where supported.

  • Concurrency control: flock prevents overlapping local executions. Across hosts, use a distributed lock or configure the orchestrator to serialize the operation.

  • Atomic updates: Write a temporary file in the destination filesystem and rename it into place, preventing readers from seeing a partially written configuration.

  • Change-triggered actions: Restart a service or reload configuration only when the managed state changes, not on every execution.

  • Auditability and monitoring: Record timestamps, current state, changes, errors, and duration. Emit metrics for changed, no-op, and failed runs, and alert on repeated failures.

  • Versioned artifacts: Record configuration or deployment versions so upgrades, incidents, and rollbacks can be correlated.

Failure Modes and Security

  • set -euo pipefail detects unset variables, failed commands, and failed pipeline components. It does not replace explicit handling of expected failures.

  • Run with the minimum required privileges. This example needs root because it writes under /opt and /var/lock; ordinary applications should use a dedicated service account.

  • Avoid eval, unquoted expansions, predictable temporary names, and embedded secrets. Supply secrets through protected environment files, secret managers, or mounted credentials.

  • A local lock does not coordinate with other machines or protect against an administrator manually changing the destination.

  • Keep previous versions when replacement must be reversible. The example demonstrates atomicity, not a complete rollback system.

Bash Mechanics and POSIX sh

  • Variables, quoting, and arguments: Quote expansions such as "$variable" and "$@". Validate $# before using positional arguments such as "$1".

  • Conditionals, functions, loops, and exits: Use [[ ... ]] for Bash tests, functions for reusable validated operations, and while IFS= read -r line; do ...; done < "$file" for line-safe input. Return 0 for success and a nonzero status for failure.

  • Text processing: Use grep -F for literal matching, sed -E for simple substitutions, and awk for record or field processing. Prefer a schema-aware parser for JSON or XML.

  • Portability: POSIX sh does not provide [[ ... ]] or set -o pipefail; portable scripts use [ ... ] and set -eu. flock, GNU mv -T, stat -c, and mktemp are also non-POSIX. RHEL 9 ships Bash 4.4 by default; this example remains compatible with it.

Best Practices

  • Make desired state explicit and repair drift rather than assuming resources are absent.

  • Guard destructive operations and make every operation safely repeatable.

  • Run the script twice in testing and verify that the second execution reports no change.

  • Validate ShellCheck results and exercise permission failures, unavailable dependencies, and lock contention.

  • Keep backups, versioned artifacts, and rollback procedures separate from idempotency; together they provide convergence and operational recoverability.

49

How do you prevent concurrent execution of the same Bash script?

Use an advisory file lock with flock: every invocation opens the same lock file and atomically acquires an exclusive lock before performing the protected work. This avoids the check-then-write race associated with PID files, provided every cooperating instance uses the same lock.

Read full answer

Applies toBash 5.x and util-linux 2.38+ (`flock`) on Linux

How flock Works

The flock utility places an advisory lock on an open file description. Separate processes opening the same lock file contend for that lock, and the kernel performs acquisition atomically.

The lock is cooperative rather than mandatory:

  • Every instance must use the same lock path and lock operation.

  • Code that ignores the advisory lock can still modify the protected resource.

  • File permissions and service isolation remain separate security controls.

Blocking and Nonblocking Modes

Mode Command Behavior
Nonblocking flock --exclusive --nonblock 9 Returns immediately with a conflict status if the lock is held.
Blocking flock --exclusive 9 Waits until the lock can be acquired.
Timed wait flock --exclusive --timeout 30 9 Waits for at most 30 seconds.

Nonblocking mode is usually preferable for scheduled jobs because it avoids an indefinitely queued invocation. Blocking mode is useful when every invocation must eventually run, but an overrunning job can create a backlog or cause older work to execute later than expected.

Why PID Files Are Riskier

A typical PID-file implementation checks whether a recorded PID is running and then writes its own PID. Because those operations are separate, two processes can both observe an apparently free state and proceed. Stale files and PID reuse introduce further failure modes.

flock removes that acquisition race because the kernel serializes lock acquisition on the open file description.

Bash Versus POSIX sh

flock is a Linux utility rather than a shell feature, but its surrounding script syntax can differ:

  • Bash 5 supports set -euo pipefail, functions, arrays, process substitution, and [[ ... ]].

  • POSIX sh supports set -eu and basic redirection but not pipefail or Bash-specific expansions.

  • A POSIX sh wrapper can still invoke flock when the utility is installed, although locking remains platform-dependent.

Code Example

The following complete Bash script uses a dedicated file descriptor and distinguishes lock contention from other acquisition failures:

#!/usr/bin/env bash
set -euo pipefail

readonly LOCK_FILE="/run/lock/production-sync.lock"

# Every instance must open the same lock file.
exec 9>"$LOCK_FILE"

if flock --exclusive --nonblock --conflict-exit-code 75 9; then
    :
else
    status=$?
    if (( status == 75 )); then
        echo "Another instance is already running; exiting." >&2
        exit 75
    fi

    echo "Could not acquire the lock (flock exit $status)." >&2
    exit "$status"
fi

cleanup() {
    exec 9>&-
}
trap cleanup EXIT
trap 'exit 130' INT
trap 'exit 143' TERM

echo "Acquired the lock; starting the protected work."
# Replace this with the protected operation.
sleep 5
echo "Protected work completed."

First invocation:

Acquired the lock; starting the protected work.
Protected work completed.

Concurrent invocation:

Another instance is already running; exiting.

flock can also wrap a command:

flock --exclusive --nonblock --conflict-exit-code 75 \
    "$LOCK_FILE" /path/to/production-sync

After successful acquisition, this form returns the wrapped command’s exit status. Consequently, the conflict status must be reserved by the application; the file-descriptor form above separates acquisition status from protected-work status.

Production Considerations

  • Use one stable path: Every instance must use exactly the same path. Do not include a timestamp or PID in the filename.

  • Control permissions: Ensure the service account can create the lock file in /run/lock, or configure a dedicated writable runtime directory.

  • Do not unlink the lock file while holding it: Removing it allows another process to create and lock a different inode at the same path. Its mere existence does not mean the lock is held.

  • Handle inherited descriptors: Background children can inherit descriptor 9 and keep the lock after the parent exits. Close it in children that should not hold the lock.

  • Use suitable storage: Coordinate instances through a filesystem with reliable flock semantics; verify distributed-filesystem behavior before coordinating across hosts.

  • Monitor contention: Log contention and acquisition failures, and configure supervisors such as systemd to distinguish a temporary conflict from a failed protected operation.

50

How do you create temporary files safely with mktemp in Bash?

mktemp safely creates temporary files or directories with unpredictable names, avoiding the symlink and overwrite races associated with predictable paths such as /tmp/job.$$. In Bash, combine it with strict error handling, restrictive permissions, and an exit trap to clean up resources automatically.

Read full answer

Applies toBash 5.x and GNU coreutils 9.x on Linux

How mktemp Works

mktemp replaces a template containing consecutive X characters with randomly generated characters and atomically creates the resulting path. It fails rather than overwriting an existing file.

  • File creation: mktemp /tmp/job.XXXXXXXXXX creates an individual file.

  • Directory creation: mktemp -d /tmp/job.XXXXXXXXXX creates a private directory.

  • Permissions: GNU mktemp creates files with user read/write permissions and directories with user read/write/execute permissions, subject to the process umask.

  • Failure behavior: It returns a nonzero exit status if it cannot create the path.

The template must contain at least three consecutive X characters in its final path component. An explicit template is preferable to relying on the default, because GNU and BSD implementations can differ in their default behavior.

Safe Bash Pattern

Use set -euo pipefail to stop on unexpected failures, unset variables, and failed pipeline stages. Set umask 077 before creating sensitive files so that no group or other permissions are granted.

For multiple related temporary files, create a private directory first. Other processes cannot list or enter a mode 700 directory unless they have sufficient privileges.

Install the cleanup trap immediately after creating the directory. Otherwise, a failure while creating a later file could leave the directory behind.

Code Example

#!/usr/bin/env bash
set -euo pipefail

# Prevent new files from receiving group or other permissions.
umask 077

readonly app_name="orders"
readonly tmp_root="${TMPDIR:-/tmp}"

# Atomically create a private directory for this process.
work_dir="$(mktemp -d "${tmp_root}/${app_name}.XXXXXXXXXX")"

cleanup() {
  rm -rf -- "$work_dir"
}
trap cleanup EXIT
trap 'exit 129' HUP
trap 'exit 130' INT
trap 'exit 143' TERM

# Create an unpredictable file inside the private directory.
temp_file="$(mktemp "${work_dir}/payload.XXXXXXXXXX")"

# The file already exists and is writable only by its owner.
printf '%s\n' '{"order_id":123,"status":"pending"}' > "$temp_file"

printf 'Temporary file: %s\n' "$temp_file"
printf 'Content: '
cat "$temp_file"

Example output:

Temporary file: /tmp/orders.K8mP2xQa/payload.7Fg9dLs1
Content: {"order_id":123,"status":"pending"}

The signal traps convert termination signals into controlled exits. The EXIT trap then performs cleanup.

Failure Modes and Security

  • Avoid predictable names: Paths such as /tmp/job.$$ can be guessed, and code that checks before writing may still have a race window.

  • Do not pre-create the final file: Passing an existing path to mktemp causes it to fail instead of securely replacing the path.

  • Do not broaden permissions afterward: Running chmod 666 after creation exposes sensitive data. Create it with the required permissions instead.

  • Do not trust an unvalidated TMPDIR: Use a directory controlled by the service account or administrator.

  • mktemp is not encryption: Other processes running as the same user, root, and potentially swap or recovery mechanisms may access the contents.

  • Traps have limits: An EXIT trap does not run after SIGKILL, a kernel panic, or host power loss.

Production Practices

  • For credentials or sensitive payloads, prefer a service-private tmpfs, such as a dedicated RuntimeDirectory in systemd, rather than the general-purpose /tmp filesystem.

  • Use systemd sandboxing and dedicated service accounts so temporary files are not shared with unrelated applications.

  • Run temporary storage on encrypted filesystems when the host disk or swap may be exposed.

  • Add an external cleanup mechanism, such as an appropriate systemd-tmpfiles policy, for directories left behind after abnormal termination.

  • Avoid promising secure deletion through shred; journaled and copy-on-write filesystems may retain previous contents.

  • mktemp itself is not specified by POSIX. Basic template usage works on GNU and BSD systems, but options such as --tmpdir, --suffix, and --dry-run are not portable. The pipefail option also requires Bash rather than POSIX sh.

51

How do I add timestamps to Bash script logs?

Use timestamped logging through a Bash function that formats each message before writing it to standard output, standard error, or a log file. In production, timestamps should normally use UTC and an unambiguous ISO 8601 format so logs from multiple hosts can be correlated reliably.

Read full answer

Applies toBash 5.x on Linux (Ubuntu 22.04/24.04, RHEL 9)

How It Works

A practical logging function combines four operations:

  1. date -u generates the current UTC timestamp.

  2. printf formats the timestamp, severity, and message consistently.

  3. tee -a writes to a file while also displaying the message.

  4. set -euo pipefail detects unset variables, command failures, and failed pipeline components.

Messages are commonly separated by destination:

  • Standard output: Informational and successful progress messages.

  • Standard error: Warnings and errors.

  • Log file: A persistent copy, when local file logging is required.

Code Example

#!/usr/bin/env bash
set -euo pipefail

LOG_FILE="${LOG_FILE:-./application.log}"

log() {
    local level="$1"
    shift

    local timestamp
    local message

    timestamp="$(date -u '+%Y-%m-%dT%H:%M:%SZ')"
    message="$*"

    # Prevent untrusted input from creating forged log entries.
    message="${message//$'\n'/\\n}"
    message="${message//$'\r'/\\r}"

    if [[ "$level" == "ERROR" ]]; then
        printf '%s [%s] %s\n' "$timestamp" "$level" "$message" \
            | tee -a "$LOG_FILE" >&2
    else
        printf '%s [%s] %s\n' "$timestamp" "$level" "$message" \
            | tee -a "$LOG_FILE"
    fi
}

log INFO "Application started"
log INFO "Processing request id=${REQUEST_ID:-unknown}"

if [[ "${1:-}" == "--fail" ]]; then
    log ERROR "Simulated operation failure"
    exit 1
fi

log INFO "Application stopped"

Example output:

2025-03-08T14:25:36Z [INFO] Application started
2025-03-08T14:25:36Z [INFO] Processing request id=unknown
2025-03-08T14:25:36Z [ERROR] Simulated operation failure

The -a option appends to the file rather than overwriting it. Because tee writes to standard error for errors, monitoring systems can treat error output separately from normal progress output.

Production Practices

  • Prefer structured logging: Emit JSON when a centralized collector such as Loki, Elasticsearch, or Splunk parses the logs. Include fields such as timestamp, level, service, host, and request_id.

  • Use one primary sink: In containers, writing timestamped lines to stdout and stderr is usually enough for journald, Docker, or Kubernetes to collect them. Avoid keeping duplicate local files unless retention requires them.

  • Rotate files: Configure logrotate or the platform’s logging agent to limit file size, retention, and disk usage. A logging failure caused by a full disk can otherwise terminate the script.

  • Protect sensitive data: Never log passwords, access tokens, private keys, or complete request bodies. Redact sensitive fields before formatting the message.

  • Prevent log injection: Convert embedded newlines and carriage returns, as shown above, when values can come from users or remote systems.

  • Synchronize clocks: Configure NTP or chrony. Accurate host time is essential when correlating logs with metrics, traces, and incident timelines.

  • Monitor failures: Alert on repeated error entries, logging errors, rapid log growth, and unexpectedly missing events.

Failure Modes and Portability

  • If LOG_FILE points to a nonexistent directory or an unwritable file, tee fails. With set -euo pipefail, that failure can stop the script.

  • Concurrent scripts writing to the same file can interleave output. Use one writer, append-safe single-line messages, or file locking when strict ordering is required.

  • ${var//pattern/replacement} and [[ ... ]] are Bash features and are not portable to POSIX sh. The pipefail option is also not POSIX.

  • The date format used here is supported by GNU coreutils on the stated Linux distributions. A POSIX shell can still use date -u and printf, but it needs different syntax for parameter substitution, conditionals, and pipeline error handling.

52

How do you check whether a command exists in Bash?

Use the command -v builtin to test whether a command is available before invoking it. It checks the current shell environment, including PATH executables, built-ins, shell functions, and defined aliases.

Read full answer

Applies toBash 5.x; POSIX sh implementations supporting `command -v`

How command -v Works

command -v returns exit status 0 when it finds the requested command and a nonzero status when it does not:

if command -v curl >/dev/null 2>&1; then
  curl --version
else
  printf 'curl is not installed\n' >&2
  exit 127
fi

For an external command, it prints the pathname found on PATH. Redirecting output prevents the path from appearing in logs when only availability matters.

Bash checks its PATH when locating external commands. Consequently, a command may be unavailable even if it is installed in a location that is not present in the current PATH.

Built-Ins, Functions, and External Tools

command -v recognizes more than executable files:

  • Built-ins: Commands such as printf or test are available even without a separate executable file.

  • Functions: A defined Bash function causes the check to succeed.

  • External commands: Bash searches PATH for an executable file.

  • Aliases: command -v can report a defined Bash alias. Whether that alias is expanded later during command parsing is controlled separately by the expand_aliases shell option.

If the operation specifically requires an external executable, Bash’s type -P is stricter because it searches only PATH and ignores built-ins, functions, and aliases:

if type -P jq >/dev/null 2>&1; then
  jq --version
fi

type -P is Bash-specific. Avoid relying on which: it is not POSIX, may be installed separately, and its output can be misleading when multiple PATH entries contain the same command.

Robust Bash Example

The following script checks every dependency before running the main operation and reports all missing commands together.

#!/usr/bin/env bash
set -euo pipefail

required_commands=(curl jq systemctl)
missing_commands=()

for required_command in "${required_commands[@]}"; do
  if ! command -v "$required_command" >/dev/null 2>&1; then
    missing_commands+=("$required_command")
  fi
done

if ((${#missing_commands[@]} > 0)); then
  printf 'Missing required command(s):\n' >&2
  printf '  %s\n' "${missing_commands[@]}" >&2
  exit 127
fi

printf 'All required commands are available.\n'

Example output when jq is absent:

Missing required command(s):
  jq

Exit status 127 is conventionally used to indicate that a required command could not be found. The if ! command -v ... construct is safe with set -e because commands used as conditions are exempt from immediate failure.

Production and Portability Considerations

  • Check the effective PATH: sudo, service managers, containers, and CI runners may use a different PATH from an interactive login shell.

  • Aliases may not be expanded in scripts: Non-interactive shells usually have alias expansion disabled, even if an alias has been defined. Use type -P when an external executable is mandatory.

  • Presence is not readiness: A binary existing does not prove that a service is healthy or that its dependencies are reachable. Production checks should also validate versions, connectivity, and service state.

  • Avoid eval: Never use eval to execute a dynamically selected command. Quote the command variable so it remains one argument.

  • Restrict dynamic commands: If the command name comes from user input, use an explicit allowlist. Checking that a name exists does not make executing arbitrary input safe.

  • POSIX compatibility: command -v is supported by POSIX shells, but arrays, set -o pipefail, and type -P require Bash.

Best Practices

  • Use command -v for normal availability checks.

  • Use type -P when an actual executable in PATH is mandatory.

  • Place dependency checks before the operation that requires them.

  • Quote every expansion and avoid word splitting.

  • Validate the required version, not just the command’s presence.

  • Run shellcheck script.sh and test both successful and missing-command paths before deployment.

53

How does sourcing a Bash script differ from executing it?

Sourcing runs a script’s contents in the current shell, allowing its variables, functions, and other shell state to affect the caller. Executing a script runs it in a separate interpreter process, so most shell-state changes remain isolated from the caller.

Read full answer

Applies toBash 5.x and POSIX `sh` (`.` is POSIX; `source` is not POSIX-standard)

How Sourcing Works

Bash provides the source built-in, while the equivalent POSIX form is the dot command: source ./common.sh and . ./common.sh, respectively.

The sourced file is interpreted by the calling shell rather than launched as another program. It therefore does not require a shebang, although any shebang present is treated as a comment. Its syntax must nevertheless be compatible with the calling shell: Bash-specific syntax cannot be safely sourced by a POSIX sh.

If the filename contains no slash, Bash searches PATH for it. If it contains a slash, Bash resolves it relative to the current working directory. Sourcing is commonly used for function libraries, configuration files, constants, and shell initialization code.

Sourcing can intentionally or accidentally change the caller’s variables, functions, working directory, positional parameters, shell options, traps, and other state. It can also terminate the calling shell with exit, so sourced files should be treated as trusted code with the same privileges as the caller.

How Execution Differs

Running a script as ./script.sh normally uses its shebang to select an interpreter. Running bash script.sh explicitly selects Bash and starts it as a child process.

Behavior Sourcing with source or . Executing a script
Process Uses the current shell Starts a separate interpreter process
Variables Can modify the caller’s variables Changes do not propagate to the caller
Functions Become available to the caller Exist only in the child process
exit Can terminate the calling shell Terminates only the script’s process
Shebang Not required Needed for reliable interpreter selection
Exit status Final sourced-command status, 0 for an empty file, or nonzero if the file cannot be read The executed script’s exit status
Portability . is portable; source is a Bash extension The script can select its own interpreter

External effects still occur when a script is executed: it can modify files, call services, change system configuration, or send signals. Execution isolates shell state, not operating-system side effects.

Arguments and Exit Status

Arguments supplied to source temporarily become the sourced file’s positional parameters. For example, source ./set-mode.sh production makes production available as $1 inside that file. Bash restores the caller’s original positional parameters when the source operation completes.

The status of source is nonzero when the file cannot be found or read. Otherwise, it is the status of the final command executed by the sourced file; an empty file returns 0. An executed script likewise communicates success or failure through its exit status, allowing production job runners to branch on the result.

Practical Guidance and Failure Modes

  • Prefer execution for standalone automation: it prevents accidental changes to the caller’s environment and gives jobs a clear process boundary and exit status.

  • Use sourcing for intentional shared state: it is appropriate for trusted libraries, configuration, and interactive shell setup.

  • Resolve paths explicitly: relative paths are based on the caller’s working directory, not automatically on the location of the calling script. Derive a script directory from a known base when portability matters.

  • Treat sourced code as privileged code: it can overwrite variables, alter shell options, change directories, or terminate the parent shell.

  • Manage error handling deliberately: options such as set -euo pipefail already active in the caller also apply to sourced code, potentially terminating an interactive or automation shell.

  • Check scripts with ShellCheck: quote expansions, use local for function-local variables, and avoid unnecessary aliases or unquoted command substitutions.

  • Test changes to shared libraries: sourcing can affect every later command in the session, so production setup scripts should be validated in a controlled shell before deployment.

For scripts that must run under POSIX sh, use . rather than source and avoid Bash-only features such as arrays, [[ ... ]], local, and set -o pipefail.

54

How do Bash subshells differ from the current shell?

A Bash subshell is a child environment that starts with a copy of the current shell’s state, but changes made inside it do not affect the parent. The current shell is the original Bash process, so assignments, function definitions, option changes, and directory changes made there remain active.

Read full answer

Applies toBash 5.x on Linux

Process Model

The current shell executes commands, built-ins, and sourced files in its existing process. A subshell, commonly created with parenthesized commands such as (command1; command2), runs in a child Bash process with a different process ID and an initially copied working directory and shell state.

Subshells are used for:

  • Parenthesized groups: Commands enclosed in parentheses execute in a subshell.

  • Command substitution: output=$(command) evaluates the command in a subshell environment and captures its standard output.

  • Pipelines: Pipeline components normally execute in subshells, with the lastpipe exception described below.

  • Background commands: A command followed by & executes asynchronously in a child environment.

Starting bash -c 'command' creates a separate Bash process. It provides similar state isolation, but it is not technically a subshell created by the current Bash process. Similarly, an external command such as curl starts as a child process and is normally replaced with that executable using exec; it is not a Bash subshell after that replacement.

Inside a Bash-created subshell, $$ retains the original shell’s process ID, while $BASHPID identifies the current Bash process. A separately started bash -c process has its own value for both.

State and Scope

Aspect Current shell Subshell
Process Existing shell process Child Bash process
Process ID Remains unchanged Normally differs from the parent
Variable assignments Remain active Do not modify the parent
Working directory cd changes the parent’s directory cd affects only the child
Function definitions Remain available Copied initially; changes do not reach the parent
Shell options Apply directly Copied initially; changes remain local
exit Terminates the current shell Terminates only the subshell
Exported environment Inherited by child processes Copied at creation; changes remain local

A subshell can read ordinary, unexported shell variables as well as exported environment variables because it begins as a copy of the parent shell. Ordinary variables are not added to the child process’s exported environment, so an external program started by the subshell does not receive them unless they are explicitly exported.

The lastpipe option is an important exception to normal pipeline isolation. When it is enabled and job control is disabled, the final command in a pipeline can execute in the current shell and therefore retain variable and directory changes. It is enabled by default from Bash 5.1 onward; Bash 5.0 supports it but does not enable it by default.

Exit Status and Error Handling

Immediately after a subshell finishes, its exit status is available in $?. For example, (exit 7) leaves a status of 7 for the next command.

With set -euo pipefail:

  • errexit: Terminates the shell after many unsuccessful simple commands, subject to Bash’s context-sensitive exceptions, such as commands whose status is tested in an if condition.

  • nounset: Treats references to unset variables as errors.

  • pipefail: Makes a pipeline return the status of the rightmost component that exited unsuccessfully, rather than always returning the final component’s status.

These options are inherited by Bash-created subshells, but changing them inside a subshell does not reconfigure the parent. A standalone failing subshell can cause an errexit-enabled parent to terminate, so important statuses should still be handled explicitly with constructs such as if or ||.

Current Shell Versus Sourcing

The source builtin, also available as the POSIX command ., executes a file in the current shell. Variable assignments, function definitions, and other permitted changes from the file remain in that shell.

Running bash script.sh instead starts another Bash process. Changes to variables, functions, options, or the working directory in that script do not affect the caller. This distinction is useful when choosing between loading trusted configuration and running isolated automation.

Production Guidance

  • Use the current shell when configuration must intentionally change variables, functions, options, or the working directory.

  • Use a subshell when temporary state, command substitution, pipeline isolation, or background work is preferable.

  • Do not source untrusted files; sourced code runs with the current shell’s privileges and can alter security-sensitive behavior.

  • Do not rely solely on set -e; check operations whose failure requires rollback, cleanup, alerting, or a specific fallback.

  • Use trap for cleanup, mktemp for temporary files, and explicit argument validation in production scripts.

  • Avoid exposing secrets in command arguments or unnecessarily exporting them, because child processes may inherit exported values and some environments expose process metadata to other privileged users.

  • POSIX sh does not support Bash-only features such as arrays, [[ ... ]], process substitution, and pipefail; use #!/usr/bin/env bash and Bash 5 syntax explicitly when those features are required.

55

Why do Bash variables vanish inside a pipeline loop?

Bash variables appear to vanish inside a pipeline loop because the loop normally runs in a subshell, a child process with its own copy of the shell state. Assignments made there cannot update the parent shell.

Read full answer

Applies toBash 5.x on Linux and other Unix-like systems

How Bash Executes Pipelines

By default, Bash runs each command in a pipeline in a subshell environment. This applies when a while loop is a pipeline component: assignments inside the loop modify only the child shell’s copy of those variables. When that child exits, its changes are discarded, while the parent retains its previous value or remains unset.

Related Bash behaviors follow the same process-isolation principle:

  • Command substitution: result=$(command) runs the command in a subshell, so shell-state changes inside it do not affect the parent.

  • Background execution: A command started with & cannot modify the parent shell’s variables.

  • export: Exporting a variable includes its current value in the environment of subsequently launched children. It does not propagate later changes back to the parent.

  • set -euo pipefail: These options control error handling and pipeline status reporting; they do not share shell state between processes.

There is one Bash-specific exception: shopt -s lastpipe allows the final command of a pipeline to run in the current shell when job control is inactive. It is Bash-only and can change shell state unexpectedly, so it is generally unsuitable as a portable default.

Alternatives and Portability

Method Preserves parent variables Portability Operational notes
Process substitution with while ... done < <(command) Yes Bash only Concise, but the producer’s exit status is not automatically propagated to the loop
Redirect the loop from a regular file Yes Bash and POSIX sh Simple and portable; manage the file securely and remove temporary files reliably
Capture output with command substitution Yes, indirectly Bash and POSIX sh Suitable for output data rather than arbitrary shell-state changes; trailing newlines require care
Enable lastpipe and use an ordinary pipeline Sometimes Bash only The final command runs in the current shell only when job control is inactive
Use an ordinary pipeline No Bash and POSIX sh Each pipeline component retains its isolated shell environment

For POSIX sh, use redirection from a file or a here-document instead of process substitution. Process substitution is convenient in Bash but is not part of the POSIX shell standard.

Production Practices

  • Use IFS= read -r value when processing line-oriented input. An empty IFS preserves leading and trailing whitespace, while -r prevents backslashes from being interpreted as escape sequences.

  • Do not rely on set -e to detect every producer failure. Test important commands explicitly with if or case, and use pipefail when evaluating the status of an entire pipeline.

  • Treat process-substitution producers carefully: their failure status is not automatically the status of the consuming loop. If the failure matters, run the producer separately, check its status, and then consume a securely created temporary file.

  • For text processing with grep, sed, or awk, remember that pipeline output can be transformed successfully while shell-variable changes remain isolated. Materialize the required output before updating parent-shell state.

  • Quote expansions such as "$value" and "$@" to prevent word splitting and glob expansion. Pass function arguments through positional parameters, return meaningful nonzero statuses, and test them explicitly.

  • When using temporary files, create them with secure permissions, restrict access where data is sensitive, and arrange cleanup with a trap so failures or interrupts do not leave artifacts behind.

56

How do Bash scripts handle signals such as SIGINT gracefully?

Use a Bash trap to catch SIGINT and other catchable signals, set a shutdown flag, and clean up child processes before exiting. This creates a graceful shutdown rather than allowing the script and its workloads to terminate abruptly.

Read full answer

Applies toBash 5.x on Linux (Ubuntu 22.04/24.04, RHEL 9)

Signal Handling Model

Signal Typical source Can be trapped? Recommended behavior
SIGINT User presses Ctrl-C Yes Stop work and shut down cleanly
SIGTERM systemd, Kubernetes, process managers Yes Begin graceful termination
SIGHUP Terminal disconnect or session change Yes Persist state or shut down
SIGKILL Forced termination No Use only when graceful shutdown times out
SIGSTOP Process suspension No Cannot be caught or ignored

Bash executes trap commands between shell operations. A trap handler should therefore perform minimal work: update state, record the signal, or forward termination to a child. Do not perform long-running operations directly inside the handler.

EXIT is different from signal traps: it runs during normal shell exit and during a handled shutdown path, but it cannot run after SIGKILL.

Production Behavior

When a user presses Ctrl-C, the terminal sends SIGINT to the entire foreground process group. Consequently, the Bash script and its foreground child may receive the signal independently.

By contrast, kill -TERM <script-pid> targets only the script. If the script does not forward the request, its child process may continue running after the parent exits. A production script should therefore:

  • Trap SIGINT and SIGTERM.

  • Stop starting new work.

  • Allow in-flight work to finish within a defined deadline.

  • Forward SIGTERM to child processes.

  • Clean up temporary files, locks, and other resources.

  • Record enough information for monitoring and incident diagnosis.

When running under systemd or Kubernetes, configure the termination grace period to exceed the script’s cleanup time. If cleanup exceeds that deadline, the supervisor may send SIGKILL, which no script can handle.

Code Example

The following script waits for a background child and handles either SIGINT or SIGTERM. It also uses set -euo pipefail, quoted variables, a function argument, conditional logic, and explicit exit codes.

#!/usr/bin/env bash
set -euo pipefail

child_pid=''
shutdown_requested=0

handle_signal() {
  local signal_name="$1"

  shutdown_requested=1
  printf 'Received %s; shutting down\n' "$signal_name" >&2
}

cleanup() {
  local exit_code="$1"

  # Prevent recursive execution of the EXIT trap.
  trap - EXIT INT TERM

  if [[ -n "$child_pid" ]] && kill -0 "$child_pid" 2>/dev/null; then
    printf 'Stopping child PID %s\n' "$child_pid" >&2
    kill -TERM "$child_pid" 2>/dev/null || true
    wait "$child_pid" 2>/dev/null || true
  fi

  exit "$exit_code"
}

trap 'handle_signal SIGINT' INT
trap 'handle_signal SIGTERM' TERM
trap 'cleanup "$?"' EXIT

# The optional argument controls how long the child runs.
sleep "${1:-300}" &
child_pid=$!
printf 'Started child PID %s\n' "$child_pid"

# Putting wait in an if condition prevents set -e from exiting
# before the signal handler can request graceful shutdown.
if wait "$child_pid"; then
  child_pid=''
else
  child_status=$?

  if (( shutdown_requested == 1 )); then
    exit 0
  fi

  exit "$child_status"
fi

Example output:

Started child PID 12345
^C
Received SIGINT; shutting down
Stopping child PID 12345

The script exits with 0 after handling a requested shutdown. If callers require signal-style status, the shutdown branches can instead use 130 for SIGINT and 143 for SIGTERM, following the conventional 128 + signal number convention.

Best Practices and Failure Modes

  • Keep handlers short: Set variables such as shutdown_requested=1; let a main loop such as while (( ! shutdown_requested )); do ...; done perform the actual shutdown work.

  • Handle child processes explicitly: A signal sent to the parent does not automatically reach children. Forward SIGTERM and wait for them to stop.

  • Do not rely on set -e for signal handling: set -euo pipefail detects command failures, but it does not replace traps or cleanup logic.

  • Avoid SIGKILL as the first choice: It prevents cleanup, state flushing, and diagnostic logging. Use it only as a supervisor’s final fallback.

  • Prefer structured logs: Emit timestamped messages to stderr. For quick incident searches, filter them with journalctl -u <service> | grep -E 'Received SIG(INT|TERM)|Stopping child'.

  • Handle PID 1 carefully: Explicit traps are required because Linux gives PID 1 special default signal behavior. For containers, an init or supervisor that reaps orphaned children is safer than relying on a standalone Bash script.

  • Test every termination path: Exercise Ctrl-C, kill -TERM, child failure, invalid arguments, and supervisor-initiated shutdown. Verify exit codes, orphaned processes, logs, and cleanup of locks or temporary files.

57

How do Bash scripts handle dates, times, and time zones?

Bash scripts typically use GNU date for parsing timestamps and performing calendar calculations, and Bash’s printf time format for lightweight formatting. Store and compare instants in UTC or Unix epoch seconds, then convert to a local time zone only for display or scheduling.

Read full answer

Applies toBash 5.x on Linux with GNU coreutils 9.x

How It Works

Bash does not implement a complete calendar system. Its built-in time facilities work with the system clock and timezone database:

  • printf '%(%Y-%m-%d)T': Formats the current time using strftime-style directives. This is a Bash built-in and avoids starting an external process.

  • EPOCHREALTIME: Contains the current Unix timestamp with microsecond granularity.

  • EPOCHSECONDS: Contains the current Unix timestamp as an integer.

  • SECONDS: Tracks the number of seconds since the shell started, useful for coarse script-relative timing.

  • date: Provides timezone conversion, parsing, epoch conversion, and relative-date calculations.

  • TZ: Controls the timezone used by GNU date and Bash’s time-formatting functions when the system default is not wanted.

Internally, Unix timestamps represent elapsed seconds since 1970-01-01T00:00:00Z. They do not encode leap seconds.

Core Operations

Operation GNU date command
Current UTC timestamp date --utc '+%Y-%m-%dT%H:%M:%SZ'
Current epoch seconds date +%s
Convert epoch to UTC date --utc --date="@1710000000" '+%Y-%m-%dT%H:%M:%SZ'
Parse an ISO timestamp date --utc --date='2025-03-08T12:30:00Z' +%s
Add one UTC calendar day date --utc --date='2025-03-08 00:00:00 UTC +1 day' '+%Y-%m-%dT%H:%M:%SZ'
Add 24 elapsed hours to epoch 1710000000 date --utc --date="@$((1710000000 + 86400))" '+%Y-%m-%dT%H:%M:%SZ'
Format using Bash printf '%(%Y-%m-%d %H:%M:%S %Z)Tn' -1
Convert UTC to a named zone TZ=America/New_York date --date='2025-03-08T12:30:00Z' '+%Y-%m-%d %H:%M:%S %Z'

For arithmetic, prefer integer epoch seconds. Adding 86400 means exactly 24 elapsed hours; a local “tomorrow” can be 23 or 25 hours across daylight-saving transitions.

Code Example

#!/usr/bin/env bash
set -euo pipefail
export LC_ALL=C

parse_iso_utc() {
  local value="$1"
  local pattern='^[0-9]{4}-[0-9]{2}-[0-9]{2}T[0-9]{2}:[0-9]{2}:[0-9]{2}Z$'
  local epoch

  [[ "$value" =~ $pattern ]] || return 1
  epoch="$(date --utc --date="$value" +%s)" || return 1

  # Reject dates that GNU date silently normalized, such as February 31.
  [[ "$(date --utc --date="@${epoch}" '+%Y-%m-%dT%H:%M:%SZ')" == "$value" ]] || return 1
  printf '%s\n' "$epoch"
}

now_epoch="$(date +%s)"
now_utc="$(date --utc '+%Y-%m-%dT%H:%M:%SZ')"
local_now="$(printf '%(%Y-%m-%d %H:%M:%S %Z)T' -1)"
input='2025-03-08T12:30:00Z'

if ! parsed_epoch="$(parse_iso_utc "$input")"; then
  printf 'Invalid UTC timestamp: %s\n' "$input" >&2
  exit 2
fi

# Find the next UTC midnight independently of the host timezone.
tomorrow_midnight="$(date --utc --date='tomorrow 00:00 UTC' +%s)"
seconds_until_midnight=$((tomorrow_midnight - now_epoch))

printf 'UTC now: %s\n' "$now_utc"
printf 'Local now: %s\n' "$local_now"
printf 'Input as epoch: %s\n' "$parsed_epoch"
printf 'Seconds until next UTC midnight: %s\n' "$seconds_until_midnight"

if ((now_epoch < parsed_epoch)); then
  printf 'The input timestamp is in the future.\n'
else
  printf 'The input timestamp is in the past.\n'
fi

Illustrative output on a host configured for UTC:

UTC now: 2025-03-08T10:15:30Z
Local now: 2025-03-08 10:15:30 UTC
Input as epoch: 1741437000
Seconds until next UTC midnight: 49470
The input timestamp is in the future.

Production Practices

  • Use UTC for storage and comparisons: Avoid daylight-saving ambiguity when converting local deployment schedules.

  • Use epoch seconds for elapsed-time calculations: They are simpler to compare and subtract than formatted strings. Use date with an explicit timezone for calendar arithmetic.

  • Validate before parsing: GNU date --date accepts flexible expressions such as tomorrow; strict validation prevents accidental interpretation of malformed input.

  • Handle failures explicitly: With set -euo pipefail, failed date commands normally stop the script. Use if ! value="$(command)" when failure is an expected validation path.

  • Specify recurring schedules and zones: For example, use 02:00 UTC or 02:00 America/New_York. Test schedules around daylight-saving transitions because local 02:00 can be skipped or occur twice.

  • Account for clock synchronization: Epoch calculations depend on accurate system time. Production hosts should use NTP or chrony, and timestamp-related alerts should monitor clock skew.

  • Be explicit about locale: Set LC_ALL=C when parsing predictable numeric or date output.

Portability and Failure Modes

GNU date options are not portable to BSD/macOS:

  • GNU uses date --date='...'.

  • BSD uses date -j -f '<format>' '<value>' '<format>' for parsing.

  • BSD relative calculations use expressions such as date -v+1d.

  • POSIX sh and basic POSIX date provide limited formatting but do not standardize GNU-style parsing or relative-date arithmetic.

For scripts that must run across Linux and macOS, isolate date operations behind functions or use a portable runtime such as Python. Also avoid relying on date for precise elapsed-time measurement in performance-sensitive tooling, because wall-clock adjustments can affect results.

58

How can curl make reliable HTTP requests in Bash scripts?

Use curl with strict error handling, explicit timeouts, controlled retries, and secure authentication options. In production, scripts should distinguish transport failures from HTTP error responses and avoid leaking credentials or retrying non-idempotent operations unintentionally.

Read full answer

Applies toBash 5.x and curl 8.x on Linux (Ubuntu 22.04/24.04, RHEL 9)

How It Works

A typical curl request consists of:

  • URL: The resource to request.

  • Request method: Defaults to GET; use --request POST for other methods.

  • Headers: Added with --header.

  • Payload: Supplied with --data, --data-binary, or --upload-file.

  • Output handling: The response body is written to standard output by default or to a file with --output.

  • Failure detection: --fail-with-body makes curl exit unsuccessfully for HTTP 4xx and 5xx responses while preserving the response body.

Important exit behavior:

  • Network, DNS, TLS, and timeout failures produce a nonzero curl exit status.

  • HTTP 404 or 500 responses normally produce exit status 0 unless --fail or --fail-with-body is used.

  • 3xx responses are not failures; --location follows redirects.

  • --retry retries transient failures such as timeouts and selected HTTP responses, but applications should still use idempotency controls for writes.

Production Script

The following script handles errors, bounds execution time, retries transient failures, restricts redirects and protocols, and stores the response body separately from the HTTP status.

#!/usr/bin/env bash
set -euo pipefail

url="${1:?Usage: $0 https://example.com}"
body_file="$(mktemp)"
trap 'rm -f "$body_file"' EXIT

curl_args=(
  --silent
  --show-error
  --fail-with-body
  --location
  --max-redirs 5
  --connect-timeout 5
  --max-time 30
  --retry 3
  --retry-delay 1
  --retry-max-time 30
  --proto '=https'
  --tlsv1.2
  --output "$body_file"
)

if ! status="$(curl "${curl_args[@]}" --write-out '%{http_code}' "$url")"; then
  printf 'Request failed (HTTP status %s)\n' "$status" >&2
  cat "$body_file" >&2 || true
  exit 1
fi

printf 'HTTP %s\n' "$status"
cat "$body_file"

--connect-timeout limits connection establishment, while --max-time limits the entire request. --retry-max-time prevents retries from extending the operation indefinitely.

Code Example

For a JSON POST request:

#!/usr/bin/env bash
set -euo pipefail

curl \
  --silent \
  --show-error \
  --fail-with-body \
  --request POST \
  --header 'Content-Type: application/json' \
  --data '{"name":"example"}' \
  'https://api.example.com/items'

Use --data-binary @payload.json when the payload must be sent without curl’s newline or form-processing behavior. For large responses, use --output response.json rather than retaining the entire body in a shell variable.

Security and Reliability

  • Credentials: Prefer a secret manager or a protected curl configuration file. Avoid putting tokens in URLs, because URLs commonly appear in logs and browser history.

  • TLS: Prefer HTTPS, validate certificates, and avoid disabling verification with --insecure.

  • Redirects: Restrict redirect targets and limit redirects with --max-redirs. An unrestricted redirect can unexpectedly send credentials to another host.

  • Logging: Record the status code, duration, and a request or trace ID, but do not log authorization headers or sensitive response bodies.

  • Retries: Retrying a POST can create duplicate records. Use an idempotency key or perform the operation in a controlled workflow.

  • Monitoring: Alert on elevated HTTP error rates, latency, timeout counts, and certificate failures rather than treating every nonzero curl exit as the same incident type.

For credentials stored in a mode-0600 curl configuration file:

curl --config "$CURL_CONFIG" 'https://api.example.com/items'

The configuration file should contain the required header entries and should be readable only by the account running the script.

Bash and POSIX Portability

The script uses Bash-specific features such as arrays and set -o pipefail. pipefail is not part of POSIX sh, and arrays are not portable either. A POSIX sh script should use simpler arguments and explicit status checks:

#!/bin/sh
set -eu

curl --silent --show-error --fail-with-body \
  --connect-timeout 5 \
  --max-time 30 \
  'https://example.com'

Use #!/usr/bin/env bash when Bash features are intentional; use #!/bin/sh when the script must run on minimal or non-Bash Unix systems.

Common Pitfalls

  • Omitting --fail-with-body causes HTTP 404 and 500 responses to look successful.

  • Omitting timeouts can leave CI jobs or production workers blocked indefinitely.

  • Using --retry without an idempotency strategy can duplicate writes.

  • Using --insecure hides certificate or hostname validation failures.

  • Interpolating untrusted input directly into a shell command can cause command injection; pass URLs and payloads as quoted arguments.

  • Piping curl directly into another command can obscure failures unless pipefail and explicit error checks are used.

59

How do you parse JSON in Bash with jq?

jq is a command-line JSON processor that parses JSON into values, transforms them with a filter language, and emits JSON or raw text. In Bash, it is commonly used to extract fields, validate response structures, and safely modify configuration data.

Read full answer

Applies tojq 1.7.x, Bash 5.x, and curl 8.x

How jq Processes JSON

jq reads one or more JSON values and applies a filter to each value:

jq '.user.name' response.json

The filter .user.name means:

  • .user: Select the user object.

  • .name: Select its name field.

Common filter operations include:

  • .field: Access an object field.

  • .items[]: Iterate over an array.

  • select(.enabled == true): Keep matching elements.

  • map(...): Transform every element in an array.

  • length: Return the length of an array, object, or string.

  • keys, has, and type: Inspect an object’s structure and values.

A missing field normally evaluates to null, so parsing success does not prove the JSON has the expected schema. Validate important fields explicitly.

Useful jq Options

Option Purpose
-r Print strings without JSON quotation marks.
-c Emit compact, single-line JSON.
-e Set the exit status based on whether jq produced a usable result.
-n Do not read input; useful when constructing JSON.
-s, --slurp Read all input documents into one array.
--arg name value Pass a string argument to a filter.
--argjson name value Pass a JSON value to a filter.
-S Sort object keys for deterministic output.

For extracting a scalar string:

jq -r '.user.name' response.json

For passing shell data into jq, use --arg rather than interpolating the value into the filter:

jq --arg environment production \
  '.environment = $environment' config.json

This prevents shell or input data from being interpreted as jq program code.

Error Handling and Exit Status

Use strict Bash settings in automation scripts:

set -Eeuo pipefail
  • -e: Exit when a command fails.

  • -u: Treat unset variables as errors.

  • pipefail: Detect failures in any stage of a pipeline.

  • -E: Preserve function and trap behavior for error handlers.

With jq -e:

  • Exit status 0 means the final result was neither false nor null.

  • Exit status 1 means the final result was false or null.

  • Exit status 4 means no valid result was produced.

  • Invalid JSON and invalid jq programs also produce non-zero statuses.

Do not rely exclusively on set -e; explicitly handle expected jq failures. This is especially important in conditionals and command substitutions, where Bash temporarily disables the usual errexit behavior.

Code Example

The following Bash 5 script creates sample JSON, extracts data safely, and produces formatted output:

#!/usr/bin/env bash
set -Eeuo pipefail

json_file=$(mktemp)
readonly json_file
trap 'rm -f "$json_file"' EXIT

cat >"$json_file" <<'JSON'
{
  "environment": "production",
  "services": [
    {"id": "svc-1", "name": "api", "status": "active", "enabled": true},
    {"id": "svc-2", "name": "worker", "status": "active", "enabled": false},
    {"id": "svc-3", "name": "legacy", "status": "stopped", "enabled": true}
  ]
}
JSON

if ! active_name=$(jq -er '
  first(
    .services[]
    | select(.status == "active" and .enabled == true)
    | .name
  ) // error("no active enabled service")
' "$json_file"); then
  printf 'Unable to find an active enabled service.\n' >&2
  exit 1
fi

total_services=$(jq -er '.services | length' "$json_file")

printf 'active_service=%s\n' "$active_name"
printf 'total_services=%d\n' "$total_services"

jq -c '.services[] | {id: .id, name: .name, status: .status}' "$json_file"

Example output:

active_service=api
total_services=3
{"id":"svc-1","name":"api","status":"active"}
{"id":"svc-2","name":"worker","status":"active"}
{"id":"svc-3","name":"legacy","status":"stopped"}

Production Best Practices

  • Validate before using values: Check types and required fields with jq -e. jq is not a complete JSON Schema validator; use a schema validator such as ajv for complex contracts.

  • Bound remote input: Download with strict transport and a timeout. --proto-redir '=https' prevents HTTPS redirects to HTTP, while --max-filesize rejects a body whose advertised size exceeds the limit. It is not a hard cap when the response size is unknown or the body is chunked, so enforce a strict limit at a proxy, streaming wrapper, or storage boundary as well:

curl --fail --silent --show-error --location \
  --proto '=https' --proto-redir '=https' \
  --max-time 30 --max-filesize 10485760 \
  'https://api.example.com/payload' -o payload.json
  • Write atomically: When updating configuration, write to a temporary file in the same filesystem, validate it with jq, and then use mv to replace the original.

  • Protect sensitive data: Avoid logging complete payloads or passing credentials directly in command-line arguments. Redact fields before sending JSON to logs or monitoring systems.

  • Preserve exit codes: Treat jq failures as pipeline or deployment failures, and emit useful context to stderr without exposing secrets.

  • Remember portability: jq itself works from POSIX sh, but the example uses Bash features. A POSIX shell script should use #!/bin/sh and avoid arrays, [[ ... ]], and other Bash-only syntax.

60

How can a Bash script load values from a configuration file?

Bash has no built-in parser for arbitrary configuration formats. For simple deployments, key=value parsing with read and an associative array loads values without executing the file; source is appropriate only when the file contains trusted Bash syntax.

Read full answer

Applies toBash 5.x on Linux (Ubuntu 22.04/24.04, RHEL 9)

How Bash Loads Configuration

There are two common approaches:

Approach Executes file contents Supported format Appropriate use
source "$file" Yes Bash syntax Trusted configuration owned by the same team
read plus an associative array No Restricted KEY=VALUE Simple application settings and automation
A dedicated parser No JSON, YAML, TOML, INI Structured or nested configuration

source runs the file in the current shell, so it can overwrite variables, change shell options, define functions, or execute arbitrary commands. For example, source "$CONFIG_FILE" executes the referenced file. Use an absolute or explicitly controlled path because Bash searches PATH when source receives a filename without a slash.

A parser using read treats configuration values as ordinary strings. It does not evaluate commands, expand embedded $variables, remove quotes, or implement JSON/YAML semantics.

Define a Restricted Format

For a simple parser, establish strict rules before loading the file:

  • Each setting uses KEY=VALUE.

  • Keys contain only letters, digits, and underscores and cannot begin with a digit.

  • Blank lines and lines beginning with # are ignored.

  • Values are literal strings; they may contain spaces or additional = characters.

  • Duplicate keys are rejected.

  • Optional whitespace around = and inline comments are not supported unless explicitly parsed.

These restrictions prevent surprising behavior and make validation straightforward.

Code Example

Configuration file:

# service.conf
listen_port=8080
environment=production
allowed_cidr=10.0.0.0/16

Bash loader:

#!/usr/bin/env bash
set -euo pipefail

usage() {
  printf 'Usage: %s CONFIG_FILE\n' "${0##*/}" >&2
}

die() {
  printf 'Error: %s\n' "$*" >&2
  exit 1
}

if (($# != 1)); then
  usage
  exit 2
fi

config_file=$1
[[ -f $config_file ]] || die "not a regular file: $config_file"
[[ -r $config_file ]] || die "cannot read: $config_file"

declare -A config=()
line_number=0

# Read one complete line without interpreting whitespace or backslashes.
while IFS= read -r line || [[ -n $line ]]; do
  ((line_number += 1))

  # Normalize Windows CRLF input.
  line=${line%$'\r'}

  [[ -z $line || $line == \#* ]] && continue
  [[ $line == *=* ]] || die "line $line_number: expected KEY=VALUE"

  key=${line%%=*}
  value=${line#*=}

  # Reject invalid or duplicate keys before storing them.
  [[ $key =~ ^[A-Za-z_][A-Za-z0-9_]*$ ]] ||
    die "line $line_number: invalid key: $key"

  if [[ ${config[$key]+present} ]]; then
    die "line $line_number: duplicate key: $key"
  fi

  config["$key"]=$value
done < "$config_file"

# Require every setting used by this script.
for required_key in listen_port environment allowed_cidr; do
  [[ -n ${config[$required_key]-} ]] ||
    die "missing required key: $required_key"
done

# Validate the port as decimal, including values with leading zeroes.
port=${config[listen_port]}
[[ $port =~ ^[0-9]{1,5}$ ]] || die "listen_port must be numeric"
((10#$port >= 1 && 10#$port <= 65535)) ||
  die "listen_port must be between 1 and 65535"

# Validate IPv4 CIDR notation and the prefix length.
cidr=${config[allowed_cidr]}
[[ $cidr =~ ^([0-9]{1,3})[.]([0-9]{1,3})[.]([0-9]{1,3})[.]([0-9]{1,3})/([0-9]{1,2})$ ]] ||
  die "allowed_cidr must use IPv4 CIDR notation"

for octet in "${BASH_REMATCH[@]:1:4}"; do
  ((10#$octet <= 255)) || die "allowed_cidr contains an invalid octet: $cidr"
done
prefix=${BASH_REMATCH[5]}
((10#$prefix <= 32)) || die "allowed_cidr prefix must be between 0 and 32"

printf 'listen_port=%s\n' "$port"
printf 'environment=%s\n' "${config[environment]}"
printf 'allowed_cidr=%s\n' "$cidr"

Output:

listen_port=8080
environment=production
allowed_cidr=10.0.0.0/16

Production Practices

  • Restrict permissions: Use mode 0600 for files containing secrets, with ownership limited to the service account. The containing directory should also prevent unauthorized replacement.

  • Validate values: Successful parsing does not make a port, timeout, hostname, or CIDR valid. Apply domain-specific checks before operational use.

  • Avoid executing untrusted files: source executes code. Parse non-trusted input instead, and never use eval to process it.

  • Handle secrets deliberately: Prefer a secret manager or restricted runtime secret injection. Environment variables may be visible to the same user or privileged processes.

  • Deploy atomically: Replace configuration using an atomic rename or versioned mount. Retain the previous version for rollback and record the deployed version for incident diagnosis.

  • Monitor and back up: Monitor startup and validation failures without logging values. Maintain access-controlled backups of known-good configuration, and version the schema separately so upgrades can validate or migrate old files safely.

  • Use text tools only for simple extraction: sed -n 's/^listen_port=//p' "$config_file" works for an exact prefix, but it does not detect duplicates or enforce a schema. grep, sed, and awk are useful for inspection, not structured parsing.

Failure Modes and Portability

  • set -euo pipefail catches many failures early, but it is not a substitute for explicit validation. Commands used in conditional tests may intentionally return nonzero.

  • pipefail affects pipelines only; it does not detect an invalid configuration line by itself.

  • read stores strings. Validate them before arithmetic use, and use 10# to force decimal interpretation of leading-zero values.

  • Bash associative arrays require Bash 4 or newer and are not available in POSIX sh.

  • POSIX sh also lacks [[ ... ]], =~, pipefail, and source; use portable test expressions and . instead.

  • For JSON, YAML, TOML, or nested INI files, use a dedicated parser rather than extending a KEY=VALUE parser with increasingly complicated substitutions.

61

How do you validate Bash arguments and print usage?

Argument validation checks required options, rejects unexpected values, and returns meaningful exit codes before the script performs any operation. A centralized usage function provides consistent help output, sent to stdout for -h and stderr for errors.

Read full answer

Applies toBash 5.x (including Bash 5.1 on RHEL 9); the shown syntax also works in Bash 4.4

Argument Validation Model

A robust Bash script should:

  • Use getopts for options and validate each option explicitly.

  • Check that required options were supplied and reject unexpected positional arguments.

  • Validate semantic constraints such as file existence, readability, and destination permissions.

  • Quote every variable expansion to preserve filenames containing spaces or shell metacharacters.

  • Return 0 for success, a usage error for invalid arguments, and distinct codes for input or output failures.

set -euo pipefail provides early failure for unhandled errors, but it does not replace explicit argument checks. In particular, set -e is suppressed for commands tested inside conditionals such as if and while.

Usage and Exit Codes

  • Successful help request: usage writes to stdout, and the script exits 0.

  • Invalid arguments: errors and usage are written to stderr, with exit code 64, conventionally EX_USAGE.

  • Invalid source: exit code 66, conventionally EX_NOINPUT.

  • Invalid destination: exit code 73, conventionally EX_CANTCREAT.

  • Successful validation: exit code 0.

Keeping help on stdout and errors on stderr lets callers distinguish documentation from failure and supports Unix pipelines and CI systems. The example below performs validation only; it does not copy the source file.

Bash and POSIX Compatibility

The example uses Bash features such as [[ ... ]], (( ... )), and set -o pipefail. POSIX sh requires [ ... ], no arithmetic (( ... )), and has no pipefail option. Although getopts is POSIX-compatible, rewriting the example for sh still requires replacing the Bash-specific syntax and avoiding Bash arrays and process substitution.

Code Example

#!/usr/bin/env bash
set -euo pipefail

readonly PROGRAM_NAME="${0##*/}"

usage() {
    cat <<EOF
Usage: ${PROGRAM_NAME} -s <source> -d <destination> [-v] [-h]

Options:
  -s <source>      Readable source file
  -d <destination> Destination file
  -v               Print the validated paths
  -h               Show this help message
EOF
}

source_file=''
destination_file=''
verbose=false

while getopts ':s:d:vh' option; do
    case "$option" in
        s)
            source_file="$OPTARG"
            ;;
        d)
            destination_file="$OPTARG"
            ;;
        v)
            verbose=true
            ;;
        h)
            usage
            exit 0
            ;;
        :)
            printf 'Error: option "-%s" requires an argument.\n' "$OPTARG" >&2
            usage >&2
            exit 64
            ;;
        \?)
            printf 'Error: unknown option "-%s".\n' "$OPTARG" >&2
            usage >&2
            exit 64
            ;;
    esac
done

shift "$((OPTIND - 1))"

# Reject positional arguments left after option parsing.
if (( $# != 0 )); then
    printf 'Error: unexpected argument "%s".\n' "$1" >&2
    usage >&2
    exit 64
fi

if [[ -z "$source_file" || -z "$destination_file" ]]; then
    printf 'Error: both -s and -d are required.\n' >&2
    usage >&2
    exit 64
fi

if [[ ! -f "$source_file" ]]; then
    printf 'Error: source is not a regular file: %s\n' "$source_file" >&2
    exit 66
fi

if [[ ! -r "$source_file" ]]; then
    printf 'Error: source is not readable: %s\n' "$source_file" >&2
    exit 66
fi

if [[ -d "$destination_file" ]]; then
    printf 'Error: destination is a directory: %s\n' "$destination_file" >&2
    exit 73
fi

destination_dir=$(dirname -- "$destination_file")
readonly destination_dir

if [[ ! -d "$destination_dir" ]]; then
    printf 'Error: destination directory does not exist: %s\n' "$destination_dir" >&2
    exit 73
fi

if [[ ! -w "$destination_dir" ]]; then
    printf 'Error: destination directory is not writable: %s\n' "$destination_dir" >&2
    exit 73
fi

if [[ "$verbose" == true ]]; then
    printf 'Validated: "%s" -> "%s"\n' "$source_file" "$destination_file"
fi

Example output:

$ printf 'id,name\n1,alice\n' > report.csv
$ ./validate-args.sh -s report.csv -d report-copy.csv -v
Validated: "report.csv" -> "report-copy.csv"

Production Practices

  • Keep validation close to the start of the script so invalid input cannot partially modify production state.

  • Validate permissions and destination directories before beginning a potentially long copy or deployment.

  • Use dirname --, mkdir --, cp --, and similar commands to handle paths beginning with -.

  • Avoid eval; it can reinterpret filenames as commands and creates injection risks.

  • Treat validation as a time-of-check/time-of-use boundary. In concurrent systems, recheck conditions immediately before the destructive operation.

  • Define overwrite behavior explicitly. The example permits an existing regular destination, but production scripts should state whether replacement, backup, or failure is required.

  • If a script supports both options and positional arguments, remember that simple getopts loops generally stop processing options once a positional argument is encountered.

62

How do you build portable POSIX-compliant shell scripts?

POSIX portability means writing shell code that runs consistently under a conforming POSIX shell, such as dash or ksh93, rather than relying on shell-specific extensions. Quote expansions, use standardized syntax and utilities, handle failures explicitly, and test with the target POSIX shell.

Read full answer

Applies toPOSIX.1-2024 (IEEE Std 1003.1-2024) Shell; Bash 5.x for comparison

Core Portability Rules

  • Use #!/usr/bin/env bash for Bash 5 scripts. For a genuinely POSIX-only deployment, normally use #!/bin/sh; the example below can also be tested with sh backup_file.sh.

  • Use POSIX syntax:

  • Use [ ... ] or case, not Bash’s [[ ... ]].

  • Use name() rather than the non-POSIX function name() form.

  • Use . script instead of Bash’s source.

  • Avoid arrays, mapfile, readarray, process substitution, and here-strings.

  • Quote variable expansions: "$file", "$value", "$@". Unquoted expansions can undergo field splitting and pathname expansion.

  • Use printf rather than echo, whose handling of options and backslashes varies between implementations.

  • Set IFS explicitly when parsing predictable input, and use read -r to preserve backslashes.

  • Use POSIX utilities and options. Avoid extensions such as grep -P, GNU-specific sed behavior, and non-POSIX date formats.

Error Handling and Exit Codes

For a POSIX script, use:

set -eu
  • -e exits when an ordinary simple command fails. Commands used as if, while, or until conditions, non-final commands in && or || lists, and other specified tested contexts are exempt. A failure in an ordinary loop-body command can still terminate the script.

  • -u rejects references to unset variables. Use ${value:-default} when an unset value is acceptable.

  • POSIX sh does not support set -o pipefail; a pipeline’s status is the status of its final command. Check important intermediate operations explicitly.

  • Bash 5 supports set -euo pipefail, but it does not replace explicit validation.

  • Use exit 0 for success and document distinct nonzero statuses, such as 2 for invalid usage and 1 for an operational failure.

Portable Text Processing

  • Use [[:space:]] instead of non-portable regular-expression shorthands such as \s or \d.

  • Use grep -E for POSIX extended regular expressions; grep -P is not portable.

  • Use awk for field-based processing. Its fields are whitespace-separated by default; use -F when a specific delimiter is required.

  • Set LC_ALL=C for deterministic sorting and character classification. Sort unordered output when stable ordering matters:

LC_ALL=C awk '$9 ~ /^[45][0-9][0-9]$/ { count[$9]++ }
END { for (code in count) print code, count[code] }' access.log | LC_ALL=C sort

Code Example

This script validates arguments, checks the source file, creates a private backup directory, and reports the copied file.

#!/usr/bin/env bash
# POSIX-compatible body; test with: sh backup_file.sh
set -eu
umask 077

usage() {
    printf 'Usage: %s SOURCE_FILE DESTINATION_DIRECTORY\n' "$0" >&2
}

copy_file() {
    src=$1
    dst_dir=$2
    name=${src##*/}

    if [ ! -f "$src" ] || [ ! -r "$src" ]; then
        printf 'Source file is missing or unreadable: %s\n' "$src" >&2
        return 1
    fi

    if [ -z "$name" ]; then
        printf 'Invalid source path: %s\n' "$src" >&2
        return 1
    fi

    if ! mkdir -p "$dst_dir"; then
        printf 'Cannot create destination directory: %s\n' "$dst_dir" >&2
        return 1
    fi

    backup_dir="$dst_dir/$(date -u '+%Y%m%dT%H%M%SZ').$$"

    if ! mkdir "$backup_dir"; then
        printf 'Cannot create backup directory: %s\n' "$backup_dir" >&2
        return 1
    fi

    if ! cp "$src" "$backup_dir/$name"; then
        printf 'Backup copy failed: %s\n' "$src" >&2
        return 1
    fi

    printf 'Created backup: %s\n' "$backup_dir/$name"
}

if [ "$#" -ne 2 ]; then
    usage
    exit 2
fi

copy_file "$1" "$2"

Example output:

Created backup: /var/backups/myapp/20250308T142233Z.1234/config.yaml

Production Practices and Bash Differences

Bash feature POSIX-compatible alternative
[[ ... ]] [ ... ] or case
${value,,} printf '%s' "$value" | tr '[:upper:]' '[:lower:]'
Arrays Positional parameters or a controlled temporary file
local Avoid it or use uniquely prefixed variables
source file . file
set -o pipefail Check critical operations explicitly
mapfile or readarray while IFS= read -r line
echo printf
  • Run shellcheck -s sh script.sh in CI and execute tests with the target /bin/sh.

  • Before OS upgrades, retest scripts because the implementation or version of /bin/sh may change.

  • Avoid eval, untrusted input, and logging of passwords, tokens, or secrets.

  • Use restrictive permissions such as umask 077 for backups and temporary files.

  • POSIX does not define an atomic temporary-file utility; mktemp is widely available but non-POSIX.

  • Coordinate concurrent runs with locking and define stale-lock recovery.

  • During incidents, rely on explicit error messages and exit statuses, monitor repeated failures, and verify backup integrity and restoration separately.

63

When should you prefer Python over Bash for automation?

Python is preferable when automation requires complex data handling, structured error handling, reusable logic, or integrations beyond simple command orchestration. Bash remains the better choice for lightweight Linux automation that primarily connects command-line tools.

Read full answer

Applies toBash 5.x and Python 3.x

Core Decision

Use Bash when the script mainly performs operational tasks such as:

  • Running commands and checking their exit statuses.

  • Managing files, directories, permissions, and processes.

  • Piping command output through grep, sed, awk, or similar tools.

  • Implementing straightforward conditionals, loops, and functions.

  • Acting as glue for existing CLI tools in CI/CD or incident-response workflows.

Choose Python when the task involves:

  • Parsing JSON, YAML, XML, CSV, or binary data reliably.

  • Calling REST APIs or maintaining state across multiple operations.

  • Implementing reusable business logic, data models, or algorithms.

  • Working with complex data structures such as dictionaries and lists.

  • Requiring unit tests, type checking, structured exceptions, or richer logging.

  • Supporting multiple operating systems without relying on Bash-specific behavior.

  • Running longer-lived processes that need timeouts, retries, or concurrency.

Bash and Python Compared

Concern Bash Python
Primary strength Connecting Unix command-line tools General-purpose programming and structured automation
Data handling Strings, streams, and command output Lists, dictionaries, objects, and typed data
Error handling Exit codes, trap, and set -euo pipefail Exceptions and explicit exception handling
Reusability Functions work, but shell state can become fragile Modules, packages, classes, and dependency management
Portability Best on Linux; Bash features may not work with POSIX sh Highly portable across Linux, macOS, and Windows
Dependencies Usually only standard shell utilities May require packages such as requests or PyYAML
Operational overhead Minimal; suitable for ephemeral containers Requires the Python runtime and installed dependencies
Best fit Small system-administration and CI/CD scripts Complex integrations, data processing, and maintainable tooling

How They Behave Under the Hood

Bash is a shell interpreter optimized for command execution. A command such as curl or jq is normally a separate process. Quoting, word splitting, glob expansion, and pipelines can therefore change behavior unexpectedly. With set -o pipefail, a pipeline fails if any participating command fails rather than only checking the final command.

Python manages control flow and data directly in its runtime. It can decode JSON into dictionaries, validate values, catch specific exceptions, and decide whether an operation should be retried. When Python launches another process, arguments are passed as a list without a shell by default, avoiding accidental shell expansion.

This difference is especially important for untrusted input:

  • In Bash, quote expansions such as "$user_input" and avoid eval.

  • In Python, use subprocess.run(["command", argument], check=True) instead of constructing a shell command string.

  • Never pass secrets as command-line arguments when they may appear in process listings or logs.

Production Considerations

Python scripts generally provide better observability and failure isolation, but they also introduce operational responsibilities:

  • Dependencies: Pin dependencies, build reproducible virtual environments or containers, and verify them in CI.

  • Secrets: Read credentials from the platform’s secret manager or environment variables rather than source files.

  • Retries: Retry only transient failures and use exponential backoff with jitter; do not blindly repeat non-idempotent operations.

  • Timeouts: Set explicit timeouts for network calls and external commands.

  • Logging: Emit structured logs with relevant request IDs, resource names, and timestamps, while excluding credentials.

  • Exit behavior: Return a nonzero exit status for failed automation so CI/CD and orchestration systems can detect the failure.

  • Upgrades: Test Python scripts against the runtime version included in the production image.

Bash has a smaller deployment footprint and often fits naturally into Linux images. However, a growing Bash script becomes difficult to test and may hide important state in files, environment variables, and temporary directories.

Portability and Common Pitfalls

  • Bash is not POSIX sh. Features such as arrays, [[ ... ]], process substitution, and pipefail require Bash. Use #!/usr/bin/env bash and set -euo pipefail for Bash scripts.

  • set -e has contextual behavior and is not a complete substitute for explicit error handling. Check important commands, preserve failure details, and clean up temporary resources.

  • Avoid parsing JSON with grep or sed; use jq or Python’s json module.

  • Do not select Python merely because it is “more powerful.” For a short script that only invokes a few trusted Linux commands, Bash is simpler and easier to operate.

  • Select Python when the increased maintainability, portability, and reliability justify the additional runtime and dependency management.

64

How do I create timestamped directory backups with 7-day retention?

Timestamped directory backups combine an immutable archive name with age-based cleanup. The Bash script below creates a compressed tar.gz archive for each source directory and removes matching archives older than seven days.

Read full answer

Applies toBash 5 with GNU coreutils, GNU tar, and GNU findutils on Linux (Ubuntu 22.04/24.04, RHEL 9)

How It Works

  • The script accepts a source directory and a backup root.

  • It resolves canonical paths with realpath and uses a hash of the canonical source path so sources with identical basenames remain separate.

  • A UTC timestamp with nanosecond precision identifies when the backup was created.

  • An mktemp-derived suffix and no-clobber publication prevent an existing archive from being overwritten.

  • The archive is written to a temporary file and renamed only after tar succeeds.

  • find removes matching archives whose modification time is older than 10080 minutes, approximately seven days.

Code Example

#!/usr/bin/env bash
set -Eeuo pipefail
umask 077

usage() {
  printf 'Usage: %s SOURCE_DIRECTORY BACKUP_ROOT\n' "$0"
}

if (( $# != 2 )); then
  usage >&2
  exit 2
fi

source_dir=$1
backup_root=$2

if [[ ! -d "$source_dir" ]]; then
  printf 'Source is not a directory: %s\n' "$source_dir" >&2
  exit 1
fi

if ! source_dir=$(realpath -e -- "$source_dir"); then
  printf 'Unable to resolve source directory: %s\n' "$1" >&2
  exit 1
fi

if [[ "$source_dir" == "/" ]]; then
  printf 'Backing up the filesystem root is not supported.\n' >&2
  exit 1
fi

source_parent=$(dirname -- "$source_dir")
source_name=$(basename -- "$source_dir")

mkdir -p -- "$backup_root"

if ! backup_root=$(realpath -e -- "$backup_root"); then
  printf 'Unable to resolve backup root: %s\n' "$2" >&2
  exit 1
fi

source_hash=$(printf '%s' "$source_dir" | sha256sum)
source_hash=${source_hash%% *}
source_hash=${source_hash:0:12}

backup_dir="$backup_root/${source_name}_${source_hash}"
mkdir -p -- "$backup_dir"

if ! backup_dir=$(realpath -e -- "$backup_dir"); then
  printf 'Unable to resolve backup directory: %s\n' "$backup_dir" >&2
  exit 1
fi

if [[ "$backup_dir" == "$source_dir" || "$backup_dir" == "$source_dir/"* ]]; then
  printf 'Backup directory must not be inside the source directory.\n' >&2
  exit 1
fi

tmp_archive=""
cleanup() {
  local exit_status=$?

  if [[ -n "$tmp_archive" ]]; then
    if ! rm -f -- "$tmp_archive"; then
      printf 'Failed to remove temporary archive: %s\n' "$tmp_archive" >&2
    fi
  fi

  return "$exit_status"
}
trap cleanup EXIT

timestamp=$(date -u +%Y%m%dT%H%M%S.%NZ)
tmp_archive=$(mktemp "$backup_dir/.backup.XXXXXXXX")
archive_suffix=${tmp_archive##*.}
final_archive="$backup_dir/backup_${timestamp}_${archive_suffix}.tar.gz"

tar -czf "$tmp_archive" -C "$source_parent" -- "$source_name"

if ! mv -n -- "$tmp_archive" "$final_archive"; then
  printf 'Unable to publish backup archive: %s\n' "$final_archive" >&2
  exit 1
fi

if [[ -e "$tmp_archive" ]]; then
  printf 'Archive name collision; refusing to overwrite: %s\n' "$final_archive" >&2
  exit 1
fi
tmp_archive=""

find "$backup_dir" \
  -maxdepth 1 \
  -type f \
  -name 'backup_*.tar.gz' \
  -mmin +10080 \
  -exec rm -f -- {} +

printf 'Created backup: %s\n' "$final_archive"

Run it with:

chmod 750 ./backup.sh
./backup.sh /srv/app /var/backups

A typical result is:

Created backup: /var/backups/app_a1b2c3d4e5f6/backup_20250308T142530.123456789Z_a1B2c3D4.tar.gz

Production Behavior

The archive contains the source directory itself. Restore it under a chosen parent directory with:

tar -xzf /var/backups/app_a1b2c3d4e5f6/backup_20250308T142530.123456789Z_a1B2c3D4.tar.gz \
  -C /restore/location

Cleanup uses the archive file’s modification time, not timestamps inside the archive. A successful backup therefore becomes eligible for deletion approximately seven days after creation. The script can run under cron or a systemd timer; capture exit status and output in centralized logging and monitor free disk space, backup age, creation failures, and cleanup failures.

Failure Modes and Security

  • set -Eeuo pipefail stops the script on errors such as permission failures, disk exhaustion, or a failed tar operation.

  • The EXIT trap removes an unpublished temporary archive after failure.

  • umask 077 prevents newly created directories and archives from being broadly readable.

  • Mutable application data may produce an inconsistent archive. Quiesce writes or use filesystem and application-consistent snapshots where required.

  • Local backups do not protect against host or filesystem loss; replicate encrypted copies offsite.

  • tar does not encrypt archives. Use encryption-at-rest or encrypt the copy before offsite transfer.

  • Periodically test a full restore; tar -tzf validates readability but does not replace a restore test.

Portability and Best Practices

This is Bash 5-specific rather than POSIX sh: pipefail, GNU realpath, GNU date nanoseconds, GNU find, and GNU mv options are used. It should not be run unchanged on minimal BSD or BusyBox systems.

Use flock when multiple invocations could run concurrently, retain more than one successful copy for recovery testing, and alert on failed or overdue backups rather than assuming that the absence of an alert means backups are healthy.

65

How to monitor disk usage and alert above 80 percent in Bash?

The script uses disk-usage threshold monitoring: df reads filesystem capacity metadata, Bash compares the reported percentage against 80, and an alert is sent when usage is strictly greater than 80%. It supports multiple mount points and can deliver alerts through stderr or an HTTPS webhook.

Read full answer

Applies toBash 5.x and GNU coreutils 8.32+ on Linux (Ubuntu 22.04/24.04, RHEL 9)

How It Works

  • df --output=pcent returns a header plus the percentage used by one filesystem. GNU df rejects -P combined with --output, so the two are not mixed.

  • df uses filesystem metadata such as statfs or statvfs; it does not walk every file, so checks are inexpensive.

  • GNU df rounds the displayed percentage upward. The script checks the reported whole-number percentage against the threshold.

  • Each configured mount point is checked independently. A different filesystem can have a different threshold.

  • If df or the webhook fails, the script records a nonzero exit status while continuing to check remaining mount points.

  • No webhook URL means alerts are written to stderr, which cron can email or a container platform can collect.

Code Example

Save as disk_usage_monitor.sh:

#!/usr/bin/env bash
set -Eeuo pipefail

THRESHOLD_PERCENT="${THRESHOLD_PERCENT:-80}"
ALERT_WEBHOOK_URL="${ALERT_WEBHOOK_URL:-}"

if ! [[ "$THRESHOLD_PERCENT" =~ ^[0-9]+$ ]] ||
   (( THRESHOLD_PERCENT < 1 || THRESHOLD_PERCENT > 100 )); then
  printf 'ERROR: THRESHOLD_PERCENT must be an integer from 1 to 100\n' >&2
  exit 2
fi

readonly THRESHOLD_PERCENT
readonly ALERT_WEBHOOK_URL

HOSTNAME_VALUE="$(hostname -f 2>/dev/null || hostname)"
readonly HOSTNAME_VALUE

json_escape() {
  local value="$1"
  value=${value//\\/\\\\}
  value=${value//\"/\\\"}
  value=${value//$'\n'/\\n}
  value=${value//$'\r'/\\r}
  value=${value//$'\t'/\\t}
  printf '%s' "$value"
}

send_alert() {
  local mountpoint="$1"
  local usage="$2"
  local message="$3"
  local payload
  local usage_json

  if [[ "$usage" == "unknown" ]]; then
    usage_json=null
  else
    usage_json="$usage"
  fi

  printf -v payload \
    '{"host":"%s","mountpoint":"%s","usage_percent":%s,"threshold_percent":%s,"message":"%s"}' \
    "$(json_escape "$HOSTNAME_VALUE")" \
    "$(json_escape "$mountpoint")" \
    "$usage_json" \
    "$THRESHOLD_PERCENT" \
    "$(json_escape "$message")"

  printf 'DISK ALERT: %s\n' "$message" >&2

  if [[ -n "$ALERT_WEBHOOK_URL" ]]; then
    if ! command -v curl >/dev/null 2>&1; then
      printf 'ERROR: curl is required for webhook delivery\n' >&2
      return 1
    fi

    if ! curl --fail --silent --show-error \
      --connect-timeout 5 \
      --max-time 10 \
      --header 'Content-Type: application/json' \
      --data "$payload" \
      --url "$ALERT_WEBHOOK_URL" >/dev/null; then
      printf 'ERROR: webhook delivery failed for %s\n' "$mountpoint" >&2
      return 1
    fi
  fi
}

MOUNTPOINTS=("$@")
if (( ${#MOUNTPOINTS[@]} == 0 )); then
  MOUNTPOINTS=("/")
fi

status=0

for mountpoint in "${MOUNTPOINTS[@]}"; do
  printf 'Checking %s\n' "$mountpoint"

  if ! df_output="$(df --output=pcent -- "$mountpoint" 2>&1)"; then
    printf 'ERROR: df failed for %s: %s\n' "$mountpoint" "$df_output" >&2
    send_alert "$mountpoint" unknown \
      "Unable to query disk usage for ${mountpoint}" || status=1
    continue
  fi

  usage="$(awk 'NR == 2 { gsub(/%/, "", $1); print $1 }' <<< "$df_output")"

  if ! [[ "$usage" =~ ^[0-9]+$ ]] || (( usage > 100 )); then
    send_alert "$mountpoint" unknown \
      "Invalid disk-usage result for ${mountpoint}" || status=1
    continue
  fi

  if (( usage > THRESHOLD_PERCENT )); then
    send_alert "$mountpoint" "$usage" \
      "Disk usage on ${mountpoint} is ${usage}%, above threshold ${THRESHOLD_PERCENT}%" ||
      status=1
  fi
done

exit "$status"

Example execution:

export ALERT_WEBHOOK_URL='https://alerts.example.com/webhooks/disk'
chmod 0700 disk_usage_monitor.sh
shellcheck disk_usage_monitor.sh
./disk_usage_monitor.sh / /var/lib/docker

Possible output:

Checking /
DISK ALERT: Disk usage on / is 84%, above threshold 80%

Production Considerations

  • Schedule the check every five minutes with a systemd timer or cron. A cron entry can run:

cron */5 * * * * /usr/local/sbin/disk_usage_monitor.sh / /var/lib/docker >>/var/log/disk-usage-monitor.log 2>&1

  • A successful alert leaves exit status 0; query or delivery failures return 1. This avoids treating a correctly delivered threshold alert as a script failure.

  • The script alerts on every execution while usage remains high. Configure grouping, suppression, or deduplication in the alert provider to prevent notification storms.

  • Store webhook credentials outside the script in a protected systemd EnvironmentFile or a secret manager. Require HTTPS and do not disable TLS certificate verification.

  • Check inode exhaustion separately with df -i; a filesystem can run out of inodes while retaining free data blocks.

  • Capacity alerts do not replace backups, filesystem health monitoring, or proactive expansion. Use per-filesystem thresholds because application data, logs, and container storage often grow at different rates.

Portability and Common Pitfalls

  • This is a Bash script, not POSIX sh: it uses arrays, [[ ... ]], pipefail, and Bash-specific parameter expansion.

  • df --output=pcent is a GNU coreutils feature. BSD or macOS df generally lacks it; use df -P "$mountpoint" and parse the standard output instead.

  • In containers, check the host’s mount namespace. A container may see only its overlay filesystem while the host’s data volume is approaching capacity.

  • Avoid hard-coded webhook URLs, credentials, or mount paths in source control. Separate environment-specific configuration from the reusable script.

66

How do you script top IP counts from nginx access logs?

Top IP counting identifies which client addresses generated the most requests by aggregating the first field of each nginx access-log entry. The following Bash script handles arguments safely, fails on pipeline errors, supports IPv4 and IPv6 addresses, and returns the highest-volume addresses first.

Read full answer

Applies toBash 5.x on Linux (Ubuntu 22.04/24.04, RHEL 9); nginx open-source 1.x using the default `combined` log format

Approach and Assumptions

The default nginx combined log format begins with $remote_addr, so awk can treat the first whitespace-separated field as the client IP.

The script:

  • Accepts a log file as the first argument.

  • Accepts the number of addresses to display as the second argument, defaulting to 10.

  • Uses set -euo pipefail so undefined variables, unsuccessful commands, and pipeline failures are detected.

  • Counts every request represented by a log line; it does not calculate unique visitors.

  • Sorts by request count descending and breaks ties by IP address.

Code Example

Save this as count_nginx_ips.sh:

#!/usr/bin/env bash
set -euo pipefail
export LC_ALL=C

usage() {
  printf 'Usage: %s ACCESS_LOG [TOP_COUNT]\n' "${0##*/}" >&2
}

if (( $# < 1 || $# > 2 )); then
  usage
  exit 64
fi

log_file=$1
top_n=${2:-10}

if [[ ! $top_n =~ ^[1-9][0-9]*$ ]]; then
  printf 'TOP_COUNT must be a positive integer.\n' >&2
  exit 64
fi

if [[ ! -f $log_file || ! -r $log_file ]]; then
  printf 'Cannot read regular log file: %s\n' "$log_file" >&2
  exit 66
fi

# Prevent a filename beginning with "-" from being interpreted as an option.
case "$log_file" in
  /*) input_path=$log_file ;;
  *)  input_path="./$log_file" ;;
esac

awk '
{
  ip = $1
  if (ip != "" && ip != "-") {
    counts[ip]++
  }
}
END {
  for (ip in counts) {
    printf "%.0f\t%s\n", counts[ip], ip
  }
}' "$input_path" |
  sort -t $'\t' -k1,1nr -k2,2 |
  awk -v limit="$top_n" 'NR <= limit { print }'

Run it with:

chmod 0755 count_nginx_ips.sh
./count_nginx_ips.sh /var/log/nginx/access.log 20

Example output has the request count followed by the address:

  • 95 203.0.113.10

  • 71 2001:db8::42

  • 43 198.51.100.25

Under the Hood

awk stores each address in an associative array and increments its counter. The END block emits one count<TAB>IP record per unique address. sort orders the records numerically by descending count, while the final awk retains the requested number of results.

Unlike piping into head, this implementation lets the final awk consume the complete sorted stream. That avoids SIGPIPE-related false failures when pipefail is enabled.

Production Considerations

  • Proxies and realip: $remote_addr may represent a load balancer rather than the end client. Enable nginx’s realip module only for explicitly trusted proxy ranges; trusting arbitrary X-Forwarded-For values allows clients to spoof addresses.

  • Shared addresses: One IP can represent many users through NAT, VPN, mobile carrier, or CDN infrastructure. Do not block an address solely because it has a high request count.

  • Active-file consistency: Log rotation can occur while the file is being read. For incident analysis, preserve and process the relevant rotated file or an immutable snapshot.

  • Volume: The awk array uses memory proportional to the number of unique IPs, while sort may require temporary disk space. Very high-cardinality logs may need an external analytics pipeline.

  • Compression: This script expects an uncompressed regular file. Compressed rotated logs must first be processed with gzip -cd.

Failure Modes and Best Practices

  • Keep the log on the same reliable storage used during incidents, and retain rotated archives according to the organization’s investigation requirements.

  • Monitor sudden changes in request volume per address; a sharp increase may indicate scanning or abuse, but should be correlated with nginx status codes, request paths, and upstream logs.

  • Validate the log format after nginx configuration or version upgrades. If the format changes and $remote_addr is no longer first, update the extractor.

  • Use UTC timestamps across infrastructure so request spikes can be correlated with deployments, incidents, and monitoring alerts.

  • Treat IP addresses as operational and potentially personal data. Restrict access to detailed logs and avoid publishing raw client-address reports publicly.

  • Keep PATH predictable when running the script from automation, but avoid unnecessary privilege. Read-only access to the log file is normally sufficient.

67

How can Bash check whether websites return HTTP 200?

A reliable HTTP 200 check in Bash sends an HTTP request to each URL with curl and inspects the returned status code. The script below handles timeouts, network errors, invalid URLs, optional redirects, and meaningful exit codes.

Read full answer

Applies toBash 5.x with curl 8.x on Linux

How It Works

  • The script reads one URL per line from a file and uses curl’s default GET method.

  • For each URL, curl resolves DNS, establishes a TCP or TLS connection, sends the request, and writes the status code using %{http_code}.

  • A URL passes only when the response code is exactly 200 and curl completed without a transport or protocol error.

  • Redirects are not followed by default. Set FOLLOW_REDIRECTS=1 to check the final destination after up to five redirects.

  • The script exits 0 only when at least one valid URL was checked and every request and validation check passed. It exits 1 for a failed check when at least one valid URL exists, 64 for usage or configuration errors, 65 when no valid URL was found, and 66 when the input file is unreadable.

Failure Modes and Production Behavior

  • Timeouts: --connect-timeout limits connection establishment, while --max-time limits the complete operation, including DNS, connection, TLS, and transfer.

  • HTTP errors: Responses such as 404, 500, or 503 are reported as failures even though curl successfully received them.

  • Network errors: DNS failures, refused connections, TLS errors, and timeouts produce a status such as 000 plus the curl exit code.

  • Error handling: set -euo pipefail stops on unhandled command failures, unset-variable errors, and failed pipeline components. The curl invocation is deliberately placed in an if condition so its exit status can be captured without terminating the script.

  • Monitoring: Exit status works for CI/CD, but production monitoring should also capture per-URL results, timestamps, duration, and the distinction between HTTP and network failures.

  • Load: Requests run sequentially, limiting accidental load. For large lists, use bounded concurrency rather than unbounded parallel requests.

Security and Operational Trade-offs

  • --proto '=http,https' prevents protocols such as file and ftp from being requested.

  • --disable prevents a user-level ~/.curlrc from silently changing curl behavior.

  • URLs containing embedded credentials are rejected to avoid leaking secrets into output.

  • An untrusted URL list can still target internal services or cloud metadata endpoints. Validate destinations against an allowlist and enforce network egress controls.

  • Avoid placing tokens or sensitive query parameters in URLs because successful and failed URLs are printed.

Code Example

Save this as check-http-200.sh:

#!/usr/bin/env bash
set -euo pipefail

CONNECT_TIMEOUT_SECONDS=5
MAX_TIME_SECONDS=15
FOLLOW_REDIRECTS=${FOLLOW_REDIRECTS:-0}

readonly CONNECT_TIMEOUT_SECONDS
readonly MAX_TIME_SECONDS
readonly FOLLOW_REDIRECTS

usage() {
  printf 'Usage: %s <url-list-file>\n' "${0##*/}" >&2
  exit 64
}

[[ $# -eq 1 ]] || usage
input_file=$1

if [[ ! -r $input_file ]]; then
  printf 'ERROR: cannot read %s\n' "$input_file" >&2
  exit 66
fi

curl_options=(
  --disable
  --silent
  --show-error
  --globoff
  --output /dev/null
  --write-out '%{http_code}'
  --connect-timeout "$CONNECT_TIMEOUT_SECONDS"
  --max-time "$MAX_TIME_SECONDS"
  --proto '=http,https'
)

if [[ $FOLLOW_REDIRECTS == 1 ]]; then
  curl_options+=(--location --max-redirs 5 --proto-redir '=http,https')
elif [[ $FOLLOW_REDIRECTS != 0 ]]; then
  printf 'ERROR: FOLLOW_REDIRECTS must be 0 or 1\n' >&2
  exit 64
fi

error_file=$(mktemp)
trap 'rm -f "$error_file"' EXIT

failures=0
checked=0
line_number=0

while IFS= read -r url || [[ -n $url ]]; do
  line_number=$((line_number + 1))

  if [[ -z $url || $url == \#* ]]; then
    continue
  fi

  if [[ $url =~ [[:space:]] ]] ||
     [[ ! $url =~ ^https?://([^/?#]+)([/?#].*)?$ ]]; then
    printf 'INVALID URL on line %d\n' "$line_number" >&2
    failures=$((failures + 1))
    continue
  fi

  authority=${BASH_REMATCH[1]}
  if [[ $authority == *@* ]]; then
    printf 'INVALID URL on line %d: embedded credentials are not allowed\n' \
      "$line_number" >&2
    failures=$((failures + 1))
    continue
  fi

  checked=$((checked + 1))
  : >"$error_file"

  if http_code=$(curl "${curl_options[@]}" --url "$url" 2>"$error_file"); then
    curl_exit=0
  else
    curl_exit=$?
  fi

  error_message=$(tr '\n' ' ' <"$error_file")

  if [[ $http_code == 200 && $curl_exit -eq 0 ]]; then
    printf 'PASS %s HTTP 200\n' "$url"
  else
    reason="HTTP ${http_code:-000}"
    if (( curl_exit != 0 )); then
      reason+="; curl exit ${curl_exit}: ${error_message:-request failed}"
    fi
    printf 'FAIL %s %s\n' "$url" "$reason"
    failures=$((failures + 1))
  fi
done <"$input_file"

if (( checked == 0 )); then
  printf 'ERROR: no valid URLs found\n' >&2
  exit 65
fi

if (( failures > 0 )); then
  exit 1
fi

Create sites.txt:

https://example.com
https://www.example.org

Run it with:

chmod +x check-http-200.sh
./check-http-200.sh sites.txt

Illustrative output:

PASS https://example.com HTTP 200
FAIL https://www.example.org HTTP 301

Best Practices

  • Run shellcheck check-http-200.sh before deployment.

  • Use FOLLOW_REDIRECTS=1 only when the final destination should determine success.

  • Add controlled retries for transient network failures, but avoid retrying permanent 4xx responses.

  • Keep timeout values conservative so one slow site cannot stall a monitoring run.

  • Upgrade curl and the operating system’s CA trust store regularly so TLS validation remains reliable.

  • Portability: This script intentionally uses Bash-only arrays, [[ ... ]], regular expressions, and BASH_REMATCH; run it with bash or its shebang, not with sh. A POSIX sh implementation must replace those constructs.

  • Text processing: Use grep to select matching records, sed for simple substitutions, and awk for field-based extraction or aggregation. Quote patterns and filenames, and prefer their portable POSIX options over GNU-specific behavior.

68

How do I rename all .txt files to .md in Bash?

A safe Bash rename script should quote filenames, detect destination conflicts before making changes, and avoid overwriting existing .md files. The script below processes regular files and symbolic links directly inside one directory without recursion.

Read full answer

Applies toBash 5.x on Linux with GNU coreutils 9.x

How It Works

  • Accepts one directory argument and changes into it with cd -P.

  • Uses nullglob so unmatched patterns do not produce literal filenames.

  • Uses dotglob so hidden entries such as .draft.txt are included.

  • Preflights every rename and aborts before changing anything if a corresponding .md file already exists.

  • Uses GNU mv -n -T to avoid overwriting a destination and prevent accidental placement inside a directory.

  • Handles spaces, tabs, and newlines in filenames through Bash arrays and quoted expansions.

Code Example

Save as rename_txt_to_md.sh:

#!/usr/bin/env bash
set -euo pipefail

usage() {
    printf 'Usage: %s DIRECTORY\n' "${0##*/}" >&2
}

if (( $# != 1 )); then
    usage
    exit 64
fi

directory=$1

if [[ ! -d "$directory" ]]; then
    printf 'Not a directory: %q\n' "$directory" >&2
    exit 66
fi

cd -P -- "$directory"

shopt -s nullglob dotglob
declare -a rename_pairs=()

# Precalculate all operations and reject conflicts before renaming.
for source in ./*.txt; do
    if [[ ! -f "$source" && ! -L "$source" ]]; then
        continue
    fi

    target="${source%.txt}.md"

    if [[ -e "$target" || -L "$target" ]]; then
        printf 'Refusing to overwrite: %q\n' "$target" >&2
        exit 1
    fi

    rename_pairs+=("$source" "$target")
done

if (( ${#rename_pairs[@]} == 0 )); then
    printf 'No .txt files found in %q\n' "$PWD"
    exit 0
fi

for ((i = 0; i < ${#rename_pairs[@]}; i += 2)); do
    source="${rename_pairs[$i]}"
    target="${rename_pairs[$i + 1]}"

    # Recheck immediately before the operation.
    if [[ -e "$target" || -L "$target" ]]; then
        printf 'Destination appeared before rename: %q\n' "$target" >&2
        exit 1
    fi

    if ! command mv -n -T -- "$source" "$target"; then
        printf 'Failed to rename %q to %q\n' "$source" "$target" >&2
        exit 1
    fi

    # GNU mv -n can report success when it skips an existing destination.
    if [[ -e "$source" || -L "$source" ]]; then
        printf 'Rename was not completed: %q -> %q\n' \
            "$source" "$target" >&2
        exit 1
    fi

    printf 'Renamed %q -> %q\n' "$source" "$target"
done

Run it with:

chmod +x rename_txt_to_md.sh
./rename_txt_to_md.sh /srv/reports

Example output:

Renamed report.txt -> report.md
Renamed notes\ draft.txt -> notes\ draft.md

Operational Behavior

  • The operation is non-recursive; files in subdirectories are unchanged.

  • Only the final lowercase .txt suffix is replaced. .TXT is not matched on case-sensitive Linux filesystems.

  • mv performs a same-filesystem rename, changing the directory entry rather than rewriting file contents. Permissions, ownership, timestamps, and inode data are normally preserved.

  • Exit status 0 indicates success; nonzero status indicates invalid arguments, an existing destination, or a failed rename.

  • Output goes to standard output, while errors go to standard error, making the script suitable for CI logs and operational tooling.

Failure Modes and Best Practices

  • Partial completion: Preflighting prevents destination conflicts, but an I/O or permission failure can still occur after earlier files have been renamed. For critical directories, maintain a manifest and use version control or backups.

  • Concurrent changes: Another process can create a destination between the preflight and rename. mv -n prevents overwriting it, while the post-operation check detects a skipped rename. A directory-level lock provides stronger isolation.

  • Privileged execution: Run with the least privileges required. Avoid running the script as root over directories writable by untrusted users.

  • Portability: Arrays, shopt, and nullglob are Bash features rather than POSIX sh features. GNU mv -T is also Linux-specific; BSD and macOS versions of mv do not support that option.

69

How do I recursively delete empty directories with Bash?

A Bash script can use GNU find to locate empty directories and remove them safely. The script should validate the target, protect the filesystem root, avoid descending into mounted filesystems, and quote every path to prevent word-splitting or glob-expansion errors.

Read full answer

Applies toBash 5.x with GNU findutils 4.x and GNU coreutils 8.32+ on Linux

How It Works

The script processes one target directory:

  1. set -euo pipefail makes the script stop on unexpected errors, unset variables, and failed pipeline components.

  2. realpath -e -- "$target" converts the argument into an absolute canonical path and resolves symbolic links.

  3. The filesystem root / is rejected to prevent an accidental recursive cleanup of the entire machine.

  4. find traverses the target with these important options:

  5. -mindepth 1: excludes the target directory itself.

  6. -depth: visits children before their parents, allowing a parent to become empty and be removed later.

  7. -type d: considers only directories.

  8. -empty: matches directories containing no entries, including hidden files.

  9. -xdev: prevents traversal into nested mounted filesystems.

  10. -delete: removes each matching directory; GNU find exits nonzero if deletion fails.

Code Example

#!/usr/bin/env bash
set -euo pipefail

if (( $# != 1 )); then
    printf 'Usage: %s DIRECTORY\n' "${0##*/}" >&2
    exit 64
fi

target="$1"

if [[ ! -d "$target" ]]; then
    printf 'Error: target is not an existing directory: %s\n' "$target" >&2
    exit 66
fi

# Resolve the target to an absolute path and resolve directory symlinks.
target="$(realpath -e -- "$target")"

# Never allow the filesystem root to be used as the cleanup target.
if [[ "$target" == "/" ]]; then
    printf 'Error: refusing to operate on the filesystem root\n' >&2
    exit 1
fi

# Delete matching descendants bottom-up without descending into nested mounts.
find "$target" \
    -xdev \
    -depth \
    -mindepth 1 \
    -type d \
    -empty \
    -delete

Save it as remove-empty-dirs.sh, make it executable, and run it against a specific path:

chmod +x remove-empty-dirs.sh
./remove-empty-dirs.sh /srv/app/data

A successful run normally produces no output. The script’s exit status indicates success or failure.

Dry Run and Operational Safety

Preview the directories that would be removed before executing the destructive operation:

find /srv/app/data \
    -xdev \
    -depth \
    -mindepth 1 \
    -type d \
    -empty \
    -print

Deletion is not transactional. GNU find can report an individual deletion error and continue processing other entries; because find is the final command, set -e makes the script exit with its nonzero status after find completes. In production:

  • Run the cleanup with an account that has only the required permissions.

  • Avoid running it as root unless that is explicitly necessary.

  • Take a snapshot or backup when the directory tree is operationally important.

  • Schedule the operation outside peak-traffic periods.

  • Capture stderr and the exit status in the automation or job-monitoring system.

  • Remember that deleting a child requires write and execute permission on its parent directory.

Failure Modes and Security

  • Permission denied: find reports the failure and exits nonzero.

  • Concurrent changes: A directory may stop being empty after find evaluates it; -delete then fails rather than deleting non-empty content.

  • Symbolic links: GNU find does not follow directory symlinks by default, reducing the risk of traversal outside the intended tree.

  • Mounted filesystems: -xdev prevents descending into nested mount points. The mount-point directory itself can still be tested, so -delete may fail there with a busy-device error.

  • Hidden content: Hidden files and directories count as content, so directories containing them are not considered empty.

  • Path safety: The canonical-path and root checks reduce accidental damage, but they do not replace backups, least privilege, or change-management controls.

Portability

This implementation targets Bash 5 and GNU tools. POSIX sh does not support set -o pipefail, and the script depends on GNU realpath -e --. BSD and macOS find do provide -empty and -delete, but the complete script is still not portable to those environments: macOS uses Bash 3.2 rather than Bash 5 and does not provide the same GNU toolchain.

70

How to write a Bash script that restarts an inactive systemd service?

A Bash script can use systemctl to check whether a service is active and restart it when necessary. For an inactive service, systemctl restart starts it; for a failed service, it clears the failed state and attempts a fresh start.

Read full answer

Applies toBash 5.x and systemd on Ubuntu 22.04/24.04 or RHEL 9

How It Works

The script performs four operations:

  1. Accepts an optional service unit, defaulting to nginx.service.

  2. Validates the unit name and confirms that systemd has loaded it.

  3. Uses systemctl is-active --quiet to check the current state.

  4. Restarts an inactive service and verifies that it becomes active.

set -Eeuo pipefail stops execution on undefined variables, unexpected failures, and failed pipeline components. Because systemctl is-active intentionally returns a nonzero status for an inactive service, it is used inside an if condition, where set -e does not terminate the script.

Code Example

#!/usr/bin/env bash
set -Eeuo pipefail

usage() {
  printf 'Usage: %s [service.unit]\n' "${0##*/}" >&2
}

if (( $# > 1 )); then
  usage
  exit 64
fi

SERVICE=${1:-nginx.service}
readonly SERVICE

# Permit conventional unit names and prevent option injection.
if [[ ! $SERVICE =~ ^[a-zA-Z0-9][a-zA-Z0-9_.@-]*$ ]]; then
  printf 'Invalid service name: %s\n' "$SERVICE" >&2
  usage
  exit 64
fi

load_state="$(
  systemctl show "$SERVICE" --property=LoadState --value 2>/dev/null
)" || {
  printf 'Unable to query systemd unit: %s\n' "$SERVICE" >&2
  exit 1
}

if [[ $load_state != "loaded" ]]; then
  printf 'Systemd unit is not loaded: %s\n' "$SERVICE" >&2
  exit 1
fi

if systemctl is-active --quiet "$SERVICE"; then
  printf '%s is active; no action needed.\n' "$SERVICE"
  exit 0
fi

printf '%s is not active; restarting...\n' "$SERVICE"

if ! systemctl restart "$SERVICE"; then
  printf 'Failed to restart %s.\n' "$SERVICE" >&2
  exit 1
fi

if systemctl is-active --quiet "$SERVICE"; then
  printf '%s restarted successfully.\n' "$SERVICE"
else
  printf '%s did not become active after restart.\n' "$SERVICE" >&2
  exit 1
fi

Example output:

$ sudo ./ensure-service-running.sh nginx.service
nginx.service is not active; restarting...
nginx.service restarted successfully.

Production Considerations

  • Permissions: The script normally needs root or an appropriate sudoers rule to restart a system service. Do not make the script setuid; use a controlled privileged runner instead.

  • Boot behavior: This script does not enable the service at boot. Use systemctl enable nginx.service when the service should start automatically after a reboot.

  • Health checks: is-active reports systemd’s process state, not application health. A service can be active while its HTTP endpoint returns errors. Use an application-specific health check when that distinction matters.

  • Existing restart policy: A unit configured with Restart=on-failure already asks systemd to restart failed processes. This script is mainly useful for boot reconciliation, detecting an externally stopped service, or implementing an external health check.

  • Monitoring: Record restart events and alert on repeated failures. Relevant logs are available through journalctl -u nginx.service.

Failure Modes and Security

  • A missing or masked unit is rejected before attempting a restart.

  • A restart can still fail because of invalid configuration, missing dependencies, resource exhaustion, or a failed ExecStart; the script returns a nonzero exit status in that case.

  • Repeated automatic restarts can create a restart loop and hide the underlying defect. Production automation should include backoff, thresholds, and alerting.

  • Service names are validated and variables are quoted to prevent word splitting and option injection. A privileged wrapper should still restrict callers to an allowlist of units they may restart.

  • systemctl restart can briefly interrupt traffic. It is not a substitute for graceful application draining, rolling deployment, or database-aware failover.

Bash and POSIX Compatibility

This script intentionally uses Bash features such as [[ ... =~ ... ]], arithmetic conditions, and set -o pipefail, so it is not portable to POSIX sh. A simplified version can use #!/bin/sh with set -eu, but Bash remains preferable when complex validation or consistent error handling is required. The dependency on systemctl also limits the script to Linux systems using systemd.

71

How can a Bash script sum a numeric column from a CSV file?

A production-ready approach is to use Bash for argument validation and orchestration while delegating CSV parsing to Python’s standard csv module. This handles quoted fields, embedded commas, escaped quotes, and newlines while retaining exact arithmetic for decimal inputs.

Read full answer

Applies toBash 5.x with Python 3.9+

How It Works

The script:

  • Accepts a CSV file and a 1-based column number.

  • Treats the first row as the header and ignores blank records.

  • Rejects inconsistent row widths, invalid decimal values, and common malformed CSV records.

  • Prints only the sum to standard output; errors and diagnostics go to standard error.

  • Uses set -euo pipefail, quoted expansions, a usage function, positional arguments, conditionals, and explicit exit codes: 64 for invalid usage, 127 for a missing dependency, and 1 for processing failures.

Python’s csv module is used because splitting a CSV line with IFS=, is unreliable: quoted values may contain commas, and quoted fields may span multiple lines.

Bash and POSIX Differences

The script relies on Bash-only features such as [[ ... ]], (( ... )), and set -o pipefail; it will not run unchanged under a POSIX /bin/sh. In POSIX sh, use [ ... ], case, and awk, and do not rely on pipefail.

A line-oriented Bash loop commonly uses while IFS= read -r line || [[ -n $line ]]; do with done < "$file". However, splitting such lines with IFS=, is not CSV parsing, and Bash arithmetic is integer-only, so it is unsuitable for decimal totals.

For a prevalidated CSV containing only simple, single-line numeric fields, POSIX awk can calculate a total:

awk -F, -v column=2 'NR == 1 { next } { total += $column } END { print total }' data.csv

This is simpler but uses floating-point arithmetic and does not correctly parse quoted CSV records. grep and sed are useful for line-oriented prevalidation or preprocessing, but they cannot safely parse multiline CSV fields.

Code Example

#!/usr/bin/env bash
set -euo pipefail

usage() {
  printf 'Usage: %s <csv-file> <column-number>\n' "$0" >&2
}

if (( $# != 2 )); then
  usage
  exit 64
fi

csv_file=$1
column_number=$2

if [[ ! -f "$csv_file" ]]; then
  printf 'error: CSV file not found: %s\n' "$csv_file" >&2
  exit 1
fi

if [[ ! "$column_number" =~ ^[1-9][0-9]*$ ]]; then
  printf 'error: column number must be a positive integer\n' >&2
  exit 64
fi

if ! command -v python3 >/dev/null 2>&1; then
  printf 'error: python3 is required\n' >&2
  exit 127
fi

python3 - "$csv_file" "$column_number" <<'PY'
import csv
import sys
from decimal import Decimal, InvalidOperation
from fractions import Fraction

path = sys.argv[1]
column_number = int(sys.argv[2])

try:
    # utf-8-sig removes an optional UTF-8 BOM; newline="" preserves CSV semantics.
    with open(path, "r", encoding="utf-8-sig", newline="") as csv_file:
        reader = csv.reader(csv_file, strict=True)

        try:
            header = next(reader)
        except StopIteration:
            raise ValueError("CSV file is empty")

        if column_number > len(header):
            raise ValueError(
                f"column {column_number} exceeds header width {len(header)}"
            )

        total = Fraction(0)
        values_read = 0

        for row_number, row in enumerate(reader, start=2):
            if not row:
                continue

            if len(row) != len(header):
                raise ValueError(
                    f"row {row_number}: expected {len(header)} fields, got {len(row)}"
                )

            raw_value = row[column_number - 1].strip()
            if not raw_value:
                continue

            try:
                # Decimal rejects rational forms such as 1/2 and preserves
                # the exact decimal value without applying context rounding.
                value = Fraction(Decimal(raw_value))
            except (InvalidOperation, OverflowError, ValueError, ZeroDivisionError):
                raise ValueError(
                    f"row {row_number}: invalid numeric value {raw_value!r}"
                )

            total += value
            values_read += 1

    # Finite decimal inputs have denominators containing only factors of 2
    # and 5, allowing the exact rational total to be rendered as a decimal.
    denominator = total.denominator
    power_of_two = 0
    power_of_five = 0

    while denominator % 2 == 0:
        denominator //= 2
        power_of_two += 1

    while denominator % 5 == 0:
        denominator //= 5
        power_of_five += 1

    if denominator != 1:
        raise ValueError("result cannot be represented as a finite decimal")

    scale = max(power_of_two, power_of_five)
    scaled_numerator = total.numerator * (2 ** (scale - power_of_two))
    scaled_numerator *= 5 ** (scale - power_of_five)

    sign = "-" if scaled_numerator < 0 else ""
    digits = str(abs(scaled_numerator)).zfill(scale + 1)

    if scale:
        result = f"{sign}{digits[:-scale]}.{digits[-scale:]}"
    else:
        result = f"{sign}{digits}"

    print(result)
    print(f"values summed: {values_read}", file=sys.stderr)

except (OSError, UnicodeError, csv.Error, ValueError) as error:
    print(f"error: {error}", file=sys.stderr)
    sys.exit(1)
PY

Example input:

region,amount
us-east,1250.50
us-west,749.25

Run it with:

chmod +x sum_csv_column.sh
./sum_csv_column.sh data.csv 2

Standard output is:

1999.75

The script also writes values summed: 2 to standard error.

Production Considerations

  • Failure behavior: The script fails closed on malformed data instead of silently producing an incorrect total. This is important for billing, capacity, and financial calculations.

  • Observability: Keep standard output machine-readable for command substitution. Record the exit status, runtime, input identity, and checksums in CI/CD or scheduler logs.

  • Large files: csv.reader streams records rather than loading the whole file. Only the running total and current record are retained.

  • Security: Inputs are quoted and no eval is used. Avoid logging sensitive CSV contents, and restrict permissions on files containing confidential data.

  • Maintenance: Store the script in version control, run ShellCheck, and test fixtures containing quoted commas, CRLF endings, blank cells, BOMs, and malformed rows. Pin and test the Python version used by CI before upgrading it.

  • Recovery: The script never modifies its input, so it needs no local backup. During incident analysis, preserve the original file and its checksum; any upstream data-retention or backup policy remains authoritative.

72

How to wait for a TCP port with Bash

A TCP readiness check repeatedly attempts a TCP connection until the handshake succeeds or a deadline expires. This script uses Bash’s /dev/tcp support and GNU timeout, avoiding a dependency on nc.

Read full answer

Applies toBash 5.x built with network-redirection support on Linux, with GNU coreutils 9.x

How It Works

  1. Validates the host, port, and timeout values.

  2. Attempts a TCP connection to HOST:PORT.

  3. Limits each connection attempt with timeout so an unreachable or filtered host cannot block indefinitely.

  4. Retries with a configurable delay until the total timeout expires.

  5. Exits with 0 when the port becomes reachable, 1 on timeout, and 2 for invalid arguments.

This checks whether a TCP listener is reachable. It does not mean the local port is free to bind, nor does it prove that the application behind the port is healthy.

Code Example

Save this as wait-for-port.sh:

#!/usr/bin/env bash
set -euo pipefail

usage() {
  printf 'Usage: %s HOST PORT [TOTAL_TIMEOUT] [CONNECT_TIMEOUT] [RETRY_DELAY]\n' \
    "$0" >&2
}

if (( $# < 2 || $# > 5 )); then
  usage
  exit 2
fi

host=$1
port=$2
total_timeout=${3:-60}
connect_timeout=${4:-1}
retry_delay=${5:-2}

if [[ -z "$host" ]]; then
  printf 'HOST must not be empty.\n' >&2
  exit 2
fi

if ! [[ "$port" =~ ^[0-9]+$ ]]; then
  printf 'PORT must be an integer.\n' >&2
  exit 2
fi

port=$((10#$port))
if (( port < 1 || port > 65535 )); then
  printf 'PORT must be between 1 and 65535.\n' >&2
  exit 2
fi

for value_name in total_timeout connect_timeout retry_delay; do
  value=${!value_name}

  if ! [[ "$value" =~ ^[0-9]+$ ]]; then
    printf '%s must be an integer.\n' "$value_name" >&2
    exit 2
  fi

  printf -v "$value_name" '%d' "$((10#$value))"
done

if (( total_timeout < 1 || connect_timeout < 1 || retry_delay < 1 )); then
  printf 'Timeouts and retry delay must be at least 1 second.\n' >&2
  exit 2
fi

printf 'Waiting up to %d seconds for TCP port %d...\n' \
  "$total_timeout" "$port"

start_time=$SECONDS

while :; do
  remaining=$((total_timeout - (SECONDS - start_time)))
  (( remaining > 0 )) || break

  attempt_timeout=$connect_timeout
  if (( attempt_timeout > remaining )); then
    attempt_timeout=$remaining
  fi

  # The child receives host and port as positional arguments.
  # Bash opens a TCP stream through /dev/tcp when the handshake succeeds.
  if timeout "${attempt_timeout}s" \
      bash -c "exec 3<>\"/dev/tcp/\$1/\$2\"" \
      _ "$host" "$port" 2>/dev/null; then
    if [[ "$host" == *:* ]]; then
      printf 'TCP port is reachable: [%s]:%s\n' "$host" "$port"
    else
      printf 'TCP port is reachable: %s:%s\n' "$host" "$port"
    fi
    exit 0
  fi

  remaining=$((total_timeout - (SECONDS - start_time)))
  (( remaining > 0 )) || break

  sleep_for=$retry_delay
  if (( sleep_for > remaining )); then
    sleep_for=$remaining
  fi
  sleep "$sleep_for"
done

printf 'Timed out after %d seconds waiting for TCP port %s:%s\n' \
  "$total_timeout" "$host" "$port" >&2
exit 1

Run it with:

chmod +x wait-for-port.sh
./wait-for-port.sh db.example.com 5432 60 1 2

Example output:

Waiting up to 60 seconds for TCP port 5432...
TCP port is reachable: db.example.com:5432

Under the Hood

  • /dev/tcp/HOST/PORT is a Bash-special redirection target rather than a normal device file. Bash performs a TCP connection and opens file descriptor 3 when the handshake succeeds.

  • The child closes the socket immediately after a successful check, so it sends no application data and does not keep the connection open. The peer can still accept and log the transient connection.

  • GNU timeout terminates an individual connection attempt that exceeds CONNECT_TIMEOUT.

  • The outer loop recalculates the remaining time, keeping retries bounded by TOTAL_TIMEOUT.

Production Considerations

Check What it establishes What remains unverified
TCP handshake A listener is reachable Protocol and application health
HTTP health check The endpoint returns an acceptable response Other application dependencies
Local ss check A local process is listening Remote reachability and service health

For an HTTP service, supplement the port check with an application-level request such as curl -fsS --max-time 2 https://example.com/health. For Kubernetes, prefer startupProbe and readinessProbe; for systemd, an ExecStartPre= wait script can prevent premature startup.

If the requirement is specifically that a local port can be bound, a connection check is the wrong test. Inspect local listeners with ss, or attempt the actual bind because any check-then-bind sequence has a race.

Failure Modes and Best Practices

  • Use finite deadlines: Infinite waits can leave CI jobs and deployments hanging indefinitely.

  • Distinguish readiness from liveness: A reachable port does not guarantee that workers, databases, or downstream dependencies are healthy.

  • Monitor timeouts: Emit metrics or alerts when the script exits with status 1; repeated timeouts indicate a startup or dependency incident.

  • Avoid secrets in arguments: Host and port are safe operational metadata, but credentials should not be exposed through process arguments or logs.

  • Validate untrusted destinations: Positional arguments are quoted and no eval is used, but an allowlist is still appropriate when the host comes from untrusted input.

  • Expect DNS changes: Each attempt resolves the hostname again, which is useful when service discovery records change.

  • Remember Bash versus POSIX sh: /dev/tcp is a Bash feature and is unavailable in portable POSIX sh.

74

How do you manually rotate a log file with Bash?

A manual log rotation script renames the active log file, creates a fresh file in its place, and preserves the original file for archival or compression. The script below uses mv, mktemp, strict Bash error handling, and same-directory temporary files so rotation does not cross filesystem boundaries.

Read full answer

Applies toBash 5 on Linux with GNU coreutils

How It Works

  1. The script accepts exactly one log-file path as an argument.

  2. It resolves the path with realpath, so symlinks are rotated at their target rather than replacing the symlink itself.

  3. It creates a temporary file in the same directory as the log.

  4. It copies the original file’s permissions and, when run as root, ownership.

  5. It renames the active log to a timestamped name such as app.log.20250308T120000Z.

  6. It renames the temporary file to the original log path.

Each mv is a same-filesystem rename and is therefore atomic individually. The complete rotation is not one atomic exchange, so the script includes recovery logic if replacing the original path fails.

Production Behavior

An application may continue writing to the rotated file if it keeps an open file descriptor. For example, after:

app.log -> app.log.20250308T120000Z

an existing process might still write to app.log.20250308T120000Z until it reopens the pathname. Conversely, a process that opens the log for every write will immediately use the new app.log.

The script deliberately does not send SIGHUP or another signal because signal behavior is application-specific. In production, coordinate rotation with the service’s documented reload mechanism, such as:

  • systemctl reload app.service

  • an application-specific SIGHUP handler

  • a controlled service restart during a maintenance window

Text Processing and Portability

  • Use grep -F for fixed-string searches and grep -E for extended regular expressions when investigating log events.

  • Use sed for line-oriented substitutions, not for rotating files.

  • Use awk for field-based analysis, such as extracting timestamps or status codes.

  • Avoid parsing arbitrary logs with shell read loops; records can contain spaces, tabs, control characters, or multiline entries.

This script requires Bash and GNU utilities. It is not POSIX sh compatible because it uses [[, set -o pipefail, realpath, mktemp, and GNU-specific mv -T and chmod --reference options. A POSIX version would need [ ] instead of [[ ]], no pipefail, and platform-specific temporary-file handling.

Code Example

#!/usr/bin/env bash
set -euo pipefail

usage() {
    printf 'Usage: %s LOG_FILE\n' "${0##*/}" >&2
    exit 2
}

if (( $# != 1 )); then
    usage
fi

log_file=$1

if [[ ! -f "$log_file" ]]; then
    printf 'Error: log file does not exist or is not a regular file: %s\n' \
        "$log_file" >&2
    exit 1
fi

# Resolve symlinks and convert the path to an absolute path.
log_file=$(realpath -e -- "$log_file")
log_dir=$(cd -- "$(dirname -- "$log_file")" && pwd -P)
log_name=${log_file##*/}

timestamp=$(date -u +%Y%m%dT%H%M%SZ)
rotated="${log_dir}/${log_name}.${timestamp}"

# Avoid overwriting an existing archived file.
if [[ -e "$rotated" ]]; then
    suffix=1
    while [[ -e "${rotated}.${suffix}" ]]; do
        suffix=$((suffix + 1))
    done
    rotated="${rotated}.${suffix}"
fi

# Create the replacement in the same directory and filesystem.
tmp_file=$(mktemp -- "${log_dir}/.${log_name}.XXXXXX")

cleanup() {
    if [[ -n "${tmp_file:-}" && -e "$tmp_file" ]]; then
        rm -f -- "$tmp_file"
    fi
}
trap cleanup EXIT

# Preserve metadata. Ownership can normally be changed only by root.
if [[ "$(id -u)" -eq 0 ]]; then
    chown --reference="$log_file" "$tmp_file"
fi
chmod --reference="$log_file" "$tmp_file"

# Rename the active log out of the way.
mv -T -- "$log_file" "$rotated"

# Install the new empty log at the original path.
if ! mv -T -- "$tmp_file" "$log_file"; then
    # Attempt to restore the original path if replacement failed.
    if ! mv -T -- "$rotated" "$log_file"; then
        printf 'Critical: failed to restore original log path: %s\n' \
            "$log_file" >&2
    fi
    exit 1
fi

tmp_file=
trap - EXIT

printf 'Rotated %s -> %s\n' "$log_file" "$rotated"

Example usage and output:

chmod 700 rotate-log.sh
./rotate-log.sh /var/log/myapp/app.log
Rotated /var/log/myapp/app.log -> /var/log/myapp/app.log.20250308T120000Z

Operational Best Practices

  • Run the script as the account that owns the log, or run it with narrowly scoped sudo permissions.

  • Ensure the log directory is writable but not writable by untrusted users; predictable archive names are only safe in a trusted directory.

  • Monitor filesystem usage. Rotating files without retention or compression can fill the disk.

  • Compress archived files only after the application has stopped writing to them.

  • Define retention, alerting, and archival procedures. For example, keep local rotations for seven days and ship logs to centralized storage for incident analysis.

  • Prefer the system’s logrotate integration or an application-native rotation mechanism when compression, copytruncate, post-rotate scripts, and retention policies are required.

  • Treat copytruncate as a compatibility fallback, not a universally safe alternative: it can lose records written between copying and truncating the original file.

75

How do you retry a command with exponential backoff in Bash?

A Bash retry function should run the command up to a configured number of attempts, increasing the delay exponentially between failures and returning the command’s final exit status. The implementation below uses exponential backoff while preserving the command’s arguments safely.

Read full answer

Applies toBash 5.x on Linux (Ubuntu 22.04/24.04, RHEL 9)

How It Works

  • The function receives the maximum attempts, initial delay, maximum delay, and the command plus its arguments.

  • The command is executed directly with "$@", so arguments are not subject to word splitting or shell glob expansion.

  • A successful command returns 0 immediately.

  • A failed command causes the function to sleep, double the delay, and try again.

  • The delay is capped at max_delay.

  • When all attempts are exhausted, the function returns the last command exit status.

Attempt Delay before the next attempt
1 1 second
2 2 seconds
3 4 seconds
4 8 seconds

Code Example

#!/usr/bin/env bash
set -euo pipefail

retry() {
    local max_attempts="$1"
    local initial_delay="$2"
    local max_delay="$3"
    shift 3

    if (( $# < 1 )); then
        printf 'Usage: retry MAX_ATTEMPTS INITIAL_DELAY MAX_DELAY COMMAND [ARG...]\n' >&2
        return 2
    fi

    if ! [[ "$max_attempts" =~ ^[1-9][0-9]*$ ]] ||
       ! [[ "$initial_delay" =~ ^[1-9][0-9]*$ ]] ||
       ! [[ "$max_delay" =~ ^[1-9][0-9]*$ ]]; then
        printf 'retry: attempts and delays must be positive integers\n' >&2
        return 2
    fi

    if (( max_delay < initial_delay )); then
        printf 'retry: maximum delay must be at least the initial delay\n' >&2
        return 2
    fi

    local attempt=1
    local delay="$initial_delay"
    local status=0

    while (( attempt <= max_attempts )); do
        if "$@"; then
            return 0
        else
            status=$?
        fi

        if (( attempt == max_attempts )); then
            return "$status"
        fi

        printf 'Command failed (attempt %d/%d, exit %d); retrying in %ds\n' \
            "$attempt" "$max_attempts" "$status" "$delay" >&2

        sleep "$delay"

        delay=$((delay * 2))
        if (( delay > max_delay )); then
            delay="$max_delay"
        fi

        attempt=$((attempt + 1))
    done
}

retry 4 1 8 /bin/false

Example output:

Command failed (attempt 1/4, exit 1); retrying in 1s
Command failed (attempt 2/4, exit 1); retrying in 2s
Command failed (attempt 3/4, exit 1); retrying in 4s

The final failure returns exit status 1; with set -e, a top-level call exits the script at that point.

Production Behavior

Use retries for failures that may be transient, such as a temporary network timeout, a restarting service, or a rate-limited API. Do not blindly retry permanent failures such as invalid credentials, malformed requests, or most HTTP 4xx responses.

A command that hangs indefinitely will never be retried by this function. On Linux, wrap it with a timeout when appropriate:

retry 3 2 10 timeout 30 curl --fail --silent --show-error https://example.com/

For pipelines involving grep, sed, or awk, invoke a fixed script through bash -c rather than concatenating untrusted input into a shell string. For example, avoid constructing commands such as "grep ... $user_input"; pass the input as a separately quoted argument instead.

Best Practices

  • Keep set -euo pipefail enabled, but do not rely on it alone. The function explicitly handles the command failure inside an if statement.

  • Use if retry ...; then when the caller needs to handle the final failure itself.

  • Add random jitter in distributed systems to prevent many clients retrying at the same time and creating a thundering herd.

  • Choose a finite attempt count and a capped delay so retries do not amplify an outage.

  • Log the attempt count, exit status, and delay, but never log secrets or authentication headers.

  • Make retried operations idempotent where possible. Retrying a non-idempotent write can create duplicate data.

  • If the command produces sensitive output, redirect or filter it separately; this function does not capture or redact command output.

  • On POSIX systems using sh rather than Bash, replace Bash features such as [[ ... ]], local, and arithmetic conditions with POSIX-compatible syntax.

76

How do I find the 10 largest files on a Linux system?

The most direct approach is to use GNU find to report each regular file’s apparent size and path, sort numerically by size in descending order, and print the first 10 results. The example below searches the root filesystem; remove -xdev only when all mounted filesystems should be included.

Read full answer

Applies toGNU findutils 4.x, GNU coreutils 9.x, GNU sed 4.x, and Bash 5.x on Linux

How the Command Works

  • find / -xdev -type f searches the root filesystem for regular files without descending into other mounted filesystems.

  • -printf '%s\t%p\n' prints the apparent size in bytes followed by a tab and the path.

  • sort -t $'\t' -k1,1nr -k2,2 sorts by size numerically in descending order, using ascending path order as the tie-breaker.

  • head -n 10 prints the 10 largest entries.

  • %s reports logical or apparent size, not allocated disk space. Consequently, a sparse file can appear large while occupying relatively little storage.

Code Example

#!/usr/bin/env bash

sudo find / -xdev -type f -printf '%s\t%p\n' 2>/dev/null \
  | sort -t $'\t' -k1,1nr -k2,2 \
  | head -n 10

Abbreviated example output:

1073741824	/var/lib/example/image-layer.tar
524288000	/var/log/archive/application.log
314572800	/var/lib/example/database-backup.dump

Filesystem and Permission Behavior

  • -xdev: Searches only the filesystem containing /. This excludes files on separate data volumes, as well as filesystems such as /proc, /sys, and remote mounts.

  • Other mounts: Remove -xdev to search every mounted filesystem. This can be slow on network storage and may include pseudo-filesystems whose reported sizes are not meaningful.

  • Selected volumes: For important data filesystems, run the same pipeline separately for each mount point and compare the resulting top-10 lists. The global top 10 must appear in one of those per-filesystem lists.

  • sudo: Allows the scan to traverse directories that an unprivileged user cannot read.

  • Error suppression: 2>/dev/null hides permission and transient filesystem errors. Remove it while investigating incomplete coverage.

Production and Portability Considerations

Running a broad find scan can create substantial disk, network, CPU, and inode activity. Schedule it during low-traffic periods, monitor its duration and exit status, and remember that GNU sort may use temporary disk space while sorting a large result set.

With set -euo pipefail, head can close the pipe early and cause sort to receive SIGPIPE, making the pipeline fail. GNU sed without an early-exit command consumes the complete input, so use it when pipe-failure status matters:

#!/usr/bin/env bash
set -euo pipefail

sudo find / -xdev -type f -printf '%s\t%p\n' \
  | sort -t $'\t' -k1,1nr -k2,2 \
  | sed -n '1,10p'

Do not suppress find errors in this fail-fast script: if find exits nonzero, pipefail causes the script to exit after displaying the diagnostic.

GNU find extensions such as -printf are not available in strictly POSIX find. Bash’s $'\t' is also non-POSIX; under POSIX sh, use sort -t "$(printf '\t')" instead. Filenames containing tabs or newlines make line-oriented output ambiguous and require NUL-delimited processing.

Before deleting anything, verify the file’s purpose, retention requirements, and whether a service still has it open. Investigate large deleted-but-open files with lsof +L1; because they have no discoverable directory entry, find cannot report them.

77

How do I replace a string in every file in a directory?

Use an in-place text replacement over a safely enumerated set of regular files. The command below treats the old and new values as literal strings and correctly handles filenames containing spaces or other shell metacharacters.

Read full answer

Applies toBash 5.x, GNU findutils 4.x, and Perl 5.32+ on Linux

How It Works

  • find enumerates the files while avoiding directory traversal and symbolic links.

  • -maxdepth 1 limits the operation to files directly inside the current directory.

  • -type f selects regular files, excluding directories, devices, sockets, and symbolic links.

  • -exec ... {} + executes Perl in batches and passes filenames safely rather than interpolating them into a shell command.

  • \Q...\E makes the old value literal, so characters such as ., *, [, and ? are not treated as regular-expression metacharacters.

  • Perl’s -i option performs the replacement in place, while -p processes the file line by line.

Code Example

#!/usr/bin/env bash
set -euo pipefail

# These temporary environment variables are inherited by find and Perl.
OLD_TEXT='old string' NEW_TEXT='new string' find . -maxdepth 1 -type f \
  -exec perl -pi -e 's/\Q$ENV{OLD_TEXT}\E/$ENV{NEW_TEXT}/g' -- {} +

Example before and after:

$ grep -nF -- 'old string' ./app.conf
./app.conf:3:old string

$ grep -nF -- 'new string' ./app.conf
./app.conf:3:new string

For a recursive replacement, remove -maxdepth 1:

find . -type f -exec perl -pi -e 's/\Q$ENV{OLD_TEXT}\E/$ENV{NEW_TEXT}/g' -- {} +

If the old value is intentionally a regular expression and GNU sed is available, the shorter equivalent is:

find . -maxdepth 1 -type f -exec sed -i 's/old_string/new_string/g' -- {} +

Operational and Failure Modes

  • Unintended scope: Removing -maxdepth 1 also changes files in every nested directory. Use an explicit root such as ./config to constrain the operation.

  • Non-text files: The command does not distinguish text from binary files. Replacing bytes in a binary file can corrupt it.

  • Rewrite behavior: Perl rewrites every selected file, even when it contains no match. File modification times and inode numbers may therefore change.

  • Line-based matching: Perl processes one line at a time, so a value spanning multiple lines will not match.

  • Empty old value: An empty OLD_TEXT is unsafe because it can match at every position. Validate that it is non-empty.

  • Partial failure: set -e causes the script to stop if find or Perl reports an error, but some earlier files may already have been modified.

  • No cross-file atomicity: Each file is replaced separately. A failure halfway through leaves earlier files updated and later files unchanged.

Best Practices

  • Back up first: Use version control, snapshots, or an explicit backup before modifying production configuration.

  • Review the diff: Run git diff --check and inspect git diff before committing or reloading services.

  • Validate configuration: Run the relevant syntax checker, such as nginx -t, before reloading a service.

  • Handle symlinks deliberately: The shown command skips symbolic links, preventing accidental modification of files outside the directory.

  • Prefer staging: Test the replacement against representative files before applying it to production.

  • Check portability: set -o pipefail is a Bash feature, not part of POSIX sh. The shown find syntax also relies on GNU findutils behavior.

  • Coordinate writers: Avoid editing files while another process is writing them. For critical live configuration, deploy versioned files and switch to them atomically instead.

78

How can large-file shell scripts be optimized for speed?

Bash optimization means reducing unnecessary process creation, moving repetitive work into efficient tools such as awk, and parallelizing independent work without exhausting host resources. The largest gains usually come from profiling first, avoiding per-line shell loops, and tuning the script for the actual I/O and CPU bottleneck.

Read full answer

Applies toBash 5.x with GNU coreutils 9.x and GNU awk 5.x on Linux

How to Find the Bottleneck

Do not optimize blindly. Test with a representative file and measure where time is spent:

/usr/bin/time -v ./process-logs.sh access.log
strace -c ./process-logs.sh access.log

Look for these symptoms:

  • High system time: The script creates many processes or performs excessive reads and writes.

  • High user time: Expensive operations such as regular expressions run repeatedly.

  • High I/O wait: The workload is disk-bound, so CPU optimization will have limited effect.

  • Slow startup: The script launches many small commands even when the file is small.

Also use bash -x for short traces when necessary, but avoid it in production because expanded arguments and secrets may appear in logs.

Key Optimizations

Technique Benefit Trade-off
Use builtins where practical Avoids fork and exec overhead Builtins are less expressive than external tools
Replace shell line loops with awk Moves repeated processing into a compiled, streaming tool Field positions depend on the input format
Combine commands into one awk program Avoids intermediate pipelines and repeated file scans Can become difficult to maintain
Use grep -F for fixed strings Avoids regular-expression processing Only suitable when no pattern syntax is needed
Batch work with xargs -P Uses multiple CPU cores Can overload disks, memory, or downstream services
Cache reusable values Avoids repeated calculations or lookups Stale cached data can cause correctness problems
Use streaming processing Keeps memory usage bounded More complex aggregation may require memory

Bash builtins such as printf, read, test, [[, and parameter expansion are faster than equivalent external commands because they run inside the shell. However, a Bash loop that invokes an external command for every line is still expensive.

For Unicode-heavy data, avoid forcing LC_ALL=C; it can speed up byte-oriented matching but changes locale-sensitive matching and collation behavior. Reserve it for explicitly ASCII data where that behavior is acceptable.

Code Example

The following script counts HTTP failures in one streaming pass. It avoids a Bash while read loop and does not load the file into memory:

#!/usr/bin/env bash
set -euo pipefail

if (( $# != 1 )); then
    printf 'Usage: %s FILE\n' "$0" >&2
    exit 64
fi

input=$1
export LC_ALL=C

awk '
{
    status = $9
    if (status ~ /^[45][0-9][0-9]$/) {
        failures++
        counts[status]++
    }
}
END {
    printf "total_failed=%d\n", failures + 0

    for (i = 400; i < 600; i++) {
        status = sprintf("%d", i)
        if (status in counts) {
            printf "%s=%d\n", status, counts[status]
        }
    }
}
' -- "$input"

Example output:

total_failed=1253
404=1100
500=153

awk reads the input incrementally, so memory usage depends mainly on the number of distinct matching statuses rather than the file size. The script assumes the ninth whitespace-separated field is the HTTP status, which is common for standard combined access logs but must be validated for the actual format.

Unlike many Bash features, this awk structure is broadly portable to POSIX awk. Features such as arrays, [[ ... ]], and (( ... )) are Bash-specific and are not available in POSIX sh.

Parallel Processing

Use parallelism only when records can be processed independently:

  • GNU xargs -P: Suitable for many small files or command batches. Its concurrency may exceed the available CPU count, so set a conservative limit.

  • File sharding: Split a large file into independent chunks, process them concurrently, and merge the results.

  • Producer-consumer pipelines: Commands in a normal shell pipeline already run concurrently, but this usually overlaps I/O rather than parallelizing one sequential stage.

  • Aggregation strategy: Combine per-chunk counts before loading all results into one associative array.

Parallelism is often ineffective for a single disk-bound file because concurrent readers may compete for the same I/O budget. It is more useful for many independent files, network requests, or CPU-heavy transformations.

Production Practices

  • Validate inputs: Check file existence, readability, expected format, and output directory before processing large datasets.

  • Fail explicitly: Use set -euo pipefail, but also validate commands and inputs because set -e has contextual behavior and is not a substitute for explicit error handling.

  • Write atomically: Create output with mktemp, write it completely, validate it, and then rename it into place.

  • Protect temporary data: Set a restrictive umask, avoid predictable temporary filenames, and clean them up with a trap.

  • Check completion: Compare record counts or checksums when the source provides an expected total.

  • Support incremental runs: Store checkpoints or processed offsets so a failed multi-hour job does not restart from the beginning.

  • Monitor throughput: Record processed bytes, records per second, elapsed time, and estimated completion time during long operations.

  • Test before deployment: Run shellcheck, test with malformed and empty input, and benchmark using production-sized data.

  • Plan upgrades: Pin critical tool versions during rollout, compare outputs before and after upgrades, and retain rollback procedures.

The safest optimization strategy is to measure the current script, replace the dominant bottleneck, rerun the same benchmark, and verify that output remains identical.

79

How do you unit test Bash scripts with Bats?

Bats is a TAP-based test runner for Bash that executes each test in an isolated subshell. Tests follow the arrange-act-assert pattern: prepare inputs, invoke the script with run, and assert its exit status and output.

Read full answer

Applies toBash 5.x, bats-core 1.10+, and bats-assert 2.x on Ubuntu 24.04 (Ubuntu 22.04 ships bats 1.2, which lacks `BATS_TEST_TMPDIR` and `--separate-stderr`)

How Bats Works

Bats discovers files ending in .bats and converts each @test block into a separate test case. Each test receives a clean process, preventing exported variables or filesystem changes from leaking into another test.

  • setup: Runs before every test; use it to create temporary files or initialize state.

  • teardown: Runs after every test; use it for cleanup.

  • run: Executes a command without immediately terminating the test, capturing $status, $output, and $lines.

  • bats-assert: Provides assertions such as assert_success, assert_failure, and assert_output.

  • BATS_TEST_DIRNAME: The directory containing the current test file.

  • BATS_TEST_TMPDIR: A temporary directory isolated for the current test.

By default, run combines standard output and standard error. Newer bats-core versions can separate them with run --separate-stderr.

Test Design

A focused Bash unit test should exercise one decision or side effect:

  • Arrange: Create a controlled input, temporary directory, or command stub.

  • Act: Run the production script or function.

  • Assert: Check the exit code, stdout, stderr, created files, or mocked API calls.

For infrastructure scripts, mock dependencies such as aws, kubectl, terraform, or gcloud. Place stub executables in $BATS_TEST_TMPDIR/bin and prepend that directory to PATH. This prevents unit tests from modifying real cloud resources.

Do not use set -e in the Bats test file. Bats relies on explicit assertions to report failures, while production scripts should use set -euo pipefail and deliberate exit codes.

Code Example

Install the test dependencies on Ubuntu:

sudo apt-get update
sudo apt-get install -y bats bats-support bats-assert

Create validate_release.sh:

#!/usr/bin/env bash
set -euo pipefail

readonly RELEASE_PATTERN='^[a-z0-9][a-z0-9._-]{0,63}$'

validate_release() {
    if (( $# != 1 )); then
        printf 'usage: %s RELEASE\n' "${0##*/}" >&2
        return 2
    fi

    local release="$1"
    if [[ ! $release =~ $RELEASE_PATTERN ]]; then
        # Escape the value so newlines cannot forge additional log records.
        printf 'invalid release name: %q\n' "$release" >&2
        return 1
    fi

    printf 'release=%s\n' "$release"
}

main() {
    validate_release "$@"
}

# Allow the functions to be sourced without running the CLI.
if [[ ${BASH_SOURCE[0]} == "$0" ]]; then
    main "$@"
fi

Create tests/validate_release.bats:

#!/usr/bin/env bash
# shellcheck shell=bash

# Ubuntu's bats-support and bats-assert packages install under /usr/lib/bats.
load '/usr/lib/bats/bats-support/load'
load '/usr/lib/bats/bats-assert/load'

setup() {
    RELEASE_SCRIPT="$BATS_TEST_DIRNAME/../validate_release.sh"
}

@test "accepts a valid release name" {
    run bash "$RELEASE_SCRIPT" "web-2.1"

    assert_success
    assert_output "release=web-2.1"
}

@test "rejects an invalid release name" {
    run bash "$RELEASE_SCRIPT" "Web_2"

    assert_failure 1
    assert_output --partial "invalid release name"
}

@test "returns a usage error when the argument is missing" {
    run bash "$RELEASE_SCRIPT"

    assert_failure 2
    assert_output --partial "usage:"
}

Running the Tests

Run the complete suite from the repository root:

shellcheck validate_release.sh
bats tests/validate_release.bats

Successful TAP-style output is similar to:

1..3
ok 1 accepts a valid release name
ok 2 rejects an invalid release name
ok 3 returns a usage error when the argument is missing

Bats also supports filtering tests by name and selecting a formatter:

bats --filter "invalid release" tests/validate_release.bats
bats --formatter tap tests/

Production and Tooling Choices

Tool Primary use Key consideration
bats-core Testing Bash behavior Best fit for Bash-specific features
bats-assert Assertions for Bats Cleaner assertions and failure messages
Shellspec Testing several POSIX-compatible shells Useful when sh compatibility matters
shunit2 Bourne-shell unit testing Lightweight, but less feature-rich than Bats

In CI, pin the Bats and assertion-library versions to keep results reproducible. Run unit tests on every pull request, then run separate integration or staging tests against version-pinned infrastructure providers. Unit tests should detect control-flow regressions quickly; integration tests detect credential, API, image, and provider drift that mocks cannot reproduce.

Common Pitfalls

  • Testing live infrastructure: Mock external commands so tests remain deterministic and safe.

  • Overusing set -e: Use explicit run and assertions in Bats; use set -euo pipefail in production scripts.

  • Losing failure output: Inspect $status, $output, and $lines when an assertion fails.

  • Leaking state: Create fresh files under $BATS_TEST_TMPDIR for every test.

  • Exposing secrets: Never place real credentials in fixtures, TAP logs, or command arguments.

  • Assuming validation proves trust: A valid release name does not prove artifact authenticity; production deployments still need signature or digest verification.

Sign in to keep reading

You've opened 2 full answers. Sign in to read all 79 Shell Scripting answers, mark what you know and pick up where you left off.

Report a mistake

What's wrong?

All Cloud, DevOps & Infrastructure subjects