What is a shell? How do Bash, sh, zsh, and fish differ?
KnownRevise
A shell is a command interpreter that reads commands, expands variables and patterns, starts programs, manages input/output, and returns exit codes. Bash is a widely used GNU shell for Linux automation, while sh, zsh, and fish differ in compatibility, features, and intended use.
Read full answerHide full answer
Applies toGNU Bash 5.2, POSIX sh (implementation-dependent), Z shell 5.9, fish 4.x
How a Shell Works
When you run a command, the shell generally:
Reads a command line or script.
Performs expansions, including variable, command, arithmetic, and pathname expansion.
Splits the result into words when appropriate.
Applies redirections and pipelines.
Executes a shell function, builtin, or external program.
Waits for completion and exposes an exit status.
External commands usually run in a child process. For example, grep is not part of the shell; the shell starts grep and waits for it. Builtins such as cd, printf, export, and exit are executed by the shell itself, which is why cd changes the current shell’s directory but cannot change the directory of its parent process.
A successful command normally returns 0; a failure returns a nonzero status. These statuses are essential in CI/CD jobs, deployment scripts, health checks, and orchestration systems.
Shell Comparison
Shell
Main characteristics
Typical use
sh
POSIX-oriented, small, portable interface; the actual implementation varies
System administration scripts where POSIX portability matters
Bash
GNU shell with arrays, [[ ]], brace expansion, process substitution, advanced job control, and pipefail
Linux automation, CI/CD, deployment scripts, and interactive shells
zsh
Powerful interactive shell with extensive completion, themes, history, recursive globbing, and many configurable options
Interactive development and customization
fish
User-friendly shell with syntax emphasizing variables, pipelines, and interactive features; intentionally not Bash-compatible
Interactive use and fish-specific automation
sh is not necessarily an older version of Bash. On Ubuntu, /bin/sh commonly points to dash; on some other distributions it may point to Bash, BusyBox ash, or another implementation. A script that works under one sh may fail under another if it depends on implementation-specific behavior.
Bash is largely Bourne-compatible, but it is not a strict POSIX shell. zsh offers excellent interactive features but is not a drop-in Bash replacement. fish has a deliberately different syntax, such as set -x NAME value instead of export NAME=value, and does not support Bash’s set -euo pipefail convention.
Bash Scripting Capabilities
Bash provides the features commonly needed in production automation:
Variables: name=value, expansion with $name or ${name}, and environment variables with export.
Quoting: "$name" preserves one argument; "${array[@]}" safely expands every array element.
Conditionals: if, case, [[ ... ]], and command exit-status tests.
Loops: for, while, and until, including reading files with while IFS= read -r.
Functions: Reusable blocks with positional parameters such as $1, $2, and $#.
Arguments: "$@" passes each argument separately; "$*" joins them into one string.
Error handling: set -Eeuo pipefail enables stricter error behavior, unset-variable detection, and pipeline failure detection.
Text processing: grep selects matching lines, sed transforms text, and awk performs field-based or aggregate processing.
Be careful with set -e: it does not make every possible failure fatal. For example, a command used as an if condition is intentionally allowed to return nonzero. Use explicit checks for operations where partial failure is unacceptable.
Production and Portability Pitfalls
Quote every expansion: Prefer "$variable" and "${array[@]}"; unquoted values can cause word splitting and glob expansion.
Avoid eval: It is difficult to audit and can enable command injection.
Use mktemp, trap, and atomic renames when updating temporary files or deploying artifacts, so interruptions do not leave partial state.
Make scripts idempotent: Retrying a deployment should not duplicate resources or corrupt existing state.
Log deliberately: Send meaningful progress and failure information to standard error, and avoid exposing secrets.
Monitor failures: A nonzero exit status should fail a CI job or alert an orchestrator; silent failures are more dangerous than crashes.
Test under the target interpreter: A Bash script may work on a developer laptop but fail on a minimal container where /bin/sh is dash.
Prefer portable tools for broad deployment: POSIX utilities and a /bin/sh script are safer across heterogeneous systems; Bash is preferable when arrays, robust conditionals, and pipefail are required.
2
What is a shebang line and why use #!/usr/bin/env bash?
KnownRevise
A shebang line is the first line of an executable script that tells a Unix-like operating system which interpreter to run. #!/usr/bin/env bash delegates the search for bash to /usr/bin/env, which uses the caller’s PATH; this helps when Bash is installed outside a fixed location, but it also makes interpreter selection depend on PATH.
Read full answerHide full answer
Applies toall Bash versions on Unix-like systems; exact shebang execution semantics are platform-specific
What a Shebang Line Does
The shebang, also called a hashbang, must begin at the first byte of the file for direct execution to recognize it.
Its conventional form is #! followed by an interpreter pathname and, on some systems, an optional argument.
When a script is executed directly, the operating system starts the selected interpreter with the script pathname. Once Bash starts, it treats the shebang as a comment.
The shebang is not required when explicitly invoking an interpreter, such as bash script.sh; it matters when running ./script.sh.
Without a shebang, a direct execve normally fails with ENOEXEC. An invoking shell may then interpret the file as a shell script, but that fallback is shell behavior rather than kernel behavior.
Direct execution also requires suitable filesystem permissions, normally including an execute bit for the script.
How #!/usr/bin/env bash Works
On Linux, the kernel recognizes #! and starts the named interpreter. Linux supports one optional interpreter argument.
With #!/usr/bin/env bash, the kernel starts /usr/bin/env with bash and the script pathname.
env then searches PATH for an executable named bash and executes it with the script pathname.
If /usr/bin/env does not exist, or no executable bash is found in PATH, execution fails before the script body starts.
Other Unix-like systems have broadly similar behavior, but details such as supported arguments and error handling can differ.
Why #!/usr/bin/env bash Is Often Used
Interpreter discovery: Bash may be installed at /bin/bash, /usr/bin/bash, or another location, so searching PATH can avoid assuming one fixed path.
Version selection: In development environments, PATH can deliberately select a preferred Bash version.
Environment consistency: The same script can work across systems with different installation layouts, provided /usr/bin/env exists and the required interpreter is available through PATH.
Limitations: The form still hard-codes /usr/bin/env, and an unsafe or unexpected PATH can select the wrong executable. For example, stock macOS provides Bash 3.2 unless a newer version is installed separately.
Hard-Coded Path vs. env Lookup
Aspect
#!/bin/bash
#!/usr/bin/env bash
Interpreter discovery
Uses exactly /bin/bash
Uses /usr/bin/env, then searches PATH for bash
Determinism
Higher when the target path is standardized
Depends on the invocation environment
Security
Does not use PATH to select Bash
Vulnerable to manipulated PATH
Portability
Requires Bash at that path
Requires /usr/bin/env and Bash on PATH
Startup
One interpreter startup
Adds the env lookup and execution step
Best fit
Controlled production hosts with a known package-managed path
Development or scripts running across known environments
Production Best Practices and Portability
Prefer determinism on managed hosts: Install Bash through the platform package manager and use its known absolute path when the execution environment is controlled.
Control PATH before launch: If using env, sanitize or set PATH in the launcher before executing the script. Changing PATH inside the script is too late to affect interpreter selection.
Use POSIX sh when required: For broad Unix portability, write POSIX-compliant code and use #!/bin/sh; Bash syntax such as arrays, [[ ... ]], and process substitution will not be portable.
Preserve file metadata: Keep Unix line endings, the shebang on line 1, and the required execute permission.
Test on target platforms: Run the script in CI for every supported OS and architecture, and use bash -n plus ShellCheck for Bash scripts.
Manage upgrades: When upgrading Bash or changing the standard system path, retest scripts and automation that rely on interpreter-specific behavior.
3
How do you make a Bash script executable and run it?
KnownRevise
Set the script’s executable permission with chmod, then run it using its path, such as ./script.sh. The file must also contain a valid shebang so the operating system knows which interpreter should execute it.
Read full answerHide full answer
Applies toBash 5.x on Linux (Ubuntu 22.04/24.04, RHEL 9)
Permissions and Commands
Create or edit the script with a shebang as its first line:
#!/usr/bin/env bash
Make it executable and run it:
chmod 755 deploy.sh
./deploy.sh production
Possible output:
Deploying to production
Common permission modes are:
Mode
Meaning
Typical use
700
Owner can read, write, and execute; group and others have no access
Private administrative script
750
Owner has full access; group can read and execute; others have no access
Team-managed operational script
755
Owner has full access; group and others can read and execute
Public or generally accessible script
744
Owner has full access; group and others can read only, so only the owner (or root) can run it directly
Script readable by the team but runnable only by its owner
To grant only the owner execute permission, use:
chmod u+x deploy.sh
Verify the permissions with:
ls -l deploy.sh
An executable script normally appears like this:
-rwxr-xr-x 1 sre sre 2048 Mar 10 09:30 deploy.sh
The permission triplet after the file-type character is owner/group/other: the third character of the owner triplet (x here) is the owner’s execute bit.
How Execution Works
The shell executes ./deploy.sh and verifies that the file has an applicable execute bit.
The kernel reads the #!/usr/bin/env bash shebang and starts /usr/bin/env with the argument bash.
Because that argument is not an absolute path, env searches PATH for bash.
Bash opens the script and starts executing its commands.
A path containing /, such as ./deploy.sh, is resolved directly. A command without a slash, such as deploy.sh, is searched through PATH:
export PATH="$HOME/bin:$PATH"
mkdir -p "$HOME/bin"
mv deploy.sh "$HOME/bin/deploy.sh"
deploy.sh production
If the script is not executable, it can still be run explicitly by the interpreter:
bash deploy.sh production
This bypasses the execute-bit requirement, although the script must still be readable.
Code Example
#!/usr/bin/env bash
set -euo pipefail
if (( $# != 1 )); then
printf 'Usage: %s <environment>\n' "$(basename "$0")" >&2
exit 64
fi
readonly environment="$1"
printf 'Deploying to %s\n' "$environment"
Make and run it:
chmod 755 deploy.sh
./deploy.sh production
Check it with ShellCheck before deployment:
shellcheck deploy.sh
The script uses:
Quoted expansions: "$1" and "$(basename "$0")" prevent word splitting and pathname expansion.
Strict error handling: set -euo pipefail stops on unexpected command failures, rejects unset variables, and detects failures in pipeline components.
Script arguments: "$1" is the first argument, "$@" represents all arguments, and "$#" is the argument count.
Production and Portability Considerations
Use /usr/bin/env bash for portability across Bash installation locations, but ensure PATH is trusted. For privileged automation, a fixed interpreter path such as #!/bin/bash can reduce PATH-manipulation risk.
Bash features such as [[ ... ]], arrays, (( ... )), and local are not portable to POSIX sh. Use #!/bin/sh and POSIX syntax when the script must run on minimal systems.
Do not rely on set -e as a complete error-handling strategy. Check critical operations explicitly, propagate meaningful exit codes, and use trap for cleanup when appropriate.
Keep deployment scripts idempotent so retries do not create duplicate resources. For production automation, add locking, validation, dry-run support, structured logging, and rollback procedures.
Avoid embedding credentials in scripts. Prefer a secret manager, protected environment file, or workload identity mechanism.
Preserve the executable bit through source control by committing the file’s executable mode; otherwise, configure the deployment pipeline to apply chmod explicitly.
4
How are variables declared and accessed in Bash?
KnownRevise
In Bash, variables are named, case-sensitive storage slots for strings: assign them with name=value and access them as $name or ${name}. They are untyped, should normally be expanded inside double quotes, and are passed to child processes only when exported or supplied as a command-prefix assignment.
Read full answerHide full answer
Applies toBash 5.x on Linux
Declaring and Assigning Variables
Plain assignment creates a shell variable; no separate declaration is required. Do not put spaces around the first =.
declare -r name=value: create a read-only variable.
declare -x name=value: create an exported variable.
declare -a items=("api" "worker"): create an indexed array.
declare -A settings=([region]="eu-west-1"): create an associative array.
Variable names begin with a letter or underscore and contain only letters, digits, and underscores. Values are strings, although arithmetic contexts evaluate them as numeric expressions.
Accessing Variables and Scope
Reading: Prefix a name with $; use ${name} when adjacent text could obscure the variable boundary.
Global and local scope: Ordinary assignments are global. Use local name=value inside a function. Bash uses dynamic scoping: a callee can read and modify a caller’s local variable unless it declares its own local variable with that name.
Child processes: export NAME=value marks a variable for the environment of subsequently executed commands. NAME=value command passes it only to that command without exporting it in the current shell.
Arguments: $1 is the first positional argument, $# is their count, and "$@" expands each argument as a separate quoted word.
Defaults, Error Handling, and Control Flow
Parameter expansions provide defaults and required-value checks:
Expansion
Behaviour
${name:-default}
Use default when name is unset or empty
${name-default}
Use default only when name is unset
${name:=default}
Assign default when name is unset or empty
${name:?message}
Report an error when name is unset or empty
set -euo pipefail is a useful automation baseline:
-e requests that the shell exit when a command, pipeline, or list returns non-zero, subject to Bash’s documented exceptions.
-u treats expansion of an unset variable as an error.
pipefail makes a pipeline return non-zero when any component fails.
Do not rely on set -e alone; explicitly check expected failures. Bash conditionals include if/elif/else and case, while loops include for, while, and until. A function can use return N, and a script can use exit N; $? contains the previous command’s status and should be captured immediately.
Text Processing and Portability
Use field- and pattern-aware tools rather than fragile shell splitting:
grep -q '^api_' "$file" tests for matching lines; grep returns 0 for a match, 1 for no match, and 2 for an error.
sed -E 's/^DEBUG=true$/DEBUG=false/' "$file" performs a stream transformation.
awk -F: '$1 == "environment" { print $2 }' "$file" extracts a field from structured records.
Bash-specific features include arrays, [[ ... ]], local, process substitution, and set -o pipefail. A POSIX sh script should use #!/bin/sh, [ ... ], and set -eu, but cannot rely on Bash arrays, local, or pipefail.
Code Example
#!/usr/bin/env bash
set -euo pipefail
# Fail early when required arguments are missing.
if (( $# < 2 )); then
printf 'Usage: %s <release> <config-file>...\n' "${0##*/}" >&2
exit 64
fi
readonly release="$1"
shift
readonly app_name="${APP_NAME:-payments-api}"
readonly backup_root="${BACKUP_ROOT:-/var/backups/payments}"
# Restrict permissions for newly created backups.
umask 077
timestamp="$(date -u +%Y%m%dT%H%M%SZ)"
readonly timestamp
readonly backup_dir="${backup_root}/${app_name}/${timestamp}"
mkdir -p -- "$backup_dir"
backup_one() {
local source_file="$1"
local destination="$2"
if [[ ! -f "$source_file" || ! -r "$source_file" ]]; then
printf 'Config is not a readable file: %s\n' "$source_file" >&2
return 1
fi
cp -- "$source_file" "$destination"
}
# Copy each input into the timestamped backup directory.
index=0
for config_file in "$@"; do
index=$((index + 1))
destination="${backup_dir}/${index}-${config_file##*/}"
backup_one "$config_file" "$destination"
done
printf 'Backed up %d config file(s) for %s release %s to %s\n' \
"$index" "$app_name" "$release" "$backup_dir"
Backed up 1 config file(s) for checkout-api release v2025.03.1 to /tmp/payments-backups/checkout-api/20250308T101530Z
Best Practices and Common Pitfalls
Always quote expansions such as "$variable" and "${array[@]}".
Use printf instead of echo for predictable formatting and safe handling of user-controlled values.
Validate arguments, paths, URLs, and numeric values before using them in production operations.
Avoid eval; it can execute attacker-controlled input and obscure failure handling.
Do not store secrets directly in scripts or ordinary environment variables. Use a secret manager and inject credentials at runtime.
Use read-only variables for values that must not change, but do not treat read-only as a security boundary.
In production, add retention, alerting, disk-capacity checks, cleanup, concurrency protection, and recovery testing. A timestamped backup directory alone does not guarantee retention, consistency, or recoverability.
5
Bash quoting: single vs double vs unquoted values?
KnownRevise
In Bash, quoting controls whether special characters are expanded and whether whitespace, wildcard characters, and command substitutions are treated as literal text or shell syntax. Single quotes produce literal values, double quotes permit expansions while preventing word splitting and pathname expansion, and unquoted values can be split and globbed.
Read full answerHide full answer
Applies toBash 5.x
Core Difference
Form
Expands variables?
Preserves spaces?
Allows globbing?
Typical use
'text'
No
Yes
No
Literal strings
"text"
Yes
Yes
No
Variables and command substitutions
text
Yes, when applicable
No
Yes
Deliberate shell syntax or splitting
Single quotes: Everything between the quotes is literal, including $, backticks, *, and spaces. A literal single quote can be embedded with '\''.
Double quotes: Parameter, command, and arithmetic expansions are evaluated, but word splitting and pathname expansion are not performed on the resulting text. The special parameter "$@" is an exception: it expands to separate positional arguments.
Unquoted text: Expansion results may be split into multiple arguments using IFS and may be interpreted as glob patterns.
Ordinary words: An unquoted literal such as production behaves like 'production'; differences appear when the value contains whitespace, expansions, or special characters.
Expansion and Word Splitting
Bash performs expansions before executing a command. Unquoted expansion results can undergo word splitting and then pathname expansion, while quoted results remain single fields. The exception is "$@", which deliberately preserves the positional-parameter boundaries.
For example, with name="Ada Lovelace", printf '<%s>\n' $name passes Ada and Lovelace as separate arguments, producing <Ada> and <Lovelace>. Using printf '<%s>\n' "$name" passes one argument and produces <Ada Lovelace>.
Inside double quotes, backslash is special only when followed by $, a backtick, ", another backslash, or a newline. Characters such as * remain literal, preventing accidental filename matching.
Security and Production Risks
Unquoted values can cause argument-count, whitespace, and glob-related failures in commands such as kubectl, aws, or deployment scripts. Quoting preserves boundaries, but it does not stop an argument beginning with - from being interpreted as an option. Validate option-like values and use -- before user-controlled arguments when the command supports it.
Command substitution used as an argument should also be quoted: tool "$(printf 'healthy')" passes the output as one argument. In an assignment such as message=$(printf 'healthy'), Bash does not split the command-substitution result. However, quoting the assignment does not preserve it later; expanding "$message" is what prevents splitting at the point of use. Command substitution also removes trailing newline characters.
Never use eval or construct a command string from untrusted input such as HTTP responses, CI variables, cloud metadata, or user-provided parameters. These values can inject shell syntax. Prefer direct argument passing, arrays, explicit validation, and allowlists.
Best Practices and Portability
Quote every variable expansion: use "$variable", not $variable.
Use single quotes for literals that should not be expanded.
Use "${array[@]}" to pass every array element as a separate, preserved argument.
Use "$@" to forward positional arguments safely.
Use [[ ... ]] for Bash-specific conditionals. For an optional variable under set -u, use [[ -z "${value:-}" ]]; require mandatory variables earlier with an explicit validation or ${variable:?message} check.
Use set -euo pipefail, while remembering that set -e does not report every failed command used in a conditional context.
Run ShellCheck in CI to detect missing quotes and portability problems.
POSIX sh supports single and double quotes but lacks Bash arrays, [[ ... ]], and $'...' quoting. POSIX also does not require set -o pipefail, and common /bin/sh implementations such as dash do not support it. Keep deployment scripts POSIX-compatible when they must run under /bin/sh.
6
Why should variables be quoted in Bash?
KnownRevise
Quoting variables in Bash preserves argument boundaries and prevents unintended word splitting and pathname expansion. Unquoted expansions can create extra arguments, disappear when empty, or unexpectedly expand into filenames.
Read full answerHide full answer
Applies toBash 5.x; the core behavior also applies to POSIX-compliant shells
How Bash Processes Variables
During command processing, Bash recognizes quoted regions before performing expansions. For an unquoted variable expansion, the resulting text can be:
Split into multiple words according to the current IFS.
Subjected to pathname expansion, so wildcard characters can select matching files.
Removed entirely when the variable is set to an empty string and the expansion is the entire word.
A double-quoted expansion remains one word. For example, a value of alpha beta passed as printf '<%s>\n' "$value" is one argument containing a space, whereas printf '<%s>\n' $value normally produces two arguments.
Quoting also prevents expanded content from becoming shell syntax. If an ordinary variable contains a semicolon, redirection operator, or similar character, that character remains data rather than being reinterpreted as shell syntax. Command substitutions written directly inside double quotes are still executed, however, because those substitutions are part of the command before variable expansion occurs.
Common Failure Modes
Extra arguments: A path containing spaces becomes several arguments if expanded without quotes.
Unexpected globbing: A value containing *.log can expand into every matching file in the current directory.
Missing empty values: "$empty" preserves one empty argument when empty is set but empty; $empty normally removes the word.
Incorrect loops: Iterating over an unquoted space-separated string can split complete items into fragments.
Option interpretation: Quoting keeps a value as one argument, but it does not stop a value beginning with - from being interpreted as an option or operator. Commands differ in how they handle such arguments.
IFS-dependent splitting: Unquoted values are split only on characters currently present in IFS. By default, these are space, tab, and newline, not arbitrary Unicode whitespace.
set -euo pipefail does not disable word splitting or pathname expansion. set -u detects references to unset variables, but it does not make unquoted expansions safe.
Choosing the Correct Quote
Form
Behavior
Typical use
"$variable"
Expands the value while preserving one argument
Default choice for strings
'$variable'
Treats the text literally, including $variable
Literal strings and regex fragments
"${array[@]}"
Preserves each array element as a separate argument
Iterating over Bash arrays
"${array[*]}"
Joins elements using the first character of IFS
Deliberate serialization
"$@"
Preserves each positional parameter as a separate argument
Forwarding script arguments
Double quotes permit parameter expansion, command substitution, and specific escape sequences. Single quotes suppress these forms of interpretation. Bash array forms are not portable to POSIX sh.
Security and Operational Considerations
Quoting is an argument-handling control, not a complete security boundary. Do not pass untrusted input to eval, bash -c, or another shell interpreter because those interfaces parse their input as shell code.
When a command supports an end-of-options marker such as --, use it before user-controlled arguments when appropriate. Also validate expected values and prefer APIs that accept structured data instead of constructing shell command strings. Quoting prevents expansion accidents; it does not replace validation, authorization, or safe command design.
For production automation, use quoting consistently rather than relying on assumptions about current directory, IFS, locale, or the values supplied by an upstream job. ShellCheck and tests containing spaces, empty strings, glob characters, and leading hyphens can catch regressions before deployment.
Bash Versus POSIX sh
The basic "$variable" quoting rules are the same in POSIX sh. Bash-specific features such as arrays, [[ ... ]], process substitution, and local are not specified by POSIX and should not be used in portable scripts. Likewise, set -o pipefail is a Bash extension rather than a POSIX option.
A portable script should use POSIX parameter expansion, positional parameters, and loop constructs. The script shebang determines which shell interprets it, so #!/bin/sh should not contain Bash-only behavior.
Best Practices
Default to double quotes for string expansions: "$variable".
Quote command substitutions: "$(command)".
Use "$@" when forwarding positional arguments.
Use "${array[@]}" when iterating over Bash arrays.
Quote values passed to commands such as printf, test, and external utilities.
Use read -r when input should preserve literal backslashes.
Remember that Bash assignments do not undergo word splitting or pathname expansion, but quote the right-hand side for clarity and quote the stored value wherever it is later expanded.
Run ShellCheck in CI and treat its findings as errors.
7
How do environment variables and export work in Bash?
KnownRevise
An environment variable is a string entry in a process’s startup environment, while export gives a Bash shell variable the export attribute so that Bash includes it in the environment of child processes. Each child receives its own environment copy, so changes made by a child do not modify its parent’s variables.
Read full answerHide full answer
Applies toGNU Bash 5.x on Linux
Environment Variables and Shell Variables
Bash distinguishes variables by how they are used:
A shell variable is a named value stored by the current Bash shell. It is visible to shell functions and subshells but is not automatically supplied to newly started external commands.
An exported variable is a shell variable with the export attribute. Bash includes it in the environment constructed for child processes.
When Bash starts, it imports inherited environment variables as shell variables and retains their export attributes.
Common environment variables include PATH, HOME, LANG, TZ, and application-specific values such as DATABASE_URL. Environment values are strings; Bash does not preserve numeric or Boolean types for child processes. Arrays are shell features and cannot be exported as arrays to external programs.
How export Works
export NAME marks an existing shell variable for export without changing its value. export NAME=value both assigns a value and marks it for export. Exporting does not start a process or send the value anywhere by itself; it affects commands Bash starts afterward.
Related operations include:
export -p: Prints variables that have the export attribute.
export -n NAME: Removes the export attribute without deleting the shell variable.
unset NAME: Removes the shell variable and therefore removes it from the environment inherited by future children.
readonly NAME: Prevents later modification of a variable but does not export it; export and readonly are separate attributes.
VAR=value command: For an external command, supplies VAR in that command’s environment without permanently changing the current shell variable. Functions and special built-ins have different assignment-persistence semantics.
env VAR=value command: Starts an external command with the current environment plus the supplied assignment.
Inheritance and Process Boundaries
When Bash starts an external program, it constructs the program’s environment and passes it during process creation. The child receives a copy, so modifying or unsetting an environment variable in the child does not affect the parent.
This inheritance continues through scripts and descendant processes. For example, if Bash exports DEPLOY_ENV=staging, a Bash script started by that shell imports DEPLOY_ENV, and commands started by the script inherit it unless the script deliberately changes or removes the export attribute.
Changing an exported variable in the parent affects only commands started afterward; it cannot alter the environment of an already-running process.
Production and Security Considerations
Validate configuration early: Use an explicit check or ${VAR:?message} so scripts fail clearly when required configuration is missing or empty.
Quote expansions: Write "$VAR" to prevent word splitting and pathname expansion.
Export only necessary values: Every child inherits exported variables, increasing exposure through diagnostics, logs, env output, or same-user process inspection.
Avoid exporting secrets by default: Prefer workload identity or restricted secret delivery where practical. Redact tracing and diagnostic output.
Avoid accidental bulk export: set -a automatically exports subsequently assigned variables and can expose temporary or internal values.
Use readonly carefully: It helps protect validated configuration from accidental changes but is not a substitute for validation.
Handle failures explicitly: Production Bash scripts commonly use set -euo pipefail. set -e has intentional exceptions, including commands in if conditions and non-final commands in && or || lists, so critical operations should still be checked explicitly.
Protect logs: set -x can print expanded values, including credentials, so it should not be enabled around secrets.
Portability and Common Pitfalls
export is supported by POSIX shells, but export -n, Bash arrays, [[ ... ]], and set -o pipefail are Bash-specific or otherwise not portable. For example, /bin/sh on Ubuntu commonly points to dash, which does not support pipefail.
Exporting a variable does not make it global across unrelated shells, preserve its original type, or encrypt its value. It only controls whether Bash includes that variable in the environment of commands it starts.
8
What are Bash positional parameters like `$1`, `$@`, and `$#`?
KnownRevise
Bash positional parameters are the arguments supplied to a script or function. $1 through $9 represent individual arguments, $# contains their count, and "$@" expands all arguments while preserving their boundaries.
Read full answerHide full answer
Applies toBash 5.x on Linux and Unix systems
How They Work
When Bash starts a script, it assigns the script’s command-line arguments to positional parameters. For example, invoking ./deploy.sh production api-service --verbose provides:
$0: the script name used for invocation, such as ./deploy.sh.
$1: production.
$2: api-service.
$3: --verbose.
$#: 3, the number of positional parameters.
Positional parameters above nine require braces: ${10} refers to the tenth argument, while $10 would be interpreted as $1 followed by the character 0.
Each function has its own positional parameters. Bash restores the caller’s parameters when the function returns. The complete positional-parameter set can also be replaced with set --, but syntax such as 1=value is invalid.
Important Parameter Expansions
Expression
Meaning
$0
Name used to invoke the script; inside a function, it normally remains the script or shell name
$1–$9
Individual positional arguments
${10}, ${11}
Positional arguments with two or more digits
$#
Number of positional parameters
"$@"
All arguments, each preserved as a separate word
"$*"
All arguments joined into one word using the first character of IFS
$?
Exit status of the previous command; not a positional parameter
$$
Process ID of the current shell; not a positional parameter
$!
Process ID of the most recently started background job; not a positional parameter
The distinction between "$@" and "$*" matters when arguments contain spaces. Given arguments such as web server and api server, "$@" preserves them as two arguments, while "$*" combines them into the single string web server api server. Unquoted $@ and $* are unsafe for user-supplied values because they undergo word splitting and pathname expansion.
Functions and Argument Forwarding
The shift command removes parameters from the beginning of the positional-parameter list and renumbers those that remain. A plain shift removes one argument, while shift 2 removes two. After either operation, $# and "$@" reflect the updated list. The argument count must be checked before shifting to avoid an error.
When forwarding arguments to another command or function, use "$@". This preserves each argument as a distinct value and avoids word splitting, pathname expansion, and accidental execution of shell metacharacters.
Production Practices and Common Pitfalls
Quote expansions: Use "$@", "$*", "$1", and "$#" where appropriate. Quoting prevents values containing spaces or wildcard characters from being altered.
Validate argument counts: Check $# before accessing required arguments. Under set -u, expanding an unset positional parameter can terminate the script.
Use shift carefully: Confirm that enough arguments exist before shifting, and account for the updated numbering afterward.
Prefer "$@" for forwarding: It preserves argument boundaries and safely passes empty or space-containing values.
Use getopts for short options: Bash’s getopts is suitable for options such as -v or -n value. Long options generally require a separate case parser.
Avoid eval: Never use eval to process arguments because attacker-controlled input can become executable shell code.
Plan for portability: "$@", $#, and shift are supported by POSIX shells, but arrays, brace expansion such as {1..5}, [[ ... ]], and process substitution are Bash-specific.
9
How do positional parameters $@ and $* differ in Bash?
KnownRevise
In Bash, quoted expansion is the decisive distinction: "$@" preserves each positional parameter as a separate argument, while "$*" joins the positional parameters into one argument. This distinction matters when values contain whitespace, empty strings, or wildcard characters.
Read full answerHide full answer
Applies toBash 5.x, including Bash 5.2
Expansion Rules
Positional parameters are the arguments supplied to a script or function.
Expression
Expansion behavior
"$@"
Produces one field per positional parameter and preserves their exact boundaries. With no positional parameters, it produces zero fields.
"$*"
Joins the positional parameters into one field using the first character of IFS. With no positional parameters, it produces one empty field.
$@
Starts with separate positional parameters, but because the expansion is unquoted, word splitting and pathname expansion may alter their boundaries and contents.
$*
Joins the positional parameters first, then word splitting and pathname expansion may alter the resulting text.
The quoted forms "$@" and "$*" are not subject to word splitting or pathname expansion.
IFS and Empty Arguments
The default IFS begins with a space, so "$*" normally uses a space as its separator. If IFS is changed, its first character becomes the separator; for example, an IFS beginning with a colon makes "$*" join arguments with colons. Setting IFS to the empty string removes the separator.
These behaviors must be distinguished from an explicitly supplied empty argument. If a script receives one empty-string argument, "$@" produces one empty field and "$*" also produces one empty field.
Changing IFS affects other parsing operations in the shell, so production scripts should avoid changing it globally merely to format one message.
Forwarding Arguments
Use "$@" when forwarding arguments to a function or local command because each value remains a distinct argument. Use "$*" only when flattening the values into one human-readable string is intentional.
Avoid reconstructing shell command strings from arguments or passing them through eval; that can introduce quoting errors, word-splitting vulnerabilities, and command-injection risks.
SSH requires an additional caveat: ssh concatenates the remote command arguments into a string, and the remote shell parses that string again. Consequently, "$@" does not guarantee preservation of remote argument boundaries. Untrusted values should not be embedded in a remote shell command without a transport or API designed for structured arguments.
Production Guidance
Prefer "$@" for forwarding; reserve "$*" for deliberate joining.
Do not rely on unquoted $@ or $*, especially in automation involving spaces, tabs, newlines, or glob characters.
set -euo pipefail does not make unsafe expansions safe; retain explicit status checks for failures that are expected or recoverable.
Under set -u, "$@" and "$*" remain safe when no positional parameters exist, while directly expanding an unset parameter such as $1 fails.
Run ShellCheck in CI to detect quoting defects and unused assignments.
Avoid exposing secrets directly in command-line arguments where possible, because arguments may be visible in process listings or operational logs.
10
What do Bash special parameters like $?, $$, and $! mean?
KnownRevise
Special parameters are built-in Bash variables that expose process, argument, and execution-status information. They are expanded by the shell, so their values often change depending on the most recently executed command or background job.
Read full answerHide full answer
Applies toBash 5.x on Linux
Core Special Parameters
Parameter
Meaning
$?
Exit status of the previously executed command: 0 generally means success; a non-zero value means failure.
$$
Process ID of the current shell. In a Bash subshell, $$ normally retains the parent shell’s PID.
$!
Process ID of the most recently started background job. It may be empty if no background job exists.
$0
Name of the script as invoked, such as ./deploy.sh, or the shell name when running interactively.
$1–$9
Positional arguments. Use ${10} for arguments numbered ten or higher.
$#
Number of positional arguments passed to the script or function.
$@
All positional arguments, preserving individual argument boundaries. Prefer "$@".
$*
All positional arguments joined using the first character of IFS. Use "$*" when one combined string is intended.
$-
Options currently active in the shell.
$_
Last argument of the previous command, with some context-dependent behavior; avoid using it for critical logic.
Bash also provides ${PIPESTATUS[@]}, containing the individual exit statuses of commands in the most recently executed pipeline. This differs from $?, which contains the pipeline’s overall status.
Status Handling and Error Handling
The shell updates $? after a command completes. Capture it before running another command; for example, after false, $? expands to 1, and immediately assigning that value to a variable preserves it.
Within an if condition, the else branch can capture the failed command’s status before performing other work. This is preferable to checking $? after an intervening command because that command may overwrite it.
set -euo pipefail provides common production safeguards:
-e: Exit when a command fails, except where failure is explicitly handled in contexts such as if, while, &&, or ||.
-u: Treat references to unset variables as errors. Use ${variable:-default} for optional values.
-o pipefail: Return failure when any component of a pipeline fails.
set -e is not a substitute for explicit error handling. Important operations should use if checks, functions should return meaningful statuses, and cleanup should normally be registered with trap.
Functions have their own positional parameters. Calling function_name "$@" preserves argument boundaries, while return "$status" communicates failure to the caller.
Production Practices
Quote expansions: Use "$variable", "$@", and "$1". Unquoted values can undergo word splitting and pathname expansion.
Handle statuses explicitly: grep returns 0 for a match, 1 for no match, and 2 for an error; these cases should be treated differently.
Wait for jobs: Use wait "$background_pid" rather than assuming a background operation completed.
Do not use $$ for locking: Multiple subshells can share it. Use flock or an atomic filesystem operation instead.
Prefer BASHPID inside subshells: $BASHPID identifies the current Bash process, whereas $$ may identify its parent shell.
Use ${BASH_SOURCE[0]} when sourcing: $0 can still refer to the calling script.
Process files safely: Prefer commands such as grep -F -- "$needle" "$file" for literal matching and always place user-controlled data in variables rather than constructing shell code.
Bash and POSIX Compatibility
$?, $$, $!, $0, $#, $@, $*, and ordinary positional arguments are defined by POSIX and are broadly supported by /bin/sh. Features not specified by POSIX—including ${PIPESTATUS[@]}, ${BASH_SOURCE[0]}, arrays, [[ ... ]], pipefail, and local—are not guaranteed across all /bin/sh implementations.
For POSIX scripts, use set -eu, avoid Bash-only syntax, and check pipeline components individually when pipeline failure matters.
11
How do exit status codes work in Bash scripts?
KnownRevise
In Bash, every command produces an integer exit status: 0 conventionally indicates success, while a non-zero value indicates failure or an error. A script normally returns the status of its last command unless an explicit exit statement specifies the result.
Read full answerHide full answer
Applies toBash 5.0–5.3 on Linux and POSIX sh where noted
Exit Status Fundamentals
The special parameter $? contains the exit status of the most recently completed foreground command:
Read $? immediately, because running another command replaces it.
Common conventions include:
Situation
Typical status
Successful command
0
General failure
1
Incorrect command-line usage
64
Command not found
127
Command found but not executable
126
Terminated by SIGINT
130
Terminated by SIGTERM
143
Terminated by SIGKILL
137
These values are conventions rather than universal guarantees. For signals, Bash normally reports 128 + signal_number, unless a signal trap handles the event.
How Exit Statuses Propagate
Script completion: Without an explicit exit, the script returns the status of its last command.
exit n: Terminates the script with status n.
exit without an argument: Uses the status of the previous command.
return n: Returns status n from a function. Without an argument, it returns the previous command’s status.
Pipeline: Returns the status of the last command by default. With set -o pipefail, it returns the rightmost non-zero status, or zero if every command succeeds.
Negation: ! command reverses the status, so it returns zero when command fails.
Background process: $! contains the process ID, not its exit status. Use wait "$pid" to obtain the status.
Subshell: A subshell has its own environment, but exits with the status of its last command or an explicit exit.
For example, false | true normally returns 0, while the same pipeline with pipefail returns 1.
Error Handling with set -euo pipefail
The common Bash preamble is:
set -euo pipefail
-e: Exits when an unhandled command returns non-zero.
-u: Treats references to unset variables as errors.
pipefail: Makes pipelines fail when any component fails.
However, set -e has important exceptions. Bash does not exit for a failing command used as:
The condition of if, while, or until
Part of an && or || list, except the final command
A command prefixed by !
A function called from an if condition also runs with -e effectively suppressed throughout its body. Therefore, explicitly checking important commands is safer than depending only on set -e.
Code Example
#!/usr/bin/env bash
set -euo pipefail
usage() {
printf 'Usage: %s <url>\n' "$0" >&2
exit 64
}
check_url() {
local url=$1
local output=$2
local status=0
if curl --fail --silent --show-error --max-time 10 \
"$url" >"$output"; then
return 0
else
status=$?
fi
return "$status"
}
main() {
if (( $# != 1 )); then
usage
fi
local url=$1
local output
output=$(mktemp)
# Remove the temporary file on normal or explicit-exit paths.
trap 'rm -f -- "$output"' EXIT
if check_url "$url" "$output"; then
printf 'Health check succeeded.\n'
exit 0
else
local status=$?
printf 'Health check failed (exit %d).\n' "$status" >&2
exit "$status"
fi
}
main "$@"
The script captures the failure status in an else block because using if ! command discards the original status. For an HTTP 500 response, for example, curl --fail normally returns 22, which the script propagates.
Production Best Practices and Portability
Use explicit exits: End scripts with exit 0 on success and meaningful non-zero statuses on failure.
Handle expected failures: Check network calls, file operations, and validation commands instead of relying exclusively on set -e.
Clean up resources: Use an EXIT trap for temporary files, but do not rely on it for SIGKILL, which cannot be trapped.
Handle termination signals: In long-running services, trap SIGTERM and SIGINT to stop gracefully and report the final status.
Make failures observable: CI systems and service managers use the exit status to decide whether a deployment, job, or service succeeded.
Avoid ambiguous statuses: Reserve documented codes for categories such as configuration errors, dependency failures, and timeouts.
Remember portability: Exit statuses and $? are POSIX concepts, but pipefail is not required by POSIX sh. Scripts requiring Bash should retain #!/usr/bin/env bash rather than using #!/bin/sh.
12
How do you write if, elif, and else conditionals in Bash?
KnownRevise
Conditional statements in Bash execute command lists and branch on their exit status: zero indicates success, while a non-zero status indicates failure. An if chain can contain optional elif branches and an optional else catch-all, and every chain must end with fi.
Read full answerHide full answer
Applies toBash 5.x; POSIX `sh` portability notes
How It Works
if: Executes its body when the preceding command list returns status zero.
then and fi: then introduces the first body and fi closes the chain. then may follow the condition on the same line or the next line.
elif: Tests another condition only if the previous conditions failed.
else: Runs when no preceding condition succeeded.
&& and ||: Chain commands based on status; they are not substitutes for a multi-branch conditional.
The Three Test Contexts
Context
Typical use
Important behavior
POSIX sh
[ "$x" = "y" ]
Portable string and file tests
Quote expansions to preserve argument boundaries
Supported
[[ "$x" =~ regex ]]
Bash patterns, regular expressions and arrays
No word splitting or pathname expansion; leave the regex right-hand side unquoted for regex matching
Bash-only
(( count > 10 ))
Integer arithmetic
Variable names normally omit $
Bash-only
[ ] versus [[ ]]: [ is a command, so its arguments must be quoted where necessary. [[ ]] is a Bash keyword with safer expansion rules and additionally supports =~, internal &&/||, and array-element tests.
Common Tests
String: [ -z "$s" ] for empty and [ -n "$s" ] for non-empty.
Equality: [ "$a" = "$b" ] or [[ "$a" == "$b" ]].
Numeric: [ "$n" -gt 10 ] or (( n > 10 )); non-numeric input produces an error.
Glob: [[ "$filename" == *.log ]]; use case for broader multi-way dispatch.
Failure Modes and Pitfalls
Unquoted expansions: [ $value = "text" ] can change the argument count. An empty expansion can produce a misleading result, while a multi-word expansion can cause “too many arguments.”
Regular expressions: In Bash 5.x, [[ $text =~ ^[0-9]+$ ]] treats the right-hand side as a regular expression. Quoting part of that expression makes the quoted text match literally.
Invalid integers: Numeric tests such as [ "$x" -gt 5 ] return an error for non-numeric input; validate the value before comparing it.
Missing fi: This causes a parse error, so the script does not execute.
set -e behavior: The if and elif condition lists are exempt because their statuses are tested. Branch-body commands are normally checked, but commands in tested &&, ||, !, if, or while contexts are also exempt.
Code Example
This complete Bash 5 script demonstrates if, elif, and else while validating unsafe input:
#!/usr/bin/env bash
# Validate deployment arguments and report service health.
set -euo pipefail
usage() {
printf 'Usage: %s <service-name> <replica-count>\n' "${0##*/}" >&2
exit 64
}
[[ $# -eq 2 ]] || usage
service=$1
replicas=$2
# Restrict the name so it cannot contain path separators or newlines.
if [[ ! $service =~ ^[A-Za-z0-9][A-Za-z0-9._-]*$ ]]; then
printf 'Error: invalid service name: %s\n' "$service" >&2
exit 64
fi
if [[ ! $replicas =~ ^[0-9]+$ ]]; then
printf 'Error: replica-count must be a non-negative integer\n' >&2
exit 64
fi
# Force base 10 so values such as 08 are not interpreted as octal.
replicas=$((10#$replicas))
if (( replicas < 1 )); then
status="CRITICAL"
elif (( replicas < 3 )); then
status="DEGRADED"
elif (( replicas < 6 )); then
status="WARNING"
else
status="HEALTHY"
fi
printf 'service=%s replicas=%s status=%s\n' \
"$service" "$replicas" "$status"
config_dir="${CONFIG_DIR:-/etc}"
config="$config_dir/$service.conf"
if [ -f "$config" ]; then
printf 'config found: %s\n' "$config"
elif [ -d "$config_dir" ]; then
printf 'no config for %s, but %s exists\n' "$service" "$config_dir"
else
printf 'configuration directory does not exist: %s\n' "$config_dir" >&2
exit 69
fi
With api 2 as the arguments and no /etc/api.conf, the output is:
service=api replicas=2 status=DEGRADED
no config for api, but /etc exists
Best Practices for Production Scripts
Start Bash scripts with #!/usr/bin/env bash and normally use set -euo pipefail; pipefail is not POSIX.
Quote expansions where word splitting or pathname expansion can occur. Bash’s [[ ]] and arithmetic contexts have different, safer rules.
Prefer [[ ]] in Bash and [ ] in portable POSIX sh scripts.
Use case for multi-way dispatch rather than a long elif chain.
Treat exit codes as an API, send diagnostics to stderr, and keep machine-readable output on stdout.
For production automation, add explicit timeouts, retries with backoff, audit logging, dry-run support, monitoring and alerting rather than relying solely on set -e.
13
What do [ ], [[ ]], and (( )) do differently in Bash?
KnownRevise
In Bash, [ ... ] is the test command, [[ ... ]] is a conditional expression, and (( ... )) evaluates an arithmetic expression. They differ in supported operators, expansion behavior, exit-status semantics, and portability.
Read full answerHide full answer
Applies toBash 5.x; all POSIX-compliant `sh` implementations for `[ ... ]` only
Core Differences
Construct
Purpose
Example
Portable to POSIX sh
[ ... ]
Tests strings, integers, and filesystem properties
[ -f "$file" ]
Yes
[[ ... ]]
Bash conditional expression with pattern and regular-expression matching
[[ "$name" == api-* ]]
No
(( ... ))
Evaluates an arithmetic expression
(( count > 0 ))
No
[ is a command name. Bash provides it as a builtin, while POSIX systems may also provide it as /usr/bin/[. Its arguments undergo normal shell expansion, so variable expansions should be quoted.
[[ and ]] are Bash reserved words handled directly by the parser. Words inside them do not undergo normal word splitting or pathname expansion.
(( and )) use Bash’s arithmetic evaluator. Arithmetic variables are referenced by name rather than with the usual $ prefix.
Supported Operators
String tests with [ ]: use -z, -n, =, and !=.
String tests with [[ ]]: use -z, -n, ==, !=, and pattern matching.
Regular expressions with [[ ]]: use =~, which matches against a POSIX extended regular expression.
Integer tests with [ ]: use -eq, -ne, -lt, -le, -gt, and -ge.
Arithmetic with (( )): use C-style operators such as +, %, &&, ||, and comparisons.
Unlike [ "$name" = api-* ], the right side of [[ "$name" == api-* ]] is interpreted as a glob pattern rather than a literal string.
Expansion and Security
Normal command expansion applies to [ ... ]. For example, [ $status = ready ] is unsafe when status is empty or contains spaces. Quote every expansion, as in [ "$status" = "ready" ].
Inside [[ ... ]], splitting and glob expansion do not occur, although quoting remains useful for clarity. Leave the right-hand regular expression unquoted so its metacharacters remain active, as in [[ "$email" =~ ^[a-z]+@[a-z]+\.example\.com$ ]].
Bash recursively evaluates arithmetic variable values as arithmetic expressions. Do not place untrusted input directly into (( ... )); validate that it is an expected integer first. Bash also interprets leading-zero integers as octal, so a value such as 08 can cause an arithmetic error.
Exit Status
All three constructs can be used for conditional execution through if, while, &&, and ||.
[ ... ] and [[ ... ]] return 0 for true, 1 for false, and an error status for invalid operations.
(( ... )) returns 0 when the arithmetic result is nonzero and 1 when the result is zero.
That distinction matters with set -e: a standalone (( count == 0 )) can terminate a script because the expression returned status 1. Inside an if condition, the status is handled normally and does not trigger errexit.
Production and Portability
Use [ ... ] in scripts deployed through /bin/sh, POSIX containers, or portable CI runners. Use [[ ... ]] when Bash-specific pattern matching, regular expressions, or safer expansion semantics are useful. Use (( ... )) for counters, thresholds, retries, and resource-limit calculations.
For operational scripts, report failures on stderr, return meaningful exit codes, and avoid relying on set -e as a substitute for explicit error handling. A failure inside a conditional list may be intentional and therefore ignored by errexit.
Best Practices and Pitfalls
Start Bash scripts with #!/usr/bin/env bash and use set -euo pipefail where appropriate.
Quote variables in [ ... ]; use "${value:-}" when an unset variable should be treated as empty under set -u.
Prefer [[ ... ]] over [ ... ] in Bash scripts when it provides clearer pattern or regular-expression behavior.
Use (( ... )) for arithmetic, not general string evaluation.
Never use eval as a substitute for structured tests or parsing.
Run ShellCheck in CI and test failure paths because malformed values, unset variables, and nonzero arithmetic results are common production failure modes.
14
How does Bash compare strings and numeric values?
KnownRevise
Bash provides separate syntax for string comparison and numeric comparison. Use [[ ... ]] for string operations and the arithmetic command (( ... )) for integer comparisons.
Read full answerHide full answer
Applies toBash 5.x on Linux
How It Works
Bash shell variables hold strings; the operator and evaluation context determine whether a value is treated as text or an integer.
Inside [[ ... ]], == and != perform string matching. The right-hand operand can act as a glob pattern unless it is quoted. Thus, [[ "$environment" == "prod" ]] is an exact literal comparison, while [[ "$environment" == prod-* ]] matches names beginning with prod-.
The < and > operators perform lexical ordering rather than numeric ordering. Their results can depend on the locale’s collation rules, which matters when scripts must produce deterministic comparisons.
The numeric operators -eq, -ne, -lt, -le, -gt, and -ge compare integers. The arithmetic command (( ... )) is usually clearer for compound expressions such as (( version >= minimum_version && version < maximum_version )). Within that command, ==, !=, <, <=, >, and >= are arithmetic operators, not string operators.
Bash arithmetic supports signed integers, not floating-point numbers.
Choosing an Operator
Requirement
Bash form
Notes
Exact string equality
[[ "$a" == "$b" ]]
Quote both operands for literal comparison
Glob-style string match
[[ "$name" == prod-* ]]
Unquoted right side is a pattern
Lexical string ordering
[[ "$a" < "$b" ]]
Uses locale collation
Integer comparison
(( a < b ))
Evaluates an arithmetic expression
POSIX numeric comparison
[ "$a" -lt "$b" ]
Works in Bash and POSIX sh
POSIX string equality
[ "$a" = "$b" ]
Use =, not Bash-specific ==
Failure Modes and Security
Validate numeric operands: Numeric comparisons require valid integer expressions. Inside [[ ... ]], -lt evaluates its operands as arithmetic expressions: a value such as 1.5 causes an error (status 1 with a diagnostic), while a word such as abc is treated as a variable name and silently becomes 0.
Do not evaluate untrusted values as arithmetic expressions: Bash recursively evaluates referenced variable values as arithmetic expressions. Validate or normalize input before using it.
Quote patterns when literal matching is intended: [[ "$value" == "literal" ]] prevents the right-hand side from being interpreted as a glob.
Avoid leading-zero surprises: Ordinary arithmetic interprets leading-zero values as octal, so 08 is invalid. The 10# base prefix, such as 10#$value, forces decimal interpretation after validation.
Watch exit statuses: [[ ... ]] returns 0 for true and 1 for false. An arithmetic command returns 1 when its expression evaluates to zero and can return a greater status for evaluation errors.
Understand set -e exceptions: A failed [[ ... ]] or (( ... )) used as an if, while, or until condition does not terminate the script. In an && or || list, errexit is ignored for commands except the final one, so a failing final command can terminate the script. A standalone failed comparison can also terminate it.
Portability and Best Practices
Prefer [[ ... ]] in Bash scripts. It does not perform word splitting or pathname expansion on its operands.
For portable POSIX sh, use [ ... ], single = for string equality, and escape ordering operators as in [ "$a" < "$b" ] (accepted by Bash and dash, though not specified by POSIX).
Set a deterministic locale when comparison order matters, such as export LC_ALL=C for bytewise ordering. This is not equivalent to natural Unicode linguistic sorting.
Pass function inputs as quoted positional parameters such as "$1" and "$@", validate them at the function boundary, and use comparison statuses normally in conditionals and loops.
Use grep, sed, and awk to extract or transform text, then validate the resulting values before comparing them numerically.
Production Bash scripts commonly use set -euo pipefail, but this does not replace explicit error handling or validation.
In CI/CD and orchestration systems, emit errors to stderr and return meaningful nonzero exit codes so failed comparisons and invalid inputs can trigger alerts or remediation.
15
How do Bash for, while, and until loops work?
KnownRevise
Bash loops repeat one or more commands according to a collection, condition, or success check. A for loop is usually best for known collections, while for condition-based or input-driven processing, and until for repeating an operation until it succeeds.
Read full answerHide full answer
Applies toBash 5.x on Linux
Choosing a Loop
Loop
Typical use
Continues when
for
Iterate over files, array elements, or positional parameters
Items remain in the collection
Arithmetic for
Repeat a fixed or calculated number of times
Its arithmetic condition remains true
while
Process streams or repeat while a condition is true
The condition returns success
until
Retry an operation until it succeeds
The condition returns failure
A list-based for loop expands a collection before executing its body. It can iterate over shell words, pathname expansions, command substitutions, or Bash array elements. An arithmetic for loop uses initialization, condition, and update expressions; it is a Bash extension rather than POSIX sh syntax.
How Each Loop Works
A while loop evaluates its condition before every iteration. If the condition succeeds, the body runs; if it fails, the loop ends. A while loop therefore performs zero iterations when its initial condition is false.
An until loop has the inverse condition behavior. Its body runs while the condition fails and stops when the condition succeeds. It also performs zero iterations when the initial condition succeeds. This makes until useful for retries, readiness checks, and waiting for an asynchronous operation to complete.
A list-based for loop consumes the remaining elements of its expanded collection. Quoting an array expansion such as "${items[@]}" preserves each element as one argument and prevents unwanted word splitting and pathname expansion.
Control Flow and Exit Status
All three loop forms support break and continue. break exits the current loop, while continue skips to its next iteration. Numeric variants such as break 2 and continue 2 affect two enclosing loop levels.
A loop normally returns the exit status of the last command executed in its body, or zero if its body never executes. Consequently, callers that check a loop’s status should not assume that reaching the end of the loop means success; the final body command must succeed deliberately.
With set -euo pipefail, set -e normally terminates a script after an unchecked command fails. Commands used directly as while or until conditions are exempt from that behavior, which allows an expected failure to control the loop. Commands in the loop body are generally subject to errexit. Bash also suppresses errexit throughout a function or compound command when that function or command is itself invoked in an exempt context, so production scripts should test critical operations explicitly rather than relying only on set -e.
set -u rejects references to unset variables, while set -o pipefail makes a pipeline fail when any component fails, rather than considering only the final component’s status. A grep exit status of 1, meaning no match, can therefore terminate a script unexpectedly; place such a command in an if condition when no match is an expected outcome.
Production Practices and Portability
Bound retries: An until loop can run forever if success never occurs. Add a maximum attempt count, timeout, or deadline.
Quote expansions: Use quoted parameter and array expansions to prevent word splitting and pathname expansion.
Preserve input: Use read -r when backslashes should remain literal. A while read loop should also handle a final unterminated line when processing files or streams.
Keep iterations observable: Send progress and diagnostic messages to stderr, and ensure the final exit status reflects the operation’s outcome.
Handle cleanup: Use trap to remove temporary files, release locks, and perform other required cleanup on failure.
Avoid Bash-only syntax in POSIX scripts: Arrays, arithmetic for loops, and several expansion features require Bash. POSIX sh supports list-based for, while, and until, but not Bash arithmetic-loop syntax.
16
How do I use a safe Bash read loop for file lines?
KnownRevise
Use a while IFS= read -r loop to process a file one line at a time without losing whitespace or interpreting backslashes. Adding || [[ -n "$line" ]] also processes a final line that does not end with a newline.
Read full answerHide full answer
Applies toBash 5.x on Linux
How the Read Loop Works
read: Reads one line from standard input.
IFS=: Prevents leading and trailing whitespace from being removed.
-r: Treats backslashes literally instead of interpreting escape sequences.
|| [[ -n "$line" ]]: Handles a nonempty final line when the file does not end with a newline.
Redirection: < "$file" connects the loop directly to the file, so the loop runs in the current shell.
The pattern preserves indentation and literal backslashes such as C:\temp\file. By contrast, for line in $(cat "$file") splits on whitespace, removes blank lines, and may perform pathname expansion.
Why Not Pipe into the Loop?
With Bash’s default lastpipe setting disabled, each pipeline component runs in a subshell, so assignments made inside the loop do not affect the parent shell. set -o pipefail can report a failing cat, but it does not preserve variables changed inside the loop.
Direct redirection is preferable because it keeps the loop in the current shell and makes failures from opening or reading the file explicit. Process substitution can also keep the loop in the current shell, but direct redirection is simpler and avoids obscuring failures in the producing command.
Edge Cases and Portability
Situation
Behavior or recommendation
Empty file
The loop body does not execute.
Blank lines
Blank lines are preserved and passed to the loop.
| Final line lacks a newline | The || [[ -n "$line" ]] condition processes it. | | Windows CRLF files | read retains the \r; remove it only when CRLF is expected. | | NUL bytes | Bash variables cannot represent NUL bytes reliably; use a binary-aware tool instead. | | Very large files | The read loop uses constant memory and is preferable to mapfile, which stores the input in memory. | | POSIX sh | Use read -r with [ -n "$line" ]; [[ ... ]], pipefail, and mapfile are Bash extensions. |
For a known CRLF file, remove the trailing carriage return inside the loop with line=${line%$'\r'}. Do this only when the input format is known, because it modifies otherwise valid content.
Code Example
The core pattern is:
while IFS= read -r line || [[ -n "$line" ]]; do
# Process "$line"
done < "$file"
A complete, ShellCheck-compatible script is:
#!/usr/bin/env bash
set -euo pipefail
if (( $# != 1 )); then
printf 'Usage: %s FILE\n' "$0" >&2
exit 2
fi
file=$1
if [[ $file == "-" ]]; then
input_file=/dev/stdin
else
if [[ ! -f $file || ! -r $file ]]; then
printf 'Not a readable regular file: %s\n' "$file" >&2
exit 1
fi
input_file=$file
fi
line_number=0
line=''
while IFS= read -r line || [[ -n $line ]]; do
line_number=$((line_number + 1))
printf '%d:%s\n' "$line_number" "$line"
done < "$input_file"
Save it as process-lines.sh, then run it without requiring executable permissions:
Avoid this pattern because the loop runs in a subshell:
count=0
cat "$file" | while IFS= read -r line; do
count=$((count + 1))
done
Use redirection instead so that count is retained:
count=0
while IFS= read -r line || [[ -n $line ]]; do
count=$((count + 1))
done < "$file"
Production Best Practices
Always quote filenames: < "$file" safely handles spaces, glob characters, and filenames beginning with -.
Use set -euo pipefail for strict error handling, but handle expected failures explicitly inside the loop.
Avoid eval, unquoted expansions, and for line in $(cat "$file").
If the file is rewritten in place while being read, the loop may observe a mixture of old and new content. For configuration and state files, write a new file and atomically rename it into place.
For log rotation, use a tool designed for that behavior, such as tail -F, rather than repeatedly reopening a fixed path.
Run the script through ShellCheck and test empty files, blank lines, missing final newlines, and filenames containing spaces.
17
How do you loop over files with spaces in Bash?
KnownRevise
Use NUL-delimited iteration: have find emit each pathname terminated by \0, then read each value with IFS= read -r -d ''. This keeps filenames containing spaces, tabs, newlines, quotes, and backslashes intact as single values.
Read full answerHide full answer
Applies toBash 5 on Linux (Ubuntu 22.04/24.04, RHEL 9)
How It Works
Filenames are strings, so spaces inside them are valid. Problems usually occur when a filename is incorrectly separated into multiple arguments through:
Unquoted variable expansion, such as for file in $files.
Newline-based input, such as find ... | while read file.
Command substitution, such as for file in $(find ...).
The robust pattern uses find -print0 and read -d '':
-print0: Separates paths with the NUL byte.
-d '': Reads until the NUL byte because Bash represents an empty delimiter as \0.
-r: Prevents backslashes from being interpreted.
IFS=: Prevents leading or trailing whitespace from being removed.
Approach Comparison
Approach
Handles spaces
Handles newlines
Scope
Portability
for file in "$dir"/*.txt
Yes
Yes
One directory, fixed pattern
POSIX-compatible
find -print0 with read -d ''
Yes
Yes
Recursive and flexible
Bash-specific
find -exec command {} +
Yes
Yes
Recursive
POSIX-compatible
find | while IFS= read -r
Yes, with IFS=
No
Recursive
Bash-specific
Code Example
This script safely processes every regular file below the supplied directory:
#!/usr/bin/env bash
set -euo pipefail
target_dir=${1:-.}
if [[ ! -d "$target_dir" ]]; then
printf 'Not a directory: %s\n' "$target_dir" >&2
exit 1
fi
find "$target_dir" -type f -print0 |
while IFS= read -r -d '' file; do
# The complete pathname is preserved in one variable.
printf 'Processing: <%s>\n' "$file"
done
Example output:
Processing: <quarterly report.txt>
Processing: <Q1 final (approved).txt>
For a single directory and a known extension, a Bash glob is simpler:
#!/usr/bin/env bash
set -euo pipefail
target_dir=${1:-.}
shopt -s nullglob
for file in "$target_dir"/*.txt; do
printf 'Processing: <%s>\n' "$file"
done
nullglob prevents an unmatched pattern from being processed as the literal string *.txt. The quoted directory prefix ensures its spaces are not split.
A portable alternative for simple per-file operations is:
find "$target_dir" -type f -exec printf 'Processing: <%s>\n' {} +
Production Best Practices and Pitfalls
Avoid $(find ...): It loses the original argument boundaries and introduces command-substitution parsing problems.
Avoid parsing ls: Its output is intended for human reading and can be ambiguous or paginated.
Handle traversal errors: find reports permission and I/O failures to stderr. With set -o pipefail, its nonzero status causes the pipeline to fail.
Decide on symlink behavior: By default, find -type f does not process symlinks to regular files. Use the appropriate -H, -L, or -P behavior deliberately.
Use explicit status handling around destructive operations. In production, record processed counts and failures, and send structured logs rather than printing every filename.
Remember Bash differences: read -d '', nullglob, arrays, and pipefail are not supported by POSIX sh. Prefer find -exec when portability to systems such as Alpine’s default ash matters.
18
How do you use case statements in Bash?
KnownRevise
A case statement in Bash matches a single value against multiple glob patterns and runs the commands attached to the first match, making it the right tool for branching on string values, command-line flags, or menu choices.
Read full answerHide full answer
Applies toBash 5.x (also largely compatible with POSIX sh, with noted differences)
How It Works
The case keyword evaluates one word, then tests it against each pattern in order. The first matching branch runs, and execution continues after esac. Patterns are glob expressions, not regular expressions, so *, ?, and [...] work. Use | to list alternative patterns in one branch; extended globs such as @(a|b) or +(pattern) require shopt -s extglob.
case "$1" in
start)
echo "Starting service"
;;
stop|restart)
echo "Stopping or restarting"
;;
*)
echo "Unknown command: $1" >&2
exit 1
;;
esac
Each branch ends with ;;. Use ;& to fall through to the next branch (Bash 4+) or ;;& to test subsequent patterns too. The *) catch-all should always be present so unexpected input fails loudly instead of silently doing nothing.
Pattern Matching Details
Patterns are tested branch by branch; the first matching branch wins.
Quote variables inside the test word ("$1") but do not quote patterns, or they become literal strings.
Case is sensitive by default. Set shopt -s nocasematch for case-insensitive matching.
Enable extended globs with shopt -s extglob to use @(a|b), +(pattern), etc.
Common Pitfalls
Forgetting ;; causes a syntax error.
Using regex syntax (^start$) silently never matches because case uses globs.
Leaving out the catch-all branch hides invalid input.
Comparing numbers with case works but (( )) or [[ ]] is clearer for arithmetic.
Code Example
#!/usr/bin/env bash
set -euo pipefail
action="${1:-}"
case "$action" in
start)
systemctl start myapp
;;
stop)
systemctl stop myapp
;;
status)
systemctl status myapp --no-pager
;;
*)
printf 'Usage: %s {start|stop|status}\n' "$0" >&2
exit 2
;;
esac
Best Practices
Always include a *) default branch that prints usage and exits non-zero.
Quote the tested variable; never quote the patterns.
Prefer case over long if/elif chains when branching on a single string.
Keep branches short; delegate complex logic to functions.
Run scripts through ShellCheck to catch unquoted variables and missing defaults.
For POSIX sh portability, avoid ;&, ;;&, and shopt; stick to basic globs and ;;.
Comparison: case vs if/elif
Aspect
case
if/elif
Best for
Single value, multiple string patterns
Complex boolean conditions
Pattern type
Glob
Any test expression
Readability
High for menus/flags
High for ranges/logic
Fall-through
;& / ;;& (Bash 4+)
Not supported
Production Notes
In production scripts, case statements commonly dispatch subcommands (start, stop, reload), parse environment names (prod, staging), or route log levels. Pair them with set -euo pipefail and explicit exit codes so failures surface in CI/CD pipelines and systemd units rather than being swallowed.
19
How are Bash functions defined and invoked?
KnownRevise
In Bash, a function groups reusable commands under a name. Functions execute in the current shell context unless invoked in a subshell, and they communicate success or failure through an exit status.
Read full answerHide full answer
Applies toBash 5.x
Defining and Calling Functions
Define a function with name() { ...; } and invoke it as a command using name or name "argument". Do not include parentheses at the call site.
Bash also accepts function name { ...; }. Unlike the name() form, the function keyword is not portable to POSIX sh.
Arguments and Variable Scope
$1, $2, …: Positional arguments passed to the function.
$@: Expands to all positional parameters as separate arguments.
$#: Number of positional parameters.
shift: Removes the first positional parameter and renumbers the others.
local: Creates or shadows a variable within the function, reducing accidental global-state changes.
Dynamic scoping: A local variable is visible to functions called by the current function and takes precedence over a global variable of the same name.
When forwarding every argument, use "$@":
run_command() {
local command_name="$1"
shift
"$command_name" "$@"
}
This preserves argument boundaries and prevents unintended word splitting or pathname expansion.
Return Codes and Error Handling
A function returns the exit status of its last executed command unless return specifies another status.
return N: Ends the function with status N.
exit N: Ends the current shell or subshell, not merely the function.
Exit statuses range from 0 through 255; 0 conventionally means success.
With set -euo pipefail, set -e exits on an unhandled simple-command failure subject to documented exceptions, set -u rejects unset variables, and pipefail makes a pipeline fail when any component fails. When a function is invoked as an if or && condition, set -e is ignored for commands in the function body, so intermediate failures may not stop it. Validate inputs explicitly and return meaningful statuses.
Code Example
#!/usr/bin/env bash
set -euo pipefail
WORK_DIR=""
log() {
local message="${1:-}"
printf '[%s] %s\n' \
"$(date -u +%Y-%m-%dT%H:%M:%SZ)" \
"$message"
}
cleanup() {
if [[ -n "$WORK_DIR" ]]; then
rm -rf -- "$WORK_DIR"
fi
}
greet() {
local person="${1:-world}"
printf 'Hello, %s!\n' "$person"
}
run_command() {
local command_name="$1"
shift
"$command_name" "$@"
}
validate_port() {
local port="${1:-}"
if [[ ! "$port" =~ ^[0-9]+$ ]] ||
(( port < 1 || port > 65535 )); then
printf 'Invalid port: %s\n' "$port" >&2
return 1
fi
return 0
}
validate_config() {
local file="$1"
if [[ ! -f "$file" ]]; then
log "Configuration file not found: $file" >&2
return 2
fi
if [[ ! -s "$file" ]]; then
log "Configuration file is empty: $file" >&2
return 3
fi
if [[ ! -r "$file" ]]; then
log "Configuration file is not readable: $file" >&2
return 4
fi
log "Configuration validated: $file"
}
main() {
local config_file
WORK_DIR="$(mktemp -d)"
trap cleanup EXIT
config_file="$WORK_DIR/app.conf"
printf 'healthy\n' > "$config_file"
greet "SRE"
run_command printf '<%s>\n' 'argument with spaces' '*'
validate_port 8080
validate_config "$config_file"
}
main
Representative output:
Hello, SRE!
<argument with spaces>
<*>
[2026-03-10T14:25:31Z] Configuration validated: /tmp/tmp.K8s3dP/app.conf
Production and Portability Considerations
Validate dependencies: Check required commands, files, permissions, and connectivity before making changes.
Use bounded retries: Retry only transient failures, with a limit and delay; do not retry permanent validation errors indefinitely.
Add timeouts: Wrap network operations with timeout or the tool’s native timeout option.
Protect secrets: Avoid exposing credentials directly in command arguments, and remember that set -x can write secrets to logs.
Separate output streams: Send diagnostics and failures to stderr; reserve standard output for requested data or successful status output.
Design idempotently: Automation should safely handle repeated deployment, backup, or cleanup operations.
Preserve diagnostics: During incidents, retain logs and meaningful exit statuses so automation platforms can identify the failed stage.
Bash also supports arrays, process substitution, local, and export -f. POSIX sh has a smaller standardized feature set; name() definitions are portable, but local, arrays, process substitution, and exported functions should not be assumed to work everywhere.
Best Practices and Common Pitfalls
Quote expansions: use "$var", "$@", and "${array[@]}".
Prefer local for temporary variables to prevent global-state corruption.
Use return inside functions; reserve exit for top-level script termination.
Do not rely on set -e as a substitute for explicit error handling.
Avoid echo for variable output; use printf.
Check command and file existence before use.
Keep functions focused so failures have clear, testable statuses.
Run scripts through ShellCheck and test both success and failure paths before deployment.
20
How do Bash functions return values?
KnownRevise
Bash functions can return a value through standard output and an exit status through return. In production scripts, use standard output for data and exit status for success or failure.
Read full answerHide full answer
Applies toBash 5.x
Returning Data with Standard Output
A function commonly prints its result with printf. The caller captures that output using command substitution:
result=$(calculate_sum 10 20)
The result is assigned to a variable because command substitution executes the function in a subshell. Variable assignments made inside the function are therefore never visible to the caller, not even through a nameref or global variable. To return values through variables, call the function directly rather than inside $(...).
Prefer printf over echo: it handles formatting predictably and avoids implementation differences between shells.
Returning an Exit Status
Use return to stop a function and provide a status code:
return 0 indicates success.
A nonzero value indicates failure.
return without an argument returns the status of the most recently executed command.
Status codes are represented as integers in the range 0 through 255.
A caller can test the status directly:
if validate_config "$config_file"; then
deploy "$config_file"
else
printf 'Configuration validation failed\n' >&2
exit 1
fi
When a function is called as a simple command, its exit status becomes the status of that command. This matters when set -e is enabled because a failing function can terminate the script.
Combining Data and Status
A robust function should print the requested value to standard output and reserve standard error for diagnostics. It should also return nonzero when the operation fails.
#!/usr/bin/env bash
set -euo pipefail
divide_numbers() {
local numerator=$1
local denominator=$2
if (( denominator == 0 )); then
printf 'Error: denominator must not be zero\n' >&2
return 1
fi
printf '%s\n' "$((numerator / denominator))"
}
if result=$(divide_numbers 20 5); then
printf 'Quotient: %s\n' "$result"
else
printf 'Division failed\n' >&2
exit 1
fi
Example successful output:
Quotient: 4
The if result=$(...) pattern is important with set -e: because the function call is part of a conditional, its expected nonzero status does not immediately terminate the script.
Key Behavior and Failure Modes
Command substitution captures stdout only: diagnostics written to standard error remain visible to the user or monitoring system.
Trailing newlines are removed: command substitution strips trailing newline characters from captured output.
Bash variables cannot store NUL bytes: use files or a structured format such as JSON when handling arbitrary binary data.
Function-local state should use local: prevents accidental modification of global variables.
Unquoted expansions are unsafe: always quote variable expansions such as "$result" and "$config_file".
A function has one exit status and one stdout stream: to return multiple values, use a nameref (declare -n) or a global variable with a directly called function (not inside command substitution), or print structured output such as JSON.
Use printf '%q' when constructing shell commands: it provides shell-safe quoting for values such as filenames with spaces.
Production Practices
Return 0 only when the function completed its requested operation successfully.
Return distinct nonzero statuses when callers need to distinguish failures, such as 1 for invalid input and 2 for an unavailable dependency.
Send errors to standard error with >&2 so command substitution does not mistake diagnostic text for a successful result.
Avoid returning formatted, multi-field text that callers must parse with awk or sed. For structured data, print JSON and parse it with a suitable tool.
Remember that return exits the current function. It does not by itself stop the parent script unless the function is invoked in a context that causes the script to exit.
Bash return is broadly supported, but the surrounding scripting features are not fully POSIX. A script using local, arrays, [[ ... ]], or process substitution should declare Bash explicitly with #!/usr/bin/env bash; a strict POSIX sh script should avoid those features.
21
How do local and global variables differ in Bash functions?
KnownRevise
In Bash, variables assigned inside a function are global by default unless explicitly declared with local. A local variable exists only for the duration of the function call and temporarily shadows a global variable with the same name.
Read full answerHide full answer
Applies toBash 5.x
Scope Fundamentals
Bash uses dynamic scoping for function-local variables. This differs from languages such as JavaScript or Python that generally use lexical scoping.
Behavior
Local variable
Global variable
Declaration
local name=value
Assignment outside a function, or declare -g inside one
Lifetime
Until the function returns
Until the shell exits or it is unset
Name conflict
Shadows a global variable during the call
Remains accessible after the function returns
Child processes
Not exported by default
Exported only when marked with export
Common use
Temporary state, arguments, intermediate results
Configuration, constants, shared counters
A function executes in the current shell, so assigning to a global variable inside it normally persists after the function returns. By contrast, a local variable is destroyed when the function returns, and any previous global value with the same name is restored.
Functions also receive their own positional parameters. After the function returns, the caller’s positional parameters are restored.
Dynamic Scoping
A local variable is visible to every function called by its declaring function. For example, if configure declares local region="eu-west-1" and then calls deploy, deploy can read region. If deploy assigns to region without declaring it local, it changes the nearest dynamically scoped variable, which in this case is configure’s local variable.
This behavior is convenient for small scripts but can make larger systems harder to reason about. Explicit parameters and return values make dependencies clearer.
Use declare -g when a global assignment inside a function is intentional, such as declare -g request_count=$((request_count + 1)).
Production Considerations
Isolate mutable state: Declare working variables with local to prevent accidental changes to deployment configuration, counters, paths, or credentials.
Return results explicitly: Write results to standard output and communicate failures through the exit status. Use return from a function rather than exit, which terminates the entire script.
Treat configuration as immutable: Mark constants with readonly, such as readonly APP_NAME="payments".
Export deliberately: export makes a global variable available to child processes. Local variables are not exported by default.
Check failures explicitly: set -euo pipefail catches many errors but does not replace explicit error handling for expected operational failures.
Avoid masking command status: Declare and assign separately because local output="$(command)" can hide the command’s failure status. Prefer declaring local output, then assigning with if ! output="$(fetch_metadata)"; then ...; fi.
Bash and POSIX Portability
local is a Bash feature and is not part of POSIX sh. Some shells, such as ksh, provide similar behavior, while dash does not. Portable scripts should use distinct variable names, positional parameters, or a subshell instead of relying on local.
For production Bash automation, prefer #!/usr/bin/env bash, set -euo pipefail, quoted expansions, printf, local function state, and ShellCheck-clean code.
22
How do Bash arrays store and expand elements?
KnownRevise
Bash arrays are indexed or associative collections whose elements are strings. Indexed arrays use integer subscripts and iterate in subscript order, while associative arrays use non-empty string keys and have no defined iteration order. Quoted "${array[@]}" expansion preserves each element’s boundaries, spaces, and empty values.
Read full answerHide full answer
Applies toBash 5.x
Array Types and Storage
Type
Declaration
Key
Typical Use
Indexed array
declare -a items=(...)
Integer subscript
Ordered lists, command results
Associative array
declare -A config=(...)
Non-empty string
Maps, lookup tables
Bash stores indexed arrays as sparse collections and associative arrays using hash-table-like structures. Array elements are strings, even when they look numeric. An index can be skipped:
declare -a items=("first" "third")
items[5]="sixth"
"${#items[@]}" reports the number of assigned elements, not the highest index plus one.
Accessing and Modifying Elements
Read an element: "${items[0]}"
Read the highest-numbered occupied indexed element: "${items[-1]}"
Append: items+=("another-item")
Assign by index: items[2]="third-item"
Delete an element: unset 'items[2]'
Delete the entire array: unset items
List assigned indexes: "${!items[@]}"
List associative keys: "${!config[@]}"
For a dense indexed array, "${items[${#items[@]} - 1]}" is equivalent to "${items[-1]}". That count-based expression is not reliable for sparse arrays.
Associative array values are accessed by key:
declare -A limits=(
[cpu]="500m"
[memory]="1Gi"
)
printf '%s\n' "${limits[cpu]}"
Associative keys have no guaranteed iteration order, so sort them before producing deterministic output.
Expansion and Quoting
Quoting is critical because unquoted expansions undergo word splitting and pathname expansion:
declare -a values=("alpha" "beta gamma")
printf '<%s>\n' "${values[@]}"
"${values[@]}" expands each element as a separate argument while preserving spaces and empty elements.
"${values[*]}" joins elements using the first character of IFS; an empty IFS produces no separator.
${#values[@]} gives the element count.
"${values[@]:1:2}" selects two elements beginning at index 1.
"${!values[@]}" returns assigned indexes.
Under set -u, accessing a missing associative key can terminate the script. Use a default when absence is expected:
declare -A limits=()
replicas="${limits[replicas]:-1}"
Passing Arrays to Functions
For an indexed array, Bash does not pass it as one function argument. Pass its elements individually and reconstruct a local indexed array from "$@":
process_items() {
local -a selected=("$@")
local item
for item in "${selected[@]}"; do
printf 'Processing %s\n' "$item"
done
}
For an associative array, preserve the key/value relationship; "${map[@]}" passes values only. Pass explicit keys and values or use a Bash nameref (local -n) carefully.
Use local -a for indexed arrays and local -A for associative arrays. Without local, assignments can modify a global variable.
Code Example
#!/usr/bin/env bash
set -euo pipefail
declare -a services=("api" "worker")
services+=("scheduler")
declare -A endpoints=(
[api]="https://api.example.com"
[worker]="https://worker.example.com"
[scheduler]="https://scheduler.example.com"
)
show_service() {
local -a selected=("$@")
local service
for service in "${selected[@]}"; do
printf '%s -> %s\n' "$service" "${endpoints[$service]}"
done
}
printf 'count=%d\n' "${#services[@]}"
printf 'second=%s\n' "${services[1]}"
printf 'slice='
printf '<%s> ' "${services[@]:1:2}"
printf '\n'
show_service "${services[@]}"
Use set -euo pipefail: set -e exits after a failed simple command except in exempt contexts such as conditionals, &&/|| lists, and loop conditions; set -u errors on unset variables; pipefail makes a pipeline return nonzero when any component fails. Still check failures whose meaning is important.
Always quote expansions: "${array[@]}" prevents word splitting and unintended pathname expansion.
Handle empty input explicitly: Check "${#array[@]}" before accessing an element. Bash 5.x permits expansion of empty arrays under set -u, but missing indexed or associative elements still require care.
Do not use eval: It can reinterpret array contents as shell syntax and introduce injection vulnerabilities.
Protect external commands: command -- "${array[@]}" only ends option parsing for command itself. Use program -- "${array[@]}" when the target program documents --, or validate and reject values beginning with -.
Avoid arrays for structured data: Bash variables cannot contain NUL bytes, and arrays cannot be exported as environment variables. Use jq for JSON or a dedicated serialization format when exchanging structured data.
Remember portability limits: POSIX sh has no arrays. Portable scripts commonly use positional parameters with set -- or an external tool such as awk.
Prefer mapfile for line-oriented input: mapfile -t lines < file reads each newline-delimited record into a separate element without invoking read in a loop.
23
How do Bash associative arrays store and retrieve key-value pairs?
KnownRevise
Associative arrays in Bash are hash-table-style arrays that map string keys to string values. They are declared with declare -A, support key-based lookup, and do not preserve insertion order.
Read full answerHide full answer
Applies toBash 5.x
How associative arrays work
Declaration: declare -A map creates an associative array.
Key-value storage: map["key"]="value" associates a string key with a value.
Lookup: map["key"] retrieves the value.
Key expansion: "${!map[@]}" expands to all keys, while "${map[@]}" expands to all values.
Internal implementation: Bash uses a hash-table-style index, making key lookup efficient for large collections, but it does not expose a deterministic key order.
Unlike an indexed array, an associative array does not use numeric positions. It is appropriate for configuration data, metadata, routing tables, and mappings between names and values.
Keys are strings. An empty string is a valid key, although empty keys are usually best avoided for clarity.
Values are stored as strings, even when they look numeric.
Quote keys and values, especially when they contain spaces, glob characters, or shell metacharacters.
declare -A map=([key]="value") is a common inline initialization form.
Inside functions, local -A map creates a function-local associative array.
A missing key requires careful handling. With set -u, directly expanding an unset key can terminate the script:
set -u
if [[ -z "${settings["timeout"]+present}" ]]; then
printf 'The timeout setting is missing\n' >&2
else
printf '%s\n' "${settings["timeout"]}"
fi
The +present test distinguishes an unset key without triggering an unbound-variable error. ${map[key]-default} supplies a default only when the key is unset, whereas ${map[key]:-default} also uses the default when the value is empty. In Bash 4.2 and later, [[ -v map[key] ]] is also a clear way to test whether a key exists.
Iteration and production use
Iterate over keys and retrieve each corresponding value:
declare -A endpoints=(
[us-east-1]="https://api-us-east-1.example.com"
[eu-west-1]="https://api-eu-west-1.example.com"
)
for region in "${!endpoints[@]}"; do
printf '%s -> %s\n' "$region" "${endpoints[$region]}"
done
The iteration order is unspecified, so sort keys explicitly when output order matters:
mapfile -t sorted_regions < <(
printf '%s\n' "${!endpoints[@]}" | sort
)
for region in "${sorted_regions[@]}"; do
printf '%s -> %s\n' "$region" "${endpoints[$region]}"
done
For text-processing pipelines, keep the associative array in Bash and use tools such as grep, sed, and awk for transformation. For example, awk -F '\t' can parse tab-separated records, while grep -F -- performs fixed-string matching without interpreting the pattern as a regular expression. Avoid eval, and avoid parsing ls output; use find, printf, and null-delimited input when filenames may contain whitespace or newlines.
Code Example
This Bash 5 script demonstrates declaration, arguments, conditionals, functions, loops, exit codes, and safe error handling:
#!/usr/bin/env bash
set -euo pipefail
declare -A endpoints=(
[us-east-1]="https://api-us-east-1.example.com"
[eu-west-1]="https://api-eu-west-1.example.com"
)
list_regions() {
local region
for region in "${!endpoints[@]}"; do
printf '%s\n' "$region"
done
}
lookup_endpoint() {
local region="$1"
if [[ ! "$region" =~ ^[a-z]{2}-[a-z]+-[0-9]+$ ]]; then
printf 'Invalid region: %s\n' "$region" >&2
return 2
fi
if [[ -z "${endpoints[$region]+present}" ]]; then
printf 'Unknown region: %s\n' "$region" >&2
return 1
fi
printf 'Region %s uses %s\n' "$region" "${endpoints[$region]}"
}
case "${1:-}" in
--list)
list_regions
;;
"")
printf 'Usage: %s <region> | --list\n' "$0" >&2
exit 64
;;
*)
lookup_endpoint "$1"
;;
esac
Example lookup output:
Region us-east-1 uses https://api-us-east-1.example.com
The --list output may list regions in either order.
Best practices, portability, and failure modes
Quote every expansion: "${value}", "${array[$key]}", and "${!array[@]}" prevent word splitting and pathname expansion.
Use set -euo pipefail: set -e improves failure handling, set -u catches unset variables, and pipefail detects failures earlier in pipelines. Still, explicitly check critical commands because set -e has conditional exceptions.
Validate external input: Treat command-line arguments and file contents as untrusted. Validate keys before using them as associative-array subscripts.
Return meaningful exit codes: For example, 64 for usage errors, 2 for invalid input, and 1 for a missing resource.
Do not rely on ordering: Sort keys when deterministic output is required.
Use Bash explicitly: The script requires Bash because POSIX sh does not provide associative arrays, [[ ... ]], local, or Bash arrays. For portable /bin/sh automation, use awk, sed, cut, or another external tool instead.
Remember process scope: Associative arrays exist only in the current shell process; they are not a persistent database or a replacement for structured configuration formats such as JSON or YAML.
24
How does Bash evaluate integer arithmetic expressions?
KnownRevise
Arithmetic expansion is Bash’s built-in mechanism for integer calculations. $((expression)) expands to an integer-valued word, while ((expression)) is an arithmetic command whose exit status reflects whether the result is zero.
Read full answerHide full answer
Applies toBash 5.0-5.3 on 64-bit Linux
How Evaluation Works
Bash supports two related constructs:
$((expression)) performs arithmetic expansion and substitutes the result into the surrounding word.
((expression)) performs arithmetic evaluation as a command. Its exit status is 0 when the result is nonzero and 1 when the result is zero.
Arithmetic expressions normally reference variables without the $ prefix. For example, with x=3 and y=x, $((x + y)) expands to 6.
A referenced variable’s value is itself evaluated as an arithmetic expression. Thus, if y='3 + 4', $((y)) expands to 7.
Shell expansions embedded directly in the expression occur before final arithmetic evaluation. Parentheses should be used when precedence would otherwise be unclear.
Without set -u, an unset arithmetic variable normally has the value 0; with set -u, referencing it is an error.
Integer division truncates toward zero, and the remainder has the same sign as the dividend. Bash uses fixed-width signed integers—64-bit on the stated platform—so values outside the representable range can overflow or wrap.
Hexadecimal and octal constants are supported. For example, $((0x10 + 010)) evaluates to 24, because 0x10 is 16 and 010 is octal 8. The 10#number form forces base-10 interpretation, so 10#08 is 8 rather than an invalid octal value.
Under the Hood and Portability
Bash evaluates arithmetic internally; $((...)) does not start an external process such as bc or awk. This makes it inexpensive for deployment checks, health probes, and resource calculations.
POSIX sh supports the $((...)) container, but not every Bash operator or extension. In particular, ** and base notation such as 10#... are not required by POSIX. Other Bash-specific constructs include the ((...)) command, [[ ... ]], local, arrays, and set -o pipefail.
Use $((...)) rather than the older external expr command. Use bc, awk, or Python for floating-point or arbitrary-precision calculations.
Code Example
This complete Bash script validates decimal syntax, handles a zero divisor explicitly, and demonstrates arithmetic expansion:
#!/usr/bin/env bash
set -euo pipefail
is_uint() {
[[ $1 =~ ^[0-9]+$ ]]
}
calculate() {
local left="$1"
local right="$2"
if ! is_uint "$left" || ! is_uint "$right"; then
printf 'error: operands must be non-negative decimal integers\n' >&2
return 2
fi
# Force base 10 so values such as 08 are not interpreted as octal.
left=$((10#$left))
right=$((10#$right))
if (( right == 0 )); then
printf 'error: division by zero\n' >&2
return 3
fi
local sum
local product
local quotient
local remainder
sum=$((left + right))
product=$((left * right))
quotient=$((left / right))
remainder=$((left % right))
printf 'sum=%d\n' "$sum"
printf 'product=%d\n' "$product"
printf 'quotient=%d\n' "$quotient"
printf 'remainder=%d\n' "$remainder"
}
main() {
if (( $# != 2 )); then
printf 'usage: %s LEFT RIGHT\n' "${0##*/}" >&2
return 2
fi
calculate "$@"
}
main "$@"
Running ./arithmetic.sh 10 3 produces:
sum=13
product=30
quotient=3
remainder=1
Failure Modes and Security
Division by zero: This is an arithmetic evaluation error, not merely an arithmetic command returning status 1. Bash prints a diagnostic to stderr and terminates a noninteractive shell even without set -e.
set -e pitfall: A standalone ((count > limit)) terminates a script under errexit when the comparison is false. Put such tests in an if condition or append an explicit || handler.
Unset variables: set -u exposes accidental use of unset arithmetic variables. Initialize operands explicitly and use ${value:-default} where appropriate.
Invalid expressions: Syntax errors are fatal to a noninteractive Bash shell because they are evaluation errors rather than ordinary command failures.
Untrusted input: Do not place unvalidated input directly into arithmetic expressions or use eval. Validate the expected decimal format and acceptable range first.
Leading zeros: Arithmetic interprets leading-zero values as octal. Use 10#$value after validating that the value contains only decimal digits.
Production Best Practices
Use set -euo pipefail in automation scripts, but do not treat it as a substitute for explicit error handling.
Quote variable expansions and pass arguments as separate parameters.
Validate both syntax and business limits; even representable operands can overflow when multiplied or added.
Prefer integer scaling for percentages, such as $((used_bytes * 100 / total_bytes)), while checking for intermediate multiplication overflow.
Send diagnostics to stderr so monitoring systems can distinguish failures from normal output.
Return distinct nonzero statuses for invalid input and invalid operational state, and never silently continue after an arithmetic evaluation failure.
25
How does Bash read capture user input from a terminal?
KnownRevise
In Bash, read is a shell builtin that reads one line of input into one or more variables. For reliable interactive input, use IFS= read -r to preserve whitespace and treat backslashes literally.
Read full answerHide full answer
Applies toBash 5.x on Linux and Unix-like systems
How read Works
The basic form is:
IFS= read -r variable
IFS= prevents leading, trailing, and repeated whitespace from being stripped or split.
-r prevents backslashes from escaping characters; without it, read interprets backslash sequences.
If variable names are supplied, Bash splits the line according to IFS.
If no variable name is supplied, the line is stored in REPLY.
Multiple variables receive successive fields, with the final variable receiving the remaining fields.
For example:
IFS= read -r username
IFS=, read -r environment region
The second command expects input such as production,eu-west-1. Bash-style comma-separated values are not the same as CSV: fields containing commas or quotes require a proper CSV parser.
Useful Options
Option
Behavior
-r
Reads input without interpreting backslash escapes.
-p prompt
Displays prompt before reading input; this is a Bash extension.
-s
Suppresses terminal echo, making it suitable for passwords and tokens.
-t timeout
Stops waiting after the specified number of seconds.
-n nchars
Returns after reading nchars, rather than waiting for a newline.
-N nchars
Reads exactly nchars, ignoring the input delimiter.
-a array
Stores the fields in an indexed array.
-d delimiter
Uses delimiter instead of newline to terminate the input.
A successful read returns exit status 0. It returns a non-zero status for EOF, timeout, an invalid file descriptor, or another read failure. A timeout produces a status greater than 128.
Reading Files and Standard Input
read is commonly used to process files or redirected input:
while IFS= read -r line; do
printf '%s\n' "$line"
done < input.txt
The || [[ -n $line ]] variant also processes a final line that does not end with a newline:
line=''
while IFS= read -r line || [[ -n $line ]]; do
printf '%s\n' "$line"
done < input.txt
For log or configuration processing, read only retrieves each line. Use grep to select matching lines, sed to substitute text, and awk for field- or record-oriented processing. Prefer jq or yq for structured JSON or YAML instead of parsing them with Bash variables.
Code Example
This interactive script uses safe quoting, validation, a function, explicit error handling, and distinct exit codes:
#!/usr/bin/env bash
set -euo pipefail
read_service() {
local service=''
if ! IFS= read -r -p 'Service name: ' service; then
printf '\nNo input received.\n' >&2
return 1
fi
if [[ ! $service =~ ^[a-z][a-z0-9-]{0,62}$ ]]; then
printf 'Invalid service name: %s\n' "$service" >&2
return 2
fi
printf 'Validated service: %s\n' "$service"
}
if read_service; then
exit 0
else
status=$?
exit "$status"
fi
Example output:
Service name: api
Validated service: api
This script processes a file supplied as its first argument and checks that the argument is a readable regular file:
#!/usr/bin/env bash
set -euo pipefail
if (( $# != 1 )); then
printf 'Usage: %s FILE\n' "$0" >&2
exit 64
fi
if [[ ! -f $1 || ! -r $1 ]]; then
printf 'Not a readable regular file: %s\n' "$1" >&2
exit 66
fi
line=''
while IFS= read -r line || [[ -n $line ]]; do
if [[ -z $line || $line == \#* ]]; then
continue
fi
printf '<%s>\n' "$line"
done < "$1"
Production and Portability Guidance
Do not rely on a bare read with set -e. EOF or timeout is a normal outcome in pipelines. Put read in an if, while, or until condition, or handle its failure explicitly.
Avoid interactive input in CI/CD and services. Accept configuration through validated arguments, environment variables, files, or standard input so automation cannot hang waiting for a terminal.
Use timeouts where appropriate.read -t prevents an unattended process from waiting indefinitely.
Protect secrets.read -s prevents terminal echo, but do not log the resulting value. Avoid passing secrets as command-line arguments because they may appear in process listings.
Do not execute input.read -r preserves backslashes, but input is still data, not code; never pass it to eval or execute it as shell code.
Bash is not POSIX sh.read -r is broadly available, but read -p, arrays, [[ ... ]], and other Bash extensions are not portable. In a POSIX sh script, use printf for prompts and avoid Bash-only options.
26
How does getopts parse Bash command-line options?
KnownRevise
Bash’s getopts builtin parses short command-line options such as -v, -o output, and -h. It processes options from the script’s positional parameters, stores option arguments in OPTARG, and leaves non-option arguments for the script to handle.
Read full answerHide full answer
Applies toBash 5.x (Linux and other Unix-like systems)
How It Works
getopts is invoked with three arguments:
getopts optstring option_variable
Optstring: Defines the supported options.
Option variable: Receives the current option letter, such as h or v.
OPTARG: Contains the argument for the current option, when applicable.
OPTIND: Tracks the next positional parameter to examine. getopts updates it automatically.
For example, this optstring defines:
:hvo:
h — no argument
v — no argument
o — requires an argument
The leading : — causes a missing option argument to be reported as : rather than ?
A while getopts ... loop continues until all options have been processed or an invalid option is encountered.
Code Example
#!/usr/bin/env bash
set -euo pipefail
usage() {
printf 'Usage: %s [-h] [-v] [-o OUTPUT] [--] [TARGET...]\n' \
"${0##*/}"
}
output=""
verbose=false
while getopts ":hvo:" option; do
case "$option" in
h)
usage
exit 0
;;
v)
verbose=true
;;
o)
output=$OPTARG
;;
:)
printf 'Error: option -%s requires an argument\n' "$OPTARG" >&2
usage >&2
exit 2
;;
\?)
printf 'Error: unknown option -%s\n' "$OPTARG" >&2
usage >&2
exit 2
;;
esac
done
# Remove the options that getopts processed.
shift "$((OPTIND - 1))"
if (($# == 0)); then
usage >&2
exit 2
fi
if [[ "$verbose" == true ]]; then
printf 'Deploying to %d target(s)\n' "$#"
fi
for target in "$@"; do
printf 'output=%s target=%s\n' "$output" "$target"
done
getopts supports short options, not GNU-style long options such as --output.
Options normally appear before positional arguments. Parsing stops at the first non-option argument.
-- explicitly ends option processing, allowing values beginning with - to be treated as operands.
An option requiring an argument obtains it from OPTARG.
An unknown option produces ?; with the leading : in the optstring, a missing argument produces :.
The loop itself must use while getopts ...; do not manually increment OPTIND.
Error Handling and Security
Use set -euo pipefail in production scripts, but still handle getopts error cases explicitly. A while condition does not normally trigger errexit when getopts simply finishes parsing.
Quote every expansion, especially "$OPTARG", "$@", and "$#".
Avoid eval; it can reinterpret user-controlled input as shell code.
Do not pass passwords or tokens directly as command-line arguments when avoidable, because other users may inspect the process list. Use an environment variable, protected file, or secret manager instead.
Return conventional exit codes: 0 for success, 2 for invalid usage, and application-specific nonzero codes for operational failures.
Bash and POSIX Differences
getopts is available in POSIX sh, so the basic parsing pattern is portable. However, this example also uses Bash features such as [[ ... ]] and pipefail, so it is not POSIX sh code. For portable automation, use POSIX conditionals, avoid pipefail, and rely only on the POSIX getopts behavior.
27
What is Bash command substitution, and how does it differ from backticks?
KnownRevise
Command substitution runs a command and replaces the substitution expression with its standard output. Bash’s $(...) is the preferred form because its quoting and nesting rules are clearer; backticks are a legacy form that remains supported but has more complicated escaping rules.
Read full answerHide full answer
Applies toGNU Bash 5.x and POSIX.1-2024 `sh`
How It Works
Command substitution captures a command’s standard output:
$(command) executes command.
Its standard output becomes text used by the surrounding command or assignment.
Standard error is not captured unless it is explicitly redirected.
Trailing newline characters are removed from the captured output.
Bash variables cannot represent NUL bytes, so NUL bytes in captured output are not preserved.
For example, timestamp="$(date -u +'%Y-%m-%dT%H:%M:%SZ')" stores a UTC timestamp in timestamp.
$(...) Versus Backticks
Both forms perform command substitution, but $(...) is preferred.
Behavior
$(...)
Backticks
Readability
Clear delimiters
Visually noisy
Nested substitution
Supports direct nesting
Nested backticks require escaping
Quoting
Balanced quotes are handled predictably
Has special legacy backslash rules
ShellCheck
Does not trigger SC2006
SC2006 recommends replacing them with $(...)
Recommendation
Standard modern choice
Use only for legacy compatibility
POSIX sh supports both forms, but $(...) is easier to read, quote, and maintain. For example, $(printf '%s' "$(hostname)") nests a substitution without the fragile escaping required for nested legacy backticks.
Quoting and Expansion
Always quote a substitution when its output should remain one argument, such as files="$(find /var/log -maxdepth 1 -type f)". Quoting preserves whitespace and prevents word splitting and pathname expansion.
Without quotes, word splitting and pathname expansion can occur when the substitution is used as an unquoted command argument. This may produce unexpected arguments or expand filenames containing characters such as *, ?, or [.
Assignment is a common exception: value="$(command)" does not split the captured output into multiple words or apply pathname expansion to it.
Failure Handling and Exit Codes
Under set -euo pipefail, a simple assignment whose only operation is command substitution receives the substitution’s exit status. Embedding it in another command can hide that failure. For example, in printf '%s\n' "$(some-tool)", the successful status of printf masks a failure by some-tool.
A common Bash pitfall is local output="$(some-tool)": local can mask the substitution’s exit status. Inside a function, declare and assign separately when failure must be detected: use local output, followed by output="$(some-tool)".
The set -o pipefail option is a Bash feature, not a POSIX sh option.
Production Best Practices
Quote substitutions: Use "$(command)" when the output must remain one argument; use arrays or mapfile when capturing multiple records.
Do not use eval: Never build a command string from untrusted input. Pass values as separately quoted arguments instead.
Handle stderr explicitly: For example, output="$(tool 2>>"$error_log")" captures output while preserving diagnostics in a log.
Check exit status: Use an assignment-only form or test the command explicitly when failures matter.
Avoid unnecessary sudo: Perform privileged operations only where required, and validate paths and input.
Preserve diagnostics: Capturing standard output without handling standard error can make incident diagnosis harder.
Prefer $(...) consistently: It avoids the quoting problems associated with legacy backticks and does not trigger ShellCheck SC2006.
28
What is process substitution in Bash?
KnownRevise
Process substitution is a Bash feature that lets a command’s input or output be treated as a file path, using <(command) to provide readable input and >(command) to provide writable output.
Read full answerHide full answer
Applies toBash 5.x (also available in ksh and zsh; not POSIX sh)
How It Works
<(list) runs list and connects its standard output to a file-like path, such as /dev/fd/63 or a named pipe.
>(list) runs list and connects its standard input to a writable file-like path.
The shell passes that generated path as an argument to another command.
The substituted command runs asynchronously, so the main command can consume or produce data while it runs.
Common Uses
Compare transformed outputs without creating temporary files, such as diff <(sort a.txt) <(sort b.txt).
Feed multiple generated inputs to a command that expects filenames.
Avoid the subshell problem when reading command output with while read ...; do ...; done < <(find ...).
Send a command’s output to several consumers with tee >(grep ERROR > errors.log).
Pitfalls and Failure Modes
The exit status of the substituted process is not part of the main command’s status, so set -e and pipefail do not catch its failure.
With >(...), the main command may finish before the consumer has processed all data, causing truncated or lost output.
Large FIFO-based transfers can block if the reader and writer are not coordinated.
Do not quote the process substitution construct itself; quote variables and paths inside it.
Portability
Process substitution is a Bash, ksh, and zsh feature. POSIX sh does not support it.
In portable scripts, use temporary files, named pipes created with mkfifo, or standard pipelines instead.
Best Practices
Use process substitution to avoid temporary files and unwanted subshells.
Do not rely on it for critical error propagation; check important subprocesses separately.
Prefer while read ... done < <(command) over command | while read ... when variables must persist.
Run scripts through ShellCheck to catch quoting and portability issues.
29
How do pipes and redirection work in Bash scripts?
KnownRevise
Pipes connect the standard output of one command to the standard input of another, while redirection changes where a command’s file descriptors read or write. Together, they are the core mechanisms for composing commands and controlling data flow in a shell script.
Read full answerHide full answer
Applies toBash 5.x (Linux, macOS with Homebrew Bash)
How Pipes and Redirection Work
A pipe is an anonymous, kernel-managed byte stream. Bash starts the commands as separate processes, connects the left command’s standard output to the right command’s standard input, and waits for the pipeline to finish. Linux pipes typically have a 64 KiB default capacity, allowing producer and consumer to run concurrently.
cmd1 | cmd2 connects stdout of cmd1 to stdin of cmd2.
cmd1 |& cmd2 also connects stderr of cmd1 into the pipe.
|& is Bash-specific; POSIX sh uses cmd1 2>&1 | cmd2.
Redirections are processed from left to right before the command runs.
Operator
Meaning
> file
Truncate the file and write stdout to it
>> file
Append stdout to the file
< file
Read stdin from the file
2> file
Redirect stderr to the file
&> file
Redirect stdout and stderr to the file
2>&1
Duplicate stderr onto stdout’s current target
1>&2
Send stdout to stderr
> /dev/null 2>&1
Discard stdout and stderr
<> file
Open the file for reading and writing on fd 0
Order matters: cmd > file 2>&1 redirects stderr to the file because stdout already points there. In cmd 2>&1 > file, stderr still points to the terminal when it is duplicated.
File Descriptors and Process Substitution
Bash can open custom file descriptors with exec:
exec 3> /var/log/app.log opens fd 3 for writing.
echo "msg" >&3 writes to it.
exec 3>&- closes it.
This keeps stdout available for machine-readable data while diagnostics go to a log.
Process substitution exposes command streams as file-like paths:
diff <(sort a.txt) <(sort b.txt) compares two sorted streams.
tee >(gzip > out.gz) copies stdin to stdout and to a compressor.
The exit status of a process-substitution command is not normally propagated through the command consuming its path.
Exit Status, pipefail, and SIGPIPE
By default, a pipeline’s status is the status of its last command, so an upstream failure can be hidden. set -o pipefail makes the pipeline return the rightmost nonzero stage status when any stage fails.
set -e normally exits when such a pipeline returns nonzero, except when it is used as a condition or in another Bash context that exempts failures from errexit.
PIPESTATUS is a Bash array containing each stage’s status. Capture it immediately because another command can replace it:
cmd1 | cmd2
statuses=("${PIPESTATUS[@]}")
If a downstream command exits early, such as head, the producer can receive SIGPIPE and exit with status 141. This is expected in some pipelines and should not automatically be ignored.
Code Example
#!/usr/bin/env bash
set -euo pipefail
log_file="$(mktemp)"
archive_file="$(mktemp)"
trap 'rm -f "$log_file" "$archive_file"' EXIT
# Keep stdout available for data; send diagnostics to fd 3.
exec 3>"$log_file"
# tar's stdout enters the pipe; gzip writes compressed data to archive_file.
tar -czf - /etc/hosts 2>&3 | gzip -c >"$archive_file"
# Process substitution exposes each stream as a file-like path.
if diff -q \
<(printf 'b\na\n' | sort) \
<(printf 'a\nb\n') >/dev/null; then
printf 'streams match\n' >&3
fi
# Capture stage statuses immediately. grep intentionally finds no match.
set +e
printf 'error\n' | grep -i fatal | sort | uniq -c
statuses=("${PIPESTATUS[@]}")
set -e
printf 'grep=%d sort=%d uniq=%d\n' \
"${statuses[0]}" "${statuses[1]}" "${statuses[2]}" >&3
exec 3>&-
Common Pitfalls
Subshell scope: Every stage of a pipeline normally runs in a subshell, so variable assignments do not propagate. Bash’s lastpipe option can change this for the final stage when job control is disabled.
Process-substitution failures: diff <(false) other can succeed because diff does not receive false‘s status.
Truncation and permissions: > truncates the target before the command runs. Redirection-created files use the current umask.
Concurrent writes: PIPE_BUF applies to pipes, not regular files. >> opens files with O_APPEND, but separate writes can still interleave; use flock for coordinated records or read-modify-write operations.
Best Practices
Always quote filenames in redirections: >"$file".
Use set -o pipefail in production Bash scripts and handle expected failures explicitly.
Use |&, &>, process substitution, arrays, PIPESTATUS, lastpipe, and pipefail only in Bash-specific scripts; POSIX sh requires constructs such as 2>&1 |.
Prefer structured output such as --json or --format over parsing human-readable text.
Add timeout around potentially hanging stages and use pv when throughput visibility is useful.
Close custom descriptors and avoid leaking them into long-lived child processes.
For backup pipelines, verify the resulting artifact, monitor freshness and failures, and test restores rather than trusting pipeline status alone.
30
How do you redirect both stdout and stderr to a file in Bash?
KnownRevise
To send both stdout and stderr to the same file in Bash, use command > file 2>&1 (portable) or command &> file (Bash shorthand). The order of redirections matters: 2>&1 must come after the stdout redirection so stderr is pointed at the already-opened file descriptor.
Read full answerHide full answer
Applies toBash 5.x (portable `> file 2>&1` also works in POSIX sh)
How It Works
File descriptor 1 is stdout, 2 is stderr.
> file opens file on fd 1 (truncating it).
2>&1 makes fd 2 a duplicate of fd 1, so both streams write to the same open file description.
&> file is Bash syntax sugar for the same thing; &>> file appends.
> file 2>&1 is the POSIX-portable form and is what you should use in scripts that may run under sh.
Common Variants
Goal
Command
Notes
Overwrite file with both streams
cmd > out.log 2>&1
Portable; truncates
Append both streams
cmd >> out.log 2>&1
Portable
Bash shorthand (overwrite)
cmd &> out.log
Bash-only
Bash shorthand (append)
cmd &>> out.log
Bash-only
Discard both
cmd > /dev/null 2>&1
Classic idiom
Split streams to two files
cmd > out.log 2> err.log
Separate capture
Order Matters (Pitfall)
cmd 2>&1 > file does not do what you want: stderr is first duped to the current stdout (the terminal), then stdout is redirected to the file. stderr still goes to the terminal. Always put 2>&1 after > file.
Production Considerations
Atomicity and buffering: stdout is typically block-buffered when not a TTY, stderr is unbuffered. Interleaving in the file may not reflect true execution order. Use stdbuf -oL or line-buffered logging when ordering matters for incident analysis.
Log rotation: redirecting to a plain file bypasses logrotate/journald. For long-running services, prefer writing to journald or a logging daemon; if you must redirect, use a wrapper like rotatelogs or reopen on SIGHUP.
Permissions: the shell opens the file as the invoking user before execing the command. Ensure the directory is writable and not world-writable (sticky bit issues). Avoid redirecting to paths under /tmp without mktemp.
set -euo pipefail: redirection failures (e.g., permission denied) cause the shell to exit under set -e, which is usually what you want. Note that cmd | tee pipelines need pipefail to surface cmd‘s exit code.
Capturing exit codes: cmd > out.log 2>&1; rc=$? preserves the status. With set -e, a failing command aborts before you can inspect $? unless guarded with if cmd > out.log 2>&1; then ....
Code Example
#!/usr/bin/env bash
set -euo pipefail
log=/var/log/myjob.log
# Append both streams; mkdir -p the dir first in real scripts.
myjob --config /etc/myjob.conf >> "$log" 2>&1
# Capture output while still showing it on the terminal:
myjob --config /etc/myjob.conf 2>&1 | tee -a "$log"
Best Practices
Prefer > file 2>&1 over &> in any script with a #!/bin/sh shebang or that may be sourced by dash/ash.
Quote the filename: > "$log" not > $log.
Use exec >file 2>&1 at the top of a script to redirect all subsequent output of the script itself.
For systemd units, prefer StandardOutput=journal/StandardError=journal rather than shell redirection, so logs integrate with journalctl.
When you need both a file and the terminal, use tee with 2>&1 placed before the pipe.
31
What are here documents and here strings in Bash?
KnownRevise
A here document and a here string are Bash redirections that supply text to a command’s standard input. A here document uses a delimited block of lines, while a here string supplies an expanded word and automatically appends a newline.
Read full answerHide full answer
Applies toBash 5.x
How They Feed Standard Input
Both forms let a command receive data without the caller first creating a separate file:
Here document: Uses << followed by a delimiter. Bash reads subsequent lines until it encounters the delimiter by itself.
Here string: Uses <<< followed by a word. Bash expands that word and supplies the result to the command.
Standard input: The receiving command can consume the entire input or process it incrementally. For example, cat displays everything, while read consumes one line at a time.
Bash may implement these redirections using pipes or temporary files, depending on the implementation and circumstances. Large payloads therefore should not be assumed to remain exclusively in memory.
Syntax and Expansion
Feature
Here document
Here string
Syntax
command <<EOF
command <<<"$text"
Input shape
Zero or more lines
One expanded word or string
Expansion
Performed unless the delimiter is quoted
Performed on the supplied word
Trailing newline
Preserves the body’s line structure
Appends a newline
POSIX support
Supported
Not supported
A here-document delimiter must appear by itself on its own line. Its two common forms are:
Unquoted delimiter, such as <<EOF: Parameter, command, arithmetic, and backslash expansions are performed in the body.
Quoted delimiter, such as <<'EOF': No expansions are performed in the body. This is safer for fixed templates containing $, backticks, or shell-like syntax.
The <<- variant removes leading tab characters from both the body and the delimiter line. It does not remove leading spaces, which makes it useful for indented scripts.
For variable data, <<<"$message" is the conventional form and makes the intent to preserve the exact value explicit. Bash does not perform word splitting or pathname expansion on a here-string word. Quoting the expression itself also prevents expansion: <<<'$HOME' passes the literal text $HOME, whereas <<<"$HOME" expands the variable.
If the supplied value has no trailing newline, the here string causes the command to receive one. If the value already ends with a newline, the here string appends another.
Bash Versus POSIX sh
POSIX sh supports here-documents but not the <<< here-string syntax. A portable equivalent is printf '%s\n' "$text" | command.
The set -o pipefail option is Bash-specific rather than POSIX. Scripts intended for multiple shells should either detect the available shell explicitly or avoid Bash-only features.
Production Best Practices and Pitfalls
Handle errors explicitly: set -euo pipefail catches many unset-variable, failed-command, and failed-pipeline conditions, but it is not a substitute for checking expected failures. Put commands such as grep inside if statements when a nonzero result is valid.
Preserve input faithfully: Use IFS= read -r when whitespace and backslashes must remain unchanged.
Avoid accidental expansion: Prefer a quoted here-document for fixed content. For dynamic values, printf with quoted arguments is often easier to validate than embedding values in an unquoted template.
Do not use eval: Here-documents and here strings do not require evaluating their contents as shell code.
Watch trailing newlines: Command substitution strips trailing newlines from its output, while here-document bodies and here strings have different newline behavior.
Generate configuration safely: For production configuration updates, write to a temporary file with restrictive permissions, validate it, and atomically rename it into place. Keep a backup and a tested rollback procedure.
Protect secrets: Standard input avoids placing the value directly in the target command’s arguments, but secrets can still leak through scripts, shell history, logs, or process inspection. Use an appropriate secret-management mechanism rather than embedding credentials in templates.
32
What does set -euo pipefail do in Bash?
KnownRevise
Strict mode combines Bash’s errexit, nounset, and pipefail options. set -euo pipefail is parsed as enabling -e, enabling -u, and setting -o pipefail.
Read full answerHide full answer
Applies toBash 5.x
Option Semantics
Option
Full name
Behavior
-e
errexit
Exits when a command returns a nonzero status, subject to Bash’s documented exceptions.
-u
nounset
Treats expansion of an unset variable as an error. An empty variable is still valid.
-o pipefail
pipefail
Returns the status of the rightmost command in a pipeline that failed. If every command succeeds, the pipeline returns 0.
These are shell options. Use set +e, set +u, or set +o pipefail to disable them.
Failure and Exit Behavior
With errexit enabled, a failing simple command, pipeline, or list normally terminates the script. Bash suppresses errexit in contexts where failure is intentionally tested, including:
The condition of if, while, or until.
Commands on the left side of && or ||.
Commands prefixed with !.
For example, command -v aws || exit 127 safely handles a missing command because command -v is part of a tested || list.
Without pipefail, a pipeline normally returns the status of its final command. For example:
false | true
returns 0 because true succeeds. With pipefail, it returns 1 because false failed. This is important in deployments where an earlier stage—such as curl—fails but a later stage—such as tee—still succeeds.
errexit does not replace explicit error handling. Expected failures should be checked deliberately, especially when distinguishing “not found” from an operational error.
Bash Versus POSIX sh
pipefail is a Bash feature and is not required by POSIX sh.
[[ ... ]], arrays, and local are also Bash-specific.
POSIX sh supports portable -e and -u, but portable scripts should not rely on Bash-only behavior.
Use #!/usr/bin/env bash when Bash features such as pipefail are required.
Code Example
#!/usr/bin/env bash
set -euo pipefail
usage() {
printf 'Usage: %s LOG_FILE\n' "$0" >&2
exit 64
}
[[ $# -eq 1 ]] || usage
readonly log_file=$1
if [[ ! -r "$log_file" ]]; then
printf 'Cannot read log file: %s\n' "$log_file" >&2
exit 1
fi
for tool in grep awk; do
if ! command -v "$tool" >/dev/null 2>&1; then
printf 'Required tool not found: %s\n' "$tool" >&2
exit 127
fi
done
error_count="$(
grep -E 'ERROR|FATAL' "$log_file" |
awk 'END { print NR + 0 }'
)" || {
status=$?
# grep returns 1 when no lines match, which is not an operational failure.
if [[ $status -eq 1 ]]; then
error_count=0
else
exit "$status"
fi
}
printf 'Found %d ERROR/FATAL records in %s\n' "$error_count" "$log_file"
For a log containing one ERROR and one FATAL record, the output is:
Found 2 ERROR/FATAL records in service.log
The script also demonstrates:
A function for consistent usage errors.
Quoted arguments and variable expansions.
Explicit handling of an expected grep exit status.
Distinct exit codes for usage, input, and missing-tool failures.
Production Best Practices
Quote expansions: Use "$variable" and "$@" to prevent word splitting and glob expansion.
Handle expected failures explicitly: Do not rely on errexit for business logic or recoverable conditions.
Preserve exit status: Cleanup traps and wrappers should not accidentally turn a failed deployment into a successful one.
Use idempotent cleanup: Remove temporary files with mktemp, register cleanup with trap, and design cleanup so it is safe to repeat during incidents.
Do not rely on errexit alone for rollback: A failed command may have already changed remote or cloud resources. Use a transactional deployment strategy and compensating actions.
Monitor meaningful status codes: Distinguish configuration errors, unavailable dependencies, failed health checks, and command failures so alerts and runbooks can respond appropriately.
Test strict mode locally: Run scripts through bash -n and ShellCheck, and test paths where commands fail, pipelines return partial results, and required environment variables are missing.
33
What are the pitfalls of set -e in Bash?
KnownRevise
set -e (errexit) is useful, but it is not a complete error-handling strategy: it exits only in specific contexts and can hide failures in pipelines, functions, and command substitutions.
Read full answerHide full answer
Applies toBash 5.x
How set -e Actually Behaves
set -e makes the shell exit when a simple command, pipeline, list, or compound command returns nonzero. However, it is deliberately disabled in several contexts, including commands used as if/while/until tests, commands in &&/|| lists except the final one, and commands whose status is inverted with !. This means a script can continue after failures that occur inside those constructs.
Common Pitfalls
Pipelines mask failures: without set -o pipefail, only the last command’s status matters, so false | true returns 0.
local hides command-substitution failures: local x=$(false) often returns 0 even though the substitution failed; assign first, then declare local.
Arithmetic can exit unexpectedly: (( count=0 )) returns 1, which can trigger errexit even though no real error occurred.
Expected nonzero statuses: commands such as grep return 1 when no match is found, so they need || true or an explicit conditional.
Functions called in conditionals lose errexit: if a function is invoked as an if test, errexit is disabled inside that function.
Background jobs are not checked automatically: a failing background job does not immediately exit the script; check it with wait.
Comparison of Risky vs Safer Patterns
Risky pattern
Why it is risky
Safer pattern
false | true
Pipeline appears successful
set -o pipefail plus explicit status handling
local x=$(cmd)
local may return 0
x=$(cmd); local x or validate x
(( n=0 ))
Arithmetic status is 1
n=0 or (( n=0 )) || true
grep pattern file
No match returns 1
grep pattern file || true when absence is valid
Production Guidance
Use set -euo pipefail as a baseline, but still add explicit checks for commands whose failure is expected or meaningful. Combine it with an ERR trap for diagnostics, validate inputs, and log failures with context. In production, errexit should be treated as a safety net, not as the primary error-handling design.
34
How do Bash traps handle errors and cleanup?
KnownRevise
Traps let Bash run handlers when errors, signals, or shell termination occur. Use set -euo pipefail for strict failure handling, an ERR trap for diagnostics, and an EXIT trap for cleanup that preserves the original exit status.
Read full answerHide full answer
Applies toBash 5.x on Linux
Error Handling Model
set -e: Causes the shell to exit when an eligible command fails.
set -u: Treats an unset variable as an error.
set -o pipefail: Makes a pipeline fail if any component fails, rather than only considering the final command.
set -E: Makes the ERR trap inherit into functions, command substitutions, and subshells.
ERR trap: Runs when a command failure meets the same conditions as errexit.
EXIT trap: Runs when the shell exits normally, exits explicitly, or exits because of an enabled failure condition.
errexit has exceptions. For example, a failing command used in an if, while, &&, ||, or ! condition does not normally terminate the shell. Important status checks should therefore be written explicitly.
Trap Lifecycle
A trap uses the form:
trap 'command_or_function' SIGNAL
Common signals include:
Signal
Typical meaning
Conventional status
HUP
Terminal disconnected
129
INT
Interrupt, normally Ctrl+C
130
TERM
Graceful termination request
143
EXIT
Shell is exiting
Preserves previous status
Signal handlers should usually call exit with an appropriate status. This allows the EXIT trap to perform cleanup. SIGKILL cannot be trapped, so cleanup cannot be guaranteed after kill -9, a kernel panic, or power loss.
Code Example
This script creates a private temporary directory, reports the failing command, removes the directory, and preserves the original failure status.
#!/usr/bin/env bash
set -Eeuo pipefail
work_dir=""
cleanup() {
local exit_code="$1"
# Prevent recursive EXIT handling and ignore termination signals during cleanup.
trap - EXIT HUP INT TERM
if [[ -n "${work_dir}" && -d "${work_dir}" ]]; then
if ! rm -rf -- "${work_dir}"; then
printf 'Cleanup failed for %s\n' "${work_dir}" >&2
else
printf 'Cleanup: removed %s\n' "${work_dir}" >&2
fi
fi
# Return the status that caused the shell to exit.
exit "${exit_code}"
}
report_error() {
local exit_code="$1"
local failed_command="$2"
printf 'Command failed with status %d: %s\n' \
"${exit_code}" "${failed_command}" >&2
exit "${exit_code}"
}
# "$?" is expanded when the trap executes.
trap 'cleanup "$?"' EXIT
trap 'report_error "$?" "$BASH_COMMAND"' ERR
trap 'exit 129' HUP
trap 'exit 130' INT
trap 'exit 143' TERM
work_dir="$(mktemp -d)"
readonly work_dir
printf 'Working in %s\n' "${work_dir}"
printf 'Simulated failure\n' >&2
false
Example output:
Working in /tmp/tmp.K8m2s7D91
Simulated failure
Command failed with status 1: false
Cleanup: removed /tmp/tmp.K8m2s7D91
The script exits with status 1. Passing $? into the cleanup function preserves the original failure instead of accidentally returning the status of a cleanup command.
Common Pitfalls
Assuming trap catches every error: It does not. Some failures are explicitly tested, and SIGKILL cannot be handled.
Losing the original status: A cleanup command can overwrite $?. Capture the status first and return it explicitly.
Deleting an unsafe path: Never run rm -rf against an unchecked variable. Validate the path and use -- to prevent option-like arguments.
Running cleanup repeatedly: Make cleanup idempotent and disable the EXIT trap inside the handler to prevent recursion.
Ignoring cleanup failures: Log cleanup errors, but preserve the primary exit status so incidents retain the original cause.
Using Bash-only syntax with sh: POSIX sh uses trap cleanup 0 rather than EXIT, and does not support pipefail, [[ ... ]], or Bash arrays.
Production Practices
Keep EXIT cleanup local, fast, and bounded; avoid lengthy network calls inside signal handlers.
Use mktemp -d for isolated temporary storage. Its restrictive permissions also reduce the risk of exposing credentials.
For remote resources, use idempotent cleanup APIs with timeouts and retries. A process killed with SIGKILL can still leave orphaned cloud resources.
Under systemd, place required teardown in ExecStop or ExecStopPost rather than relying exclusively on the application’s shell trap.
In containers, ensure the entrypoint process receives and forwards SIGTERM; a shell that does not forward signals can prevent graceful termination.
Emit cleanup failures to standard error and monitor them. Repeated cleanup errors often indicate permissions, storage, or orchestration problems.
35
How do you trace Bash execution with set -x?
KnownRevise
set -x enables Bash xtrace, which prints each command after variable expansion and immediately before execution. It is useful for locating failures, verifying control flow, and confirming the actual arguments passed to commands.
Read full answerHide full answer
Applies toBash 5.x on Linux
How Bash Xtrace Works
With xtrace enabled, Bash writes diagnostic output to standard error by default. A trace entry typically contains the command and its expanded arguments:
+ printf 'Checking target: %s\n' /srv/app
For each command, Bash performs this sequence:
Expands variables, command substitutions, and other shell expansions.
Writes the expanded command to the trace stream.
Executes the command.
Continues or exits according to options such as set -e and set -o pipefail.
Xtrace does not change a command’s exit status. However, when combined with set -e, the script may stop at the first failing command. set -o pipefail makes a pipeline fail when any component fails, including an earlier command hidden by a successful final command.
Xtrace also does not show branches that were skipped or explain business-level errors. Use it alongside application logs, exit-code checks, and monitoring.
Enabling Xtrace
Use bash -x to trace a script from startup:
bash -x ./deploy-check.sh /srv/app
Enable tracing inside the script with set -x:
set -Eeuo pipefail
set -x
Disable it temporarily with set +x:
set -x
diagnostic_command
set +x
For a production script, tracing can be conditional:
if [[ "${DEBUG:-0}" == "1" ]]; then
set -x
fi
The PS4 variable controls the trace prefix. A useful prefix includes the script name and line number:
PS4='+ ${BASH_SOURCE##*/}:${LINENO}: '
BASH_XTRACEFD can redirect traces to an open file descriptor instead of standard error:
exec 9>/tmp/application.trace
BASH_XTRACEFD=9
set -x
The descriptor must remain open while tracing is active. Disable tracing before closing it:
set +x
exec 9>&-
Reading Trace Output
Trace entries reveal the expanded command, which is often the fastest way to diagnose quoting and path problems. For example:
Run shellcheck ./deploy-check.sh before using the script in CI/CD or production.
Production Practices
Protect secrets: Xtrace can expose passwords, tokens, customer data, and signed URLs. Avoid tracing secret-bearing commands, and remember that disabling xtrace does not remove secrets from process arguments.
Bound the output: Long-running production traces can generate large volumes and fill disk space. Redirect them to a controlled temporary file with an expiration policy.
Reproduce safely: Prefer staging or a read-only diagnostic command. Do not repeatedly execute a destructive script merely to obtain a trace.
Use strict mode correctly: set -Eeuo pipefail provides early failure detection, but set -e has context-specific behavior. Explicitly validate important commands and handle expected failures.
Check portability: set -x is supported by POSIX shells, but PS4, BASH_XTRACEFD, [[ ... ]], and Bash-specific error-trap behavior are not. Use the required #!/usr/bin/env bash shebang when relying on Bash features.
36
What is ShellCheck and why use it?
KnownRevise
ShellCheck is a static analysis tool for shell scripts that parses Bash, POSIX sh, dash, and ksh, then reports syntax errors, quoting bugs, portability issues, and common pitfalls before the script ever runs.
A linter written in Haskell that understands shell grammar rather than just matching text, so it tracks variable scope, quoting context, and command substitutions.
Runs locally from the CLI, inside editors such as VS Code and Vim, and in CI pipelines.
Emits diagnostics with stable codes (SC1xxx–SC3xxx) that link to detailed wiki explanations.
Why Use It
Catches bugs that only surface at runtime, such as unquoted variables causing word splitting or globbing.
Enforces portability when a script targets #!/bin/sh instead of Bash.
Blocks defective scripts from reaching production by failing CI on warnings or errors.
Teaches better shell idioms through its explanations, which is valuable for teams with mixed scripting experience.
What It Catches
Category
Example issue
ShellCheck code
Quoting
rm $file splits on spaces
SC2086
Unused variables
count=0 never read
SC2034
Portability
[[ ]] used in a sh script
SC3010
Deprecated syntax
Backticks instead of $()
SC2006
Logic
== inside single-bracket [ ]
SC3014
Masked failures
Ignoring exit status of a command
SC2181
Production Usage
Run it in CI with a severity threshold, e.g. shellcheck -S warning scripts/*.sh, and fail the build on findings.
Use inline directives such as # shellcheck disable=SC2086 only with a comment explaining why, never blanket-disable a rule file-wide.
Pair it with set -euo pipefail, trap cleanup handlers, and actual execution tests; ShellCheck finds static issues but cannot prove runtime behavior.
Keep scripts ShellCheck-clean so reviewers focus on logic rather than style.
Best Practices and Pitfalls
Treat warnings as errors in CI rather than letting them accumulate into noise.
Fix the root cause (usually missing quotes) instead of suppressing the diagnostic.
Re-run ShellCheck after refactoring, since moving code between functions changes quoting and scoping analysis.
Remember that ShellCheck cannot detect issues requiring runtime data, such as missing files or wrong permissions, so execution tests remain necessary.
37
How do you get substring, length, and replace in Bash?
KnownRevise
Bash performs these operations with parameter expansion, so substring extraction, length calculation, and replacement work directly on string variables without starting external commands.
Read full answerHide full answer
Applies toBash 5.x
Core Parameter Expansions
Operation
Syntax
Behavior
Length
${#var}
Returns the number of characters according to the current locale. In the C locale, each byte counts as one character.
Substring
${var:offset:length}
Returns length characters beginning at zero-based offset. ${var:offset} returns the remainder.
First replacement
${var/pattern/replacement}
Replaces the first glob-pattern match.
All replacements
${var//pattern/replacement}
Replaces every glob-pattern match.
Beginning match
${var/#pattern/replacement}
Replaces only a match at the beginning.
Ending match
${var/%pattern/replacement}
Replaces only a match at the end.
Patterns are shell glob patterns, not regular expressions. An empty replacement deletes the matched text. A negative substring offset counts from the end and must be separated from the colon, such as ${var: -3}, to avoid ambiguity with default-value syntax.
Case modifiers are also available: ${var^} uppercases the first character, ${var^^} uppercases all characters, while ${var,} and ${var,,} lowercase the first character or all characters. These modifiers require Bash 4 or later.
Scripting and Error Handling
Bash variables are untyped and created by assignment. Quote assignments and expansions:
name="production server"
printf '%s\n' "$name"
if and [[ ... ]] handle conditional tests, for and while implement loops, and functions group reusable logic. Positional arguments are available as $1, $2, and "$@".
Every command returns an exit status: zero indicates success and nonzero indicates failure. set -euo pipefail enables early termination on command failures, unset-variable checks, and pipeline-failure detection. It improves scripts but does not replace explicit checks for expected failures.
Use grep for selecting matching text, sed for stream transformations, and awk for record or field processing. These tools are appropriate for files and structured input; parameter expansion is more efficient for edits to one in-memory string.
Bash and POSIX sh
${#var} is POSIX, but Bash substring slicing and pattern replacement are extensions. In strict POSIX sh, use alternatives such as expr substr "$var" 2 5, cut -c2-6, or sed. Debian and Ubuntu normally provide dash as /bin/sh, so these Bash-only expansions fail in a #!/bin/sh script.
Code Example
#!/usr/bin/env bash
set -euo pipefail
s="${1:-production-server-42}"
echo "length: ${#s}" # 20
echo "substring: ${s:11:7}" # server-
echo "tail: ${s: -2}" # 42
echo "first replace: ${s/server/node}" # production-node-42
echo "all replace: ${s//-/_}" # production_server_42
echo "prefix replace: ${s/#production/staging}" # staging-server-42
echo "delete: ${s/-42/}" # production-server
echo "upper: ${s^^}" # PRODUCTION-SERVER-42
slugify() {
local input="$1"
echo "${input//-/_}"
}
slugify "$s"
if [[ "$s" == *-42 ]]; then
echo "has numeric suffix"
fi
IFS='-' read -r -a parts <<< "$s"
for part in "${parts[@]}"; do
echo "part: $part"
done
if grep -q -- 'server' <<< "$s"; then
echo "contains server"
else
echo "server not found" >&2
exit 1
fi
Running it without arguments produces:
length: 20
substring: server-
tail: 42
first replace: production-node-42
all replace: production_server_42
prefix replace: staging-server-42
delete: production-server
upper: PRODUCTION-SERVER-42
production_server_42
has numeric suffix
part: production
part: server
part: 42
contains server
Production Best Practices
Quote every expansion. Unquoted results can undergo word splitting and pathname expansion.
Use set -euo pipefail in operational scripts, but handle expected failures explicitly with if, case, or &&/||.
Prefer parameter expansion for simple single-string edits; it avoids subprocess overhead in loops.
Since Bash 5.2, the patsub_replacement option is enabled by default. An unquoted & in a replacement is then replaced by the matched text. Use shopt -u patsub_replacement when literal ampersands must not receive that substitution.
Validate scripts with bash -n script.sh and shellcheck script.sh before deployment.
38
What is parameter expansion in Bash?
KnownRevise
Parameter expansion is Bash’s mechanism for reading, transforming, and substituting variable values using the ${...} syntax, including forms like ${var:-default} that supply fallbacks when a variable is unset or empty.
Read full answerHide full answer
Applies toBash 5.x (forms marked POSIX also work in dash/POSIX sh)
How It Works
When Bash parses a word containing ${...}, it expands the parameter before executing the command. The colon (:) in forms like :-, :=, :+, :? tests for both “unset” and “null” (empty string); without the colon, only “unset” is tested. Because expansion happens during word parsing, quoting matters: an unquoted expansion is subject to word splitting and globbing, while "${var}" preserves the value as a single word.
Common Forms
Form
Meaning
POSIX?
${var}
Value of var
Yes
${var:-default}
default if var unset or null
Yes
${var-default}
default only if var unset
Yes
${var:=default}
Assign default to var if unset/null, then expand
Yes
${var:+replacement}
replacement if var set and non-null
Yes
${var:?message}
Print error and exit if unset/null
Yes
${#var}
Length of value
Yes
${var:offset:length}
Substring
No
${var#pat} / ${var##pat}
Remove shortest/longest prefix matching pat
Yes
${var%pat} / ${var%%pat}
Remove shortest/longest suffix matching pat
Yes
${var/old/new} / ${var//old/new}
Replace first/all matches
No
${var^^} / ${var,,}
Uppercase/lowercase entire value
No (Bash 4+)
${!var}
Indirect expansion (value of variable named by var)
No
Production Usage
In scripts, ${var:?} is the idiomatic way to enforce required inputs: : "${DB_HOST:?DB_HOST is required}" fails fast with a clear message instead of running with an empty value. ${var:-} is commonly used to normalize optional flags, e.g. debug=${DEBUG:-0}. Prefix/suffix removal (${file%.*}, ${path##*/}) avoids spawning basename/dirname in tight loops and is faster and safer than parsing with external tools.
Best Practices and Pitfalls
Always quote expansions: "${var}", not $var, to prevent word splitting and globbing.
Prefer ${var:-} over ${var-} when an empty string should also trigger the default.
Use ${var:?} for required values; it exits non-zero under set -e and prints to stderr.
Avoid ${var:=...} on read-only or positional parameters; it errors on $1 etc.
Remember set -u treats unset variables as errors, so ${var:-} is the safe idiom for optional values.
POSIX scripts cannot use substring, substitution, case conversion, or indirect expansion; stick to the POSIX subset or require Bash explicitly with #!/usr/bin/env bash.
39
How does grep use regular expressions in Bash scripts?
KnownRevise
Extended regular expressions (ERE) let grep select lines containing specific text patterns. In Bash, use grep -E for ERE syntax, quote the pattern to prevent shell expansion, and handle grep’s exit status explicitly when automation must distinguish “no match” from an actual error.
Read full answerHide full answer
Applies toBash 5.x, GNU grep 3.x, and POSIX ERE for portable scripts
Choosing a Regular Expression Mode
Option
Pattern language
Example
Use case
grep -G
Basic regular expression (BRE)
^(ERROR):
Default in GNU grep; portable BRE grouping
grep -E
Extended regular expression (ERE)
^(ERROR|FATAL):
Portable grouping, alternation and repetition
grep -F
Fixed string
grep -F 'connection refused'
Literal text without regex interpretation
grep -P
Perl-compatible regex (PCRE)
grep -P 'ERROR(?=()'
GNU extension for lookarounds and other PCRE features
ERE is usually the most readable choice for Bash scripts. It supports alternation (|), grouping (()), repetition (*, +, ?, {m,n}) and anchors (^, $). Prefer portable classes such as [[:digit:]], [[:space:]] and [[:alpha:]].
POSIX BRE has no portable alternation operator; GNU’s \| extension should not be used in strictly portable scripts. GNU grep -P is also non-POSIX and may not be available in every build, so use it only when the target systems are known to support it.
How Matching Works
grep is an external utility, not a Bash builtin. The shell first performs expansions, quoting and word splitting; grep then compiles the pattern and scans its input line by line. A match prints the entire line unless another output mode is selected.
Useful options include:
-n: Prefix output with line numbers.
-i: Perform case-insensitive matching.
-c: Count matching lines.
-v: Select nonmatching lines.
-r: Search files recursively.
-I: Skip binary files.
-q: Suppress output and return only a status.
ERE matching is line-oriented. ^ and $ anchor to the beginning and end of a line, not to the complete file. Scripts should rely only on documented matching behavior rather than GNU grep’s internal choice of matching engine.
Exit Status and Error Handling
grep returns:
0: At least one line matched.
1: No lines matched.
2: An error occurred, such as an unreadable file or invalid pattern.
This distinction matters with set -euo pipefail. A grep exit status of 1 can fail an entire pipeline even though it only means that nothing matched. Also, grep -q may stop reading early, causing an upstream producer to receive SIGPIPE; pipefail can report that as a pipeline failure.
Code Example
#!/usr/bin/env bash
set -euo pipefail
if (( $# < 2 )); then
printf 'Usage: %s REGEX LOG_FILE[...]\n' "${0##*/}" >&2
exit 64
fi
pattern=$1
shift
search_log() {
local file=$1
local regex=$2
local status=0
grep -nE -e "$regex" -- "$file" || status=$?
return "$status"
}
overall_status=0
for log_file in "$@"; do
if [[ ! -r $log_file ]]; then
printf 'Cannot read log file: %s\n' "$log_file" >&2
overall_status=2
continue
fi
grep_status=0
search_log "$log_file" "$pattern" || grep_status=$?
case "$grep_status" in
0)
;;
1)
printf 'No matching entries in %s\n' "$log_file" >&2
if (( overall_status == 0 )); then
overall_status=1
fi
;;
*)
printf 'grep failed for %s with status %d\n' \
"$log_file" "$grep_status" >&2
overall_status=$grep_status
;;
esac
done
exit "$overall_status"
42:ERROR: database connection timed out
87:FATAL: worker process exited unexpectedly
The invocation quotes the regex so the shell does not interpret its metacharacters. Inside the script, grep -e "$pattern" also permits a pattern beginning with -, while -- protects the filename from being interpreted as an option.
Production Best Practices
Use grep -F for literal text; it avoids regex semantics and often avoids regex-processing overhead.
Use sed -E for regex-based substitutions and awk for field, record and transformation logic; grep primarily selects lines.
Set LC_ALL=C when byte-oriented matching must remain stable across locales.
Never pass user-supplied patterns to eval; grep treats the value as regex data, not as shell commands.
Restrict untrusted pattern length and complexity, especially for PCRE features involving backtracking.
Decide explicitly whether “no match” should produce success or failure in CI, monitoring and incident-response scripts.
This script uses non-POSIX features such as [[, ((, local and pipefail; replace them and explicitly capture pipeline statuses when targeting POSIX sh.
40
How do you find and replace text in files using sed?
KnownRevise
sed is a stream editor that transforms text line by line; its s (substitute) command is the standard way to find and replace text in files or pipelines. By default it prints the result to stdout, so it never modifies a file unless you explicitly request in-place editing.
Read full answerHide full answer
Applies toGNU sed 4.x on Linux; BSD sed (macOS) differs in `-i` syntax.
How It Works
Basic form: sed 's/pattern/replacement/flags' file
The s command takes a delimiter (commonly /), a regex pattern, a replacement, and optional flags.
Without -i, output goes to stdout and the original file is untouched — safe for previewing.
Addresses (line numbers or ranges) can restrict where substitution happens: sed '2,5s/foo/bar/' file.
sed exits 0 even when no match is found, so do not rely on its exit status to detect matches; use grep -q first if needed.
Common Patterns
Global replace on every line: sed 's/foo/bar/g'
Case-insensitive (GNU sed): sed 's/foo/bar/gi'
Alternate delimiter for paths/URLs: sed 's|/var/log|/var/log/archive|g'
Capture groups with extended regex: sed -E 's/^([A-Za-z]+): (.*)$/\2 (\1)/'
Replace only on lines matching a condition: sed '/ERROR/s/foo/bar/'
Delete matching lines instead: sed '/pattern/d'
In-Place Editing
GNU sed: sed -i 's/foo/bar/g' file.txt
With backup: sed -i.bak 's/foo/bar/g' file.txt
GNU sed writes a temp file and renames it into place, preserving mode but changing ownership to the user running sed. In production, running as root or the file owner avoids surprising ownership changes.
BSD/macOS requires a backup argument: sed -i '' 's/foo/bar/g' file.txt (empty string means no backup).
For critical files, prefer writing to a temp file and mv atomically, or keep a .bak.
Portability: GNU vs BSD sed
Feature
GNU sed
BSD sed (macOS)
In-place, no backup
sed -i 's/.../.../'
sed -i '' 's/.../.../'
Extended regex
-E (also -r)
-E
Case-insensitive flag
I
not supported
n in replacement
supported
limited
Use -E for extended regex on both; avoid GNU-only flags in portable scripts.
Code Example
#!/usr/bin/env bash
set -euo pipefail
# Preview substitution (no file change)
sed 's/localhost/127.0.0.1/g' config.ini
# In-place global replace with backup (GNU sed)
sed -i.bak 's/old_value/new_value/g' config.ini
# Replace a path using an alternate delimiter
sed -i 's|/opt/app/v1|/opt/app/v2|g' deploy.conf
# Capture groups: "key: value" -> "value (key)"
sed -E 's/^([A-Za-z_]+): (.*)$/\2 (\1)/' env.txt
Sample input env.txt:
HOST: localhost
PORT: 8080
Sample output:
localhost (HOST)
8080 (PORT)
Best Practices and Pitfalls
Always quote the script to prevent shell expansion of $, *, or backticks.
Escape &, </code>, and the delimiter in the replacement; <code>& means “the whole match.”
Test without -i first, then add -i once output looks correct.
sed is line-oriented; it cannot easily handle multi-line or binary data — use perl or awk for those.
Under set -o pipefail, a failing upstream command in cmd | sed ... still surfaces the error.
41
How can awk extract and transform tabular text fields?
KnownRevise
awk is a stream-oriented text-processing tool that splits each record into fields and lets you filter, select, aggregate, or reformat those fields using patterns and actions. It is well suited to logs, inventories, metrics, and other line-oriented operational data.
Read full answerHide full answer
Applies toGNU awk 5.x and Bash 5.x on Linux
Processing Model
awk processes a file one record at a time:
It reads a record, normally one line.
It splits the record into fields using FS.
It evaluates each pattern-action rule.
It executes the action for matching records.
Important built-in variables include:
$0: The complete current record.
$1, $2, …: Individual fields.
NF: Number of fields in the current record.
NR: Current record number across all input files.
FNR: Current record number within the current file.
FS: Input field separator.
OFS: Output field separator used by print.
ORS: Output record separator, normally a newline.
With the default FS, runs of spaces and tabs separate fields. Set FS="\t" for tab-separated data or use -F ',' for simple comma-separated data.
Common Column Operations
Goal
awk expression
Print one column
awk -F 't' '{ print $3 }' data.tsv
Filter by a column
awk -F 't' '$3 == "ERROR" { print }' data.tsv
Select multiple columns
awk -F 't' '{ print $2, $4 }' data.tsv
Aggregate a numeric column
awk -F 't' 'NR > 1 { total += $4 } END { print total + 0 }' data.tsv
awk -F 't' 'BEGINFILE { total = 0 } { total += $4 } ENDFILE { print FILENAME, total }' data.tsv
The BEGINFILE and ENDFILE features in the final example are GNU awk extensions.
Code Example
The following Bash script reads tab-separated service data, selects the service and duration columns, calculates totals, and rejects malformed records.
#!/usr/bin/env bash
set -euo pipefail
export LC_ALL=C
input_file=${1:?"Usage: $0 INPUT.tsv"}
if [[ ! -f "$input_file" || ! -r "$input_file" ]]; then
printf 'Input is not a readable file: %s\n' "$input_file" >&2
exit 1
fi
# Prefix relative paths so filenames cannot be interpreted as options.
case "$input_file" in
/*) ;;
*) input_file="./$input_file" ;;
esac
awk '
BEGIN {
FS = "\t"
OFS = "\t"
}
NR == 1 {
expected = NF
if (expected != 4) {
printf "Header has %d fields; expected 4\n", expected > "/dev/stderr"
bad = 1
}
next
}
{
if (NF != expected) {
printf "Record %d has %d fields; expected %d\n", NR, NF, expected > "/dev/stderr"
bad = 1
next
}
# Remove a Windows CRLF line ending from the final field.
sub(/\r$/, "", $4)
if ($4 !~ /^[0-9]+$/) {
printf "Record %d has an invalid duration: %s\n", NR, $4 > "/dev/stderr"
bad = 1
next
}
printf "%s\t%s\n", $2, $4
total += $4
if ($3 == "ERROR") {
errors++
}
}
END {
if (bad) {
exit 2
}
printf "TOTAL_DURATION_MS\t%.0f\n", total
printf "ERROR_ROWS\t%d\n", errors
}
' "$input_file"
For this input:
timestamp service status duration_ms
2025-01-10T10:00:00Z api OK 120
2025-01-10T10:01:00Z worker ERROR 85
2025-01-10T10:02:00Z api OK 430
The output is:
api 120
worker 85
api 430
TOTAL_DURATION_MS 635
ERROR_ROWS 1
Failure Modes and Security
Incorrect separator: Verify the actual delimiter before processing. Using -F ',' is not safe for RFC 4180 CSV containing quoted commas; use a proper CSV parser such as Python’s csv module or Miller for that format.
Malformed records: Check NF and validate numeric fields. Exit nonzero rather than silently converting invalid values to zero.
Whitespace in values: Default whitespace-based FS cannot distinguish a tab from spaces or preserve empty fields reliably.
Memory growth: awk streams records, but associative arrays consume memory for every unique key.
Untrusted input: Quote filenames and never concatenate untrusted values directly into the awk program text.
Partial artifacts: Write to a temporary file and publish it only after awk exits successfully. This prevents consumers from reading incomplete output.
Production and Portability
For scheduled jobs and CI/CD pipelines, log malformed-record counts, monitor nonzero exit statuses, and preserve the raw input for forensic replay under the required retention policy. Pin Bash and GNU awk versions in container images and regression-test scripts after upgrades.
The script uses Bash-specific [[ ... ]] and set -o pipefail. In POSIX sh, replace [[ ... ]] with [ ... ] and use set -eu; POSIX sh does not support pipefail. The awk logic is mostly POSIX-compatible, except for /dev/stderr, which is not a POSIX file path.
42
How do sort, uniq, and wc transform and count text lines?
KnownRevise
sort, uniq, and wc form a text-processing pipeline: sort orders lines, uniq collapses adjacent duplicates or counts them, and wc counts lines, words, or bytes. Global deduplication and accurate frequency counts require sorting the input before passing it to uniq.
Read full answerHide full answer
Applies toGNU coreutils 9.x on Linux; Bash 5.x or POSIX sh as the invoking shell
How the Tools Work
Tool
Primary behavior
Important detail
sort
Orders lines lexicographically by default
Duplicate lines remain unless -u is used
uniq
Collapses or reports adjacent duplicate lines
It does not find duplicates separated by other lines
wc
Counts lines, words, and bytes
-l counts newline characters, not necessarily logical records
A typical global deduplication pipeline is sort data.txt | uniq. The command sort -u data.txt provides the same unique result more directly. To count occurrences, use sort data.txt | uniq -c | sort -k1,1nr: the first sort makes equal lines adjacent, uniq -c counts each run, and the final sort orders the results by descending numeric frequency.
Common Command Patterns
Sort numerically: sort -n values.txt
Sort in reverse order: sort -r values.txt
Sort by a numeric field in simple, unquoted CSV: sort -t ',' -k2,2nr records.csv. Use a CSV-aware parser for quoted fields, embedded commas, or other RFC-style CSV features.
Deduplicate globally: sort -u data.txt
Count each unique line: sort data.txt | uniq -c
Show only repeated lines: sort data.txt | uniq -d
Show lines occurring once: sort data.txt | uniq -u
Count lines, words, bytes, or characters: wc -l data.txt, wc -w data.txt, wc -c data.txt, or wc -m data.txt
Count without printing the filename: wc -l < data.txt
Check whether input is sorted: sort -c data.txt
For locale-independent byte-value ordering, set LC_ALL=C. For locale-aware ordering, use an installed UTF-8 locale such as en_US.UTF-8.
Operational and Security Considerations
Pipeline failures: In Bash, set -o pipefail makes a pipeline fail if sort, uniq, or wc fails, rather than reporting only the final command’s status.
Temporary storage: For large inputs, GNU sort can use temporary files. Ensure TMPDIR points to storage with sufficient space and appropriate permissions.
Missing final newline: wc -l counts newline characters, so a final unterminated record is not included. Use awk 'END { print NR }' when logical-record counting is required.
Line endings: CRLF files retain \r as part of each record, so CRLF and LF versions of the same text do not compare as identical. Normalize line endings deliberately.
Large files: uniq and wc are streaming operations, while sort may consume substantial I/O and temporary disk space.
Automation: Check command and pipeline exit statuses, and emit logs or metrics before using results to trigger deployments, cleanup, or incident workflows.
Safe arguments: Quote filenames and use -- before user-controlled filenames to prevent them from being interpreted as options.
Bash and POSIX sh Portability
The utilities and their basic options are available in POSIX environments. Bash 5.x additionally provides shell features such as [[ ... ]], arithmetic evaluation with (( ... )), and set -o pipefail. POSIX sh instead uses test, arithmetic expansion, and set -eu, but has no direct equivalent of pipefail; detecting an intermediate pipeline failure requires another mechanism.
43
How do you use cut, tr, and paste in Bash?
KnownRevise
cut, tr, and paste are text-streaming utilities: cut extracts fields or characters, tr transforms or removes individual characters, and paste combines lines from multiple files or streams. They are useful for lightweight shell automation but are not CSV- or JSON-aware parsers.
Read full answerHide full answer
Applies toGNU coreutils 9.x on Linux and Bash 5.x
Command Roles
Command
Primary purpose
Common options
cut
Select fields or character positions
-d, -f, -c, -s
tr
Translate, delete, or squeeze characters
-d, -s, -c
paste
Merge corresponding lines from multiple inputs
-d, -s
These commands read from files, accept - for standard input, and normally write to standard output, making them easy to compose in pipelines.
Using cut
cut selects columns separated by a delimiter. Without -d, it uses tabs.
Translation: tr SET1 SET2 maps characters in SET1 to corresponding characters in SET2.
Deletion: tr -d SET removes every character in SET.
Squeezing: tr -s SET replaces repeated characters with one occurrence.
Complement: tr -c SET acts on characters outside SET.
If SET2 is shorter than SET1, the last character of SET2 is repeated; extra characters in SET2 are ignored. tr cannot replace substrings directly; use sed, parameter expansion, or a structured parser for that.
Using paste
paste reads corresponding lines from two or more inputs and writes them horizontally.
Use set -euo pipefail to terminate on an unset variable reference, an unhandled command failure, or a failed pipeline stage. It is not a universal error handler: a command whose status is tested in an if or while condition, or used in an && or || list, may not trigger -e.
Define reusable operations as functions. Pass arguments through $1, $2, and so on, declare them with local, and quote every expansion.
Use if for expected failures and for loops for repeated work. When handling an expected failure, capture $? immediately in the else branch if the specific status is needed.
Quote file names and variables, especially when paths may contain spaces.
Use LC_ALL=C when byte-oriented behavior must remain consistent across hosts.
Create temporary files under a restrictive mktemp directory and clean them with a trap.
Do not put credentials directly in command arguments because they may appear in process listings or shell history.
Use awk, jq, or a dedicated CSV parser instead of these utilities for quoted CSV, JSON, XML, or key-based joins.
For long-running log processing, stream records through pipelines and monitor failures rather than loading entire files into memory.
The cut --complement option is a GNU coreutils extension and is unavailable in POSIX-only cut implementations, regardless of whether Bash or POSIX sh invokes it.
44
How does xargs work compared to a shell loop?
KnownRevise
xargs reads items from standard input and passes them as arguments to another command, batching them efficiently and optionally running commands in parallel. A shell loop processes one item at a time and is better when each item needs conditional logic, variable handling, or shell builtins.
Read full answerHide full answer
Applies toBash 5.x and GNU xargs (findutils) on Linux
How xargs Works
xargs collects input, usually separated by whitespace or newlines, and invokes the target command with as many arguments as the system allows.
Use find ... -print0 | xargs -0 so filenames containing spaces, quotes, or newlines are handled safely.
Use -r (--no-run-if-empty) to avoid running the command when there is no input.
Use -n to limit arguments per invocation and -P to run multiple commands in parallel.
Use -I {} only when each item must be placed in a specific position; it usually runs one command per item and loses batching.
How a Loop Differs
A while read loop gives full control over each item, including conditionals, retries, logging, and shell variables.
Loops are safer for complex per-item logic but invoke the command once per item unless you deliberately batch inside the loop.
Loops can use shell functions and builtins directly; xargs generally needs an external command or sh -c.
xargs automatically respects ARG_MAX; loops avoid argument-list limits by invoking commands one at a time.
Comparison
Aspect
xargs
Shell loop
Invocation count
Batches many items
Usually one per item
Parallelism
Built-in with -P
Manual with background jobs
Complex logic
Limited
Full shell logic
Safe filenames
Requires -0
Requires read -d ''
Exit status
Aggregated across invocations
Controlled per item
Failure Modes and Pitfalls
Default xargs input splitting breaks on spaces and newlines; always pair find -print0 with xargs -0.
Without -r, an empty input can cause commands such as rm to run with no file arguments.
xargs -I {} is convenient but can be much slower because it disables batching.
In production, prefer xargs -P for CPU-bound or I/O-bound batch work, but cap parallelism to avoid overwhelming disks, APIs, or databases.
For destructive operations, log the generated command first with xargs -t or use -- before filenames.
Best Practices
Use find -print0 | xargs -0 for any filename processing.
Add -r to prevent empty-input command execution.
Use -n and -P deliberately, and test with xargs -t before destructive runs.
Use a loop when per-item decisions, retries, or shell functions are needed.
Prefer while IFS= read -r -d '' over parsing find output in a for loop.
45
How do run commands in parallel with Bash?
KnownRevise
Parallel execution in Bash means starting independent commands in the background and then synchronizing with wait. Bash provides concurrent process execution; actual simultaneous CPU use depends on the number of available cores and the workload.
Read full answerHide full answer
Applies toBash 5.x on Linux; GNU coreutils 9.x for `xargs -P`
How Bash Runs Background Commands
Append & to start a command asynchronously:
command_one &
command_two &
wait
When Bash encounters &, it:
Starts the command in a separate process or subshell.
Stores its process ID in $!.
Continues executing the parent script without waiting.
Adds the process to the shell’s job table.
For example:
curl --fail https://api.example.com/health &
pid=$!
# Perform other work while the request is running.
wait "$pid"
The value of $! must be captured immediately after launching the background command because later background commands overwrite it.
Waiting and Exit Statuses
wait "$pid" waits for one specific background process and returns that process’s exit status. This is important when the script must detect failures:
if wait "$pid"; then
printf 'Job completed successfully\n'
else
status=$?
printf 'Job failed with status %d\n' "$status" >&2
fi
By contrast, wait without arguments waits for all background jobs but normally returns zero, so it can hide individual job failures. Under set -e, a failing wait can also terminate the script unless it is used in a conditional such as if wait "$pid"; then.
Bash also provides:
wait -n: Waits for the next background job to finish (Bash 4.3+).
wait -n -p finished_pid: Bash 5.1+ also records the finished job’s process ID.
kill "$pid": Requests termination of a specific background process.
Bash and POSIX sh Differences
POSIX sh supports background commands with & and wait, but it does not provide Bash arrays or wait -n. POSIX does not standardise local (dash, ash and ksh implement it as an extension), and job control is only active in interactive shells — a script normally runs without it. Because the example below relies on arrays and per-PID status handling, it should run with #!/usr/bin/env bash rather than #!/bin/sh.
Code Example
Parallel health checks with per-job exit status — the script records every process ID and reports individual failures:
#!/usr/bin/env bash
set -euo pipefail
check_service() {
local name=$1
local url=$2
local attempt
for attempt in 1 2 3; do
if curl --fail --silent --show-error \
--connect-timeout 5 --max-time 10 "$url" >/dev/null; then
printf '%s: healthy\n' "$name"
return 0
fi
sleep $((attempt * 2))
done
printf '%s: failed after retries\n' "$name" >&2
return 1
}
declare -a pids=()
declare -a names=()
for name in api worker db; do
check_service "$name" "https://${name}.example.com/health" &
pids+=("$!")
names+=("$name")
done
failed=0
for index in "${!pids[@]}"; do
if wait "${pids[$index]}"; then
printf '%s: completed successfully\n' "${names[$index]}"
else
status=$?
printf '%s: failed with exit status %d\n' \
"${names[$index]}" "$status" >&2
failed=1
fi
done
if (( failed != 0 )); then
exit 1
fi
Output ordering is nondeterministic because the jobs finish independently.
For a large workload, bound the concurrency with GNU xargs -P (at most four curl processes at a time):
xargs exits with 123 if an invocation exits with status 1-125, but it does not offer the same convenient per-item status collection as storing Bash PIDs.
Production Best Practices
Launching an unbounded number of jobs can overload CPUs, APIs, databases or network connections; cap concurrency (xargs -P, a semaphore in coproc, or a job pool).
Apply explicit timeouts, bounded retries and exponential backoff to every remote call.
Write each job’s output to a separate file to prevent interleaved or corrupted logs.
Make retried operations idempotent, especially during deployments or incident recovery.
Use trap handlers to terminate child processes when the parent receives a termination signal.
Avoid eval; validate inputs and quote every variable expansion.
Monitor duration, failure rate and resource saturation rather than treating parallel completion alone as success.
46
How do you run a Bash command in the background and wait?
KnownRevise
Append & to start a background process, save $! if you need its process ID, and use the wait builtin to block until it finishes. wait "$pid" waits for a specific child, while wait without arguments waits for all active child processes.
Read full answerHide full answer
Applies toBash 5.x on Linux
How Bash Tracks Background Jobs
When Bash executes a command containing &, it starts the command asynchronously and immediately continues with the next statement:
long_running_command &
pid=$!
&: Places the command in the background.
$!: Contains the process ID of the most recently started background job.
wait "$pid": Blocks the current shell until that child process exits.
$?: Contains the exit status returned by wait.
For example:
sleep 5 &
pid=$!
echo "Waiting for PID $pid"
wait "$pid"
echo "The job exited with status $?"
Waiting for Multiple Jobs
Bash provides several forms of wait:
Form
Behavior
wait "$pid"
Waits for the specified child process.
wait "$pid1" "$pid2"
Waits for both processes; the returned status is from the last process waited for.
wait
Waits for all active child processes and returns zero after they complete.
wait -n
Waits for the next available background job to finish.
When every job must succeed, check each status separately. A multi-PID wait can hide an earlier failure if the final job succeeds.
job_one &
pid_one=$!
job_two &
pid_two=$!
failed=0
wait "$pid_one" || failed=$?
wait "$pid_two" || failed=$?
if (( failed != 0 )); then
echo "A background job failed" >&2
exit "$failed"
fi
Error Handling with set -euo pipefail
set -euo pipefail is a common foundation for production Bash scripts:
-e: Exits when an unhandled command fails, although commands used as conditions are exempt.
-u: Treats use of an unset variable as an error.
pipefail: Makes a pipeline fail when any component fails.
A background command does not immediately trigger errexit in the parent shell because the parent continues executing. The failure appears when wait returns the child’s nonzero status.
Use wait inside an if condition when you need to handle that failure. Calling wait "$pid" as a standalone command under set -e can terminate the script before custom cleanup or reporting runs.
Code Example
This script starts a job, redirects its output, waits for it, and preserves the job’s exit status:
#!/usr/bin/env bash
set -euo pipefail
log_dir="$(mktemp -d)"
trap 'rm -rf "$log_dir"' EXIT
sleep 2 >"$log_dir/job.log" 2>&1 &
pid=$!
printf 'Started background job with PID %s\n' "$pid"
if wait "$pid"; then
printf 'Job completed successfully\n'
else
status=$?
printf 'Job failed with exit status %d\n' "$status" >&2
exit "$status"
fi
Expected output, with a variable PID:
Started background job with PID 12345
Job completed successfully
Production Best Practices
Redirect output: Background output can interleave with the parent script. Write it to a log file or another managed destination.
Quote expansions: Use "$pid" and "$log_dir"; unquoted values can cause word splitting or pathname expansion.
Avoid eval: Pass command arguments directly rather than constructing shell code from untrusted input.
Use mktemp: Create temporary directories securely instead of predicting filenames in /tmp.
Add operational ownership: wait provides no progress reporting, health checks, or timeout. For long-running production work, use systemd, a Kubernetes Job, or a CI runner.
Understand shell lifetime: wait works only for children of the current Bash process. Exiting the shell can terminate or detach from jobs depending on their signal handling and environment.
Monitor failures: A successful wait means only that the process exited; inspect logs, metrics, and exit codes to determine whether the job actually completed its intended work.
47
How do you schedule a Bash script with cron?
KnownRevise
A cron job is scheduled by adding an entry to a user’s crontab. For a Bash script, use its absolute path, configure explicit error handling and logging, and account for cron’s restricted environment.
Read full answerHide full answer
Applies toDebian `cron` on Ubuntu 22.04/24.04, Cronie on RHEL 9, and Bash 5.x
How Cron Schedules Jobs
A crontab entry contains five time fields followed by the command: minute hour day-of-month month day-of-week command.
Common shorthand schedules include @hourly, @daily, @weekly, @monthly, and @reboot. If both day-of-month and day-of-week are restricted, standard cron behavior is to run the job when either field matches.
Use crontab -e to edit the current user’s schedule and crontab -l to inspect it. Schedule scripts as a dedicated non-login service account rather than as root.
Cron Execution Environment
Cron starts jobs with a minimal environment and normally executes the command through /bin/sh.
Use an absolute path for the script and important external commands.
Do not depend on the current working directory.
Set or override PATH inside the script when necessary.
Redirect both standard output and standard error to a log file.
Treat a non-zero exit status as a failed job and monitor it externally; cron does not provide production alerting by itself.
On Vixie-style implementations, including Debian cron and Cronie, % is special in a crontab command. Escape it as \% when it must be passed literally.
Production Behavior
The cron daemon’s service name differs by distribution:
Distribution
Service
Commands
Ubuntu/Debian
cron
sudo systemctl enable --now cron
RHEL-family
crond
sudo systemctl enable --now crond
Production considerations include:
Time zone: Schedules use the daemon host’s local time by default. Debian cron and Cronie support CRON_TZ for a per-crontab override; daylight-saving changes can cause local times to be skipped or repeated.
Missed execution: Ordinary crontab jobs missed while the daemon or host was stopped are not replayed.
Overlapping runs: Use flock when a job can exceed its interval.
Logging: Configure logrotate and monitor logs for failures or missing completion heartbeats.
Backups: Archive every user crontab together with the system schedules, for example sudo tar -czf "crontabs-$(date +%F).tar.gz" /var/spool/cron /etc/crontab /etc/cron.d (Ubuntu keeps user crontabs under /var/spool/cron/crontabs, RHEL under /var/spool/cron). Listing one crontab with crontab -l does not create a backup.
Upgrades: Re-test scripts and schedules after operating-system upgrades.
Code Example
Create a dedicated account, state directory, and Bash script:
sudo useradd --system \
--home-dir /var/lib/report-job \
--create-home \
--shell /usr/sbin/nologin \
report-job
sudo install -d -m 0750 -o report-job -g report-job /var/lib/report-job
sudo tee /usr/local/bin/report-job >/dev/null <<'EOF'
#!/usr/bin/env bash
set -euo pipefail
export PATH=/usr/local/bin:/usr/bin:/bin
readonly STATE_DIR="/var/lib/report-job"
readonly LOCK_FILE="${STATE_DIR}/report-job.lock"
# Prevent overlapping executions of this job.
exec 9>"${LOCK_FILE}"
if ! /usr/bin/flock --nonblock 9; then
printf '%s report-job skipped: another instance is running\n' \
"$(/usr/bin/date --iso-8601=seconds)"
exit 0
fi
printf '%s report-job started\n' "$(/usr/bin/date --iso-8601=seconds)"
# Run the actual reporting or maintenance work here.
printf '%s report-job completed\n' "$(/usr/bin/date --iso-8601=seconds)"
EOF
sudo chmod 0755 /usr/local/bin/report-job
sudo touch /var/log/report-job.log
sudo chown report-job:report-job /var/log/report-job.log
sudo chmod 0640 /var/log/report-job.log
This runs the script every day at 02:17 using the configured cron time zone.
Bash Scripting and Portability
Variables and quoting: Use readonly for constants and quote expansions such as "${LOCK_FILE}" to prevent word splitting and pathname expansion.
Conditionals, functions, and arguments: Use if or case for validation, organize larger jobs into functions, pass arguments with main "$@", and declare function-local variables with local.
Loops and text processing: Read records with while IFS= read -r line; use grep for selection, sed for substitutions, and awk for field processing or aggregation. Quote patterns and protect filenames from option-like values.
Exit codes:set -euo pipefail catches common failures, but critical operations still need explicit checks because set -e has conditional and function-related edge cases. Run ShellCheck in CI.
Portability: Keep the Bash shebang: set -o pipefail, arrays, [[ ... ]], and process substitution are not portable to a /bin/sh script such as dash.
48
How do you write an idempotent Bash script for production?
KnownRevise
An idempotent script converges to the same desired state when repeated; a second run should make no additional changes to the managed resources. It may still validate state, acquire locks, and report status. Production scripts should combine state comparison, concurrency control, atomic updates, explicit error handling, and least-privilege execution.
Read full answerHide full answer
Applies toBash 5.x on Ubuntu 22.04/24.04; Bash 4.4 on RHEL 9
How It Works
An idempotent automation script generally follows this pattern:
Validate inputs, permissions, and required dependencies.
Acquire a lock so concurrent executions cannot modify the same state.
Compare the current state with the desired state.
Apply only missing or incorrect configuration.
Return a meaningful exit code and record what changed.
Idempotency does not mean the script performs no work or produces no output. It means repeated executions converge on the same managed state without accumulating additional changes.
Code Example
The following script installs a regular configuration file at a fixed path. It replaces a symlink, or a regular file whose content or mode differs; a matching regular file is left untouched.
#!/usr/bin/env bash
set -Eeuo pipefail
readonly APP_DIR="/opt/myapp"
readonly APP_FILE="${APP_DIR}/app.conf"
readonly LOCK_FILE="/var/lock/myapp-deploy.lock"
tmp_file=""
cleanup() {
if [[ -n "$tmp_file" ]]; then
rm -f -- "$tmp_file"
fi
}
trap 'printf "Deployment failed at line %s\n" "$LINENO" >&2' ERR
trap cleanup EXIT
main() {
if (( EUID != 0 )); then
printf "This script must run as root.\n" >&2
exit 1
fi
# Prevent two deployments from modifying the application concurrently.
exec 9>"$LOCK_FILE"
if ! flock -n 9; then
printf "Another deployment is already running.\n" >&2
exit 1
fi
install -d -m 0755 "$APP_DIR"
# Create the candidate file on the same filesystem for an atomic rename.
tmp_file="$(mktemp "${APP_DIR}/.app.conf.XXXXXX")"
cat >"$tmp_file" <<'EOF'
listen 8080
workers 4
EOF
chmod 0644 "$tmp_file"
if [[ ! -f "$APP_FILE" || -L "$APP_FILE" ]] ||
! cmp -s -- "$tmp_file" "$APP_FILE" ||
[[ "$(stat -c '%a' "$APP_FILE")" != "644" ]]; then
# -T treats the destination as a file; GNU mv replaces it atomically.
mv -fT -- "$tmp_file" "$APP_FILE"
tmp_file=""
printf "Updated %s\n" "$APP_FILE"
else
printf "Unchanged %s\n" "$APP_FILE"
fi
}
main
The script requires Bash, GNU coreutils, GNU diffutils for cmp, and flock from util-linux. Run it through ShellCheck before deployment.
Operational Design
Desired-state comparison: Inspect existing resources before modifying them. For cloud APIs, prefer idempotent PUT operations over repeated POST requests; use stable resource identifiers or idempotency keys where supported.
Concurrency control: flock prevents overlapping local executions. Across hosts, use a distributed lock or configure the orchestrator to serialize the operation.
Atomic updates: Write a temporary file in the destination filesystem and rename it into place, preventing readers from seeing a partially written configuration.
Change-triggered actions: Restart a service or reload configuration only when the managed state changes, not on every execution.
Auditability and monitoring: Record timestamps, current state, changes, errors, and duration. Emit metrics for changed, no-op, and failed runs, and alert on repeated failures.
Versioned artifacts: Record configuration or deployment versions so upgrades, incidents, and rollbacks can be correlated.
Failure Modes and Security
set -euo pipefail detects unset variables, failed commands, and failed pipeline components. It does not replace explicit handling of expected failures.
Run with the minimum required privileges. This example needs root because it writes under /opt and /var/lock; ordinary applications should use a dedicated service account.
Avoid eval, unquoted expansions, predictable temporary names, and embedded secrets. Supply secrets through protected environment files, secret managers, or mounted credentials.
A local lock does not coordinate with other machines or protect against an administrator manually changing the destination.
Keep previous versions when replacement must be reversible. The example demonstrates atomicity, not a complete rollback system.
Bash Mechanics and POSIX sh
Variables, quoting, and arguments: Quote expansions such as "$variable" and "$@". Validate $# before using positional arguments such as "$1".
Conditionals, functions, loops, and exits: Use [[ ... ]] for Bash tests, functions for reusable validated operations, and while IFS= read -r line; do ...; done < "$file" for line-safe input. Return 0 for success and a nonzero status for failure.
Text processing: Use grep -F for literal matching, sed -E for simple substitutions, and awk for record or field processing. Prefer a schema-aware parser for JSON or XML.
Portability: POSIX sh does not provide [[ ... ]] or set -o pipefail; portable scripts use [ ... ] and set -eu. flock, GNU mv -T, stat -c, and mktemp are also non-POSIX. RHEL 9 ships Bash 4.4 by default; this example remains compatible with it.
Best Practices
Make desired state explicit and repair drift rather than assuming resources are absent.
Guard destructive operations and make every operation safely repeatable.
Run the script twice in testing and verify that the second execution reports no change.
Validate ShellCheck results and exercise permission failures, unavailable dependencies, and lock contention.
Keep backups, versioned artifacts, and rollback procedures separate from idempotency; together they provide convergence and operational recoverability.
49
How do you prevent concurrent execution of the same Bash script?
KnownRevise
Use an advisory file lock with flock: every invocation opens the same lock file and atomically acquires an exclusive lock before performing the protected work. This avoids the check-then-write race associated with PID files, provided every cooperating instance uses the same lock.
Read full answerHide full answer
Applies toBash 5.x and util-linux 2.38+ (`flock`) on Linux
How flock Works
The flock utility places an advisory lock on an open file description. Separate processes opening the same lock file contend for that lock, and the kernel performs acquisition atomically.
The lock is cooperative rather than mandatory:
Every instance must use the same lock path and lock operation.
Code that ignores the advisory lock can still modify the protected resource.
File permissions and service isolation remain separate security controls.
Blocking and Nonblocking Modes
Mode
Command
Behavior
Nonblocking
flock --exclusive --nonblock 9
Returns immediately with a conflict status if the lock is held.
Blocking
flock --exclusive 9
Waits until the lock can be acquired.
Timed wait
flock --exclusive --timeout 30 9
Waits for at most 30 seconds.
Nonblocking mode is usually preferable for scheduled jobs because it avoids an indefinitely queued invocation. Blocking mode is useful when every invocation must eventually run, but an overrunning job can create a backlog or cause older work to execute later than expected.
Why PID Files Are Riskier
A typical PID-file implementation checks whether a recorded PID is running and then writes its own PID. Because those operations are separate, two processes can both observe an apparently free state and proceed. Stale files and PID reuse introduce further failure modes.
flock removes that acquisition race because the kernel serializes lock acquisition on the open file description.
Bash Versus POSIX sh
flock is a Linux utility rather than a shell feature, but its surrounding script syntax can differ:
Bash 5 supports set -euo pipefail, functions, arrays, process substitution, and [[ ... ]].
POSIX sh supports set -eu and basic redirection but not pipefail or Bash-specific expansions.
A POSIX sh wrapper can still invoke flock when the utility is installed, although locking remains platform-dependent.
Code Example
The following complete Bash script uses a dedicated file descriptor and distinguishes lock contention from other acquisition failures:
#!/usr/bin/env bash
set -euo pipefail
readonly LOCK_FILE="/run/lock/production-sync.lock"
# Every instance must open the same lock file.
exec 9>"$LOCK_FILE"
if flock --exclusive --nonblock --conflict-exit-code 75 9; then
:
else
status=$?
if (( status == 75 )); then
echo "Another instance is already running; exiting." >&2
exit 75
fi
echo "Could not acquire the lock (flock exit $status)." >&2
exit "$status"
fi
cleanup() {
exec 9>&-
}
trap cleanup EXIT
trap 'exit 130' INT
trap 'exit 143' TERM
echo "Acquired the lock; starting the protected work."
# Replace this with the protected operation.
sleep 5
echo "Protected work completed."
First invocation:
Acquired the lock; starting the protected work.
Protected work completed.
After successful acquisition, this form returns the wrapped command’s exit status. Consequently, the conflict status must be reserved by the application; the file-descriptor form above separates acquisition status from protected-work status.
Production Considerations
Use one stable path: Every instance must use exactly the same path. Do not include a timestamp or PID in the filename.
Control permissions: Ensure the service account can create the lock file in /run/lock, or configure a dedicated writable runtime directory.
Do not unlink the lock file while holding it: Removing it allows another process to create and lock a different inode at the same path. Its mere existence does not mean the lock is held.
Handle inherited descriptors: Background children can inherit descriptor 9 and keep the lock after the parent exits. Close it in children that should not hold the lock.
Use suitable storage: Coordinate instances through a filesystem with reliable flock semantics; verify distributed-filesystem behavior before coordinating across hosts.
Monitor contention: Log contention and acquisition failures, and configure supervisors such as systemd to distinguish a temporary conflict from a failed protected operation.
50
How do you create temporary files safely with mktemp in Bash?
KnownRevise
mktemp safely creates temporary files or directories with unpredictable names, avoiding the symlink and overwrite races associated with predictable paths such as /tmp/job.$$. In Bash, combine it with strict error handling, restrictive permissions, and an exit trap to clean up resources automatically.
Read full answerHide full answer
Applies toBash 5.x and GNU coreutils 9.x on Linux
How mktemp Works
mktemp replaces a template containing consecutive X characters with randomly generated characters and atomically creates the resulting path. It fails rather than overwriting an existing file.
File creation: mktemp /tmp/job.XXXXXXXXXX creates an individual file.
Directory creation: mktemp -d /tmp/job.XXXXXXXXXX creates a private directory.
Permissions: GNU mktemp creates files with user read/write permissions and directories with user read/write/execute permissions, subject to the process umask.
Failure behavior: It returns a nonzero exit status if it cannot create the path.
The template must contain at least three consecutive X characters in its final path component. An explicit template is preferable to relying on the default, because GNU and BSD implementations can differ in their default behavior.
Safe Bash Pattern
Use set -euo pipefail to stop on unexpected failures, unset variables, and failed pipeline stages. Set umask 077 before creating sensitive files so that no group or other permissions are granted.
For multiple related temporary files, create a private directory first. Other processes cannot list or enter a mode 700 directory unless they have sufficient privileges.
Install the cleanup trap immediately after creating the directory. Otherwise, a failure while creating a later file could leave the directory behind.
Code Example
#!/usr/bin/env bash
set -euo pipefail
# Prevent new files from receiving group or other permissions.
umask 077
readonly app_name="orders"
readonly tmp_root="${TMPDIR:-/tmp}"
# Atomically create a private directory for this process.
work_dir="$(mktemp -d "${tmp_root}/${app_name}.XXXXXXXXXX")"
cleanup() {
rm -rf -- "$work_dir"
}
trap cleanup EXIT
trap 'exit 129' HUP
trap 'exit 130' INT
trap 'exit 143' TERM
# Create an unpredictable file inside the private directory.
temp_file="$(mktemp "${work_dir}/payload.XXXXXXXXXX")"
# The file already exists and is writable only by its owner.
printf '%s\n' '{"order_id":123,"status":"pending"}' > "$temp_file"
printf 'Temporary file: %s\n' "$temp_file"
printf 'Content: '
cat "$temp_file"
The signal traps convert termination signals into controlled exits. The EXIT trap then performs cleanup.
Failure Modes and Security
Avoid predictable names: Paths such as /tmp/job.$$ can be guessed, and code that checks before writing may still have a race window.
Do not pre-create the final file: Passing an existing path to mktemp causes it to fail instead of securely replacing the path.
Do not broaden permissions afterward: Running chmod 666 after creation exposes sensitive data. Create it with the required permissions instead.
Do not trust an unvalidated TMPDIR: Use a directory controlled by the service account or administrator.
mktemp is not encryption: Other processes running as the same user, root, and potentially swap or recovery mechanisms may access the contents.
Traps have limits: An EXIT trap does not run after SIGKILL, a kernel panic, or host power loss.
Production Practices
For credentials or sensitive payloads, prefer a service-private tmpfs, such as a dedicated RuntimeDirectory in systemd, rather than the general-purpose /tmp filesystem.
Use systemd sandboxing and dedicated service accounts so temporary files are not shared with unrelated applications.
Run temporary storage on encrypted filesystems when the host disk or swap may be exposed.
Add an external cleanup mechanism, such as an appropriate systemd-tmpfiles policy, for directories left behind after abnormal termination.
Avoid promising secure deletion through shred; journaled and copy-on-write filesystems may retain previous contents.
mktemp itself is not specified by POSIX. Basic template usage works on GNU and BSD systems, but options such as --tmpdir, --suffix, and --dry-run are not portable. The pipefail option also requires Bash rather than POSIX sh.
51
How do I add timestamps to Bash script logs?
KnownRevise
Use timestamped logging through a Bash function that formats each message before writing it to standard output, standard error, or a log file. In production, timestamps should normally use UTC and an unambiguous ISO 8601 format so logs from multiple hosts can be correlated reliably.
Read full answerHide full answer
Applies toBash 5.x on Linux (Ubuntu 22.04/24.04, RHEL 9)
How It Works
A practical logging function combines four operations:
date -u generates the current UTC timestamp.
printf formats the timestamp, severity, and message consistently.
tee -a writes to a file while also displaying the message.
set -euo pipefail detects unset variables, command failures, and failed pipeline components.
Messages are commonly separated by destination:
Standard output: Informational and successful progress messages.
Standard error: Warnings and errors.
Log file: A persistent copy, when local file logging is required.
Code Example
#!/usr/bin/env bash
set -euo pipefail
LOG_FILE="${LOG_FILE:-./application.log}"
log() {
local level="$1"
shift
local timestamp
local message
timestamp="$(date -u '+%Y-%m-%dT%H:%M:%SZ')"
message="$*"
# Prevent untrusted input from creating forged log entries.
message="${message//$'\n'/\\n}"
message="${message//$'\r'/\\r}"
if [[ "$level" == "ERROR" ]]; then
printf '%s [%s] %s\n' "$timestamp" "$level" "$message" \
| tee -a "$LOG_FILE" >&2
else
printf '%s [%s] %s\n' "$timestamp" "$level" "$message" \
| tee -a "$LOG_FILE"
fi
}
log INFO "Application started"
log INFO "Processing request id=${REQUEST_ID:-unknown}"
if [[ "${1:-}" == "--fail" ]]; then
log ERROR "Simulated operation failure"
exit 1
fi
log INFO "Application stopped"
The -a option appends to the file rather than overwriting it. Because tee writes to standard error for errors, monitoring systems can treat error output separately from normal progress output.
Production Practices
Prefer structured logging: Emit JSON when a centralized collector such as Loki, Elasticsearch, or Splunk parses the logs. Include fields such as timestamp, level, service, host, and request_id.
Use one primary sink: In containers, writing timestamped lines to stdout and stderr is usually enough for journald, Docker, or Kubernetes to collect them. Avoid keeping duplicate local files unless retention requires them.
Rotate files: Configure logrotate or the platform’s logging agent to limit file size, retention, and disk usage. A logging failure caused by a full disk can otherwise terminate the script.
Protect sensitive data: Never log passwords, access tokens, private keys, or complete request bodies. Redact sensitive fields before formatting the message.
Prevent log injection: Convert embedded newlines and carriage returns, as shown above, when values can come from users or remote systems.
Synchronize clocks: Configure NTP or chrony. Accurate host time is essential when correlating logs with metrics, traces, and incident timelines.
Monitor failures: Alert on repeated error entries, logging errors, rapid log growth, and unexpectedly missing events.
Failure Modes and Portability
If LOG_FILE points to a nonexistent directory or an unwritable file, tee fails. With set -euo pipefail, that failure can stop the script.
Concurrent scripts writing to the same file can interleave output. Use one writer, append-safe single-line messages, or file locking when strict ordering is required.
${var//pattern/replacement} and [[ ... ]] are Bash features and are not portable to POSIX sh. The pipefail option is also not POSIX.
The date format used here is supported by GNU coreutils on the stated Linux distributions. A POSIX shell can still use date -u and printf, but it needs different syntax for parameter substitution, conditionals, and pipeline error handling.
52
How do you check whether a command exists in Bash?
KnownRevise
Use the command -v builtin to test whether a command is available before invoking it. It checks the current shell environment, including PATH executables, built-ins, shell functions, and defined aliases.
Read full answerHide full answer
Applies toBash 5.x; POSIX sh implementations supporting `command -v`
How command -v Works
command -v returns exit status 0 when it finds the requested command and a nonzero status when it does not:
if command -v curl >/dev/null 2>&1; then
curl --version
else
printf 'curl is not installed\n' >&2
exit 127
fi
For an external command, it prints the pathname found on PATH. Redirecting output prevents the path from appearing in logs when only availability matters.
Bash checks its PATH when locating external commands. Consequently, a command may be unavailable even if it is installed in a location that is not present in the current PATH.
Built-Ins, Functions, and External Tools
command -v recognizes more than executable files:
Built-ins: Commands such as printf or test are available even without a separate executable file.
Functions: A defined Bash function causes the check to succeed.
External commands: Bash searches PATH for an executable file.
Aliases: command -v can report a defined Bash alias. Whether that alias is expanded later during command parsing is controlled separately by the expand_aliases shell option.
If the operation specifically requires an external executable, Bash’s type -P is stricter because it searches only PATH and ignores built-ins, functions, and aliases:
if type -P jq >/dev/null 2>&1; then
jq --version
fi
type -P is Bash-specific. Avoid relying on which: it is not POSIX, may be installed separately, and its output can be misleading when multiple PATH entries contain the same command.
Robust Bash Example
The following script checks every dependency before running the main operation and reports all missing commands together.
#!/usr/bin/env bash
set -euo pipefail
required_commands=(curl jq systemctl)
missing_commands=()
for required_command in "${required_commands[@]}"; do
if ! command -v "$required_command" >/dev/null 2>&1; then
missing_commands+=("$required_command")
fi
done
if ((${#missing_commands[@]} > 0)); then
printf 'Missing required command(s):\n' >&2
printf ' %s\n' "${missing_commands[@]}" >&2
exit 127
fi
printf 'All required commands are available.\n'
Example output when jq is absent:
Missing required command(s):
jq
Exit status 127 is conventionally used to indicate that a required command could not be found. The if ! command -v ... construct is safe with set -e because commands used as conditions are exempt from immediate failure.
Production and Portability Considerations
Check the effective PATH: sudo, service managers, containers, and CI runners may use a different PATH from an interactive login shell.
Aliases may not be expanded in scripts: Non-interactive shells usually have alias expansion disabled, even if an alias has been defined. Use type -P when an external executable is mandatory.
Presence is not readiness: A binary existing does not prove that a service is healthy or that its dependencies are reachable. Production checks should also validate versions, connectivity, and service state.
Avoid eval: Never use eval to execute a dynamically selected command. Quote the command variable so it remains one argument.
Restrict dynamic commands: If the command name comes from user input, use an explicit allowlist. Checking that a name exists does not make executing arbitrary input safe.
POSIX compatibility: command -v is supported by POSIX shells, but arrays, set -o pipefail, and type -P require Bash.
Best Practices
Use command -v for normal availability checks.
Use type -P when an actual executable in PATH is mandatory.
Place dependency checks before the operation that requires them.
Quote every expansion and avoid word splitting.
Validate the required version, not just the command’s presence.
Run shellcheck script.sh and test both successful and missing-command paths before deployment.
53
How does sourcing a Bash script differ from executing it?
KnownRevise
Sourcing runs a script’s contents in the current shell, allowing its variables, functions, and other shell state to affect the caller. Executing a script runs it in a separate interpreter process, so most shell-state changes remain isolated from the caller.
Read full answerHide full answer
Applies toBash 5.x and POSIX `sh` (`.` is POSIX; `source` is not POSIX-standard)
How Sourcing Works
Bash provides the source built-in, while the equivalent POSIX form is the dot command: source ./common.sh and . ./common.sh, respectively.
The sourced file is interpreted by the calling shell rather than launched as another program. It therefore does not require a shebang, although any shebang present is treated as a comment. Its syntax must nevertheless be compatible with the calling shell: Bash-specific syntax cannot be safely sourced by a POSIX sh.
If the filename contains no slash, Bash searches PATH for it. If it contains a slash, Bash resolves it relative to the current working directory. Sourcing is commonly used for function libraries, configuration files, constants, and shell initialization code.
Sourcing can intentionally or accidentally change the caller’s variables, functions, working directory, positional parameters, shell options, traps, and other state. It can also terminate the calling shell with exit, so sourced files should be treated as trusted code with the same privileges as the caller.
How Execution Differs
Running a script as ./script.sh normally uses its shebang to select an interpreter. Running bash script.sh explicitly selects Bash and starts it as a child process.
Behavior
Sourcing with source or .
Executing a script
Process
Uses the current shell
Starts a separate interpreter process
Variables
Can modify the caller’s variables
Changes do not propagate to the caller
Functions
Become available to the caller
Exist only in the child process
exit
Can terminate the calling shell
Terminates only the script’s process
Shebang
Not required
Needed for reliable interpreter selection
Exit status
Final sourced-command status, 0 for an empty file, or nonzero if the file cannot be read
The executed script’s exit status
Portability
. is portable; source is a Bash extension
The script can select its own interpreter
External effects still occur when a script is executed: it can modify files, call services, change system configuration, or send signals. Execution isolates shell state, not operating-system side effects.
Arguments and Exit Status
Arguments supplied to source temporarily become the sourced file’s positional parameters. For example, source ./set-mode.sh production makes production available as $1 inside that file. Bash restores the caller’s original positional parameters when the source operation completes.
The status of source is nonzero when the file cannot be found or read. Otherwise, it is the status of the final command executed by the sourced file; an empty file returns 0. An executed script likewise communicates success or failure through its exit status, allowing production job runners to branch on the result.
Practical Guidance and Failure Modes
Prefer execution for standalone automation: it prevents accidental changes to the caller’s environment and gives jobs a clear process boundary and exit status.
Use sourcing for intentional shared state: it is appropriate for trusted libraries, configuration, and interactive shell setup.
Resolve paths explicitly: relative paths are based on the caller’s working directory, not automatically on the location of the calling script. Derive a script directory from a known base when portability matters.
Treat sourced code as privileged code: it can overwrite variables, alter shell options, change directories, or terminate the parent shell.
Manage error handling deliberately: options such as set -euo pipefail already active in the caller also apply to sourced code, potentially terminating an interactive or automation shell.
Check scripts with ShellCheck: quote expansions, use local for function-local variables, and avoid unnecessary aliases or unquoted command substitutions.
Test changes to shared libraries: sourcing can affect every later command in the session, so production setup scripts should be validated in a controlled shell before deployment.
For scripts that must run under POSIX sh, use . rather than source and avoid Bash-only features such as arrays, [[ ... ]], local, and set -o pipefail.
54
How do Bash subshells differ from the current shell?
KnownRevise
A Bash subshell is a child environment that starts with a copy of the current shell’s state, but changes made inside it do not affect the parent. The current shell is the original Bash process, so assignments, function definitions, option changes, and directory changes made there remain active.
Read full answerHide full answer
Applies toBash 5.x on Linux
Process Model
The current shell executes commands, built-ins, and sourced files in its existing process. A subshell, commonly created with parenthesized commands such as (command1; command2), runs in a child Bash process with a different process ID and an initially copied working directory and shell state.
Subshells are used for:
Parenthesized groups: Commands enclosed in parentheses execute in a subshell.
Command substitution: output=$(command) evaluates the command in a subshell environment and captures its standard output.
Pipelines: Pipeline components normally execute in subshells, with the lastpipe exception described below.
Background commands: A command followed by & executes asynchronously in a child environment.
Starting bash -c 'command' creates a separate Bash process. It provides similar state isolation, but it is not technically a subshell created by the current Bash process. Similarly, an external command such as curl starts as a child process and is normally replaced with that executable using exec; it is not a Bash subshell after that replacement.
Inside a Bash-created subshell, $$ retains the original shell’s process ID, while $BASHPID identifies the current Bash process. A separately started bash -c process has its own value for both.
State and Scope
Aspect
Current shell
Subshell
Process
Existing shell process
Child Bash process
Process ID
Remains unchanged
Normally differs from the parent
Variable assignments
Remain active
Do not modify the parent
Working directory
cd changes the parent’s directory
cd affects only the child
Function definitions
Remain available
Copied initially; changes do not reach the parent
Shell options
Apply directly
Copied initially; changes remain local
exit
Terminates the current shell
Terminates only the subshell
Exported environment
Inherited by child processes
Copied at creation; changes remain local
A subshell can read ordinary, unexported shell variables as well as exported environment variables because it begins as a copy of the parent shell. Ordinary variables are not added to the child process’s exported environment, so an external program started by the subshell does not receive them unless they are explicitly exported.
The lastpipe option is an important exception to normal pipeline isolation. When it is enabled and job control is disabled, the final command in a pipeline can execute in the current shell and therefore retain variable and directory changes. It is enabled by default from Bash 5.1 onward; Bash 5.0 supports it but does not enable it by default.
Exit Status and Error Handling
Immediately after a subshell finishes, its exit status is available in $?. For example, (exit 7) leaves a status of 7 for the next command.
With set -euo pipefail:
errexit: Terminates the shell after many unsuccessful simple commands, subject to Bash’s context-sensitive exceptions, such as commands whose status is tested in an if condition.
nounset: Treats references to unset variables as errors.
pipefail: Makes a pipeline return the status of the rightmost component that exited unsuccessfully, rather than always returning the final component’s status.
These options are inherited by Bash-created subshells, but changing them inside a subshell does not reconfigure the parent. A standalone failing subshell can cause an errexit-enabled parent to terminate, so important statuses should still be handled explicitly with constructs such as if or ||.
Current Shell Versus Sourcing
The source builtin, also available as the POSIX command ., executes a file in the current shell. Variable assignments, function definitions, and other permitted changes from the file remain in that shell.
Running bash script.sh instead starts another Bash process. Changes to variables, functions, options, or the working directory in that script do not affect the caller. This distinction is useful when choosing between loading trusted configuration and running isolated automation.
Production Guidance
Use the current shell when configuration must intentionally change variables, functions, options, or the working directory.
Use a subshell when temporary state, command substitution, pipeline isolation, or background work is preferable.
Do not source untrusted files; sourced code runs with the current shell’s privileges and can alter security-sensitive behavior.
Do not rely solely on set -e; check operations whose failure requires rollback, cleanup, alerting, or a specific fallback.
Use trap for cleanup, mktemp for temporary files, and explicit argument validation in production scripts.
Avoid exposing secrets in command arguments or unnecessarily exporting them, because child processes may inherit exported values and some environments expose process metadata to other privileged users.
POSIX sh does not support Bash-only features such as arrays, [[ ... ]], process substitution, and pipefail; use #!/usr/bin/env bash and Bash 5 syntax explicitly when those features are required.
55
Why do Bash variables vanish inside a pipeline loop?
KnownRevise
Bash variables appear to vanish inside a pipeline loop because the loop normally runs in a subshell, a child process with its own copy of the shell state. Assignments made there cannot update the parent shell.
Read full answerHide full answer
Applies toBash 5.x on Linux and other Unix-like systems
How Bash Executes Pipelines
By default, Bash runs each command in a pipeline in a subshell environment. This applies when a while loop is a pipeline component: assignments inside the loop modify only the child shell’s copy of those variables. When that child exits, its changes are discarded, while the parent retains its previous value or remains unset.
Related Bash behaviors follow the same process-isolation principle:
Command substitution: result=$(command) runs the command in a subshell, so shell-state changes inside it do not affect the parent.
Background execution: A command started with & cannot modify the parent shell’s variables.
export: Exporting a variable includes its current value in the environment of subsequently launched children. It does not propagate later changes back to the parent.
set -euo pipefail: These options control error handling and pipeline status reporting; they do not share shell state between processes.
There is one Bash-specific exception: shopt -s lastpipe allows the final command of a pipeline to run in the current shell when job control is inactive. It is Bash-only and can change shell state unexpectedly, so it is generally unsuitable as a portable default.
Alternatives and Portability
Method
Preserves parent variables
Portability
Operational notes
Process substitution with while ... done < <(command)
Yes
Bash only
Concise, but the producer’s exit status is not automatically propagated to the loop
Redirect the loop from a regular file
Yes
Bash and POSIX sh
Simple and portable; manage the file securely and remove temporary files reliably
Capture output with command substitution
Yes, indirectly
Bash and POSIX sh
Suitable for output data rather than arbitrary shell-state changes; trailing newlines require care
Enable lastpipe and use an ordinary pipeline
Sometimes
Bash only
The final command runs in the current shell only when job control is inactive
Use an ordinary pipeline
No
Bash and POSIX sh
Each pipeline component retains its isolated shell environment
For POSIX sh, use redirection from a file or a here-document instead of process substitution. Process substitution is convenient in Bash but is not part of the POSIX shell standard.
Production Practices
Use IFS= read -r value when processing line-oriented input. An empty IFS preserves leading and trailing whitespace, while -r prevents backslashes from being interpreted as escape sequences.
Do not rely on set -e to detect every producer failure. Test important commands explicitly with if or case, and use pipefail when evaluating the status of an entire pipeline.
Treat process-substitution producers carefully: their failure status is not automatically the status of the consuming loop. If the failure matters, run the producer separately, check its status, and then consume a securely created temporary file.
For text processing with grep, sed, or awk, remember that pipeline output can be transformed successfully while shell-variable changes remain isolated. Materialize the required output before updating parent-shell state.
Quote expansions such as "$value" and "$@" to prevent word splitting and glob expansion. Pass function arguments through positional parameters, return meaningful nonzero statuses, and test them explicitly.
When using temporary files, create them with secure permissions, restrict access where data is sensitive, and arrange cleanup with a trap so failures or interrupts do not leave artifacts behind.
56
How do Bash scripts handle signals such as SIGINT gracefully?
KnownRevise
Use a Bash trap to catch SIGINT and other catchable signals, set a shutdown flag, and clean up child processes before exiting. This creates a graceful shutdown rather than allowing the script and its workloads to terminate abruptly.
Read full answerHide full answer
Applies toBash 5.x on Linux (Ubuntu 22.04/24.04, RHEL 9)
Signal Handling Model
Signal
Typical source
Can be trapped?
Recommended behavior
SIGINT
User presses Ctrl-C
Yes
Stop work and shut down cleanly
SIGTERM
systemd, Kubernetes, process managers
Yes
Begin graceful termination
SIGHUP
Terminal disconnect or session change
Yes
Persist state or shut down
SIGKILL
Forced termination
No
Use only when graceful shutdown times out
SIGSTOP
Process suspension
No
Cannot be caught or ignored
Bash executes trap commands between shell operations. A trap handler should therefore perform minimal work: update state, record the signal, or forward termination to a child. Do not perform long-running operations directly inside the handler.
EXIT is different from signal traps: it runs during normal shell exit and during a handled shutdown path, but it cannot run after SIGKILL.
Production Behavior
When a user presses Ctrl-C, the terminal sends SIGINT to the entire foreground process group. Consequently, the Bash script and its foreground child may receive the signal independently.
By contrast, kill -TERM <script-pid> targets only the script. If the script does not forward the request, its child process may continue running after the parent exits. A production script should therefore:
Trap SIGINT and SIGTERM.
Stop starting new work.
Allow in-flight work to finish within a defined deadline.
Forward SIGTERM to child processes.
Clean up temporary files, locks, and other resources.
Record enough information for monitoring and incident diagnosis.
When running under systemd or Kubernetes, configure the termination grace period to exceed the script’s cleanup time. If cleanup exceeds that deadline, the supervisor may send SIGKILL, which no script can handle.
Code Example
The following script waits for a background child and handles either SIGINT or SIGTERM. It also uses set -euo pipefail, quoted variables, a function argument, conditional logic, and explicit exit codes.
#!/usr/bin/env bash
set -euo pipefail
child_pid=''
shutdown_requested=0
handle_signal() {
local signal_name="$1"
shutdown_requested=1
printf 'Received %s; shutting down\n' "$signal_name" >&2
}
cleanup() {
local exit_code="$1"
# Prevent recursive execution of the EXIT trap.
trap - EXIT INT TERM
if [[ -n "$child_pid" ]] && kill -0 "$child_pid" 2>/dev/null; then
printf 'Stopping child PID %s\n' "$child_pid" >&2
kill -TERM "$child_pid" 2>/dev/null || true
wait "$child_pid" 2>/dev/null || true
fi
exit "$exit_code"
}
trap 'handle_signal SIGINT' INT
trap 'handle_signal SIGTERM' TERM
trap 'cleanup "$?"' EXIT
# The optional argument controls how long the child runs.
sleep "${1:-300}" &
child_pid=$!
printf 'Started child PID %s\n' "$child_pid"
# Putting wait in an if condition prevents set -e from exiting
# before the signal handler can request graceful shutdown.
if wait "$child_pid"; then
child_pid=''
else
child_status=$?
if (( shutdown_requested == 1 )); then
exit 0
fi
exit "$child_status"
fi
Example output:
Started child PID 12345
^C
Received SIGINT; shutting down
Stopping child PID 12345
The script exits with 0 after handling a requested shutdown. If callers require signal-style status, the shutdown branches can instead use 130 for SIGINT and 143 for SIGTERM, following the conventional 128 + signal number convention.
Best Practices and Failure Modes
Keep handlers short: Set variables such as shutdown_requested=1; let a main loop such as while (( ! shutdown_requested )); do ...; done perform the actual shutdown work.
Handle child processes explicitly: A signal sent to the parent does not automatically reach children. Forward SIGTERM and wait for them to stop.
Do not rely on set -e for signal handling: set -euo pipefail detects command failures, but it does not replace traps or cleanup logic.
Avoid SIGKILL as the first choice: It prevents cleanup, state flushing, and diagnostic logging. Use it only as a supervisor’s final fallback.
Prefer structured logs: Emit timestamped messages to stderr. For quick incident searches, filter them with journalctl -u <service> | grep -E 'Received SIG(INT|TERM)|Stopping child'.
Handle PID 1 carefully: Explicit traps are required because Linux gives PID 1 special default signal behavior. For containers, an init or supervisor that reaps orphaned children is safer than relying on a standalone Bash script.
Test every termination path: Exercise Ctrl-C, kill -TERM, child failure, invalid arguments, and supervisor-initiated shutdown. Verify exit codes, orphaned processes, logs, and cleanup of locks or temporary files.
57
How do Bash scripts handle dates, times, and time zones?
KnownRevise
Bash scripts typically use GNU date for parsing timestamps and performing calendar calculations, and Bash’s printf time format for lightweight formatting. Store and compare instants in UTC or Unix epoch seconds, then convert to a local time zone only for display or scheduling.
Read full answerHide full answer
Applies toBash 5.x on Linux with GNU coreutils 9.x
How It Works
Bash does not implement a complete calendar system. Its built-in time facilities work with the system clock and timezone database:
printf '%(%Y-%m-%d)T': Formats the current time using strftime-style directives. This is a Bash built-in and avoids starting an external process.
EPOCHREALTIME: Contains the current Unix timestamp with microsecond granularity.
EPOCHSECONDS: Contains the current Unix timestamp as an integer.
SECONDS: Tracks the number of seconds since the shell started, useful for coarse script-relative timing.
date: Provides timezone conversion, parsing, epoch conversion, and relative-date calculations.
TZ: Controls the timezone used by GNU date and Bash’s time-formatting functions when the system default is not wanted.
Internally, Unix timestamps represent elapsed seconds since 1970-01-01T00:00:00Z. They do not encode leap seconds.
Core Operations
Operation
GNU date command
Current UTC timestamp
date --utc '+%Y-%m-%dT%H:%M:%SZ'
Current epoch seconds
date +%s
Convert epoch to UTC
date --utc --date="@1710000000" '+%Y-%m-%dT%H:%M:%SZ'
Parse an ISO timestamp
date --utc --date='2025-03-08T12:30:00Z' +%s
Add one UTC calendar day
date --utc --date='2025-03-08 00:00:00 UTC +1 day' '+%Y-%m-%dT%H:%M:%SZ'
Add 24 elapsed hours to epoch 1710000000
date --utc --date="@$((1710000000 + 86400))" '+%Y-%m-%dT%H:%M:%SZ'
Format using Bash
printf '%(%Y-%m-%d %H:%M:%S %Z)Tn' -1
Convert UTC to a named zone
TZ=America/New_York date --date='2025-03-08T12:30:00Z' '+%Y-%m-%d %H:%M:%S %Z'
For arithmetic, prefer integer epoch seconds. Adding 86400 means exactly 24 elapsed hours; a local “tomorrow” can be 23 or 25 hours across daylight-saving transitions.
Code Example
#!/usr/bin/env bash
set -euo pipefail
export LC_ALL=C
parse_iso_utc() {
local value="$1"
local pattern='^[0-9]{4}-[0-9]{2}-[0-9]{2}T[0-9]{2}:[0-9]{2}:[0-9]{2}Z$'
local epoch
[[ "$value" =~ $pattern ]] || return 1
epoch="$(date --utc --date="$value" +%s)" || return 1
# Reject dates that GNU date silently normalized, such as February 31.
[[ "$(date --utc --date="@${epoch}" '+%Y-%m-%dT%H:%M:%SZ')" == "$value" ]] || return 1
printf '%s\n' "$epoch"
}
now_epoch="$(date +%s)"
now_utc="$(date --utc '+%Y-%m-%dT%H:%M:%SZ')"
local_now="$(printf '%(%Y-%m-%d %H:%M:%S %Z)T' -1)"
input='2025-03-08T12:30:00Z'
if ! parsed_epoch="$(parse_iso_utc "$input")"; then
printf 'Invalid UTC timestamp: %s\n' "$input" >&2
exit 2
fi
# Find the next UTC midnight independently of the host timezone.
tomorrow_midnight="$(date --utc --date='tomorrow 00:00 UTC' +%s)"
seconds_until_midnight=$((tomorrow_midnight - now_epoch))
printf 'UTC now: %s\n' "$now_utc"
printf 'Local now: %s\n' "$local_now"
printf 'Input as epoch: %s\n' "$parsed_epoch"
printf 'Seconds until next UTC midnight: %s\n' "$seconds_until_midnight"
if ((now_epoch < parsed_epoch)); then
printf 'The input timestamp is in the future.\n'
else
printf 'The input timestamp is in the past.\n'
fi
Illustrative output on a host configured for UTC:
UTC now: 2025-03-08T10:15:30Z
Local now: 2025-03-08 10:15:30 UTC
Input as epoch: 1741437000
Seconds until next UTC midnight: 49470
The input timestamp is in the future.
Production Practices
Use UTC for storage and comparisons: Avoid daylight-saving ambiguity when converting local deployment schedules.
Use epoch seconds for elapsed-time calculations: They are simpler to compare and subtract than formatted strings. Use date with an explicit timezone for calendar arithmetic.
Validate before parsing: GNU date --date accepts flexible expressions such as tomorrow; strict validation prevents accidental interpretation of malformed input.
Handle failures explicitly: With set -euo pipefail, failed date commands normally stop the script. Use if ! value="$(command)" when failure is an expected validation path.
Specify recurring schedules and zones: For example, use 02:00 UTC or 02:00 America/New_York. Test schedules around daylight-saving transitions because local 02:00 can be skipped or occur twice.
Account for clock synchronization: Epoch calculations depend on accurate system time. Production hosts should use NTP or chrony, and timestamp-related alerts should monitor clock skew.
Be explicit about locale: Set LC_ALL=C when parsing predictable numeric or date output.
Portability and Failure Modes
GNU date options are not portable to BSD/macOS:
GNU uses date --date='...'.
BSD uses date -j -f '<format>' '<value>' '<format>' for parsing.
BSD relative calculations use expressions such as date -v+1d.
POSIX sh and basic POSIX date provide limited formatting but do not standardize GNU-style parsing or relative-date arithmetic.
For scripts that must run across Linux and macOS, isolate date operations behind functions or use a portable runtime such as Python. Also avoid relying on date for precise elapsed-time measurement in performance-sensitive tooling, because wall-clock adjustments can affect results.
58
How can curl make reliable HTTP requests in Bash scripts?
KnownRevise
Use curl with strict error handling, explicit timeouts, controlled retries, and secure authentication options. In production, scripts should distinguish transport failures from HTTP error responses and avoid leaking credentials or retrying non-idempotent operations unintentionally.
Read full answerHide full answer
Applies toBash 5.x and curl 8.x on Linux (Ubuntu 22.04/24.04, RHEL 9)
How It Works
A typical curl request consists of:
URL: The resource to request.
Request method: Defaults to GET; use --request POST for other methods.
Headers: Added with --header.
Payload: Supplied with --data, --data-binary, or --upload-file.
Output handling: The response body is written to standard output by default or to a file with --output.
Failure detection: --fail-with-body makes curl exit unsuccessfully for HTTP 4xx and 5xx responses while preserving the response body.
Important exit behavior:
Network, DNS, TLS, and timeout failures produce a nonzero curl exit status.
HTTP 404 or 500 responses normally produce exit status 0 unless --fail or --fail-with-body is used.
3xx responses are not failures; --location follows redirects.
--retry retries transient failures such as timeouts and selected HTTP responses, but applications should still use idempotency controls for writes.
Production Script
The following script handles errors, bounds execution time, retries transient failures, restricts redirects and protocols, and stores the response body separately from the HTTP status.
--connect-timeout limits connection establishment, while --max-time limits the entire request. --retry-max-time prevents retries from extending the operation indefinitely.
Use --data-binary @payload.json when the payload must be sent without curl’s newline or form-processing behavior. For large responses, use --output response.json rather than retaining the entire body in a shell variable.
Security and Reliability
Credentials: Prefer a secret manager or a protected curl configuration file. Avoid putting tokens in URLs, because URLs commonly appear in logs and browser history.
TLS: Prefer HTTPS, validate certificates, and avoid disabling verification with --insecure.
Redirects: Restrict redirect targets and limit redirects with --max-redirs. An unrestricted redirect can unexpectedly send credentials to another host.
Logging: Record the status code, duration, and a request or trace ID, but do not log authorization headers or sensitive response bodies.
Retries: Retrying a POST can create duplicate records. Use an idempotency key or perform the operation in a controlled workflow.
Monitoring: Alert on elevated HTTP error rates, latency, timeout counts, and certificate failures rather than treating every nonzero curl exit as the same incident type.
For credentials stored in a mode-0600 curl configuration file:
The configuration file should contain the required header entries and should be readable only by the account running the script.
Bash and POSIX Portability
The script uses Bash-specific features such as arrays and set -o pipefail. pipefail is not part of POSIX sh, and arrays are not portable either. A POSIX sh script should use simpler arguments and explicit status checks:
Use #!/usr/bin/env bash when Bash features are intentional; use #!/bin/sh when the script must run on minimal or non-Bash Unix systems.
Common Pitfalls
Omitting --fail-with-body causes HTTP 404 and 500 responses to look successful.
Omitting timeouts can leave CI jobs or production workers blocked indefinitely.
Using --retry without an idempotency strategy can duplicate writes.
Using --insecure hides certificate or hostname validation failures.
Interpolating untrusted input directly into a shell command can cause command injection; pass URLs and payloads as quoted arguments.
Piping curl directly into another command can obscure failures unless pipefail and explicit error checks are used.
59
How do you parse JSON in Bash with jq?
KnownRevise
jq is a command-line JSON processor that parses JSON into values, transforms them with a filter language, and emits JSON or raw text. In Bash, it is commonly used to extract fields, validate response structures, and safely modify configuration data.
Read full answerHide full answer
Applies tojq 1.7.x, Bash 5.x, and curl 8.x
How jq Processes JSON
jq reads one or more JSON values and applies a filter to each value:
jq '.user.name' response.json
The filter .user.name means:
.user: Select the user object.
.name: Select its name field.
Common filter operations include:
.field: Access an object field.
.items[]: Iterate over an array.
select(.enabled == true): Keep matching elements.
map(...): Transform every element in an array.
length: Return the length of an array, object, or string.
keys, has, and type: Inspect an object’s structure and values.
A missing field normally evaluates to null, so parsing success does not prove the JSON has the expected schema. Validate important fields explicitly.
Useful jq Options
Option
Purpose
-r
Print strings without JSON quotation marks.
-c
Emit compact, single-line JSON.
-e
Set the exit status based on whether jq produced a usable result.
-n
Do not read input; useful when constructing JSON.
-s, --slurp
Read all input documents into one array.
--arg name value
Pass a string argument to a filter.
--argjson name value
Pass a JSON value to a filter.
-S
Sort object keys for deterministic output.
For extracting a scalar string:
jq -r '.user.name' response.json
For passing shell data into jq, use --arg rather than interpolating the value into the filter:
jq --arg environment production \
'.environment = $environment' config.json
This prevents shell or input data from being interpreted as jq program code.
Error Handling and Exit Status
Use strict Bash settings in automation scripts:
set -Eeuo pipefail
-e: Exit when a command fails.
-u: Treat unset variables as errors.
pipefail: Detect failures in any stage of a pipeline.
-E: Preserve function and trap behavior for error handlers.
With jq -e:
Exit status 0 means the final result was neither false nor null.
Exit status 1 means the final result was false or null.
Exit status 4 means no valid result was produced.
Invalid JSON and invalid jq programs also produce non-zero statuses.
Do not rely exclusively on set -e; explicitly handle expected jq failures. This is especially important in conditionals and command substitutions, where Bash temporarily disables the usual errexit behavior.
Code Example
The following Bash 5 script creates sample JSON, extracts data safely, and produces formatted output:
Validate before using values: Check types and required fields with jq -e. jq is not a complete JSON Schema validator; use a schema validator such as ajv for complex contracts.
Bound remote input: Download with strict transport and a timeout. --proto-redir '=https' prevents HTTPS redirects to HTTP, while --max-filesize rejects a body whose advertised size exceeds the limit. It is not a hard cap when the response size is unknown or the body is chunked, so enforce a strict limit at a proxy, streaming wrapper, or storage boundary as well:
Write atomically: When updating configuration, write to a temporary file in the same filesystem, validate it with jq, and then use mv to replace the original.
Protect sensitive data: Avoid logging complete payloads or passing credentials directly in command-line arguments. Redact fields before sending JSON to logs or monitoring systems.
Preserve exit codes: Treat jq failures as pipeline or deployment failures, and emit useful context to stderr without exposing secrets.
Remember portability: jq itself works from POSIX sh, but the example uses Bash features. A POSIX shell script should use #!/bin/sh and avoid arrays, [[ ... ]], and other Bash-only syntax.
60
How can a Bash script load values from a configuration file?
KnownRevise
Bash has no built-in parser for arbitrary configuration formats. For simple deployments, key=value parsing with read and an associative array loads values without executing the file; source is appropriate only when the file contains trusted Bash syntax.
Read full answerHide full answer
Applies toBash 5.x on Linux (Ubuntu 22.04/24.04, RHEL 9)
How Bash Loads Configuration
There are two common approaches:
Approach
Executes file contents
Supported format
Appropriate use
source "$file"
Yes
Bash syntax
Trusted configuration owned by the same team
read plus an associative array
No
Restricted KEY=VALUE
Simple application settings and automation
A dedicated parser
No
JSON, YAML, TOML, INI
Structured or nested configuration
source runs the file in the current shell, so it can overwrite variables, change shell options, define functions, or execute arbitrary commands. For example, source "$CONFIG_FILE" executes the referenced file. Use an absolute or explicitly controlled path because Bash searches PATH when source receives a filename without a slash.
A parser using read treats configuration values as ordinary strings. It does not evaluate commands, expand embedded $variables, remove quotes, or implement JSON/YAML semantics.
Define a Restricted Format
For a simple parser, establish strict rules before loading the file:
Each setting uses KEY=VALUE.
Keys contain only letters, digits, and underscores and cannot begin with a digit.
Blank lines and lines beginning with # are ignored.
Values are literal strings; they may contain spaces or additional = characters.
Duplicate keys are rejected.
Optional whitespace around = and inline comments are not supported unless explicitly parsed.
These restrictions prevent surprising behavior and make validation straightforward.
#!/usr/bin/env bash
set -euo pipefail
usage() {
printf 'Usage: %s CONFIG_FILE\n' "${0##*/}" >&2
}
die() {
printf 'Error: %s\n' "$*" >&2
exit 1
}
if (($# != 1)); then
usage
exit 2
fi
config_file=$1
[[ -f $config_file ]] || die "not a regular file: $config_file"
[[ -r $config_file ]] || die "cannot read: $config_file"
declare -A config=()
line_number=0
# Read one complete line without interpreting whitespace or backslashes.
while IFS= read -r line || [[ -n $line ]]; do
((line_number += 1))
# Normalize Windows CRLF input.
line=${line%$'\r'}
[[ -z $line || $line == \#* ]] && continue
[[ $line == *=* ]] || die "line $line_number: expected KEY=VALUE"
key=${line%%=*}
value=${line#*=}
# Reject invalid or duplicate keys before storing them.
[[ $key =~ ^[A-Za-z_][A-Za-z0-9_]*$ ]] ||
die "line $line_number: invalid key: $key"
if [[ ${config[$key]+present} ]]; then
die "line $line_number: duplicate key: $key"
fi
config["$key"]=$value
done < "$config_file"
# Require every setting used by this script.
for required_key in listen_port environment allowed_cidr; do
[[ -n ${config[$required_key]-} ]] ||
die "missing required key: $required_key"
done
# Validate the port as decimal, including values with leading zeroes.
port=${config[listen_port]}
[[ $port =~ ^[0-9]{1,5}$ ]] || die "listen_port must be numeric"
((10#$port >= 1 && 10#$port <= 65535)) ||
die "listen_port must be between 1 and 65535"
# Validate IPv4 CIDR notation and the prefix length.
cidr=${config[allowed_cidr]}
[[ $cidr =~ ^([0-9]{1,3})[.]([0-9]{1,3})[.]([0-9]{1,3})[.]([0-9]{1,3})/([0-9]{1,2})$ ]] ||
die "allowed_cidr must use IPv4 CIDR notation"
for octet in "${BASH_REMATCH[@]:1:4}"; do
((10#$octet <= 255)) || die "allowed_cidr contains an invalid octet: $cidr"
done
prefix=${BASH_REMATCH[5]}
((10#$prefix <= 32)) || die "allowed_cidr prefix must be between 0 and 32"
printf 'listen_port=%s\n' "$port"
printf 'environment=%s\n' "${config[environment]}"
printf 'allowed_cidr=%s\n' "$cidr"
Restrict permissions: Use mode 0600 for files containing secrets, with ownership limited to the service account. The containing directory should also prevent unauthorized replacement.
Validate values: Successful parsing does not make a port, timeout, hostname, or CIDR valid. Apply domain-specific checks before operational use.
Avoid executing untrusted files: source executes code. Parse non-trusted input instead, and never use eval to process it.
Handle secrets deliberately: Prefer a secret manager or restricted runtime secret injection. Environment variables may be visible to the same user or privileged processes.
Deploy atomically: Replace configuration using an atomic rename or versioned mount. Retain the previous version for rollback and record the deployed version for incident diagnosis.
Monitor and back up: Monitor startup and validation failures without logging values. Maintain access-controlled backups of known-good configuration, and version the schema separately so upgrades can validate or migrate old files safely.
Use text tools only for simple extraction: sed -n 's/^listen_port=//p' "$config_file" works for an exact prefix, but it does not detect duplicates or enforce a schema. grep, sed, and awk are useful for inspection, not structured parsing.
Failure Modes and Portability
set -euo pipefail catches many failures early, but it is not a substitute for explicit validation. Commands used in conditional tests may intentionally return nonzero.
pipefail affects pipelines only; it does not detect an invalid configuration line by itself.
read stores strings. Validate them before arithmetic use, and use 10# to force decimal interpretation of leading-zero values.
Bash associative arrays require Bash 4 or newer and are not available in POSIX sh.
POSIX sh also lacks [[ ... ]], =~, pipefail, and source; use portable test expressions and . instead.
For JSON, YAML, TOML, or nested INI files, use a dedicated parser rather than extending a KEY=VALUE parser with increasingly complicated substitutions.
61
How do you validate Bash arguments and print usage?
KnownRevise
Argument validation checks required options, rejects unexpected values, and returns meaningful exit codes before the script performs any operation. A centralized usage function provides consistent help output, sent to stdout for -h and stderr for errors.
Read full answerHide full answer
Applies toBash 5.x (including Bash 5.1 on RHEL 9); the shown syntax also works in Bash 4.4
Argument Validation Model
A robust Bash script should:
Use getopts for options and validate each option explicitly.
Check that required options were supplied and reject unexpected positional arguments.
Validate semantic constraints such as file existence, readability, and destination permissions.
Quote every variable expansion to preserve filenames containing spaces or shell metacharacters.
Return 0 for success, a usage error for invalid arguments, and distinct codes for input or output failures.
set -euo pipefail provides early failure for unhandled errors, but it does not replace explicit argument checks. In particular, set -e is suppressed for commands tested inside conditionals such as if and while.
Usage and Exit Codes
Successful help request: usage writes to stdout, and the script exits 0.
Invalid arguments: errors and usage are written to stderr, with exit code 64, conventionally EX_USAGE.
Keeping help on stdout and errors on stderr lets callers distinguish documentation from failure and supports Unix pipelines and CI systems. The example below performs validation only; it does not copy the source file.
Bash and POSIX Compatibility
The example uses Bash features such as [[ ... ]], (( ... )), and set -o pipefail. POSIX sh requires [ ... ], no arithmetic (( ... )), and has no pipefail option. Although getopts is POSIX-compatible, rewriting the example for sh still requires replacing the Bash-specific syntax and avoiding Bash arrays and process substitution.
Code Example
#!/usr/bin/env bash
set -euo pipefail
readonly PROGRAM_NAME="${0##*/}"
usage() {
cat <<EOF
Usage: ${PROGRAM_NAME} -s <source> -d <destination> [-v] [-h]
Options:
-s <source> Readable source file
-d <destination> Destination file
-v Print the validated paths
-h Show this help message
EOF
}
source_file=''
destination_file=''
verbose=false
while getopts ':s:d:vh' option; do
case "$option" in
s)
source_file="$OPTARG"
;;
d)
destination_file="$OPTARG"
;;
v)
verbose=true
;;
h)
usage
exit 0
;;
:)
printf 'Error: option "-%s" requires an argument.\n' "$OPTARG" >&2
usage >&2
exit 64
;;
\?)
printf 'Error: unknown option "-%s".\n' "$OPTARG" >&2
usage >&2
exit 64
;;
esac
done
shift "$((OPTIND - 1))"
# Reject positional arguments left after option parsing.
if (( $# != 0 )); then
printf 'Error: unexpected argument "%s".\n' "$1" >&2
usage >&2
exit 64
fi
if [[ -z "$source_file" || -z "$destination_file" ]]; then
printf 'Error: both -s and -d are required.\n' >&2
usage >&2
exit 64
fi
if [[ ! -f "$source_file" ]]; then
printf 'Error: source is not a regular file: %s\n' "$source_file" >&2
exit 66
fi
if [[ ! -r "$source_file" ]]; then
printf 'Error: source is not readable: %s\n' "$source_file" >&2
exit 66
fi
if [[ -d "$destination_file" ]]; then
printf 'Error: destination is a directory: %s\n' "$destination_file" >&2
exit 73
fi
destination_dir=$(dirname -- "$destination_file")
readonly destination_dir
if [[ ! -d "$destination_dir" ]]; then
printf 'Error: destination directory does not exist: %s\n' "$destination_dir" >&2
exit 73
fi
if [[ ! -w "$destination_dir" ]]; then
printf 'Error: destination directory is not writable: %s\n' "$destination_dir" >&2
exit 73
fi
if [[ "$verbose" == true ]]; then
printf 'Validated: "%s" -> "%s"\n' "$source_file" "$destination_file"
fi
Keep validation close to the start of the script so invalid input cannot partially modify production state.
Validate permissions and destination directories before beginning a potentially long copy or deployment.
Use dirname --, mkdir --, cp --, and similar commands to handle paths beginning with -.
Avoid eval; it can reinterpret filenames as commands and creates injection risks.
Treat validation as a time-of-check/time-of-use boundary. In concurrent systems, recheck conditions immediately before the destructive operation.
Define overwrite behavior explicitly. The example permits an existing regular destination, but production scripts should state whether replacement, backup, or failure is required.
If a script supports both options and positional arguments, remember that simple getopts loops generally stop processing options once a positional argument is encountered.
62
How do you build portable POSIX-compliant shell scripts?
KnownRevise
POSIX portability means writing shell code that runs consistently under a conforming POSIX shell, such as dash or ksh93, rather than relying on shell-specific extensions. Quote expansions, use standardized syntax and utilities, handle failures explicitly, and test with the target POSIX shell.
Read full answerHide full answer
Applies toPOSIX.1-2024 (IEEE Std 1003.1-2024) Shell; Bash 5.x for comparison
Core Portability Rules
Use #!/usr/bin/env bash for Bash 5 scripts. For a genuinely POSIX-only deployment, normally use #!/bin/sh; the example below can also be tested with sh backup_file.sh.
Use POSIX syntax:
Use [ ... ] or case, not Bash’s [[ ... ]].
Use name() rather than the non-POSIX function name() form.
Use . script instead of Bash’s source.
Avoid arrays, mapfile, readarray, process substitution, and here-strings.
Quote variable expansions: "$file", "$value", "$@". Unquoted expansions can undergo field splitting and pathname expansion.
Use printf rather than echo, whose handling of options and backslashes varies between implementations.
Set IFS explicitly when parsing predictable input, and use read -r to preserve backslashes.
Use POSIX utilities and options. Avoid extensions such as grep -P, GNU-specific sed behavior, and non-POSIX date formats.
Error Handling and Exit Codes
For a POSIX script, use:
set -eu
-e exits when an ordinary simple command fails. Commands used as if, while, or until conditions, non-final commands in && or || lists, and other specified tested contexts are exempt. A failure in an ordinary loop-body command can still terminate the script.
-u rejects references to unset variables. Use ${value:-default} when an unset value is acceptable.
POSIX sh does not support set -o pipefail; a pipeline’s status is the status of its final command. Check important intermediate operations explicitly.
Bash 5 supports set -euo pipefail, but it does not replace explicit validation.
Use exit 0 for success and document distinct nonzero statuses, such as 2 for invalid usage and 1 for an operational failure.
Portable Text Processing
Use [[:space:]] instead of non-portable regular-expression shorthands such as \s or \d.
Use grep -E for POSIX extended regular expressions; grep -P is not portable.
Use awk for field-based processing. Its fields are whitespace-separated by default; use -F when a specific delimiter is required.
Set LC_ALL=C for deterministic sorting and character classification. Sort unordered output when stable ordering matters:
LC_ALL=C awk '$9 ~ /^[45][0-9][0-9]$/ { count[$9]++ }
END { for (code in count) print code, count[code] }' access.log | LC_ALL=C sort
Code Example
This script validates arguments, checks the source file, creates a private backup directory, and reports the copied file.
#!/usr/bin/env bash
# POSIX-compatible body; test with: sh backup_file.sh
set -eu
umask 077
usage() {
printf 'Usage: %s SOURCE_FILE DESTINATION_DIRECTORY\n' "$0" >&2
}
copy_file() {
src=$1
dst_dir=$2
name=${src##*/}
if [ ! -f "$src" ] || [ ! -r "$src" ]; then
printf 'Source file is missing or unreadable: %s\n' "$src" >&2
return 1
fi
if [ -z "$name" ]; then
printf 'Invalid source path: %s\n' "$src" >&2
return 1
fi
if ! mkdir -p "$dst_dir"; then
printf 'Cannot create destination directory: %s\n' "$dst_dir" >&2
return 1
fi
backup_dir="$dst_dir/$(date -u '+%Y%m%dT%H%M%SZ').$$"
if ! mkdir "$backup_dir"; then
printf 'Cannot create backup directory: %s\n' "$backup_dir" >&2
return 1
fi
if ! cp "$src" "$backup_dir/$name"; then
printf 'Backup copy failed: %s\n' "$src" >&2
return 1
fi
printf 'Created backup: %s\n' "$backup_dir/$name"
}
if [ "$#" -ne 2 ]; then
usage
exit 2
fi
copy_file "$1" "$2"
Example output:
Created backup: /var/backups/myapp/20250308T142233Z.1234/config.yaml
Production Practices and Bash Differences
Bash feature
POSIX-compatible alternative
[[ ... ]]
[ ... ] or case
${value,,}
printf '%s' "$value" | tr '[:upper:]' '[:lower:]'
Arrays
Positional parameters or a controlled temporary file
local
Avoid it or use uniquely prefixed variables
source file
. file
set -o pipefail
Check critical operations explicitly
mapfile or readarray
while IFS= read -r line
echo
printf
Run shellcheck -s sh script.sh in CI and execute tests with the target /bin/sh.
Before OS upgrades, retest scripts because the implementation or version of /bin/sh may change.
Avoid eval, untrusted input, and logging of passwords, tokens, or secrets.
Use restrictive permissions such as umask 077 for backups and temporary files.
POSIX does not define an atomic temporary-file utility; mktemp is widely available but non-POSIX.
Coordinate concurrent runs with locking and define stale-lock recovery.
During incidents, rely on explicit error messages and exit statuses, monitor repeated failures, and verify backup integrity and restoration separately.
63
When should you prefer Python over Bash for automation?
KnownRevise
Python is preferable when automation requires complex data handling, structured error handling, reusable logic, or integrations beyond simple command orchestration. Bash remains the better choice for lightweight Linux automation that primarily connects command-line tools.
Read full answerHide full answer
Applies toBash 5.x and Python 3.x
Core Decision
Use Bash when the script mainly performs operational tasks such as:
Running commands and checking their exit statuses.
Managing files, directories, permissions, and processes.
Piping command output through grep, sed, awk, or similar tools.
Implementing straightforward conditionals, loops, and functions.
Acting as glue for existing CLI tools in CI/CD or incident-response workflows.
Choose Python when the task involves:
Parsing JSON, YAML, XML, CSV, or binary data reliably.
Calling REST APIs or maintaining state across multiple operations.
Implementing reusable business logic, data models, or algorithms.
Working with complex data structures such as dictionaries and lists.
Requiring unit tests, type checking, structured exceptions, or richer logging.
Supporting multiple operating systems without relying on Bash-specific behavior.
Running longer-lived processes that need timeouts, retries, or concurrency.
Bash and Python Compared
Concern
Bash
Python
Primary strength
Connecting Unix command-line tools
General-purpose programming and structured automation
Data handling
Strings, streams, and command output
Lists, dictionaries, objects, and typed data
Error handling
Exit codes, trap, and set -euo pipefail
Exceptions and explicit exception handling
Reusability
Functions work, but shell state can become fragile
Modules, packages, classes, and dependency management
Portability
Best on Linux; Bash features may not work with POSIX sh
Highly portable across Linux, macOS, and Windows
Dependencies
Usually only standard shell utilities
May require packages such as requests or PyYAML
Operational overhead
Minimal; suitable for ephemeral containers
Requires the Python runtime and installed dependencies
Best fit
Small system-administration and CI/CD scripts
Complex integrations, data processing, and maintainable tooling
How They Behave Under the Hood
Bash is a shell interpreter optimized for command execution. A command such as curl or jq is normally a separate process. Quoting, word splitting, glob expansion, and pipelines can therefore change behavior unexpectedly. With set -o pipefail, a pipeline fails if any participating command fails rather than only checking the final command.
Python manages control flow and data directly in its runtime. It can decode JSON into dictionaries, validate values, catch specific exceptions, and decide whether an operation should be retried. When Python launches another process, arguments are passed as a list without a shell by default, avoiding accidental shell expansion.
This difference is especially important for untrusted input:
In Bash, quote expansions such as "$user_input" and avoid eval.
In Python, use subprocess.run(["command", argument], check=True) instead of constructing a shell command string.
Never pass secrets as command-line arguments when they may appear in process listings or logs.
Production Considerations
Python scripts generally provide better observability and failure isolation, but they also introduce operational responsibilities:
Dependencies: Pin dependencies, build reproducible virtual environments or containers, and verify them in CI.
Secrets: Read credentials from the platform’s secret manager or environment variables rather than source files.
Retries: Retry only transient failures and use exponential backoff with jitter; do not blindly repeat non-idempotent operations.
Timeouts: Set explicit timeouts for network calls and external commands.
Logging: Emit structured logs with relevant request IDs, resource names, and timestamps, while excluding credentials.
Exit behavior: Return a nonzero exit status for failed automation so CI/CD and orchestration systems can detect the failure.
Upgrades: Test Python scripts against the runtime version included in the production image.
Bash has a smaller deployment footprint and often fits naturally into Linux images. However, a growing Bash script becomes difficult to test and may hide important state in files, environment variables, and temporary directories.
Portability and Common Pitfalls
Bash is not POSIX sh. Features such as arrays, [[ ... ]], process substitution, and pipefail require Bash. Use #!/usr/bin/env bash and set -euo pipefail for Bash scripts.
set -e has contextual behavior and is not a complete substitute for explicit error handling. Check important commands, preserve failure details, and clean up temporary resources.
Avoid parsing JSON with grep or sed; use jq or Python’s json module.
Do not select Python merely because it is “more powerful.” For a short script that only invokes a few trusted Linux commands, Bash is simpler and easier to operate.
Select Python when the increased maintainability, portability, and reliability justify the additional runtime and dependency management.
64
How do I create timestamped directory backups with 7-day retention?
KnownRevise
Timestamped directory backups combine an immutable archive name with age-based cleanup. The Bash script below creates a compressed tar.gz archive for each source directory and removes matching archives older than seven days.
Read full answerHide full answer
Applies toBash 5 with GNU coreutils, GNU tar, and GNU findutils on Linux (Ubuntu 22.04/24.04, RHEL 9)
How It Works
The script accepts a source directory and a backup root.
It resolves canonical paths with realpath and uses a hash of the canonical source path so sources with identical basenames remain separate.
A UTC timestamp with nanosecond precision identifies when the backup was created.
An mktemp-derived suffix and no-clobber publication prevent an existing archive from being overwritten.
The archive is written to a temporary file and renamed only after tar succeeds.
find removes matching archives whose modification time is older than 10080 minutes, approximately seven days.
Code Example
#!/usr/bin/env bash
set -Eeuo pipefail
umask 077
usage() {
printf 'Usage: %s SOURCE_DIRECTORY BACKUP_ROOT\n' "$0"
}
if (( $# != 2 )); then
usage >&2
exit 2
fi
source_dir=$1
backup_root=$2
if [[ ! -d "$source_dir" ]]; then
printf 'Source is not a directory: %s\n' "$source_dir" >&2
exit 1
fi
if ! source_dir=$(realpath -e -- "$source_dir"); then
printf 'Unable to resolve source directory: %s\n' "$1" >&2
exit 1
fi
if [[ "$source_dir" == "/" ]]; then
printf 'Backing up the filesystem root is not supported.\n' >&2
exit 1
fi
source_parent=$(dirname -- "$source_dir")
source_name=$(basename -- "$source_dir")
mkdir -p -- "$backup_root"
if ! backup_root=$(realpath -e -- "$backup_root"); then
printf 'Unable to resolve backup root: %s\n' "$2" >&2
exit 1
fi
source_hash=$(printf '%s' "$source_dir" | sha256sum)
source_hash=${source_hash%% *}
source_hash=${source_hash:0:12}
backup_dir="$backup_root/${source_name}_${source_hash}"
mkdir -p -- "$backup_dir"
if ! backup_dir=$(realpath -e -- "$backup_dir"); then
printf 'Unable to resolve backup directory: %s\n' "$backup_dir" >&2
exit 1
fi
if [[ "$backup_dir" == "$source_dir" || "$backup_dir" == "$source_dir/"* ]]; then
printf 'Backup directory must not be inside the source directory.\n' >&2
exit 1
fi
tmp_archive=""
cleanup() {
local exit_status=$?
if [[ -n "$tmp_archive" ]]; then
if ! rm -f -- "$tmp_archive"; then
printf 'Failed to remove temporary archive: %s\n' "$tmp_archive" >&2
fi
fi
return "$exit_status"
}
trap cleanup EXIT
timestamp=$(date -u +%Y%m%dT%H%M%S.%NZ)
tmp_archive=$(mktemp "$backup_dir/.backup.XXXXXXXX")
archive_suffix=${tmp_archive##*.}
final_archive="$backup_dir/backup_${timestamp}_${archive_suffix}.tar.gz"
tar -czf "$tmp_archive" -C "$source_parent" -- "$source_name"
if ! mv -n -- "$tmp_archive" "$final_archive"; then
printf 'Unable to publish backup archive: %s\n' "$final_archive" >&2
exit 1
fi
if [[ -e "$tmp_archive" ]]; then
printf 'Archive name collision; refusing to overwrite: %s\n' "$final_archive" >&2
exit 1
fi
tmp_archive=""
find "$backup_dir" \
-maxdepth 1 \
-type f \
-name 'backup_*.tar.gz' \
-mmin +10080 \
-exec rm -f -- {} +
printf 'Created backup: %s\n' "$final_archive"
Created backup: /var/backups/app_a1b2c3d4e5f6/backup_20250308T142530.123456789Z_a1B2c3D4.tar.gz
Production Behavior
The archive contains the source directory itself. Restore it under a chosen parent directory with:
tar -xzf /var/backups/app_a1b2c3d4e5f6/backup_20250308T142530.123456789Z_a1B2c3D4.tar.gz \
-C /restore/location
Cleanup uses the archive file’s modification time, not timestamps inside the archive. A successful backup therefore becomes eligible for deletion approximately seven days after creation. The script can run under cron or a systemd timer; capture exit status and output in centralized logging and monitor free disk space, backup age, creation failures, and cleanup failures.
Failure Modes and Security
set -Eeuo pipefail stops the script on errors such as permission failures, disk exhaustion, or a failed tar operation.
The EXIT trap removes an unpublished temporary archive after failure.
umask 077 prevents newly created directories and archives from being broadly readable.
Mutable application data may produce an inconsistent archive. Quiesce writes or use filesystem and application-consistent snapshots where required.
Local backups do not protect against host or filesystem loss; replicate encrypted copies offsite.
tar does not encrypt archives. Use encryption-at-rest or encrypt the copy before offsite transfer.
Periodically test a full restore; tar -tzf validates readability but does not replace a restore test.
Portability and Best Practices
This is Bash 5-specific rather than POSIX sh: pipefail, GNU realpath, GNU date nanoseconds, GNU find, and GNU mv options are used. It should not be run unchanged on minimal BSD or BusyBox systems.
Use flock when multiple invocations could run concurrently, retain more than one successful copy for recovery testing, and alert on failed or overdue backups rather than assuming that the absence of an alert means backups are healthy.
65
How to monitor disk usage and alert above 80 percent in Bash?
KnownRevise
The script uses disk-usage threshold monitoring: df reads filesystem capacity metadata, Bash compares the reported percentage against 80, and an alert is sent when usage is strictly greater than 80%. It supports multiple mount points and can deliver alerts through stderr or an HTTPS webhook.
Read full answerHide full answer
Applies toBash 5.x and GNU coreutils 8.32+ on Linux (Ubuntu 22.04/24.04, RHEL 9)
How It Works
df --output=pcent returns a header plus the percentage used by one filesystem. GNU df rejects -P combined with --output, so the two are not mixed.
df uses filesystem metadata such as statfs or statvfs; it does not walk every file, so checks are inexpensive.
GNU df rounds the displayed percentage upward. The script checks the reported whole-number percentage against the threshold.
Each configured mount point is checked independently. A different filesystem can have a different threshold.
If df or the webhook fails, the script records a nonzero exit status while continuing to check remaining mount points.
No webhook URL means alerts are written to stderr, which cron can email or a container platform can collect.
Code Example
Save as disk_usage_monitor.sh:
#!/usr/bin/env bash
set -Eeuo pipefail
THRESHOLD_PERCENT="${THRESHOLD_PERCENT:-80}"
ALERT_WEBHOOK_URL="${ALERT_WEBHOOK_URL:-}"
if ! [[ "$THRESHOLD_PERCENT" =~ ^[0-9]+$ ]] ||
(( THRESHOLD_PERCENT < 1 || THRESHOLD_PERCENT > 100 )); then
printf 'ERROR: THRESHOLD_PERCENT must be an integer from 1 to 100\n' >&2
exit 2
fi
readonly THRESHOLD_PERCENT
readonly ALERT_WEBHOOK_URL
HOSTNAME_VALUE="$(hostname -f 2>/dev/null || hostname)"
readonly HOSTNAME_VALUE
json_escape() {
local value="$1"
value=${value//\\/\\\\}
value=${value//\"/\\\"}
value=${value//$'\n'/\\n}
value=${value//$'\r'/\\r}
value=${value//$'\t'/\\t}
printf '%s' "$value"
}
send_alert() {
local mountpoint="$1"
local usage="$2"
local message="$3"
local payload
local usage_json
if [[ "$usage" == "unknown" ]]; then
usage_json=null
else
usage_json="$usage"
fi
printf -v payload \
'{"host":"%s","mountpoint":"%s","usage_percent":%s,"threshold_percent":%s,"message":"%s"}' \
"$(json_escape "$HOSTNAME_VALUE")" \
"$(json_escape "$mountpoint")" \
"$usage_json" \
"$THRESHOLD_PERCENT" \
"$(json_escape "$message")"
printf 'DISK ALERT: %s\n' "$message" >&2
if [[ -n "$ALERT_WEBHOOK_URL" ]]; then
if ! command -v curl >/dev/null 2>&1; then
printf 'ERROR: curl is required for webhook delivery\n' >&2
return 1
fi
if ! curl --fail --silent --show-error \
--connect-timeout 5 \
--max-time 10 \
--header 'Content-Type: application/json' \
--data "$payload" \
--url "$ALERT_WEBHOOK_URL" >/dev/null; then
printf 'ERROR: webhook delivery failed for %s\n' "$mountpoint" >&2
return 1
fi
fi
}
MOUNTPOINTS=("$@")
if (( ${#MOUNTPOINTS[@]} == 0 )); then
MOUNTPOINTS=("/")
fi
status=0
for mountpoint in "${MOUNTPOINTS[@]}"; do
printf 'Checking %s\n' "$mountpoint"
if ! df_output="$(df --output=pcent -- "$mountpoint" 2>&1)"; then
printf 'ERROR: df failed for %s: %s\n' "$mountpoint" "$df_output" >&2
send_alert "$mountpoint" unknown \
"Unable to query disk usage for ${mountpoint}" || status=1
continue
fi
usage="$(awk 'NR == 2 { gsub(/%/, "", $1); print $1 }' <<< "$df_output")"
if ! [[ "$usage" =~ ^[0-9]+$ ]] || (( usage > 100 )); then
send_alert "$mountpoint" unknown \
"Invalid disk-usage result for ${mountpoint}" || status=1
continue
fi
if (( usage > THRESHOLD_PERCENT )); then
send_alert "$mountpoint" "$usage" \
"Disk usage on ${mountpoint} is ${usage}%, above threshold ${THRESHOLD_PERCENT}%" ||
status=1
fi
done
exit "$status"
A successful alert leaves exit status 0; query or delivery failures return 1. This avoids treating a correctly delivered threshold alert as a script failure.
The script alerts on every execution while usage remains high. Configure grouping, suppression, or deduplication in the alert provider to prevent notification storms.
Store webhook credentials outside the script in a protected systemd EnvironmentFile or a secret manager. Require HTTPS and do not disable TLS certificate verification.
Check inode exhaustion separately with df -i; a filesystem can run out of inodes while retaining free data blocks.
Capacity alerts do not replace backups, filesystem health monitoring, or proactive expansion. Use per-filesystem thresholds because application data, logs, and container storage often grow at different rates.
Portability and Common Pitfalls
This is a Bash script, not POSIX sh: it uses arrays, [[ ... ]], pipefail, and Bash-specific parameter expansion.
df --output=pcent is a GNU coreutils feature. BSD or macOS df generally lacks it; use df -P "$mountpoint" and parse the standard output instead.
In containers, check the host’s mount namespace. A container may see only its overlay filesystem while the host’s data volume is approaching capacity.
Avoid hard-coded webhook URLs, credentials, or mount paths in source control. Separate environment-specific configuration from the reusable script.
66
How do you script top IP counts from nginx access logs?
KnownRevise
Top IP counting identifies which client addresses generated the most requests by aggregating the first field of each nginx access-log entry. The following Bash script handles arguments safely, fails on pipeline errors, supports IPv4 and IPv6 addresses, and returns the highest-volume addresses first.
Read full answerHide full answer
Applies toBash 5.x on Linux (Ubuntu 22.04/24.04, RHEL 9); nginx open-source 1.x using the default `combined` log format
Approach and Assumptions
The default nginx combined log format begins with $remote_addr, so awk can treat the first whitespace-separated field as the client IP.
The script:
Accepts a log file as the first argument.
Accepts the number of addresses to display as the second argument, defaulting to 10.
Uses set -euo pipefail so undefined variables, unsuccessful commands, and pipeline failures are detected.
Counts every request represented by a log line; it does not calculate unique visitors.
Sorts by request count descending and breaks ties by IP address.
Code Example
Save this as count_nginx_ips.sh:
#!/usr/bin/env bash
set -euo pipefail
export LC_ALL=C
usage() {
printf 'Usage: %s ACCESS_LOG [TOP_COUNT]\n' "${0##*/}" >&2
}
if (( $# < 1 || $# > 2 )); then
usage
exit 64
fi
log_file=$1
top_n=${2:-10}
if [[ ! $top_n =~ ^[1-9][0-9]*$ ]]; then
printf 'TOP_COUNT must be a positive integer.\n' >&2
exit 64
fi
if [[ ! -f $log_file || ! -r $log_file ]]; then
printf 'Cannot read regular log file: %s\n' "$log_file" >&2
exit 66
fi
# Prevent a filename beginning with "-" from being interpreted as an option.
case "$log_file" in
/*) input_path=$log_file ;;
*) input_path="./$log_file" ;;
esac
awk '
{
ip = $1
if (ip != "" && ip != "-") {
counts[ip]++
}
}
END {
for (ip in counts) {
printf "%.0f\t%s\n", counts[ip], ip
}
}' "$input_path" |
sort -t $'\t' -k1,1nr -k2,2 |
awk -v limit="$top_n" 'NR <= limit { print }'
Example output has the request count followed by the address:
95 203.0.113.10
71 2001:db8::42
43 198.51.100.25
Under the Hood
awk stores each address in an associative array and increments its counter. The END block emits one count<TAB>IP record per unique address. sort orders the records numerically by descending count, while the final awk retains the requested number of results.
Unlike piping into head, this implementation lets the final awk consume the complete sorted stream. That avoids SIGPIPE-related false failures when pipefail is enabled.
Production Considerations
Proxies and realip: $remote_addr may represent a load balancer rather than the end client. Enable nginx’s realip module only for explicitly trusted proxy ranges; trusting arbitrary X-Forwarded-For values allows clients to spoof addresses.
Shared addresses: One IP can represent many users through NAT, VPN, mobile carrier, or CDN infrastructure. Do not block an address solely because it has a high request count.
Active-file consistency: Log rotation can occur while the file is being read. For incident analysis, preserve and process the relevant rotated file or an immutable snapshot.
Volume: The awk array uses memory proportional to the number of unique IPs, while sort may require temporary disk space. Very high-cardinality logs may need an external analytics pipeline.
Compression: This script expects an uncompressed regular file. Compressed rotated logs must first be processed with gzip -cd.
Failure Modes and Best Practices
Keep the log on the same reliable storage used during incidents, and retain rotated archives according to the organization’s investigation requirements.
Monitor sudden changes in request volume per address; a sharp increase may indicate scanning or abuse, but should be correlated with nginx status codes, request paths, and upstream logs.
Validate the log format after nginx configuration or version upgrades. If the format changes and $remote_addr is no longer first, update the extractor.
Use UTC timestamps across infrastructure so request spikes can be correlated with deployments, incidents, and monitoring alerts.
Treat IP addresses as operational and potentially personal data. Restrict access to detailed logs and avoid publishing raw client-address reports publicly.
Keep PATH predictable when running the script from automation, but avoid unnecessary privilege. Read-only access to the log file is normally sufficient.
67
How can Bash check whether websites return HTTP 200?
KnownRevise
A reliable HTTP 200 check in Bash sends an HTTP request to each URL with curl and inspects the returned status code. The script below handles timeouts, network errors, invalid URLs, optional redirects, and meaningful exit codes.
Read full answerHide full answer
Applies toBash 5.x with curl 8.x on Linux
How It Works
The script reads one URL per line from a file and uses curl’s default GET method.
For each URL, curl resolves DNS, establishes a TCP or TLS connection, sends the request, and writes the status code using %{http_code}.
A URL passes only when the response code is exactly 200 and curl completed without a transport or protocol error.
Redirects are not followed by default. Set FOLLOW_REDIRECTS=1 to check the final destination after up to five redirects.
The script exits 0 only when at least one valid URL was checked and every request and validation check passed. It exits 1 for a failed check when at least one valid URL exists, 64 for usage or configuration errors, 65 when no valid URL was found, and 66 when the input file is unreadable.
Failure Modes and Production Behavior
Timeouts: --connect-timeout limits connection establishment, while --max-time limits the complete operation, including DNS, connection, TLS, and transfer.
HTTP errors: Responses such as 404, 500, or 503 are reported as failures even though curl successfully received them.
Network errors: DNS failures, refused connections, TLS errors, and timeouts produce a status such as 000 plus the curl exit code.
Error handling: set -euo pipefail stops on unhandled command failures, unset-variable errors, and failed pipeline components. The curl invocation is deliberately placed in an if condition so its exit status can be captured without terminating the script.
Monitoring: Exit status works for CI/CD, but production monitoring should also capture per-URL results, timestamps, duration, and the distinction between HTTP and network failures.
Load: Requests run sequentially, limiting accidental load. For large lists, use bounded concurrency rather than unbounded parallel requests.
Security and Operational Trade-offs
--proto '=http,https' prevents protocols such as file and ftp from being requested.
--disable prevents a user-level ~/.curlrc from silently changing curl behavior.
URLs containing embedded credentials are rejected to avoid leaking secrets into output.
An untrusted URL list can still target internal services or cloud metadata endpoints. Validate destinations against an allowlist and enforce network egress controls.
Avoid placing tokens or sensitive query parameters in URLs because successful and failed URLs are printed.
Code Example
Save this as check-http-200.sh:
#!/usr/bin/env bash
set -euo pipefail
CONNECT_TIMEOUT_SECONDS=5
MAX_TIME_SECONDS=15
FOLLOW_REDIRECTS=${FOLLOW_REDIRECTS:-0}
readonly CONNECT_TIMEOUT_SECONDS
readonly MAX_TIME_SECONDS
readonly FOLLOW_REDIRECTS
usage() {
printf 'Usage: %s <url-list-file>\n' "${0##*/}" >&2
exit 64
}
[[ $# -eq 1 ]] || usage
input_file=$1
if [[ ! -r $input_file ]]; then
printf 'ERROR: cannot read %s\n' "$input_file" >&2
exit 66
fi
curl_options=(
--disable
--silent
--show-error
--globoff
--output /dev/null
--write-out '%{http_code}'
--connect-timeout "$CONNECT_TIMEOUT_SECONDS"
--max-time "$MAX_TIME_SECONDS"
--proto '=http,https'
)
if [[ $FOLLOW_REDIRECTS == 1 ]]; then
curl_options+=(--location --max-redirs 5 --proto-redir '=http,https')
elif [[ $FOLLOW_REDIRECTS != 0 ]]; then
printf 'ERROR: FOLLOW_REDIRECTS must be 0 or 1\n' >&2
exit 64
fi
error_file=$(mktemp)
trap 'rm -f "$error_file"' EXIT
failures=0
checked=0
line_number=0
while IFS= read -r url || [[ -n $url ]]; do
line_number=$((line_number + 1))
if [[ -z $url || $url == \#* ]]; then
continue
fi
if [[ $url =~ [[:space:]] ]] ||
[[ ! $url =~ ^https?://([^/?#]+)([/?#].*)?$ ]]; then
printf 'INVALID URL on line %d\n' "$line_number" >&2
failures=$((failures + 1))
continue
fi
authority=${BASH_REMATCH[1]}
if [[ $authority == *@* ]]; then
printf 'INVALID URL on line %d: embedded credentials are not allowed\n' \
"$line_number" >&2
failures=$((failures + 1))
continue
fi
checked=$((checked + 1))
: >"$error_file"
if http_code=$(curl "${curl_options[@]}" --url "$url" 2>"$error_file"); then
curl_exit=0
else
curl_exit=$?
fi
error_message=$(tr '\n' ' ' <"$error_file")
if [[ $http_code == 200 && $curl_exit -eq 0 ]]; then
printf 'PASS %s HTTP 200\n' "$url"
else
reason="HTTP ${http_code:-000}"
if (( curl_exit != 0 )); then
reason+="; curl exit ${curl_exit}: ${error_message:-request failed}"
fi
printf 'FAIL %s %s\n' "$url" "$reason"
failures=$((failures + 1))
fi
done <"$input_file"
if (( checked == 0 )); then
printf 'ERROR: no valid URLs found\n' >&2
exit 65
fi
if (( failures > 0 )); then
exit 1
fi
Run shellcheck check-http-200.sh before deployment.
Use FOLLOW_REDIRECTS=1 only when the final destination should determine success.
Add controlled retries for transient network failures, but avoid retrying permanent 4xx responses.
Keep timeout values conservative so one slow site cannot stall a monitoring run.
Upgrade curl and the operating system’s CA trust store regularly so TLS validation remains reliable.
Portability: This script intentionally uses Bash-only arrays, [[ ... ]], regular expressions, and BASH_REMATCH; run it with bash or its shebang, not with sh. A POSIX sh implementation must replace those constructs.
Text processing: Use grep to select matching records, sed for simple substitutions, and awk for field-based extraction or aggregation. Quote patterns and filenames, and prefer their portable POSIX options over GNU-specific behavior.
68
How do I rename all .txt files to .md in Bash?
KnownRevise
A safe Bash rename script should quote filenames, detect destination conflicts before making changes, and avoid overwriting existing .md files. The script below processes regular files and symbolic links directly inside one directory without recursion.
Read full answerHide full answer
Applies toBash 5.x on Linux with GNU coreutils 9.x
How It Works
Accepts one directory argument and changes into it with cd -P.
Uses nullglob so unmatched patterns do not produce literal filenames.
Uses dotglob so hidden entries such as .draft.txt are included.
Preflights every rename and aborts before changing anything if a corresponding .md file already exists.
Uses GNU mv -n -T to avoid overwriting a destination and prevent accidental placement inside a directory.
Handles spaces, tabs, and newlines in filenames through Bash arrays and quoted expansions.
Code Example
Save as rename_txt_to_md.sh:
#!/usr/bin/env bash
set -euo pipefail
usage() {
printf 'Usage: %s DIRECTORY\n' "${0##*/}" >&2
}
if (( $# != 1 )); then
usage
exit 64
fi
directory=$1
if [[ ! -d "$directory" ]]; then
printf 'Not a directory: %q\n' "$directory" >&2
exit 66
fi
cd -P -- "$directory"
shopt -s nullglob dotglob
declare -a rename_pairs=()
# Precalculate all operations and reject conflicts before renaming.
for source in ./*.txt; do
if [[ ! -f "$source" && ! -L "$source" ]]; then
continue
fi
target="${source%.txt}.md"
if [[ -e "$target" || -L "$target" ]]; then
printf 'Refusing to overwrite: %q\n' "$target" >&2
exit 1
fi
rename_pairs+=("$source" "$target")
done
if (( ${#rename_pairs[@]} == 0 )); then
printf 'No .txt files found in %q\n' "$PWD"
exit 0
fi
for ((i = 0; i < ${#rename_pairs[@]}; i += 2)); do
source="${rename_pairs[$i]}"
target="${rename_pairs[$i + 1]}"
# Recheck immediately before the operation.
if [[ -e "$target" || -L "$target" ]]; then
printf 'Destination appeared before rename: %q\n' "$target" >&2
exit 1
fi
if ! command mv -n -T -- "$source" "$target"; then
printf 'Failed to rename %q to %q\n' "$source" "$target" >&2
exit 1
fi
# GNU mv -n can report success when it skips an existing destination.
if [[ -e "$source" || -L "$source" ]]; then
printf 'Rename was not completed: %q -> %q\n' \
"$source" "$target" >&2
exit 1
fi
printf 'Renamed %q -> %q\n' "$source" "$target"
done
The operation is non-recursive; files in subdirectories are unchanged.
Only the final lowercase .txt suffix is replaced. .TXT is not matched on case-sensitive Linux filesystems.
mv performs a same-filesystem rename, changing the directory entry rather than rewriting file contents. Permissions, ownership, timestamps, and inode data are normally preserved.
Exit status 0 indicates success; nonzero status indicates invalid arguments, an existing destination, or a failed rename.
Output goes to standard output, while errors go to standard error, making the script suitable for CI logs and operational tooling.
Failure Modes and Best Practices
Partial completion: Preflighting prevents destination conflicts, but an I/O or permission failure can still occur after earlier files have been renamed. For critical directories, maintain a manifest and use version control or backups.
Concurrent changes: Another process can create a destination between the preflight and rename. mv -n prevents overwriting it, while the post-operation check detects a skipped rename. A directory-level lock provides stronger isolation.
Privileged execution: Run with the least privileges required. Avoid running the script as root over directories writable by untrusted users.
Portability: Arrays, shopt, and nullglob are Bash features rather than POSIX sh features. GNU mv -T is also Linux-specific; BSD and macOS versions of mv do not support that option.
69
How do I recursively delete empty directories with Bash?
KnownRevise
A Bash script can use GNU find to locate empty directories and remove them safely. The script should validate the target, protect the filesystem root, avoid descending into mounted filesystems, and quote every path to prevent word-splitting or glob-expansion errors.
Read full answerHide full answer
Applies toBash 5.x with GNU findutils 4.x and GNU coreutils 8.32+ on Linux
How It Works
The script processes one target directory:
set -euo pipefail makes the script stop on unexpected errors, unset variables, and failed pipeline components.
realpath -e -- "$target" converts the argument into an absolute canonical path and resolves symbolic links.
The filesystem root / is rejected to prevent an accidental recursive cleanup of the entire machine.
find traverses the target with these important options:
-mindepth 1: excludes the target directory itself.
-depth: visits children before their parents, allowing a parent to become empty and be removed later.
-type d: considers only directories.
-empty: matches directories containing no entries, including hidden files.
-xdev: prevents traversal into nested mounted filesystems.
-delete: removes each matching directory; GNU find exits nonzero if deletion fails.
Code Example
#!/usr/bin/env bash
set -euo pipefail
if (( $# != 1 )); then
printf 'Usage: %s DIRECTORY\n' "${0##*/}" >&2
exit 64
fi
target="$1"
if [[ ! -d "$target" ]]; then
printf 'Error: target is not an existing directory: %s\n' "$target" >&2
exit 66
fi
# Resolve the target to an absolute path and resolve directory symlinks.
target="$(realpath -e -- "$target")"
# Never allow the filesystem root to be used as the cleanup target.
if [[ "$target" == "/" ]]; then
printf 'Error: refusing to operate on the filesystem root\n' >&2
exit 1
fi
# Delete matching descendants bottom-up without descending into nested mounts.
find "$target" \
-xdev \
-depth \
-mindepth 1 \
-type d \
-empty \
-delete
Save it as remove-empty-dirs.sh, make it executable, and run it against a specific path:
Deletion is not transactional. GNU find can report an individual deletion error and continue processing other entries; because find is the final command, set -e makes the script exit with its nonzero status after find completes. In production:
Run the cleanup with an account that has only the required permissions.
Avoid running it as root unless that is explicitly necessary.
Take a snapshot or backup when the directory tree is operationally important.
Schedule the operation outside peak-traffic periods.
Capture stderr and the exit status in the automation or job-monitoring system.
Remember that deleting a child requires write and execute permission on its parent directory.
Failure Modes and Security
Permission denied: find reports the failure and exits nonzero.
Concurrent changes: A directory may stop being empty after find evaluates it; -delete then fails rather than deleting non-empty content.
Symbolic links: GNU find does not follow directory symlinks by default, reducing the risk of traversal outside the intended tree.
Mounted filesystems: -xdev prevents descending into nested mount points. The mount-point directory itself can still be tested, so -delete may fail there with a busy-device error.
Hidden content: Hidden files and directories count as content, so directories containing them are not considered empty.
Path safety: The canonical-path and root checks reduce accidental damage, but they do not replace backups, least privilege, or change-management controls.
Portability
This implementation targets Bash 5 and GNU tools. POSIX sh does not support set -o pipefail, and the script depends on GNU realpath -e --. BSD and macOS find do provide -empty and -delete, but the complete script is still not portable to those environments: macOS uses Bash 3.2 rather than Bash 5 and does not provide the same GNU toolchain.
70
How to write a Bash script that restarts an inactive systemd service?
KnownRevise
A Bash script can use systemctl to check whether a service is active and restart it when necessary. For an inactive service, systemctl restart starts it; for a failed service, it clears the failed state and attempts a fresh start.
Read full answerHide full answer
Applies toBash 5.x and systemd on Ubuntu 22.04/24.04 or RHEL 9
How It Works
The script performs four operations:
Accepts an optional service unit, defaulting to nginx.service.
Validates the unit name and confirms that systemd has loaded it.
Uses systemctl is-active --quiet to check the current state.
Restarts an inactive service and verifies that it becomes active.
set -Eeuo pipefail stops execution on undefined variables, unexpected failures, and failed pipeline components. Because systemctl is-active intentionally returns a nonzero status for an inactive service, it is used inside an if condition, where set -e does not terminate the script.
Code Example
#!/usr/bin/env bash
set -Eeuo pipefail
usage() {
printf 'Usage: %s [service.unit]\n' "${0##*/}" >&2
}
if (( $# > 1 )); then
usage
exit 64
fi
SERVICE=${1:-nginx.service}
readonly SERVICE
# Permit conventional unit names and prevent option injection.
if [[ ! $SERVICE =~ ^[a-zA-Z0-9][a-zA-Z0-9_.@-]*$ ]]; then
printf 'Invalid service name: %s\n' "$SERVICE" >&2
usage
exit 64
fi
load_state="$(
systemctl show "$SERVICE" --property=LoadState --value 2>/dev/null
)" || {
printf 'Unable to query systemd unit: %s\n' "$SERVICE" >&2
exit 1
}
if [[ $load_state != "loaded" ]]; then
printf 'Systemd unit is not loaded: %s\n' "$SERVICE" >&2
exit 1
fi
if systemctl is-active --quiet "$SERVICE"; then
printf '%s is active; no action needed.\n' "$SERVICE"
exit 0
fi
printf '%s is not active; restarting...\n' "$SERVICE"
if ! systemctl restart "$SERVICE"; then
printf 'Failed to restart %s.\n' "$SERVICE" >&2
exit 1
fi
if systemctl is-active --quiet "$SERVICE"; then
printf '%s restarted successfully.\n' "$SERVICE"
else
printf '%s did not become active after restart.\n' "$SERVICE" >&2
exit 1
fi
Example output:
$ sudo ./ensure-service-running.sh nginx.service
nginx.service is not active; restarting...
nginx.service restarted successfully.
Production Considerations
Permissions: The script normally needs root or an appropriate sudoers rule to restart a system service. Do not make the script setuid; use a controlled privileged runner instead.
Boot behavior: This script does not enable the service at boot. Use systemctl enable nginx.service when the service should start automatically after a reboot.
Health checks: is-active reports systemd’s process state, not application health. A service can be active while its HTTP endpoint returns errors. Use an application-specific health check when that distinction matters.
Existing restart policy: A unit configured with Restart=on-failure already asks systemd to restart failed processes. This script is mainly useful for boot reconciliation, detecting an externally stopped service, or implementing an external health check.
Monitoring: Record restart events and alert on repeated failures. Relevant logs are available through journalctl -u nginx.service.
Failure Modes and Security
A missing or masked unit is rejected before attempting a restart.
A restart can still fail because of invalid configuration, missing dependencies, resource exhaustion, or a failed ExecStart; the script returns a nonzero exit status in that case.
Repeated automatic restarts can create a restart loop and hide the underlying defect. Production automation should include backoff, thresholds, and alerting.
Service names are validated and variables are quoted to prevent word splitting and option injection. A privileged wrapper should still restrict callers to an allowlist of units they may restart.
systemctl restart can briefly interrupt traffic. It is not a substitute for graceful application draining, rolling deployment, or database-aware failover.
Bash and POSIX Compatibility
This script intentionally uses Bash features such as [[ ... =~ ... ]], arithmetic conditions, and set -o pipefail, so it is not portable to POSIX sh. A simplified version can use #!/bin/sh with set -eu, but Bash remains preferable when complex validation or consistent error handling is required. The dependency on systemctl also limits the script to Linux systems using systemd.
71
How can a Bash script sum a numeric column from a CSV file?
KnownRevise
A production-ready approach is to use Bash for argument validation and orchestration while delegating CSV parsing to Python’s standard csv module. This handles quoted fields, embedded commas, escaped quotes, and newlines while retaining exact arithmetic for decimal inputs.
Read full answerHide full answer
Applies toBash 5.x with Python 3.9+
How It Works
The script:
Accepts a CSV file and a 1-based column number.
Treats the first row as the header and ignores blank records.
Rejects inconsistent row widths, invalid decimal values, and common malformed CSV records.
Prints only the sum to standard output; errors and diagnostics go to standard error.
Uses set -euo pipefail, quoted expansions, a usage function, positional arguments, conditionals, and explicit exit codes: 64 for invalid usage, 127 for a missing dependency, and 1 for processing failures.
Python’s csv module is used because splitting a CSV line with IFS=, is unreliable: quoted values may contain commas, and quoted fields may span multiple lines.
Bash and POSIX Differences
The script relies on Bash-only features such as [[ ... ]], (( ... )), and set -o pipefail; it will not run unchanged under a POSIX /bin/sh. In POSIX sh, use [ ... ], case, and awk, and do not rely on pipefail.
A line-oriented Bash loop commonly uses while IFS= read -r line || [[ -n $line ]]; do with done < "$file". However, splitting such lines with IFS=, is not CSV parsing, and Bash arithmetic is integer-only, so it is unsuitable for decimal totals.
For a prevalidated CSV containing only simple, single-line numeric fields, POSIX awk can calculate a total:
awk -F, -v column=2 'NR == 1 { next } { total += $column } END { print total }' data.csv
This is simpler but uses floating-point arithmetic and does not correctly parse quoted CSV records. grep and sed are useful for line-oriented prevalidation or preprocessing, but they cannot safely parse multiline CSV fields.
Code Example
#!/usr/bin/env bash
set -euo pipefail
usage() {
printf 'Usage: %s <csv-file> <column-number>\n' "$0" >&2
}
if (( $# != 2 )); then
usage
exit 64
fi
csv_file=$1
column_number=$2
if [[ ! -f "$csv_file" ]]; then
printf 'error: CSV file not found: %s\n' "$csv_file" >&2
exit 1
fi
if [[ ! "$column_number" =~ ^[1-9][0-9]*$ ]]; then
printf 'error: column number must be a positive integer\n' >&2
exit 64
fi
if ! command -v python3 >/dev/null 2>&1; then
printf 'error: python3 is required\n' >&2
exit 127
fi
python3 - "$csv_file" "$column_number" <<'PY'
import csv
import sys
from decimal import Decimal, InvalidOperation
from fractions import Fraction
path = sys.argv[1]
column_number = int(sys.argv[2])
try:
# utf-8-sig removes an optional UTF-8 BOM; newline="" preserves CSV semantics.
with open(path, "r", encoding="utf-8-sig", newline="") as csv_file:
reader = csv.reader(csv_file, strict=True)
try:
header = next(reader)
except StopIteration:
raise ValueError("CSV file is empty")
if column_number > len(header):
raise ValueError(
f"column {column_number} exceeds header width {len(header)}"
)
total = Fraction(0)
values_read = 0
for row_number, row in enumerate(reader, start=2):
if not row:
continue
if len(row) != len(header):
raise ValueError(
f"row {row_number}: expected {len(header)} fields, got {len(row)}"
)
raw_value = row[column_number - 1].strip()
if not raw_value:
continue
try:
# Decimal rejects rational forms such as 1/2 and preserves
# the exact decimal value without applying context rounding.
value = Fraction(Decimal(raw_value))
except (InvalidOperation, OverflowError, ValueError, ZeroDivisionError):
raise ValueError(
f"row {row_number}: invalid numeric value {raw_value!r}"
)
total += value
values_read += 1
# Finite decimal inputs have denominators containing only factors of 2
# and 5, allowing the exact rational total to be rendered as a decimal.
denominator = total.denominator
power_of_two = 0
power_of_five = 0
while denominator % 2 == 0:
denominator //= 2
power_of_two += 1
while denominator % 5 == 0:
denominator //= 5
power_of_five += 1
if denominator != 1:
raise ValueError("result cannot be represented as a finite decimal")
scale = max(power_of_two, power_of_five)
scaled_numerator = total.numerator * (2 ** (scale - power_of_two))
scaled_numerator *= 5 ** (scale - power_of_five)
sign = "-" if scaled_numerator < 0 else ""
digits = str(abs(scaled_numerator)).zfill(scale + 1)
if scale:
result = f"{sign}{digits[:-scale]}.{digits[-scale:]}"
else:
result = f"{sign}{digits}"
print(result)
print(f"values summed: {values_read}", file=sys.stderr)
except (OSError, UnicodeError, csv.Error, ValueError) as error:
print(f"error: {error}", file=sys.stderr)
sys.exit(1)
PY
The script also writes values summed: 2 to standard error.
Production Considerations
Failure behavior: The script fails closed on malformed data instead of silently producing an incorrect total. This is important for billing, capacity, and financial calculations.
Observability: Keep standard output machine-readable for command substitution. Record the exit status, runtime, input identity, and checksums in CI/CD or scheduler logs.
Large files: csv.reader streams records rather than loading the whole file. Only the running total and current record are retained.
Security: Inputs are quoted and no eval is used. Avoid logging sensitive CSV contents, and restrict permissions on files containing confidential data.
Maintenance: Store the script in version control, run ShellCheck, and test fixtures containing quoted commas, CRLF endings, blank cells, BOMs, and malformed rows. Pin and test the Python version used by CI before upgrading it.
Recovery: The script never modifies its input, so it needs no local backup. During incident analysis, preserve the original file and its checksum; any upstream data-retention or backup policy remains authoritative.
72
How to wait for a TCP port with Bash
KnownRevise
A TCP readiness check repeatedly attempts a TCP connection until the handshake succeeds or a deadline expires. This script uses Bash’s /dev/tcp support and GNU timeout, avoiding a dependency on nc.
Read full answerHide full answer
Applies toBash 5.x built with network-redirection support on Linux, with GNU coreutils 9.x
How It Works
Validates the host, port, and timeout values.
Attempts a TCP connection to HOST:PORT.
Limits each connection attempt with timeout so an unreachable or filtered host cannot block indefinitely.
Retries with a configurable delay until the total timeout expires.
Exits with 0 when the port becomes reachable, 1 on timeout, and 2 for invalid arguments.
This checks whether a TCP listener is reachable. It does not mean the local port is free to bind, nor does it prove that the application behind the port is healthy.
Code Example
Save this as wait-for-port.sh:
#!/usr/bin/env bash
set -euo pipefail
usage() {
printf 'Usage: %s HOST PORT [TOTAL_TIMEOUT] [CONNECT_TIMEOUT] [RETRY_DELAY]\n' \
"$0" >&2
}
if (( $# < 2 || $# > 5 )); then
usage
exit 2
fi
host=$1
port=$2
total_timeout=${3:-60}
connect_timeout=${4:-1}
retry_delay=${5:-2}
if [[ -z "$host" ]]; then
printf 'HOST must not be empty.\n' >&2
exit 2
fi
if ! [[ "$port" =~ ^[0-9]+$ ]]; then
printf 'PORT must be an integer.\n' >&2
exit 2
fi
port=$((10#$port))
if (( port < 1 || port > 65535 )); then
printf 'PORT must be between 1 and 65535.\n' >&2
exit 2
fi
for value_name in total_timeout connect_timeout retry_delay; do
value=${!value_name}
if ! [[ "$value" =~ ^[0-9]+$ ]]; then
printf '%s must be an integer.\n' "$value_name" >&2
exit 2
fi
printf -v "$value_name" '%d' "$((10#$value))"
done
if (( total_timeout < 1 || connect_timeout < 1 || retry_delay < 1 )); then
printf 'Timeouts and retry delay must be at least 1 second.\n' >&2
exit 2
fi
printf 'Waiting up to %d seconds for TCP port %d...\n' \
"$total_timeout" "$port"
start_time=$SECONDS
while :; do
remaining=$((total_timeout - (SECONDS - start_time)))
(( remaining > 0 )) || break
attempt_timeout=$connect_timeout
if (( attempt_timeout > remaining )); then
attempt_timeout=$remaining
fi
# The child receives host and port as positional arguments.
# Bash opens a TCP stream through /dev/tcp when the handshake succeeds.
if timeout "${attempt_timeout}s" \
bash -c "exec 3<>\"/dev/tcp/\$1/\$2\"" \
_ "$host" "$port" 2>/dev/null; then
if [[ "$host" == *:* ]]; then
printf 'TCP port is reachable: [%s]:%s\n' "$host" "$port"
else
printf 'TCP port is reachable: %s:%s\n' "$host" "$port"
fi
exit 0
fi
remaining=$((total_timeout - (SECONDS - start_time)))
(( remaining > 0 )) || break
sleep_for=$retry_delay
if (( sleep_for > remaining )); then
sleep_for=$remaining
fi
sleep "$sleep_for"
done
printf 'Timed out after %d seconds waiting for TCP port %s:%s\n' \
"$total_timeout" "$host" "$port" >&2
exit 1
Waiting up to 60 seconds for TCP port 5432...
TCP port is reachable: db.example.com:5432
Under the Hood
/dev/tcp/HOST/PORT is a Bash-special redirection target rather than a normal device file. Bash performs a TCP connection and opens file descriptor 3 when the handshake succeeds.
The child closes the socket immediately after a successful check, so it sends no application data and does not keep the connection open. The peer can still accept and log the transient connection.
GNU timeout terminates an individual connection attempt that exceeds CONNECT_TIMEOUT.
The outer loop recalculates the remaining time, keeping retries bounded by TOTAL_TIMEOUT.
Production Considerations
Check
What it establishes
What remains unverified
TCP handshake
A listener is reachable
Protocol and application health
HTTP health check
The endpoint returns an acceptable response
Other application dependencies
Local ss check
A local process is listening
Remote reachability and service health
For an HTTP service, supplement the port check with an application-level request such as curl -fsS --max-time 2 https://example.com/health. For Kubernetes, prefer startupProbe and readinessProbe; for systemd, an ExecStartPre= wait script can prevent premature startup.
If the requirement is specifically that a local port can be bound, a connection check is the wrong test. Inspect local listeners with ss, or attempt the actual bind because any check-then-bind sequence has a race.
Failure Modes and Best Practices
Use finite deadlines: Infinite waits can leave CI jobs and deployments hanging indefinitely.
Distinguish readiness from liveness: A reachable port does not guarantee that workers, databases, or downstream dependencies are healthy.
Monitor timeouts: Emit metrics or alerts when the script exits with status 1; repeated timeouts indicate a startup or dependency incident.
Avoid secrets in arguments: Host and port are safe operational metadata, but credentials should not be exposed through process arguments or logs.
Validate untrusted destinations: Positional arguments are quoted and no eval is used, but an allowlist is still appropriate when the host comes from untrusted input.
Expect DNS changes: Each attempt resolves the hostname again, which is useful when service discovery records change.
Remember Bash versus POSIX sh: /dev/tcp is a Bash feature and is unavailable in portable POSIX sh.
73
How do you deploy an application with zero downtime using symlinks?
KnownRevise
A zero-downtime symlink deployment publishes immutable release directories and switches a single current symlink using an atomic symlink swap. On Linux, GNU mv -Tf replaces the symlink through a same-filesystem rename, so requests see either the old release or the new one, never a missing current link.
Read full answerHide full answer
Applies toBash 5.x on Ubuntu 22.04/24.04 or RHEL 9
How It Works
Releases live under /opt/myapp/releases/<release-id>.
/opt/myapp/current is an absolute symlink to the active release.
A temporary symlink is created beside current and renamed over it.
Linux performs the operation with rename(2), atomically replacing the directory entry.
The previous release remains available for rollback and graceful request draining.
The application or web server must resolve the current path when handling requests. For example, Nginx can use root /opt/myapp/current/public;. A server that resolves its document root only at startup must be gracefully reloaded after the switch.
Operational Requirements
Build each release into a new immutable directory; never overwrite a release that may be active.
The release health URL must check that exact release, ideally returning its release ID rather than an unconditional HTTP 200.
The public health URL checks the release selected through current.
All deployments must use the same flock lock to prevent concurrent switches.
Retain several previous releases. Delete old releases only after the longest request, worker, and rollback window has elapsed.
Database changes should use backward-compatible expand-and-contract migrations; a symlink switch cannot make an incompatible schema migration safe.
Code Example
The script assumes the release is already copied into the application’s releases directory.
#!/usr/bin/env bash
set -euo pipefail
IFS=$'\n\t'
usage() {
printf 'Usage: %s RELEASE_DIR APP_ROOT RELEASE_HEALTH_URL PUBLIC_HEALTH_URL\n' \
"${0##*/}"
}
if (( $# != 4 )); then
usage >&2
exit 64
fi
required_commands=(
curl flock ln mkdir mktemp mv readlink realpath rm rmdir
)
for command_name in "${required_commands[@]}"; do
if ! command -v "$command_name" >/dev/null 2>&1; then
printf 'Required command not found: %s\n' "$command_name" >&2
exit 69
fi
done
release_arg=$1
app_root_arg=$2
release_health_url=$3
public_health_url=$4
mkdir -p -- "$app_root_arg/releases"
app_root=$(realpath -e -- "$app_root_arg")
release_dir=$(realpath -e -- "$release_arg")
case "$release_dir" in
"$app_root/releases/"*) ;;
*)
printf 'Release must be inside %s/releases\n' "$app_root" >&2
exit 65
;;
esac
if [[ ! -d "$release_dir" ]]; then
printf 'Release directory does not exist: %s\n' "$release_dir" >&2
exit 66
fi
current="$app_root/current"
if [[ -e "$current" && ! -L "$current" ]]; then
printf '%s exists but is not a symlink\n' "$current" >&2
exit 73
fi
old_target=""
if [[ -L "$current" ]]; then
old_target=$(readlink -f -- "$current" 2>/dev/null || true)
case "$old_target" in
"$app_root/releases/"*) ;;
*)
printf 'Current symlink points outside the releases directory\n' >&2
exit 73
;;
esac
fi
if [[ "$release_dir" == "$old_target" ]]; then
printf 'Release is already active: %s\n' "$release_dir" >&2
exit 0
fi
exec 9>"$app_root/.deploy.lock"
if ! flock -n 9; then
printf 'Another deployment is already running\n' >&2
exit 75
fi
health_check() {
local url=$1
curl --fail --silent --show-error \
--connect-timeout 2 --max-time 5 \
--retry 2 --retry-delay 1 --retry-connrefused \
"$url" >/dev/null
}
temp_dir=""
cleanup() {
if [[ -n "$temp_dir" ]]; then
rm -rf -- "$temp_dir"
fi
}
trap cleanup EXIT
switch_to() {
local target=$1
temp_dir=$(mktemp -d "$app_root/.deploy.XXXXXX")
ln -s -- "$target" "$temp_dir/current"
mv -Tf -- "$temp_dir/current" "$current"
rmdir -- "$temp_dir"
temp_dir=""
}
if ! health_check "$release_health_url"; then
printf 'Release health check failed; current release was not changed\n' >&2
exit 70
fi
switch_to "$release_dir"
if ! health_check "$public_health_url"; then
printf 'Public health check failed; rolling back\n' >&2
if [[ -n "$old_target" ]]; then
switch_to "$old_target"
if ! health_check "$public_health_url"; then
printf 'Rollback completed, but the application remains unhealthy\n' >&2
fi
else
printf 'No previous release is available for rollback\n' >&2
fi
exit 1
fi
printf 'Deployed %s\n' "$release_dir"
Rollback: Keep the previous release intact and automate rollback when post-switch health checks fail.
Backups: Release retention enables application rollback, but database backups are still required for destructive data operations.
Upgrades: Test release-specific health checks in staging before production and coordinate application, worker, and web-server upgrades.
Portability: This is Bash and Linux-specific. POSIX sh lacks [[, arrays, pipefail, and GNU mv -T.
Failure Modes and Trade-offs
The atomic rename prevents partial symlink states, but it does not guarantee application compatibility. A broken release can still be served briefly before the health check detects it and triggers rollback. Health endpoints must exercise meaningful dependencies and verify the running version; otherwise an old worker or cached response can produce a false-positive health result.
74
How do you manually rotate a log file with Bash?
KnownRevise
A manual log rotation script renames the active log file, creates a fresh file in its place, and preserves the original file for archival or compression. The script below uses mv, mktemp, strict Bash error handling, and same-directory temporary files so rotation does not cross filesystem boundaries.
Read full answerHide full answer
Applies toBash 5 on Linux with GNU coreutils
How It Works
The script accepts exactly one log-file path as an argument.
It resolves the path with realpath, so symlinks are rotated at their target rather than replacing the symlink itself.
It creates a temporary file in the same directory as the log.
It copies the original file’s permissions and, when run as root, ownership.
It renames the active log to a timestamped name such as app.log.20250308T120000Z.
It renames the temporary file to the original log path.
Each mv is a same-filesystem rename and is therefore atomic individually. The complete rotation is not one atomic exchange, so the script includes recovery logic if replacing the original path fails.
Production Behavior
An application may continue writing to the rotated file if it keeps an open file descriptor. For example, after:
app.log -> app.log.20250308T120000Z
an existing process might still write to app.log.20250308T120000Z until it reopens the pathname. Conversely, a process that opens the log for every write will immediately use the new app.log.
The script deliberately does not send SIGHUP or another signal because signal behavior is application-specific. In production, coordinate rotation with the service’s documented reload mechanism, such as:
systemctl reload app.service
an application-specific SIGHUP handler
a controlled service restart during a maintenance window
Text Processing and Portability
Use grep -F for fixed-string searches and grep -E for extended regular expressions when investigating log events.
Use sed for line-oriented substitutions, not for rotating files.
Use awk for field-based analysis, such as extracting timestamps or status codes.
Avoid parsing arbitrary logs with shell read loops; records can contain spaces, tabs, control characters, or multiline entries.
This script requires Bash and GNU utilities. It is not POSIX sh compatible because it uses [[, set -o pipefail, realpath, mktemp, and GNU-specific mv -T and chmod --reference options. A POSIX version would need [ ] instead of [[ ]], no pipefail, and platform-specific temporary-file handling.
Code Example
#!/usr/bin/env bash
set -euo pipefail
usage() {
printf 'Usage: %s LOG_FILE\n' "${0##*/}" >&2
exit 2
}
if (( $# != 1 )); then
usage
fi
log_file=$1
if [[ ! -f "$log_file" ]]; then
printf 'Error: log file does not exist or is not a regular file: %s\n' \
"$log_file" >&2
exit 1
fi
# Resolve symlinks and convert the path to an absolute path.
log_file=$(realpath -e -- "$log_file")
log_dir=$(cd -- "$(dirname -- "$log_file")" && pwd -P)
log_name=${log_file##*/}
timestamp=$(date -u +%Y%m%dT%H%M%SZ)
rotated="${log_dir}/${log_name}.${timestamp}"
# Avoid overwriting an existing archived file.
if [[ -e "$rotated" ]]; then
suffix=1
while [[ -e "${rotated}.${suffix}" ]]; do
suffix=$((suffix + 1))
done
rotated="${rotated}.${suffix}"
fi
# Create the replacement in the same directory and filesystem.
tmp_file=$(mktemp -- "${log_dir}/.${log_name}.XXXXXX")
cleanup() {
if [[ -n "${tmp_file:-}" && -e "$tmp_file" ]]; then
rm -f -- "$tmp_file"
fi
}
trap cleanup EXIT
# Preserve metadata. Ownership can normally be changed only by root.
if [[ "$(id -u)" -eq 0 ]]; then
chown --reference="$log_file" "$tmp_file"
fi
chmod --reference="$log_file" "$tmp_file"
# Rename the active log out of the way.
mv -T -- "$log_file" "$rotated"
# Install the new empty log at the original path.
if ! mv -T -- "$tmp_file" "$log_file"; then
# Attempt to restore the original path if replacement failed.
if ! mv -T -- "$rotated" "$log_file"; then
printf 'Critical: failed to restore original log path: %s\n' \
"$log_file" >&2
fi
exit 1
fi
tmp_file=
trap - EXIT
printf 'Rotated %s -> %s\n' "$log_file" "$rotated"
Run the script as the account that owns the log, or run it with narrowly scoped sudo permissions.
Ensure the log directory is writable but not writable by untrusted users; predictable archive names are only safe in a trusted directory.
Monitor filesystem usage. Rotating files without retention or compression can fill the disk.
Compress archived files only after the application has stopped writing to them.
Define retention, alerting, and archival procedures. For example, keep local rotations for seven days and ship logs to centralized storage for incident analysis.
Prefer the system’s logrotate integration or an application-native rotation mechanism when compression, copytruncate, post-rotate scripts, and retention policies are required.
Treat copytruncate as a compatibility fallback, not a universally safe alternative: it can lose records written between copying and truncating the original file.
75
How do you retry a command with exponential backoff in Bash?
KnownRevise
A Bash retry function should run the command up to a configured number of attempts, increasing the delay exponentially between failures and returning the command’s final exit status. The implementation below uses exponential backoff while preserving the command’s arguments safely.
Read full answerHide full answer
Applies toBash 5.x on Linux (Ubuntu 22.04/24.04, RHEL 9)
How It Works
The function receives the maximum attempts, initial delay, maximum delay, and the command plus its arguments.
The command is executed directly with "$@", so arguments are not subject to word splitting or shell glob expansion.
A successful command returns 0 immediately.
A failed command causes the function to sleep, double the delay, and try again.
The delay is capped at max_delay.
When all attempts are exhausted, the function returns the last command exit status.
Attempt
Delay before the next attempt
1
1 second
2
2 seconds
3
4 seconds
4
8 seconds
Code Example
#!/usr/bin/env bash
set -euo pipefail
retry() {
local max_attempts="$1"
local initial_delay="$2"
local max_delay="$3"
shift 3
if (( $# < 1 )); then
printf 'Usage: retry MAX_ATTEMPTS INITIAL_DELAY MAX_DELAY COMMAND [ARG...]\n' >&2
return 2
fi
if ! [[ "$max_attempts" =~ ^[1-9][0-9]*$ ]] ||
! [[ "$initial_delay" =~ ^[1-9][0-9]*$ ]] ||
! [[ "$max_delay" =~ ^[1-9][0-9]*$ ]]; then
printf 'retry: attempts and delays must be positive integers\n' >&2
return 2
fi
if (( max_delay < initial_delay )); then
printf 'retry: maximum delay must be at least the initial delay\n' >&2
return 2
fi
local attempt=1
local delay="$initial_delay"
local status=0
while (( attempt <= max_attempts )); do
if "$@"; then
return 0
else
status=$?
fi
if (( attempt == max_attempts )); then
return "$status"
fi
printf 'Command failed (attempt %d/%d, exit %d); retrying in %ds\n' \
"$attempt" "$max_attempts" "$status" "$delay" >&2
sleep "$delay"
delay=$((delay * 2))
if (( delay > max_delay )); then
delay="$max_delay"
fi
attempt=$((attempt + 1))
done
}
retry 4 1 8 /bin/false
Example output:
Command failed (attempt 1/4, exit 1); retrying in 1s
Command failed (attempt 2/4, exit 1); retrying in 2s
Command failed (attempt 3/4, exit 1); retrying in 4s
The final failure returns exit status 1; with set -e, a top-level call exits the script at that point.
Production Behavior
Use retries for failures that may be transient, such as a temporary network timeout, a restarting service, or a rate-limited API. Do not blindly retry permanent failures such as invalid credentials, malformed requests, or most HTTP 4xx responses.
A command that hangs indefinitely will never be retried by this function. On Linux, wrap it with a timeout when appropriate:
For pipelines involving grep, sed, or awk, invoke a fixed script through bash -c rather than concatenating untrusted input into a shell string. For example, avoid constructing commands such as "grep ... $user_input"; pass the input as a separately quoted argument instead.
Best Practices
Keep set -euo pipefail enabled, but do not rely on it alone. The function explicitly handles the command failure inside an if statement.
Use if retry ...; then when the caller needs to handle the final failure itself.
Add random jitter in distributed systems to prevent many clients retrying at the same time and creating a thundering herd.
Choose a finite attempt count and a capped delay so retries do not amplify an outage.
Log the attempt count, exit status, and delay, but never log secrets or authentication headers.
Make retried operations idempotent where possible. Retrying a non-idempotent write can create duplicate data.
If the command produces sensitive output, redirect or filter it separately; this function does not capture or redact command output.
On POSIX systems using sh rather than Bash, replace Bash features such as [[ ... ]], local, and arithmetic conditions with POSIX-compatible syntax.
76
How do I find the 10 largest files on a Linux system?
KnownRevise
The most direct approach is to use GNU find to report each regular file’s apparent size and path, sort numerically by size in descending order, and print the first 10 results. The example below searches the root filesystem; remove -xdev only when all mounted filesystems should be included.
Read full answerHide full answer
Applies toGNU findutils 4.x, GNU coreutils 9.x, GNU sed 4.x, and Bash 5.x on Linux
How the Command Works
find / -xdev -type f searches the root filesystem for regular files without descending into other mounted filesystems.
-printf '%s\t%p\n' prints the apparent size in bytes followed by a tab and the path.
sort -t $'\t' -k1,1nr -k2,2 sorts by size numerically in descending order, using ascending path order as the tie-breaker.
head -n 10 prints the 10 largest entries.
%s reports logical or apparent size, not allocated disk space. Consequently, a sparse file can appear large while occupying relatively little storage.
-xdev: Searches only the filesystem containing /. This excludes files on separate data volumes, as well as filesystems such as /proc, /sys, and remote mounts.
Other mounts: Remove -xdev to search every mounted filesystem. This can be slow on network storage and may include pseudo-filesystems whose reported sizes are not meaningful.
Selected volumes: For important data filesystems, run the same pipeline separately for each mount point and compare the resulting top-10 lists. The global top 10 must appear in one of those per-filesystem lists.
sudo: Allows the scan to traverse directories that an unprivileged user cannot read.
Error suppression: 2>/dev/null hides permission and transient filesystem errors. Remove it while investigating incomplete coverage.
Production and Portability Considerations
Running a broad find scan can create substantial disk, network, CPU, and inode activity. Schedule it during low-traffic periods, monitor its duration and exit status, and remember that GNU sort may use temporary disk space while sorting a large result set.
With set -euo pipefail, head can close the pipe early and cause sort to receive SIGPIPE, making the pipeline fail. GNU sed without an early-exit command consumes the complete input, so use it when pipe-failure status matters:
#!/usr/bin/env bash
set -euo pipefail
sudo find / -xdev -type f -printf '%s\t%p\n' \
| sort -t $'\t' -k1,1nr -k2,2 \
| sed -n '1,10p'
Do not suppress find errors in this fail-fast script: if find exits nonzero, pipefail causes the script to exit after displaying the diagnostic.
GNU find extensions such as -printf are not available in strictly POSIX find. Bash’s $'\t' is also non-POSIX; under POSIX sh, use sort -t "$(printf '\t')" instead. Filenames containing tabs or newlines make line-oriented output ambiguous and require NUL-delimited processing.
Before deleting anything, verify the file’s purpose, retention requirements, and whether a service still has it open. Investigate large deleted-but-open files with lsof +L1; because they have no discoverable directory entry, find cannot report them.
77
How do I replace a string in every file in a directory?
KnownRevise
Use an in-place text replacement over a safely enumerated set of regular files. The command below treats the old and new values as literal strings and correctly handles filenames containing spaces or other shell metacharacters.
Read full answerHide full answer
Applies toBash 5.x, GNU findutils 4.x, and Perl 5.32+ on Linux
How It Works
find enumerates the files while avoiding directory traversal and symbolic links.
-maxdepth 1 limits the operation to files directly inside the current directory.
-type f selects regular files, excluding directories, devices, sockets, and symbolic links.
-exec ... {} + executes Perl in batches and passes filenames safely rather than interpolating them into a shell command.
\Q...\E makes the old value literal, so characters such as ., *, [, and ? are not treated as regular-expression metacharacters.
Perl’s -i option performs the replacement in place, while -p processes the file line by line.
Code Example
#!/usr/bin/env bash
set -euo pipefail
# These temporary environment variables are inherited by find and Perl.
OLD_TEXT='old string' NEW_TEXT='new string' find . -maxdepth 1 -type f \
-exec perl -pi -e 's/\Q$ENV{OLD_TEXT}\E/$ENV{NEW_TEXT}/g' -- {} +
If the old value is intentionally a regular expression and GNU sed is available, the shorter equivalent is:
find . -maxdepth 1 -type f -exec sed -i 's/old_string/new_string/g' -- {} +
Operational and Failure Modes
Unintended scope: Removing -maxdepth 1 also changes files in every nested directory. Use an explicit root such as ./config to constrain the operation.
Non-text files: The command does not distinguish text from binary files. Replacing bytes in a binary file can corrupt it.
Rewrite behavior: Perl rewrites every selected file, even when it contains no match. File modification times and inode numbers may therefore change.
Line-based matching: Perl processes one line at a time, so a value spanning multiple lines will not match.
Empty old value: An empty OLD_TEXT is unsafe because it can match at every position. Validate that it is non-empty.
Partial failure: set -e causes the script to stop if find or Perl reports an error, but some earlier files may already have been modified.
No cross-file atomicity: Each file is replaced separately. A failure halfway through leaves earlier files updated and later files unchanged.
Best Practices
Back up first: Use version control, snapshots, or an explicit backup before modifying production configuration.
Review the diff: Run git diff --check and inspect git diff before committing or reloading services.
Validate configuration: Run the relevant syntax checker, such as nginx -t, before reloading a service.
Handle symlinks deliberately: The shown command skips symbolic links, preventing accidental modification of files outside the directory.
Prefer staging: Test the replacement against representative files before applying it to production.
Check portability: set -o pipefail is a Bash feature, not part of POSIX sh. The shown find syntax also relies on GNU findutils behavior.
Coordinate writers: Avoid editing files while another process is writing them. For critical live configuration, deploy versioned files and switch to them atomically instead.
78
How can large-file shell scripts be optimized for speed?
KnownRevise
Bash optimization means reducing unnecessary process creation, moving repetitive work into efficient tools such as awk, and parallelizing independent work without exhausting host resources. The largest gains usually come from profiling first, avoiding per-line shell loops, and tuning the script for the actual I/O and CPU bottleneck.
Read full answerHide full answer
Applies toBash 5.x with GNU coreutils 9.x and GNU awk 5.x on Linux
How to Find the Bottleneck
Do not optimize blindly. Test with a representative file and measure where time is spent:
High system time: The script creates many processes or performs excessive reads and writes.
High user time: Expensive operations such as regular expressions run repeatedly.
High I/O wait: The workload is disk-bound, so CPU optimization will have limited effect.
Slow startup: The script launches many small commands even when the file is small.
Also use bash -x for short traces when necessary, but avoid it in production because expanded arguments and secrets may appear in logs.
Key Optimizations
Technique
Benefit
Trade-off
Use builtins where practical
Avoids fork and exec overhead
Builtins are less expressive than external tools
Replace shell line loops with awk
Moves repeated processing into a compiled, streaming tool
Field positions depend on the input format
Combine commands into one awk program
Avoids intermediate pipelines and repeated file scans
Can become difficult to maintain
Use grep -F for fixed strings
Avoids regular-expression processing
Only suitable when no pattern syntax is needed
Batch work with xargs -P
Uses multiple CPU cores
Can overload disks, memory, or downstream services
Cache reusable values
Avoids repeated calculations or lookups
Stale cached data can cause correctness problems
Use streaming processing
Keeps memory usage bounded
More complex aggregation may require memory
Bash builtins such as printf, read, test, [[, and parameter expansion are faster than equivalent external commands because they run inside the shell. However, a Bash loop that invokes an external command for every line is still expensive.
For Unicode-heavy data, avoid forcing LC_ALL=C; it can speed up byte-oriented matching but changes locale-sensitive matching and collation behavior. Reserve it for explicitly ASCII data where that behavior is acceptable.
Code Example
The following script counts HTTP failures in one streaming pass. It avoids a Bash while read loop and does not load the file into memory:
#!/usr/bin/env bash
set -euo pipefail
if (( $# != 1 )); then
printf 'Usage: %s FILE\n' "$0" >&2
exit 64
fi
input=$1
export LC_ALL=C
awk '
{
status = $9
if (status ~ /^[45][0-9][0-9]$/) {
failures++
counts[status]++
}
}
END {
printf "total_failed=%d\n", failures + 0
for (i = 400; i < 600; i++) {
status = sprintf("%d", i)
if (status in counts) {
printf "%s=%d\n", status, counts[status]
}
}
}
' -- "$input"
Example output:
total_failed=1253
404=1100
500=153
awk reads the input incrementally, so memory usage depends mainly on the number of distinct matching statuses rather than the file size. The script assumes the ninth whitespace-separated field is the HTTP status, which is common for standard combined access logs but must be validated for the actual format.
Unlike many Bash features, this awk structure is broadly portable to POSIX awk. Features such as arrays, [[ ... ]], and (( ... )) are Bash-specific and are not available in POSIX sh.
Parallel Processing
Use parallelism only when records can be processed independently:
GNU xargs -P: Suitable for many small files or command batches. Its concurrency may exceed the available CPU count, so set a conservative limit.
File sharding: Split a large file into independent chunks, process them concurrently, and merge the results.
Producer-consumer pipelines: Commands in a normal shell pipeline already run concurrently, but this usually overlaps I/O rather than parallelizing one sequential stage.
Aggregation strategy: Combine per-chunk counts before loading all results into one associative array.
Parallelism is often ineffective for a single disk-bound file because concurrent readers may compete for the same I/O budget. It is more useful for many independent files, network requests, or CPU-heavy transformations.
Production Practices
Validate inputs: Check file existence, readability, expected format, and output directory before processing large datasets.
Fail explicitly: Use set -euo pipefail, but also validate commands and inputs because set -e has contextual behavior and is not a substitute for explicit error handling.
Write atomically: Create output with mktemp, write it completely, validate it, and then rename it into place.
Protect temporary data: Set a restrictive umask, avoid predictable temporary filenames, and clean them up with a trap.
Check completion: Compare record counts or checksums when the source provides an expected total.
Support incremental runs: Store checkpoints or processed offsets so a failed multi-hour job does not restart from the beginning.
Monitor throughput: Record processed bytes, records per second, elapsed time, and estimated completion time during long operations.
Test before deployment: Run shellcheck, test with malformed and empty input, and benchmark using production-sized data.
Plan upgrades: Pin critical tool versions during rollout, compare outputs before and after upgrades, and retain rollback procedures.
The safest optimization strategy is to measure the current script, replace the dominant bottleneck, rerun the same benchmark, and verify that output remains identical.
79
How do you unit test Bash scripts with Bats?
KnownRevise
Bats is a TAP-based test runner for Bash that executes each test in an isolated subshell. Tests follow the arrange-act-assert pattern: prepare inputs, invoke the script with run, and assert its exit status and output.
Read full answerHide full answer
Applies toBash 5.x, bats-core 1.10+, and bats-assert 2.x on Ubuntu 24.04 (Ubuntu 22.04 ships bats 1.2, which lacks `BATS_TEST_TMPDIR` and `--separate-stderr`)
How Bats Works
Bats discovers files ending in .bats and converts each @test block into a separate test case. Each test receives a clean process, preventing exported variables or filesystem changes from leaking into another test.
setup: Runs before every test; use it to create temporary files or initialize state.
teardown: Runs after every test; use it for cleanup.
run: Executes a command without immediately terminating the test, capturing $status, $output, and $lines.
bats-assert: Provides assertions such as assert_success, assert_failure, and assert_output.
BATS_TEST_DIRNAME: The directory containing the current test file.
BATS_TEST_TMPDIR: A temporary directory isolated for the current test.
By default, run combines standard output and standard error. Newer bats-core versions can separate them with run --separate-stderr.
Test Design
A focused Bash unit test should exercise one decision or side effect:
Arrange: Create a controlled input, temporary directory, or command stub.
Act: Run the production script or function.
Assert: Check the exit code, stdout, stderr, created files, or mocked API calls.
For infrastructure scripts, mock dependencies such as aws, kubectl, terraform, or gcloud. Place stub executables in $BATS_TEST_TMPDIR/bin and prepend that directory to PATH. This prevents unit tests from modifying real cloud resources.
Do not use set -e in the Bats test file. Bats relies on explicit assertions to report failures, while production scripts should use set -euo pipefail and deliberate exit codes.
#!/usr/bin/env bash
set -euo pipefail
readonly RELEASE_PATTERN='^[a-z0-9][a-z0-9._-]{0,63}$'
validate_release() {
if (( $# != 1 )); then
printf 'usage: %s RELEASE\n' "${0##*/}" >&2
return 2
fi
local release="$1"
if [[ ! $release =~ $RELEASE_PATTERN ]]; then
# Escape the value so newlines cannot forge additional log records.
printf 'invalid release name: %q\n' "$release" >&2
return 1
fi
printf 'release=%s\n' "$release"
}
main() {
validate_release "$@"
}
# Allow the functions to be sourced without running the CLI.
if [[ ${BASH_SOURCE[0]} == "$0" ]]; then
main "$@"
fi
Create tests/validate_release.bats:
#!/usr/bin/env bash
# shellcheck shell=bash
# Ubuntu's bats-support and bats-assert packages install under /usr/lib/bats.
load '/usr/lib/bats/bats-support/load'
load '/usr/lib/bats/bats-assert/load'
setup() {
RELEASE_SCRIPT="$BATS_TEST_DIRNAME/../validate_release.sh"
}
@test "accepts a valid release name" {
run bash "$RELEASE_SCRIPT" "web-2.1"
assert_success
assert_output "release=web-2.1"
}
@test "rejects an invalid release name" {
run bash "$RELEASE_SCRIPT" "Web_2"
assert_failure 1
assert_output --partial "invalid release name"
}
@test "returns a usage error when the argument is missing" {
run bash "$RELEASE_SCRIPT"
assert_failure 2
assert_output --partial "usage:"
}
1..3
ok 1 accepts a valid release name
ok 2 rejects an invalid release name
ok 3 returns a usage error when the argument is missing
Bats also supports filtering tests by name and selecting a formatter:
bats --filter "invalid release" tests/validate_release.bats
bats --formatter tap tests/
Production and Tooling Choices
Tool
Primary use
Key consideration
bats-core
Testing Bash behavior
Best fit for Bash-specific features
bats-assert
Assertions for Bats
Cleaner assertions and failure messages
Shellspec
Testing several POSIX-compatible shells
Useful when sh compatibility matters
shunit2
Bourne-shell unit testing
Lightweight, but less feature-rich than Bats
In CI, pin the Bats and assertion-library versions to keep results reproducible. Run unit tests on every pull request, then run separate integration or staging tests against version-pinned infrastructure providers. Unit tests should detect control-flow regressions quickly; integration tests detect credential, API, image, and provider drift that mocks cannot reproduce.
Common Pitfalls
Testing live infrastructure: Mock external commands so tests remain deterministic and safe.
Overusing set -e: Use explicit run and assertions in Bats; use set -euo pipefail in production scripts.
Losing failure output: Inspect $status, $output, and $lines when an assertion fails.
Leaking state: Create fresh files under $BATS_TEST_TMPDIR for every test.
Exposing secrets: Never place real credentials in fixtures, TAP logs, or command arguments.
Assuming validation proves trust: A valid release name does not prove artifact authenticity; production deployments still need signature or digest verification.