Skip to content
Cloud, DevOps & Infrastructure

Linux interview questions and answers

80 questions, 48 with code. About 7 h to read them all.

By TestLab · Updated

Plan for an interview
80 questions

Track what you know and what to revise.

Sign in to track progress

You can open a few full answers without signing in. Sign in to read all

1

What is Linux, and how are distributions different?

Linux is the open-source operating-system kernel that manages hardware resources and provides core services for processes, memory, filesystems, networking, and security. A Linux distribution packages the Linux kernel with user-space tools, libraries, a package manager, usually installation tools (Arch, Gentoo and NixOS take a more manual approach), documentation, and optional desktop or server software.

Read full answer

Applies toLinux on Ubuntu 22.04/24.04 and RHEL 9

Core Architecture

The Linux kernel runs in kernel space, while applications and utilities run in user space. The kernel provides:

  • Process management: scheduling processes, creating them through system calls such as fork and clone, and coordinating them with functions such as wait.

  • Signals: delivering asynchronous events such as SIGTERM, SIGKILL, and SIGCHLD.

  • Memory management: virtual memory, paging, memory mappings, and configurable overcommit behavior.

  • Filesystem support: the Virtual File System, or VFS, which gives processes a common interface to filesystems such as ext4, XFS, and Btrfs.

  • Networking: sockets, routing, firewall hooks, and protocol implementations.

  • Device access: character and block devices exposed through /dev.

  • Security primitives: user and group permissions, namespaces, capabilities, mandatory access control, and isolation mechanisms.

A kernel alone is not a complete general-purpose operating system. It implements system calls for operations such as creating files, while user-space programs and services such as bash, sshd, and systemd provide the operating environment.

Distribution Differences

Area Ubuntu RHEL
Package format Debian packages (.deb) RPM packages (.rpm)
Package manager apt, apt-get, dpkg dnf, yum, rpm
Installation apt install or apt-get install dnf install
Repositories Ubuntu archive and security repositories RHEL BaseOS, AppStream, and CodeReady repositories
Mandatory access control AppArmor is commonly enabled SELinux is normally enabled in enforcing mode
Init/service manager systemd systemd
Target General-purpose server, cloud, and desktop systems Long-lived enterprise and certified environments

Distributions can also use different kernel versions, kernel configurations, filesystem defaults, compiler versions, system defaults, and backported patches. Consequently, two hosts running Linux may behave differently even when their kernel version strings appear similar.

Filesystem, Permissions, and Services

The filesystem hierarchy separates operating-system files from variable data:

  • /bin and /sbin: essential commands, usually linked into /usr on these modern distributions.

  • /etc: system-wide configuration.

  • /usr: most user-space programs and libraries.

  • /var: logs, package databases, queues, and other frequently changing data.

  • /home or /root: user home directories.

  • /run: volatile runtime state that normally disappears after reboot.

  • /proc, /sys, and /dev: virtual or device-backed filesystems representing processes, kernel state, and devices.

Files have an owner, a group, and permissions for the owner, group, and others. Inspect ownership and mode with ls -ld /etc/ssh. For a private application directory, a restrictive mode such as chmod 750 is generally preferable to world-writable permissions. Avoid chmod 777: it makes files writable by every account and removes meaningful access boundaries. Use groups, ACLs, or narrowly scoped capabilities when appropriate.

systemd manages services through unit files under /etc/systemd/system and /usr/lib/systemd/system. Administrator-defined units in /etc take precedence over vendor-provided units. systemd-journald collects journal entries and supports querying by unit, priority, and time. Processes communicate through signals: SIGTERM requests a graceful shutdown, while SIGKILL cannot be handled or ignored.

Inspecting and Updating a Host

Useful diagnostic commands include cat /etc/os-release for distribution metadata, uname -srmo for the kernel release and architecture, findmnt -no SOURCE,FSTYPE,OPTIONS / for the root filesystem, ps -eo pid,ppid,user,stat,%mem,comm --sort=-%mem for process inspection, and systemctl is-system-running for the overall service-manager state. The degraded state means the manager is active but at least one unit has failed.

Updates should be inspected before installation. On Ubuntu, refresh metadata with sudo apt-get update and simulate the upgrade with apt-get -s upgrade. On RHEL, use sudo dnf check-update; it returns exit code 100 when updates are available, so automation must distinguish that result from an actual command failure.

Production Operations and Pitfalls

  • Monitoring: Track CPU throttling, memory pressure, OOM kills, disk and inode usage, packet errors, service restarts, and failed systemd units.

  • Backups: Back up /etc, application configuration, and persistent data. Use application-consistent database backups rather than copying live database files blindly.

  • Upgrades: Apply security updates regularly. Kernel upgrades normally require a reboot and should be tested because modules, drivers, and applications may depend on specific versions.

  • Rollback: Retain a known-good kernel and relevant boot artifacts until the new kernel has been validated. LVM, ZFS, or virtual-machine snapshots can support controlled rollback when their consistency requirements are understood.

  • Security: Use least privilege, patch packages, configure SSH securely, monitor authentication logs, and preserve AppArmor or SELinux protections.

  • Common failures: Disk or inode exhaustion under /var, incompatible kernel modules, broken package dependencies, service misconfiguration, OOM kills, and storage errors can all cause outages.

2

What are /etc, /var, /usr, /opt and /proc used for in Linux?

The Linux filesystem hierarchy is standardized by the Filesystem Hierarchy Standard (FHS) to organize directories consistently, ensuring predictable locations for configuration, binaries, data, and runtime information.

Read full answer

Applies toLinux (all distributions, including Ubuntu 22.04/24.04 and RHEL 9)

The Filesystem Hierarchy Standard

The FHS defines the purpose of each directory under the root /, separating static data (binaries, libraries) from variable data (logs, caches) and configuration. This standardization allows scripts and documentation to reference fixed paths across distributions.

/etc: System Configuration Files

Contains text-based configuration files for the system and installed services, typically owned by root with permissions 644. Sensitive files like /etc/shadow use 600. Examples include /etc/passwd, /etc/hostname, and service configs like /etc/nginx/nginx.conf. On Ubuntu, network configuration lives in /etc/netplan/; on RHEL, it’s in /etc/sysconfig/network-scripts/ or /etc/NetworkManager/conf.d/. Changes often require service restarts, and configuration management tools (Ansible, Puppet) help enforce consistency. Always back up /etc before upgrades.

/var: Variable Data and Logs

Stores variable-length files that change frequently: logs, databases, caches. Key subdirectories: /var/log (system logs; Ubuntu uses /var/log/syslog, RHEL uses /var/log/messages), /var/lib (persistent service data like MySQL), /var/spool (print/mail queues), /var/cache (package and web caches). Monitor disk usage with df -h /var to prevent out-of-space errors. Implement log rotation and back up /var/lib databases.

/usr: Static System Data and Installed Software

Contains read-only, static data: user commands, libraries, documentation. /usr/bin holds most executables (e.g., ls), /usr/lib holds shared libraries, /usr/share holds architecture-independent data (man pages, fonts). /usr/local is for locally compiled software. Historically mounted read-only after boot; if unavailable, the system may not reach multi-user target. Package managers update /usr automatically; custom software in /usr/local requires manual management.

/opt: Optional Add-on Packages

Used for third-party or add-on software not part of the standard distribution. Each package gets its own subdirectory (e.g., /opt/google/chrome), containing a self-contained application. Packages are usually installed manually via tarballs or custom installers. Upgrades require manual intervention; back up /opt and add its bin directories to PATH as needed.

/proc: Kernel and Process Information

A virtual filesystem created by the kernel at boot, providing a window into kernel and process data. Contains per-process directories /proc/<pid>/ with files like cmdline, status, and fd/. System information includes /proc/cpuinfo, /proc/meminfo, /proc/loadavg. Kernel parameters are exposed under /proc/sys/ and can be tuned with sysctl. Used for troubleshooting and performance tuning; not backed up.

Summary Table

Directory Purpose Typical Contents Operational Considerations
/etc System configuration Config files, passwd, hostname Sensitive; use config management; restart services after changes
/var Variable data Logs, databases, caches Monitor disk usage; rotate logs; backup databases
/usr Static binaries and data Executables, libraries, documentation Read-only; critical for boot; upgrade via package manager
/opt Optional add-on software Third-party applications Manual management; self-contained; backup custom software
/proc Kernel and process info Virtual files, PID dirs, sysctl Real-time; used for tuning and troubleshooting; no backup

Best Practices and Common Pitfalls

  • Permissions: Never use chmod 777; use minimal required permissions. Protect /etc/shadow with 600.

  • Disk monitoring: Set up alerts for /var and / filesystem usage. Use df -h and du -sh regularly.

  • Backups: Prioritize /etc, /var, and /opt for backups. /usr can be reinstalled from packages, but custom files in /usr/local need backup.

  • Configuration management: Use tools like Ansible, Puppet, or Chef to manage /etc across servers to avoid configuration drift.

  • Security: Restrict access to /proc/sys/ entries that affect security (e.g., kernel.randomize_va_space). Use sysctl carefully.

  • Path management: When adding custom binaries from /opt or /usr/local, modify PATH in shell profiles or use symlinks in /usr/local/bin.

3

How do absolute and relative paths differ in Linux?

An absolute path starts at the filesystem root (/), while a relative path is resolved from the current working directory of the process performing the lookup. Both forms are subject to the same file ownership, permission, symbolic-link, and mount-namespace rules.

Read full answer

Applies toLinux (Ubuntu 22.04/24.04, RHEL 9), Bash 5.x, GNU coreutils 8.32/9.x

Core Difference

Property Absolute path Relative path
Starting point Filesystem root (/) Process’s current working directory
Syntax Begins with / Does not begin with /
Effect of changing the working directory None The resolved target may change
Example /etc/ssh/sshd_config config/app.yaml
Typical use System configuration and service resources Project files and commands run within a known directory

Common relative-path components include:

  • .: The current directory.

  • ..: The parent of the current directory.

  • config/app.yaml: A child named config, followed by its app.yaml child.

  • ~/logs: The shell expands ~ to the user’s home directory before lookup; ~ itself is not an absolute-path prefix.

Path Resolution and Shell Behavior

The Linux kernel resolves a relative path against the process’s actual current working directory. An interactive shell’s pwd normally reports its logical $PWD, which can preserve symlink components, while pwd -P reports the physical path. This distinction matters when diagnosing services or scripts that appear to start in different directories.

Shell expansion is separate from pathname resolution. Tilde, variable, and other shell expansions occur before the resulting path is passed to the kernel.

Executable lookup is also separate. When Bash runs appctl, it searches the directories listed in PATH; the current directory is not searched automatically unless it is explicitly included in PATH.

GNU realpath canonicalizes a path and resolves symbolic links. Its --relative-to=DIR option expresses the resolved target relative to the specified directory.

Process and Filesystem Context

  • Every Linux process has a current working directory. A child normally inherits it, while a shell’s cd changes the working directory of that shell and subsequently created children.

  • Permissions apply to both path forms. Using ../ does not bypass search or execute permissions on parent directories.

  • Symbolic links can redirect a relative path unexpectedly. For example, logs/current may resolve into /var/log/app-v2.

  • Mount namespaces give containers and services different filesystem views. Consequently, /data/app inside one namespace need not refer to the same storage as /data/app elsewhere.

  • In a chroot or container, / means the root of that process’s filesystem view rather than necessarily the host’s root.

Operational and Security Guidance

  • Use absolute paths for services: Set systemd’s WorkingDirectory= explicitly and use an absolute executable path in ExecStart=. If WorkingDirectory= is omitted, systemd defaults the service’s working directory to /.

  • Avoid depending on the caller’s directory: Scripts should accept explicit paths or establish a known base directory before reading configuration.

  • Validate untrusted paths: Reject traversal components and verify that the canonical target remains within an approved directory. A string-prefix check alone is insufficient because paths such as /data/app-other can resemble /data/app.

  • Check existence and symlinks separately: Use test -e when the target must already exist, then use realpath to canonicalize it. realpath -e means --no-symlinks; it is not an existence check.

  • Do not confuse spelling with resolution: Equivalent absolute and relative spellings of the same resolved location reference the same object. Changing a symlink component, however, can change the object reached.

  • Test under the production context: Validate paths as the service account and within the same mount namespace, working directory, and security restrictions used in production.

4

How do Linux file permissions and chmod 755 work?

Linux uses POSIX discretionary access control (DAC) to assign read, write, and execute permissions to the file owner, a group, and everyone else. chmod 755 gives the owner full access while giving the group and others read and execute access.

Read full answer

Applies toLinux (Ubuntu 22.04/24.04, RHEL 9), GNU coreutils `chmod`

Permission Model

Each file has a numeric owner, a primary group, and three classes of permissions:

Class r w x
User/owner Read contents Modify or truncate Execute the file
Group Read contents Modify or truncate Execute the file
Other Read contents Modify or truncate Execute the file

The kernel applies the user class when the process UID owns the file. Otherwise, it applies the group class when the process belongs to the file’s group or a supplementary group. If neither matches, the other class applies.

For directories, the meanings differ:

  • Read (r): list directory entry names.

  • Write (w): create, rename, or delete entries in the directory.

  • Execute (x): traverse the directory and access entries whose names are known.

A writable and traversable directory can allow a user to delete a file even when that user lacks write permission on the file itself. The sticky bit can restrict such deletion in shared directories.

What chmod 755 Sets

755 is a three-digit octal representation:

  • 7 = rwx = 4 + 2 + 1

  • 5 = r-x = 4 + 1

  • 5 = r-x = 4 + 1

Therefore:

Class Numeric Symbolic
Owner 7 rwx
Group 5 r-x
Other 5 r-x

For a regular executable, this lets any user run the file but only its owner can modify it. For a directory, it permits listing and traversal by everyone but prevents users from creating or deleting entries.

The symbolic equivalent is clearer and less error-prone:

chmod u=rwx,g=rx,o=rx /srv/example/bin/deploy.sh

How chmod Works

chmod updates permission metadata stored in the file’s inode; it does not change the file’s contents. The Linux kernel’s permission checks enforce those mode bits whenever a process opens, modifies, or executes the file.

Common modes include:

Mode Typical use
600 Private keys and secrets readable only by the owner
640 Restricted configuration readable by the owner and a dedicated group
644 Public, non-sensitive files such as static web content
755 Public executables and traversable directories
775 Group-writable directories used by collaborating service accounts

Avoid chmod 777. It grants every user write access and commonly turns a permissions mistake into code injection, data tampering, or malware placement.

Production Troubleshooting

Use stat and namei to inspect permissions and every directory in a path:

stat -c '%A %a %U:%G %n' /srv/example/bin/deploy.sh
namei -l /srv/example/bin/deploy.sh

Typical output:

-rwxr-xr-x 755 app:app /srv/example/bin/deploy.sh
drwxr-xr-x root root /
drwxr-xr-x root root srv
drwxr-xr-x app app srv/example
drwxr-xr-x app app srv/example/bin
-rwxr-xr-x app app srv/example/bin/deploy.sh

Common causes of access failures include:

  • Wrong ownership: A service running as app cannot read a 600 file owned by root.

  • Missing directory execute permission: A user may have file access but lack permission to traverse a parent directory.

  • Incorrect group membership: Supplementary groups may differ from those available in an interactive shell.

  • POSIX ACLs: ls -ld showing a + indicates extended ACLs; inspect them with getfacl.

  • SELinux denial: DAC permissions can be correct while SELinux still blocks access. Check journalctl and audit logs rather than disabling SELinux.

  • Mount options or attributes: Read-only mounts and immutable attributes can prevent expected writes or changes.

For systemd services, verify the configured identity:

systemctl show example.service -p User -p Group

Code Example

WORKDIR="$HOME/demo/bin"
mkdir -p "$WORKDIR"

# Create a small executable script.
printf '#!/usr/bin/env bash\necho "Deploying application"\n' > "$WORKDIR/deploy.sh"

# Owner can read/write/execute; group and others can read/execute.
chmod 0755 "$WORKDIR/deploy.sh"

stat -c '%A %a %U:%G %n' "$WORKDIR/deploy.sh"
"$WORKDIR/deploy.sh"

Example output:

-rwxr-xr-x 755 dev:dev /home/dev/demo/bin/deploy.sh
Deploying application

Best Practices

  • Apply least privilege: use 600 or 640 for sensitive data and 755 only when public execution or traversal is required.

  • Give services dedicated user accounts and groups; avoid running production processes as root.

  • Set ownership and mode during installation with tools such as install, rather than applying broad recursive chmod commands afterward.

  • Remember that umask controls permissions for newly created files and directories; it does not change existing files.

  • Audit unexpected world-writable files:

sudo find /srv/example -xdev -type f -perm -0002 -ls
  • Preserve ownership and permissions in backups and verify them after restores.

  • Treat permission drift as a security event, especially for web roots, deployment directories, configuration files, and secret stores.

5

What do setuid, setgid, and sticky bits mean in Linux?

The setuid, setgid, and sticky bits are special Linux permission bits that change how executables, directories, and file ownership behave. They are represented numerically as 4000, 2000, and 1000, respectively, and are commonly combined with ordinary chmod permissions.

Read full answer

Applies toLinux (Ubuntu 22.04/24.04, RHEL 9)

Permission Bit Behavior

Bit Octal value Common use Behavior
Setuid 4000 Privilege-aware executables A program executes with the effective UID of the file’s owner.
Setgid 2000 Shared directories and executables A program executes with the effective GID of the file’s group; new directory entries inherit the directory’s group.
Sticky 1000 Shared temporary directories Restricts deletion and renaming of directory entries by unprivileged users.
  • Setuid on an executable: If /usr/local/bin/tool belongs to root and has mode 4755, an ordinary user executing it retains their real UID but receives root as the effective UID.

  • Setgid on an executable: The process receives the file’s group as its effective GID. Group execute permission still applies, so setgid does not by itself let an unrelated user execute the file.

  • Setgid on a directory: Newly created files inherit the directory’s group, and newly created subdirectories normally inherit both the group and the setgid bit.

  • Sticky bit on a directory: An unprivileged user may delete or rename an entry only if they own the file, own the containing directory, or hold CAP_FOWNER. In every case, the appropriate write and search permissions on the containing directory are also required.

  • The sticky bit does not prevent users from reading or modifying files where file permissions allow it, and it does not itself prevent users from creating new entries.

Seeing and Changing the Bits

ls -l displays these bits through special permission characters. /usr/bin/passwd typically appears as -rwsr-xr-x, while /tmp appears as drwxrwxrwt.

  • The first s represents setuid.

  • The second s represents setgid.

  • The final t represents the sticky bit.

  • Uppercase S means setuid or setgid is set without the corresponding owner or group execute/search permission. Uppercase T means the sticky bit is set without the other execute/search permission.

The command stat -c '%A %a %U:%G %n' /usr/bin/passwd /tmp typically reports octal modes 4755 and 1777, respectively.

Set or remove the bits with symbolic or numeric chmod modes:

  • sudo chmod u+s,g+s /usr/local/bin/tool enables setuid and setgid while preserving the existing base permissions.

  • sudo chmod 6750 /usr/local/bin/tool enables both special bits and sets the base mode to 0750; its symbolic mode is -rwsr-s---.

  • sudo chown app-team:app-team /srv/app/shared assigns the shared directory’s group, and sudo chmod 2770 /srv/app/shared enables setgid directory inheritance.

  • sudo chmod +t /srv/app/shared adds the sticky bit, while sudo chmod 3770 /srv/app/shared enables setgid and sticky together.

  • sudo chmod u-s,g-s,o-t /usr/local/bin/tool removes the special bits.

The fourth octal digit contains the special bits: 6 means setuid plus setgid, 2 means setgid, and 1 means sticky.

Important Implementation Details

  • Kernel restriction: Linux intentionally ignores setuid and setgid bits when executing interpreted scripts. Setting u+s on a shell script does not make that script run as root.

  • Directory ownership controls deletion: File deletion is controlled primarily by write and search permissions on the containing directory. The sticky bit adds ownership and capability checks to that rule.

  • Setuid is not an unrestricted privilege bypass: The program remains subject to Linux capabilities, the capability bounding set, mandatory access control such as SELinux or AppArmor, mount restrictions, and its own privilege-dropping logic.

  • Mount options matter: Filesystems mounted with nosuid ignore setuid and setgid bits. noexec prevents execution, while nodev prevents device files from being interpreted.

  • Regular files: The sticky bit has no standard effect on ordinary regular files; its important use is on shared directories.

Security and Operational Guidance

  • Audit special files: sudo find / -xdev -type f -perm -4000 -ls 2>/dev/null finds setuid files on the root filesystem, while sudo find / -xdev -type f -perm -2000 -ls 2>/dev/null finds setgid files. Repeat the audit on separately mounted filesystems that -xdev excludes.

  • Minimize privilege: Use the narrowest required permissions and prefer Linux file capabilities over setuid when the operation can be expressed safely as a capability.

  • Control shared directories: Directory ownership and group management determine who can modify setgid content. Verify that the owning group is appropriate and membership is managed deliberately.

  • Do not use chmod 777 as a fix: It grants excessive write access. For shared temporary directories, the usual pattern is a sticky directory such as /tmp with mode 1777.

  • Monitor changes: Alert on unexpected changes to privileged binaries, ownership of shared directories, and newly introduced setuid or setgid files.

6

How does Linux umask affect permissions on newly created files?

umask is a process-level bitmask that removes permissions when files or directories are created. It does not grant permissions and does not change existing objects; it only determines which requested permission bits are withheld.

Read full answer

Applies toLinux (Ubuntu 22.04/24.04, RHEL 9)

How umask Works

A program requests an initial mode when creating an object. On traditional Linux filesystems, the effective permissions are calculated as:

effective_mode = requested_mode & ~umask

Typical requested modes are:

  • Files: 0666 (rw-rw-rw-)

  • Directories: 0777 (rwxrwxrwx)

The umask specifies which bits to remove. For example, 0022 removes group-write and other-write permission:

umask New files New directories Typical use
0000 0666 0777 Insecure for most systems
0022 0644 0755 Common system default
0027 0640 0750 Restrict access to owner and group
0077 0600 0700 Private application data
0002 0664 0775 Shared group collaboration

The application’s requested mode still matters. If an application explicitly creates a file with 0600, a permissive umask cannot turn it into 0666; umask only removes permissions.

Inspecting and Applying umask

$ umask
0022

$ umask -S
u=rwx,g=rx,o=rx

$ touch report.txt
$ mkdir reports
$ stat -c '%A %a %n' report.txt reports
-rw-r--r-- 644 report.txt
drwxr-xr-x 755 reports
  • umask prints the mask in octal.

  • umask -S prints its symbolic representation. These bits represent permissions being masked, not the resulting file permissions.

  • stat -c displays symbolic permissions, octal permissions, and the filename.

Change the umask for the current shell and its future child processes:

$ umask 0027
$ touch service.conf
$ mkdir service-data
$ stat -c '%A %a %n' service.conf service-data
-rw-r----- 640 service.conf
drwxr-x--- 750 service-data

The shell’s existing umask is inherited by child processes. Changing a child’s umask does not affect its parent or siblings.

Persistence Across Shells and Services

For login shells, the distribution commonly defines the default in /etc/login.defs:

$ grep -E '^[[:space:]]*UMASK' /etc/login.defs
UMASK 022

Administrators may override this in /etc/profile, /etc/profile.d/*.sh, or a user’s ~/.profile. Shell startup files do not necessarily affect services started outside a login session.

For systemd services, set UMask= in the unit file:

[Service]
UMask=0077
ExecStart=/usr/local/bin/app

After changing a unit:

sudo systemctl daemon-reload
sudo systemctl restart app.service
systemctl show app.service -p UMask

Use 0022 for ordinary public content, 0027 or 0077 for sensitive service data, and 0002 only when a controlled group needs shared write access.

Security and Operational Trade-offs

  • Secrets: SSH keys, tokens, certificates, and credential files should normally be 0600; use umask 0077 while creating them.

  • Shared directories: 0027 produces 0750 directories, preventing unrelated users from listing or traversing them.

  • Avoid 0000: It creates world-writable files and directories, allowing unauthorized modification and data disclosure.

  • Permissions are not access control by themselves: Directory traversal requires execute permission, ownership, groups, and ACLs also affect effective access.

  • Existing data remains unchanged: Applying a safer umask does not secure previously created files. Audit and correct them explicitly with chmod, ownership, and ACLs.

Common Pitfalls and Production Practices

  • Umask affects only newly created objects; modifying an existing file’s mode does not automatically protect future versions of that file.

  • POSIX ACLs can influence inherited permissions. Inspect them with getfacl when expected results differ from the umask calculation.

  • Do not use chmod 777 as a shortcut for permission problems. Identify the required owner, group, directory traversal rights, and write requirements instead.

  • Include umask settings in service reviews, deployment checks, and security baselines.

  • After an incident involving exposed data, remember that umask prevents future exposure but does not remediate files already created.

7

How do I change file ownership with chown and chgrp in Linux?

chown changes a file’s owner and optionally its group, while chgrp changes only the group. Both operate on the numeric user and group IDs stored in the file’s inode. Changing the owner to another user normally requires root or CAP_CHOWN; an owner may change the group to a group of which they are a member.

Read full answer

Applies toGNU coreutils 8.32+ on Linux (Ubuntu 22.04/24.04, RHEL 9, Fedora)

Commands and Options

Command Purpose
sudo chown user:group FILE Sets both owner and group
sudo chown user FILE Sets the owner and leaves the group unchanged
sudo chgrp group FILE Sets only the group
sudo chown -R user:group DIRECTORY Recursively changes ownership
sudo chown -h user:group LINK Changes the symbolic link itself rather than its target
sudo chown -v user:group FILE Displays each affected file
sudo chown -c user:group FILE Reports only files that were changed

User and group names are resolved through NSS, which can include local files, LDAP, or SSSD. Numeric IDs such as 1001:1001 are also accepted, but symbolic names are generally clearer and less likely to be mapped incorrectly.

Changing ownership preserves the file’s user, group, and other read, write, and execute mode bits. However, Linux can clear setuid and setgid bits when the corresponding owner or group changes. A regular file with mode 0640 remains 0640, although its effective accessibility can change because the new owner or group may gain or lose relevant permissions.

Code Example

Assuming app exists and app-admin is a valid group, change an application file’s owner and group, then update only its group:

$ ls -l /srv/myapp/config.env
-rw-r----- 1 app app 842 Jun 10 09:14 /srv/myapp/config.env

# Set both owner and group.
$ sudo chown app:app /srv/myapp/config.env

# Set only the group.
$ sudo chgrp app-admin /srv/myapp/config.env

$ ls -l /srv/myapp/config.env
-rw-r----- 1 app app-admin 842 Jun 10 09:14 /srv/myapp/config.env

For a release directory:

# Change directory contents recursively and affect symlinks themselves.
$ sudo chown -R -h app:app /srv/myapp/releases/2026-06-10
  • -R: recursively processes directory contents.

  • -h: acts on symbolic links themselves rather than dereferencing them.

  • A successful chown or chgrp normally produces no output.

Inspect ownership and permissions with:

$ stat -c '%U %G %u %g %a %n' /srv/myapp/config.env
app app-admin 1001 1005 640 /srv/myapp/config.env

$ ls -ln /srv/myapp/config.env
-rw-r----- 1 1001 1005 842 Jun 10 09:14 /srv/myapp/config.env

Permissions and Under-the-Hood Behavior

The filesystem stores numeric UID and GID values in the inode. Names such as app and app-admin are mappings resolved from account databases and are not stored in the inode itself.

Ownership metadata is separate from file contents, but it can determine access:

  • The owner may read or write a file according to the owner permission bits.

  • Group access applies when the process belongs to the file’s group through its primary or supplementary groups.

  • Directory traversal requires appropriate x permission.

  • Ownership changes update the inode’s change timestamp, or ctime.

Only privileged processes can normally transfer ownership to another user. A file owner can change the group to a group of which they are a member; CAP_CHOWN permits other group changes.

Production Failure Modes and Security

  • Partial recursive changes: chown -R may update some files before encountering an error. Use narrowly scoped paths and verify the result.

  • Symlink attacks: In attacker-writable directory trees, symbolic links can redirect operations toward sensitive files. Avoid following links and operate only in trusted directories.

  • Broken service access: Incorrect ownership can prevent applications, deployment agents, or backup jobs from reading configuration files.

  • SELinux is separate: On RHEL and Fedora, incorrect SELinux labels can still deny access after ownership is fixed. Use sudo restorecon -Rv /srv/myapp to restore the policy’s default labels. If the service requires a non-default label, define a persistent file-context rule with semanage fcontext and then run restorecon; chcon changes the current label but is not a persistent policy definition and can be overwritten by restorecon.

  • Restores may change ownership: Validate ownership after restoring backups, especially when archives were created or extracted by different accounts.

Operational Best Practices

  • Prefer user:group over numeric IDs for readability and auditability.

  • Review recursive ownership changes with a dry inventory, such as find /srv/myapp -xdev -printf '%u:%g %p\n'.

  • Use ls -ln or stat to distinguish names from numeric IDs.

  • Record changes through change management and auditd; filesystem metadata alone does not identify the operator.

  • Prefer creating or deploying files with the correct owner using tools such as install -o app -g app -m 0640, rather than applying a broad recursive correction afterward.

  • Never use shortcuts such as chmod 777 or disabling SELinux to compensate for incorrect ownership.

8

What are Linux POSIX access control lists (ACLs)?

Access control lists (ACLs) extend Linux’s traditional owner, group, and other permission model with per-user and per-group entries. They allow permissions to be granted to specific identities without changing a file’s owning group.

Read full answer

Applies toLinux 5.x/6.x kernels with POSIX ACL support; Ubuntu 22.04/24.04 and RHEL 9

How ACLs Work

A standard Linux permission set has three classes:

Class Meaning
Owner Permissions for the file’s owning user.
Group Permissions for the owning group.
Other Permissions for identities that do not match a more specific ACL entry.

An extended POSIX ACL can add:

  • user:alice:r-x: Alice can read and traverse the object.

  • group:developers:r-x: members of developers can read and traverse it.

  • mask::r-x: The maximum effective permissions granted by named users, named groups, and the owning group in an extended ACL.

The owner and other entries are not restricted by the mask. The kernel evaluates ACL entries during each access check. On filesystems such as ext4 and XFS, ACL metadata is commonly stored in the system.posix_acl_access and system.posix_acl_default extended attributes.

Effective Permissions

An extended ACL retains the three traditional base entries and may contain one or more named-user or named-group entries. When multiple entries apply, the kernel combines the applicable permissions and then applies the mask.

The effective permissions visible to a process can therefore be narrower than the permissions displayed in a named entry. The getfacl command exposes this by marking permissions limited by the mask as #effective.

The + character shown by ls -l indicates that an object has an extended ACL. Use getfacl PATH to inspect its complete access and default ACLs.

Default ACLs and Inheritance

  • Access ACL: Controls access to the directory or file itself.

  • Default ACL: Exists only on directories and provides the template for newly created children.

  • A newly created regular file inherits an access ACL derived from its parent’s default ACL; it does not receive a default ACL because files cannot contain one.

  • A newly created subdirectory inherits both an access ACL and a default ACL, allowing it to pass the template to later descendants.

  • Existing objects are not changed when a default ACL is added.

  • The mode requested when creating an object constrains the inherited entries. When no parent default ACL exists, the normal umask calculation determines access permissions.

  • Use getfacl to verify inherited entries and distinguish active access rules from default: entries.

Production Operations

  • Prefer group ownership: Use ACLs primarily for exceptions rather than replacing a coherent ownership and group model.

  • Audit access: getfacl -R /srv recursively reviews access and default ACLs under /srv; look for unexpected named-user entries and overly broad masks.

  • Manage declaratively: Store required ACLs in configuration management or infrastructure-as-code to prevent drift.

  • Preserve metadata during transfers: tar --acls preserves ACLs in GNU tar archives, rsync -A preserves ACLs during synchronization, and cp --preserve=all preserves supported metadata when copying.

  • Include ACLs in backups: A byte-for-byte backup may not retain filesystem metadata unless the backup format and tooling support ACLs.

  • Monitor provisioning: Verify that newly created files receive the expected inherited permissions, especially after filesystem, application, or deployment changes.

Failure Modes and Security

  • The filesystem must support POSIX ACLs; ext4 and XFS do. Check the mount and filesystem configuration if ACLs cannot be set.

  • chmod can change the ACL mask on an extended ACL, reducing the effective permissions of named entries even though those named entries themselves remain unchanged.

  • ACLs store numeric user and group IDs. If an account is deleted and its UID is later reused, the new account can inherit the old account’s permissions.

  • A broad default ACL can grant unintended access to every future object in a directory.

  • NFSv4 ACLs are a separate protocol with different semantics. Samba and Windows ACL mappings are not automatically interchangeable with local POSIX ACLs.

  • POSIX ACLs are independent of SELinux labels. On RHEL-family systems, the security context must also permit the operation.

9

How do you manage Linux users and groups?

Linux user and group administration uses useradd, usermod, userdel, groupadd, groupmod, and groupdel, normally with sudo. Accounts can use local files or centralized sources such as LDAP, Active Directory, or FreeIPA.

Read full answer

Applies toLinux (Ubuntu 22.04/24.04 and RHEL 9)

Core Concepts

  • Primary group: Every user has one default group, recorded in /etc/passwd.

  • Supplementary groups: Grant additional access, commonly through groups such as developers or docker.

  • Account database: /etc/passwd contains public account metadata, /etc/shadow stores password hashes, and /etc/group and /etc/gshadow contain group membership and protected group data.

  • Name Service Switch: getent queries the same identity sources applications use, so it can reveal local, LDAP, or SSSD-backed accounts.

  • Home directories: The FHS normally places personal home directories under /home, while shared application data may belong under /srv or /var/lib.

Command Behavior

  • useradd creates an account; usermod changes an existing account; userdel removes it.

  • groupadd, groupmod, and groupdel manage groups.

  • passwd sets or changes passwords interactively.

  • Common options:

  • -m: Create or move the home directory.

  • -U: Create a group with the same name as the user.

  • -s: Set the login shell.

  • -aG: Add supplementary groups without removing existing memberships.

  • -d: Set the home directory path.

  • -g: Set the primary group.

  • -r: Remove home and mail data when deleting a user.

  • groupmod -n renames a group while retaining its GID.

  • gpasswd -d user group removes supplementary membership.

  • chage manages password age and account-expiry policy.

  • On Ubuntu, install the tools with sudo apt install passwd; on RHEL/Fedora, use sudo dnf install shadow-utils.

Code Example

Create a group, user, password, and supplementary membership:

sudo groupadd developers
sudo useradd -m -U -s /bin/bash alice
sudo passwd alice
sudo usermod -aG developers alice

id alice
getent passwd alice

Typical output; numeric IDs vary:

uid=1001(alice) gid=1001(alice) groups=1001(alice),1002(developers)
alice:x:1001:1001::/home/alice:/bin/bash

Modify the account and rename the group:

sudo install -d -m 0755 /srv/home
sudo usermod -m -d /srv/home/alice alice
sudo usermod -s /bin/bash alice
sudo chage -M 90 alice
sudo groupmod -n platform developers

After stopping any processes owned by alice, back up the home directory and remove the account and group:

pgrep -a -u alice
sudo install -d -m 0700 /var/backups/local-accounts
sudo tar -C /srv/home -czf \
  /var/backups/local-accounts/alice-home-$(date +%F).tar.gz alice

sudo gpasswd -d alice platform
sudo groupdel platform
sudo userdel -r alice

getent passwd alice || echo "account removed"
getent group platform || echo "group removed"

Common Pitfalls

  • Replacing group memberships: usermod -G developers alice replaces all supplementary groups. Always use usermod -aG when adding one.

  • Ignoring active processes: userdel removes account records but does not terminate running processes. Stop services first to avoid deleted UIDs owning live resources.

  • Confusing lock and disable: passwd -l locks password authentication but does not constitute full account disablement. Also set the shell to /usr/sbin/nologin and apply an account-expiry policy.

  • Changing primary group ownership: usermod -g does not change existing files to the new group. Use chown selectively.

  • Using overly broad permissions: Avoid chmod 777. For shared directories, prefer chown, group permissions, and a setgid directory such as chmod 2775.

  • Ignoring stale memberships: Before deleting a group, inspect it with getent group groupname and remove remaining members.

Production Best Practices

  • Prefer centralized identity management for multi-host environments; local accounts can cause configuration drift.

  • Provision users and groups with Ansible or another idempotent configuration-management system rather than repeated manual commands.

  • Give each service a dedicated non-login account, for example:

sudo useradd --system --no-create-home \
  --home-dir /var/lib/appsvc \
  --shell /usr/sbin/nologin appsvc
  • Audit changes with auditd and review them using sudo ausearch -m USER_ACCT -ts recent.

  • Back up account data before destructive changes; encrypt sensitive exports and restrict their permissions.

  • On RHEL, preserve and restore SELinux contexts during backups. For the nonstandard /srv/home/alice path, define a persistent semanage fcontext rule for user_home_t before running restorecon -RFv; restorecon alone only applies the default policy match.

  • Prefer filesystem snapshots for complete metadata backups; a tar archive may not preserve all extended attributes and security contexts.

  • Review inactive accounts, password expiry, home-directory quotas, disk usage, and SSH-key access during regular access reviews.

10

How do /etc/passwd and /etc/shadow differ in Linux?

/etc/passwd is the general user account database, while /etc/shadow stores password hashes and password-aging information in a protected file. Modern Linux systems keep password verification data out of the world-readable /etc/passwd file.

Read full answer

Applies toLinux (Ubuntu 22.04/24.04, RHEL 9)

File Roles and Layout

Both files reside under /etc, which contains host-specific system configuration.

File Typical permissions Purpose
/etc/passwd 0644, root:root Public user metadata such as UID, primary GID, home directory and login shell
/etc/shadow 0640, root:shadow Password hashes, password status and account-expiration settings

The exact permissions can vary by distribution, but /etc/shadow must not be world-readable. root can read it regardless of ordinary file permissions.

A typical /etc/passwd entry has the format username:password-marker:UID:GID:GECOS:home-directory:login-shell; for example, deploy:x:1001:1001:Deploy Account:/home/deploy:/bin/bash. The x is a marker indicating that the password verifier is stored in /etc/shadow. Some service accounts use * or ! instead to prevent normal password authentication.

Password and Aging Data

A typical /etc/shadow entry contains fields separated by colons: username:password-hash:last-change:minimum-age:maximum-age:warning-period:inactive-period:expiration-date:reserved. Important fields include:

  • Password hash: Includes the algorithm identifier, salt and resulting hash. Modern Ubuntu commonly uses yescrypt ($y$), while other systems may use SHA-512 ($6$), SHA-256 ($5$) or older algorithms.

  • Last change: Number of days since the password was last changed.

  • Minimum and maximum age: Enforced password lifetimes.

  • Warning period: How many days before expiration the user is warned.

  • Inactive period: How long an expired password may remain unused before the account is disabled.

  • Expiration date: Absolute account expiration date.

Password hashes remain sensitive because attackers can attempt offline guessing. Restrict backups and diagnostic access accordingly.

Authentication and Permissions

During a normal login, PAM performs account-policy checks and verifies the supplied password against the protected hash. SSH public keys are not stored in /etc/shadow; SSH maintains separate key and authorization configuration.

Use getent rather than directly reading /etc/passwd when resolving users because it consults the configured Name Service Switch sources; for example, getent passwd deploy can return LDAP, NIS or other centrally managed identity as well as local accounts. Do not edit local copies of centrally managed accounts; make changes in the identity-management system. To inspect account status without exposing password hashes, use sudo passwd -S deploy or chage -l.

Production Operations and Failure Modes

  • Use supported tools: Create and modify accounts with useradd, usermod, passwd, chage and related utilities rather than appending raw text.

  • Use locking tools: For emergency access, vipw and vipw -s safely edit /etc/passwd and /etc/shadow using lock files and validation.

  • Maintain consistency: Restore or migrate /etc/passwd and /etc/shadow as a matched pair. A valid account entry with a missing shadow entry can cause login failures.

  • Protect backups: Include /etc/passwd, /etc/shadow, /etc/group, /etc/gshadow and related account files in encrypted, access-controlled backups.

  • Monitor failures: Corrupt records, broken UID mappings, incorrect ownership or malformed shadow fields can block login, cron jobs or services running as the affected user.

  • Check central identity systems: Determine whether an account is local, LDAP-backed or otherwise managed before changing it.

Best Practices

  • Keep /etc/shadow at least as restrictive as the distribution recommends, commonly 0640 root:shadow; never use chmod 777.

  • Do not weaken permissions to work around an access problem. Correct group membership, ownership or the underlying authorization policy instead.

  • Use passwd -S or chage -l to inspect account status without printing password hashes.

  • Lock credentials with supported commands such as usermod -L or passwd -l, and use chage for expiration policy.

  • Avoid disabling SELinux or other mandatory-access controls merely to permit shadow-file access.

  • Test restored account files in a staging environment before using them in production, especially when changing authentication infrastructure.

11

How does Linux sudo work, and how do you configure /etc/sudoers?

sudo is a Linux privilege-escalation utility that lets authorized users run commands as another user, usually root. It evaluates a policy in /etc/sudoers, authenticates the caller when required, and executes only the commands permitted by that policy.

Read full answer

Applies tosudo 1.9.x on Linux (Ubuntu 22.04/24.04, RHEL 9)

How sudo Works

The sudo executable is installed setuid root, allowing it to launch the requested process with the authorized user and group identities.

  1. sudo resolves the caller’s identity and parses its policy.

  2. It checks whether a matching rule permits the requested command and target identities.

  3. If authorization fails, it exits without running the command.

  4. If a valid authentication timestamp is unavailable, PAM authenticates the user, commonly with a password, MFA, or enterprise identity integration.

  5. sudo launches the command as the permitted runas user and optional runas group.

Authentication and authorization are separate: a valid password does not grant a command, and a policy entry does not bypass required authentication.

Option Purpose
sudo -u postgres command Run as user postgres
sudo -g app command Run with group app
sudo -l Show the caller’s permitted commands
sudo -v Validate or refresh the authentication timestamp
sudo -k Invalidate the cached authentication timestamp
sudo -n command Never prompt; fail if authentication is required
sudo -i Start a login shell for the target user

Understanding sudoers Configuration

/etc/sudoers uses this general format:

user host=(runas-user:runas-group) command

Important components include:

  • User: A username or %group, such as alice or %app-ops.

  • Host: The host on which the rule applies. ALL permits any host.

  • Runas user and group: The identities the command may assume. ALL:ALL allows any user and group.

  • Command: The executable and, optionally, its arguments. Use an absolute path.

The main file is /etc/sudoers; drop-in policies belong in /etc/sudoers.d. These files must be owned by root and must not be writable by unprivileged users.

$ stat -c '%A %U %G %n' /etc/sudoers /etc/sudoers.d
-r--r----- root root /etc/sudoers
drwxr-xr-x root root /etc/sudoers.d

Use visudo, not a normal editor, because it checks syntax before installing changes. Avoid direct cat > /etc/sudoers workflows because an error can lock administrators out.

Production Behavior and Auditing

Successful and failed sudo activity should be forwarded to centralized logging. On Ubuntu, sudo events commonly appear in journald and may also be written to /var/log/auth.log; RHEL commonly uses /var/log/secure and journald.

Useful commands include:

$ sudo -l -U alice
User alice may run the following commands on app-01:
    (root) /usr/bin/systemctl restart app.service

$ sudo visudo -c
/etc/sudoers: parsed OK
/etc/sudoers.d/app-operations: parsed OK

$ journalctl _COMM=sudo --since "30 minutes ago"

timestamp_timeout controls reuse of sudo’s cached authentication timestamp. passwd_timeout controls reuse of a password cached by an external PAM authentication mechanism. For sensitive administration, use short timeouts and explicit break-glass access.

In managed environments, generate sudoers policy through configuration management, review it as code, and test it with visudo -c before deployment. Preserve an independently tested emergency account and document how it is protected.

Code Example

This drop-in permits members of app-ops to restart one specific service and requires a sudo authentication on each invocation:

# /etc/sudoers.d/app-operations
Defaults:%app-ops timestamp_timeout=0
Defaults:%app-ops passwd_timeout=0
Defaults:%app-ops log_input

%app-ops ALL=(root) /usr/bin/systemctl restart app.service

Install and validate the drop-in safely:

# Verify the executable path before using it in sudoers.
$ command -v systemctl
/usr/bin/systemctl

$ sudo visudo -f /etc/sudoers.d/app-operations
$ sudo visudo -c
/etc/sudoers: parsed OK
/etc/sudoers.d/app-operations: parsed OK

On Ubuntu, %sudo commonly represents administrators. On RHEL and Fedora, %wheel is commonly used instead. Configure the appropriate group for each distribution.

Security Best Practices and Common Pitfalls

  • Grant the narrowest privilege possible: Prefer one target identity, one command, and one exact argument set.

  • Use absolute executable paths: Relative paths can introduce PATH and working-directory risks.

  • Avoid NOPASSWD: ALL: It enables unrestricted, unauthenticated root execution if an account or host is compromised.

  • Do not trust writable commands: A root-run script owned by an unprivileged user can be replaced with malicious code.

  • Understand argument matching: If arguments are specified, they must match; "" means no arguments. Omitting arguments generally permits any arguments.

  • Avoid granting editors and shells: Commands such as vi, less, or unrestricted shells can escape to a root shell. sudoedit is designed for controlled file editing.

  • Do not weaken filesystem permissions: Never make /etc/sudoers or its drop-ins writable by unprivileged users.

  • Never disable SELinux or AppArmor as a shortcut: Correct the service, path, label, or execution policy instead.

  • Review effective access: Run sudo -l after every policy change and test both allowed and denied operations without using an interactive root shell.

12

What is the difference between su and sudo on Linux?

su replaces the current session with one running under another user’s identity, while sudo runs a selected command as an authorized run-as user—normally root—without replacing the caller’s shell. sudo is generally preferable for routine administration because its policy can be narrowly scoped and its command activity is easier to audit.

Read full answer

Applies toLinux with sudo 1.9.x and util-linux 2.37+; Ubuntu 22.04/24.04, RHEL 9, and current Fedora

How They Work

  • su: Starts a shell under the target user’s UID and groups. A non-root caller normally authenticates with the target account’s password, although PAM configuration can change that behavior. su - creates a login environment, including the target user’s home directory and login-shell settings; plain su is not equivalent to a full login.

  • sudo: Starts a child process with the selected run-as identity and leaves the invoking shell intact. It normally authenticates the invoking user with that user’s own password, subject to PAM and sudo policy. A suitable NOPASSWD rule can remove the prompt for a particular command.

  • Policy: sudo rules live in /etc/sudoers and files included from /etc/sudoers.d. Their general form is user host = (runas) [options] command; ALL=(ALL:ALL) ALL grants unrestricted execution and should not be the default.

  • Authentication cache: With the default timestamp_type=tty, sudo normally caches credentials for five minutes per user and terminal, typically under /run/sudo/ts/. sudo -k invalidates the caller’s cached credential. Distros may override these defaults.

  • Auditing: sudo normally records the invoking user, run-as user, command, terminal, and working directory through the configured syslog or audit destination. A full login session with su - is easier to correlate with a user, but individual commands inside it are not guaranteed to receive equivalent command-level audit records.

Key Differences

Aspect su sudo
Scope Replaces the session, or can run one command with -c Runs a command or scoped shell while retaining the caller’s session
Authentication Normally asks for the target account’s password Normally asks for the invoking user’s password
Authorization Controlled mainly by target-account access and PAM Controlled by explicit per-user, host, run-as user, command, and option rules
Environment su - initializes a login environment; plain su may retain caller variables env_reset is enabled by default; secure_path applies when configured by sudoers
Auditability PAM may log session start and closure, but per-command auditing is not guaranteed Usually provides command-level logging through syslog, journald, or auditd
Typical use Occasional full switch to another account Routine one-off administration and tightly authorized automation

Security and Failure Modes

  • Use visudo rather than a normal editor. It validates sudoers syntax and detects unsafe ownership or permissions. Sudoers files should be owned by root:root and mode 0440; never make them or /etc/sudoers.d writable by unprivileged users.

  • Plain su may retain the caller’s PATH, while su - establishes a login environment. Sudo protects command lookup through secure_path only when that option is explicitly configured; do not assume it is present on every system.

  • A sudoers rule can be restricted to an exact binary and fixed arguments, such as /usr/bin/systemctl restart nginx. Merely naming a broadly capable binary without restricting its arguments may grant more access than intended.

  • For automation, use sudo -n so the job fails rather than waiting for input. Alternatively, provide a narrowly constrained NOPASSWD rule for a trusted service account; treat every passwordless rule as a persistent privilege-escalation path.

  • Nested sudo requires care. In CI or another non-interactive environment there may be no usable TTY timestamp, so use sudo -n, an approved NOPASSWD rule, or run the workload directly as the intended service account.

  • Distinguish authorization from mandatory access control. If sudoers permits the command but SELinux blocks it, inspect recent AVC events with ausearch -m avc -ts recent and apply a targeted policy or correct file labels with restorecon. Do not disable SELinux.

  • If sudo reports that it cannot resolve the host, correct /etc/hostname and /etc/hosts; do not ignore the warning because host matching and local audit records may depend on a consistent hostname.

Production Best Practices

  • Prefer sudo <command> for isolated work and sudo -i only when an interactive administrative shell is genuinely required. Avoid using su in automation.

  • Apply least privilege: grant only the required run-as identity, binary, fixed arguments, and options. Avoid unrestricted ALL rules and broad NOPASSWD grants.

  • On systems using @includedir /etc/sudoers.d, use dot-free filenames such as /etc/sudoers.d/deployer. The standard include behavior skips names containing a dot and names ending in ~, so deployer.sudo would not be loaded.

  • Set a reasonable timestamp_timeout for the environment, and use sudo -k when handing a privileged terminal to another operator.

  • Back up sudoers configuration before changes, validate it with visudo -c, and test access with sudo -l before deployment.

  • Monitor the appropriate destination—commonly /var/log/auth.log on Ubuntu, /var/log/secure on RHEL/Fedora, and journal or audit records where configured—and forward relevant events to centralized alerting and access-review processes.

13

How do you search for files using find and locate on Linux?

find searches the live filesystem by recursively traversing directories and evaluating expressions, while locate searches a periodically generated index of file paths. Use find when results must reflect the current filesystem; use locate for fast, broad searches where a slightly stale result is acceptable.

Read full answer

Applies toGNU findutils 4.x; locate/mlocate/plocate on Ubuntu 22.04/24.04 and RHEL 9

How find Works

find receives a starting path, followed by tests, actions, and logical operators. Without an explicit action, it prints matching paths.

find /var/log -type f -name '*.log' -mmin -60 -print

Typical output:

/var/log/app/api.log
/var/log/nginx/access.log
  • -type f: Match regular files; use -type d for directories.

  • -name '*.log': Match the filename using a shell-style pattern.

  • -iname '*.log': Perform a case-insensitive filename match.

  • -mmin -60: Match files modified less than 60 minutes ago.

  • -print: Print each matching path.

Other useful tests include:

  • -user alice: Match files owned by alice.

  • -group web: Match files belonging to the web group.

  • -perm -u+w: Match files to which the owner has write permission.

  • -size +100M: Match files larger than 100 MiB, with GNU find rounding sizes into blocks.

Tests are combined with logical AND by default. Use -o for OR and parentheses for grouping. Starting paths such as /var/log, /etc, /home, and /srv correspond to the standard Linux filesystem hierarchy.

How locate Works

locate queries a database generated by updatedb. It is much faster for repository-wide searches because it searches an index rather than walking every directory.

locate -e -b nginx.conf

Typical output:

/etc/nginx/nginx.conf
  • -e: Return only entries that still exist.

  • -b: Match only the basename rather than the complete indexed path.

  • -i: Ignore case.

  • -l 50: Limit the output to 50 results.

The index may not contain files created moments ago, and it can contain paths deleted since the last update. Therefore, locate is unsuitable for finding recently generated files or deciding exactly what to delete without verifying the result.

find Compared with locate

Tool Data source Speed Freshness Appropriate use
find Live directory tree Slower Current Recent files, exact checks, safe mutations
locate Prebuilt path database Faster Potentially stale Broad searches across many directories
fd Live directory tree Fast Current User-friendly alternative where available

Code Example

Install and refresh the appropriate locate implementation:

# Ubuntu/Debian: choose this installation
sudo apt-get install plocate

# RHEL/Fedora: alternatively choose this installation
sudo dnf install mlocate

# Refresh the index; configuration is commonly in /etc/updatedb.conf
sudo updatedb

Combine find with null-delimited output when filenames may contain spaces or newlines:

find /srv/logs -type f -name '*.gz' -print0 | xargs -0 du -h

For a potentially destructive operation, preview first:

find /tmp/job-cache -type f -mtime +7 -print
find /tmp/job-cache -type f -mtime +7 -delete

The second command permanently deletes matching files and causes find to use depth-first traversal.

Operational Practices and Pitfalls

  • Use least privilege: An unprivileged find cannot traverse unreadable directories and may emit Permission denied. Use sudo only when the full search is necessary, and avoid globally hiding errors with 2>/dev/null during production investigations.

  • Restrict the scope: find / -xdev ... prevents traversal into other mounted filesystems. Exclude virtual filesystems such as /proc, /sys, and /dev when appropriate.

  • Treat locate as a hint: Verify important paths with test -e, stat, or find before acting.

  • Monitor index maintenance: If automation depends on locate, monitor the distribution’s plocate or updatedb timer for failures. The database can always be regenerated, so backups are normally unnecessary.

  • Avoid broad deletion commands: Never use an unrestricted pattern such as find / -delete. Use a fixed directory, explicit type and age filters, and a preview step.

14

How do you search file contents with grep on Linux?

grep searches files for lines matching a pattern and prints the matching content. It is useful for troubleshooting logs, configuration files, source code, and command output.

Read full answer

Applies toGNU grep 3.6+ on Linux (Ubuntu 22.04/24.04 and RHEL 9)

Core Usage

The basic syntax is:

grep [OPTIONS] PATTERN [FILE ...]

For example, grep 'ERROR' app.log prints every line containing ERROR. Without a file argument, grep reads standard input, which makes it usable in pipelines:

journalctl -u myapp.service --since '1 hour ago' | grep 'ERROR'

Patterns are basic regular expressions (BRE) by default. Use -E for extended regular expressions or -F when the pattern should be treated as literal text.

Important Options

Option Purpose
-i Perform a case-insensitive search
-n Prefix each result with its line number
-c Print only the number of matching lines
-l Print filenames containing a match
-L Print filenames containing no match
-o Print only the matched portion of each line
-v Print nonmatching lines
-w Match the pattern as a complete word
-E Use extended regular expressions
-F Treat the pattern as fixed, literal text
-C N Show N lines before and after each match
-A N / -B N Show lines after or before each match
-r Search directories recursively, following command-line symbolic links but not links encountered during recursion
-R Search recursively while following all symbolic links
--include Restrict recursion to matching filenames
--exclude-dir Skip directories during recursion
-- End option parsing; use before patterns beginning with -

Quoting patterns protects them from shell expansion. Single quotes are usually preferable when the pattern contains spaces or regular-expression metacharacters.

Under the Hood

GNU grep reads files sequentially and evaluates the pattern against each line. Standard output receives matching lines, while standard error receives warnings and errors.

Its exit status is important for automation:

  • 0: At least one line matched.

  • 1: No lines matched.

  • 2: An error occurred, such as an unreadable file or invalid pattern.

Unlike tools such as ripgrep, grep does not automatically respect .gitignore or skip hidden files. Recursive searches can therefore be slower than expected.

Production Failure Modes

  • Unbounded searches: Avoid commands such as grep -R 'pattern' /; they traverse virtual filesystems, device files, and large log trees. Restrict the search path.

  • Permissions: A missing file can produce exit status 2. Do not solve this with chmod 777; grant only the required access or use appropriately scoped service accounts.

  • Symbolic links: -r follows symbolic links supplied on the command line but not those encountered during recursion, while -R follows all symbolic links. Use -R cautiously on untrusted trees.

  • Binary files: grep may report “binary file matches.” The -a option forces text processing, but this can produce excessive output.

  • Compressed and journal logs: Plain grep does not transparently search gzip files. Use zgrep for gzip-compressed files and journalctl -g for systemd journals.

  • Sensitive data: Search results may expose credentials or tokens. Avoid copying configuration matches into tickets, shared terminals, or monitoring logs.

Code Example

mkdir -p demo/logs

printf '%s\n' \
  '2026-03-08T10:00:00Z INFO service started' \
  '2026-03-08T10:01:12Z ERROR upstream timeout after 30s' \
  '2026-03-08T10:02:00Z FATAL worker stopped' \
  > demo/logs/app.log

grep -n 'ERROR' demo/logs/app.log
# 2:2026-03-08T10:01:12Z ERROR upstream timeout after 30s

grep -Rni --include='*.log' 'timeout' demo
# demo/logs/app.log:2:2026-03-08T10:01:12Z ERROR upstream timeout after 30s

grep -En 'ERROR|FATAL' demo/logs/app.log
# 2:2026-03-08T10:01:12Z ERROR upstream timeout after 30s
# 3:2026-03-08T10:02:00Z FATAL worker stopped

Best Practices

  • Use grep -F for literal strings; it avoids regex interpretation and is typically faster for fixed-string searches.

  • Narrow recursive searches with explicit directories, --include, and --exclude-dir.

  • Use -n during troubleshooting so results can be correlated with source files or log positions.

  • Treat exit status 1 as “no match” and exit status 2 as an operational failure; monitoring should not label both as errors.

  • For very large repositories, use ripgrep when available. For systemd-managed logs, prefer journalctl with filters before applying grep.

15

How do you view and follow Linux logs with tail, less and journalctl?

Use tail for quick file inspection, less for interactive searching, and journalctl for querying systemd-journald’s structured journal. In production, account for log rotation, permissions, disk usage, retention, centralized collection, and monitoring.

Read full answer

Applies toLinux with GNU coreutils 8.32+, systemd 249+, and less 551+ (Ubuntu 22.04/24.04, RHEL 9, and equivalent current Fedora releases)

How the Tools Differ

  • tail: Reads a file from the end. It is lightweight and useful for checking recent entries or following a file continuously.

  • less: Opens a file interactively and reads it in blocks, making it suitable for searching large logs without loading the entire file into memory.

  • journalctl: Queries systemd-journald’s structured journal and can filter by unit, boot, time range, priority, PID, executable, and other fields.

Traditional logs commonly live under /var/log; journal data is stored separately:

Log source Ubuntu/Debian RHEL/Fedora
General system messages /var/log/syslog /var/log/messages
Authentication messages /var/log/auth.log /var/log/secure
Service-specific files /var/log/<service>/ /var/log/<service>/

Operational Behavior

  • tail -n N: Prints the last N lines. With -f, it follows the opened file descriptor. With -F, it continues following by name and reopens the file after rotation.

  • less: /pattern searches forward, n selects the next match, g and G jump to the beginning or end, and q exits. -N displays line numbers, while -S chops long lines and permits horizontal scrolling instead of wrapping.

  • less +F: Starts in forward-follow mode. Press Ctrl+C to interrupt following and search or scroll; press F to resume following.

  • journalctl: Uses a pager, normally less, when writing to a terminal. -f follows new entries, -b selects a boot, -b -1 selects the previous boot, -r reverses the order, and -p warning includes warning and more severe priorities.

  • Journal persistence: Journald may use persistent storage under /var/log/journal and volatile storage under /run/log/journal, depending on its configuration.

Code Example

The following file examples use Ubuntu paths. On RHEL/Fedora, use /var/log/messages and /var/log/secure instead.

$ sudo tail -n 3 /var/log/auth.log
Sep  8 10:14:02 host CRON[2418]: pam_unix(cron:session): session opened
Sep  8 10:14:09 host sshd[2431]: Accepted publickey for deploy from 10.0.0.25 port 52214 ssh2
Sep  8 10:15:01 host systemd[1]: Starting Session 12 of user deploy.

Use less when the file is large or interactive searching is needed:

$ sudo less -N -S /var/log/auth.log

Follow a rotated file safely by name:

$ sudo tail -F /var/log/syslog
Sep  8 10:15:03 host systemd[1]: Started Session 12 of user deploy.
Sep  8 10:15:07 host cron[2440]: (root) CMD (/usr/local/bin/health-check)

Press Ctrl+C to stop. The interactive equivalent is:

$ sudo less +F /var/log/syslog

Query the journal and follow a service:

$ sudo journalctl -u ssh.service -n 3 --no-pager -o short-iso
2025-09-08T10:14:09+0000 host sshd[2431]: Accepted publickey for deploy from 10.0.0.25 port 52214 ssh2
2025-09-08T10:14:09+0000 host sshd[2431]: pam_unix(sshd:session): session opened
2025-09-08T10:14:31+0000 host sshd[2452]: Received disconnect from 10.0.0.25 port 52216:11

$ sudo journalctl -u nginx.service -f
$ sudo journalctl -b -p warning --since "1 hour ago" --no-pager

On RHEL/Fedora, OpenSSH commonly uses sshd.service, so use -u sshd.service.

Production Practices and Pitfalls

  • Rotation: tail -f can remain attached to an old, renamed inode and miss the active file. Prefer tail -F for rotated files.

  • Separate collectors: Do not assume journalctl contains entries from traditional files; journald and tools such as rsyslog are separate collectors.

  • Disk pressure: Monitor traditional log files and journal usage with journalctl --disk-usage. Apply an approved retention policy.

  • Rate limiting: Journald can suppress excessive messages. Monitor service-level symptoms rather than assuming every message was persisted.

  • Permissions: Logs can contain IP addresses, usernames, tokens, and request details; restrict access and use audited group membership such as systemd-journal where appropriate.

  • Monitoring: Ship logs to a centralized platform for alerting, long-term search, and incident correlation rather than relying on an attached terminal.

  • Maintenance: Test log collection and journal configuration before upgrades, and preserve critical logs according to retention, backup, and compliance requirements.

17

How do Linux inodes work, and why does inode exhaustion cause failures?

An inode is a filesystem record containing metadata about a file, including its owner, permissions, timestamps, size, and data mappings. Linux stores filenames separately in directories; when a filesystem cannot allocate another inode, operations requiring a new filesystem object fail with ENOSPC—usually displayed as No space left on device—even when data blocks remain available.

Read full answer

Applies toRHEL 9 (Linux 5.14) and Ubuntu 24.04 (Linux 6.8); ext4 and XFS

How Inodes Work

  • Filename versus inode: A directory entry maps a filename to an inode number. The filename is not stored in the inode itself.

  • Metadata: Ownership, mode bits, timestamps, link count, file size, and block or extent mappings are recorded in the inode.

  • Filesystem objects: Regular files, directories, symbolic links, FIFOs, sockets, and device nodes each require an inode. Hard links are different directory entries referring to the same inode, so they do not consume additional inodes.

  • Permissions: Commands such as chmod and chown modify inode metadata. Removing or renaming a directory entry is governed primarily by permissions on the parent directory, including its sticky bit and ACLs.

  • Filesystem scope: Inode numbers and inode capacity are local to a filesystem. A separate /var, /tmp, database, or container-storage filesystem can exhaust its inodes while / still has plenty available.

On ext4, the inode capacity is normally fixed when the filesystem is created, although it can be changed through filesystem resizing operations. XFS allocates inodes dynamically, subject to available filesystem space and accounting limits, but it can still fail when another inode cannot be allocated.

Why Exhaustion Causes Failures

Creating a new file, directory, symbolic link, or other object requires the filesystem to allocate an inode and a directory entry. If inode allocation fails, the operation returns ENOSPC before the operation can complete. This commonly occurs when a workload creates many small files, temporary files, cache entries, log files, container layers, or session artifacts.

Writes to an existing file may continue when sufficient data blocks are available because extending an existing inode does not require allocating another inode. Operations such as renaming an existing file within the same filesystem may also succeed, subject to directory-block space and permissions. Inode exhaustion therefore does not necessarily stop all writes immediately.

Symptoms and Diagnosis

Typical symptoms include:

  • New files, directories, uploads, and temporary files cannot be created.

  • apt or dnf installations fail while unpacking packages.

  • Log rotation fails because creating the replacement file requires another inode.

  • Applications report write failures even though filesystem byte usage appears healthy.

  • Service incidents generate files faster than retention or cleanup policies remove them.

Use df -i <mountpoint> to inspect inode usage and df -h <mountpoint> to inspect data-block usage. These must be monitored independently because inode exhaustion and block exhaustion produce the same ENOSPC error.

To locate inode-heavy directories, run du --inodes -x --max-depth=1 <path>. The --inodes option counts filesystem objects, -x prevents crossing filesystem boundaries, and --max-depth=1 limits the output to the selected directory and its immediate children. Review the results before deleting anything.

Inspect deleted-but-open files with sudo lsof +L1. An unlinked file continues consuming blocks and its inode while a process holds an open descriptor. Restarting or coordinating with the owning process closes the descriptor and releases the resources, although restarting a production service carries incident risk.

Safe Recovery

  1. Identify the affected filesystem: Check every important mount with df -i, not only /.

  2. Locate the consumer: Use du --inodes to identify directories containing large numbers of files, directories, or symbolic links.

  3. Check unlinked files: Use lsof +L1 to identify deleted files that remain open.

  4. Remove unnecessary objects safely: Apply retention policies to caches, old logs, temporary files, abandoned uploads, and container layers. Do not delete active database files or unknown system files.

  5. Avoid false remedies: Truncating a file releases its allocated data blocks but normally keeps the inode because the file still exists. chmod 777 does not recover inodes and weakens security.

  6. Verify recovery: Re-run df -i and confirm that the affected application, package-manager, and logging operations succeed.

Production Practices

  • Monitor inode utilization, inode growth rate, and data-block utilization independently; alerting only on byte usage misses inode exhaustion.

  • Investigate sudden inode growth, which often indicates runaway temporary files, excessive deployment churn, cache accumulation, or logging loops.

  • Remember that journald may fail to create a new journal file during inode exhaustion, reducing observability during an incident.

  • Use ext4 or XFS quota features, including inode limits, for workloads likely to create many small objects.

  • Plan filesystem capacity and retention policies so cleanup occurs before inode exhaustion becomes an availability incident.

18

How do you check Linux disk usage with df and du?

df reports space usage for mounted filesystems, while du calculates disk usage for files and directories. Use df to determine whether a filesystem is full and du to identify which paths are consuming space.

Read full answer

Applies toGNU coreutils 8.32–9.x and lsof 4.x on Linux (Ubuntu 22.04/24.04, RHEL 9, and current Fedora)

How df and du Differ

Tool Primary question Typical scope
df Is a mounted filesystem full? One mounted filesystem
du Which files or directories consume space? A directory tree
df -i Are filesystem inodes exhausted? One mounted filesystem
du -x How much is used without crossing filesystems? One filesystem
  • df obtains capacity information from the filesystem, including blocks, reserved space, and inodes.

  • du traverses directory entries and sums the space allocated to files.

  • du uses allocated blocks by default, not apparent file size. Sparse files and compressed or deduplicated files can therefore differ from their logical size.

  • du counts a hard-linked file only once within the same traversal, while df reflects the filesystem’s actual block allocation.

Interpreting the Results

  • A filesystem can have plenty of data space but no free inodes. Each filesystem object, such as a regular file or directory, has an inode; hard-linked names share one inode.

  • df can show used space that du cannot locate when a process still has a deleted file open. The directory entry is gone, but the filesystem cannot release its blocks until the open file descriptor is closed.

  • df and du may disagree because the commands ran in different mount or container namespaces.

  • du normally reports permission errors for directories the current user cannot read. Use sudo when necessary instead of discarding errors with 2>/dev/null.

  • du without -x can cross into mounted filesystems and attribute another mount’s usage to the starting path.

  • Some filesystems reserve blocks for privileged users, so Size, Used, and Avail may not add up exactly as a simple calculation suggests.

Code Example

First inspect filesystem capacity, filesystem type, and inode usage:

$ df -hT /
Filesystem     Type  Size  Used Avail Use% Mounted on
/dev/nvme0n1p2 ext4   49G   18G   28G  39% /

$ df -ih /
Filesystem       Inodes  IUsed IFree IUse% Mounted on
/dev/nvme0n1p2 1310720  82431 1288289    6% /
  • -h: print human-readable values.

  • -T: display the filesystem type.

  • -i: inspect inode usage rather than data blocks.

  • /: report the filesystem containing that path.

Find the largest immediate subdirectories under /var:

$ sudo du -xhd1 /var | sort -h
4.0K  /var/tmp
8.0K  /var/backups
16M   /var/cache
72M   /var/log
768M  /var/spool
12G   /var/lib
13G   /var
  • -x: remain on one filesystem.

  • -h: print human-readable sizes.

  • -d1: report totals for each immediate subdirectory, excluding deeper descendants such as /var/lib/docker.

  • sort -h: sort values according to their human-readable size.

Inspect individual large files:

$ sudo du -ahx /var | sort -h | tail -20

This command can be expensive because du must traverse the entire tree. Limit it to the relevant mount or application directory during production incidents.

Check for deleted files that are still consuming space:

$ sudo lsof +L1
COMMAND   PID USER     FD  TYPE DEVICE SIZE/OFF NLINK NODE NAME
postgres 1842 postgres  12u  REG  254,1       1.5G     0  8912 /var/lib/postgresql/data/base/16384/24576 (deleted)

+L1 selects open files whose link count is less than one. First determine which service owns the file. Then close the descriptor through a safe application restart or another documented procedure; removing the path again will not release the blocks.

Install lsof if necessary:

# Ubuntu
sudo apt install lsof

# RHEL and Fedora
sudo dnf install lsof

Production Practices

  • Monitor both block usage and inode usage for every important mount. Alerting only on df can miss inode exhaustion.

  • Track growth by mount and application. Logs, databases, package caches, container layers, and temporary files usually cause gradual growth, but deleted-but-open files create sudden df increases.

  • Plan backup capacity and destinations carefully. A backup or snapshot must not fill its source filesystem, and backup retention must account for the space it consumes.

  • Avoid running unrestricted du / during an incident. A broad traversal can generate heavy metadata I/O and make recovery slower.

  • Use application-specific cleanup procedures. Remove rotated logs through the logging system, clean package caches with the package manager, and use supported container-storage cleanup commands rather than deleting database or runtime files directly.

  • Check the host or correct container namespace. In containers, df may describe the overlay filesystem while the application data resides on a separate volume.

  • Keep upgrade and incident headroom. df reflects the host filesystem’s actual limits, including reserved blocks and filesystem-specific metadata overhead.

19

How do you find the largest files on Linux?

Use df to identify which filesystem is full, then use du to find the directories consuming space and find to locate individual large files. If filesystem usage and visible file sizes do not reconcile, check for deleted files that are still held open by processes.

Read full answer

Applies toLinux (Ubuntu 22.04/24.04, RHEL 9; GNU coreutils and findutils)

Filesystem and Directory Triage

Start with df -hT to see filesystem type, capacity, and mount point; -h uses human-readable units and -T displays the filesystem type. Then use du against the relevant mount point to determine where space is allocated.

Common locations include:

  • /var: logs, package caches, databases, and container data

  • /home: user home directories

  • /opt: manually installed applications

  • /usr: installed operating-system and application files

  • /tmp: temporary data

  • /root: files belonging to the root account

If /var or /home is a separate mount, inspect it separately. The du -x option prevents du from crossing filesystem boundaries.

Understanding the Tools

  • df: reports filesystem capacity and is the authoritative view when investigating a full disk.

  • du: reports allocated disk blocks consumed by files and directories; it does not follow symbolic links by default.

  • find: enumerates files and can report apparent byte size or allocated blocks.

  • lsof +L1: finds open files with a link count below one, typically deleted files whose blocks remain allocated.

  • df -i: checks inode usage, which matters when free bytes remain but new files cannot be created.

A sparse file can have a large apparent size while occupying relatively few blocks. Compare ordinary du output with du --apparent-size when sparse-file behavior matters.

Code Example

Check filesystem capacity and inode usage:

$ df -hT /var
Filesystem     Type  Size  Used Avail Use% Mounted on
/dev/nvme0n1p2 ext4  200G  164G   36G  82% /

$ df -i /
Filesystem      Inodes  IUsed IFree IUse% Mounted on
/dev/nvme0n1p2 13056000 821342 12134658    7% /

Find the largest entries on the root filesystem:

$ sudo du -x -h --max-depth=1 / 2>/dev/null | sort -hr | head -n 6
27G    /
12G    /var
8.4G   /usr
3.1G   /home
1.8G   /opt
680M   /tmp

The options mean:

  • -x: remain on one filesystem.

  • -h: print human-readable values.

  • --max-depth=1: include the starting directory and its immediate children.

  • 2>/dev/null: suppress permission and transient-filesystem errors.

  • sort -hr: sort using human-readable numeric ordering, largest first.

  • head -n 6: retain the six largest entries, including the aggregate / line.

Drill into /var:

$ sudo du -x -h --max-depth=1 /var 2>/dev/null | sort -hr | head -n 5
12G    /var
6.1G   /var/lib
4.0G   /var/log
820M   /var/cache
95M    /var/tmp

List large files by allocated space, using 1-KiB blocks and converting them to IEC units:

# %k is allocated 1-KiB blocks; awk converts the block count to human units.
sudo find /var -xdev -type f -printf '%k\t%p\n' 2>/dev/null \
  | sort -nr | head -n 10 \
  | awk -F '\t' '
      function human(k) {
        if (k >= 1048576) return sprintf("%.1f GiB", k / 1048576)
        if (k >= 1024) return sprintf("%.1f MiB", k / 1024)
        return sprintf("%d KiB", k)
      }
      { printf "%s\t%s\n", human($1), $2 }
    '
8.0 GiB  /var/lib/docker/overlay2/.../diff/layer.tar
4.1 GiB  /var/log/journal/ab/cd/journalfile
1.2 GiB  /var/lib/containers/storage/overlay-file

Here, -xdev prevents crossing mounts, -type f selects regular files, %k reports allocated 1-KiB blocks, %p prints the path, sort -nr sorts numerically in descending order, and head -n 10 limits the result.

Check for deleted files that still consume space:

$ sudo lsof +L1 | head -n 20
COMMAND     PID USER     FD  TYPE DEVICE  SIZE/OFF NLINK  NODE NAME
nginx      1242 www-data 14w  REG  253,1 104857600     0 12345 /var/log/nginx/access.log (deleted)

If lsof is unavailable:

# Ubuntu
sudo apt-get update && sudo apt-get install -y lsof

# RHEL
sudo dnf install -y lsof

The space is released only after every process holding the descriptor has closed it, usually by stopping or restarting all relevant owners.

Failure Modes and Operational Risks

  • Separate mounts: du -x / will not include a separate /var filesystem.

  • Deleted open files: du may not show them, while df still reports their blocks as used.

  • Inode exhaustion: many tiny files can exhaust inodes despite available bytes.

  • Long-running scans: du and find create metadata traffic; schedule them carefully and lower their I/O priority when necessary.

  • Network storage: scanning NFS, EFS, or similar mounts can be slow and generate significant traffic.

  • Sensitive paths: use appropriate privileges and avoid exposing directory names in shared logs or tickets.

Production Best Practices

  • Alert on both percentage usage and absolute free space; monitor inode usage separately.

  • Establish retention policies for logs, journal files, container layers, caches, and temporary files.

  • Investigate growth trends rather than treating a one-time snapshot as the root cause.

  • Use ncdu -x / for an interactive read-only view when ncdu is available.

  • Do not delete database files, active logs, or container layers directly; use supported rotation, cleanup, and retention mechanisms.

  • After remediation, verify with df -hT, df -i, and du, and confirm that no process still holds deleted files open.

20

How can I list and control processes using ps, top, and htop?

ps, top, and htop are essential commands for listing and managing processes on Linux systems. They provide different perspectives—from static snapshots to real-time interactive views—allowing administrators to monitor resource usage, diagnose issues, and control running processes.

Read full answer

Applies toLinux (Ubuntu 22.04/24.04, RHEL 9) with procps-ng (ps, top) and htop 3.3+

1. ps: static process listing

ps displays a snapshot of current processes. It is ideal for scripting and quick checks because its output is stable and easy to parse.

Common flags: - ps aux – BSD-style listing of all processes. - ps -ef – POSIX-style listing with full command. - ps -p PID -o pid,comm,args – custom output for a specific process.

Key columns: PID (process ID), %CPU, %MEM, STAT (status codes like S for sleeping, R for running), and COMMAND.

2. top: real-time monitoring

top provides a continuously updated view of system activity. It is perfect for spotting sudden spikes or identifying the most CPU‑intensive processes.

Run top interactively, or use batch mode for logging.

Interactive keys inside top: - M – sort by memory usage. - P – sort by CPU usage. - k – kill a process (prompts for PID and signal). - r – renice a process. - q – quit.

3. htop: interactive process management

htop is a user‑friendly rewrite of top with colour, mouse support, and easier process control. Install it with the package manager.

A typical htop screen shows a full‑screen list with colour-coded resource bars.

Interactive actions: - F5 – toggle tree view. - F6 – sort by a chosen column. - F9 – kill the selected process (choose signal). - F7/F8 – decrease/increase the nice value (raise/lower priority). - F10 – quit.

Because htop shows all processes in a full‑screen list, you can simply move the cursor and press F9 to send a signal, making it the fastest tool for ad‑hoc process management.

4. Comparison of ps, top, and htop

Tool Output style Interactive? Best use case
ps Static snapshot No Scripting, logging, quick one‑off checks
top Real‑time, text Yes (basic) Monitoring CPU/memory trends, identifying top consumers
htop Real‑time, colour Yes (advanced) Interactive process management, user‑friendly diagnostics

5. Managing processes with kill, renice, and pkill

The kill command sends signals to processes. Common signals:

Signal Name Purpose
15 (SIGTERM) Terminate Ask process to shut down gracefully
9 (SIGKILL) Kill Force immediate termination (cannot be caught)
2 (SIGINT) Interrupt Interrupt current operation (like Ctrl+C)

Examples of process control commands are shown in the code example section.

Code Example

# ps examples
ps aux
ps -ef
ps -p 1056 -o pid,comm,args

# Example ps aux output:
# USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
# root       512  0.0  0.0  40624  2540 ?        Ss   Apr19   0:01 /sbin/init
# user1     1023  0.1  0.2 123456  78912 ?        Sl   Apr19   0:45 /usr/bin/Xorg

# top batch mode (one iteration)
top -b -n 1

# Example top batch output:
# top -  10:35:21 up 10 days,  2:30,  2 users,  load average: 0.15, 0.05, 0.01
# Tasks: 123 total,   1 running, 122 sleeping,   0 stopped,   0 zombie
# %Cpu(s):  2.3 us,  1.0 sy,  0.0 ni, 96.7 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
# KiB Mem :  8094532 total,  2345678 free,  3456789 used,   2345678 buff/cache
# KiB Swap:  2097152 total,  2097152 free,        0 used.  4567890 avail Mem
#   PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND
#  1023 user1     20   0  123456   78912  12345 S   0.3  0.8   0:45.67 Xorg

# htop installation
# Ubuntu/Debian
sudo apt install htop
# RHEL/Fedora
sudo dnf install htop

# Process control
kill 1056               # send SIGTERM
kill -9 1056            # send SIGKILL
kill -TERM 1056         # same as first
renice -n 5 -p 1056     # change priority (higher number = lower priority)
pkill -TERM nginx       # kill all processes named “nginx”
pgrep -f "python script" # find PID of a process matching a pattern

Best practices and safety

  • Prefer SIGTERM over SIGKILL. A well‑behaved process will clean up files, close connections, and release resources when it receives SIGTERM. Use SIGKILL only when the process ignores SIGTERM and must be stopped immediately.

  • Verify before killing. Use ps, pgrep, or htop to confirm you are targeting the correct process. Accidentally killing a critical system process can cause an outage.

  • Use sudo sparingly. Run commands as the process owner when possible; sudo should be reserved for administrative tasks.

  • Leverage systemctl for services. For managed services (e.g., web servers, databases), use systemctl start|stop|restart instead of manual kill.

  • Monitor logs. After killing or restarting a process, check journalctl -u <service> or application logs for error messages.

  • Keep htop installed. In production environments, having htop available on your jump hosts and servers simplifies troubleshooting.

21

What are the Linux process states?

Linux process states describe the scheduler’s current view of a task: it may be running or waiting for CPU, sleeping for an event, blocked in an uninterruptible operation, stopped, or awaiting reaping after exit. The first character of the STAT column in ps output shows the primary state.

Read full answer

Applies toLinux kernel 4.18–6.x with procps-ng 3.3.17–4.0.x; Ubuntu 22.04/24.04 and RHEL 9

Common Process States

State Meaning Operational interpretation
R Running or runnable The task is executing on a CPU or is ready to run. It does not necessarily mean the task is currently consuming CPU.
S Interruptible sleep The task is waiting for an event such as a socket response, timer, or I/O completion. Most normal waiting processes use this state.
D Uninterruptible sleep The task is waiting in an operation that cannot normally be interrupted, commonly block-device, filesystem, or NFS I/O. Persistent D tasks can indicate a storage or network-filesystem problem.
T Stopped by job control The process was suspended, commonly by SIGSTOP or SIGTSTP, and resumes after SIGCONT.
t Tracing stop The task has been stopped for inspection by a debugger using ptrace.
Z Zombie The process has exited, but its parent has not yet collected its exit status.
I Idle kernel thread The task is an idle kernel worker rather than an ordinary application process.
X or x Dead The task is dead. These states are normally transient and rarely visible.
P Parked A rarely observed state in which a task has been explicitly parked by the kernel.

The old W paging or wakekill code and K wakekill code are obsolete in modern kernels and are not normal current states.

Reading Process States

Use ps -eo pid,ppid,stat,wchan:24,comm --sort=pid to show the process identifier, parent identifier, combined state and modifiers, kernel wait channel, and command name. The STAT column combines the primary state with optional modifiers: for example, Ss means interruptible sleep and session leader, while R+ means runnable and in the foreground process group.

The wchan field is a clue about where a sleeping task is waiting, not a complete root-cause diagnosis; its visibility also depends on kernel configuration and privileges. /proc/PID/status provides a human-readable state for an individual task. Counting primary state characters is useful for trend monitoring, but every result is only a point-in-time snapshot.

State Transitions and Signals

A runnable task waiting for CPU remains in R. After the scheduler runs it and it blocks for an event, it normally becomes S; an uninterruptible operation produces D. Memory pressure can appear as CPU pressure, reclaim activity, or I/O pressure rather than one universal state.

  • SIGSTOP forces a process into T; SIGTSTP normally stops it when job control permits; SIGCONT makes it eligible to run again.

  • SIGKILL cannot be caught or ignored. A process in D may retain the pending fatal signal until its uninterruptible operation ends.

  • SIGCHLD notifies a parent about a child’s state or exit, but notification does not replace reaping the child with wait() or waitpid().

Production Interpretation

  • High R count: May indicate CPU saturation, but one sample is inconclusive. Check vmstat, top, pidstat, cgroup limits, and /proc/pressure/cpu.

  • High S count: Usually normal. Determine whether tasks are waiting on sockets, timers, locks, or child processes.

  • Persistent D tasks: Investigate storage latency, filesystem errors, mount problems, and NFS servers. Brief D states during heavy I/O are normal.

  • Increasing Z count: Indicates that a parent is not reaping children. An exited zombie cannot itself be killed.

  • Unexpected T tasks: Check for administrative suspension, terminal-generated SIGTSTP, or debugging activity.

  • Stuck S tasks: Correlate wait channels, file descriptors, sockets, and thread stacks rather than assuming they are consuming CPU.

Troubleshooting and Recovery

For persistent D tasks, inspect ps -eo pid,ppid,stat,wchan:32,comm, kernel messages with sudo dmesg -T, and storage metrics with iostat -xz 1 5. Here, -x enables extended device statistics, -z suppresses devices with no activity, 1 is the sampling interval, and 5 is the number of reports. Use vmstat 1 5 for runnable and blocked counts, context switches, CPU state, and I/O wait; it does not identify the responsible PIDs. df -hT shows filesystem types and capacity, while lsblk -o NAME,TYPE,FSTYPE,MOUNTPOINTS,SIZE,STATE shows block-device relationships and health.

Do not use kill -9 as the primary response to D; the task may remain uninterruptible until the kernel operation returns. Resume an intentionally stopped process with kill -CONT PID. For zombies, identify the parent with ps -o pid,ppid,stat,comm -p PID, then correct its child-reaping behavior or restart it under a suitable supervisor. Container PID 1 implementations must reap or delegate reaping of orphaned children to prevent zombie accumulation.

22

What is the difference between SIGTERM and SIGKILL when using kill?

SIGTERM requests that a process terminate and can be handled for graceful cleanup; SIGKILL forcibly terminates a process and cannot be caught, blocked, or ignored. Linux kill sends SIGTERM by default when no signal is specified.

Read full answer

Applies toLinux (all supported kernel and userspace versions; examples include Ubuntu 22.04/24.04 and RHEL 9)

How kill and Signals Work

  • Shell builtin and utility: Most shells provide kill as a builtin. Common Linux systems also install an external utility, often /usr/bin/kill from procps-ng, but its location is distribution-dependent. killall and pkill are separate utilities that select processes by name or attributes.

  • Syntax: kill [-s signal] pid... or kill -signal pid.... If neither form specifies a signal, the default is SIGTERM, whose signal number is 15 on Linux.

  • Signal listing: kill -l lists the supported signal names and numbers.

  • Delivery: The kernel delivers the signal, and a handler runs when the target process can safely execute it. SIGTERM may be caught, blocked, or ignored, so receiving it does not guarantee immediate exit.

  • Permissions: An ordinary user can generally signal processes they own. Root and appropriately privileged processes with CAP_KILL can signal processes owned by other users.

SIGTERM vs SIGKILL

Aspect SIGTERM (15) SIGKILL (9)
Default kernel action Terminate the process Forced termination
Catchable or blockable Yes No
Ignorable Yes No
User-space cleanup Possible when a handler is installed Impossible because the handler cannot run
Resource release Kernel resources are released when the process exits Kernel resources are also released, but application buffers and in-flight work may not be settled
Timing Depends on the handler, signal disposition, and process state Usually prompt, but termination can be delayed by uninterruptible I/O
Typical use Orderly shutdown and graceful draining Last-resort termination of an unresponsive process

SIGTERM is graceful only when the program implements suitable shutdown behavior. For example, a server may stop accepting new work, finish selected in-flight requests, flush buffers, and close connections. Without a handler, the default action simply terminates the process.

Choosing and Escalating

  • Start with SIGTERM: kill PID sends SIGTERM, while kill -TERM PID states the signal explicitly. Allow enough time for the workload to complete its shutdown sequence.

  • Escalate selectively: If the process remains alive after the grace period and its crash-consistency guarantees are acceptable, use kill -KILL PID.

  • Verify process identity: Before escalating, confirm that the original PID still belongs to the intended process. PID reuse can otherwise cause an unrelated process to be killed.

  • Interpret kill -0 carefully: Signal 0 performs existence and permission checking without delivering a signal. It can succeed for an unreaped zombie and fail because of permissions, so it is not a complete liveness test.

  • Handle D state separately: A task in uninterruptible sleep cannot execute a user-space handler. SIGKILL may remain pending until the storage operation completes; investigate the underlying device, filesystem, or kernel issue.

  • Use systemd controls: For a systemd service, KillSignal=SIGTERM selects the stop signal and TimeoutStopSec=30s bounds the grace period. If the service has not exited by then, systemd normally escalates to SIGKILL.

Production and Operational Effects

  • Web servers: A SIGTERM handler can drain connections and complete requests, but this behavior is application-specific. SIGKILL drops active connections and skips deferred work.

  • Databases: SIGTERM may let the database perform an orderly shutdown, such as closing connections and flushing state according to that database’s implementation. SIGKILL skips that handling and can lose in-flight transactions or application-side side effects; however, a database with proper write-ahead logging or crash recovery should not automatically be considered corrupted.

  • Backups and batch jobs: Forced termination may leave incomplete output or partially updated application state. Backup completion and integrity should be verified independently.

  • Monitoring: Track shutdown duration, exit status, service restarts, and processes remaining in D state. For systemd services, inspect the service status and journal before escalating manually.

  • Upgrades and restarts: Prefer graceful termination for rolling restarts so clients and dependent services can observe an orderly transition.

Security and Best Practices

  • Avoid broad matching: killall and pkill can terminate several processes unintentionally. Prefer a verified PID or a narrowly constrained selection.

  • Use least privilege: Do not run workloads as root merely to simplify process management. Where appropriate, grant only the capability required to control a specific service.

  • Reserve SIGKILL: Use it when SIGTERM is ignored or blocked and the service can tolerate forced termination. Do not use it as a routine first choice.

  • Verify completion: After signalling, confirm that the process exited and inspect its logs and resulting state rather than assuming the signal was handled successfully.

23

What is a daemon in Linux?

A daemon is a background process that provides a continuing service and normally runs without a controlling terminal. Examples include sshd, cron, systemd-journald, web servers, and database servers; on modern Linux, systemd usually supervises them as services.

Read full answer

Applies toLinux with systemd (Ubuntu 22.04/24.04; RHEL 9)

How Daemons Work

Daemons commonly perform the following operations:

  • Run independently of a user’s terminal session.

  • Start during system boot, on demand, or when another process connects.

  • Handle requests through files, sockets, IPC, or network connections.

  • Run for long periods, often for the lifetime of the server.

  • Receive operating-system signals to reload configuration or shut down.

Historically, a daemon would fork, call setsid(), and often fork again. The second fork prevented it from acquiring a controlling terminal. Its standard input, output, and error streams were redirected to /dev/null or log files.

Modern daemons usually do not implement this sequence themselves. Instead, systemd starts them and manages their process, environment, logging, dependencies, and restart behavior.

Daemons Under systemd

Systemd distinguishes services from other units. A service can run continuously, exit after completing work, or report that it is ready:

Type= Behavior
simple The main process starts directly; systemd considers it active immediately.
exec Similar to simple, but successful execution is treated as the start operation.
forking The initial process exits after a child daemonizes itself.
oneshot The process runs and exits; commonly paired with RemainAfterExit=yes.
notify The daemon sends sd_notify() to systemd after initialization is complete.

Systemd is PID 1, so it starts units during boot, reaps orphaned child processes, and supervises configured daemons. For example, a service can set User=appuser, use ExecStart=/usr/local/bin/example-app, and apply Restart=on-failure with RestartSec=5s. The WantedBy=multi-user.target setting links it into the normal multi-user boot target.

Restart=on-failure does not hide a permanently broken application; it can create a crash loop. Monitor restart counts and logs rather than relying only on the service appearing active.

Filesystem Locations and Security

Daemons interact with the standard Linux filesystem hierarchy:

  • /etc/systemd/system/: Administrator-created unit files.

  • /usr/lib/systemd/system/: Unit files supplied by installed packages; /lib/systemd/system may point here on merged-/usr systems.

  • /etc/: Application configuration, such as /etc/ssh/sshd_config.

  • /run/: Volatile runtime data, including PID files and Unix sockets.

  • /var/log/: Traditional application logs.

  • /var/log/journal/: Persistent journald storage when configured.

  • /var/lib/: Persistent application state that may need backups.

Daemon accounts should have only the privileges they require. Prefer User=, Group=, Linux capabilities, and systemd sandboxing options over running everything as root. OpenSSH uses privilege separation: a privileged monitor controls privileged operations while pre-authentication child processes run under an unprivileged account. The monitor may still require sufficient privilege to bind a privileged port such as 22.

Package and service names can differ:

  • Ubuntu: sudo apt install openssh-server; the service is usually ssh.

  • RHEL/Fedora: sudo dnf install openssh-server; the service is usually sshd.

Do not disable SELinux or use permissions such as chmod 777 to solve access problems. Determine the missing SELinux policy, group membership, ownership, or capability instead.

Production Operations and Failure Modes

Production daemons require active monitoring and deliberate maintenance:

  • Monitoring: Track ActiveState, SubState, restart counts, error rates, latency, resource usage, and certificate expiration.

  • Logging: Use journalctl and forward important logs to centralized storage. A local journal is not a sufficient backup.

  • Backups: Protect configuration, TLS certificates, keys, and application state. Test restoration procedures.

  • Upgrades: Review package changes, validate configuration, reload units if required, and restart during a controlled maintenance window.

  • Graceful shutdown: Daemons should handle SIGTERM, finish active work, and exit within the systemd timeout. Otherwise, systemd sends SIGKILL.

  • Common incidents: Configuration errors, expired certificates, full disks, unavailable dependencies, port conflicts, resource exhaustion, and repeated crashes.

Best Practices and Common Pitfalls

  • Use systemd units rather than custom startup scripts so dependencies, logging, and restart policies remain consistent.

  • Run daemons as a dedicated unprivileged account whenever possible.

  • Apply configuration updates with systemctl reload when the daemon supports graceful reloads; use restart only when reload is unsupported or necessary.

  • Keep secrets out of unit files and command-line arguments, which may be visible to other users.

  • Treat systemctl is-active as a point-in-time check, not a substitute for monitoring.

  • Avoid blind restart loops. Determine the root cause using journalctl, resource metrics, configuration validation, and dependency status.

24

What are systemd and unit files in Linux?

systemd is Linux’s standard initialization and service manager. A unit file is the declarative configuration systemd uses to manage and control a system resource such as a service, socket, timer, mount, or target.

Read full answer

Applies tosystemd 249 (Ubuntu 22.04, RHEL 9) and systemd 255 (Ubuntu 24.04)

Core Concepts

On a Linux host, systemd normally runs as PID 1, starts the system, tracks processes through cgroups, and supervises services. It replaces SysVinit scripts with structured configuration and supports:

  • Automatic recovery: restarting failed processes according to configured policies.

  • Dependency management: starting related units in the required order.

  • Socket activation: starting a service when a connection arrives.

  • Timers: scheduling jobs using calendar or monotonic intervals.

  • Journaling: collecting service output and metadata in journald.

  • Resource controls: applying CPU, memory, task, and I/O limits through cgroups.

  • Parallel startup: starting independent units concurrently.

The systemd project also provides components such as systemd-networkd, systemd-resolved, systemd-timesyncd, and per-user service managers. Their availability and whether they are enabled by default depend on the distribution.

Unit Types

A unit’s filename determines its type:

Type Example Purpose
.service nginx.service Manages a server process or a one-shot task
.socket docker.socket Listens for connections and activates a service
.timer backup.timer Schedules a service to run
.target multi-user.target Groups units for startup and dependencies
.mount data.mount Mounts a filesystem
.device dev-sda.device Represents a device
.path upload.path Activates a unit when a path changes
.slice database.slice Groups units for resource management
.scope worker.scope Groups externally created processes into a cgroup

A .service unit commonly represents one long-running process, but Type=oneshot services can perform a task and exit.

Unit File Structure and Lifecycle

Unit files use an INI-style syntax with sections such as:

  • [Unit]: Description, dependencies, ordering, and start limits.

  • [Service]: Process configuration, identity, restart policy, limits, and security settings.

  • [Install]: Controls how the unit is enabled at boot.

Important options include:

  • After=: Defines ordering but does not pull another unit into the transaction.

  • Requires=: Adds a dependency; explicitly stopping the required unit can stop the dependent unit.

  • Wants=: Adds a weaker dependency whose activation failure is not fatal.

  • ExecStart=: Specifies the command systemd executes. An absolute executable path is recommended because the command is not normally interpreted by a shell.

  • Type=exec: Starts an executable directly and considers startup successful after the exec operation; it is suitable for foreground services.

  • Restart=on-failure: Restarts after a nonzero exit, termination signal, timeout, or watchdog failure, except when systemd initiated the stop.

  • TimeoutStopSec=: Controls how long systemd waits for graceful termination before sending SIGKILL.

During a normal stop, systemd sends SIGTERM first. Applications should handle it to stop accepting requests, close connections, and flush state.

File Locations and Operations

Vendor packages normally install units under /usr/lib/systemd/system. Administrator-owned configuration belongs under /etc/systemd/system, while /run/systemd/system contains runtime state. When unit names conflict, /etc/systemd/system takes precedence.

Common operations include sudo systemctl daemon-reload, sudo systemctl enable --now acme-api.service, sudo systemctl status acme-api.service, sudo systemctl restart acme-api.service, systemctl is-active acme-api.service, systemctl cat acme-api.service, and systemctl list-dependencies acme-api.service.

After changing a unit file, run daemon-reload; this reloads configuration but does not restart the running service. Use restart when the new configuration must take effect. Validate units with sudo systemd-analyze verify /etc/systemd/system/acme-api.service.

For local adjustments, prefer a drop-in such as /etc/systemd/system/acme-api.service.d/override.conf over editing a package-managed unit. This prevents package upgrades from overwriting local changes. Ubuntu uses apt, while RHEL uses dnf, but service activation should use systemctl rather than manually creating boot symlinks.

Operations, Security, and Failure Modes

  • Inspect logs: use journalctl -u acme-api.service -n 100 --no-pager and add -f to follow logs during an incident.

  • Monitor failures: track systemctl list-units --failed and alert on repeated service restarts or units entering the failed state.

  • Diagnose delays: use systemd-analyze blame and systemd-analyze critical-chain.

  • Avoid restart storms: systemd applies start limits; investigate the underlying failure instead of disabling supervision.

  • Use least privilege: run services under dedicated User= and Group= identities rather than root.

  • Apply namespace protections: ProtectSystem=strict, ProtectHome=true, and PrivateTmp=true restrict accidental access. Add only necessary writable paths with ReadWritePaths=.

  • Do not disable SELinux or AppArmor as a troubleshooting shortcut: inspect denied operations in the audit or security logs and correct the policy or file labels.

  • Plan log persistence: do not assume journald storage is persistent. With Storage=auto, logs become persistent when /var/log/journal/ exists; Storage=persistent explicitly requests persistent storage.

  • Back up configuration: preserve /etc/systemd/system and custom drop-ins in configuration management, and verify unit changes before deployment.

25

How should you create a systemd service for an application?

A systemd service is a declarative unit that tells systemd how to start, stop, supervise, and recover an application process. The unit belongs in /etc/systemd/system/ for machine-specific services and is normally managed with systemctl.

Read full answer

Applies toLinux with systemd (Ubuntu 22.04/24.04, RHEL 9; systemd 249-255)

How systemd Starts the Service

Systemd is PID 1 and reads unit files during boot. When you run systemctl enable --now myapp.service, it creates a boot-time dependency and starts the service immediately.

For a typical long-running application:

  • The application runs in the foreground as a non-root user.

  • ExecStart uses an absolute executable path.

  • Restart=on-failure recovers from crashes.

  • Standard output and standard error are written to journald.

  • systemd sends SIGTERM during a normal stop, then sends SIGKILL if TimeoutStopSec expires.

Unit File Semantics

Important directives include:

  • [Unit]: Defines the service description, dependencies, and ordering. After= controls startup order; it does not itself start another unit.

  • [Service]: Defines the process, user, environment, restart policy, and security controls.

  • [Install]: Specifies where systemd creates the enablement symlink, usually under multi-user.target.

  • Type=simple: Appropriate when the process remains in the foreground. Other types such as Type=notify require application support for systemd readiness notifications.

  • User= and Group=: Run the application without root privileges.

  • WorkingDirectory=: Sets the process working directory independently of where systemd was invoked.

  • Restart=on-failure: Restarts after crashes, signals, timeouts, or nonzero exits, but not after a clean exit.

  • EnvironmentFile=: Loads environment variables from a separate file. Secrets should not be embedded directly in the unit.

Production Behavior and Failure Modes

systemd manages the process lifetime, but it does not automatically provide application-level readiness. Type=simple means the service is considered active shortly after ExecStart; it does not prove that an HTTP endpoint is ready.

Common operational failures include:

  • Invalid executable or permissions: ExecStart must be absolute. Check systemctl status and journalctl.

  • Working-directory failure: A missing WorkingDirectory can prevent startup.

  • Port conflicts: The application exits if another process already owns the listening port.

  • Restart loops: Repeated crashes should be diagnosed from journald rather than hidden by an aggressive restart policy.

  • Unclean shutdown: Applications should handle SIGTERM to stop accepting requests and flush work. Otherwise, systemd eventually uses SIGKILL.

  • Resource exhaustion: Use MemoryMax, CPUQuota, file-descriptor limits, and monitoring appropriate to the workload.

Code Example

This example deploys a small Python HTTP service on Ubuntu. It listens only on loopback, so a reverse proxy or load balancer should provide public access.

# app.py
from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer


class Handler(BaseHTTPRequestHandler):
    def do_GET(self):
        if self.path != "/health":
            self.send_error(404)
            return

        body = b'{"status":"ok"}\n'
        self.send_response(200)
        self.send_header("Content-Type", "application/json")
        self.send_header("Content-Length", str(len(body)))
        self.end_headers()
        self.wfile.write(body)


if __name__ == "__main__":
    server = ThreadingHTTPServer(("127.0.0.1", 8080), Handler)
    server.serve_forever()

Create the service account and deploy the file:

# Ubuntu 22.04/24.04
sudo adduser --system --group \
  --home /opt/myapp --no-create-home \
  --shell /usr/sbin/nologin myapp

sudo install -d -o root -g myapp -m 0755 /opt/myapp
sudo install -o root -g myapp -m 0644 app.py /opt/myapp/app.py

On RHEL 9 or Fedora, install Python and create the account with:

sudo dnf install python3
sudo useradd --system --user-group \
  --home-dir /opt/myapp --create-home \
  --shell /sbin/nologin myapp

Create /etc/systemd/system/myapp.service:

[Unit]
Description=My application HTTP service
After=network.target

[Service]
Type=simple
User=myapp
Group=myapp
WorkingDirectory=/opt/myapp
ExecStart=/usr/bin/python3 /opt/myapp/app.py
Environment=PYTHONUNBUFFERED=1
Restart=on-failure
RestartSec=5s
TimeoutStopSec=30s
KillSignal=SIGTERM

NoNewPrivileges=true
PrivateTmp=true
ProtectSystem=strict
ProtectHome=true
ProtectKernelTunables=true
ProtectKernelModules=true
ProtectControlGroups=true
RestrictSUIDSGID=true
LockPersonality=true

[Install]
WantedBy=multi-user.target

Load and start it:

sudo systemctl daemon-reload
sudo systemctl enable --now myapp.service
sudo systemctl status --no-pager myapp.service

Typical output:

Created symlink /etc/systemd/system/multi-user.target.wants/myapp.service →
/etc/systemd/system/myapp.service.

● myapp.service - My application HTTP service
     Loaded: loaded (/etc/systemd/system/myapp.service; enabled)
     Active: active (running) since Mon 10:15:00 UTC 2025; 2s ago
   Main PID: 18342 (python3)

Verify the endpoint and inspect logs:

curl http://127.0.0.1:8080/health
sudo journalctl -u myapp.service -n 20 --no-pager
{"status":"ok"}
myapp[18342]: 127.0.0.1 - - [10/Mar/2025:10:15:03] "GET /health HTTP/1.1" 200 -

Best Practices

  • Run under a dedicated unprivileged account; never use chmod 777 or run an internet-facing service as root.

  • Apply the narrowest practical systemd sandbox controls. On RHEL, resolve SELinux denials using an appropriate file context or port label rather than disabling SELinux.

  • Store secrets outside the unit, restrict access, and prefer a secrets manager where available.

  • Monitor systemctl is-active, journald error rates, restart counts, memory use, and an application-level health endpoint.

  • Implement graceful SIGTERM handling and persist state outside the release directory, such as /var/lib/myapp.

  • Deploy versioned artifacts and switch them atomically, keeping the previous version available for rollback.

  • After every unit change, run sudo systemctl daemon-reload; after changing only application code, use sudo systemctl restart myapp.service.

  • Back up persistent state and test restoration regularly. Service restarts do not protect against data loss.

26

How do you diagnose a failed systemd service start?

A failed systemd service is diagnosed by checking the unit state, reading its journald logs, and verifying its configuration, executable, permissions, and dependencies. The most useful evidence is usually the exit status shown by systemctl status and the detailed error messages from journalctl.

Read full answer

Applies toLinux with systemd on Ubuntu 22.04/24.04 and RHEL 9

Triage Workflow

  1. Identify the failed unit

bash systemctl list-units --failed --no-pager systemctl status myapp.service --no-pager -l

Look for Result, the failing process, and its exit status.

  1. Read the current boot’s logs

bash sudo journalctl -u myapp.service -b --no-pager -n 100

Use -b for the current boot, -b -1 for the previous boot when retained, and --since when the failure occurred at a known time.

  1. Inspect relevant systemd properties

bash systemctl show myapp.service \ -p LoadState \ -p ActiveState \ -p SubState \ -p Result \ -p ExecMainCode \ -p ExecMainStatus

LoadState=not-found, for example, indicates a missing unit file, while Result=exit-code means the service or one of its commands exited unsuccessfully.

  1. Check the unit and dependencies

bash systemctl cat myapp.service systemctl list-dependencies myapp.service systemctl status postgresql.service --no-pager -l

A required dependency may have failed before the target service was started.

Interpreting Failure Evidence

Evidence Likely meaning Checks
status=203/EXEC systemd could not execute the configured program Verify the path, executable bit, mount options, and shebang
status=200/CHDIR The configured working directory was unavailable Check WorkingDirectory= and directory permissions
status=226/NAMESPACE A namespace or sandbox restriction failed Review PrivateTmp=, ProtectSystem=, and related directives
Result=timeout Startup did not complete before TimeoutStartSec= Check dependencies, blocked network calls, and slow initialization
status=1 or another application status The application or an ExecStartPre= command returned an error Inspect application logs and the failing command
Dependency failure A required unit did not start successfully Check the dependency’s status and logs

For Type=notify services, systemd expects the process to send READY=1; otherwise startup may time out. For Type=forking, an incorrect PIDFile= or process behavior can also cause startup failures.

Verify Files, Permissions, and Dependencies

Unit files normally live under /etc/systemd/system, /usr/lib/systemd/system, or /lib/systemd/system. Administrative overrides belong in /etc/systemd/system/myapp.service.d/.

sudo systemd-analyze verify /etc/systemd/system/myapp.service
namei -l /opt/myapp/bin/myapp
sudo -u appuser test -x /opt/myapp/bin/myapp && echo "Executable is accessible"

systemd-analyze verify catches many syntax and unit-level problems, although some warnings may be environment-specific. If a path is missing, Ubuntu’s dpkg -S and RHEL’s rpm -qf can identify which package owns it:

dpkg -S /usr/bin/example-command   # Ubuntu
rpm -qf /usr/bin/example-command   # RHEL

If the unit file was edited, reload systemd before retrying:

sudo systemctl daemon-reload
sudo systemctl restart myapp.service

daemon-reload reloads unit files and rebuilds the dependency tree; it does not restart the application.

Code Example

A compact first-pass diagnostic is:

sudo systemctl status myapp.service --no-pager -l
sudo journalctl -u myapp.service -b --no-pager -n 100

A typical failed unit might produce output similar to:

× myapp.service - MyApp API
     Loaded: loaded (/etc/systemd/system/myapp.service; enabled)
     Active: failed (Result: exit-code) since Wed 2025-09-10 14:02:11 UTC
    Process: 4821 ExecStartPre=/usr/local/bin/check-config (code=exited, status=203/EXEC)

The corresponding journal entry identifies the root cause:

Sep 10 14:02:11 host systemd[1]: Starting MyApp API...
Sep 10 14:02:11 host systemd[1]: myapp.service: Failed to locate executable /usr/local/bin/check-config: No such file or directory
Sep 10 14:02:11 host systemd[1]: myapp.service: Failed at step EXEC spawning /usr/local/bin/check-config: No such file or directory
Sep 10 14:02:11 host systemd[1]: myapp.service: Failed with result 'exit-code'.
Sep 10 14:02:11 host systemd[1]: Failed to start myapp.service.

systemctl reset-failed myapp.service resets the failed state and the unit’s start/restart rate-limit counters; it does not repair the cause. Use it after fixing the cause if the failed state or a rate limit prevents another attempt.

Production Practices

  • Preserve logs: Configure persistent journald storage under /var/log/journal if logs must survive reboots. Also retain application logs in the normal centralized logging system.

  • Avoid secrets in unit arguments: Command-line arguments can appear in ps and systemctl show. Use systemd credentials such as LoadCredential= or LoadCredentialEncrypted=, or a secrets manager instead. EnvironmentFile= is better reserved for non-secret configuration.

  • Control failure behavior: Use Restart=on-failure with appropriate RestartSec= and StartLimitIntervalSec=/StartLimitBurst= values. Avoid masking or disabling services merely to suppress alerts.

  • Check infrastructure failures: For timeouts or sudden exits, inspect disk space, kernel messages, memory pressure, and dependency connectivity:

bash df -h free -h sudo journalctl -k -b --no-pager ss -ltnp

  • Monitor service state: Alert on repeated ActiveState=failed, restart loops, and startup duration. During incidents, preserve the journal output before restarting the unit because logs and exit metadata can change.

  • Upgrade carefully: Validate unit changes in staging with systemd-analyze verify, canary deployments where possible, and retain the previous unit file for rollback.

27

How do systemd timers work and compare with cron?

systemd timers are separate units that activate services on calendar or monotonic schedules. Compared with cron, they integrate with unit dependencies, journald, service hardening, and optional catch-up behavior, although cron remains more portable across Unix-like systems.

Read full answer

Applies tosystemd 249-255 on Ubuntu 22.04/24.04; systemd 249 on RHEL 9

How Systemd Timers Work

A timer is normally paired with a service unit:

  • Calendar scheduling: OnCalendar= activates a unit at specified dates, times, weekdays, or recurring intervals.

  • Relative scheduling: OnBootSec= is relative to boot, OnStartupSec= to manager startup, OnActiveSec= to timer activation, OnUnitActiveSec= to the activated unit’s previous activation, and OnUnitInactiveSec= to its previous deactivation. Relative timers use monotonic time.

  • Service activation: Unit= identifies the service to activate. If omitted, systemd activates the service with the timer’s base name.

  • Persistent execution: Persistent=true applies only to OnCalendar= timers. It stores trigger state and performs one catch-up invocation after downtime if an event elapsed while the timer was inactive.

  • Randomized delay: RandomizedDelaySec= spreads timer activation across a configured interval.

  • Accuracy window: AccuracySec= controls how long systemd may coalesce or delay an activation.

Timer units are enabled by linking them into timers.target:

[Install]
WantedBy=timers.target

Systemd Timers vs Cron

Area systemd timers cron
Scheduling Calendar events and monotonic intervals Primarily five-field, minute-based schedules
Granularity Supports seconds Standard cron resolution is one minute
Process control Integrates with unit dependencies and lifecycle Starts separate processes with less dependency management
Overlap control Does not concurrently activate the same unit Each occurrence commonly starts independently
Missed executions Persistent=true can run one catch-up Usually no replay; anacron covers selected periodic jobs
Environment Associated services define users, paths, environment, and security controls Usually has a minimal environment and invokes a shell
Logging Integrated with journald Commonly uses syslog or email
Time zones Supports time-zone names in OnCalendar= CRON_TZ support varies by implementation
Portability Standard on modern systemd-based Linux Available across many Unix-like systems
Boot schedules OnBootSec= covers the @reboot use case Supports @reboot, @daily, and similar aliases

An explicit time zone makes the intended local schedule clear and avoids server-time-zone ambiguity:

OnCalendar=*-*-* 02:00:00 Europe/London

Operational Failure Modes

  • The service fails: The timer remains active, but it does not automatically retry the failed invocation. The next scheduled activation can start the failed service again.

  • The service is still running: systemd does not launch a second instance of the same unit. Make the work idempotent because it may restart after a crash or reboot.

  • A persistent timer catches up: systemd performs one catch-up invocation, not one for every missed occurrence. RandomizedDelaySec= and AccuracySec= can delay it, and fleet-wide catch-up can still create a thundering herd.

  • The clock changes: Calendar timers follow wall-clock time; monotonic relative timers are preferable for intervals that must not be affected by clock adjustments.

  • Configuration is invalid: The timer may fail to load or never trigger. Validate calendar expressions with systemd-analyze calendar.

  • Permissions are insufficient: A hardened service may run as an unprivileged user while its script expects root-owned paths or write access.

Code Example

This service deletes regular application temporary files older than exactly seven days. Before enabling it, ensure the app user and group exist and /srv/app/tmp is owned appropriately and writable by that group.

/etc/systemd/system/cleanup.service:

[Unit]
Description=Remove expired application temporary files

[Service]
Type=oneshot
# Run under a dedicated least-privilege identity.
User=app
Group=app
# Delete regular files older than exactly 7 days.
ExecStart=/usr/bin/find /srv/app/tmp -mindepth 1 -type f -mmin +10080 -delete
# Bound execution time and isolate the service.
TimeoutStartSec=15min
NoNewPrivileges=true
PrivateTmp=true
ProtectSystem=strict
ProtectHome=true
ReadWritePaths=/srv/app/tmp

/etc/systemd/system/cleanup.timer:

[Unit]
Description=Run temporary-file cleanup every day

[Timer]
# Run daily at 02:00 London time.
OnCalendar=*-*-* 02:00:00 Europe/London
# Catch up once after downtime and spread fleet-wide activation.
Persistent=true
RandomizedDelaySec=15m
AccuracySec=1m
Unit=cleanup.service

[Install]
WantedBy=timers.target

Validate and activate the timer:

# Validate the exact calendar expression and unit files.
systemd-analyze calendar '*-*-* 02:00:00 Europe/London'
sudo systemd-analyze verify /etc/systemd/system/cleanup.service /etc/systemd/system/cleanup.timer

# Load, enable, and start the timer.
sudo systemctl daemon-reload
sudo systemctl enable --now cleanup.timer
systemctl list-timers cleanup.timer

Typical abbreviated output on a UTC host, with randomized delay affecting the exact minutes:

NEXT                        LEFT     LAST                        PASSED       UNIT          ACTIVATES
Wed 2025-06-18 01:07:00 UTC 13h      Tue 2025-06-17 01:06:00 UTC 11h ago      cleanup.timer cleanup.service

Production Best Practices

  • Prefer systemd timers on modern Ubuntu, RHEL, and Fedora hosts; retain cron for compatibility or simple existing workloads.

  • Give services explicit User=, Group=, and TimeoutStartSec= settings with least-privilege filesystem access.

  • Inspect execution with journalctl -u cleanup.service --since today and monitor failed units with systemctl --failed.

  • Add an OnFailure= notification target for critical jobs, and monitor expected schedules externally because systemctl --failed does not detect a timer that never ran.

  • Test the service with sudo systemctl start cleanup.service before enabling the timer.

  • Version unit files in configuration management and run systemctl daemon-reload after deployment.

  • On Debian/Ubuntu, apt manages the cron package; RHEL and Fedora generally use dnf and package cronie.

28

How do I schedule jobs with cron and read crontab syntax?

Cron schedules commands by evaluating stored crontab expressions once per minute. Use a user crontab for personal jobs and /etc/crontab or /etc/cron.d for centrally managed schedules.

Read full answer

Applies toUbuntu 24.04 `cron` and RHEL 9 `cronie`

How Cron Scheduling Works

  • Ubuntu runs cron.service; RHEL runs crond.service.

  • A user crontab executes commands as its owner. Entries in /etc/crontab and /etc/cron.d specify the execution user explicitly.

  • Cron evaluates the configured local date and time once per minute.

  • It provides minute-level scheduling only. Missed runs are not replayed after downtime or clock adjustments.

  • @reboot is an exception: it runs when the cron daemon starts, which may be later than the machine boot.

Platform User crontab directory Package
Ubuntu 24.04 /var/spool/cron/crontabs/<user> cron
RHEL 9 /var/spool/cron/<user> cronie

Crontab Syntax

A user crontab has five whitespace-separated fields followed by the command:

minute hour day-of-month month day-of-week command
Field Allowed values Examples
Minute 0-59 0, 15, */10
Hour 0-23 0, 2, */2
Day of month 1-31 1, 15, 1-7
Month 1-12 or JAN-DEC 1, JAN, JUL
Day of week 0-7 or SUN-SAT 0, 7, MON-FRI

Fields support *, comma-separated lists, ranges, and steps such as */15. Both 0 and 7 represent Sunday. If day-of-month and day-of-week are both restricted, the command runs when either matches; if one field is *, the other determines the schedule.

*/15 * * * *       Every 15 minutes
0 * * * *          Every hour
30 2 * * *         Every day at 02:30
0 3 * * 1-5        Weekdays at 03:00
0 0 1 * *          First day of every month

Special schedules include @hourly, @daily, @midnight, @weekly, @monthly, @yearly/@annually, and @reboot.

Managing Crontabs

$ crontab -e
$ crontab -l
# m h dom mon dow command
*/15 * * * * /usr/local/bin/check-health

Use crontab -e to edit the current user’s schedule and crontab -l to inspect it. crontab -r removes the entire crontab without confirmation, so back it up first with crontab -l > "$HOME/crontab.backup".

Install or update the daemon with sudo apt install cron on Ubuntu or sudo dnf install cronie on RHEL.

$ systemctl is-active cron
active
$ journalctl -u cron -n 1 --no-pager
Jul 14 10:00:01 app01 CRON[1234]: (deploy) CMD (/usr/local/sbin/disk-usage-check)

On RHEL, use systemctl status crond and journalctl -u crond.

Code Example

This complete Ubuntu example installs a root-owned script and schedules it every ten minutes. The log is placed under the user’s home because a normal user cannot create files directly in /var/log.

cat > disk-usage-check.sh <<'EOF'
#!/bin/bash
set -Eeuo pipefail

/bin/date --iso-8601=seconds
/usr/bin/df -P /
EOF

sudo install -d -m 0755 /usr/local/sbin
sudo install -m 0755 disk-usage-check.sh /usr/local/sbin/disk-usage-check
rm -f disk-usage-check.sh
mkdir -p "$HOME/cron"

Install and inspect the crontab:

cat > disk-usage-check.cron <<'EOF'
SHELL=/bin/bash
MAILTO=""
*/10 * * * * /usr/local/sbin/disk-usage-check >> "$HOME/cron/disk-usage-check.log" 2>&1
EOF

crontab disk-usage-check.cron
rm -f disk-usage-check.cron
crontab -l

Expected listing:

SHELL=/bin/bash
MAILTO=""
*/10 * * * * /usr/local/sbin/disk-usage-check >> "$HOME/cron/disk-usage-check.log" 2>&1

Cron passes the command to the configured shell, so shell operators such as >, >>, &&, and pipes work. Escape % as \% in a crontab command; otherwise cron treats the remaining text as command input.

Production Operations and Failure Modes

  • Overlapping executions: Cron can start another run while the previous one remains active. Use flock or application-level locking when concurrency is unsafe.

  • No automatic retry: A failed job is not retried until its next scheduled occurrence. Implement bounded retries when appropriate.

  • Limited visibility: Successful jobs may produce no output. Emit logs or a heartbeat metric so monitoring detects silent failures.

  • Log growth: Redirected output can exhaust the filesystem. Configure retention or logrotate and alert on disk utilization.

  • Deployments: Replacing a script while it is running can create version inconsistencies. Use versioned or atomic deployments.

  • Clock changes: NTP corrections and daylight-saving transitions can cause skipped or repeated wall-clock executions. Monitor synchronization with timedatectl.

Security and Best Practices

  • Run jobs as a dedicated, least-privileged service account rather than root.

  • Use absolute command paths and root-owned scripts in directories not writable by job users.

  • Cron uses a minimal PATH, commonly /usr/bin:/bin, rather than inheriting the interactive shell’s PATH.

  • Never use chmod 777 or weaken SELinux/AppArmor as a shortcut.

  • Store credentials in a secrets manager or protected runtime configuration, not in a crontab.

  • Install crontabs with crontab instead of editing spool files directly. Keep /etc/cron.d files root-owned and non-writable by other users; where audit rules capture crontab changes, inspect auditd records with ausearch.

  • Prefer a systemd timer when a job needs dependencies, sandboxing, failure handling, or catch-up execution. Put Persistent=true in the timer’s [Timer] section for an OnCalendar= schedule, and put OnFailure=handler.service in the triggered service’s [Unit] section.

Requirement Cron systemd timer
Simple periodic command Excellent Excellent
Dependency ordering Limited Unit dependencies
Failure handling No built-in handler Service [Unit] OnFailure=
Catch-up after downtime No replay One catch-up run with OnCalendar= and Persistent=true
Execution sandboxing External configuration Service hardening properties
29

How do nohup, &, screen, and tmux differ in Linux?

Background execution and terminal multiplexing solve different problems. & places a command in the background, nohup additionally protects it from SIGHUP, while screen and tmux provide persistent, attachable terminal sessions.

Read full answer

Applies toLinux with GNU coreutils 9.x, Bash 5.x, GNU Screen 4.x, and tmux 3.x

Core Differences

Tool Primary purpose Survives terminal hangup? Reattachable? Interactive terminal support
& Run a shell command asynchronously Not reliably by itself No Limited by terminal and job control
nohup Ignore SIGHUP during long-running execution Yes, against SIGHUP No Not a full terminal session manager
screen Create a persistent terminal session Yes Yes Yes
tmux Create a persistent, configurable terminal session Yes Yes Yes, with stronger session and pane management

screen and tmux are terminal multiplexers: each maintains a server process and one or more pseudo-terminals. You can disconnect and later reconnect without stopping commands running inside them.

Signal and Process Behavior

  • &: Starts a command asynchronously and immediately returns control to the shell. The command remains associated with the shell’s session and, depending on job-control settings, its process-group arrangement. Terminal closure or session teardown can therefore still terminate it.

  • nohup: Makes the command ignore SIGHUP, the hangup signal commonly sent when an SSH connection closes. It does not protect against SIGTERM, SIGKILL, host reboots, or a service manager stopping the process.

  • screen: Runs commands inside a persistent Screen session. Detaching leaves the session and its processes running on the host.

  • tmux: Provides the same persistence model with named sessions, windows, panes, scripting, and session-management features.

A detached terminal session is not a substitute for supervision. If the host reboots, the screen or tmux server is gone unless its lifecycle is managed separately.

Operational Use

For a simple, unsupervised SSH job, nohup with explicit redirection is often sufficient: run nohup ./worker.sh >worker.log 2>&1 </dev/null &. The shell prints the background process ID, and explicit redirection prevents the job from depending on an interactive terminal.

Use screen or tmux when an interactive shell, compiler, database client, or debugging session must remain available after reconnecting. Start a named Screen session with screen -S incident-console, detach with Ctrl-A followed by D, and reconnect with screen -r incident-console.

For tmux, start tmux new-session -s incident-console, detach with Ctrl-B followed by D, reconnect with tmux attach-session -t incident-console, and list sessions with tmux ls.

These tools are especially useful during incidents when restarting a production process would be inconvenient. For production services, prefer systemd: it provides restart policies, dependency ordering, resource controls, structured logging, user isolation, and deployment integration. nohup and ordinary background jobs can leave unmanaged processes consuming resources or writing after a deployment.

Common Pitfalls and Best Practices

  • Redirect every standard stream: Use >file 2>&1 </dev/null for unattended jobs so they never depend on an interactive terminal.

  • Monitor and rotate logs: nohup.out or custom log files can grow indefinitely. Use the platform’s logging system or logrotate.

  • Do not confuse detachment with supervision: tmux and screen preserve interactive sessions, but they do not automatically restart failed processes.

  • Restrict access: Terminal scrollback and logs may contain credentials or customer data. Protect SSH access and avoid shared, world-accessible session sockets.

  • Clean up sessions: Use tmux kill-session -t name and screen -S name -X quit when obsolete sessions are no longer needed.

  • Manage packages correctly: Use apt on Ubuntu and dnf on RHEL/Fedora to install or upgrade coreutils, bash, screen, and tmux; the package manager does not change their session behavior.

30

What are the sources and inheritance rules for Linux environment variables?

Environment variables are per-process key-value strings used to configure applications without changing their code. Linux normally gives a child a copy of its parent’s exported environment at process creation, so later changes in either process do not alter the other process’s environment.

Read full answer

Applies toLinux (all versions); examples use Ubuntu 22.04/24.04 and RHEL 9

Sources and Scopes

  • Shell variables: A normal assignment such as APP_ENV=production creates a variable in the current shell. It is not inherited by child processes unless it is exported.

  • Exported variables: export APP_ENV marks a shell variable for inclusion in the environment of subsequently executed commands. APP_ENV=production command supplies the value only to that command’s environment.

  • Parent-process environment: A child initially receives a copy of the parent’s current environment. The child can modify its own environment, but those modifications do not propagate back to the parent.

  • Shell startup files: Interactive non-login Bash reads ~/.bashrc. A Bash login shell reads /etc/profile and then the first existing user startup file among ~/.bash_profile, ~/.bash_login, and ~/.profile; Ubuntu’s ~/.profile also commonly sources ~/.bashrc for interactive shells.

  • PAM environment files: On systems with the pam_env module enabled, /etc/environment can populate PAM sessions with simple NAME=value assignments. It is not a shell script and must not contain commands such as export.

  • systemd: Units can define Environment= entries or read EnvironmentFile= files. System services do not automatically inherit an administrator’s interactive shell environment; systemd controls which manager environment is passed, including with PassEnvironment=.

  • Containers and orchestrators: Docker supplies variables with --env or --env-file; Kubernetes uses env or envFrom.

  • Scheduled jobs: cron starts jobs with a small, implementation-defined environment. Scripts should therefore define required values, including PATH, rather than assuming a login shell environment.

Inheritance and Replacement Rules

  • A shell passes its exported environment to fork()ed children and the execve() operation that replaces them with another program.

  • execve() receives an explicit environment array. A program can therefore retain, replace, add to, or remove entries before executing the new program.

  • Environment changes made after a child starts are not synchronized automatically. Each process has its own copy.

  • env -i command starts the command with an empty environment, while env NAME=value command adds or replaces one entry for that invocation.

  • sudo commonly resets most environment variables through its env_reset policy. su - starts a login shell and loads the applicable login startup files.

  • A .env file has no standard Linux meaning. It must be interpreted explicitly by an application, framework, or deployment tool.

Inspection and systemd Configuration

  • printenv NAME prints an exported variable and returns status 1 when it is unset. env displays the current environment, while export -p shows exported shell attributes.

  • A running process’s initial environment can usually be inspected through /proc/PID/environ, represented as NUL-separated entries. Access is subject to process ownership and kernel security controls.

  • With systemd, Environment="APP_ENV=production" defines a unit-level value, while EnvironmentFile=-/etc/app/app.env loads simple assignments from a file. The leading - makes a missing file non-fatal.

  • systemctl show app --property=Environment shows unit Environment= entries but does not necessarily show values loaded from EnvironmentFile=. The EnvironmentFiles property identifies configured files.

Production Practices and Failure Modes

  • Restart after changes: Updating startup files, unit files, or environment files does not update an already-running process. Restart the relevant service or job after validation.

  • Protect sensitive values: Environment values may be exposed through /proc, diagnostic tools, crash dumps, logs, or child processes. Prefer mounted secret files or a dedicated secret manager for production credentials.

  • Restrict file access: A systemd environment file containing secrets should be owned by the service account and have permissions such as chmod 600; do not rely solely on its directory name.

  • Avoid shell syntax in systemd files: Environment= does not perform shell command substitution or general shell expansion. Use explicit values or the supported EnvironmentFile= syntax.

  • Document precedence: Applications differ in whether command-line arguments, environment variables, or configuration files win. Define and test the application’s precedence explicitly.

  • Account for restricted environments: Services, containers, cron jobs, and sudo commands may have a different PATH, locale, home directory, or proxy configuration from an interactive login shell.

31

How do .bashrc, .bash_profile, and /etc/profile differ in Bash?

Bash startup files are chosen by shell interactivity, login status, and scope. ~/.bashrc is primarily read for interactive non-login shells, ~/.bash_profile for an interactive login shell belonging to one user, and /etc/profile for system-wide interactive login initialization.

Read full answer

Applies toBash 5.1–5.2 on Linux (Ubuntu 22.04/24.04, RHEL 9)

Startup Behavior

Bash distinguishes three common shell modes:

  • Interactive login shell: Runs in a user’s login context, such as an interactive SSH session, su -, a console login, or bash -l with a terminal.

  • Interactive non-login shell: A normal terminal session started without login-shell processing.

  • Non-interactive shell: Runs a script or command, such as bash script.sh or bash -c 'command'. It normally reads neither .bashrc nor the login profile files.

For an interactive login shell, the usual order is:

  1. Bash reads /etc/profile, if it exists.

  2. The distribution’s /etc/profile usually sources scripts under /etc/profile.d/.

  3. Bash looks for ~/.bash_profile, then ~/.bash_login, then ~/.profile.

  4. It reads and executes only the first file that exists.

For an interactive non-login shell, Bash reads ~/.bashrc, if it exists. Some distributions also provide a system-wide Bash configuration file: Ubuntu uses /etc/bash.bashrc, while RHEL 9 uses /etc/bashrc. Bash does not load these files directly; a user’s .bashrc or another startup file commonly sources them.

File Responsibilities

File Scope When it is loaded Typical purpose
~/.bashrc One user Interactive non-login Bash shells; it may also be sourced explicitly by a login startup file Aliases, functions, prompt settings, history options
~/.bash_profile One user Interactive login Bash shells, when it is the first available user profile Login-specific environment variables and startup commands
/etc/profile All users Interactive login Bash shells System-wide PATH, environment settings, and sourcing /etc/profile.d
/etc/bash.bashrc on Ubuntu; /etc/bashrc on RHEL System-wide When sourced by a user or distribution startup file Global aliases, functions, and interactive shell behavior

Although Bash does not read .bashrc directly as a login startup file, Ubuntu’s default ~/.profile sources ~/.bashrc when running under Bash. RHEL-family ~/.bash_profile files commonly do the same.

Important Distinctions

  • First user profile wins: If ~/.bash_profile exists, Bash does not automatically read ~/.bash_login or ~/.profile; it stops after ~/.bash_profile.

  • Fallback behavior: Ubuntu normally uses ~/.profile when neither ~/.bash_profile nor ~/.bash_login exists. RHEL commonly provides ~/.bash_profile instead.

  • Indirect .bashrc loading: A login shell can receive .bashrc settings because its selected profile explicitly sources .bashrc.

  • Environment differences: A login shell may have a different PATH from an interactive non-login shell because it processes /etc/profile and may inherit a different parent environment.

  • Aliases are not exported: An alias created in .bashrc applies to that shell but is not automatically inherited by child processes. Use export for environment variables.

  • Automation behaves differently: Non-interactive shells do not normally load these files. Pass required values explicitly, source a controlled script, or use the BASH_ENV mechanism where appropriate.

Security and Operational Practices

  • Keep ownership correct: User startup files should belong to that user and should not be writable by other users. System startup files should be root-owned and inaccessible for modification by unprivileged users.

  • Avoid untrusted sources: Bash executes startup files with the shell user’s privileges. Never source files that another user can modify.

  • Do not embed credentials: Prefer a secret manager or protected service configuration over plaintext passwords or tokens in startup files.

  • Test before deployment: Validate changed Bash files with bash -n, then test a fresh interactive login and a fresh interactive non-login shell.

  • Remember effective scope: Editing ~/.bashrc affects that user’s interactive non-login shells and any login shells that source it. Editing /etc/profile affects future interactive login shells for every user.

  • Avoid slow startup commands: Network calls, package-manager queries, or external service lookups can delay every new shell and create operational incidents.

32

How does the PATH variable work in Linux?

PATH is a colon-separated, ordered list of directories that a shell searches when resolving a command that does not contain a slash. The first executable found is normally selected, and child processes inherit a copy of the environment when they are launched.

Read full answer

Applies toLinux (Ubuntu 22.04/24.04, RHEL 9)

How Command Resolution Works

When a user enters a command such as systemctl, the shell normally:

  1. Processes any applicable alias.

  2. Checks shell functions and built-in commands.

  3. Consults its command-location cache, if supported.

  4. Otherwise, splits PATH on : and searches each directory from left to right.

  5. Selects the first regular file with the required execute permission.

For example, if PATH contains /usr/local/bin:/usr/bin, the shell checks /usr/local/bin/systemctl before /usr/bin/systemctl.

The execve() system call does not search PATH; the shell performs that search before calling it. A command containing a slash, such as /usr/bin/systemctl, bypasses the PATH search and is resolved directly.

Linux Directory Conventions

Typical directory Intended use
/usr/local/bin Locally installed applications and administrator-managed commands
/usr/bin Commands supplied by distribution packages
/usr/sbin System administration commands
/sbin, /bin Essential system commands; often merged into /usr on modern distributions
~/.local/bin Commands installed for a particular user

These are conventions rather than a mandatory order. Distribution packages, update-alternatives, and locally installed software may create symlinks in directories such as /usr/bin and /usr/local/bin.

PATH is normally exported into the environment. A child process receives an environment copy when it starts, so later changes in the parent shell do not update already-running children.

Diagnosing Resolution Problems

Useful shell built-ins include:

  • command -v name: Reports how the current shell would resolve a command. It may identify a function, alias, built-in, or executable rather than only a file in PATH.

  • type -a name: Shows all matching definitions, including executable files found in PATH.

  • hash -r: Clears Bash’s command-location cache. Use it when a previously resolved command was removed or moved and a new copy was installed at a different pathname; replacing a file at the same pathname does not by itself require clearing the cache.

Common failures include:

  • Exit status 127: The command was not found in PATH.

  • Exit status 126: A matching file was found but could not be executed, commonly because of permissions, an invalid executable format, or a missing dynamic loader.

  • Wrong command selected: An earlier PATH directory contains another executable with the same name.

  • Stale Bash cache: Bash remembers a command’s previous pathname until the cache is cleared.

  • Different execution context: The command works in an interactive shell but not under sudo, systemd, cron, a container, or a CI job.

Inspecting and Configuring PATH

Use printf '%s\n' "$PATH" to inspect the current search path and command -v systemctl to identify the selected executable. For a private user command directory, install -d -m 700 "$HOME/.local/bin" creates it with owner-only access, and export PATH="$HOME/.local/bin:$PATH" prepends it.

For interactive Bash sessions, manage the same export in ~/.bashrc through configuration management. Other shells and login environments may use different startup files, so verify the relevant execution context rather than assuming .bashrc applies everywhere.

For a systemd service, set Environment="PATH=/opt/app/bin:/usr/local/bin:/usr/bin:/bin" in a service drop-in when nonstandard binaries are required. After systemctl daemon-reload and restarting the unit, systemctl show app.service -p Environment displays the configured value.

Security and Production Considerations

  • Directory trust: A writable directory placed before trusted system directories can allow command hijacking. Do not place directories writable by untrusted users near the front of PATH.

  • Current-directory execution: An empty PATH element, such as a trailing colon or PATH=.:..., implicitly includes the current directory. Avoid this in scripts and production environments.

  • Privilege boundaries: sudo may replace the user’s PATH with a configured secure_path. Inspect it through approved sudoers configuration rather than assuming the interactive PATH is preserved.

  • Service isolation: systemd services do not automatically use an administrator’s interactive PATH. Define the service PATH explicitly when the application depends on nonstandard binaries.

  • Containers and CI: PATH is part of the image or runner configuration. Keep it consistent across replicas and define it in Dockerfiles, Kubernetes manifests, or CI configuration.

  • Monitoring: During incidents, compare command -v output across affected and healthy execution contexts. PATH drift can explain why only one replica or deployment stage fails.

  • Upgrades and backups: Preserve PATH and custom binary locations in configuration management or infrastructure as code. Package-managed binaries are restored through the package manager, while locally installed software needs its own deployment process.

Best Practices and Common Pitfalls

  • Never use chmod 777 to solve a PATH or permission problem.

  • Avoid replacing the entire PATH with one directory; prepend or append a trusted directory instead.

  • Quote assignments such as export PATH="$HOME/.local/bin:$PATH".

  • Keep directory names free of : because it is the PATH separator and has no portable escaping mechanism. Spaces and newlines are legal but can complicate shell quoting and configuration formats, so avoid them where practical.

  • Use command -v during diagnostics instead of assuming a command exists.

  • Verify behavior as the actual runtime user, including under sudo, systemd, and the container or CI environment.

33

How do apt and dnf manage package installation and upgrades?

apt and dnf install software from configured operating-system repositories and coordinate dependency resolution, package transactions, maintainer scripts, and recorded history. Ubuntu uses apt with dpkg and .deb packages, while RHEL and Fedora use dnf with rpm and .rpm packages.

Read full answer

Applies toapt 2.x on Ubuntu 22.04/24.04; DNF 4.x on RHEL 9/10; DNF 5.x on Fedora 41 and later

Command Workflows

Operation Ubuntu with apt RHEL/Fedora with dnf
Refresh repository indexes sudo apt update sudo dnf makecache
Install a package sudo apt install nginx sudo dnf install nginx
Install without prompts sudo apt-get install -y nginx sudo dnf install -y nginx
List available upgrades apt list --upgradable dnf check-update
Upgrade installed packages sudo apt upgrade sudo dnf upgrade
Remove a package sudo apt remove nginx sudo dnf remove nginx

Run apt update before installation because apt install selects versions using the repository indexes stored on the host. It does not upgrade installed software.

dnf normally refreshes repository metadata when needed; dnf makecache makes that behavior explicit in controlled environments. dnf check-update exits with status 100 when upgrades are available, so automation must distinguish that status from execution failures.

Under the Hood

  • Repository metadata: apt reads Release metadata and package indexes from /etc/apt/sources.list or /etc/apt/sources.list.d/. Repository signatures and package checksums protect the download. dnf reads repositories under /etc/yum.repos.d/; metadata and package signature verification depend on repo_gpgcheck and gpgcheck policy.

  • Dependency resolution: apt and apt-get select compatible versions, while dnf uses the libsolv dependency solver to construct the transaction.

  • Package installation: apt delegates package-state changes to dpkg; dnf uses rpm after resolving and coordinating the transaction.

  • Maintainer scripts: Packages can create users, enable services, update configuration, and perform package-specific actions. A successful download therefore does not guarantee that every post-installation action succeeds.

  • Package-manager history: apt records operations in /var/log/apt/history.log and /var/log/apt/term.log. dnf exposes transactional history through dnf history.

Code Example

Typical Ubuntu 24.04 commands and abbreviated output:

$ sudo apt update
Hit:1 https://archive.example.com/ubuntu noble InRelease
Get:2 https://archive.example.com/ubuntu noble-updates InRelease
Reading package lists... Done

$ sudo apt install -y nginx
Setting up nginx (1.24.0-2ubuntu7) ...
Processing triggers for man-db ...

# Later, after the repository publishes a newer package:
$ sudo apt update
$ apt list --upgradable
Listing...
nginx/noble-updates,noble-security 1.24.0-2ubuntu7.1 amd64 [upgradable from: 1.24.0-2ubuntu7]

$ sudo apt upgrade -y
Calculating upgrade... Done
The following packages will be upgraded:
  nginx
Setting up nginx (1.24.0-2ubuntu7.1) ...

Typical RHEL 9 commands and abbreviated output:

$ sudo dnf makecache
Metadata cache created.

$ sudo dnf install -y nginx
Installed:
  nginx-1:1.24.0-2.el9.x86_64

# After a later repository refresh publishes another version:
$ dnf check-update
Available Upgrades
nginx.x86_64           1:1.24.0-3.el9            nginx

$ sudo dnf upgrade -y
Upgraded:
  nginx

DNF5 on current Fedora accepts the same core workflow, although output wording can differ. The -y option answers affirmative prompts automatically; production automation should first capture and review the proposed transaction.

Upgrade Semantics

  • apt upgrade: Installs newer versions and may install new dependencies, but does not remove existing packages. Use apt full-upgrade when removals may be required to satisfy dependencies.

  • dnf upgrade: Upgrades installed packages and removes packages declared obsolete by repository metadata. If another installed package would have to be erased, --allowerasing permits that only after reviewing the transaction.

  • Security updates: Both managers install updates published by configured security repositories. Ubuntu commonly uses unattended-upgrades; RHEL commonly uses dnf-automatic.

  • Version selection: Use apt install package=version or dnf install package-version to request a specific available version. Downgrades require additional administrative handling and are not automatically safe.

Production Practices and Failure Modes

  • Test first: Simulate an Ubuntu upgrade with sudo apt-get --simulate --with-new-pkgs upgrade. On RHEL, use sudo dnf upgrade --assumeno and review the transaction.

  • Protect the system: Take a snapshot or ensure the host can be rebuilt before a major upgrade. dnf history undo <transaction-id> can undo an eligible DNF transaction, but it is not a substitute for backups or snapshots; apt has no equivalent general-purpose rollback command.

  • Handle failures safely: Do not kill a package manager during installation. If dpkg reports unconfigured packages, inspect the error and use sudo dpkg --configure -a or sudo apt-get --fix-broken install as appropriate.

  • Avoid bypassing verification: Do not use --allow-unauthenticated or --nogpgcheck in production. Enforce trusted mirrors, valid GPG keys, gpgcheck=1, and repo_gpgcheck=1 where supported.

  • Plan reboots and locking: Kernel and other low-level upgrades may require a reboot. Monitor services afterward, coordinate package-manager locks, and use apt-daily-upgrade.timer or dnf-automatic.timer instead of running conflicting unattended jobs concurrently.

34

What are Snap and Flatpak package formats?

Snap and Flatpak are Linux application distribution formats that package applications independently of the host’s native package manager. Snap is maintained by Canonical and is distributed mainly through the Snap Store; Flatpak uses configured OSTree repositories such as Flathub and normally pairs applications with separately installed shared runtimes.

Read full answer

Applies toSnap (`snapd` 2.x on Ubuntu 24.04) and Flatpak 1.x (Ubuntu 24.04, RHEL 9/10, and Fedora 42)

How Packaging Works

  • Snap package: A .snap file containing the application payload, metadata, and usually a dependency on a shared base snap such as core24. Its main filesystem is a read-only SquashFS image mounted under /snap.

  • Flatpak application: Usually an OSTree object in a configured repository. The application contains its own code and metadata while depending on a separately installed runtime containing common libraries, allowing multiple applications to share one runtime.

  • Snap Store: Canonical’s primary distribution channel. Snap packages can also come from private stores.

  • Flatpak remotes: Independently configured repositories. Flathub is the largest community repository, but organizations can operate private remotes.

  • Flatpak bundles: A .flatpak file is a single-file export of an application or runtime for local or offline distribution.

Key Differences

Area Snap Flatpak
Primary distribution Snap Store Flatpak remotes, especially Flathub
Packaging model Application payload plus a base snap when a shared base is used Application plus separately installed runtime
Confinement AppArmor, seccomp, mount namespaces, and interfaces Bubblewrap namespaces, seccomp filters, and desktop portals
Permissions Granted through snap interface connections Granted through Flatpak permissions and portals
Updates Automatic refresh is enabled by default Flatpak itself normally requires flatpak update; desktop software may automate it
Rollback snap revert PACKAGE flatpak rollback APP_ID, when a previous deployment remains
Typical strength Canonical integration and broad hardware support Desktop integration through XDG portals

Filesystem and Runtime Model

Neither format replaces traditional Linux packages such as apt or dnf.

  • Snap installation data is normally stored under /var/lib/snapd/snaps.

  • Snap user data is stored under ~/snap/<application>; system daemons may also use /var/snap/<application>.

  • Flatpak system installations live under /var/lib/flatpak.

  • Flatpak user installations live under ~/.local/share/flatpak.

  • Flatpak application state is commonly stored under ~/.var/app/<application-id>.

Confinement limits what an application can access, but it does not replace systemd, /etc, /usr, or normal operating-system administration.

Security and Production Behavior

  • Snap interfaces: Applications request access to resources such as network, home, removable-media, or system-observe. Inspect requests with snap connections PACKAGE.

  • Flatpak permissions: Filesystem and device access are controlled separately, while portals mediate operations such as printing, screen sharing, and opening files.

  • Trust: Verified package and repository signatures protect integrity but do not guarantee that an application is safe. Review publishers, permissions, and remote repositories.

  • Updates: Snap automatic refreshes can change application behavior over time. Use refresh holds where an exact revision must remain deployed. Flatpak’s update model provides direct control, although desktop software or scheduled jobs can still automate updates.

  • Monitoring: Review Snap daemon activity with journalctl -u snapd.service, and monitor Flatpak helper and application logs through journalctl.

  • Backups: Back up application state such as ~/snap/<name> or ~/.var/app/<app-id>, not only installed package files.

Best Practices and Pitfalls

  • Prefer distro-managed packages for servers, system services, and infrastructure agents unless isolation provides a clear benefit.

  • Avoid classic snaps when strict confinement is possible; they receive much broader host access.

  • Do not grant broad home or filesystem permissions without reviewing the application’s requirements.

  • Test specific Snap revisions or Flatpak commits in staging. Channels and branches are mutable tracking references rather than immutable pins.

  • Remember that Snap and Flatpak are application distribution and sandboxing technologies, not general replacements for containers, immutable images, or infrastructure provisioning.

35

How do you compile and install Linux software from source?

A source build compiles software from its original code and then installs the resulting binaries, libraries, and configuration. The usual workflow is to obtain and verify the release archive, install build dependencies, configure the build, compile it, test it, and install it under a controlled filesystem path.

Read full answer

Applies toLinux (Ubuntu 22.04/24.04, RHEL 9/10); example uses GNU Hello 2.12.1

Build Workflow

Different projects use different build systems, but the general process is consistent:

  1. Install prerequisites: Compilers, make, wget or curl, and development headers such as libssl-dev.

  2. Download the release archive: Prefer a signed release or an official source repository over an arbitrary mirror.

  3. Verify integrity: Compare the archive’s checksum with the upstream-published checksum. Use a detached signature or commit signature when available.

  4. Extract into a clean directory: Build from a new directory so stale objects from another version cannot contaminate the result.

  5. Configure the build: ./configure is common for Autotools projects; CMake and Meson are also common.

  6. Compile and test: Run make or the project’s equivalent, then execute its test target before installation.

  7. Install deliberately: Choose an appropriate prefix and install as root only after the build succeeds.

Build as an unprivileged user. Although make install needs elevated permissions in system-wide directories, running configure or make as root increases the impact of a malicious or defective build script.

What Happens Under the Hood

  • The compiler converts source files such as .c or .cpp into object files.

  • The assembler converts those object files into machine code.

  • The linker combines object files and referenced libraries into an executable.

  • Shared libraries are resolved at runtime by the dynamic loader.

  • make install copies artifacts into the selected prefix, commonly /usr/local, /opt, or /usr.

  • Package-specific install rules may also create directories, configuration files, users, or systemd units.

A prefix such as /opt/example/2.4.1 is useful in production because each release remains isolated and can be replaced through a symlink during a controlled rollout.

Code Example

The following builds GNU Hello using its Autotools workflow:

# Ubuntu or Debian
sudo apt update
sudo apt install -y build-essential wget ca-certificates

# RHEL or Fedora equivalent:
# sudo dnf group install -y "Development Tools"
# sudo dnf install -y wget

build_dir="$(mktemp -d)"
cd "$build_dir"

wget https://ftp.gnu.org/gnu/hello/hello-2.12.1.tar.gz

# Compare this value with the checksum published by upstream.
sha256sum hello-2.12.1.tar.gz

tar -xzf hello-2.12.1.tar.gz
cd hello-2.12.1

./configure --prefix=/opt/hello/2.12.1
make -j"$(nproc)"
make check

sudo install -d -o root -g root /opt/hello/2.12.1
sudo make install

/opt/hello/2.12.1/bin/hello

Realistic final output:

Hello, world!

For a large project, limiting parallelism may prevent memory exhaustion:

make -j2

For a distributed system, build once in a clean CI environment and promote the resulting artifact rather than rebuilding it independently on every production server.

Production and Failure Modes

Problem Meaning Correct response
configure cannot find a library A header, linker library, or development package is missing Install the matching -dev package and rerun configure
Compiler reports an error The source may be incompatible, or a compiler flag is unsupported Review the first error, compiler version, and project requirements
Linker reports undefined symbols Libraries were omitted or have incompatible ABIs Check link flags and library versions
Program cannot find a shared library Runtime search paths are incomplete Inspect with ldd and readelf -d; use packaging or ldconfig carefully
Permission denied during installation The target directory requires elevated privileges Install to a writable staging directory first, then install as root
Service starts but immediately exits Configuration, permissions, dependencies, or library paths are wrong Inspect systemctl status and journalctl -u SERVICE_NAME

Avoid replacing files managed by dpkg, dnf, or rpm. Doing so can cause package upgrades to overwrite the binary or leave unmanaged dependencies behind.

Operational Best Practices

  • Prefer apt or dnf for maintained packages; source builds are most appropriate when a required version or feature is unavailable.

  • Use versioned directories such as /opt/service/2.4.1, with symlinks or deployment tooling controlling which release is active.

  • Keep configuration and data outside the versioned binary directory so upgrades do not overwrite them.

  • Review configure, Makefile, and install targets before running them; installation scripts execute with the privileges used by make install.

  • Do not use chmod 777. Grant only the permissions required by the service account and directories involved.

  • Test the binary in staging, validate its runtime dependencies, and configure a systemd unit with a dedicated unprivileged user, resource limits, and restart policy.

  • Monitor the service with systemd status, journalctl, health checks, and metrics. Include startup failures, crashes, and performance regressions in alerts.

  • Keep previous releases and configuration backups for rollback, but avoid rebuilding the same version in place; preserve the original artifact for incident analysis.

36

How do you inspect Linux network interfaces and IP addresses with ip?

The Linux ip command is the modern tool for inspecting network interfaces and assigned IP addresses. Use ip link for interface state and ip address for IPv4/IPv6 configuration, while ip route shows how the kernel selects routes for outgoing traffic.

Read full answer

Applies toLinux with iproute2 5.x–6.x (Ubuntu 22.04/24.04, RHEL 9)

Core Commands

  • List all interfaces and addresses: bash ip address show address can be shortened to addr.

  • List interfaces without IP details: bash ip link show This shows link-layer information such as the interface name, MAC address, MTU, operational state, and queueing settings.

  • Use compact output for a quick summary: bash ip -brief address show The -brief option produces one summary line per interface.

  • Inspect one interface: bash ip address show dev ens3 ip link show dev ens3 Replace ens3 with the relevant interface name.

  • Restrict output to an IP family: bash ip -4 address show ip -6 address show

  • Show the routes used by the kernel: bash ip route show ip -6 route show

Interpreting the Output

A typical result is:

$ ip -brief address show
lo     UNKNOWN 127.0.0.1/8 ::1/128
ens3   UP      10.0.0.15/24 fe80::5054:ff:fe12:3456/64

Important fields include:

  • Interface name: ens3 is a predictable network name, but systems may also use eth0, enp1s0, bonds, bridges, or virtual interfaces such as veth*.

  • State: UP indicates that the link is operationally up. It does not prove that the host has connectivity.

  • Address and prefix: 10.0.0.15/24 means the interface uses 10.0.0.15 with a 24-bit subnet mask.

  • Scope: host is typically used for loopback addresses. global means the address is eligible for normal routing beyond the local link, but it may still be private IPv4 space or an IPv6 unique-local address and therefore not publicly reachable.

  • IPv6 link-local address: An address beginning with fe80:: is normally used for local IPv6 control traffic rather than general Internet access.

  • Address lifetime: DHCP-assigned addresses may show dynamic and finite valid_lft or preferred_lft lifetimes.

Routes, Neighbors, and DNS

The ip command does not display DNS configuration. Check it separately with systemd-resolved on Ubuntu and RHEL systems where it is enabled:

resolvectl status

To determine which interface and source address the kernel would use for a destination, run:

$ ip route get 1.1.1.1
1.1.1.1 via 10.0.0.1 dev ens3 src 10.0.0.15 uid 1000

This only queries the routing decision; it does not send traffic to 1.1.1.1.

For local IPv4 and IPv6 neighbor entries, equivalent to ARP and NDP information, make the address family explicit:

ip -4 neighbor show
ip -6 neighbor show

Code Example

Inspect an interface in detail:

$ ip address show dev ens3
2: ens3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
    link/ether 52:54:00:12:34:56 brd ff:ff:ff:ff:ff:ff
    inet 10.0.0.15/24 brd 10.0.0.255 scope global ens3
       valid_lft forever preferred_lft forever
    inet6 fe80::5054:ff:fe12:3456/64 scope link
       valid_lft forever preferred_lft forever

The LOWER_UP flag indicates that the underlying carrier is available. The inet and inet6 sections contain protocol-level addresses.

Production Practices

  • Do not confuse link state with end-to-end connectivity: After checking ip, test name resolution, routes, and reachability with tools such as resolvectl query, ping, tracepath, mtr, and ss. ICMP may be blocked even when TCP services work.

  • Inspect counters when diagnosing faults: ip -s link show reports packet and error counters. Physical-layer problems may require ethtool.

  • Keep configuration persistent: Changes made with ip address add or ip link set are runtime changes and normally disappear after reboot. Use Netplan on Ubuntu or NetworkManager on RHEL instead.

  • Use rollback when changing remote systems: On Ubuntu, sudo netplan try tests a Netplan change and reverts it automatically if the session is not confirmed. sudo netplan apply is permanent but does not provide the same rollback behavior.

  • Monitor interfaces in production: Collect interface state, traffic, errors, and dropped packets through tools such as Prometheus node_exporter and the cloud provider’s network metrics.

  • Protect management interfaces: Keep administrative addresses private where possible, restrict access with host and cloud firewalls, and avoid exposing management services directly to public networks.

37

How to check listening ports with ss or netstat?

Listening sockets are network endpoints bound to local ports and ready to accept connections. On Linux, ss is the preferred tool because it uses the kernel’s netlink interface; netstat is a legacy alternative that reads statistics from /proc.

Read full answer

Applies toLinux (Ubuntu 22.04/24.04, RHEL 9); iproute2 5.15–6.x and net-tools 1.60–2.10

Command Options

  • -l: Show listening sockets.

  • -n: Display numeric addresses and ports without DNS resolution.

  • -t: Include TCP sockets.

  • -u: Include UDP sockets.

  • -p: Show the owning PID and process name; usually requires root.

  • -4 or -6: Restrict output to IPv4 or IPv6.

  • -x: Show Unix domain sockets.

  • -a: Show both listening and non-listening sockets; without it, ss normally shows connected sockets.

The equivalent netstat options are mostly identical. Use ss -lnt for TCP listeners, ss -lntup for TCP listeners and unconnected UDP sockets with process information, and ss -antp when investigating established or other non-listening connections.

Interpreting the Output

For TCP listeners, State is normally LISTEN. Local Address:Port identifies the bind address and port:

  • 127.0.0.1:8080: Available only through the loopback interface.

  • 0.0.0.0:8080: Available through all IPv4 interfaces.

  • *:8080 or [::]:8080: Commonly represents all applicable IPv4 or IPv6 interfaces.

  • 10.0.0.10:8080: Available only on that specific local address.

For TCP listeners, Recv-Q is the number of pending established connections waiting to be accepted, while Send-Q is the maximum listen backlog. They are not normally byte counts for a listening socket.

UDP has no LISTEN state. A bound UDP socket commonly appears as UNCONN, which does not mean that it is inactive or unreachable.

ss Versus netstat

Tool Backend Recommended use
ss Kernel netlink socket diagnostics Primary operational tool; efficient and generally faster
netstat Reads /proc/net socket and protocol files Compatibility with older scripts or runbooks

ss is installed by the iproute2 package. On Ubuntu, install the legacy tool with sudo apt update && sudo apt install net-tools; on RHEL and Fedora, use sudo dnf install net-tools.

Code Example

Inspect TCP listeners and identify their owning processes:

# TCP listening sockets with process information
sudo ss -lntp

# TCP listeners and unconnected UDP sockets with process information
sudo ss -lntup

# IPv4 TCP listeners only
sudo ss -4 -lntp

# Listeners on TCP port 443
sudo ss -lntp 'sport = :443'

# Unix-domain listening sockets
sudo ss -lnxp

Example TCP listener output:

State  Recv-Q Send-Q Local Address:Port  Peer Address:Port Process
LISTEN 0      511    127.0.0.53%lo:53     0.0.0.0:*         users:(("systemd-resolve",pid=754,fd=13))
LISTEN 0      511    0.0.0.0:22           0.0.0.0:*         users:(("sshd",pid=812,fd=3))
LISTEN 0      4096   *:443                *:*                users:(("nginx",pid=934,fd=6))

The equivalent legacy commands are:

sudo netstat -lntp
sudo netstat -lntup

Example netstat output:

Proto Recv-Q Send-Q Local Address    Foreign Address  State   PID/Program name
tcp   0      0      0.0.0.0:22      0.0.0.0:*        LISTEN  812/sshd
tcp6  0      0      :::443           :::*             LISTEN  934/nginx
udp   0      0      127.0.0.53:53    0.0.0.0:*                 754/systemd-resolve

Production Troubleshooting and Security

  • A socket is not necessarily reachable: ss reports local sockets, not whether host firewalls, cloud security groups, network ACLs, or load balancers allow access. Check sudo ufw status verbose on Ubuntu or sudo firewall-cmd --list-all on RHEL, along with cloud firewall rules.

  • Test the actual path: From another host, use nc -vz -w 3 10.0.0.10 443 for a TCP connection or curl --connect-timeout 5 https://example.com/ -o /dev/null for a validated HTTP/TLS request. UDP probing requires a protocol-specific test.

  • Use the correct namespace: Containers and network namespaces have separate socket tables. A host’s ss may not show container listeners; use docker exec, kubectl exec, or nsenter -t PID -n ss -lntp where appropriate.

  • Restrict exposure: Services should bind to 127.0.0.1 when only local access is required. Binding to 0.0.0.0 or [::] exposes the service to every reachable interface.

  • Correlate with systemd: Use systemctl status nginx --no-pager and journalctl -u nginx --since "15 minutes ago" to verify service state and startup errors.

  • Do not monitor only PIDs: Restarted services receive new PIDs, and systemd socket activation can change which process owns the socket.

38

How do you test network connectivity with ping, traceroute, and mtr?

Use ping for endpoint reachability and latency, traceroute to identify the network path, and mtr to combine path tracing with continuous loss and latency measurements. None alone proves that an application protocol is working, so production checks should also validate the relevant TCP port or HTTP/TLS connection.

Read full answer

Applies toLinux (Ubuntu 22.04/24.04, RHEL 9)

How the Tools Work

  • ping: Sends ICMP Echo Request messages and measures Echo Reply latency. It is useful for basic reachability, round-trip time, and packet loss.

  • traceroute: Sends probes with increasing Time-to-Live values. Each router normally returns an ICMP Time Exceeded message, revealing the path toward the destination.

  • mtr: Repeatedly sends TTL-limited probes and combines traceroute-style hop data with per-hop latency statistics.

  • Important distinction: Intermediate hops may show apparent loss because routers deprioritize or rate-limit ICMP control messages. If later hops and the destination continue receiving replies, the path may still be healthy.

Useful Command Options

Tool Options Purpose
ping -c 4 Send four probes and exit
ping -W 2 Wait up to two seconds for each reply
ping -4, -6 Select the IP protocol version
traceroute -n Avoid reverse DNS lookups
traceroute -m 10 Stop after ten hops
traceroute -q 2 Send two probes per hop
traceroute -w 2 Wait up to two seconds per probe
mtr -r Produce a report instead of an interactive display
mtr -c 10 Collect ten reporting cycles
mtr -n Disable reverse DNS lookups
mtr -4, -6 Select the IP protocol version

Use -n during incident response when reverse DNS is slow or unavailable. It prevents path output from being delayed by name resolution.

Code Example

On Ubuntu, install the required packages:

sudo apt-get update
sudo apt-get install -y iputils-ping traceroute mtr

Test endpoint reachability:

ping -c 4 -W 2 10.0.0.1

Typical output:

PING 10.0.0.1 (10.0.0.1) 56(84) bytes of data.
64 bytes from 10.0.0.1: icmp_seq=1 ttl=64 time=0.412 ms
64 bytes from 10.0.0.1: icmp_seq=2 ttl=64 time=0.388 ms
64 bytes from 10.0.0.1: icmp_seq=3 ttl=64 time=0.401 ms
64 bytes from 10.0.0.1: icmp_seq=4 ttl=64 time=0.379 ms

4 packets transmitted, 4 received, 0% packet loss, time 3004ms
rtt min/avg/max/mdev = 0.379/0.395/0.412/0.014 ms

Trace the route:

traceroute -n -m 10 -q 2 -w 2 10.0.0.1

Typical output:

traceroute to 10.0.0.1 (10.0.0.1), 10 hops max, 60 byte packets
 1  10.0.0.1  0.412 ms  0.388 ms  0.401 ms

Collect a short mtr report:

mtr -n -r -c 10 10.0.0.1

Typical output:

Host                    Loss%   Snt   Last   Avg  Best  Wrst StDev
10.0.0.1                 0.0%    10    0.3   0.3   0.2   0.4   0.1

On Fedora, use dnf instead of apt:

sudo dnf install -y iputils traceroute mtr

On RHEL 9, mtr may need to come from an approved repository such as EPEL.

Production Troubleshooting

Symptom Likely interpretation Next check
Name resolution fails DNS problem Test with getent ahosts example.com
Ping succeeds, application fails ICMP works but the service or port is unavailable Test with nc, curl, or openssl
Middle hops show * Routers may suppress ICMP responses Continue checking later hops
Intermediate mtr loss, no end-to-end loss ICMP control-plane rate limiting Correlate with application metrics
Sustained loss at the final hop Destination, firewall, or end-to-end path problem Check host health and firewall logs
High latency without loss Congestion, distance, or ICMP deprioritization Compare multiple cycles and packet sizes

Run tests from the same source host, network segment, region, and egress path that experienced the incident. DNS, routing, NAT, stateful firewalls, and load balancers can make results differ substantially between locations.

These tools test IP and ICMP behavior, not application correctness. Validate the actual service with commands such as:

nc -vz -w 3 10.0.0.10 443
curl --connect-timeout 5 --max-time 10 https://example.com

Operational Best Practices

  • Establish thresholds for latency and loss rather than treating every ICMP timeout as an outage.

  • Correlate results with interface counters, flow logs, firewall logs, application metrics, and provider health information.

  • Record the source address, destination, protocol, and exact command in incident notes.

  • Avoid aggressive probing against third-party or shared infrastructure; excessive traffic can trigger rate limits.

  • Do not disable firewalls or weaken ICMP filtering globally to make diagnostics work. Instead, allow the required ICMP types selectively at network boundaries.

  • Prefer sustained mtr observations over a single traceroute snapshot, and test both IPv4 and IPv6 when the service supports dual stack.

39

How to query DNS with dig and nslookup?

dig and nslookup are command-line tools used to query DNS servers and retrieve resource records such as A, AAAA, CNAME, MX, and TXT. They are essential for diagnosing DNS resolution issues, verifying zone configurations, and troubleshooting network problems.

Read full answer

Applies todig and nslookup on Linux (Ubuntu 22.04/24.04, RHEL 9)

Overview

Both tools send a DNS query to a server—by default the system resolver listed in /etc/resolv.conf—and display the response. They support all common record types and can target a specific server with the @ option (dig) or the server command (nslookup). Under the hood, they construct a DNS message, send it over UDP (or TCP for large responses), and parse the reply.

  • dig: Stands for “Domain Information Groper”. It is part of the BIND utils and is favored for its extensive options and script‑friendly output.

  • nslookup: Name server lookup. It provides both a one‑shot command and an interactive prompt, making it useful for quick checks.

Feature dig nslookup
Primary use Advanced DNS debugging, scripting Simple lookups, interactive exploration
Interactive mode No Yes
Output control +short, +noall, +answer Limited formatting options
Recursion tracing +trace Not available
DNSSEC +dnssec Limited support
Scripting Excellent with +short Possible but less convenient

On Ubuntu they are provided by the dnsutils package (apt install dnsutils); on RHEL/Fedora they come from bind-utils (dnf install bind-utils).

Using dig

  • Syntax: dig [@server] [options] name [type]

  • Common options:

  • -t type: specify record type (e.g., A, AAAA, MX, TXT)

  • +short: print only the answer section, one record per line

  • +trace: show the full recursion path from the root to the authoritative server

  • @server: query a specific DNS server (e.g., @8.8.8.8)

  • -p port: use a non‑standard port (default is 53)

  • -b address: bind to a specific source address

  • Example queries:

  • dig example.com A

  • dig +short example.com

  • dig @8.8.8.8 example.com MX

  • Output sections: A typical reply contains the QUESTION, ANSWER, AUTHORITY, and ADDITIONAL sections. The ANSWER section holds the requested records, each with a TTL.

$ dig example.com A

; <<>> DiG 9.18 <<>> example.com A
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 12345
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 1232
;; QUESTION SECTION:
;example.com.			IN	A

;; ANSWER SECTION:
example.com.			300	IN	A	93.184.216.34

Using nslookup

  • Non‑interactive mode: nslookup [-option] ... [name] [(server)]

  • -type=<type>: set the query type (e.g., MX, CNAME)

  • Example: nslookup -type=MX example.com

  • Interactive mode: start with nslookup, then use the following commands:

  • set type=<type>: change the query type

  • server <addr>: specify a DNS server

  • set querytype=<type>: same as set type

  • exit or quit: leave the prompt

  • Example interactive session: “‘bash $ nslookup > set type=MX > example.com Server: 8.8.8.8 Address: 8.8.8.8#53

example.com mail exchanger = 10 mail.example.com. “‘

Code Example

# 1. Basic A record query with dig
dig example.com A

# 2. Short output, suitable for scripts
dig +short example.com

# 3. Query a specific server for MX records
dig @8.8.8.8 example.com MX

# 4. Non‑interactive nslookup
nslookup -type=MX example.com

# 5. Interactive nslookup
nslookup
> set type=A
> example.com
> exit

Best Practices and Troubleshooting

  • Target a reliable server: When diagnosing a problem, always query a known‑good DNS server (e.g., dig @1.1.1.1 or nslookup ... 1.1.1.1) to bypass the local resolver.

  • Use +short in scripts: For automated checks, dig +short returns a clean list of answers that can be easily parsed.

  • Trace recursion: dig +trace reveals the path from the root to the authoritative server, helping to identify delegation or forwarding issues.

  • Combine with other tools: Cross‑check results with host, resolvectl, or systemd‑resolve for consistency.

  • Security: dig +dnssec requests DNSSEC records (check the ad flag from a validating resolver, or use delv to validate); consider DNS over TLS (DoT) via systemd-resolved for encrypted queries.

  • Monitoring: Schedule periodic dig queries to measure latency and detect failures; integrate with Prometheus or a similar system.

  • Common pitfalls:

  • Forgetting to specify the record type (defaults to A).

  • Misinterpreting TTL values when planning DNS changes.

  • Ignoring non‑standard ports or firewall rules that block UDP/TCP 53.

  • Relying on a single tool; use both dig and nslookup to verify results.

40

How do you transfer files with scp and rsync on Linux?

Use scp for simple, one-time file copies and rsync for repeatable synchronization, large files, or deployment workflows. Both normally transfer data over an encrypted SSH connection, but rsync can transfer only changed portions and offers better verification, partial-transfer handling, and deployment options.

Read full answer

Applies toOpenSSH 9.0–10.x and rsync 3.2.x on Ubuntu 24.04, RHEL 9, and Fedora

scp

scp copies files or directory trees between local and remote hosts:

scp ./app.log deploy@10.0.0.20:/tmp/
scp ./app.log deploy@10.0.0.20:/tmp/app.log
scp -r ./dist deploy@10.0.0.20:/srv/app/

Typical output:

app.log                                      100%  1024KB   0.0MB/s   00:00
  • Recursive copy: -r includes subdirectories.

  • Preserve metadata: -p preserves modification times and file modes.

  • Compress the stream: -C helps when text data is transferred over a slow or high-latency link.

  • Use a non-default SSH port: scp -P 2222 file deploy@host:/path/. The scp port option is uppercase P.

  • Legacy protocol: OpenSSH 9.0 and later use SFTP by default. -O selects the legacy SCP protocol, but it should only be used for compatibility with old servers because SFTP is the safer modern default.

Unlike rsync, scp is not a synchronization tool. It copies the specified files and does not efficiently update an existing large file when only a small portion changed.

rsync

rsync compares source and destination and transfers only the required differences:

rsync -avz --partial --info=stats2 ./dist/ deploy@10.0.0.20:/srv/app/

Typical output:

sending incremental file list
dist/index.html

Number of regular files transferred: 1
Total file size: 1,200 bytes
Total transferred file size: 1,200 bytes
sent 1,235 bytes  received 35 bytes  2,540.00 bytes/sec
total size is 1,200  speed-up is 0.97
  • Archive mode: -a preserves permissions, timestamps, symbolic links, and other common metadata. Ownership preservation normally requires sufficient privileges on the remote system.

  • Verbose output: -v lists transferred files.

  • Compression: -z compresses data in transit; it is usually unnecessary for local transfers or already-compressed artifacts.

  • Partial files: --partial retains partial data, reducing retransmission when a transfer is interrupted.

  • Delayed updates: --delay-updates stores completed updates temporarily and renames them into place, reducing the chance of consumers seeing half-written files.

  • Checksum verification: --checksum compares file contents instead of relying on the default size and modification-time checks.

  • Custom SSH port: use rsync -e 'ssh -p 2222' ...; the SSH port option here is lowercase p.

The trailing slash is significant. rsync -a ./dist/ host:/srv/app/ copies the contents of dist, while rsync -a ./dist host:/srv/app/ creates or updates /srv/app/dist.

Comparison

Capability scp rsync
Simple one-time copy Excellent Good
Recursive transfer -r -a
Transfers only changed data No Yes
Preserves ACLs and extended attributes Limited -A and -X
Interrupted-transfer handling Basic --partial and related options
Dry-run comparison No --dry-run
Typical transport SFTP over SSH rsync protocol over SSH
Remote software required SSH/SFTP server SSH server and rsync

For ACLs and extended attributes, use rsync -aAX where both systems and the remote account support them. On RHEL and Fedora, SELinux contexts can still prevent access even when Unix permissions allow it; restore the correct context with restorecon rather than disabling SELinux.

Code Example

Install the required clients on Ubuntu or RHEL/Fedora:

# Ubuntu
sudo apt-get update
sudo apt-get install -y openssh-client rsync

# RHEL or Fedora
sudo dnf install -y openssh-clients rsync

Test a transfer before an incident or deployment:

# Preview what rsync would change
rsync -avzn --checksum ./dist/ deploy@10.0.0.20:/srv/app/

# Transfer into a staging directory
rsync -aAXz --partial --delay-updates \
  --info=stats2 ./dist/ deploy@10.0.0.20:/srv/releases/2025-03-08/

Verify important artifacts independently:

sha256sum ./dist/app.tar.gz
ssh deploy@10.0.0.20 'sha256sum /srv/releases/2025-03-08/app.tar.gz'

Production Practices and Failure Modes

  • Use SSH keys and a dedicated deployment account; avoid storing passwords in scripts or shell history.

  • Keep rsync installed on both endpoints. Protocol incompatibilities can occur when client and remote rsync versions differ substantially.

  • Transfer application releases into a new versioned directory, validate checksums and health checks, then switch a symlink atomically. Do not update live application files in place.

  • Monitor exit status and transfer statistics. An interrupted transfer can leave the destination tree partly updated; --delay-updates keeps new files out of place until the end, and --partial keeps partial data so a retry can resume.

  • Use --dry-run --checksum before destructive synchronization operations.

  • Preserve required ACLs, ownership, and SELinux contexts, but review ownership preservation carefully when the remote account is not root.

  • Never use chmod 777 to solve permission failures. Correct the account, group ownership, directory permissions, or mandatory-access-control policy instead.

41

How do I configure SSH key-based authentication on Linux?

SSH key-based authentication replaces password authentication with a cryptographic key pair: the client proves possession of the private key, while the server verifies the matching public key stored in authorized_keys. It reduces password exposure and supports stronger policies such as disabling root login and restricting forwarding.

Read full answer

Applies toLinux (Ubuntu 22.04/24.04, RHEL 9) with OpenSSH 8.7+

How It Works

  • Private key: Secret key stored on the client, normally at ~/.ssh/id_ed25519.

  • Public key: Non-secret key copied to the server as ~/.ssh/authorized_keys.

  • Authentication flow:

  • The client sends the username and public-key identifiers.

  • The server sends a challenge.

  • The client signs it using the private key.

  • The server verifies the signature against authorized_keys.

  • The private key never transfers to the server. Its passphrase protects the key at rest, but it is not sent during authentication.

Code Example

Generate a key on the client workstation:

ssh-keygen -t ed25519 -a 100 -C "user@workstation"

Use a strong passphrase unless the key is managed by an appropriate automated-secret system.

Copy the public key to the server:

ssh-copy-id -i ~/.ssh/id_ed25519.pub user@10.0.0.10

Typical output:

user@10.0.0.10's password:
Number of key(s) added: 1

This password is used only to bootstrap the key. Verify the server’s SSH host-key fingerprint through a trusted channel rather than bypassing host-key checking.

Test the connection in a second terminal before changing server policy:

ssh -i ~/.ssh/id_ed25519 -o IdentitiesOnly=yes \
  user@10.0.0.10 'hostname; id -un'

Expected output:

app-01
user

Confirm that authentication succeeds without any password or keyboard-interactive fallback:

ssh -i ~/.ssh/id_ed25519 \
  -o IdentitiesOnly=yes \
  -o BatchMode=yes \
  -o PasswordAuthentication=no \
  -o KbdInteractiveAuthentication=no \
  user@10.0.0.10 'true'

A successful command exits with status 0 and produces no output.

Server Hardening

After key authentication works, create a drop-in configuration:

sudoedit /etc/ssh/sshd_config.d/99-key-only.conf
PubkeyAuthentication yes
PasswordAuthentication no
KbdInteractiveAuthentication no
AuthenticationMethods publickey
PermitRootLogin no

These settings require public-key authentication for every user covered by the configuration. Do not apply them until all required administrators have tested their keys. Keep the current SSH session open and access the server through a provider console so access is not lost.

Validate the configuration before reloading:

sudo sshd -t

Reload the appropriate service:

# Ubuntu/Debian
sudo systemctl reload ssh

# RHEL/Fedora
sudo systemctl reload sshd

Check the effective server settings:

sudo sshd -T | grep -E \
'^(pubkeyauthentication|passwordauthentication|kbdinteractiveauthentication|authenticationmethods|permitrootlogin) '

Expected values include:

pubkeyauthentication yes
passwordauthentication no
kbdinteractiveauthentication no
authenticationmethods publickey
permitrootlogin no

Troubleshooting and Failure Modes

  • Incorrect permissions: OpenSSH may reject otherwise valid keys. Check ownership and modes:

bash stat -c '%U:%G %a %n' /home/user /home/user/.ssh /home/user/.ssh/authorized_keys

Expected ownership is the login user and the login user’s primary group. Typical modes are 755 or stricter for the home directory, 700 for .ssh, and 600 for authorized_keys.

  • Wrong identity or excessive keys: Use IdentitiesOnly=yes and specify -i ~/.ssh/id_ed25519 when several keys are loaded.

  • SELinux denial on RHEL: Restore the expected security context:

bash sudo restorecon -Rv /home/user/.ssh

  • Server logs: Use the service name matching the distribution:

bash journalctl -u ssh -n 50 --no-pager journalctl -u sshd -n 50 --no-pager

A successful attempt contains Accepted publickey; rejected attempts commonly contain Failed publickey or Authentication refused.

Production Practices

  • Give each person or workload a separate key so access can be revoked without affecting others.

  • Remove a revoked key’s public-key line from ~/.ssh/authorized_keys.

  • Restrict authorized_keys entries by source address when appropriate, and disable forwarding with options such as no-port-forwarding and no-agent-forwarding when those features are unnecessary.

  • Store private keys in encrypted workstations, password managers, or secret-management systems; do not copy them into images or repositories.

  • Monitor accepted and failed SSH authentication, alert on unusual activity, and retain a documented break-glass recovery method.

  • On cloud instances, account for provider tooling such as cloud-init, which may install keys or regenerate SSH configuration from instance metadata.

42

How to harden an SSH server on Linux?

SSH server hardening reduces OpenSSH’s exposed authentication and network surface without losing legitimate access or recoverability. Changes should be staged, validated, and applied while retaining a tested administrative session.

Read full answer

Applies toOpenSSH 8.7p1 on RHEL 9, 8.9p1 on Ubuntu 22.04, and 9.6p1 on Ubuntu 24.04

Baseline Configuration

OpenSSH supports SSH protocol 2 in these versions, so a Protocol 2 directive provides no additional protection. Edit /etc/ssh/sshd_config or a drop-in under /etc/ssh/sshd_config.d/.

Directive Recommended value Purpose
PermitRootLogin no Blocks direct root SSH; privileged access should use audited sudo.
PasswordAuthentication no Disables SSH password authentication.
KbdInteractiveAuthentication no Prevents alternative PAM password prompts; enable only for a deliberate MFA design.
PubkeyAuthentication yes Enables public-key authentication.
AuthenticationMethods publickey Requires a successful public-key authentication.
UsePAM yes Applies PAM account, session, and password-policy handling; it does not itself provide MFA.
MaxAuthTries 3 Limits authentication attempts per connection.
LoginGraceTime 30 Limits unauthenticated connection time.
ClientAliveInterval 300 Sends a probe after five minutes without client data.
ClientAliveCount 2 Disconnects after two consecutive unanswered probes, approximately ten minutes.
X11Forwarding no Removes unnecessary X11 forwarding.
AllowAgentForwarding no Prevents SSH-agent forwarding unless explicitly required.

Keys and Accounts

Use a unique Ed25519 key per operator, protected by a passphrase; prefer a hardware-backed FIDO2 key for privileged access. Distribute and test the key before disabling passwords.

Store ~/.ssh with mode 0700 and authorized_keys with mode 0600, both owned by the login account. Remove stale or shared keys, use individual accounts with sudo, and consider AllowUsers or AllowGroups after creating the referenced accounts or groups. authorized_keys options such as from= and restrict can limit a key’s source and capabilities, but they require testing.

Network, MFA, and Monitoring

Permit SSH only from administrative networks using host firewalls. Changing port 22 reduces log noise but is not a security boundary; on RHEL, it also requires updating firewalld and SELinux port policy.

Fail2ban can limit repeated password failures, but it is supplemental and can miss distributed attacks. For MFA, configure the PAM stack and use KbdInteractiveAuthentication yes with an appropriate AuthenticationMethods value, rather than assuming UsePAM yes enables MFA.

Monitor authentication failures, accepted sessions, configuration changes, and account changes. Use journalctl -u ssh -f on Ubuntu and journalctl -u sshd -f on RHEL; /var/log/auth.log and /var/log/secure apply only when the corresponding rsyslog configuration is installed.

Code Example

# Generate a passphrase-protected key, then install it while password login is still available.
ssh-keygen -t ed25519 -a 100 -f ~/.ssh/id_ed25519_work
ssh-copy-id -i ~/.ssh/id_ed25519_work.pub deploy@host

# Replace "deploy" with the real account.
ssh deploy@host 'sudo chown deploy:deploy ~/.ssh ~/.ssh/authorized_keys && chmod 700 ~/.ssh && chmod 600 ~/.ssh/authorized_keys'
# /etc/ssh/sshd_config.d/00-hardening.conf
PermitRootLogin no
PasswordAuthentication no
KbdInteractiveAuthentication no
PubkeyAuthentication yes
AuthenticationMethods publickey
PermitEmptyPasswords no
UsePAM yes
MaxAuthTries 3
LoginGraceTime 30
ClientAliveInterval 300
ClientAliveCount 2
X11Forwarding no
AllowAgentForwarding no

Ensure /etc/ssh/sshd_config contains Include /etc/ssh/sshd_config.d/*.conf before other directives. OpenSSH uses the first obtained value for each directive, so sudo sshd -T must be used to confirm the effective policy.

# Ubuntu example; audit and remove any broader existing SSH allowance.
sudo ufw allow from 10.0.0.0/16 to any port 22 proto tcp
sudo ufw status verbose

# RHEL 9 alternative.
sudo firewall-cmd --permanent --add-rich-rule='rule family="ipv4" source address="10.0.0.0/16" port port="22" protocol="tcp" accept'
sudo firewall-cmd --reload
sudo firewall-cmd --list-all
sudo sshd -t && echo "sshd configuration is valid"
sudo sshd -T | grep -E '^(permitrootlogin|passwordauthentication|kbdinteractiveauthentication|pubkeyauthentication|authenticationmethods|maxauthtries|clientaliveinterval|clientalivecount)'

# Run only the command matching the distribution.
sudo systemctl reload ssh    # Ubuntu
sudo systemctl reload sshd   # RHEL

Production Practices

Back up the main configuration, drop-ins, firewall rules, and relevant SELinux policy before changing them. Keep the current session open, open a second session with the new policy, and verify key login, sudo, and required tunnels before closing either session.

Upgrade the server package regularly with sudo apt-get update && sudo apt-get upgrade openssh-server on Ubuntu or sudo dnf upgrade openssh-server on RHEL. Centralize logs, alert on repeated failures and unexpected accepted sessions, review failed bans and rate limits, and maintain cloud-console or physical-console access for emergency recovery.

43

What are /etc/hosts and /etc/resolv.conf used for in Linux?

/etc/hosts provides local static hostname-to-IP mappings, while /etc/resolv.conf configures the system resolver, including upstream DNS servers, search domains, and resolver options. Applications using libc and the Name Service Switch (NSS) normally consult /etc/nsswitch.conf to determine whether and in what order to use static files and DNS.

Read full answer

Applies toLinux (Ubuntu 22.04/24.04, RHEL 9; systemd-resolved 249–255 where installed)

How Name Resolution Works

  1. An application calls a resolver function such as getaddrinfo().

  2. NSS consults the hosts database configured in /etc/nsswitch.conf.

  3. If configured, the files module checks /etc/hosts.

  4. If no static match exists, the dns module sends queries to a resolver configured through /etc/resolv.conf.

  5. The resolver returns the matching address to the application.

A common NSS configuration is hosts: files dns, meaning local mappings are checked before DNS. Some systemd-resolved configurations also use the resolve or myhostname modules. Inspect the actual order with grep '^hosts:' /etc/nsswitch.conf rather than assuming it.

DNS-only clients and libraries may bypass NSS and /etc/hosts, so a successful lookup in one application does not prove that every application resolves names the same way.

/etc/hosts

Each non-comment entry contains an IP address, a canonical hostname, and optional aliases. For example, 10.20.0.15 api.internal.example.com api maps the canonical name api.internal.example.com and its alias api to 10.20.0.15.

  • Local overrides: Useful for testing a service against a specific server or temporarily redirecting a hostname.

  • No DNS dependency: A matching entry resolves even if DNS is unavailable.

  • Limited scalability: Entries must be maintained manually and can become stale.

  • Local effect: It influences name lookups on that machine but does not publish DNS records to other systems.

  • No wildcard records: Standard Linux hosts files do not support patterns such as *.example.com.

  • Separate from the system hostname: On the named systemd-based distributions, the configured hostname is stored in /etc/hostname and can be managed with hostnamectl.

Use getent hosts api.internal.example.com to test the configured NSS lookup path. For the example above, it normally reports the 10.20.0.15 mapping.

/etc/resolv.conf

This file tells the resolver which DNS servers to query and how to behave. Common directives include nameserver 10.0.0.53, search example.com, and options such as ndots:2, timeout:2, attempts:3, and rotate.

  • nameserver: The IPv4 or IPv6 address of a DNS server. Multiple nameservers provide fallback candidates.

  • search: A list of domains used to expand short names. For example, api may be tried as api.example.com.

  • options ndots:N: Controls whether a name with fewer than N dots is tried with the search domains before being treated as absolute.

  • options timeout:N: Sets the resolver timeout in seconds.

  • options attempts:N: Sets the number of times the resolver attempts each query.

  • options rotate: Rotates the order of configured nameservers.

On Ubuntu installations using systemd-resolved, /etc/resolv.conf is commonly a symlink to a runtime stub-resolver file containing nameserver 127.0.0.53. The stub forwards requests to upstream servers configured through NetworkManager or systemd-resolved. Inspect that state with resolvectl status and test a lookup with resolvectl query api.example.com.

Do not edit the runtime target as a permanent configuration change: systemd-resolved, NetworkManager, or DHCP may overwrite it during a reboot or network change.

Comparing the Two Files

File Maps names to addresses Requires DNS servers Intended use
/etc/hosts Yes, statically No Small local overrides and bootstrap resolution
/etc/resolv.conf No Yes Upstream DNS addresses, search domains, and resolver options

Use getent hosts for an application-style NSS lookup. In contrast, dig queries DNS directly and generally bypasses /etc/hosts. Install it first with sudo apt install dnsutils on Ubuntu or sudo dnf install bind-utils on RHEL, then run dig +short example.com; the returned address can vary.

Production and Security Practices

  • Prefer DNS for service discovery, dynamic infrastructure, load balancing, and failover. Static /etc/hosts entries can direct traffic to a failed or decommissioned node.

  • Use /etc/hosts for controlled exceptions, bootstrap dependencies, or emergency overrides with a documented removal plan.

  • Keep /etc/hosts and the regular file targeted by /etc/resolv.conf owned by root and not writable by unprivileged users. Writable resolver configuration can redirect traffic or suppress DNS failures.

  • Monitor DNS latency, timeouts, SERVFAIL responses, and resolver cache statistics rather than assuming that a successful ping means name resolution is healthy.

  • Back up custom configuration, but recognize that systemd-resolved, NetworkManager, or DHCP may regenerate /etc/resolv.conf.

  • Avoid unnecessary search domains. They can expand short names unexpectedly, increase lookup traffic, and expose internal naming conventions.

  • Never “fix” DNS problems by disabling SELinux, opening arbitrary firewall ports, or replacing resolver security controls. Diagnose the complete resolution path first.

44

How to configure firewall with ufw, firewalld, iptables?

The Linux kernel’s netfilter framework provides packet filtering, while ufw, firewalld, and iptables expose different management layers. Choose one firewall manager per host: current releases usually store rules through nftables, and mixing legacy, nft, and higher-level frontends can produce unexpected rule interactions.

Read full answer

Applies toufw 0.36 on Ubuntu 22.04/24.04; firewalld 1.x/2.x on RHEL 9/Fedora 42; iptables 1.8.x with nftables or legacy backends

How the Firewall Stack Works

  • IPv4 netfilter processes packets through PREROUTING, INPUT, FORWARD, OUTPUT, and POSTROUTING hooks.

  • The filter table controls traffic acceptance, nat handles network address translation, and mangle modifies packet headers.

  • iptables manages rules directly. ufw and firewalld are higher-level front ends; current Ubuntu ufw normally uses iptables-nft, and current firewalld normally uses nftables directly.

  • Inspect the backend before troubleshooting: sudo iptables --version. Do not combine iptables-legacy and iptables-nft rule sets on the same host.

  • IPv6 has separate policy and rules; configure it explicitly rather than assuming IPv4 rules cover it.

Tool Default manager Model Persistence
ufw Ubuntu Simple rules and application profiles Files under /etc/ufw, applied at boot
firewalld RHEL/Fedora Zones, services, and runtime/permanent rules XML under /etc/firewalld
iptables Not a default manager Direct table, chain, and target control Requires an explicit restore mechanism

ufw on Ubuntu

ufw is normally installed but inactive on Ubuntu. Install it with sudo apt install ufw.

  • sudo ufw allow ssh permits the profile’s SSH port, normally TCP 22.

  • sudo ufw allow 80/tcp permits HTTP, while sudo ufw deny 23 rejects Telnet.

  • sudo ufw status verbose shows the active policy and rules.

  • sudo ufw logging low enables low-volume logging, normally through /var/log/ufw.log and the journal.

  • Rules persist in /etc/ufw/user.rules and /etc/ufw/user6.rules and are reapplied during boot.

  • Test with sudo ufw --dry-run enable before enabling it remotely.

firewalld on RHEL and Fedora

Install firewalld with sudo dnf install firewalld, then use systemctl to manage it.

  • Interfaces are assigned to zones such as public, internal, and trusted.

  • Services are predefined groups of ports; http and ssh are standard firewalld services.

  • Without --permanent, changes affect only the current runtime configuration.

  • --permanent changes stored configuration, but sudo firewall-cmd --reload is required to apply those changes to the runtime.

  • --runtime-to-permanent copies runtime rules into permanent configuration; it is not a backup operation.

  • Service messages are available through sudo journalctl -u firewalld; denied-packet logging may additionally appear in the kernel journal.

iptables: Direct Control

iptables provides precise control over IPv4 packets but does not provide persistent storage by itself.

  • sudo iptables -L lists the filter table.

  • sudo iptables -A INPUT -p tcp --dport 22 -j ACCEPT appends an SSH rule.

  • sudo iptables -P INPUT DROP changes the default input policy.

  • sudo iptables-save exports rules, and sudo iptables-restore restores them.

  • On Debian/Ubuntu, iptables-persistent and netfilter-persistent can restore /etc/iptables/rules.v4 at boot.

  • On RHEL 9, use firewalld or a systemd unit that runs iptables-restore; merely writing /etc/sysconfig/iptables does not guarantee restoration.

  • Configure ip6tables separately when IPv6 is enabled.

Code Example

Allow SSH and HTTP while rejecting other incoming IPv4 traffic.

ufw:

sudo ufw default deny incoming
sudo ufw default allow outgoing
sudo ufw allow 22/tcp
sudo ufw allow 80/tcp
sudo ufw logging low
sudo ufw --dry-run enable
sudo ufw enable
sudo ufw status verbose
Status: active
Logging: on (low)
Default: deny (incoming), allow (outgoing)
To                         Action      From
--                         ------      ---
22/tcp                     ALLOW IN    Anywhere
80/tcp                     ALLOW IN    Anywhere

firewalld:

sudo systemctl enable --now firewalld
sudo firewall-cmd --set-default-zone=public
sudo firewall-cmd --permanent --zone=public --add-service=ssh
sudo firewall-cmd --permanent --zone=public --add-service=http
sudo firewall-cmd --reload
sudo firewall-cmd --list-all
public (active)
  target: default
  services: http ssh
  ports:
  protocols:

iptables:

sudo iptables -A INPUT -m conntrack --ctstate ESTABLISHED,RELATED -j ACCEPT
sudo iptables -A INPUT -p tcp --dport 22 -j ACCEPT
sudo iptables -A INPUT -p tcp --dport 80 -j ACCEPT
sudo iptables -P INPUT DROP
sudo iptables -L INPUT -n -v --line-numbers
Chain INPUT (policy DROP 0 packets, 0 bytes)
num  target     prot opt source      destination
1    ACCEPT     all  --  anywhere    anywhere    ctstate RELATED,ESTABLISHED
2    ACCEPT     tcp  --  anywhere    anywhere    tcp dpt:ssh
3    ACCEPT     tcp  --  anywhere    anywhere    tcp dpt:http

For Debian/Ubuntu persistence, do not combine this workflow with ufw:

sudo apt install -y iptables-persistent
sudo iptables-save | sudo tee /etc/iptables/rules.v4 >/dev/null
sudo systemctl enable netfilter-persistent

Production Practices

  • Allow SSH before applying a deny policy, retain console or out-of-band access, and schedule an automatic rollback for remote changes.

  • Maintain equivalent IPv4 and IPv6 policy. Test connectivity from both address families.

  • Assign one owner to the host firewall. Do not concurrently modify the same ruleset with ufw, firewalld, iptables-nft, and iptables-legacy.

  • Monitor host logs and upstream cloud security groups. Provider security groups may block traffic before it reaches the host, while the host firewall provides defense in depth.

  • Back up /etc/ufw, /etc/firewalld, and iptables-save output before upgrades or major changes.

  • Restrict administrative ports by source network where practical. fail2ban can reduce repeated authentication failures, but it complements rather than replaces a firewall policy.

45

How does nftables relate to iptables in Linux?

nftables is Linux’s native firewall framework for packet filtering and Network Address Translation (NAT). iptables is an older userspace interface that can use either the legacy kernel implementation or the nftables compatibility backend.

Read full answer

Applies tonftables 1.x on Ubuntu 22.04/24.04 and RHEL 9 (Linux 5.14–6.8)

How nftables Works

nftables manages objects inside the Linux kernel’s Netfilter subsystem:

  • Tables group related rules, commonly filter and nat.

  • Chains contain rules and may be regular chains or base chains attached to a Netfilter hook.

  • Rules match packet fields and execute actions such as accept, drop, reject, log, and counter.

  • Sets and maps efficiently store addresses, ports, interfaces, intervals, and other values.

A base chain is attached to a hook such as input, output, or forward. Its priority controls its ordering relative to other base chains at that hook. The inet family can apply the same rules to both IPv4 and IPv6.

Connection tracking allows rules to distinguish new flows from established or related traffic. For example, ct state established,related accept accepts packets belonging to tracked connections or related flows, such as response traffic after an outbound connection. This requires the nf_conntrack subsystem.

Relationship to iptables

Interface Kernel path Characteristics
iptables-legacy Legacy iptables/x_tables modules Older implementation; IPv6 requires the separate ip6tables-legacy utility, and nftables-native features are unavailable
iptables-nft nftables through an xtables compatibility layer Accepts iptables-style syntax and represents rules as nftables objects; IPv6 uses the separate ip6tables-nft utility
nft Native nftables Direct access to sets, maps, expressions, stateful objects, and unified IPv4/IPv6 rules

On many current Ubuntu and RHEL systems, iptables --version reports nf_tables, meaning the selected iptables alternative uses the nftables backend. This does not make the two interfaces equivalent: native nftables expressions and object types are not fully exposed by the iptables compatibility layer.

The selected Ubuntu alternative can be inspected with sudo update-alternatives --display iptables. On RHEL, use sudo alternatives --display iptables.

Production Behavior

  • Atomic updates: Loading rules with nft -f processes the batch as one transaction. If parsing or validation fails, the ruleset is not partially applied.

  • Counters: Rules can include counter, and sudo nft list ruleset shows packet and byte counts useful for identifying active or unexpectedly quiet paths.

  • Logging: The log statement can include a meaningful prefix, but kernel log-rate limiting can suppress repeated messages.

  • Persistence: Ubuntu commonly uses /etc/nftables.conf with nftables.service; RHEL 9 commonly uses /etc/sysconfig/nftables.conf with its nftables.service.

  • NAT: NAT base chains are attached to the appropriate Netfilter hooks, commonly prerouting and postrouting. The first packet of a tracked flow establishes the mapping, after which subsequent packets use the connection-tracking entry.

  • Firewall managers: RHEL commonly uses firewalld with the nftables backend. Direct nft rules are outside firewalld’s normal management, so mixing them can create ordering, persistence, and troubleshooting problems.

Best Practices and Common Pitfalls

  • Check which backend is active with iptables --version before diagnosing behavior or migration issues.

  • Validate configuration with sudo nft -c -f /etc/nftables.conf on Ubuntu or the corresponding RHEL configuration path.

  • Back up the active ruleset with sudo nft list ruleset > ruleset.backup.

  • Prefer named tables, atomic file-based updates, explicit policies, and reviewed changes over undocumented interactive modifications.

  • Keep an existing SSH session and use an out-of-band console when changing remote firewall rules.

  • Do not use broad wildcard allows or flush ruleset without understanding the effect on every base chain.

  • Account for IPv6, Docker, Kubernetes, load balancers, high connection rates, and connection-table capacity before deploying a restrictive policy.

  • Monitor rule counters, drops, connection-table usage, and application health during and after every change.

46

How do you mount and unmount filesystems on Linux?

Mounting attaches a filesystem to a directory in the Linux filesystem hierarchy, while unmounting detaches it after users and services have stopped accessing it. Persistent mounts are normally defined in /etc/fstab and managed by systemd.

Read full answer

Applies toLinux (Ubuntu 22.04/24.04, RHEL 9)

Filesystem and Mount-Point Model

Linux presents disks, partitions, logical volumes, network shares, and pseudo-filesystems through a single directory tree.

  • / is the root filesystem and should not normally be unmounted.

  • /home, /var, and /srv are common locations for persistent application data.

  • /mnt is traditionally used for temporary manual mounts, while /media is intended for removable media.

  • /proc, /sys, and /dev are kernel-managed filesystems mounted during boot.

  • A mount point is simply a directory. After mounting, its previous contents are hidden by the mounted filesystem.

The kernel’s virtual filesystem layer routes path access through the mounted filesystem. Unmounting prevents new accesses and succeeds only when no process has the mount in use.

Inspecting Filesystems

Use these commands before mounting:

  • lsblk -f: shows block-device topology, filesystem types, labels, UUIDs, and mount points.

  • blkid: retrieves filesystem metadata such as UUID and TYPE.

  • findmnt: displays the authoritative kernel mount table.

  • df -hT: reports filesystem type and space usage.

  • df -i: reports inode usage, which can be exhausted before disk capacity.

Prefer a UUID or a stable device name such as /dev/disk/by-uuid/... in persistent configuration. Raw names such as /dev/sdb1 can change after reboot, hardware replacement, or enclosure reordering.

Persistent Configuration

/etc/fstab contains six fields:

UUID=... /mount-point filesystem-type options dump pass

For a non-root XFS filesystem, a typical entry is:

UUID=5f6a7c8d-9e0f-4a1b-8c2d-3e4f50617283 /srv/appdata xfs defaults,nofail 0 2
  • defaults: uses standard read-write filesystem options.

  • nofail: avoids blocking boot when the device is temporarily unavailable.

  • 0: disables traditional dump backups.

  • 2: allows filesystem checking during boot when required; root and swap require special values.

For network filesystems, add _netdev; for on-demand mounts, x-systemd.automount can be used. Services that require a specific mount should declare RequiresMountsFor=/srv/appdata in their systemd unit.

After editing /etc/fstab, reload systemd and verify the configuration:

sudo systemctl daemon-reload
sudo findmnt --verify --verbose
sudo mount /srv/appdata

Code Example

The following example mounts an existing XFS partition. It assumes /dev/sdb1 has already been prepared with a filesystem and contains no important data to overwrite.

# Identify the filesystem and its UUID
$ lsblk -f /dev/sdb1
NAME FSTYPE LABEL   UUID                                 MOUNTPOINTS
sdb1 xfs    appdata 5f6a7c8d-9e0f-4a1b-8c2d-3e4f50617283

$ sudo blkid /dev/sdb1
/dev/sdb1: LABEL="appdata" UUID="5f6a7c8d-9e0f-4a1b-8c2d-3e4f50617283" TYPE="xfs"

# Create the mount point and mount the filesystem
sudo mkdir -p /srv/appdata
sudo mount -t xfs /dev/sdb1 /srv/appdata

# Confirm the active mount and available space
$ findmnt /srv/appdata
TARGET       SOURCE    FSTYPE OPTIONS
/srv/appdata /dev/sdb1 xfs    rw,relatime,attr2,inode64,noquota

$ df -hT /srv/appdata
Filesystem     Type  Size  Used Avail Use% Mounted on
/dev/sdb1      xfs  100G   12G   88G  12% /srv/appdata

# Stop the application before unmounting
sudo systemctl stop app.service

# Detach the filesystem; successful unmounting normally prints nothing
sudo umount /srv/appdata

Failure Modes

  • Target is busy: A process has the directory open, uses it as its working directory, or holds a file descriptor. Identify processes with sudo fuser -vm /srv/appdata or sudo lsof +f -- /srv/appdata, then stop them properly.

  • Lazy unmount: sudo umount -l /srv/appdata detaches the mount immediately while deferred operations finish when references are released. Use it only when immediate detachment is required, commonly for stale network mounts.

  • Forced unmount: umount -f is primarily intended for unresponsive network filesystems. It should not be used to bypass normal local cleanup.

  • Repairing filesystems: Unmount the filesystem before running tools such as e2fsck or xfs_repair. Repairing a mounted filesystem can cause corruption.

  • Network filesystems: Hard mounts can block applications during network outages; soft mounts can silently corrupt data. Timeouts, retries, and monitoring must match the workload’s durability requirements.

  • I/O errors: Check journalctl -k, dmesg -T, disk health, and storage logs. A filesystem unexpectedly becoming read-only often indicates underlying storage problems.

Production Best Practices

  • Stop or drain application traffic before unmounting; never use kill -9 merely to release a busy mount.

  • Remove the /etc/fstab entry before permanently retiring a mount, then run systemctl daemon-reload.

  • Monitor df -hT, df -i, mount status, kernel logs, and service errors. Alert when expected mounts are absent.

  • Use snapshots or application-consistent backups rather than assuming an unmount is necessary for every backup.

  • Apply nosuid, nodev, and noexec when appropriate for untrusted data, but verify that applications do not require those features.

  • Never use chmod 777 as a mounting fix. Set ownership and permissions on the mounted filesystem itself, and preserve SELinux labels with restorecon where applicable.

  • For CIFS or similar filesystems, store credentials in a root-readable credentials file rather than directly in /etc/fstab.

47

What happens when /etc/fstab contains an invalid entry?

An invalid /etc/fstab entry is either rejected while the table is parsed or converted into a mount or swap unit that fails when systemd starts it. If the affected filesystem is required for root or normal boot, the system may stop in emergency or recovery mode instead of reaching its normal target.

Read full answer

Applies toLinux with systemd on Ubuntu 24.04 and RHEL 9

Purpose and Format

Each non-comment line in /etc/fstab contains six fields:

Field Example Purpose
Device LABEL=data Block device, label, UUID, or network source
Mount point /srv/data Directory where the filesystem appears
Filesystem type xfs Driver or filesystem type, such as ext4, xfs, btrfs, nfs4, or swap
Options defaults,noatime Mount and filesystem options
Dump 0 Historically used by dump; normally 0
Filesystem check pass 2 Controls fsck ordering; root is commonly 1, other checked filesystems 2, and skipped filesystems 0

Use stable identifiers such as UUID= or LABEL= rather than /dev/sda, because device letters can change after disk additions, enclosure changes, or reboots.

Boot and Mount Behavior

On systemd-based Linux, systemd-fstab-generator reads /etc/fstab during early boot and the following happens:

  1. Valid filesystem entries produce .mount units, while valid swap entries produce .swap units.

  2. systemd adds the normal mount dependencies, commonly involving local-fs.target.

  3. Filesystem checking runs where required before the filesystem is mounted.

  4. systemd starts each unit and applies its specified mount or swap options.

A malformed line is normally logged and skipped. A syntactically valid entry with an incorrect device, filesystem type, mount point, or option instead produces a unit that can fail. mount -a reads /etc/fstab and attempts its non-swap entries, while swapon --all manually enables all listed swap entries.

Root filesystem handling can occur inside the initramfs before the normal operating system starts. Consequently, an invalid root entry may lead to the distribution’s recovery or emergency environment rather than merely a failed mount unit.

Failure Modes

Problem Typical result
Malformed line or invalid field The generator logs a warning and skips the entry
Missing or incorrect UUID or label The generated unit fails because the device cannot be found
Incorrect filesystem type The mount fails, often reporting a wrong filesystem type
Unsupported mount option The mount fails with a bad-option error
Duplicate mount point An entry may be rejected or conflict with another unit
Missing removable device Boot may wait for the device before timing out or failing
Invalid root entry Boot may stop in emergency or recovery mode
Unavailable network filesystem Boot may be delayed until networking is available or the mount fails

The resulting unit remains visible as failed, for example as srv-data.mount. Relevant details are available through systemctl status srv-data.mount and journalctl -b -u srv-data.mount.

Production Handling

  • nofail: Allows boot to continue when a nonessential mount fails. It must not be used to pretend that the root filesystem is optional.

  • x-systemd.automount: Defers access until the mount point is used. This can reduce boot coupling, but the first access may incur mount latency and can still block an application.

  • _netdev: Places network filesystem activation after basic network setup. Network applications still need their own connection and I/O timeouts.

  • x-systemd.device-timeout=30s: Can bound device-availability waits, but it is not a universal timeout for every network mount operation.

  • Security: Add noexec, nosuid, and nodev where workloads permit them. Do not use chmod 777 or disable SELinux as a shortcut.

  • Operations: Monitor failed .mount units, expected filesystems that are absent, capacity thresholds, and storage errors. Keep /etc/fstab under configuration management and retain a bootable backup and console or out-of-band recovery access.

Safe Change Procedure

  1. Back up /etc/fstab before editing it.

  2. Obtain actual identifiers with lsblk -f or blkid.

  3. Run findmnt --verify --verbose to check the table for problems.

  4. Run systemctl daemon-reload so systemd regenerates units from the updated table.

  5. Test the affected unit directly; use mount -a only during an appropriate maintenance window because it may activate every non-swap entry.

  6. Review systemctl --failed and the relevant journal, then schedule production reboots only after rollback and recovery access are confirmed.

48

How do you create partitions and filesystems on a new Linux disk?

Partitioning divides a block device into separate areas, while creating a filesystem makes one partition usable for data. On Linux, this normally means creating a GPT partition table, running mkfs, mounting the partition, and adding a persistent /etc/fstab entry.

Read full answer

Applies toLinux (Ubuntu 22.04/24.04, RHEL 9; GPT partitioning with `parted` 3.x)

How Partitioning and Filesystems Differ

  • Partition table: GPT or MBR records partition boundaries. GPT supports large disks, more partitions, and better corruption resistance.

  • Partition: A range of blocks within the disk, such as /dev/sdb1.

  • Filesystem: The on-disk structures that organize files, directories, metadata, and allocation, such as ext4 or XFS.

  • Mount: Attaches a filesystem to a directory in the Linux virtual filesystem, such as /srv/app-data.

Creating a partition does not create a filesystem. Likewise, formatting or mounting a partition does not replace the disk’s partition table.

Production-Safe Workflow

  1. Identify the correct disk. Device names such as /dev/sdb can change between boots or attachment events. Check the model, serial, size, existing signatures, and mount state.

  2. Create the partition table and partition. Creating a GPT label overwrites existing partition metadata, so never run this against an unidentified disk.

  3. Create the filesystem. mkfs writes new filesystem metadata and can destroy existing data.

  4. Mount and verify. Test the mount before relying on it.

  5. Configure persistent mounting. Use a filesystem UUID in /etc/fstab, not a device name.

  6. Validate and monitor. Check df, inode usage, kernel logs, and storage-health telemetry.

Code Example

Install the required tools:

# Ubuntu
sudo apt update
sudo apt install parted e2fsprogs

# RHEL/Fedora
sudo dnf install parted e2fsprogs xfsprogs

Inspect the new disk before modifying it:

lsblk -o NAME,SIZE,MODEL,SERIAL,TYPE,FSTYPE,MOUNTPOINTS

Example output:

NAME  SIZE  MODEL         SERIAL     TYPE  FSTYPE MOUNTPOINTS
sda   20G   QEMU HARDDISK QM00001     disk
sdb  100G   QEMU HARDDISK QM00002     disk

Assume sdb has been positively identified as the new disk. Create a GPT table and one partition:

# mklabel creates GPT; mkpart creates the partition.
# 1MiB avoids placing data at the beginning of the disk.
sudo parted /dev/sdb --script \
  mklabel gpt \
  mkpart primary ext4 1MiB 100%

# Notify the kernel about the new partition table.
sudo partprobe /dev/sdb
sudo udevadm settle

lsblk -o NAME,SIZE,MODEL,SERIAL,TYPE,FSTYPE,MOUNTPOINTS

Example output:

NAME  SIZE  MODEL         SERIAL     TYPE  FSTYPE MOUNTPOINTS
sda   20G   QEMU HARDDISK QM00001     disk
sdb  100G   QEMU HARDDISK QM00002     disk
└─sdb1 100G QEMU HARDDISK            part

Create an ext4 filesystem and mount it:

# -L sets a human-readable filesystem label.
sudo mkfs.ext4 -L app-data /dev/sdb1

sudo mkdir -p /srv/app-data

UUID="$(sudo blkid -s UUID -o value /dev/sdb1)"
ENTRY="UUID=$UUID /srv/app-data ext4 defaults,nofail 0 2"

# Avoid adding a duplicate entry when the command is rerun.
sudo grep -qF "$ENTRY" /etc/fstab || \
  printf '%s\n' "$ENTRY" | sudo tee -a /etc/fstab

sudo findmnt --verify --verbose
sudo mount -a
findmnt /srv/app-data

Example output:

TARGET        SOURCE    FSTYPE OPTIONS
/srv/app-data /dev/sdb1 ext4   rw,relatime

The /etc/fstab fields mean filesystem source, mount point, filesystem type, mount options, dump flag, and filesystem check order. nofail prevents a missing non-root data disk from forcing an emergency boot; services that require the disk should still declare a dependency such as RequiresMountsFor=/srv/app-data.

Filesystem Choices

Filesystem Common Use Important Considerations
ext4 Ubuntu default, general-purpose data Journaling, inode-based allocation, online growth supported
XFS RHEL default, large data sets Excellent large-file performance; growing requires xfs_growfs; shrinking is not supported
Btrfs Snapshots and checksums Advanced features require filesystem-specific configuration and operational experience
swap Virtual memory Not a substitute for RAM or backups; use a dedicated partition with mkswap and swapon

For RHEL, create XFS with sudo mkfs.xfs /dev/sdb1. Select the filesystem according to the operating-system convention, storage requirements, backup tooling, and recovery procedures.

Operations and Failure Modes

  • Device identity: Prefer /dev/disk/by-id/ symlinks in scripts and automation; do not assume /dev/sdb remains stable.

  • Unmounted changes: Do not run mkfs on a mounted filesystem. Unmount it first and verify with findmnt and lsblk.

  • Capacity monitoring: Track both block usage and inode usage with df -h and df -i. A filesystem can have free blocks but no free inodes.

  • Health monitoring: Collect SMART data with smartctl -a /dev/sdb and monitor kernel messages with dmesg -T.

  • Backups: GPT, partitions, and filesystems are not backups. Use application-consistent snapshots or established backup tools and regularly test restores.

  • Growth: Reusing a partition boundary leaves little room for growth. Use LVM when predictable expansion is required.

  • Recovery: Keep partition and filesystem tools available in the recovery environment. fsck normally requires an unmounted filesystem, except for limited journal-replay scenarios.

  • Security: On RHEL, preserve correct SELinux labeling for mounted data and use suitable mount options such as nosuid and nodev when the application permits them. Never use chmod 777 to solve an access problem.

49

What are LVM’s storage layers and steps for growing a logical volume?

Logical Volume Management (LVM) is Linux’s flexible storage abstraction, allowing physical storage to be combined into volume groups and presented to filesystems as logical volumes. A logical volume can usually be extended without downtime when its volume group has free capacity and the filesystem supports online growth.

Read full answer

Applies toLinux LVM2 on Ubuntu 22.04/24.04 and RHEL 9

How LVM Works

LVM builds a hierarchy of block devices:

Layer Purpose
Physical Volume (PV) A disk or partition initialized with pvcreate and managed by LVM.
Volume Group (VG) Pool of physical extents shared by one or more logical volumes.
Logical Volume (LV) A virtual block device allocated from the volume group.
Filesystem Stores files on the logical volume, such as ext4 or xfs.

The Linux device-mapper layer exposes an LV through paths such as /dev/vg0/app and /dev/mapper/vg0-app. LVM metadata maps logical extents to physical extents. Extending a conventional linear LV normally allocates additional extents and updates this metadata rather than copying existing data.

Capacity Requirements

The volume group must have enough unallocated extents. If vgs shows insufficient VFree, first add another physical volume:

  1. Verify that the new device is an empty disk with lsblk and wipefs -n.

  2. Initialize it with pvcreate.

  3. Add it to the volume group with vgextend.

  4. Extend the logical volume.

Warning: pvcreate writes LVM metadata over the device. Using the wrong disk can destroy an existing filesystem or data.

Filesystem Growth

Increasing the LV’s block-device size does not automatically enlarge the filesystem. Both layers must be updated:

  • ext4: Run resize2fs against the logical volume. Modern ext4 filesystems can normally be resized while mounted.

  • XFS: Run xfs_growfs with the filesystem’s mount point. XFS cannot grow while unmounted.

  • Thin volumes: Extending a thin LV changes its visible size, but allocated data still comes from the thin pool. Monitor and extend the thin pool separately to prevent exhaustion.

Code Example

The following example assumes /dev/vg0/app uses ext4 and the volume group already has at least 20 GiB free:

# Inspect the storage hierarchy and available capacity
lsblk -o NAME,TYPE,SIZE,FSTYPE,MOUNTPOINTS,VG_NAME
sudo pvs
sudo vgs
sudo lvs

# Confirm the filesystem type before resizing
findmnt -n -o FSTYPE /srv/app

# Increase the logical volume by 20 GiB
# "+" means add 20 GiB; without it, -L specifies an absolute new size
sudo lvextend -L +20G /dev/vg0/app

# Grow the mounted ext4 filesystem online
sudo resize2fs /dev/vg0/app

# Verify the resulting filesystem capacity
df -h /srv/app

Representative output:

Before growth:

  VG   #PV #LV #SN Attr   VSize   VFree
  vg0    1   1  20 wz--n- 100.00g 60.00g

  LV   VG   Attr       LSize
  app  vg0  -wi-ao---- 40.00g

ext4

After growth:

Size of logical volume vg0/app changed from 40.00 GiB to 60.00 GiB.
Logical volume vg0/app now 60.00 GiB.
The filesystem on /dev/mapper/vg0-app is now 15728640 (4k) blocks long.

For an XFS filesystem, replace the resize2fs command with:

sudo xfs_growfs /srv/app

If the volume group has no free capacity, use this sequence only after verifying /dev/sdc is the intended empty device:

lsblk -o NAME,SIZE,FSTYPE,MODEL,SERIAL /dev/sdc
sudo wipefs -n /dev/sdc
sudo pvcreate /dev/sdc
sudo vgextend vg0 /dev/sdc
sudo lvextend -L +20G /dev/vg0/app
sudo resize2fs /dev/vg0/app

Production Best Practices and Failure Modes

  • Back up first: LVM snapshots provide fast rollback points but are not replacements for backups. Metadata backups created under /etc/lvm/backup also do not contain user data.

  • Check capacity first: vgs, lvs, and pvs expose volume-group free space, thin-pool utilization, and logical-volume attributes.

  • Monitor continuously: Alert on volume-group free extents, thin-pool Data%, filesystem utilization, inode usage, and storage latency.

  • Expect IO impact: Filesystem growth can consume CPU and IO, although the metadata operation itself is normally short.

  • Handle thin-pool exhaustion: A thin LV may report substantial free logical space while its pool is already full.

  • Preserve device identities: Prefer /dev/<vg>/<lv> or /dev/mapper/<vg>-<lv> over /dev/sd* names, which can change after reboot.

  • Plan upgrades: Keep LVM2 installed and upgrade it through apt on Ubuntu or dnf on RHEL. Do not remove storage packages while their volumes are mounted.

  • Validate the application: After growth, confirm that the service sees the new capacity and that monitoring reflects the increased filesystem size.

50

What is Linux swap and how do I add a swap file?

Swap is disk-backed virtual memory that Linux uses to hold less-active anonymous memory pages when physical RAM is under pressure. It can delay an out-of-memory kill, but it is much slower than RAM and is not a substitute for correctly sizing memory.

Read full answer

Applies toLinux (Ubuntu 22.04/24.04, RHEL 9) with util-linux 2.37+

How Linux Uses Swap

  • Virtual memory: Linux gives each process a virtual address space backed by RAM and swap.

  • Paging: Under memory pressure, the kernel moves anonymous pages from RAM to a swap area and reclaims the RAM pages.

  • Swappiness: vm.swappiness influences the kernel’s preference for swapping anonymous pages versus reclaiming file cache. Inspect it with sysctl vm.swappiness, but tune it according to workload rather than disabling it blindly.

  • OOM behavior: Swap may delay an OOM kill, but it does not correct memory leaks, incorrect container limits, or insufficient physical memory.

  • File-backed pages: Ordinary file cache is normally reclaimed directly. Private file mappings and tmpfs pages can be written to swap.

A swap file stores swap data in a file on a local filesystem; a swap partition or block device serves the same purpose. The commands below assume a local ext4 or XFS filesystem. Do not place swap on NFS because swap requires reliable block-device semantics and stable write ordering.

Code Example

This creates and activates a 1 GiB swap file at /swapfile. Choose the capacity from workload measurements and available disk space.

Save it as create-swap.sh and run it with sudo bash create-swap.sh:

#!/usr/bin/env bash
set -euo pipefail

swap_path=/swapfile
size_mib=1024

# Never overwrite an existing file or follow an existing symlink.
if [[ -e "$swap_path" || -L "$swap_path" ]]; then
  printf '%s\n' "$swap_path already exists; inspect it before continuing." >&2
  exit 1
fi

# Write a fully allocated file; this form also works on RHEL 9's XFS.
sudo dd if=/dev/zero of="$swap_path" bs=1M count="$size_mib" conv=fsync status=progress

# Restrict access because an active swap file may contain sensitive data.
sudo chmod 600 "$swap_path"

# Initialize and activate the swap area.
sudo mkswap "$swap_path"
sudo swapon "$swap_path"

# Activate it automatically during future boots.
if ! grep -qE "^${swap_path}[[:space:]]" /etc/fstab; then
  printf '%s none swap sw 0 0\n' "$swap_path" | sudo tee -a /etc/fstab
fi

swapon --show
free -h

Representative final output:

NAME      TYPE SIZE USED PRIO
/swapfile file   1Gi   0B   -2

               total        used        free      shared  buff/cache   available
Mem:             15Gi       3.2Gi       4.1Gi       180Mi       7.7Gi       11Gi
Swap:           1.0Gi          0B       1.0Gi
  • dd bs=1M count=1024: Writes 1,024 MiB; conv=fsync flushes the file before continuing, and status=progress displays progress.

  • chmod 600: Restricts the file to root because swap can expose process memory.

  • mkswap: Writes a Linux swap signature. Never run it on a device containing a filesystem or important data: it invalidates the previous filesystem, and remaining old blocks can become swap contents.

  • swapon: Activates the initialized area immediately.

  • /etc/fstab: The fields are the file path, none, filesystem type swap, options sw, no dump flag, and no filesystem-check flag. The systemd fstab generator activates persistent swap during boot.

On Ubuntu’s ext4 filesystem, fallocate -l 1G /swapfile is faster because it reserves blocks without writing every block, although it does not guarantee physically contiguous allocation. Use the dd sequence above on RHEL 9’s XFS, where fallocate may leave unwritten extents that mkswap rejects.

Monitoring and Failure Modes

  • Activation failure: Check swapon --show and kernel logs. Common causes include insufficient privileges, an already active area, an unsupported filesystem, or a corrupted swap signature.

  • Live monitoring: Use vmstat 1 and inspect si and so, which report pages swapped in and out. If sysstat is installed—sudo apt install sysstat on Ubuntu or sudo dnf install sysstat on RHEL—use sar -W 1 and iostat -x 1 for deeper analysis.

  • OOM incidents: Review kernel messages with sudo journalctl -k | grep -Ei 'oom|out of memory|killed process'.

  • Removal: Run sudo swapoff /swapfile, remove its /etc/fstab entry, and then delete the file. swapoff can fail if there is insufficient free memory to restore the swapped pages.

  • Capacity checks: Monitor filesystem utilization, major page faults, and sustained swap activity. These can indicate memory pressure or an application-sizing problem.

Security and Performance

  • Swap files can contain credentials, tokens, encryption keys, and other memory contents. Keep them at mode 0600 and use encrypted swap through dm-crypt when the threat model requires protection from offline disk access.

  • Swap I/O can sharply increase request latency and disk saturation. Sustained si/so activity should trigger memory-capacity and application investigations.

  • Use reliable local storage with sufficient free space. Avoid placing swap on the same heavily contended device as latency-sensitive databases when alternatives exist.

  • Swap is temporary backing storage, not a backup mechanism.

Production Best Practices

  • Size swap from workload evidence, memory limits, peak usage, and recovery requirements rather than applying a universal rule.

  • Keep persistence in /etc/fstab and verify activation with sudo swapon --show after maintenance or reboot.

  • Alert on sustained swap activity, OOM kills, abnormal major page faults, and disk saturation.

  • Test the procedure in staging and maintain rollback steps. Creating a swap file alone does not configure hibernation or a valid resume device.

51

How can Linux memory usage be inspected using free and /proc/meminfo?

Use free -h for a quick system-wide view and /proc/meminfo for kernel-level counters. The most useful indicator of allocatable capacity is MemAvailable, not MemFree, because cached and other reclaimable memory is distinct from unused memory.

Read full answer

Applies toLinux (Ubuntu 22.04/24.04, RHEL 9) with procps-ng 3.3+ and systemd 249+

How free and /proc/meminfo Work

  • free is a procps-ng utility that reads and formats the kernel memory counters exposed through /proc/meminfo.

  • /proc/meminfo is a kernel-provided, read-only view of global memory statistics.

  • Its values use kB, meaning 1024-byte units despite the label.

  • Neither tool identifies the process responsible for a change or retains history. Use vmstat, sar, process metrics, and cgroup metrics for that.

Reading the Main Metrics

Metric Meaning
MemTotal Total usable RAM
MemFree Currently unused pages, excluding buffers, cache, and much reclaimable slab
MemAvailable Estimated capacity for new work without immediate swapping
Buffers Memory used for relatively small block-I/O buffers
Cached Page cache for recently used file data, including tmpfs/shmem accounting but excluding SwapCached
SReclaimable Reclaimable slab memory included in free’s cache accounting
buff/cache free’s combined buffers, cached, and reclaimable-slab view
SwapTotal / SwapFree Configured and unused swap space
SwapCached Pages that were swapped out and are now in RAM while still represented in swap
Dirty Modified pages not yet written back
Writeback Pages currently being written to storage

MemAvailable is a kernel estimate based on reclaimability and system reservations. It is generally more useful than MemFree. Likewise, free’s used column is aggregate accounting; it is not the sum of process RSS values and does not represent application memory alone.

Diagnosing Memory Pressure

Use vmstat 1 5 to examine paging and I/O wait over time. Its first report is since boot; subsequent si and so fields are per-second pages swapped in and out. wa is I/O-wait time, while b counts blocked tasks, which can include tasks waiting for page-in.

Use ps with RSS to find large processes, sar -r for historical memory statistics, pidstat -h for current process memory utilization, and pidstat -r for paging activity. Sustained swap activity, low MemAvailable, writeback backlog, and OOM kills are stronger incident signals than a single low MemFree reading.

Code Example

# Human-readable system summary
free -h

# Selected kernel counters
grep -E '^(MemTotal|MemFree|MemAvailable|Buffers|Cached|SReclaimable|SwapTotal|SwapFree|SwapCached|Dirty|Writeback):' /proc/meminfo

Representative output:

total        used        free      shared  buff/cache   available
Mem:             15Gi       6.2Gi       1.0Gi       180Mi       7.8Gi       7.5Gi
Swap:           2.0Gi          0B       2.0Gi

MemTotal:       15728640 kB
MemFree:         1048576 kB
MemAvailable:    7864320 kB
Buffers:          253952 kB
Cached:          7471104 kB
SReclaimable:     450560 kB
SwapTotal:       2097152 kB
SwapFree:        2097152 kB
SwapCached:            0 kB
Dirty:             12288 kB
Writeback:             0 kB

In this sample, used follows procps-ng 3.3.x (total - free - buff/cache, as on Ubuntu 22.04 and RHEL 9). procps-ng 4.x (Ubuntu 24.04) reports used as total - available, which would show about 7.5Gi here.

The relevant free options are:

  • -h: Uses human-readable IEC units such as MiB and GiB.

  • Default: Reports KiB.

  • -m: Reports MiB.

  • -s SECONDS: Repeats the display at the specified interval.

For live and historical diagnosis:

# Paging and I/O indicators
vmstat 1 5

# Largest processes by resident set size
ps -eo pid,comm,rss,vsz,%mem --sort=-rss | head -n 6

Representative one-second vmstat row:

2  0      0  1048576   253952  7471104    0    0     7    16    0    1   99    0    0

Representative process output:

PID COMMAND        RSS      VSZ    %MEM
   1421 postgres    1254368  2508736   8.0
    903 redis-server  524288  1048576   3.3
   2214 nginx        262144   524288   1.7
      1 systemd      125000   250000   0.8

Search recent kernel messages for OOM evidence:

sudo journalctl -k --since "-30 min" -i -g 'out of memory|oom-kill|killed process'

Example:

Aug 24 10:15:01 example-host kernel: Memory cgroup out of memory: Killed process 1421 (postgres) total-vm:2508736kB,...

Production Practices and Pitfalls

  • Alert on sustained pressure: Combine low MemAvailable, sustained swapping, OOM kills, and cgroup limit exhaustion.

  • Account for containers: free and /proc/meminfo commonly show host-wide memory. Under cgroup v2, inspect memory.current, memory.max, memory.events, and memory.stat in the container’s cgroup.

  • Do not disable swap reflexively: swapoff can cause severe pressure or an OOM kill when RAM cannot absorb the working set. Swap is not a backup and provides no durability.

  • Know package differences: Ubuntu uses procps and sysstat through apt; RHEL/Fedora use procps-ng and sysstat through dnf.

  • Protect operational data: Process listings and command-line arguments can expose credentials or other sensitive information; restrict them appropriately.

  • Correlate incidents: Compare historical behavior across deployments, package or kernel upgrades, and traffic spikes rather than diagnosing from one snapshot.

52

What do `buff/cache` and `available` mean in the Linux `free` command?

The buff/cache column combines kernel buffer, page-cache, and reclaimable slab memory, while available estimates how much memory can be allocated to new workloads without immediate swapping. A large buff/cache value is usually normal; available is generally more useful operationally than the free column alone.

Read full answer

Applies toLinux 5.15/6.x on Ubuntu 22.04/24.04 and RHEL 9/Fedora, using procps/procps-ng 3.3.12 or newer

What buff/cache Contains

Modern free combines these kernel memory counters:

  • Buffers: Memory used for block-I/O buffering. This category is usually small on modern systems because filesystems rely primarily on the page cache.

  • Cached: Page cache containing file data read from storage. Clean pages can normally be reclaimed under memory pressure.

  • SReclaimable: Reclaimable slab memory, including kernel objects such as inodes and directory entries.

This memory is not necessarily wasted. Keeping frequently accessed files in the page cache reduces subsequent disk or network-storage latency. Dirty cached pages must be written back to storage before they can be reclaimed.

What available Means

available is based on the kernel’s MemAvailable value exposed through /proc/meminfo. It estimates the memory available for starting new applications without swapping.

The estimate considers:

  • Currently unused memory

  • Reclaimable page-cache and slab memory

  • Kernel memory reservations and low-memory watermarks that should not be consumed

The exact kernel calculation can change between kernel versions. Consequently, available is not simply free + buff/cache, and it is not a precise guarantee that a specific allocation will succeed without swapping.

Reading the Output

The free -h command reports memory using human-readable units.

Column Meaning
free Memory currently unallocated
buff/cache Kernel buffers, page cache, and reclaimable slab memory
available Estimated memory available for new workloads without immediate swapping
used With procps/procps-ng 3.3.12+, calculated as total memory minus available; it is not simply total minus free and buff/cache

The underlying kernel counters can be inspected with inline tools such as grep -E '^(MemFree|MemAvailable|Buffers|Cached|SReclaimable):' /proc/meminfo.

Production Interpretation

  • Low free with high available: Usually healthy. Linux intentionally uses otherwise idle memory for cache.

  • High buff/cache: Common after file-heavy workloads and generally not a reason for intervention.

  • Low available: The system may struggle to allocate memory, especially if swap activity and out-of-memory events are also increasing.

  • High swap usage alone: Not conclusive. Use vmstat 1 5 to inspect ongoing si and so activity, and check kernel logs for OOM events.

  • Containers: free normally reports host memory from /proc/meminfo, not a container’s cgroup limit. For cgroup v2 workloads, monitor memory.current, memory.stat, and memory.events.

Common Pitfalls

  • Do not clear caches merely because buff/cache is high; the kernel reclaims cache when necessary.

  • Avoid using echo 3 > /proc/sys/vm/drop_caches as routine troubleshooting. It is intrusive, temporarily reduces performance, and does not solve underlying memory pressure.

  • Treat available as an estimate and correlate it with pressure-stall information, swap activity, OOM logs, and application latency.

  • If free is missing, install it with sudo apt install procps on Ubuntu or sudo dnf install procps-ng on RHEL/Fedora.

53

How do you interpret Linux load average?

Linux load average is a host-wide, exponentially damped indicator of recent runnable and uninterruptible scheduling entities. Interpret its 1-, 5-, and 15-minute values against available logical CPUs and alongside run-queue length, CPU utilization, I/O wait, and service latency; they are not CPU utilization percentages.

Read full answer

Applies toLinux (Ubuntu 22.04/24.04, RHEL 9)

What Load Average Measures

The Linux kernel averages scheduling entities that are running or runnable and tasks in uninterruptible sleep:

  • Runnable or running: Executing on a CPU or ready to execute.

  • Uninterruptible: Commonly shown as D state and often waiting for storage or a network filesystem, although it can also represent other kernel waits.

The values are exponentially damped averages with approximately 1-, 5-, and 15-minute time constants. They are historical indicators, not instantaneous queue lengths or counts of completed work.

Load can exceed the CPU count because runnable entities and uninterruptible tasks are counted together. A high value therefore requires additional context to distinguish CPU saturation from blocked I/O.

Interpreting the Values

Observation Likely interpretation
Load 1.00 on 4 CPUs Usually low scheduler pressure; it represents one runnable-or-uninterruptible entity on average
Load 4.00 on 4 CPUs Approximately full CPU capacity if the entities are runnable and CPU-bound
Load 8.00 on 4 CPUs Significant queueing or blocked-task pressure
1-minute load much higher than 15-minute load A recent load increase; determine whether it persists or affects service latency
All three averages steadily increasing Sustained rising pressure; compare it with available capacity
High load with low CPU usage Uninterruptible tasks probably dominate; investigate storage, filesystems, and kernel waits
Low load with poor service latency Investigate dependencies, application locks, request behavior, and unhealthy processes

Normalized load is load divided by available logical CPUs. For example, load 4.12 on 4 CPUs has normalized load 1.03, but this does not imply CPU saturation if many entities are blocked.

Reading Runtime Signals

With vmstat:

  • r: Processes running or waiting for the run queue.

  • b: Processes in uninterruptible wait, commonly blocked on I/O.

  • us and sy: User and system CPU time.

  • id: Idle CPU time.

  • wa: Time the CPUs were idle while disk I/O was outstanding; it is not a direct latency measurement.

  • st: CPU time stolen by the hypervisor in a virtualized environment.

Find uninterruptible tasks with ps -eo pid,stat,wchan:32,comm. A task remaining in D state can indicate a slow or unavailable filesystem. SIGKILL cannot complete until the uninterruptible kernel wait returns.

Production Failure Modes

  • CPU saturation: A large run queue, low idle time, and high combined user and system CPU.

  • Storage bottleneck: Blocked tasks and I/O wait increase alongside disk latency and queue depth.

  • Network filesystem latency: Tasks remain in D state while accessing NFS or another remote filesystem.

  • Kernel contention: High system CPU and run-queue pressure can indicate contention on shared kernel resources; sleeping mutex waiters alone may not contribute to load.

  • Virtualization contention: High steal time indicates that the hypervisor is taking CPU capacity from the guest.

  • Container ambiguity: Load averages are normally host-wide, so use cgroup metrics to attribute pressure to a particular container.

Code Example

# Logical CPUs available to this process
nproc
# 4

# Host-wide 1-, 5-, and 15-minute load averages
uptime
# 14:32:08 up 42 days,  3:11,  1 user, load average: 4.12, 3.87, 2.94

# Raw averages; field 4 is runnable/total scheduling entities, field 5 is the last PID created
cat /proc/loadavg
# 4.12 3.87 2.94 5/734 28419

# Scheduler queues and CPU/I/O counters
vmstat 1 3
# procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu-----
# r  b  swpd  free  buff  cache   si   so   bi   bo   in   cs us sy id wa st
# 4  2     0  1800   120  24500    0    0   18    0  120  380 85  5  2  8  0
# 5  3     0  1750   120  24400    0    0   24    0  140  420 91  4  1  4  0

For vmstat 1 3, the first report covers activity since boot; the following reports are one-second interval samples.

Operational Best Practices

  • Monitor normalized load, run-queue length, CPU saturation, I/O latency, and service-level latency together.

  • Avoid universal thresholds such as “load above 4 is bad”; compare load with usable CPU capacity and workload behavior.

  • Correlate changes with journalctl, application metrics, deployments, and dependency health.

  • Use iostat -xz, filesystem metrics, and disk queue depth to identify storage pressure.

  • Identify the bottleneck before scaling; more replicas can overload a shared database or storage system.

  • Retain time-series data so the 1-, 5-, and 15-minute trends show whether pressure is transient, sustained, or worsening.

54

What steps diagnose CPU saturation on a Linux server?

CPU saturation is sustained demand exhausting or nearly exhausting a server’s aggregate logical-CPU capacity, increasing run-queue delay and request latency. Diagnose it by distinguishing aggregate saturation from a single hot CPU and separating application work from kernel overhead, interrupts, I/O wait, hypervisor steal, and cgroup throttling.

Read full answer

Applies toLinux (Ubuntu 22.04/24.04, RHEL 9, Fedora 42); procps-ng 3.x/4.x, sysstat 12.x/13.x, systemd 249/252/255/257

Triage Workflow

  1. Confirm the symptom and baseline

  2. Determine whether one host, service, container, or cluster is affected.

  3. Correlate CPU usage with latency, errors, queue growth, traffic, and deployment activity.

  4. Compare against normal load; sustained high CPU is not automatically an incident.

  5. Separate CPU saturation from load average

  6. Linux load average includes runnable and uninterruptibly blocked tasks. High load with low CPU use and a short run queue often indicates D-state tasks rather than CPU pressure.

  7. Use vmstat, mpstat, and pidstat instead of relying on one top snapshot.

  8. Identify the responsible workload

  9. Check processes, threads, systemd units, containers, and cgroup limits.

  10. Consider traffic spikes, retry storms, garbage collection, expensive queries, infinite loops, batch jobs, and hypervisor contention.

Interpret Host-Level Signals

Signal Interpretation Investigate
High %usr User-space code is consuming CPU Application threads, GC, compression, serialization
High %sys Kernel or syscall overhead Networking, filesystem activity, excessive syscalls, tracing
High %iowait CPU was idle while I/O was outstanding Slow disks, NFS, storage timeouts, fsync calls
High %steal Hypervisor reclaimed CPU time Contention or overcommit on the physical host
Sustained run queue with little idle CPU More runnable work than available CPU Thread-pool sizing, admission control, hot loops
High b Tasks blocked in uninterruptible sleep Blocked storage, NFS, or other kernel waits
Rising nr_throttled A cgroup exhausted its CPU quota Container or service CPU limits

vmstat 1 5 uses one-second intervals and five reports; the first is a since-boot summary. mpstat -P ALL 1 5 uses -P ALL for every logical CPU. iostat -x 1 5 uses -x for extended disk statistics and five one-second samples.

Attribute Work to Processes and Services

ps calculates %CPU over the process lifetime, so a recently started process can have a misleading value. pidstat samples current activity and is usually better during an incident. Use pidstat -t -u 1 5 to include individual threads.

Systemd services run in cgroups, so host-wide metrics can hide a service-specific quota. On cgroup v2, inspect cpu.stat for usage and throttling. For containers, use docker stats --no-stream for a one-time snapshot or kubectl top pod; the latter requires the Kubernetes Metrics API, normally supplied by Metrics Server.

Code Example

Install the standard diagnostics if necessary:

# Ubuntu
sudo apt-get install -y procps sysstat

# RHEL 9 or Fedora
sudo dnf install -y procps-ng sysstat

Collect process, scheduler, and CPU samples:

# Rank processes by lifetime-average CPU utilization
ps -eo pid,ppid,user,%cpu,%mem,etime,stat,comm \
  --sort=-%cpu | head -n 11

# Per-process CPU: one-second interval, five samples
pidstat -u 1 5

# Include individual-thread activity
pidstat -t -u 1 5

# Run queue, blocked tasks, CPU classes, and context switches
vmstat 1 5

# Utilization of every logical CPU
mpstat -P ALL 1 5

# Extended disk statistics
iostat -x 1 5

Representative pidstat output:

Linux 6.8.0-45-generic (ubuntu)  06/28/24  _x86_64_  (2 CPU)

02:15:10 PM   UID       PID    %usr %system  %guest   %wait    %CPU   CPU  Command
02:15:11 PM  1001      2450    92.0    4.0    0.0    0.8    96.0    0  java
02:15:12 PM  1001      2450    95.0    3.0    0.0    0.5    98.0    0  java

Inspect the affected service and its cgroup:

# Replace api.service with the affected unit
systemctl status api.service --no-pager -l
journalctl -u api.service --since "30 min ago" --no-pager

# Batch mode, one iteration
systemd-cgtop -b -n 1

CGROUP=$(systemctl show api.service -p ControlGroup --value)
sudo cat "/sys/fs/cgroup${CGROUP}/cpu.stat"

Representative cgroup v2 output:

usage_usec 918273441
user_usec 812345678
system_usec 105927763
nr_usec 918273441
nr_throttled 1247
throttled_usec 89210000

Profile a process for 30 seconds:

PID=1234
sudo perf record -F 99 -g -p "$PID" -o /tmp/perf.data -- sleep 30
sudo perf report --stdio -i /tmp/perf.data

Here, -F 99 selects a 99 Hz sampling frequency, -g records call graphs, and -p selects the process. Profiling may require matching kernel symbols and suitable perf_event_paranoid or CAP_PERFMON permissions.

Production Practices and Pitfalls

  • Capture pidstat, vmstat, service logs, deployment markers, and cgroup counters before restarting; a restart can destroy the evidence.

  • Correlate CPU with request rate, queue depth, latency, GC pauses, database latency, and retry counts.

  • Use perf, strace, and eBPF carefully during critical incidents because they add overhead and may expose sensitive data.

  • Treat renice, cgroup quota changes, concurrency limits, and load shedding as temporary containment, not fixes for retry storms or inefficient code.

  • Monitor sustained saturation and throttling continuously, and base alerts on service SLOs rather than short CPU bursts.

55

How do you troubleshoot high memory usage and OOM kills on Linux?

High memory usage is not automatically a memory leak. An OOM kill occurs when the kernel cannot satisfy an allocation, either globally or within a constrained cgroup. Troubleshooting must distinguish reclaimable cache, process growth, swap pressure, cgroup limits, and genuine leaks.

Read full answer

Applies toLinux 5.14+ with systemd cgroup v2; Ubuntu 22.04/24.04 and RHEL 9

Troubleshooting Workflow

  1. Confirm the scope

  2. Check host memory, load, and swap.

  3. Determine whether the host, one service, or one process is affected.

  4. Correlate the incident with deployments, traffic changes, or configuration changes.

  5. Identify consumers

  6. Use ps to find processes with growing resident set size (RSS).

  7. Do not compare virtual size (VSZ) directly with physical memory; VSZ includes address space and mappings that may never be resident.

  8. Compare samples over time after traffic returns to baseline.

  9. Find kernel evidence

  10. Search for Out of memory, oom-kill, and Killed process.

  11. Check both the current and previous boot where journal history is available.

  12. A cgroup limit can trigger a kill while the host still has free memory.

  13. Inspect cgroup accounting

  14. Compare memory.current with memory.max.

  15. Check memory.events for limit and OOM counters.

  16. Use memory.stat to distinguish anonymous memory from file-backed cache.

  17. Check pressure and swap

  18. Sustained swap-in and swap-out indicates memory pressure and possible latency.

  19. Use PSI files such as /proc/pressure/memory to identify stalls even before an OOM occurs.

Reading the Evidence

Signal Meaning Operational caveat
free available Memory available for new allocations without immediate swapping More useful than the free column because Linux uses idle memory for cache
RSS Physical pages currently held by a process Useful for process-growth analysis, but shared pages may be counted in multiple processes
VSZ Total virtual address space Can greatly exceed physical usage and is not proof of a leak
anon Heap, stacks, and other anonymous pages Generally cannot be discarded without swapping or reclaiming the process
file Page cache and mapped files Usually reclaimable, although dirty pages may first require writeback
si and so in vmstat Swap-in and swap-out rates Heavy paging can turn an OOM incident into a latency incident

Evidence of a leak includes continuously increasing RSS or anonymous memory after workload and traffic return to baseline without a corresponding cache explanation.

Cgroup and Service Limits

In a unified cgroup v2 hierarchy, systemd exposes per-service memory accounting. memory.max is the hard limit; after reclaim fails, the cgroup’s OOM mechanism can kill a process. memory.high provides earlier throttling and reclaim pressure, while memory.low and memory.min can provide protection.

  • memory.current: Memory currently charged to the cgroup.

  • memory.events: Event counters such as max, oom, oom_kill, and oom_group_kill.

  • memory.stat: Detailed counters including anon, file, kernel_stack, and slab.

  • memory.swap.current: Cgroup swap usage when swap accounting is available.

systemd-oomd is a separate proactive userspace mechanism based on memory pressure. It is not the same as the kernel OOM killer; inspect journalctl -u systemd-oomd when it may have terminated a managed unit.

Code Example

Start with host and process evidence:

free -h
vmstat 1 5
ps -eo pid,ppid,user,%mem,rss,vsz,stat,comm --sort=-rss | head -n 11
sudo journalctl -k -g 'Out of memory|oom-kill|Killed process'
sudo journalctl -k -b -1 -g 'Out of memory|oom-kill|Killed process'
swapon --show

Representative output from free and ps:

total        used        free      shared  buff/cache   available
Mem:            15Gi       12Gi       1.2Gi        180Mi       2.1Gi       2.7Gi
Swap:           4.0Gi       512Mi       3.5Gi

  PID  PPID USER     %MEM     RSS     VSZ STAT COMMAND
 2841     1 java      42% 6553600 8257536 S   java
 1192     1 root       4%  655360 1258291 S   containerd

vmstat 1 5 takes five one-second samples; inspect si, so, and wa. ps reports RSS and VSZ in KiB. journalctl -k reads kernel messages, and -g applies a regular-expression filter.

Inspect an affected service’s cgroup:

unit=nginx.service                 # Replace with the affected unit
cg=$(systemctl show "$unit" -p ControlGroup --value)
if [[ -z "$cg" || ! -d "/sys/fs/cgroup${cg}" ]]; then
    printf 'No active cgroup for %s\n' "$unit" >&2
    exit 1
fi
base="/sys/fs/cgroup${cg}"

printf 'ControlGroup: %s\n' "$cg"
printf 'Current: '; cat "$base/memory.current"
printf 'Limit:    '; cat "$base/memory.max"
printf 'Swap:     '; cat "$base/memory.swap.current"
printf '\nEvents:\n'; cat "$base/memory.events"
printf '\nSelected statistics:\n'
grep -E '^(anon|file|kernel_stack|slab) ' "$base/memory.stat"

Example output:

ControlGroup: /system.slice/nginx.service
Current: 1073741824
Limit:    2147483648
Swap:     0
Events:
low 0
high 0
max 512
oom 3
oom_kill 2
oom_group_kill 0
Selected statistics:
anon 805306368
file 201326592
kernel_stack 131072
slab 33554432

Production Practices

  • Monitor trends and counters: Track process RSS, cgroup memory.current, memory.events, PSI, swap activity, and OOM kills. Alert on counter increases rather than only a percentage of the limit.

  • Set measured limits: Leave headroom for replicas, traffic spikes, and temporary allocations. Do not set memory.max equal to steady-state usage.

  • Preserve evidence: Before restarting an unhealthy service, capture relevant logs, deployment metadata, and, where appropriate, a heap or allocation profile.

  • Use swap cautiously: Swap can delay an OOM but does not fix unbounded growth and can cause severe latency.

  • Review victim selection: Do not casually raise oom_score_adj; it can make an important process more likely to be selected.

  • Plan upgrades safely: Compare memory behavior before kernel, systemd, runtime, or JVM upgrades. Use apt on Ubuntu and dnf on RHEL.

  • Protect diagnostics: Restrict access to /proc, process arguments, and heap dumps because they may contain credentials or other sensitive data.

56

How do you troubleshoot slow Linux disk I/O with iostat and iotop?

Use iostat to identify the slow block device and measure throughput, request latency, and queueing, then use iotop to identify the processes generating I/O. Correlate both tools with application latency, system metrics, kernel logs, and storage-provider telemetry before changing the system.

Read full answer

Applies toLinux (Ubuntu 22.04/24.04, RHEL 9, Fedora) with sysstat 12.x and iotop (0.6 on Ubuntu, 1.x on Fedora/EPEL)

Troubleshooting Workflow

  1. Confirm the impact: Establish whether users, databases, backups, or builds are experiencing elevated latency.

  2. Measure the device: Sample iostat over multiple intervals to distinguish sustained congestion from a transient spike.

  3. Attribute the activity: Run iotop with root privileges to find the responsible processes.

  4. Correlate the evidence: Check CPU pressure, swapping, filesystem utilization, kernel errors, and hypervisor or cloud metrics.

  5. Test the hypothesis: Stop, throttle, reschedule, or move a confirmed workload, then verify that end-to-end latency improves.

Reading iostat Results

iostat reads Linux block-device counters from /proc/diskstats and calculates rates and averages between samples. Important extended columns include:

  • r/s, w/s: Completed read and write operations per second.

  • rkB/s, wkB/s: Read and write throughput in kilobytes per second.

  • rrqm/s, wrqm/s: Read and write requests merged per second.

  • %rrqm, %wrqm: Percentage of reads and writes that were merged.

  • r_await, w_await: Average read and write request latency in milliseconds.

  • aqu-sz: Average number of requests queued for the device.

  • rareq-sz: Average size of read requests in kilobytes.

  • %util: Sysstat’s device-utilization percentage; it is not a reliable saturation metric by itself for parallel NVMe devices.

Observation Likely interpretation
High aqu-sz and high r_await/w_await Requests are queueing behind device congestion
High %util, high throughput, and rising latency The device may be approaching its practical limit
Low throughput with high latency Device, filesystem, or virtualization latency may dominate
High r_await with low aqu-sz Per-request latency may dominate rather than queue depth
Healthy device metrics with slow application Investigate CPU, locks, memory, network, or application behavior

Compare sustained samples with a known baseline. A single busy interval may represent a checkpoint, backup, log rotation, or writeback burst.

Reading iotop Results

iotop samples per-process I/O counters, primarily from /proc/<pid>/io, and displays reads and writes by process. Look for sustained database writes, backup traffic, cache warm-up, reporting scans, or logging amplification.

iotop is best for interactive investigation. For production collection that is easier to parse and retain, use pidstat -d 1 from sysstat.

Process totals do not necessarily match device totals from iostat. Page-cache effects, delayed writeback, kernel I/O, and attribution timing can produce differences, so use iotop to identify candidates rather than as the sole physical-device accounting source.

Correlating System and Storage Evidence

Use vmstat to check CPU wait, run-queue depth, blocked tasks, and swapping. Use df -h for human-readable filesystem capacity and df -i for inode usage. lsblk lists block devices, findmnt shows mount relationships, and journalctl -k exposes kernel I/O and filesystem errors.

For physical or virtual storage, also inspect SMART or NVMe health, controller errors, and cloud-provider metrics. In cloud environments, the guest normally sees a virtual disk, so underlying latency or throttling may be visible only through provider monitoring.

Code Example

Install the tools using the package manager for the distribution:

# Ubuntu 22.04/24.04
sudo apt-get update
sudo apt-get install -y sysstat iotop
# Fedora, run as root
dnf install -y sysstat iotop
# RHEL 9, run as root; iotop is supplied by EPEL
dnf install -y epel-release
dnf install -y sysstat iotop

The package-manager -y option accepts prompts automatically. Capture a device baseline and inspect active processes:

# Extended statistics, no boot-to-date report, hide inactive devices,
# then sample every second for five reports.
iostat -y -xz 1 5

# Show only processes currently doing I/O; run as root.
sudo iotop -o

Illustrative iostat output:

Device            r/s     w/s     rkB/s    wkB/s   rrqm/s   wrqm/s  %rrqm  %wrqm  r_await   w_await  aqu-sz rareq-sz   %util
nvme0n1         12.00   85.33   1536.00   6553.60      0.00      1.20   0.00   1.39      3.82     22.15    1.41  256.00   98.67
sda              0.20    0.40      8.00     16.00      0.00      0.00   0.00   0.00      1.10      2.30    0.02   80.00    0.50

Here, nvme0n1 has sustained utilization, queueing, and elevated write latency, while sda is nearly idle. Correlate this with an illustrative iotop snapshot:

Total DISK READ: 8.50 M/s | Total DISK WRITE: 42.10 M/s
PID    USER       DISK READ       DISK WRITE  COMMAND
1841   postgres      1.20 M/s         38.40 M/s  postgres
2931   systemd       64.00 k/s          3.10 M/s  systemd-journal

Production Practices and Pitfalls

  • Sample over several intervals and compare against a baseline; one busy second is not sufficient evidence.

  • Monitor whole devices rather than counting both parent disks and partitions, which can double-count activity.

  • Investigate inode exhaustion and filesystem errors; metadata bottlenecks can resemble device latency.

  • Avoid unnecessary scans against a struggling device. Prefer smartctl, NVMe health commands, and provider diagnostics.

  • Treat iotop as sensitive because process names, users, and command details may reveal application internals.

  • Alert on sustained latency, queueing, and application impact rather than a transient %util spike.

  • If the device is saturated, throttle or reschedule the confirmed workload rather than repeatedly restarting the affected service.

57

How can strace be used to debug a process?

strace is a command-line utility that traces system calls and signal-related events made by a process, providing a detailed view of its interaction with the kernel. It is useful for diagnosing failed file access, unexpected networking, blocking operations, crashes, and performance overhead.

Read full answer

Applies tostrace 5.16–6.x on Linux (Ubuntu 22.04/24.04, RHEL 9/10, Fedora)

How strace works

strace uses the kernel’s ptrace mechanism. The kernel stops the traced process on system-call entry and exit, allowing strace to decode the call arguments and return values before resuming it. Signal deliveries can also be displayed. This does not expose every kernel interaction, scheduler activity, or process state transition.

Installing strace

On Ubuntu or Debian:

sudo apt update
sudo apt install strace

On RHEL or Fedora:

sudo dnf install strace

strace is available in the standard repositories but is not normally installed on minimal production hosts.

Common options

Option Description
-p PID Attach to a running process with the given PID.
-f Also trace child processes created by the traced process.
-e trace=SET Trace only selected system calls or classes, such as file, network, process, or signal.
-c Print a summary of call counts, errors, and time spent when the program exits.
-o FILE Write trace output to FILE instead of standard error.
-s SIZE Print up to SIZE bytes of string arguments; the default is 32.
-tt Prefix each line with a wall-clock timestamp.
-T Show the duration of each system call.
-E VAR=VAL Set an environment variable for the traced program.

Code Example

Trace a new program and save the syscall trace:

strace -f -o trace.log ./my_program
tail -20 trace.log

A representative excerpt is:

openat(AT_FDCWD, "/etc/hostname", O_RDONLY) = 3
read(3, "myhost\n", 1024)       = 9
write(1, "myhost\n", 9)         = 9
close(3)                        = 0

Attach to an existing process and filter for file operations:

strace -p 1234 -e trace=file

Possible output includes:

openat(AT_FDCWD, "/var/log/app.log", O_WRONLY|O_CREAT|O_APPEND, 0644) = 5
write(5, "2025-01-15 10:23:01 Starting\n", 37) = 37

Produce a system-call summary:

strace -c ./my_program

A representative summary is:

% time     seconds  usecs/call     calls    errors syscall
------ ----------- ----------- --------- --------- ----------------
 95.00    0.000950           95         10         0 openat
  5.00    0.000050           50          1         0 read
------ ----------- ----------- --------- --------- ----------------
100.00    0.001000                     11         0 total

Best practices and security considerations

  • Access control: Tracing is subject to ptrace access checks, Yama, and SELinux. Under kernel.yama.ptrace_scope=1, same-UID ownership alone does not permit attaching to an unrelated process; an allowed parent-child relationship or CAP_SYS_PTRACE in the target’s user namespace is required. Root on the host may be needed to trace another user’s process.

  • SELinux: Inspect access-vector denials with ausearch -m AVC -ts recent and interpret them with audit2why. Prefer a narrow policy exception for the specific source and target domains; do not disable SELinux.

  • Data leakage: Arguments and buffers can contain credentials, tokens, and personal data. Restrict access to trace files, delete them promptly, and redact sensitive values before sharing.

  • Performance impact: Every traced system call introduces synchronization and formatting overhead. Use filters, avoid tracing busy production services without approval, and monitor latency and CPU usage.

  • Long-running processes: Use -o to direct output to a controlled log path, but also monitor disk consumption and remove the file when investigation ends.

  • Detaching: When attached with -p, press Ctrl+C to detach. The target normally resumes after strace disconnects.

58

How do you use lsof to inspect open files and network connections?

lsof lists open file descriptors and network sockets, showing which processes are using files, directories, pipes, devices, ports, and Unix sockets. On Linux, it correlates kernel state exposed through /proc with file-descriptor, device, and inode information.

Read full answer

Applies toLinux (Ubuntu 22.04/24.04, RHEL 9) with lsof 4.x

How It Works

lsof reads information such as /proc/<pid>/fd, /proc/<pid>/maps, and Linux network socket tables. For each descriptor, it reports the owning process and relevant details:

  • Process: command, PID, and user.

  • Descriptor: file-descriptor number and access mode.

  • Type: REG, DIR, FIFO, IPv4, IPv6, unix, and others.

  • Device and inode: identify the underlying filesystem object.

  • Name: file path, socket address, or socket state.

Because /proc is a live view, results are a point-in-time snapshot. Files can be opened or closed while lsof is running, so important checks should be repeated during an incident.

Common Commands

  • List all visible open descriptors: bash lsof Unprivileged users see only descriptors they can access.

  • Find the process listening on TCP port 5432: bash sudo lsof -nP -iTCP:5432 -sTCP:LISTEN

  • -iTCP:5432 selects TCP endpoints using port 5432.

  • -sTCP:LISTEN restricts results to listening sockets.

  • -n prevents hostname resolution.

  • -P displays numeric port numbers instead of service names.

  • Inspect one process: bash sudo lsof -p 1234 This shows regular files, sockets, directories, devices, and mapped files associated with PID 1234.

  • Combine selections explicitly: bash sudo lsof -a -p 1234 -i -a means that the -p and -i conditions must both match. Without it, lsof can use logical OR between selection options.

  • Inspect a process’s working directory and executable: bash sudo lsof -a -p 1234 -d cwd,txt -d selects descriptor types such as cwd and txt.

  • Find users of a particular file: bash sudo lsof /var/log/nginx/access.log

  • List descriptors owned by a user: bash sudo lsof -u nginx

  • Find deleted files that remain open: bash sudo lsof +L1 +L1 selects open files whose Unix link count is less than one. This is useful when a process continues writing to a rotated or removed log.

Code Example

Finding a PostgreSQL listener:

$ sudo lsof -nP -iTCP:5432 -sTCP:LISTEN
COMMAND   PID USER      FD   TYPE DEVICE SIZE/OFF NODE NAME
postgres  842 postgres   4u  IPv4 119617      0t0  TCP *:5432 (LISTEN)

The output shows that PostgreSQL PID 842 owns the listening socket. It also reveals that the server listens on *:5432, meaning all available IPv4 interfaces, rather than only 127.0.0.1.

Inspecting that process’s working directory and executable:

$ sudo lsof -a -p 842 -d cwd,txt
COMMAND   PID USER      FD  TYPE DEVICE SIZE/OFF     NODE NAME
postgres  842 postgres cwd   DIR  8,2     4096      131089 /var/lib/postgresql/16/main
postgres  842 postgres txt   REG  8,1   890112      262147 /usr/lib/postgresql/16/bin/postgres

Finding a deleted log file still held open:

$ sudo lsof +L1
COMMAND   PID USER   FD   TYPE DEVICE SIZE/OFF NODE NAME
rsyslogd  711 syslog  7w   REG  8,1     8192 524301 /var/log/app.log (deleted)

The file was unlinked from the directory, but rsyslogd still holds the descriptor. Continuing writes succeed, but the data is no longer reachable through the original pathname and is reclaimed only after the last descriptor closes.

Production Practices and Pitfalls

  • Use the least privilege required: start without sudo; escalate only when descriptors are hidden by permissions. Avoid permanently running services or shells as root.

  • Install with the platform package manager: “‘bash # Ubuntu or Debian sudo apt-get update sudo apt-get install lsof

# RHEL, Rocky Linux, AlmaLinux, or Fedora sudo dnf install lsof “ - **Usessfor socket overview**:ss -lntpis often faster and more focused for listening sockets. Uselsofwhen you need detailed descriptors, deleted files, working directories, or process-wide file mappings. - **Avoid indiscriminate recursive scans**: operations such aslsof +D /large/pathcan be expensive on large directory trees. - **Account for namespaces**: run inside the relevant container or PID namespace when diagnosing a container. A host-level view may not represent the process’s complete socket view. - **Correlate with other evidence**: checkjournalctl, service logs,ss, process state, disk usage, and application metrics.lsof` shows what is open, but not necessarily why it is open or whether the application is healthy. - Interpret permissions carefully: inaccessible descriptors can cause incomplete results, especially for other users’ processes and kernel-owned resources.

59

What happens when a file is unlinked while a process has it open?

When you delete a file that a process still has open, the unlink system call removes the directory entry, but the process can continue reading and writing to the file through its open file descriptor. The file’s data remains on disk until all file descriptors referencing it are closed. This behavior is fundamental to Unix-like operating systems.

Read full answer

Applies toLinux (all kernel versions), also other Unix-like systems.

How File Deletion Works

  • The unlink() system call removes the directory entry (the file name) from the filesystem. If the file has multiple hard links, only that specific name is removed; the inode and data blocks remain until the last link is gone.

  • If the file is the last link and no process has it open, the kernel frees the inode and the data blocks, returning the space to the filesystem.

  • When a process has the file open, the inode and data are kept alive because the open file descriptor holds a reference. The rm command is simply a user-space wrapper around unlink().

Process Perspective

  • The process’s file descriptor table still points to the same inode. All system calls that operate on the file descriptor (read, write, fstat, lseek, etc.) continue to work normally.

  • The file is no longer accessible by its original name from the filesystem. Any attempt to open the file by name after deletion will fail with ENOENT (No such file or directory).

  • You can inspect the open file descriptors of a process via /proc/<PID>/fd. Files that have been deleted show up with the original name followed by (deleted) in parentheses, e.g., /tmp/log.txt (deleted).

  • If the process holds an open file descriptor, it can create a new hard link to the inode using that descriptor (e.g., via /proc/self/fd/N), effectively restoring a directory entry and allowing access by a new name.

  • When a process forks, the child inherits the file descriptors, which increments the reference count on the inode. The inode is not freed until all descriptors (in all processes) are closed.

Disk Space and Monitoring

  • The deleted file continues to occupy disk space until the process closes the file descriptor or terminates. This can lead to disk full situations that are not immediately obvious because the file is no longer visible in directory listings.

  • Use lsof +L1 to list files with a link count less than 1 (i.e., deleted but still open). Alternatively, ls -l /proc/<PID>/fd shows the (deleted) marker.

  • In production, this behavior is common during log rotation. If a daemon does not reopen its log files after receiving a signal (e.g., USR1), the old log file remains open and continues to grow, consuming disk space.

  • To recover disk space without killing the process, you can truncate the file by opening it via /proc/<PID>/fd/<N> and calling ftruncate(0), or using truncate -s 0 /proc/<PID>/fd/<N>. However, not all applications handle truncated open files gracefully; some may continue writing at the original offset, causing sparse files.

Security and Isolation

  • Because the file data persists as long as the process holds it open, sensitive data may remain on disk longer than expected. An attacker with physical access could potentially recover the data from the raw block device.

  • To securely delete a file that a process has open, you must first ensure the process closes the file descriptor (e.g., by restarting the process) and then overwrite or delete the file.

  • The /proc/<PID>/fd interface allows a process (with appropriate permissions) to open a new file descriptor pointing to the same inode, which can be used to read or modify the “deleted” file. This is a potential privilege escalation vector if not properly restricted.

Common Use Cases

  • Temporary files: Many programs create a temporary file, immediately unlink it, and keep it open. This ensures the file is automatically cleaned up if the process crashes, and it prevents other processes from accessing the file by name. The mkstemp function is a standard way to do this.

  • Log rotation: Daemons like nginx, Apache, and syslogd are designed to receive a signal (e.g., USR1) that causes them to close the current log file, reopen a new one, and then the old file can be safely deleted. If the daemon does not support this, you must restart it.

  • File descriptors as handles: Some network servers use open file descriptors as handles for client connections, and the file itself is irrelevant after opening.

Best Practices and Common Pitfalls

  • Monitor open deleted files: Regularly check lsof +L1 or scan /proc for processes with open deleted files, especially on systems with tight disk quotas.

  • Use proper log rotation: Prefer signal-based log reopening over blind deletion. If you must delete a log file, first move it (rename) to a backup, then truncate the original if the application supports it, or restart the service.

  • Do not assume deletion frees space: When you delete a large file that a process has open, the disk usage will not change until the process releases the file. Use du and df together with lsof to diagnose.

  • Secure temporary files: Use mkstemp or similar functions that create and unlink the file atomically, and set appropriate permissions to avoid race conditions.

  • Avoid manual intervention: Killing a process to free a deleted file may cause data loss or service disruption. Evaluate the impact before doing so.

60

What information does the Linux /proc filesystem expose?

/proc is a virtual filesystem through which Linux exposes process, memory, CPU, network, device, and kernel runtime information. It is generated by the kernel rather than stored as ordinary files on disk.

Read full answer

Applies toLinux kernel 5.x and 6.x, including Ubuntu 22.04/24.04 and RHEL 9

Purpose and Mounting

/proc is a Linux-specific directory in the filesystem hierarchy, alongside /sys, /dev, and /run. It provides a standardized view of kernel and process state through paths such as /proc/cpuinfo, /proc/meminfo, and /proc/<pid>/status.

Most entries are synthesized by the kernel when accessed. Their size and contents can therefore change between reads, and deleting an entry does not permanently remove the corresponding runtime information.

The kernel implements this interface through the procfs filesystem. It is commonly mounted with options such as rw,nosuid,nodev,noexec,relatime; findmnt /proc confirms the active mount and its options. Commands such as cat and grep access procfs directly, while tools from procps, iproute2, and other packages provide more convenient views.

Important Interfaces

Path Information exposed
/proc/<pid>/ State and resources for one process
/proc/self/ The process performing the access
/proc/cpuinfo Processor details, architecture, features, and CPU flags
/proc/meminfo Physical memory, cache, huge-page, and swap statistics
/proc/stat Aggregate CPU-time, process, block-I/O, and paging counters
/proc/loadavg Scheduler load averages and running/total process counts
/proc/uptime System uptime and cumulative idle time
/proc/net/ Network-namespace socket and protocol state
/proc/sys/ Kernel tunables exposed through sysctl
/proc/mounts Mount table for the current mount namespace
/proc/partitions Block devices and partitions recognized by the kernel
/proc/diskstats Per-device I/O counters
/proc/version Kernel build and compiler information

Many network tools obtain equivalent socket information through netlink rather than by parsing /proc/net.

Each process directory contains files such as cmdline, status, stat, io, fd, maps, and smaps_rollup. The status file provides a readable summary including the process name, state, parent PID, memory usage, capability sets, and context-switch counts. Scheduling priority is instead available in the priority and nice fields of /proc/<pid>/stat.

The /proc/<pid>/exe symlink identifies the executable, while /proc/<pid>/cmdline contains its NUL-separated arguments. Access to sensitive entries such as environ, fd, maps, and io is subject to permission and ptrace-style access checks.

Inspecting Processes and Resources

Higher-level commands include ps -p 1234 -o pid,ppid,user,%cpu,%mem,stat,cmd, ss -lntp, and sysctl vm.swappiness. With ps, -p selects a PID and -o selects output columns. The ss command lists listening TCP sockets with numeric addresses and, where permitted, process ownership. sysctl reads or writes values exposed through /proc/sys; results depend on the kernel, namespace, and host configuration.

The $$ shell variable expands to the current shell’s PID. Quoting it as "$$" does not prevent expansion; it only groups the expansion when constructing a path.

Operational and Security Considerations

  • Least privilege: The kernel restricts many process details and tunables. Avoid running monitoring or diagnostic services as root when narrower privileges suffice.

  • Namespace isolation: A container using a PID namespace sees a different process set under /proc. Mount namespaces can also change which filesystem information is visible.

  • Secret exposure: /proc/<pid>/cmdline and /proc/<pid>/environ may contain credentials passed as arguments or environment variables. Prefer protected secret files or a secrets manager.

  • Do not publish procfs: Never expose /proc through a web server or other user-facing file service. Files such as /proc/kcore can expose physical-memory contents to processes with CAP_SYS_RAWIO; that capability should not be granted unnecessarily.

  • Monitoring: Established collectors such as node_exporter, procps, and sysstat are generally safer than custom polling. Large or expensive per-process files such as smaps require particular care because frequent collection can add CPU and scheduling overhead.

  • Process visibility: Use procfs hidepid mount options or service-level access controls when users must not see other processes. Do not attempt to solve visibility problems with broad chmod changes.

Common Pitfalls

  • It is not persistent: Runtime state represented under /proc is cleared or regenerated across reboots.

  • Output is machine-dependent: CPU models, memory sizes, kernel features, namespace settings, and tunables vary between systems.

  • Formatting is not always stable: Some human-readable files have no cross-version format guarantee. Prefer documented interfaces or specialized tools over parsing assumptions.

  • Polling has a cost: Repeatedly reading large or expensive files can affect a busy production host. Set sensible collection intervals and monitor collector overhead.

  • Kernel upgrades can change details: Revalidate custom parsers and dashboards after kernel upgrades, especially when namespace configuration or enabled kernel features differ.

61

How do Linux kernel parameters work with sysctl?

Kernel parameters control Linux kernel behavior at different stages, while sysctl manages the subset exposed through the /proc/sys interface. Most sysctl changes take effect immediately, but they must be persisted separately to survive a reboot.

Read full answer

Applies toLinux kernel 4.18+ on Ubuntu 22.04/24.04 and RHEL 9

Types of Kernel Parameters

Linux exposes kernel configuration through three main mechanisms:

Type Where it is configured Example Reboot required
Compile-time options Kernel build configuration CONFIG_NETFILTER Kernel rebuild and reboot
Boot-time parameters Kernel command line in GRUB console=ttyS0 Yes
Runtime parameters /proc/sys, usually with sysctl vm.swappiness, net.ipv4.ip_forward Only if not persisted

sysctl handles runtime parameters. It cannot set every kernel option: some are compile-time, some are boot-time, and others are read-only. Boot-time parameters must be configured through the bootloader instead.

How sysctl Works

A sysctl name maps directly to a file under /proc/sys. For example:

  • vm.swappiness maps to /proc/sys/vm/swappiness

  • net.ipv4.ip_forward maps to /proc/sys/net/ipv4/ip_forward

The kernel defines each parameter’s type, permissions, valid range, and behavior. Values such as booleans commonly accept 0 or 1; numeric parameters enforce their own minimum and maximum values.

Changes made with sysctl -w affect only the running kernel. They disappear after reboot unless the same settings are stored in a configuration file processed during boot.

Code Example

Read a parameter and change it at runtime:

$ sysctl net.ipv4.ip_forward
net.ipv4.ip_forward = 0

$ sudo sysctl -w net.ipv4.ip_forward=1
net.ipv4.ip_forward = 1

$ sysctl net.ipv4.ip_forward
net.ipv4.ip_forward = 1

Persist the setting using a dedicated file under /etc/sysctl.d/:

sudo tee /etc/sysctl.d/99-production.conf >/dev/null <<'EOF'
vm.swappiness = 10
net.ipv4.ip_forward = 1
EOF

sudo sysctl --system

Representative output is:

* Applying /etc/sysctl.d/99-production.conf ...
vm.swappiness = 10
net.ipv4.ip_forward = 1

During boot, systemd-sysctl.service applies these files automatically on both Ubuntu and RHEL. Files are processed in lexicographical order, so a higher-numbered filename can override settings from a lower-numbered file.

For a focused validation, load only the changed file:

sudo sysctl -p /etc/sysctl.d/99-production.conf

Avoid placing application-specific tuning in /etc/sysctl.conf; /etc/sysctl.d/ makes ownership, review, and rollback clearer.

Runtime and Production Behavior

  • Permissions: Writing normally requires root privileges.

  • Containers: A container may have /proc/sys mounted read-only or restricted by its runtime. A setting that works on the host might therefore fail inside a container.

  • Namespaces: Some network-related parameters are scoped to a network namespace, so the value visible inside a container may differ from the host.

  • Validation: Invalid values are rejected by the kernel rather than silently accepted.

  • Overwrites: Cloud-init, NetworkManager, DHCP services, or image-bootstrap scripts may reapply settings. Configuration ownership must be reviewed during incidents.

  • Operational impact: Tuning such as vm.swappiness, vm.dirty_ratio, net.core.somaxconn, or net.ipv4.ip_forward can affect performance, connection queues, routing, and security. Test changes on representative workloads first.

For example, enabling net.ipv4.ip_forward does not by itself provide working container routing or Internet access. Firewall rules, NAT configuration, security groups, routes, and upstream cloud networking must also be correct.

Failure Modes and Best Practices

  • Immediate loss after reboot: The value was changed with sysctl -w but was not added to /etc/sysctl.d/.

  • Unknown parameter: The name is misspelled, removed from the kernel, or unavailable in the current namespace.

  • Permission denied: The command lacks sufficient privileges or /proc/sys is read-only.

  • Conflicting configuration: Multiple files define the same key; the last loaded file wins.

  • Unverified persistence: After reboot, compare the effective value with sysctl <parameter-name>.

  • Unsafe rollout: Do not copy tuning blindly from an unrelated workload. Establish a baseline, test under load, document the reason for each change, and keep a rollback copy.

  • Incomplete incident evidence: Record the previous value, new value, source file, hostname, kernel version, and time of change. These details distinguish host configuration from runtime overrides.

Useful inspection commands include:

uname -r
sysctl net.ipv4.ip_forward
sysctl vm.swappiness
sudo sysctl -a
cat /proc/cmdline

Treat sysctl changes as production configuration changes: review them like code, apply them consistently, validate their runtime and post-reboot values, and monitor the workloads they affect.

62

How do you inspect and load Linux kernel modules?

The standard Linux tools for managing kernel modules are lsmod and modinfo for inspection, and modprobe for loading or unloading modules with dependency resolution. In production, prefer modprobe over insmod, and validate dependencies, parameters, hardware support, and the impact of unloading before making changes.

Read full answer

Applies toLinux kernel 5.14–6.x on Ubuntu 22.04/24.04, RHEL 9, and current Fedora

Inspecting Loaded Modules

lsmod provides a formatted view of /proc/modules, while modinfo reads a module’s metadata without loading it. The uname -r output identifies the running kernel; module files must match its kernel release and architecture.

  • Module: Kernel module name.

  • Size: Kernel-reported module size in bytes, not a process-style resident-memory measurement.

  • Used by: Kernel reference or use count. Additional columns list modules holding references; a nonzero count normally prevents unloading.

  • depends: Modules that modprobe must load first.

  • vermagic: Kernel compatibility requirements; an incompatible module normally fails to load.

  • parm: Supported module parameters.

  • alias: Hardware or feature aliases used for automatic and manual loading.

Module trees normally live under /lib/modules/$(uname -r)/, commonly symlinked to /usr/lib/modules/. Ubuntu commonly uses compressed .ko.zst files; RHEL 9 and many Fedora releases use .ko.xz. Both formats are supported by modern kmod.

Loading, Unloading, and Automation

Use the module name or alias—not normally its full filename—with modprobe. It consults the dependency and alias databases generated by depmod, loads required dependencies, and supports compressed module files.

Module parameters can be supplied during the initial load. They are not reapplied by a later modprobe invocation if the module is already loaded. After loading, documented parameter values are exposed under /sys/module/<module>/parameters/.

Unloading fails when the module still has references or active resources. For example, unloading overlay while an OverlayFS mount is active is unsafe. Avoid insmod and rmmod for routine administration because insmod does not provide normal dependency resolution and rmmod does not provide modprobe‘s alias handling.

Hardware modules can be requested automatically by udev through device aliases. Persistent boot-time loading is configured under /etc/modules-load.d/; systemd-modules-load applies those configurations.

Code Example

# Identify the running kernel and inspect its module tree.
$ uname -r
6.8.0-40-generic

$ ls -ld /lib/modules/"$(uname -r)"
drwxr-xr-x 4 root root 4096 ... /lib/modules/6.8.0-40-generic

# List loaded modules; /proc/modules exposes the raw form.
$ lsmod | head
Module                  Size  Used by
overlay               153600    1
nf_conntrack          139296    1
veth                   40960    0
xt_nat                  40960    0
# Read module metadata without loading it.
$ modinfo overlay
filename:       /lib/modules/6.8.0-40-generic/kernel/fs/overlay/overlay.ko.zst
license:        GPL
description:    Overlay filesystem
author:         Miklos Szeredi <miklos@szeredi.hu>
vermagic:       6.8.0-40-generic SMP mod_unload modversions
# Preview resolution, load the module, and verify the result.
$ sudo modprobe --dry-run overlay
$ sudo modprobe --show-depends overlay
overlay

$ sudo modprobe overlay
$ lsmod | grep '^overlay'
overlay               153600    1

# Unload only a module whose Used by count is 0 (overlay above is in use).
$ sudo modprobe --remove veth
# Inspect a real module's parameters.
$ modinfo bonding | grep '^parm:'
parm: mode: (int)
parm: miimon: (int)
parm: updelay: (int)
parm: downdelay: (int)
parm: fail_over_mac: (int)

# Set mode at initial load; 4 selects IEEE 802.3ad aggregation.
$ sudo modprobe bonding mode=4
# /etc/modules-load.d/example.conf
nf_conntrack

# Review the merged configuration and load configured modules.
$ systemd-modules-load --cat-config
$ sudo systemd-modules-load

Production and Security Considerations

  • Match the running kernel: Rebuild or reinstall out-of-tree modules after every kernel upgrade. DKMS can automate this, but monitor its build failures during package updates.

  • Rebuild dependencies: Run sudo depmod -a after manually placing a correctly built module in the module tree. Package-managed installations normally handle this automatically.

  • Use least privilege: Loading a kernel module grants substantial kernel-level authority. Restrict module installation and access to root or narrowly delegated administrative accounts.

  • Avoid unnecessary unloading: Drivers may manage I/O, networking, filesystems, or security controls; unloading one can interrupt active workloads.

  • Test safely: Check dmesg, journalctl -k, service state, and application health before and after changes. Keep a rollback path and validate changes before applying them to production hosts.

  • Prefer supported tooling: Configure storage, networking, and hardware through supported OS tooling when possible rather than manually manipulating modules.

63

How does SELinux work and what modes does it support?

SELinux is a Linux mandatory access control (MAC) framework that uses policy rules and security labels to restrict what users and processes can access. It supports enforcing, permissive, and disabled modes.

Read full answer

Applies toSELinux on RHEL 9/10 and Fedora 41/42

How It Works

SELinux adds a second access-control layer alongside traditional Linux discretionary access control (DAC).

  • DAC: Owner, group, and permission bits such as rwx determine access.

  • MAC: SELinux policy can deny an action even when DAC permits it.

  • Security context: Files, processes, ports, and other objects have a label containing a user, role, type, and, where applicable, sensitivity level. For example, system_u:object_r:httpd_sys_content_t:s0 identifies an object with type httpd_sys_content_t.

  • Type enforcement: Most policy rules permit access only between approved source and target types for a particular object class and permission.

  • Policy store: The loaded policy contains the allowed relationships and is consulted by the kernel’s SELinux security server. Frequently used decisions are cached in the Access Vector Cache.

For example, the Apache process runs in the httpd_t domain and may be allowed to read files labeled httpd_sys_content_t, while access to files under an administrator’s home directory is restricted. An access must be allowed by both DAC and SELinux when SELinux is enabled.

The commonly used targeted policy confines many system services to restricted domains. Other services and processes may remain in unconfined domains, so targeted is not equivalent to applying strong confinement to every process.

SELinux Modes

Mode Enforcement Audit logging Production use
Enforcing Unauthorized actions are blocked AVC denials are logged when SELinux auditing is enabled Recommended default
Permissive Unauthorized actions are allowed but recorded as AVC denials when auditing is enabled Yes, when auditing is enabled Temporary diagnosis only
Disabled SELinux is not enforcing policy No SELinux enforcement decisions or AVC denials Generally inappropriate

targeted, minimum, standard, and mls are policy types, not enforcement modes. For example, a normal RHEL configuration uses SELINUX=enforcing with SELINUXTYPE=targeted. The MLS status reported by sestatus is also separate from these three top-level modes.

Inspecting and Changing the Mode

getenforce reports the current mode as Enforcing, Permissive, or Disabled. sestatus provides additional information, including whether SELinux is enabled, the loaded policy name, the current mode, the mode configured for the next boot, and MLS status. ls -Z displays the SELinux context of a file or process.

Use sudo setenforce 0 to switch a running, enabled system to permissive mode and sudo setenforce 1 to return it to enforcing mode. These commands change only the runtime state; they cannot enable SELinux if it was disabled during boot.

For a persistent change, edit /etc/selinux/config and reboot. Changing SELINUX=enforcing to SELINUX=permissive does not require relabeling. Re-enabling SELinux after a period in disabled mode does: existing files may have missing or invalid labels, so create /.autorelabel or schedule fixfiles -F onboot before rebooting, and verify that the policy packages are installed.

Ubuntu normally uses AppArmor and does not install SELinux utilities by default. If the SELinux tools are installed but SELinux is disabled, getenforce reports Disabled.

Diagnosing and Fixing Denials

Use sudo ausearch -m AVC -ts recent to inspect recent SELinux denial records. A denial identifies the complete source and target contexts, the object class, and the requested permission. The scontext and tcontext fields contain the full contexts; their type components, such as httpd_t and admin_home_t, identify the process and object types involved.

Prefer a persistent file-context rule over broad permission changes. For a new local rule, use sudo semanage fcontext -a -t httpd_sys_content_t "/srv/www(/.*)?", then apply the policy’s expected labels with sudo restorecon -Rv /srv/www. If the regular expression already has a local rule, modify it with -m instead of adding it with -a.

restorecon reapplies the labels expected by policy. chcon changes a label directly, but that change is generally temporary and should not replace a persistent semanage fcontext rule.

Production Practices

  • Keep SELinux in enforcing mode in production.

  • Use permissive mode only temporarily while diagnosing an incident, then restore enforcing mode and address the underlying label or policy requirement.

  • Enable and monitor auditd, collect AVC denials centrally, and alert on unexpected denial patterns.

  • Preserve SELinux labels during backups, image creation, and file transfers; restoring files with incorrect labels can cause service failures.

  • Test policy updates and custom file contexts in staging before deployment.

  • During operating-system upgrades, update SELinux policy packages, inspect new denials, and verify that critical services start in their intended domains.

  • Do not use chmod 777, disable auditing, or turn off SELinux as a substitute for correcting service labels and access requirements.

64

What does AppArmor provide for Linux security?

AppArmor is one of Linux’s mandatory access control (MAC) systems. It confines applications with path-based profiles, restricting file, process, capability, and supported network operations even when ordinary Unix permissions would otherwise allow them.

Read full answer

Applies toAppArmor 3.0.x on Ubuntu 22.04 LTS and AppArmor 4.0.x on Ubuntu 24.04 LTS

How AppArmor Works

AppArmor is implemented through the Linux kernel’s LSM framework. When a confined program performs a protected operation, the kernel checks the operation against the active AppArmor profile before allowing it.

Profiles commonly control access to:

  • Specific files and directories.

  • Executable programs and libraries.

  • Application data and temporary files.

  • Selected Linux capabilities.

  • Network operations supported by the installed AppArmor version.

AppArmor complements discretionary access control (DAC)—the permissions represented by ownership, chmod, and groups. DAC can grant broad access because the owner or an administrator controls it; AppArmor policy can deny an operation even when DAC permits it.

Profiles and Enforcement Modes

Profiles are stored under /etc/apparmor.d/. Distribution packages commonly provide profiles for services such as Apache and nginx, while site-specific rules may be added under /etc/apparmor.d/local/.

Each confined process operates in one of three modes:

Mode Behaviour Typical use
enforce Denies prohibited operations and records violations Production after validation
complain Allows operations but records violations Initial testing and troubleshooting
unconfined Applies no AppArmor confinement Legacy or deliberately unrestricted applications

Profile names often identify the executable they protect, such as usr.sbin.nginx. Permissions can use wildcards, but broad rules such as /** rw, weaken the security boundary and should be avoided.

Operational Commands

Useful production commands include:

  • sudo aa-status: Summarizes loaded profiles and counts profiles in each mode. Exact counts depend on installed packages.

  • sudo journalctl -k --since "15 min ago" | grep -iE "apparmor|audit": Reads recent kernel messages, limits the time range with --since, and performs a case-insensitive extended-regex search with grep -iE.

  • sudo aa-complain /etc/apparmor.d/usr.myapp: Places a profile in complain mode for controlled testing.

  • sudo aa-enforce /etc/apparmor.d/usr.myapp: Returns a tested profile to enforce mode.

  • sudo apparmor_parser -r /etc/apparmor.d/usr.myapp: Compiles and reloads a profile after it has been edited.

  • sudo aa-logprof: Guides interactive rule creation from recent denials. Proposed permissions must be reviewed rather than accepted blindly.

  • If auditd is installed, sudo grep -i 'apparmor="DENIED"' /var/log/audit/audit.log searches the traditional audit log. Ubuntu systems without auditd should use journalctl instead.

Comparison with SELinux

Feature AppArmor SELinux
Primary model Path-based profiles Security labels and type enforcement
Common distributions Ubuntu, Debian RHEL, Fedora
Typical profile location /etc/apparmor.d/ /etc/selinux/ and policy modules
Administration tools aa-status, aa-enforce, aa-logprof semanage, restorecon, setsebool
Security emphasis Practical application confinement Fine-grained label-based least privilege

Ubuntu normally uses AppArmor, while RHEL and Fedora normally use SELinux. They are alternative MAC mechanisms; replacing one with the other requires migrating policy and validating the complete workload.

Production Practices and Failure Modes

  • Roll out gradually: Test a profile in complain mode, exercise normal and failure paths, and then move it to enforce.

  • Monitor denials: Correlate AppArmor messages with application logs and security monitoring. Alerts should distinguish intentional policy violations from unexpected profile behavior.

  • Test upgrades: Package upgrades can change executable paths, helper binaries, libraries, or configuration locations, causing previously valid profiles to stop matching.

  • Plan rollback: Preserve the original profile and establish how to restore the previous application or policy version if enforcement causes an outage.

  • Account for containers: Host profiles may encounter different mount paths or namespace assumptions inside containers. Validate confinement in the actual deployment topology.

  • Do not treat AppArmor as the only control: Continue using patching, least-privilege service accounts, network firewalls, encryption, backups, and secure secret management.

If an application suddenly loses access, check whether its profile became unconfined, whether an upgrade moved the executable, and whether the denial appears in journalctl or the audit log. Do not disable AppArmor globally; narrow the affected rule or use a carefully tested local override instead.

65

How does logrotate rotate logs on Linux?

logrotate rotates logs according to scheduled policies, usually by renaming the active file, shifting and compressing older generations, and deleting files beyond the configured retention count. Rename-based rotation requires the application to reopen its log path; copytruncate avoids that requirement but introduces a race that can lose writes.

Read full answer

Applies toLogrotate 3.x on Ubuntu 22.04/24.04 and RHEL 9

How Rotation Works

  • Scheduling: Ubuntu normally invokes Logrotate through logrotate.timer; RHEL 9 normally uses /etc/cron.daily/logrotate. The scheduler starts the process, while each policy determines whether its logs are eligible for rotation.

  • Policy location: System policies are stored in /etc/logrotate.conf and /etc/logrotate.d/.

  • State tracking: Logrotate stores the last rotation state in /var/lib/logrotate/logrotate.status, preventing a daily policy from rotating more than once merely because the command was invoked repeatedly.

  • Typical sequence:

  • Existing generations such as .2 are shifted to .3, .4, and so on.

  • The active file is renamed to .1.

  • The newest generation may be compressed; delaycompress leaves it uncompressed until the following rotation.

  • A replacement file is created by create only when no postrotate script is defined. With postrotate, the application or script normally creates it; copytruncate does not replace the original inode.

  • The postrotate script notifies the application to reopen its log.

  • Generations beyond rotate N are removed.

Configuration Directives

Directive Behaviour
daily, weekly, monthly Establish a rotation interval, subject to the recorded state
rotate 14 Retain up to 14 previous generations in addition to the active file
compress Compress rotated generations, normally with gzip
delaycompress Leave the newest generation uncompressed until the next rotation
missingok Ignore a matching log that does not exist
notifempty Do not rotate an empty log
create 0640 syslog adm Set the replacement file’s mode, owner, and group when create is applicable
nocreate Prevent Logrotate from creating a replacement file
sharedscripts Run rotation scripts once per invocation for the logs that actually require rotation
postrotate Run commands after rotation, commonly to trigger log reopening
copytruncate Copy the active file and truncate the original instead of renaming it

Rotation Strategies

Strategy Reopen required Main trade-off
Rename plus postrotate Yes Preferred when the application can reopen logs correctly; stale descriptors can send writes to the wrong or an unlinked inode
copytruncate No Preserves the inode but writes made between copying and truncation can be lost

For systemd-managed applications, prefer systemctl reload myapp.service when the unit defines a working ExecReload=. If the application explicitly handles SIGHUP, signal the main process with systemctl kill --kill-who=main --signal=HUP myapp.service.

Production Failure Modes

  • Stale file descriptors: An application that does not reopen continues writing to the renamed or unlinked file, so new records may not appear in the expected archive.

  • Disk pressure: Retention must account for log volume, inode consumption, and temporary space required during copying and compression.

  • Copy/truncate data loss: copytruncate can lose records written during its race window; rename-and-reopen is safer when supported.

  • Script failures: A failed postrotate command can make the Logrotate invocation fail and leave the application using the old descriptor.

  • Permissions and labels: The application must be able to create and reopen the log. On RHEL, preserve appropriate SELinux labels rather than disabling SELinux.

  • Compression load: Compressing large logs can cause CPU and I/O spikes; monitor rotation duration and adjust scheduling or retention accordingly.

Operational Best Practices

  • Validate changes without changing files with sudo logrotate --debug /etc/logrotate.conf.

  • Test the reopen procedure under the real service account and confirm that new records reach the active file.

  • Monitor filesystem space, inode usage, Logrotate failures, and the relevant timer or cron execution.

  • Keep sensitive logs in a dedicated directory with restrictive permissions.

  • Use sudo logrotate --force /etc/logrotate.conf only for controlled recovery or testing; it bypasses normal interval checks.

  • Retain off-host backups or centralized log collection because rotation only manages local files and retention.

66

How do you archive and compress files with tar, gzip, or zip?

A tar archive groups files, directories, permissions, and filesystem metadata, while gzip compresses one file or stream. zip provides both archiving and compression and is convenient for cross-platform distribution; Linux backups commonly use tar with gzip compression.

Read full answer

Applies toGNU tar 1.34-1.35, gzip 1.12-1.13, and Info-ZIP `zip`/`unzip` 3.0 on Ubuntu 24.04 and RHEL 9

Choosing the Right Tool

Tool Primary use Metadata Multiple files/directories Cross-platform
tar Linux backups and package archives Strong POSIX and GNU metadata support Yes Widely supported
gzip Compressing one file or a stream Only compressed-file metadata No archive structure Widely supported
zip Windows/Linux interchange Less consistent across implementations Yes Excellent

Use tar for production Linux backups when ownership, modes, timestamps, symbolic links, ACLs, and extended attributes matter. Use zip when recipients may not have tar, especially on Windows. ZIP should not replace tar for backups that depend on Linux ownership, ACLs, or security contexts.

Essential Command Options

  • tar -czf archive.tar.gz directory/: Create a gzip-compressed tar archive.

  • -c: Create; -x: Extract; -t: List.

  • -z: Use gzip compression or decompression.

  • -f: Select the archive filename.

  • -v: Show processed files.

  • -C: Change to a specified directory.

  • --exclude: Exclude matching paths.

  • gzip -k file: Compress while retaining the original. Plain gzip file replaces it with file.gz; gzip -d file.gz decompresses it; gzip -t file.gz tests the stream.

  • gzip -c file > file.gz: Write compressed output to standard output.

  • zip -r archive.zip directory/: Archive recursively. -1 favors speed, while -9 favors compression ratio at greater CPU cost; gzip offers the same tradeoff with -1 and -9.

  • unzip -l archive.zip: List contents; unzip -t archive.zip tests member CRCs; unzip -d directory archive.zip selects the destination.

How the Formats Work

  • tar stores filesystem entries and their metadata, including headers and, for GNU/POSIX formats, optional extended metadata.

  • gzip produces a DEFLATE-compressed stream with a CRC32 and size trailer. It does not provide a directory or independent multi-file archive structure.

  • zip stores local member headers, compressed data, and a central directory that supports listing and selective extraction.

  • gzip -t and unzip -t detect corruption by validating their respective integrity data.

Code Example

# Run as an unprivileged user from a directory containing app/.
# Create a compressed tar archive, excluding its cache directory.
tar --exclude='app/cache' -czf app.tar.gz app/

# List the archive without extracting it.
tar -tzf app.tar.gz
# Typical listing:
# app/
# app/config/
# app/config/app.yaml
# app/bin/start.sh

# Extract into a fresh, isolated directory without restoring ownership.
restore_dir="$(mktemp -d)"
tar --no-same-owner --keep-old-files -xzf app.tar.gz -C "$restore_dir"

# Compress one log without removing the original, then verify it.
gzip -k app.log
gzip -t app.log.gz

# Create and test a cross-platform ZIP archive.
zip -r app.zip app/
# Typical output:
#   adding: app/ (stored 0%)
#   adding: app/config/app.yaml (deflated 54%)
#   adding: app/bin/start.sh (deflated 43%)

unzip -t app.zip
# No errors detected in compressed data of app.zip.

# Restore the ZIP into another fresh directory.
zip_restore_dir="$(mktemp -d)"
unzip -d "$zip_restore_dir" app.zip

# Generate and verify a detached checksum.
sha256sum app.tar.gz > app.tar.gz.sha256
sha256sum -c app.tar.gz.sha256
# app.tar.gz: OK

Production Practices and Security

  • Create consistent backups: Use application-aware backups or database dumps for live data. An LVM or filesystem snapshot is normally crash-consistent; quiesce the application, flush databases, or coordinate them with the snapshot when application consistency is required.

  • Preserve Linux metadata deliberately: Add --xattrs --acls when those attributes must survive a backup, and --selinux on systems where SELinux contexts must be retained. Test restoration on the target platform; privileged controlled restores may need to restore ownership and labels.

  • Protect archives: Compression is not encryption. Use umask 077, restrict access, and encrypt sensitive archives with age or GPG.

  • Inspect untrusted archives: List members first, use a fresh unprivileged destination, and retain --keep-old-files. Do not rely on a shared restore directory to prevent overwrites.

  • Control decompression resources: Stock tar and unzip do not provide extraction quotas. Enforce limits with an isolated filesystem or container and monitor expanded size and member count.

  • Test restoration and monitor jobs: Verify checksums, perform scheduled restore tests, and alert on backup-job failures. Run scheduled jobs with systemd timers or services and monitor unit status and journald output.

  • Use the Linux hierarchy deliberately: Store routine archives under /var/backups or a dedicated backup mount, configuration under /etc, and logs under /var/log; keep large backups off the root filesystem.

67

How do pipes and redirection work in Linux?

In Linux, pipes and redirection control where commands read from and write to by connecting or replacing their standard file descriptors. A pipe connects one command’s standard output to another command’s standard input, while redirection sends standard input or output to files, devices, or other descriptors.

Read full answer

Applies toLinux (Ubuntu 22.04/24.04, RHEL 9) with Bash 5.x

How Pipes Work

  • The standard streams are file descriptor 0 for standard input, 1 for standard output, and 2 for standard error.

  • Bash obtains an anonymous kernel pipe with a read end and a write end. After creating the child processes, it uses dup2() to connect the write end to the first command’s standard output and the read end to the second command’s standard input.

  • The kernel buffers pipe data, commonly around 64 KiB on Linux, although the actual capacity is kernel and configuration dependent. A writer blocks when the buffer is full until the reader consumes data.

  • The reader receives EOF only after all write ends have been closed. A pipeline remains open if another process still holds a write descriptor.

  • If a process writes to a pipe whose read end has no open readers, the write fails with EPIPE and normally raises SIGPIPE. Programs may ignore or handle that signal instead of terminating.

  • Anonymous pipes have no filesystem name and are unidirectional. Named pipes, or FIFOs, are created with mkfifo and provide similar byte-stream behavior through a filesystem path.

Redirection Operators

Operator Effect
> file Redirect standard output to file, truncating it if it exists.
>> file Append standard output to file.
< file Redirect standard input from file.
2> file Redirect standard error to file.
2>> file Append standard error to file.
&> file Redirect both standard output and standard error to file; this is Bash-specific.
| Connect the left command’s standard output to the right command’s standard input.

Bash also supports here-strings with <<< and here-documents with <<. The exec builtin can open additional descriptors, such as exec 3>file.

A newly created redirection target normally starts with mode 0666, modified by the process’s umask. If the target already exists, redirection truncates or appends to it without changing its existing ownership or permissions.

Combining Pipes and Redirection

Redirections attached to a command may be interspersed with its arguments, but their order matters:

  • cmd 2>&1 > file duplicates the current standard output into standard error before replacing standard output. Consequently, stderr goes to the original destination while stdout goes to file.

  • cmd > file 2>&1 sends both stdout and stderr to file, because 2>&1 duplicates the already redirected stdout.

  • cmd1 | cmd2 > result.txt sends cmd2‘s stdout to the file.

  • cmd1 2>&1 | tee log.txt sends cmd1‘s stdout and stderr through tee, which writes them to the log and standard output. Stderr from tee itself remains separate.

  • cmd < input.txt | cmd2 supplies input.txt as cmd‘s standard input.

  • while IFS= read -r line; do printf '%s\n' "$line"; done < file reads a file safely without treating backslashes as escape characters or trimming leading and trailing whitespace.

Bash process substitution, using <(cmd) and >(cmd), exposes command output as a file-like path.

Production Considerations

  • Pipeline failures: A pipeline normally returns the exit status of its final command. In scripts, set -o pipefail makes the pipeline return failure if any component fails. Inspect ${PIPESTATUS[@]} immediately after the pipeline to obtain each command’s status.

  • Error handling: Do not silently discard standard error. Send it to an appropriate log or monitoring system; use /dev/null only when the omission is intentional and documented.

  • File permissions: Precreate sensitive output files with install -m 0640 output.log or enforce a restrictive umask. Redirection alone does not correct an existing file’s permissions.

  • Blocking and resource use: A pipeline can block when its buffer fills or a consumer stops reading. Processes stuck in uninterruptible sleep can indicate storage or I/O contention and should be investigated with system monitoring.

  • Buffering: stdbuf -oL command can request line-buffered standard output for programs that use standard buffering, but it cannot change buffering implemented internally by an application.

  • Logging: Shell redirection creates ordinary files that require rotation and retention management. For systemd services, StandardOutput=journal and StandardError=journal provide structured journald capture and operational tooling.

  • Signal behavior: Early pipeline termination can produce SIGPIPE, for example when head exits before consuming all input. Ensure applications handle the signal deliberately when such pipelines are expected.

  • Security: Avoid eval and untrusted command construction. Quote variable expansions to prevent word splitting and pathname expansion, and prefer printf over echo for predictable output.

68

How do >, >>, and 2>&1 differ in Linux shell redirection?

These operators control file-descriptor redirection in the Linux shell: > truncates a file, >> appends to it, and 2>&1 duplicates standard error to the current standard-output destination. The order of redirections matters because the shell processes them from left to right.

Read full answer

Applies toBash 5.x and POSIX-compatible shells on Linux

Redirection Operators

Syntax Meaning Common use
command > file Redirect stdout to file, creating or truncating it Save normal output
command >> file Redirect stdout to file, creating it or appending to the end Preserve previous output
command 2> file Redirect stderr to file Save error messages separately
command 2>&1 Duplicate stderr to wherever stdout currently points Combine normal and error output
command > file 2>&1 Redirect stdout to file, then point stderr to that same file Capture both streams

The standard file descriptors are:

  • 0: standard input

  • 1: standard output

  • 2: standard error

2>&1 does not mean “redirect both stdout and stderr to a file” by itself. It means “make stderr use the destination currently assigned to stdout.”

Redirection Order

The shell evaluates redirections from left to right. With command > output.log 2>&1, stdout is first redirected to output.log. Then stderr is duplicated to stdout’s destination, so both streams go to output.log.

The reverse order behaves differently. With command 2>&1 > output.log, stderr is first duplicated to stdout’s existing destination, normally the terminal. Stdout is then redirected to output.log, so stderr remains on the terminal while stdout goes to the file.

For appending both streams, command >> output.log 2>&1 opens the file for appending and subsequently directs stderr to the same file.

Production Best Practices

  • Use >> for persistent logs: It preserves earlier records, but configure rotation because an unbounded log can exhaust disk space.

  • Prefer service managers for daemons: Let a systemd service write to stdout and stderr, then use journalctl to inspect it. Avoid inventing separate shell log files for every service.

  • Check redirection failures: The shell opens the target before executing the command. Permission errors, a full filesystem, a directory used as a target, or an invalid path can prevent the command from running.

  • Protect temporary files: Use mktemp rather than predictable names in shared directories such as /tmp, and avoid permissive permissions such as chmod 777.

  • Monitor the chosen destination: Combining streams with 2>&1 simplifies collection but removes the ability to distinguish normal output from errors unless the application adds its own severity fields.

  • Avoid accidental truncation: Remember that > is destructive. In scripts where preserving history matters, use >>, or explicitly truncate only after confirming the target is correct.

69

How do awk and sed process Linux text streams?

awk and sed are line-oriented text-processing tools: sed transforms or filters lines, while awk parses records into fields and supports aggregation, reporting, and more complex logic. Both are useful for log analysis, configuration cleanup, and data preparation in Linux automation.

Read full answer

Applies toGNU sed 4.x and POSIX awk, including mawk 1.3.4 and gawk 5.x

How They Work

Tool Processing model Typical uses
sed Reads input sequentially into a pattern space and applies pattern-and-action commands Substitutions, deletions, insertions, line filtering
awk Reads records, splits them into fields, and executes rules Counting, grouping, summarizing, generating reports
  • sed pattern space: Initially holds the current input line; commands such as N can extend it to contain multiple lines.

  • sed address: Selects lines using a line number, regular expression, or range.

  • awk record: Normally one line, but RS can change the record separator.

  • awk field: A numbered component of the current record, accessed as $1, $2, and so on.

  • awk variables: $0 contains the whole record, while NF contains the number of fields.

Essential Operations

Common sed operations include:

  • sed -n '/ERROR/p' file: Print only lines matching ERROR.

  • sed -n '5p' file: Print line 5.

  • sed '/^#/d' file: Delete lines beginning with #.

  • sed -E 's/old/new/g' file: Replace every occurrence using extended regular expressions.

  • sed -n '/WARN/{s/timeout/connection timeout/;p}' file: Apply multiple commands to selected lines.

  • -n: Suppress automatic printing, useful with the explicit p command.

Common awk operations include:

  • awk -F',' '{print $1}' file: Print the first comma-separated field.

  • awk '$9 >= 500 {count++} END {print count}' file: Count matching records.

  • awk '{sum += $3} END {print sum}' file: Sum a numeric field.

  • awk -F'|' '{counts[$2]++} END {for (key in counts) print key, counts[key]}' file: Build a frequency report.

  • BEGIN and END: Run rules before processing input or after all input has been read.

Production Usage

  • Log triage: Use sed to normalize formats and awk to count failures by service, endpoint, status code, or region.

  • Configuration preparation: Replace placeholders or remove disabled directives before deployment.

  • Metrics extraction: Convert application logs into counts suitable for dashboards or alert thresholds.

  • Data conversion: Reformat delimited output for legacy scripts or downstream tools.

For high-volume production logs, prefer a single awk invocation over starting one process per line. Measure transformation latency and backpressure, and avoid duplicating sensitive logs into temporary files. Use jq for JSON, CSV-aware tools for irregular CSV, and database-native tooling for structured queries.

Code Example

The following creates representative application events and normalizes them into pipe-separated fields:

cat > /tmp/app-events.log <<'EOF'
2025-03-08T10:15:00Z INFO 200 /health
2025-03-08T10:15:03Z ERROR 500 /api/users
2025-03-08T10:15:07Z WARN 404 /missing
2025-03-08T10:15:10Z ERROR 502 /api/users
2025-03-08T10:15:12Z ERROR 500 /login
EOF

sed -E 's/^([^ ]+) ([^ ]+) ([0-9]{3}) ([^ ]+)$/\1|\2|\3|\4/' \
  /tmp/app-events.log | tee /tmp/app-events.psv

Expected output:

2025-03-08T10:15:00Z|INFO|200|/health
2025-03-08T10:15:03Z|ERROR|500|/api/users
2025-03-08T10:15:07Z|WARN|404|/missing
2025-03-08T10:15:10Z|ERROR|502|/api/users
2025-03-08T10:15:12Z|ERROR|500|/login

Count server failures by endpoint:

awk -F'|' '
  $3 >= 500 { failures[$4]++ }
  END {
    for (path in failures) {
      print path, failures[path]
    }
  }
' /tmp/app-events.psv | sort

Expected output:

/api/users 2
/login 1

The awk array iteration order is unspecified, so sort is used for deterministic output.

Best Practices and Pitfalls

  • Test expressions: Use sample data and compare output before modifying production files.

  • Back up in-place edits: GNU sed -i.bak -E 's/old/new/g' file creates file.bak; validate the result and securely remove obsolete backups.

  • Preserve originals: Keep authoritative source data unchanged and derive transformed copies for processing.

  • Check pipeline failures: In Bash, use set -o pipefail so a failed sed or awk is not hidden by a successful final command.

  • Avoid unsafe interpolation: Never concatenate untrusted input directly into an awk program or sed expression. Pass data through files or controlled fields and validate it first.

  • Handle line endings: CRLF input can leave a trailing \r; normalize it only when the format permits.

  • Control resource usage: sed and awk are streaming tools, but large awk associative arrays can consume substantial memory.

  • Respect file permissions: Do not broaden access with chmod 777; protect logs and configuration files using least-privilege ownership and modes.

70

What do Linux file descriptors 0, 1, and 2 represent?

Linux file descriptors 0, 1, and 2 are the conventional process descriptors for standard input, standard output, and standard error. They are integer indexes into a process’s table of open files, allowing programs to read input and write results or diagnostics without knowing whether the underlying destination is a terminal, file, pipe, socket, or service manager.

Read full answer

Applies toLinux (Ubuntu 22.04/24.04, RHEL 9)

Standard Descriptor Meanings

Descriptor Conventional name Direction Typical destination
0 Standard input (stdin) Read Keyboard, terminal, file, pipe, socket
1 Standard output (stdout) Write Terminal, log file, pipe, socket
2 Standard error (stderr) Write Terminal, log file, pipe, socket

The kernel does not require these descriptors to exist or have these meanings. Programs may close them, replace them, or open additional descriptors such as 3, 4, and 5. The convention exists so standard Unix tools can communicate predictably.

How File Descriptors Work

A descriptor is a small integer maintained in the process’s file-descriptor table. It references an open file description, which contains information such as:

  • The underlying file, socket, device, or pipe

  • The current read/write position

  • Open status flags

  • References to an inode or driver object

Duplicating a descriptor with dup() or dup2() creates another descriptor referencing the same open file description. Consequently, duplicated descriptors share the file offset. Each descriptor still has its own descriptor flags, such as FD_CLOEXEC.

A terminal session commonly has all three descriptors connected to the controlling terminal. Running ls -l /proc/$$/fd would show symlinks like 0 -> /dev/pts/0, 1 -> /dev/pts/0, 2 -> /dev/pts/0. In a service, they might instead point to a socket, journal, file, or /dev/null.

Redirection and Ordering

The shell changes descriptor destinations using redirection:

  • >output.log redirects descriptor 1 to the file.

  • 2>error.log redirects descriptor 2 to a separate file.

  • 2>&1 duplicates the current destination of descriptor 1 into descriptor 2.

  • 1>&2 duplicates the current destination of descriptor 2 into descriptor 1.

  • >/dev/null discards the selected output stream.

Redirection order matters because the shell processes redirections from left to right:

  • command >application.log 2>&1 – both stdout and stderr go to application.log.

  • command 2>&1 >application.log – stdout goes to application.log, but stderr remains on the terminal (because 2>&1 first makes descriptor 2 share the terminal’s destination, then >application.log changes descriptor 1 to the file).

Pipes and Production Behavior

In a pipeline, the producer’s standard output becomes the consumer’s standard input. For example, printf 'one\ntwo\n' | wc -l outputs 2. The producer’s stderr is normally inherited separately rather than automatically sent into the pipe. Use 2>&1 when the consumer should receive diagnostic output too. In Bash, enable set -o pipefail so a failure in an upstream pipeline component is not hidden by a successful final command.

For systemd services, stdout and stderr are normally captured by journald, where they can be queried with journalctl. Separating the streams is useful when only normal output should enter an application data pipeline while errors require alerts. Avoid placing credentials or sensitive payloads in either stream because service logs often have broader access and longer retention than application data.

Operational Practices

  • Inspect effective descriptors: Use ls -l /proc/<pid>/fd or lsof -p <pid> during incident analysis.

  • Keep diagnostics separate: Preserve stderr for actionable errors rather than mixing it into normal output.

  • Use process supervision: Let systemd, containers, or CI systems capture and rotate logs instead of writing unmanaged files.

  • Check exit status: A zero exit code does not guarantee that an error message was absent.

  • Close sensitive descriptors: Prevent unnecessary credentials or network connections from leaking into child processes.

  • Do not assume descriptor stability: After redirection, verify the actual destinations with /proc when behavior appears incorrect.

71

How to check boot process and boot logs in Linux?

Boot diagnostics combine the systemd journal for userspace events, dmesg for the current kernel ring buffer, and systemd-analyze for boot timing and dependency analysis. Together they show which units started, what the kernel reported, and where startup latency accumulated.

Read full answer

Applies toLinux (Ubuntu 22.04/24.04, RHEL 9) with systemd 249/252/255

Boot Sources and Process

After firmware and the bootloader load the kernel, systemd becomes PID 1, starts units according to their dependencies, and reaches the default target. systemctl get-default identifies that target, while systemctl is-system-running reports running, degraded, or another state.

Source What it provides Important limitation
journalctl Structured systemd, service, kernel, priority, and boot-filtered records Previous boots require persistent or otherwise retained journals
dmesg Kernel messages from the current boot, including drivers and hardware probes It does not retain messages across reboots
/var/log/syslog, /var/log/messages, /var/log/boot.log Distribution-specific text logs Contents and generation vary; early messages may be absent
last -x Recorded login, reboot, and shutdown sessions from wtmp/btmp It is a boot history, not a detailed event log

With Storage=auto, systemd stores journals persistently under /var/log/journal/ when that directory exists; otherwise, it uses volatile storage under /run/log/journal/. Do not assume persistence without checking the directory and journald configuration.

Reading Boot Logs

  • journalctl -b 0 shows the current boot; journalctl -b -1 shows the previous retained boot.

  • journalctl --list-boots identifies available boot sessions.

  • journalctl -b 0 -k shows kernel messages associated with that boot.

  • journalctl -b 0 -u ssh.service filters by unit on Ubuntu; RHEL normally uses sshd.service.

  • journalctl -b 0 -p 0..3 shows emergency through error priorities.

  • journalctl -f follows new records, which is useful while reproducing a service failure.

  • dmesg -T prints human-readable kernel timestamps, while dmesg --level=err,warn restricts output to warning and error levels.

Do not clear the kernel ring buffer during diagnosis. dmesg -C permanently discards messages that may explain the failure.

Measuring and Diagnosing Startup

  • systemd-analyze reports total kernel and userspace startup time.

  • systemd-analyze blame ranks units by initialization time; a long duration does not always mean the unit caused the delay.

  • systemd-analyze critical-chain identifies the dependency chain containing the largest initialization time.

  • systemctl --failed lists failed units.

  • systemd-analyze plot > boot.svg produces an SVG boot timeline for offline analysis.

Common delay sources include storage discovery, filesystem checks, encrypted-volume unlocking, generated device units, network-online ordering, and unit timeouts.

Code Example

# Boot target, overall state, and failed units
sudo systemctl get-default
sudo systemctl is-system-running
sudo systemctl --failed

# Current, retained, and kernel-specific journal records
sudo journalctl -b 0 --no-pager
sudo journalctl --list-boots
sudo journalctl -b -1 --no-pager       # Run if a previous boot is listed
sudo journalctl -b 0 -k --no-pager
sudo journalctl -b 0 -p 0..3 --no-pager

# Distribution-specific SSH unit
sudo journalctl -b 0 -u ssh.service --no-pager   # Ubuntu

# Current kernel messages and boot-session history
sudo dmesg -T
sudo dmesg --level=err,warn
last -x | head

# Optional: persist journals and apply conservative limits
sudo mkdir -p /var/log/journal
sudo install -d -m 0755 /etc/systemd/journald.conf.d
sudo tee /etc/systemd/journald.conf.d/limits.conf >/dev/null <<'EOF'
[Journal]
Storage=auto
SystemMaxUse=1G
MaxFileSec=1month
EOF
sudo systemctl restart systemd-journald
$ systemctl get-default
graphical.target
$ systemctl is-system-running
running
$ systemctl --failed
0 loaded units listed.
$ systemd-analyze
Startup finished in 3.214s (kernel) + 1.856s (userspace) = 5.070s.

Production Troubleshooting and Hardening

  • For a failed boot, inspect journalctl -k -b -1, persistent console recordings, and /sys/fs/pstore. An early panic may occur before journald is available.

  • Apply journal changes through reviewed configuration management. SystemMaxUse limits disk consumption; MaxFileSec controls retention and rotation.

  • Preserve journald’s intended ownership, permissions, access groups, and SELinux labels. Do not blanket-chmod journal files to 0600; instead, grant controlled access through systemd-journal group membership and ACLs.

  • Use the distribution’s logrotate policy for text logs; journald manages its own storage.

  • Forward important logs to a protected central collector and retain off-host copies so a node failure does not destroy incident evidence.

  • Monitor journalctl --disk-usage, filesystem capacity, failed units, boot duration, and kernel errors.

  • On RHEL, investigate SELinux denials with ausearch and audit2why; review any proposed audit2allow rules before applying them. On Ubuntu, inspect AppArmor denials instead. Do not disable MAC security as a troubleshooting shortcut.

72

How does GRUB enable recovery from a broken Linux boot?

GRUB (GNU GRand Unified Bootloader) enables recovery by retaining boot choices after the operating system fails: it can load another kernel or recovery mode, pass recovery parameters to systemd, and provide a command line for diagnosing boot problems. GRUB does not repair a damaged filesystem by itself; that requires filesystem tools, package repair, or reinstallation of the bootloader.

Read full answer

Applies toGRUB 2 on Ubuntu 22.04/24.04 and RHEL 9

How GRUB Creates a Recovery Path

  • Firmware handoff: UEFI loads GRUB from the EFI System Partition, commonly mounted at /boot/efi; BIOS loads its initial boot code from the disk’s boot area.

  • Boot configuration: GRUB reads a generated menu configuration. Ubuntu normally uses /boot/grub/grub.cfg, while RHEL 9 uses /boot/grub2/grub.cfg. These files are generated from /etc/default/grub and scripts under /etc/grub.d/.

  • Kernel loading: Each menu entry identifies a kernel such as /boot/vmlinuz-<version> and an initial RAM filesystem such as /boot/initrd.img-<version>. The initial RAM filesystem can load storage drivers and contain early recovery tools.

  • System startup: Under a normal Ubuntu or RHEL configuration, the kernel starts systemd as PID 1. GRUB passes kernel parameters such as the root filesystem UUID and the target to activate through systemd.unit.

Recovery Options Provided by GRUB

  • Older kernel: The Advanced options menu can contain previous kernels, allowing boot with a known-working kernel or initramfs. The entry exists only if that kernel remains installed and GRUB regenerated its configuration.

  • Recovery mode: A recovery entry can start the distribution’s rescue tooling while retaining the installed root filesystem.

  • Editable boot entries: Pressing e exposes the kernel command line. Removing quiet splash exposes diagnostic output, and adding systemd.unit=rescue.target starts systemd’s rescue target. Pressing Ctrl+X boots the edited entry.

  • Manual kernel selection: GRUB’s command line can load a kernel and initramfs manually when an existing menu entry is incorrect. This is an emergency technique, not a substitute for regenerating the configuration.

  • Fallback to external media: If GRUB itself cannot read its configuration or filesystems, recovery requires booting Ubuntu live media, another installation, or provider rescue facilities and repairing the installation from there.

Practical Recovery Workflow

  1. Preserve evidence and data: Prefer the provider’s serial console, take a snapshot if possible, and avoid repeated boot attempts that could worsen filesystem damage.

  2. Try available GRUB entries: Select an older kernel or recovery mode before modifying the installation.

  3. Isolate the failing layer: Determine whether the failure occurred in firmware, GRUB, kernel initialization, the root filesystem, or a systemd service.

  4. Collect the correct logs: In a live environment, dmesg -T describes the live kernel, not the failed installation. To inspect the installation’s persistent journal, mount it and use journalctl --directory=/mnt/var/log/journal -b -1; this works only when persistent journals were enabled.

  5. Repair storage safely: Identify filesystems with lsblk -f and blkid, activate any required LVM volumes, and run the appropriate repair tool while the target filesystem is unmounted. Repairing a mounted filesystem can cause data loss.

  6. Repair GRUB from a mounted installation: Mount the target root, any separate /boot and EFI partitions, and the required pseudo-filesystems, then enter a chroot. Regenerate configuration with update-grub on Ubuntu or grub2-mkconfig -o /boot/grub2/grub.cfg on RHEL 9. Reinstall GRUB with grub-install or grub2-install only when its files are missing or damaged.

  7. Verify recovery: Reboot normally, then check systemctl --failed, journalctl -b, filesystem usage, disk-health reports, and boot timing before closing the incident.

Failure Boundaries

Observed symptom What GRUB can do Additional recovery required
Firmware does not list a boot device Nothing Fix firmware boot order, hardware, or EFI configuration
GRUB menu is missing Boot using live media Restore or reinstall GRUB and regenerate grub.cfg
GRUB menu appears but no kernels are listed Inspect GRUB’s stored paths Restore /boot, install a matching kernel/initramfs, and regenerate the menu
Kernel panics before the root filesystem is usable Try another kernel or edit parameters Repair filesystem, encryption mappings, storage drivers, or initramfs
The system reaches emergency.target Use rescue.target Correct fstab or the failed systemd service

Production and Security Practices

  • Back up /boot, /etc/fstab, /etc/default/grub, /etc/grub.d/, and relevant storage metadata; use provider snapshots for full-disk rollback.

  • Keep provider serial consoles and out-of-band recovery access available for production hosts.

  • Monitor failed units, filesystem errors, boot-time regressions, and repeated kernel fallbacks through the journal and monitoring platform.

  • Preserve Secure Boot by using signed distribution packages rather than disabling it or replacing GRUB with an unsigned build.

  • Do not weaken SELinux to hide permission problems; restore the correct labels and modes.

  • Never use chmod 777 as a generic recovery shortcut. Repair ownership, modes, and filesystem metadata instead.

73

How do Linux runlevels and systemd targets work?

Runlevels are numbered boot modes inherited from SysV init. systemd targets are the modern, dependency-based equivalent for selecting boot states, although some targets have no exact runlevel counterpart.

Read full answer

Applies tosystemd on Ubuntu 24.04 and RHEL 9; historical SysV init

Historical Runlevels

SysV init used /etc/inittab to select a default runlevel from 0 through 6. PID 1 started or stopped services through symlinks in /etc/rcN.d, where N was the selected runlevel.

  • Runlevel 0: Halt the system.

  • Runlevel 1: Single-user maintenance.

  • Runlevel 2: Multi-user mode without a graphical interface; networking distinctions varied by distribution.

  • Runlevel 3: Full multi-user, typically server-oriented mode.

  • Runlevel 4: Unused or site-specific.

  • Runlevel 5: Graphical desktop.

  • Runlevel 6: Reboot.

Modern systemd hosts support legacy SysV init scripts through systemd-sysv-generator; /etc/inittab does not control the boot process.

How systemd Targets Work

A systemd target is a unit that groups services, sockets, mounts, timers, and other targets into a boot state.

  • Wants=: Queues activation of another unit without requiring that unit to activate successfully.

  • Requires=: Starts another unit and propagates an explicit stop. If the required unit fails and an After= ordering dependency is also present, the requiring unit will not start.

  • After=: Orders startup after another unit when both jobs are pending; it does not start that unit.

  • Before=: Creates the reverse ordering relationship.

  • WantedBy=: Used under [Install] to create symlinks that pull a unit into another target.

The persistent default is normally selected by /etc/systemd/system/default.target. An active target means that the target’s own activation job succeeded; it does not prove that every application or weakly required dependency is healthy.

Runlevel Mapping

SysV runlevel Traditional purpose Closest systemd target
0 Halt poweroff.target or halt.target
1 Single-user recovery rescue.target; emergency.target is more minimal
2 Multi-user, non-graphical multi-user.target
3 Full multi-user server multi-user.target
4 Unused or custom No standard target
5 Graphical desktop graphical.target
6 Reboot reboot.target

kexec.target is a separate mechanism for replacing the running kernel, not an ordinary runlevel counterpart.

Starting, Isolating, and Selecting Targets

Command Behaviour
systemctl start TARGET Starts the target and its dependencies without stopping unrelated units from the previous target
systemctl isolate TARGET Activates the target and stops units no longer required by it
systemctl set-default TARGET Changes the persistent default for the next boot

Code Example

Inspect targets, change the persistent default, and switch the active target:

$ systemctl get-default
multi-user.target

$ systemctl list-units --type=target --state=active
basic.target            loaded active active
multi-user.target       loaded active active
sysinit.target          loaded active active

# Change the persistent default; systemctl may report the symlink it created.
$ sudo systemctl set-default multi-user.target

# Activate immediately and stop units no longer required by this target.
$ sudo systemctl isolate multi-user.target

# Inspect direct and reverse dependency relationships.
$ systemctl list-dependencies multi-user.target
$ systemctl list-dependencies --reverse multi-user.target

--state=active is appropriate because targets normally use the active state rather than the running state. Before using isolate, ensure ssh.service remains enabled or is otherwise required by the destination target; otherwise, a remote session can be terminated.

Production Practices and Common Pitfalls

  • Inspect before changing: Use systemctl list-dependencies, systemctl cat, and systemctl --failed. Target activation is not an application health check.

  • Keep recovery access: Maintain console or out-of-band access when isolating targets, especially on remote production hosts.

  • Monitor services independently: Check application endpoints and journalctl -b -u app.service. Previous-boot data from journalctl -b -1 requires persistent journal storage.

  • Reload, do not restart: Run systemctl daemon-reload after editing units; it only rereads unit configuration.

  • Preserve package files: Keep local units in /etc/systemd/system; treat /usr/lib/systemd/system or Ubuntu’s /lib/systemd/system as package-managed.

  • Back up boot configuration: Include the default-target symlink and local unit files in configuration backups and disaster-recovery procedures.

  • Validate changes: Test target changes on non-production hosts and verify required units with systemd-analyze verify.

  • Retain mandatory access controls: Never use chmod 777 or disable SELinux or AppArmor to mask permission failures.

74

How do you configure Linux hostname, timezone, and time synchronization?

Use hostnamectl for the system hostname and timedatectl for timezone and clock settings. Configure systemd-timesyncd on Ubuntu or chrony on RHEL to synchronize time over NTP; only one time-synchronization daemon should run on a host.

Read full answer

Applies toLinux on Ubuntu 24.04 (systemd 255) and RHEL 9 (systemd 252)

How It Works

  • Hostname: hostnamectl updates the static and transient hostnames and writes /etc/hostname. It does not automatically create matching DNS records.

  • Timezone: timedatectl set-timezone updates /etc/localtime and /etc/timezone. UTC is usually preferable on servers; convert to local time only when displaying it.

  • RTC mode: Keep the hardware real-time clock in UTC with timedatectl set-local-rtc 0, avoiding daylight-saving ambiguity.

  • Time synchronization: systemd-timesyncd provides a lightweight SNTP client, while chrony supports NTP client/server operation, better diagnostics, and controlled clock stepping.

Code Example

Set the hostname, local hostname mapping, timezone, and RTC mode:

sudo hostnamectl set-hostname web-01
sudo timedatectl set-timezone Etc/UTC
sudo timedatectl set-local-rtc 0
sudoedit /etc/hosts

Use this /etc/hosts entry:

127.0.1.1 web-01 web-01.example.com

On Ubuntu, configure systemd-timesyncd:

sudo apt update
sudo apt install -y systemd-timesyncd
sudo systemctl enable --now systemd-timesyncd
sudo timedatectl set-ntp true
sudoedit /etc/systemd/timesyncd.conf

Apply the following configuration, replacing the placeholder with an approved organization or provider:

[Time]
NTP=time.example.com
FallbackNTP=pool.ntp.org

Restart it and check the service:

sudo systemctl restart systemd-timesyncd
systemctl status systemd-timesyncd --no-pager

On RHEL, configure chrony instead:

sudo dnf install -y chrony
sudo systemctl enable --now chronyd
sudoedit /etc/chrony.conf

Example /etc/chrony.conf settings:

pool time.example.com iburst
makestep 1.0 3
rtcsync

The makestep 1.0 3 directive steps the clock if it differs by more than one second during the first three updates. After initial synchronization, chrony normally slews the clock gradually rather than causing disruptive jumps.

sudo systemctl restart chronyd
systemctl status chronyd --no-pager

Do not run systemd-timesyncd and chrony simultaneously. If switching implementations, stop and disable the other daemon first.

Validate and Monitor

Representative Ubuntu output:

$ hostnamectl --static
web-01

$ timedatectl status
               Local time: Fri 2025-03-14 14:32:10 UTC
           Universal time: Fri 2025-03-14 14:32:10 UTC
                 RTC time: Fri 2025-03-14 14:32:10
                Time zone: Etc/UTC (UTC, +0000)
System clock synchronized: yes
              NTP service: active
           RTC in local TZ: no

For chrony, inspect synchronization quality:

$ chronyc tracking
Reference ID    : A9FEA9FE (time.example.com)
Stratum         : 3
Ref time (UTC)  : Fri Mar 14 14:32:08 2025
System time     : 0.000012093 seconds fast of NTP time
Leap status     : Normal

Useful production checks include:

timedatectl show --property=NTPSynchronized --property=NTP
chronyc sources -v
journalctl -u systemd-timesyncd --since today
journalctl -u chronyd --since today

Alert on sustained clock offsets, failed NTP reachability, or System clock synchronized: no.

Production and Security

  • Store hostname and timezone settings in image building or configuration management so rebuilt hosts remain consistent.

  • On cloud instances, configure the hostname through cloud-init or set preserve_hostname: true in /etc/cloud/cloud.cfg before provisioning; cloud-init may otherwise replace it.

  • Prefer a redundant internal NTP service with multiple upstream sources. Public NTP pools are useful for small installations but provide weaker operational control.

  • Restrict UDP port 123 to required peers. NTP servers should not be exposed unnecessarily because open services can be abused for reflection and amplification.

  • Avoid obsolete tools such as ntpdate. Large clock corrections can disrupt TLS validation, distributed locks, scheduled jobs, log ordering, and databases.

Common Pitfalls

  • Changing only /etc/hostname: Use hostnamectl so systemd-managed state remains consistent.

  • Expecting DNS updates: A local hostname change does not create or modify external DNS records.

  • Using local time for storage: Store timestamps in UTC and apply the user’s timezone at presentation time.

  • Checking only service state: “Active” does not prove successful synchronization; verify synchronization status and source offsets.

  • Leaving multiple NTP daemons enabled: They can continuously fight over clock adjustments.

75

How do ulimit and cgroups limit Linux processes?

ulimit sets resource ceilings inherited by individual processes, while cgroups control and account for groups of processes. On modern Linux, systemd is usually the preferred interface for cgroup limits because it provides declarative configuration, monitoring, and lifecycle management.

Read full answer

Applies toLinux with cgroups v2 and systemd 249+ (Ubuntu 22.04/24.04; RHEL 9/Fedora)

How ulimit and cgroups differ

Mechanism Scope Common controls Main limitation
ulimit One process and normally its descendants Open files, user processes, virtual memory, CPU time Easily changed by the owning user; not a security boundary
cgroups v2 A group of processes Memory, CPU, I/O, task count Requires privileged or delegated cgroup access
systemd Service lifecycle using cgroups Resource limits, isolation, monitoring Controls apply through the unit and its cgroup

ulimit manages POSIX RLIMIT values. Each limit has a soft limit and a hard limit:

  • The soft limit is the value currently enforced.

  • The hard limit is the maximum the process may request.

  • An unprivileged process can lower either limit but normally cannot raise its hard limit.

  • Limits are inherited when a process is executed, but already-running children are not retroactively changed.

Important ulimit examples include:

  • -n: maximum number of open file descriptors.

  • -u: maximum processes for the real user ID.

  • -v: maximum virtual address space in KiB.

  • -t: accumulated CPU time in seconds.

What happens under the hood

With cgroups v2, processes are placed in a cgroup directory under the unified hierarchy, normally /sys/fs/cgroup. Controllers expose control files such as:

  • memory.max: maximum memory usage.

  • cpu.max: CPU quota and scheduling period.

  • pids.max: maximum number of tasks, including threads.

  • io.max: I/O bandwidth and IO-weight limits.

When a cgroup limit is exceeded, the behavior depends on the controller:

  • Memory: the cgroup’s memory allocation is reclaimed; if reclaim is insufficient, the kernel’s OOM killer may terminate a process.

  • CPU: tasks are throttled when they exhaust their quota.

  • Tasks: process or thread creation fails.

When a ulimit is exceeded, the affected system call fails:

  • File descriptors: open, socket, or accept fails with an error such as EMFILE.

  • RLIMIT_AS: memory mappings may fail even when physical memory is available because virtual address space is exhausted.

A ulimit failure generally affects one process, while a cgroup limit affects every process placed in that cgroup. Existing file descriptors are not closed when RLIMIT_NOFILE is lowered.

Configure limits with ulimit and systemd

The following commands set limits in the current shell. A new shell starts with the defaults again unless the limits are configured in a login profile or inherited from a parent process.

# Show the current soft and hard open-file limits
$ ulimit -Sn
4096
$ ulimit -Hn
4096

# Set both limits for this shell and future children
$ ulimit -n 4096
$ ulimit -u 512
$ ulimit -v 1048576

# Change the open-file limit of an existing process
$ pid=1234
$ sudo prlimit --pid "$pid" --nofile=4096:4096

# Verify the live process limits
$ sudo grep 'Max open files' /proc/1234/limits
Max open files            4096                 4096                 files

RLIMIT_NPROC is counted against processes owned by the same real user ID, not against a service cgroup. Consequently, it can affect unrelated programs owned by that user and is generally unsuitable as a precise service-level process limit.

Code Example

The following systemd unit combines POSIX limits with cgroup controls. The sample command is runnable; replace ExecStart with the actual workload.

sudo tee /etc/systemd/system/limited-app.service >/dev/null <<'EOF'
[Unit]
Description=Resource-limited demonstration service

[Service]
Type=simple
ExecStart=/usr/bin/sleep 300
LimitNOFILE=4096
LimitNPROC=512
MemoryMax=512M
CPUQuota=50%
TasksMax=256

[Install]
WantedBy=multi-user.target
EOF

sudo systemd-analyze verify /etc/systemd/system/limited-app.service
sudo systemctl daemon-reload
sudo systemctl enable --now limited-app.service

Inspect the resulting limits and cgroup:

$ systemctl show limited-app.service -p ControlGroup
ControlGroup=/system.slice/limited-app.service

$ cg=/sys/fs/cgroup/system.slice/limited-app.service
$ cat "$cg/memory.max"
536870912
$ cat "$cg/cpu.max"
50000 100000
$ cat "$cg/pids.max"
256

Here, MemoryMax=512M caps memory, CPUQuota=50% provides roughly half a CPU’s time, and TasksMax=256 caps total tasks. CPUQuota=50% does not mean 50% of every CPU; it represents 50% of one CPU’s scheduling capacity.

Production practices and failure modes

  • Measure before setting limits: Use historical peak usage, service-level objectives, and node capacity. Leave headroom for garbage collection, caches, and traffic spikes.

  • Monitor the actual controller state: Inspect memory.current, memory.events, cpu.stat, systemd-cgtop, and service logs with journalctl -u limited-app.service.

  • Recognize OOM termination: memory.events can show max, oom, and oom_kill events. A killed container or service should be investigated rather than simply given unlimited memory.

  • Treat limits as configuration: Store systemd units in version control, test them with systemd-analyze verify, and reload changes with systemctl daemon-reload.

  • Do not rely on ulimit for containment: A user can generally lower limits and may raise them if privileged. Use systemd sandboxing, dedicated users, namespaces, and cgroup delegation for stronger isolation.

  • Avoid broad root cgroup manipulation: Prefer systemd-run or unit properties over moving processes between manually created cgroups. Otherwise, services can escape their expected lifecycle and accounting.

76

How do Linux namespaces enable container isolation?

Linux namespaces provide isolated views of kernel resources, allowing processes to see separate process IDs, filesystems, network stacks, users, and other system state. Container runtimes combine namespaces with a root filesystem, capability restrictions, security policies, and cgroups to create isolated execution environments.

Read full answer

Applies toLinux kernel 5.6+; util-linux 2.37+; Ubuntu 24.04 and RHEL 9

How Namespaces Work

A namespace is a kernel mechanism that virtualizes selected system resources. For example, one process can have PID 1 inside its PID namespace while the host sees it as PID 12345.

The namespaces used by a process are visible through /proc/<pid>/ns/:

$ ls -l /proc/12345/ns/
lrwxrwxrwx 1 root root 64 Jan 15 10:20 pid -> pid:[4026531836]
lrwxrwxrwx 1 root root 64 Jan 15 10:20 net -> net:[4026531992]
lrwxrwxrwx 1 root root 64 Jan 15 10:20 mnt -> mnt:[4026531840]
lrwxrwxrwx 1 root root 64 Jan 15 10:20 user -> user:[4026531837]

Different namespace identifiers indicate that the processes belong to separate namespace instances.

Common Linux Namespace Types

Namespace Isolated resources Container use
PID Process IDs and process hierarchy The container’s main process becomes PID 1 inside its namespace
Mount Mount points and filesystem propagation Provides the container root filesystem and private mounts
Network Interfaces, routes, sockets, and firewall state Isolates each container’s network stack
UTS Hostname and domain name Each container can have a distinct hostname
IPC System V IPC and POSIX message queues Prevents cross-container IPC interference
User User and group ID mappings Supports rootless containers and reduces kernel-visible privilege
Cgroup Cgroup hierarchy view Gives the container a relative view of its cgroup resources
Time Selected virtual clocks Isolates timekeeping offsets from the host

PID and mount namespaces are especially important. The runtime places the container filesystem in a mount namespace and mounts a /proc instance associated with the container’s PID namespace, so the application sees its own process hierarchy.

How Container Runtimes Use Them

Docker, containerd, and Kubernetes-managed runtimes create or join namespaces when starting a container:

  1. The runtime prepares a root filesystem, often using OverlayFS for copy-on-write storage.

  2. It creates or joins PID, mount, network, IPC, UTS, and often cgroup namespaces, and may create a user namespace.

  3. It applies cgroup limits for CPU, memory, and other resources.

  4. It restricts capabilities and applies seccomp, AppArmor, or SELinux policy.

  5. It launches the application inside those namespaces.

Namespaces isolate visibility, but they are not complete security boundaries. A privileged container, excessive Linux capabilities, or unsafe device access can weaken isolation.

Security and Operational Boundaries

  • User namespaces: Map an unprivileged host user to root inside the container when permitted by host policy. Root inside the namespace does not automatically mean root on the host.

  • Network namespaces: Give each container a private virtual network stack. Avoid hostNetwork, hostPID, and privileged containers in production because they intentionally weaken isolation.

  • Cgroups: Enforce resource limits; namespaces alone generally do not enforce CPU or memory quotas.

  • Mounts: Avoid exposing sensitive host paths. Mount only required directories, preferably read-only, and avoid host filesystem or container-runtime socket mounts.

  • Observability: Administrators need host-level visibility into container processes, logs, metrics, and cgroups. Kubernetes audit logs, systemd/journald, and runtime inspection commands are commonly used during incidents.

Code Example

The following creates isolated PID, mount, UTS, and user namespaces. It uses sudo because Ubuntu 24.04 can block unprivileged user-namespace creation through AppArmor; where policy permits it, omit sudo to exercise the rootless path.

$ printf 'Parent user namespace: %s\n' "$(readlink /proc/self/ns/user)"

$ sudo unshare --fork --pid --mount-proc --mount --uts \
    --user --map-root-user sh -c '
        printf "Child user namespace: %s\n" "$(readlink /proc/self/ns/user)"
        hostname isolated-host
        printf "Container hostname: %s\n" "$(hostname)"
        printf "PID inside namespace: %s\n" "$$"
    '

Example output:

Parent user namespace: user:[4026531837]
Child user namespace: user:[4026532992]
Container hostname: isolated-host
PID inside namespace: 1
  • --fork: Starts a child process so the shell can become PID 1 in the new PID namespace.

  • --pid: Creates a separate process ID namespace.

  • --mount-proc: Mounts a /proc appropriate for the new PID namespace.

  • --mount: Creates a separate mount namespace.

  • --uts: Creates an independent hostname namespace.

  • --user --map-root-user: Creates a user namespace and maps the invoking user to UID 0 inside it. Under sudo, that invoking user is host root, so this does not demonstrate privilege reduction.

The exact namespace numbers vary between executions.

Production Best Practices

  • Use rootless containers or non-root users where the application supports them.

  • Drop unnecessary capabilities with container security profiles.

  • Keep SELinux or AppArmor enabled; never disable it merely to resolve permission errors.

  • Apply CPU and memory limits through cgroups.

  • Store secrets outside images and inject them through an appropriate secret mechanism.

  • Restrict host mounts, devices, and access to container-runtime sockets.

  • Monitor container restarts, OOM kills, throttling, and namespace or cgroup placement.

  • Test upgrades because kernel, runtime, and security-policy changes can affect namespace behavior.

77

How do you harden a fresh Linux server for production?

Harden a fresh Linux server through a staged, reversible process: preserve console access, minimize identity and network exposure, patch packages, restrict services and filesystem permissions, then verify logging, monitoring, and recovery. The goal is least privilege with tested operational resilience, not simply closing ports.

Read full answer

Applies toLinux (Ubuntu 24.04 LTS, RHEL 9.x)

Secure the Boot Order

  • Establish recovery access: Keep the provider console, IPMI, or out-of-band console available while changing SSH or firewall rules.

  • Create a recovery point: Take a provider snapshot before major changes. For a physical server, test rescue media and ensure the boot disk and firmware are accessible.

  • Patch before exposing the server: Update the base image, kernel, bootloader, and security packages. Schedule a controlled reboot when /var/run/reboot-required.pkgs indicates a kernel update.

  • Install only required packages: On Ubuntu use apt; on RHEL/Fedora use dnf. Remove unnecessary compilers, daemons, and management agents unless they are actively used.

  • Keep MAC security controls enabled: Use AppArmor on Ubuntu and SELinux in enforcing mode on RHEL. Fix policy denials rather than disabling the framework.

Identity, SSH, and Permissions

  • Use individual administrator accounts: Create named accounts with sudo access through /etc/sudoers using visudo. Do not share accounts or give broad sudo membership.

  • Separate service identities: Run services under dedicated, non-login accounts such as app or svc; their shell should normally be /usr/sbin/nologin.

  • Disable SSH passwords: Require hardware-backed or centrally managed public keys, disable root login, and restrict access to a dedicated SSH group.

  • Protect private material: Keep keys in a secrets manager, use file mode 0600, and avoid embedding credentials in images, shell history, or process arguments.

  • Understand the filesystem hierarchy: /etc contains configuration, /home user data, /var persistent state and logs, /run runtime state, and /tmp temporary files. Protect writable application directories rather than applying blanket permissions.

  • Never use chmod 777: It grants every local user write access. Prefer chown, narrowly scoped chmod, ACLs, and a default umask 077 for private files.

Network and Service Exposure

  • Layer controls: Apply cloud security groups, host firewalls, and application authentication independently.

  • Default-deny inbound traffic: Permit only required protocols and sources. Account for IPv4 and IPv6 separately.

  • Discover listeners: Run sudo ss -lntup, ip -br address, and ip route. Investigate every externally reachable process and bind sensitive services to private interfaces where possible.

  • Manage services with systemd: Disable unused units using sudo systemctl disable --now service. Run systemd-analyze security service and apply suitable sandboxing such as NoNewPrivileges=true, PrivateTmp=true, and ProtectSystem=strict.

  • Handle termination correctly: Systemd normally sends SIGTERM for graceful shutdown and SIGKILL after TimeoutStopSec. Configure application draining and readiness behavior accordingly.

  • Control egress too: Restrict outbound destinations when practical, especially access to cloud metadata endpoints such as IMDS.

Storage, Performance, and Recovery

  • Monitor capacity and health: Check lsblk -f, df -hT, df -i, and findmnt. Track disk space, inode exhaustion, I/O latency, SMART or NVMe health, and unclean filesystem mounts.

  • Measure saturation: Use uptime, free -h, vmstat 1 5, iostat -xz 1 5, and ss -s. Alert on sustained CPU throttling, memory pressure, swap activity, disk latency, and socket backlog growth.

  • Persist and centralize logs: Configure journald with Storage=persistent in /etc/systemd/journald.conf.d/99-production.conf, then restart systemd-journald. Forward logs to a protected SIEM over authenticated TLS.

  • Enable auditability: Run auditd on Ubuntu and RHEL for changes to identities, privileged commands, and security configuration. Central retention protects evidence from local tampering.

  • Test recovery: Maintain encrypted storage and versioned backups following the 3-2-1 model. Test application-consistent restores regularly; volume snapshots alone are not backups.

  • Define failure behavior: Alert on failed updates, stopped services, reboot-required state, full disks, certificate expiry, backup failures, and authentication anomalies.

Code Example

The following establishes a key-only administrator and restricts SSH on Ubuntu. Run it from the directory containing the administrator’s public key, and keep an existing session open.

sudo apt update
sudo apt full-upgrade -y
sudo apt install -y ufw unattended-upgrades auditd
sudo systemctl enable --now unattended-upgrades auditd

sudo useradd --create-home --shell /bin/bash deploy
sudo groupadd --system ssh-users
sudo usermod -aG ssh-users deploy
sudo install -d -m 0700 -o deploy -g deploy /home/deploy/.ssh
sudo install -m 0600 -o deploy -g deploy approved_deploy_key.pub \
  /home/deploy/.ssh/authorized_keys

sudo install -d -m 0755 /etc/ssh/sshd_config.d
sudo tee /etc/ssh/sshd_config.d/00-production.conf >/dev/null <<'EOF'
PermitRootLogin no
PasswordAuthentication no
KbdInteractiveAuthentication no
PubkeyAuthentication yes
X11Forwarding no
MaxAuthTries 3
AllowGroups ssh-users
EOF

sudo sshd -t && sudo systemctl reload ssh
sudo sshd -T | grep -E 'permitrootlogin|passwordauthentication|pubkeyauthentication'
permitrootlogin no
passwordauthentication no
pubkeyauthentication yes

Then configure the firewall before changing the SSH port:

sudo ufw default deny incoming
sudo ufw default allow outgoing
sudo ufw allow 22/tcp
sudo ufw enable
sudo ufw status verbose
Default: deny (incoming), allow (outgoing), disabled (routed)
22/tcp   ALLOW IN   Anywhere
Status: active

On RHEL/Fedora, use dnf upgrade --refresh, firewalld, and sudo systemctl reload sshd. Apply firewall-cmd --permanent --add-service=ssh before reloading the firewall.

Production Best Practices

  • Test every change through the cloud console or a second administrator session; an invalid SSH, firewall, or fstab change can make a server inaccessible.

  • Verify configuration with sshd -t, ufw status, and systemctl status rather than assuming commands succeeded.

  • Roll updates through a small canary fleet, preserve rollback images, and reboot deliberately after kernel changes.

  • Integrate host metrics, logs, vulnerability findings, and inventory into centralized monitoring.

  • Periodically review users, SSH keys, sudo rules, listening ports, enabled services, secrets, firewall rules, and backup restoration results.

78

How do you troubleshoot a suddenly slow Linux server?

Investigate a suddenly slow Linux server with staged performance triage: confirm scope and recent changes, measure CPU, memory, storage, and network behavior, attribute pressure to a process or dependency, and preserve evidence before remediation. In production, correlate host metrics with application, database, and network telemetry rather than restarting services immediately.

Read full answer

Applies toLinux (Ubuntu 22.04/24.04, RHEL 9) with systemd and sysstat 12.x

Triage Order and Key Signals

  1. Establish impact: Determine whether all users or only one endpoint, tenant, region, or request type is affected. Check latency, errors, queue depth, and dependency latency.

  2. Correlate changes: Review deployments, configuration changes, packages, certificate or DNS expiry, traffic spikes, and provider events.

  3. Measure against a baseline: One high metric does not prove saturation.

  4. Load average: Counts runnable and uninterruptible tasks; compare it with the CPU count and normal load.

  5. CPU: Sustained low idle time, cgroup throttling, or CPU steal on a VM indicates compute pressure.

  6. Memory: Distinguish cache from committed memory; check swap activity, cgroup limits, PSI, and OOM events.

  7. Disk: High await, growing queues, and processes in D state indicate blocked storage I/O.

  8. Network: Check interface errors, drops, retransmissions, connection queues, DNS latency, and upstream saturation.

Processes, systemd, and journald

Use ps, pidstat, top, and systemd-cgtop to attribute pressure. A process in D state is waiting for uninterruptible I/O and cannot terminate until that I/O completes.

Inspect services with systemctl status and journalctl. Journald uses /var/log/journal for persistent storage or /run/log/journal for volatile storage, depending on its configuration. Check journalctl --disk-usage and retention before logs consume the filesystem.

Send SIGTERM first for graceful shutdown. Wait for systemd’s TimeoutStopSec= or the application’s documented grace period; use SIGKILL only as a last resort because it prevents cleanup and may leave partial writes.

Storage, Filesystem, and Network Context

Use df -hT for filesystem capacity and df -ih for inode exhaustion. Full filesystems, exhausted inodes, stale NFS mounts, or read-only remounts can cause broad slowdown. Use du only on a constrained path because scanning large trees can worsen the incident.

The Filesystem Hierarchy Standard places configuration in /etc, mutable data and logs in /var, runtime state in /run, user data in /home, and third-party software in /opt. Do not place durable databases or application logs in /tmp.

Check ownership and permissions with ls -ld, stat -c '%U %G %a %n', and the service account. Use ss -lntp for listeners, ss -s for socket-state totals, ip route for routing, ip -s link for interface counters, dig for DNS, and openssl s_client for TLS checks.

Run diagnostics with minimum privilege. Prefer targeted sudo commands over a root shell, and never use chmod 777; it weakens confidentiality and integrity without fixing the underlying fault.

Production Observability

Monitor service RED signals—request rate, errors, and duration—and host USE signals—utilization, saturation, and errors. Include CPU throttling, memory pressure, disk latency, inode usage, network drops, load, and OOM kills.

Application logs should include request IDs, upstream timings, retries, and dependency failures. During an incident, retain timestamps, command output, metrics, and changes in a shared timeline.

Code Example

Use already-installed tools during the incident. On Ubuntu, vmstat comes from procps; on RHEL it comes from procps-ng. pidstat, iostat, and sar come from sysstat.

date -Is
uptime
vmstat 1
free -h
ps -eo pid,ppid,user,%cpu,%mem,stat,etimes,comm --sort=-%cpu | head -n 12
sudo pidstat -dru 1 3
sudo iostat -dx 1
df -hT
df -ih
sudo ss -lntp
sudo journalctl --since=-30min -p warning --no-pager
sudo journalctl -u nginx.service --since=-30min --no-pager -n 50

vmstat 1 samples once per second. pidstat -dru 1 3 reports disk, memory, and CPU activity three times; iostat -dx 1 adds extended device fields. df -ih includes inode usage, while journalctl -p warning selects warning and higher-priority records.

Relevant output excerpts:

$ vmstat 1
procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu-----
 r b   swpd   free   buff  cache   si  so    bi    bo    in   cs us sy id wa st
 6 3   4096  92144  1228  38412    0   0    12  342   18  204  3  2  4 90  1
 7 4   4096  90432  1216  37204    0   0    0  356   22  220  2  4  2 91  1

$ iostat -dx 1
Device            r/s      w/s      rkB/s      wkB/s      r_await      w_await      aqu-sz      %util
sda             12.00    340.00      96.00     3328.00        85.10        12.40        3.90     99.80
...
sda             13.00    351.00     100.00     3440.00        88.30        12.90        4.10     99.90

The blocked-task count, high I/O wait, extended-device latency, queue depth, and utilization indicate storage as the first bottleneck to investigate.

Remediation and Operational Practices

  • Fix the demonstrated cause: Scale capacity, correct a bad query, restore storage, renew an expired certificate, or fail over a degraded dependency.

  • Preserve evidence: Capture logs, metrics, process state, and recent configuration before restarting components.

  • Separate upgrades: Use apt on Ubuntu and dnf on RHEL; schedule tested updates outside the incident.

  • Protect backups: Verify coverage and restoration, coordinate database-consistent snapshots, and account for backup load.

  • Harden operations: Restrict log access, control journald retention, rate-limit noisy clients, and remove credentials from diagnostic output.

79

How to recover from a 100% full root filesystem on Linux?

Root-filesystem recovery requires stopping active writers, freeing or expanding storage, and then identifying what caused the disk to fill.

Read full answer

Applies toLinux on Ubuntu 22.04/24.04 and RHEL 9, using systemd with ext4 or XFS

Immediate Recovery

  1. Regain access: Use SSH, a virtual console, or out-of-band management. If the system will not boot, use rescue media or a live environment and mount the root filesystem.

  2. Confirm capacity and inode exhaustion: bash $ df -h / Filesystem Size Used Avail Use% Mounted on /dev/nvme0n1p1 20G 19G 0 100% / Run df -i / as well: a filesystem can have free data blocks but no free inodes.

  3. Stop active writers before deleting logs or temporary files. Identify the affected service with systemctl --failed, journalctl, and application logs.

  4. Free space immediately with non-critical package caches and expired journals: bash sudo apt-get clean sudo journalctl --vacuum-size=100M Use sudo dnf clean all on RHEL. Journal vacuuming removes archived journals; the active journal file is not truncated.

Diagnosing the Cause

Measure the root filesystem without crossing into other mounts:

$ sudo du -xhd1 / 2>/dev/null | sort -hr | head -n 11
19G /
12G /var
4.1G /usr
1.6G /home
1.2G /opt
542M /root
...
  • Deleted-open files: df may show space used even when du cannot find it. Run sudo lsof +L1; the space is reclaimed after the owning process closes the file or restarts.

  • Logs: Inspect /var/log, especially journal, application, and container logs. Verify that logrotate has rotate, compress, and retention rules; logrotate -f alone may not delete old files.

  • Containers and applications: Inspect /var/lib/docker, /var/lib/containers, and application data directories. docker system df reports Docker storage usage when Docker is installed.

  • Background jobs: Review /etc/cron.d/, /etc/cron.daily/, systemd timers, and scripts that generate large output.

  • Packages: Review removable packages with apt-get -s autoremove --purge on Ubuntu or list kernels with rpm -q kernel-core on RHEL and remove specific old versions with dnf remove kernel-core-<version>, never the running kernel (uname -r) or the known-good fallback.

Freeing and Expanding Storage

Use policy-aware temporary-file cleanup rather than deleting an active /tmp tree:

sudo systemd-tmpfiles --clean

For urgent cleanup, first identify and stop the service owning the files, then remove only verified stale artifacts under /tmp or /var/tmp.

For LVM, first confirm free capacity with vgs. If the volume group already has free extents:

sudo lvextend -L +20G /dev/vg0/root
sudo resize2fs /dev/vg0/root

For XFS, grow the mounted filesystem instead:

sudo xfs_growfs /

resize2fs can extend a mounted ext4 filesystem. xfs_growfs requires the mount point, such as / or /var, and XFS cannot be shrunk for rollback.

For cloud systems, create a provider-side snapshot, resize the existing root volume, and then start or reboot the instance as required: enlarge the existing EBS volume on AWS, managed disk on Azure, or persistent disk on GCP. Do not attach a different disk and expect it to become root automatically; boot-device mappings and fstab may still reference the original device.

After the backing volume grows, inspect it:

$ lsblk -f

If root is on a partition, use growpart from cloud-guest-utils on Ubuntu or cloud-utils-growpart on RHEL:

sudo growpart /dev/nvme0n1 1
sudo resize2fs /dev/nvme0n1p1

Skip growpart when the filesystem is directly on a block device. Use the provider’s documented rescan procedure when the running kernel does not immediately detect the new size.

Code Example

This conservative Ubuntu emergency script frees predictable cache and temporary data without automatically removing kernels:

#!/usr/bin/env bash
set -euo pipefail

echo "Filesystem usage:"
df -h /

echo "Largest root directories:"
du -xhd1 / 2>/dev/null | sort -hr | head -n 11 || true

# Remove downloaded APT packages.
apt-get clean

# Remove expired archived journal files.
journalctl --vacuum-size=100M

# Remove temporary files only when permitted by tmpfiles policy.
systemd-tmpfiles --clean

echo "Filesystem usage after cleanup:"
df -h /

Run it as root. On RHEL, replace apt-get clean with dnf clean all.

Production Practices and Pitfalls

  • Monitor both bytes and inodes using Prometheus node_exporter, Zabbix, or the platform’s monitoring service; alert before the filesystem reaches 100%.

  • Configure journald’s SystemMaxUse and SystemKeepFree, and ensure logrotate removes or compresses old logs.

  • Keep 10–20% free capacity, document emergency ownership and approval procedures, and test restoration procedures regularly.

  • Take provider-side snapshots before destructive cleanup or expansion when possible, but do not treat backups as a substitute for capacity monitoring.

  • Never use chmod 777 or disable SELinux/AppArmor to resolve disk pressure. These weaken security without directly reclaiming space.

  • Investigate the producer of the growth and record the incident; otherwise the same exhaustion is likely to recur.

80

What happens when Linux runs the ls -l command?

ls -l is a GNU coreutils command that displays directory entries in long format. The shell resolves and launches the executable, the Linux kernel loads it, and the program retrieves metadata for each entry before formatting the result.

Read full answer

Applies toGNU coreutils `ls` 8.32 on Ubuntu 22.04 and RHEL 9; GNU coreutils 9.4 on Ubuntu 24.04

How the Shell Starts the Command

  1. Parsing and expansion: Bash tokenizes ls -l and performs applicable expansions, including parameter, command-substitution, pathname, and—where enabled—alias expansion. It also prepares any redirections or pipelines present in the command line.

  2. Command lookup: Because ls is not a Bash builtin, Bash searches the directories in PATH, normally finding /usr/bin/ls. Bash may cache that lookup in its command hash table.

  3. Process creation: Bash normally forks a child process and waits for it. The child calls execve() to replace itself with ls, retaining the same process ID. Bash can optimize some final commands by executing them directly without an additional fork.

  4. Exit handling: The shell waits for the child and collects its exit status. A successful listing normally returns zero, while errors such as an inaccessible directory produce a nonzero status and a diagnostic on standard error.

Bash passes ls, -l, and any directory arguments as separate arguments. It does not interpret -l; that option is processed by ls itself.

How the Kernel Loads ls

The kernel’s ELF loader handles execve():

  • It resolves the executable through the filesystem and checks execute permission.

  • It verifies the ELF format and maps executable segments into the new process’s virtual address space.

  • Because ls is dynamically linked, the loader also maps the ELF interpreter, commonly /lib64/ld-linux-x86-64.so.2, and required shared libraries, including the C library.

  • Control eventually reaches the C runtime startup code and then GNU coreutils’ main().

A noexec mount, SELinux or AppArmor policy, missing execute permission, or invalid executable can prevent execution. Relevant errors include EACCES, ENOEXEC, and Operation not permitted; Bash may apply its script fallback when execve() reports ENOEXEC, so the final diagnostic can differ.

How the Directory Listing Is Built

With no path argument, ls uses its inherited working directory. Otherwise, it opens the requested directory and:

  • Reads entries through directory-reading system calls such as getdents64().

  • Retrieves metadata using calls such as fstatat()—commonly observed as newfstatat() in strace—or statx().

  • Resolves numeric user and group IDs into names through the Name Service Switch, normally using /etc/passwd and /etc/group plus any configured NSS providers.

  • Formats the metadata according to locale, terminal width, and command-line options.

ls -l does not read the regular files’ contents. Listing a directory requires read and execute permission on that directory; resolving a path also requires execute permission on each ancestor. A file’s own read permission is not required to retrieve its metadata.

Understanding the Long Format

A representative entry is -rw-r--r-- 1 alice developers 1250 Jul 14 09:32 report.txt.

Field Meaning
- or d Entry type, such as regular file or directory
rw-r--r-- Owner, group, and other permission bits
1 Number of hard links to the entry
alice Owning user name
developers Owning group name
1250 Size in bytes for a regular file
Jul 14 09:32 Modification time, rendered using the locale and local timezone
report.txt Entry name; a symbolic link is followed by -> target

For a symbolic link, the displayed size is normally the length of the target pathname in bytes, not the target file’s size. Special mode characters can indicate setuid, setgid, or sticky bits. The total line reports allocated filesystem blocks rather than the directory’s logical file-content size; its exact units and rounding follow the active ls options.

Production Failure Modes

  • PATH hijacking: An unexpected PATH can cause a malicious ls to run. Use type -a ls, command -v ls, or a trusted absolute path such as /usr/bin/ls in scripts.

  • Slow metadata or name resolution: NFS, FUSE, network home directories, and failing NSS lookups can make ls -l appear hung even when directory entries are available.

  • Concurrent changes: Entries can disappear or change between enumeration and metadata retrieval, producing diagnostics such as cannot access or No such file or directory.

  • Large directories: Extensive output and per-entry metadata retrieval can create substantial I/O and terminal overhead.

  • Signals and pipelines: Ctrl-C sends SIGINT to the foreground process group. If ls writes to a closed pipe, SIGPIPE can terminate it.

  • Operational visibility: ls is a short-lived process, not a systemd service, so its output does not automatically enter journald. During incidents, capture standard output and standard error separately and preserve the exit status.

Sign in to keep reading

You've opened 2 full answers. Sign in to read all 80 Linux answers, mark what you know and pick up where you left off.

Report a mistake

What's wrong?

All Cloud, DevOps & Infrastructure subjects