← Interview Prep

Linux Troubleshooting — Scenarios

Five reproducible Linux failures worked symptom→diagnosis→fix→prevent with exact commands: log forensics (aggregate with awk/sort/uniq), disk full but no files (deleted-but-open, lsof +L1, truncate via /proc/PID/fd), ENOSPC from inode exhaustion (df -i), Too many open files (EMFILE, RLIMIT_NOFILE, systemd LimitNOFILE), and undying zombies vs D-state.

Five reproducible Linux failures — the kind that page you at 2 a.m. — worked the way an interviewer wants to hear: symptom → diagnosis → fix → prevent, with the exact commands. Each is a hands-on scenario (break it, diagnose it, fix it); what matters is the reasoning, not memorizing output. Companion to Linux Basics and SRE Linux Q&A.

The meta-skill across all five: don't stare at output — reduce it to a distribution (filter → key → sort | uniq -c | sort -rn) or ask the kernel directly (/proc/<pid>/…, lsof, df -i). And always separate the symptom from the root cause.

1 · Log forensics — aggregate, don't read

Symptom: a 5xx spike since 14:00; the access log has tens of thousands of lines. Who's causing it?

Method: you never read a log line by line — you collapse it into counters. The pipeline is always filter → key → sort | uniq -c | sort -rn | head. For a standard access line IP - - [time] "METHOD path HTTP/1.1" status size, splitting on spaces: $1=client IP, $7=path (the quote makes it $7, not $6 — the classic trap), $9=status.
# how many 5xx?
awk '$9 ~ /^5/' access.log | wc -l

# which client generates them (top offenders among 5xx)
awk '$9 ~ /^5/ {print $1}' access.log | sort | uniq -c | sort -rn | head

# on which endpoint
awk '$9 ~ /^5/ {print $7}' access.log | sort | uniq -c | sort -rn | head

# the time window — bucket 5xx by minute (HH:MM out of [time])
awk '$9 ~ /^5/ {print substr($4,14,5)}' access.log | sort | uniq -c | sort -rn | head
Interview line: "I reduce to a distribution and confirm the top-1 actually separates from the baseline — normalizing against total traffic avoids fingering the busiest client instead of the faulty one." (For big files, LC_ALL=C speeds up sort/grep a lot.)

2 · Disk full, but no files (deleted-but-open)

Symptom: the partition is ~100% full, but du shows a few MB and ls is empty. Reboot?

Diagnosis: a process holds an open fd to a file that was unlinked (rm / log rotation). In Unix the inode and its blocks are freed only when the link count and the open-fd count both hit zero. df counts blocks (sees it); du walks directory names (can't). That's the whole discrepancy.
df -h /mnt      # ~full
du -sh /mnt     # ~nothing
lsof +L1 /mnt   # NLINK=0, marked (deleted)  ← the culprit
PID=$(lsof -t /mnt | head -1); ls -l /proc/$PID/fd | grep deleted
Fix (no downtime): truncate through the fd — space is reclaimed instantly, the process keeps writing:
: > /proc/$PID/fd/N        # N = the fd number from ls -l /proc/$PID/fd
# or make the daemon reopen its log:  kill -HUP $PID    (why logrotate needs copytruncate/reopen)
Why reboot is the worst fix: it "works" only because it closes the fd — downtime for something a one-second truncate solves. Root cause is almost always log rotation doing rm without telling the process (needs copytruncate or a reopen signal), so the symptom returns.

3 · "No space left on device" — but there's space

Symptom: apps fail with ENOSPC, touch fails, yet df -h shows the partition a third full.

Diagnosis: you're out of inodes, not blocks. Every file (even empty) uses one inode, and the count is fixed at filesystem creation. No space left is the same ENOSPC for two different causes — df -h checks blocks, df -i checks inodes.
df -h /mnt          # Use% low
df -i /mnt          # IUse% = 100%   ← the real cause

# find the over-populated directory (by file count, not size), staying on one FS
find /mnt -xdev -type d -exec sh -c 'echo "$(find "$1" -maxdepth 1 | wc -l) $1"' _ {} \; \
  | sort -rn | head
Fix: delete via find — a bare rm * chokes on the argument-length limit:
find /mnt/cache/sessions -type f -delete
# millions of files, faster in batches:
find /mnt/cache/sessions -type f -print0 | xargs -0 rm -f
Prevent: TTL/cache cleanup, don't dump millions of files in one directory (also bad for perf). On ext4 the inode count is set at mkfs — you can't "add inodes" without recreating the FS.

4 · "Too many open files" (EMFILE)

Symptom: the service stops accepting connections, log says Too many open files; disk, memory, CPU all fine.

Diagnosis: it hit the per-process descriptor limit (RLIMIT_NOFILE) and gets EMFILE (errno 24) on the next open. Usually two things: the limit is too low and there's an fd leak. (Don't confuse with ENFILE = errno 23, the system-wide limit.)
PID=$(pgrep -f myservice)
ls /proc/$PID/fd | wc -l                 # how many it holds
cat /proc/$PID/limits | grep 'open files' # the effective soft/hard limit
ls -l /proc/$PID/fd | awk '{print $NF}' | sort | uniq -c | sort -rn | head  # where they leak
The three levels to name:

Level Where

Shell / new processes ulimit -n, /etc/security/limits.conf (pam_limits)

Under systemd LimitNOFILE= in the unit — limits.conf does NOT apply (the classic trap)

System-wide ceiling /proc/sys/fs/file-max; current usage /proc/sys/fs/file-nr

Fix: raise the limit (symptom) and fix the leak (cause) — use pools / context managers. If fd count grows linearly over time it's a leak, and a bigger limit only postpones the crash.

5 · Zombies that won't die (and D-state)

Symptom: <defunct> processes in ps; kill -9 doesn't remove them, and they're multiplying.

Diagnosis: a parent forked children but never called wait(). The child exited but its process-table entry lingers until the parent reaps the exit status — that's a zombie (STAT Z). It's already dead: no code, no memory — just a PID slot. kill -9 can't help — there's nothing to kill. The only cure is to make the parent reap it.
# find zombies AND their parent (the real culprit)
ps -eo pid,ppid,stat,comm | awk '$3 ~ /Z/'
kill -CHLD <PPID>      # nudge the parent to reap
# or, if the parent is broken:  kill <PPID>  → zombies are reparented to init (PID 1), which reaps them
Root cause is the parent's code: call wait()/waitpid(), handle SIGCHLD. A couple of zombies is normal; growth means the parent isn't reaping and you'll eventually run out of PIDs.

Don't confuse with D-state (uninterruptible sleep): that process is alive, stuck in a kernel syscall (usually I/O — hung NFS, bad disk). It also resists kill -9, but for a different reason (signals aren't delivered until the syscall returns).

Zombie (Z) Uninterruptible (D)

Alive? No — already dead Yes — waiting on I/O

Holds Only a PID slot Memory; stuck on I/O

kill -9 Useless (no one to signal) Useless (signal undeliverable)

Fix Reap/kill the parent Remove the I/O cause (/proc/<pid>/wchan, stack)

The five one-liners

Log spike: aggregate, don't read — filter → key → sort | uniq -c | sort -rn.

df full, du empty: deleted-but-open file — lsof +L1, truncate via /proc/PID/fd.

ENOSPC with free space: inodes — df -i, find -xdev, find -delete.

Too many open files: EMFILE, per-process — /proc/PID/limits; under systemd it's LimitNOFILE.

Undying <defunct>: zombie — fix the parent, not the zombie; ≠ D-state.

Related: Linux Basics & Troubleshooting · SRE Linux Q&A · Performance Triage (USE).

Level	Where
Shell / new processes	`ulimit -n`, `/etc/security/limits.conf` (pam_limits)
Under systemd	`LimitNOFILE=` in the unit — limits.conf does NOT apply (the classic trap)
System-wide ceiling	`/proc/sys/fs/file-max`; current usage `/proc/sys/fs/file-nr`

	Zombie (Z)	Uninterruptible (D)
Alive?	No — already dead	Yes — waiting on I/O
Holds	Only a PID slot	Memory; stuck on I/O
`kill -9`	Useless (no one to signal)	Useless (signal undeliverable)
Fix	Reap/kill the parent	Remove the I/O cause (`/proc/<pid>/wchan`, `stack`)