A repeatable method for "the box is slow": Brendan Gregg's 60-second checklist (vmstat/mpstat/pidstat/iostat/sar), the USE method per resource, and the deeper tools (perf, strace, eBPF).
"A box is slow / an alert fired — walk me through it." The answer interviewers want is a method, not a random pile of commands. Two frameworks cover almost every performance question: Brendan Gregg's 60-second checklist to triage fast, and the USE method to be systematic about resources.
Ten commands, in order, to form a first hypothesis on an unfamiliar Linux host:
| # | Command | What you learn |
|---|---|---|
| 1 | uptime | Load average trend (1/5/15 min). Rising = getting worse. Compare to core count. |
| 2 | dmesg | tail | Kernel errors: OOM kills, TCP drops, disk/hardware resets. |
| 3 | vmstat 1 | r (run queue) vs cores, si/so (swapping!), us/sy/id/wa CPU split. |
| 4 | mpstat -P ALL 1 | Per-CPU balance — one core pegged at 100% (single-threaded bottleneck) vs evenly loaded. |
| 5 | pidstat 1 | Per-process CPU over time — which process, without top's refresh churn. |
| 6 | iostat -xz 1 | Per-disk: %util (saturation), await (latency), r/s w/s (IOPS). |
| 7 | free -m | Real headroom via available; is memory going to cache or swap? |
| 8 | sar -n DEV 1 | Per-NIC throughput — near line rate? packet rate for small-packet loads? |
| 9 | sar -n TCP,ETCP 1 | New connections/s, retransmits (retrans/s = loss), resets. |
| 10 | top / htop | Sanity-check the leaders against everything above. |
The tell-tales:wahigh → disk-bound;si/sonon-zero → memory pressure/swap; one hot CPU inmpstat→ single-threaded;retrans/sup → network loss. Each points you at the next, deeper tool.
For every resource, check three things — Utilization, Saturation, Errors:
| Resource | Utilization | Saturation | Errors |
|---|---|---|---|
| CPU | mpstat, top (%busy) | run queue: vmstat r, load avg | dmesg (MCE/throttling) |
| Memory | free (used vs available) | swap si/so, major faults, OOM | dmesg OOM/ECC (EDAC) |
| Disk | iostat -xz %util | await, queue depth (aqu-sz) | smartctl, dmesg I/O errors |
| Network | sar -n DEV vs link speed | drops/backlog: ss, netstat -s, qdisc | ethtool -S, ip -s link, retransmits |
Errors first, then utilization, then saturation — and remember saturation is what users feel as latency even when utilization looks "only" 80%.
| Symptom | Next tool |
|---|---|
Which syscalls / why a process is stuck in D | strace -p, /proc/<pid>/stack, /proc/<pid>/wchan |
| Where CPU time really goes | perf top, perf record → flame graph |
| Packet-level network truth | tcpdump, then Wireshark |
| Historical (what happened at 03:00?) | sar archives, atop -r (records per-process history) |
| Kernel-level tracing | bpftrace / bcc (eBPF): biolatency, execsnoop, tcpretrans |