Linux Basics & Troubleshooting

The Linux a network engineer actually needs: the modern ip/ss/ethtool toolset, virtual networking, tuning for latency, and a layer-by-layer troubleshooting drill.

As a network engineer you rarely write C on Linux, but you live on it: your traffic generators, probes, route reflectors, load balancers, containers and NFV all run there. The interview goal is to be fluent at the shell — read interface counters, follow a route, inspect sockets, capture packets, and reason about where a problem sits. This page front-loads a basics primer, then goes deep on networking.

The single most valuable habit: know the iproute2 tools (ip, ss, bridge) that replaced ifconfig/netstat/route, and be able to walk a fault L1 → L7 naming the exact command at each layer. Everything below builds to that.

Basics primer

Filesystem hierarchy

Everything hangs off a single root / (no drive letters). The directories worth knowing:

Processes & signals

A process has a PID; you inspect and control it by PID. ps is a snapshot, top/htop are live.

ps aux                      # every process, BSD-style columns
ps -ef | grep bird          # find a routing daemon
top                         # live CPU/mem; press 1 to expand per-CPU, M sort by mem
htop                        # nicer top: tree view, per-core meters, F9 to kill
pidof frr ; pgrep -a snmpd  # PID by name

kill 4021                   # send SIGTERM (15) — polite "please exit"
kill -9 4021                # SIGKILL — cannot be caught, last resort
kill -HUP 4021              # SIGHUP (1) — many daemons reload config on this
pkill -f "tcpdump.*eth0"    # match on full command line

File permissions

ls -l shows rwx triads for user / group / other. Octal: r=4, w=2, x=1.

-rw-r--r--  1 root root  648  config.yaml      # 644: owner rw, others r
-rwxr-xr-x  1 root root 1.2M  /usr/sbin/ip     # 755: owner rwx, others rx

chmod 600 ~/.ssh/id_rsa      # lock a private key to owner-only
chmod +x deploy.sh           # make executable
chown netops:netops app.cfg  # change owner:group
umask 022                    # default new-file mask

Inodes & file descriptors

An inode is the on-disk record of a file: its type, permissions, owner, size, timestamps, link count, and pointers to the data blocks — everything except the name. The name lives in a directory entry that maps a filename → an inode number. That indirection explains hard links, and why mv within a filesystem is instant (it only rewrites a directory entry).

ls -i file            # show the inode number
stat file             # inode + all metadata (perms, size, atime/mtime/ctime, links)
df -i                 # inode usage per filesystem

• Hard link = a second name pointing at the same inode (link count > 1). The data is freed only when the link count hits 0 and no process holds it open. A symlink is a separate little file that stores a path.
• "Disk full" but df shows free space? You're out of inodes — check df -i (millions of tiny files, e.g. a mail spool or cache).
• "df full but du doesn't add up? A deleted file still held open by a process — space isn't reclaimed until the fd closes. Find it with lsof | grep deleted.

A file descriptor (fd) is a per-process integer handle to an open file — and in Linux "everything is a file," so sockets, pipes, and devices are fds too. 0/1/2 are stdin/stdout/stderr.

ls -l /proc/<pid>/fd   # every fd a process holds (files, sockets, pipes)
lsof -p <pid>          # same, human-readable
lsof -i :179          # who has the BGP port open
ulimit -n             # this shell's open-fd limit (soft)

A busy network daemon (many connections = many socket fds) can hit EMFILE "too many open files" — raise the per-process limit (ulimit -n / a systemd unit's LimitNOFILE=) and the system ceiling (fs.file-max). A classic production outage.

Package managers

which tcpdump / dpkg -S $(which ip) / rpm -qf $(which ip) tell you which package owns a binary — handy when a tool is missing.

systemd & the journal

systemd is the init system and service manager on nearly all modern distros. Services are "units."

systemctl status frr             # is it running? recent log lines, PID, memory
systemctl start  frr             # start now
systemctl stop   frr             # stop now
systemctl restart frr            # bounce it
systemctl reload frr             # re-read config without a full restart (if supported)
systemctl enable  frr            # start automatically at boot
systemctl disable frr            # don't start at boot
systemctl enable --now frr       # enable + start in one shot
systemctl list-units --type=service --state=running

journalctl -u frr                # all logs for one unit
journalctl -u frr -f             # follow (tail -f) live
journalctl -u frr --since "10 min ago"
journalctl -k -b                 # kernel messages, this boot (like dmesg)
journalctl -p err -b             # only errors this boot

The boot process

The chain from power-on to a login prompt — know it so you can say where a boot hangs:

systemctl get-default            # multi-user.target (≈ old runlevel 3)
systemd-analyze                  # total boot time (firmware/loader/kernel/userspace)
systemd-analyze blame            # slowest units — find what delays boot
systemd-analyze critical-chain   # the dependency path that gated boot
journalctl --list-boots          # every recorded boot
journalctl -k -b -1              # kernel log of the *previous* boot (diagnose a crash)

systemd targets replaced SysV runlevels; a unit starts at boot when it's enabled (a symlink into the target's .wants). Networking is brought up here by whatever's in use — systemd-networkd, NetworkManager, netplan, or ifupdown.

Users & sudo

whoami ; id                      # who am I; my uid/gid/groups
sudo ip link set eth0 down       # run one command as root
sudo -i                          # interactive root shell
usermod -aG sudo alice           # add user to a group (needs re-login)
visudo                           # edit /etc/sudoers safely (syntax-checked)

Most networking commands that change state (bring a link up, add a route, capture packets) need root or sudo; read-only inspection (ip addr, ss) usually does not.

Modern networking commands (old → new)

The classic net-tools suite (ifconfig, route, arp, netstat) is deprecated and often not even installed on new systems. Learn the iproute2 replacements — interviewers notice when you reach for ifconfig.

Beyond iproute2, the tools you carry everywhere:

# Addresses & links
ip addr show                       # all interfaces + IPs (short: ip a)
ip -br addr                        # one tidy line per interface (-brief)
ip link show eth0                  # L2 state: UP/DOWN, MTU, MAC
ip link set eth0 up                # admin-up the interface
ip addr add 10.0.0.5/24 dev eth0   # add an IP
ip link set eth0 mtu 9000          # jumbo frames

# Neighbors (ARP/NDP)
ip neigh show                      # REACHABLE / STALE / FAILED entries
ip neigh flush dev eth0            # clear the cache (force re-ARP)

# Sockets
ss -tulpn                          # TCP+UDP listeners, numeric, with PID
ss -tn state established           # established TCP, no name resolution
ss -s                              # summary totals by state

Interface state & counters

When someone says "the link looks flaky," this is where you go. Two sources: the kernel's per-interface counters (ip -s link) and the driver's detailed counters (ethtool -S).

ip -s link show eth0
# 2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 ...
#     RX: bytes  packets  errors  dropped  missed  mcast
#     ...        ...      0       0        0       ...
#     TX: bytes  packets  errors  dropped  carrier collsns
#     ...        ...      0       0        0       0

UP = admin-up; LOWER_UP = physical carrier present. Admin-up but no LOWER_UP means no light / no link partner — a Layer-1 problem.

Driver counters with ethtool -S

ethtool eth0                 # Speed, Duplex, Link detected: yes/no, autoneg
ethtool -S eth0 | grep -E "err|drop|crc|miss|no_buffer|fifo"

Ring buffers, MTU, optics, flaps

ethtool -g eth0              # ring buffer: current vs max (RX/TX descriptors)
ethtool -G eth0 rx 4096      # enlarge RX ring to reduce rx_missed under bursts

ethtool -m eth0             # SFP/QSFP DOM: Rx/Tx optical power, temp, vendor
                            # low/absent Rx power => dirty/bent fiber or dead far end

ip link show eth0 | grep mtu # confirm MTU matches both ends (mismatch => black-holed
                             # large frames: small pings work, big transfers hang)

dmesg -T | grep -i eth0     # link up/down events with timestamps == flaps
journalctl -k | grep -i "link is"   # same from the journal

Routing on Linux

Linux is a real router. The kernel keeps a FIB (forwarding table) you read with ip route, and it can consult multiple tables selected by policy rules.

ip route                        # main table: default via ..., connected + static
ip route get 8.8.8.8            # which route/source/dev would be used for this dst
ip route add 10.20.0.0/16 via 10.0.0.1 dev eth0 metric 100
ip route add default via 10.0.0.1
ip route replace default via 10.0.0.1 metric 100  # idempotent change

# Lower metric wins between two candidate routes of equal prefix length;
# longest-prefix match wins across different prefix lengths.

Policy routing (multiple tables)

Beyond the destination, Linux can route on source address, incoming interface, fwmark, etc. Rules (ip rule) map traffic to routing tables (in /etc/iproute2/rt_tables).

ip rule show
#   0:      from all lookup local
#   32766:  from all lookup main
#   32767:  from all lookup default

# Source-based routing: traffic sourced from 10.0.0.5 uses table 100
echo "100 uplink2" >> /etc/iproute2/rt_tables
ip route add default via 10.0.1.1 dev eth1 table 100
ip rule add from 10.0.0.5 table 100 priority 1000

ip route show table 100         # inspect a specific table
ip rule show                    # confirm rule priority ordering (lower = first)

Multiple gateways / ECMP: give default several nexthops for load-sharing, or use ip rule for deterministic policy. Always verify with ip route get <dst> — it resolves rules + tables + metrics and tells you the actual forwarding decision, source IP included. That one command settles most "why did it go out the wrong interface" questions.

Virtual networking — the container/VM building blocks

Containers and VMs are built from a handful of kernel primitives. Knowing them explains how Docker/Kubernetes/OpenStack networking works under the hood.

# Network namespaces
ip netns add red
ip netns exec red ip addr        # run a command inside the namespace
ip netns list

# veth pair connecting host to namespace
ip link add veth0 type veth peer name veth1
ip link set veth1 netns red
ip addr add 10.10.0.1/24 dev veth0 ; ip link set veth0 up
ip netns exec red ip addr add 10.10.0.2/24 dev veth1
ip netns exec red ip link set veth1 up

# Linux bridge (software switch)
ip link add br0 type bridge
ip link set eth0 master br0      # enslave a port
ip link set br0 up
bridge link                      # bridge ports + state
bridge fdb show                  # learned MAC table (forwarding database)

# 802.1Q VLAN subinterface
ip link add link eth0 name eth0.100 type vlan id 100
ip addr add 192.168.100.2/24 dev eth0.100

# Bonding (active-backup)
ip link add bond0 type bond mode active-backup
ip link set eth1 master bond0 ; ip link set eth2 master bond0
cat /proc/net/bonding/bond0      # active slave, link status, failover count

Performance & latency tuning

On a trading or HFT box, the NIC and kernel path are tuned for predictable low latency, sometimes trading throughput and CPU for it. See Low-Latency & Trading Networks for the market-data context; here are the Linux knobs.

IRQ affinity & RSS

RSS (Receive Side Scaling) spreads incoming flows across multiple RX queues, each with its own interrupt, so multiple cores share the load. IRQ affinity pins each queue's interrupt to a specific core — you keep NIC interrupts off your latency-critical application cores (and on the same NUMA node as the NIC).

ethtool -l eth0                  # number of RX/TX queues (channels)
cat /proc/interrupts | grep eth0 # which CPU services each queue's IRQ
# pin IRQ 129 to CPU 2:
echo 4 > /proc/irq/129/smp_affinity      # bitmask: bit2 = CPU2
# (disable irqbalance so it doesn't move them back)

NIC offloads — GRO/GSO/TSO/LRO

Offloads batch packets to cut per-packet CPU. Great for throughput, but they add and hide latency and can mangle what a local capture shows.

ethtool -k eth0                  # list offload states
ethtool -K eth0 tso off gso off gro off lro off   # turn them off (note -K = set)

Interrupt coalescing & busy-poll

ethtool -c eth0                  # current coalescing settings
ethtool -C eth0 rx-usecs 0 rx-frames 1   # fire an IRQ per packet: lowest latency,
                                          # highest CPU/interrupt rate

Coalescing waits to batch interrupts (good for throughput, bad for latency). Setting rx-usecs 0 minimizes delay. Busy-polling goes further: the app spins asking the NIC for packets instead of sleeping on an interrupt — set net.core.busy_poll/busy_read, or use SO_BUSY_POLL. Zero interrupt latency at the cost of a pegged core.

Relevant sysctls

sysctl net.core.rmem_max net.core.wmem_max      # max socket buffer sizes
sysctl net.ipv4.tcp_rmem net.ipv4.tcp_wmem      # min/default/max TCP buffers
sysctl net.ipv4.tcp_congestion_control          # cubic (default) / bbr
sysctl net.core.netdev_max_backlog              # per-CPU input queue depth
sysctl -w net.ipv4.tcp_low_latency=1            # (legacy) favor latency
sysctl -w net.ipv4.tcp_timestamps=1             # RTT/PAWS; sometimes off for HFT
# persist in /etc/sysctl.conf or /etc/sysctl.d/

For a fat, high-RTT path you raise rmem/wmem so the TCP window can reach the bandwidth-delay product. For low latency you keep buffers modest to avoid bufferbloat, and pick the right congestion control (BBR on lossy/buffered paths, CUBIC otherwise).

NUMA & kernel bypass

• NUMA pinning: on multi-socket boxes, memory attached to the "wrong" socket adds latency. Pin the app and its NIC IRQs to the NUMA node the NIC is attached to (lscpu, numactl --hardware, cat /sys/class/net/eth0/device/numa_node, run with numactl --cpunodebind=0 --membind=0).
• Kernel bypass: for the lowest latency the kernel stack is skipped entirely. DPDK maps the NIC into userspace with poll-mode drivers and hugepages; Solarflare Onload transparently runs the TCP/UDP stack in userspace under an existing socket API. Both remove syscalls, interrupts and copies from the hot path — the standard HFT approach.

tcpdump -ni eth0 host 10.0.0.5 and port 443   # numeric, one iface, filtered
tcpdump -ni eth0 'tcp[tcpflags] & (tcp-syn|tcp-rst) != 0'  # SYNs and RSTs only
tcpdump -ni eth0 -w /tmp/cap.pcap host 10.0.0.5           # write for Wireshark
tshark -ni eth0 -f "port 179" -Y "bgp"                    # capture + decode BGP

Firewall basics — iptables, nftables, conntrack

Linux packet filtering runs in netfilter. The old front-end is iptables (per-protocol, ordered rules in chains); the modern replacement is nftables (nft) with one unified syntax, sets, and maps. On new distros iptables is often a compatibility shim over nftables.

# iptables — inspect & add
iptables -L -n -v --line-numbers        # list filter table with counters
iptables -t nat -L -n -v                # inspect NAT rules
iptables -A INPUT -p tcp --dport 22 -j ACCEPT
iptables -I INPUT 1 -s 10.0.0.0/8 -j DROP

# nftables — inspect & add
nft list ruleset                        # everything, current state
nft add rule inet filter input tcp dport 22 accept

# conntrack — the stateful connection table (NAT + stateful rules rely on it)
conntrack -L                            # list tracked flows
conntrack -S                            # per-CPU stats: inserts, drops, early_drop
sysctl net.netfilter.nf_conntrack_max   # table size limit
sysctl net.netfilter.nf_conntrack_count # current entries (full table => new
                                         # connections dropped — a subtle outage)

Conntrack is the state engine behind NAT and -m state ESTABLISHED,RELATED rules. A conntrack table filling up (count nearing max, rising drops) silently drops new connections on busy boxes — a favorite "the load balancer randomly refuses connections" scenario.