← Interview Prep

TCP — Theory & Mechanisms

Everything worth being able to explain about TCP at a whiteboard: connection lifecycle, reliability, flow and congestion control.

TCP turns an unreliable, unordered, best-effort IP layer into a reliable, ordered, byte-stream between two endpoints. Nearly every TCP interview question reduces to one of four jobs it performs, so keep them straight:

Reliability (sequence numbers + ACKs + retransmission), ordering (reassembly by sequence number), flow control (don't overrun the receiver — the advertised window), and congestion control (don't overrun the network — the congestion window). Flow control and congestion control are different problems solved by different windows.

Segment header

TCP is a Layer-4 protocol (IP protocol number 6). Its header is 20 bytes without options, up to 60 with them.

Connection establishment — the three-way handshake

Client                              Server
  │   SYN  seq=x                      │   (client: SYN-SENT)
  │ ────────────────────────────────▶│   (server: SYN-RECEIVED)
  │   SYN, ACK  seq=y, ack=x+1        │
  │ ◀────────────────────────────────│
  │   ACK  seq=x+1, ack=y+1           │   (both: ESTABLISHED)
  │ ────────────────────────────────▶│

• Each side picks a random Initial Sequence Number (ISN) — randomized to prevent old/forged segments from being accepted and to make blind spoofing hard.
• The SYN and FIN flags each consume one sequence number (they are "phantom" bytes), which is why the ACK is x+1.
• Options that must be negotiated up front (MSS, window scale, SACK-permitted, timestamps) are carried on the SYN and SYN-ACK — they can only be set there.

Connection termination & TIME_WAIT

  │  FIN  seq=u                 │   active closer: FIN_WAIT_1
  │ ──────────────────────────▶│   passive: CLOSE_WAIT
  │  ACK  ack=u+1              │
  │ ◀──────────────────────────│   active: FIN_WAIT_2
  │  FIN  seq=v                 │   (passive app closes) LAST_ACK
  │ ◀──────────────────────────│
  │  ACK  ack=v+1              │   active: TIME_WAIT ── 2·MSL ──▶ CLOSED
  │ ──────────────────────────▶│   passive: CLOSED

TCP is full-duplex, so each direction is closed independently — hence four segments (the middle two can coalesce, giving three). The side that sends the first FIN is the active closer.

• TIME_WAIT (on the active closer) lasts 2×MSL (Maximum Segment Lifetime). Two reasons: (1) ensure the final ACK arrived — if it was lost, the peer retransmits its FIN and we can re-ACK; (2) let stray/duplicate segments from this connection die out before the 4-tuple can be reused.
• CLOSE_WAIT piling up = a bug in your application: the peer closed, but your code never called close(). A very common interview "what does this state mean" question.
• Lots of TIME_WAIT on a busy client/proxy is normal but can exhaust ephemeral ports; mitigations include connection reuse, tcp_tw_reuse, and timestamps (PAWS).
• RST aborts immediately (no graceful close) — sent on a closed port, a bad segment, or SO_LINGER=0.

State machine (the states to know)

CLOSED → LISTEN → SYN-RECEIVED → ESTABLISHED (server) and CLOSED → SYN-SENT → ESTABLISHED (client); teardown walks FIN-WAIT-1/2 → TIME-WAIT → CLOSED (active) and CLOSE-WAIT → LAST-ACK → CLOSED (passive).

Reliability: ACKs, retransmission, RTO

The receiver sends a cumulative ACK naming the next in-order byte it expects. Data is retransmitted when the sender infers loss, by one of two mechanisms:

• RTT estimation: RTO is derived from a smoothed RTT (SRTT) and its variance (RTTVAR), per Jacobson/Karels: RTO = SRTT + 4·RTTVAR, with backoff on repeated loss.
• Karn's algorithm: don't sample RTT from a retransmitted segment (you can't tell which copy was ACKed).
• SACK (Selective ACK): the receiver reports the specific non-contiguous blocks it has, so the sender retransmits only the true gaps instead of everything after the loss. Big win on multiple losses per window.
• Delayed ACK: the receiver may wait up to ~200 ms (or until a second full segment) before ACKing, to piggyback the ACK and reduce ACK traffic.

Flow control — the receive window

Flow control protects a slow receiver from a fast sender. The receiver advertises a window (rwnd) = free space in its buffer; the sender may have at most that many unacknowledged bytes in flight. This is the sliding window.

• Window Scale option: the 16-bit window maxes at 64 KB, far too small for modern bandwidth×delay products. Window scaling shifts it left by up to 14 bits (up to ~1 GB). Negotiated on the SYN only.
• Zero window: a receiver whose buffer is full advertises window=0; the sender stops and periodically sends a zero-window probe to learn when space frees up (so it doesn't deadlock if the window-update ACK is lost).
• Silly window syndrome: avoid advertising / sending tiny windows/segments; addressed by the receiver (Clark's solution) and the sender (Nagle).

Bandwidth-delay product is the number to know: BDP = bandwidth × RTT is the amount of in-flight data needed to fill the pipe; if the window < BDP, throughput is capped regardless of link speed.

Congestion control — the congestion window

Congestion control protects the network. The sender maintains a congestion window (cwnd); the amount it may send is min(cwnd, rwnd). Classic (Reno/NewReno) has four phases:

• The overall shape is AIMD (Additive Increase, Multiplicative Decrease) — the classic TCP "sawtooth."
• A timeout (RTO) is treated as worse than dup-ACK loss: cwnd collapses to 1 MSS and slow start restarts.
• Congestion algorithms: Reno/NewReno (loss-based, textbook); CUBIC (Linux default, cubic growth — good on high-BDP links); BBR (Google; models bottleneck bandwidth and RTT instead of treating loss as the only signal — better on lossy/buffer-bloated paths).
• ECN (Explicit Congestion Notification): routers mark packets (CE) instead of dropping; the receiver echoes it (ECE) so the sender backs off without a loss.
• Bufferbloat: oversized buffers hide loss and inflate RTT, defeating loss-based control — the motivation for BBR and AQM (CoDel/FQ-CoDel).

Nagle, delayed ACK, and PSH

• Nagle's algorithm: with unacknowledged data outstanding, buffer small writes into one larger segment to avoid flooding the net with tiny packets. Great for bulk, bad for latency-sensitive small messages.
• Nagle × delayed ACK interaction: a classic ~200 ms stall — the sender waits for an ACK (Nagle) while the receiver waits to piggyback the ACK (delayed ACK). Fix: disable Nagle with TCP_NODELAY for interactive/RPC traffic (and why trading/RPC stacks set it).
• PSH asks the receiver to deliver buffered data to the application promptly rather than waiting for more.

MSS, MTU, and PMTUD

• MSS (Maximum Segment Size) = the largest payload TCP will put in one segment, advertised on the SYN. Typically MTU − 40 (1500 − 20 IP − 20 TCP = 1460).
• PMTUD: TCP sets DF; if a smaller-MTU link is hit, the router returns ICMP "fragmentation needed" (type 3 code 4) with the MTU, and the sender lowers its segment size. If that ICMP is filtered → a PMTUD black hole: the handshake works (small packets) but bulk transfer stalls.
• MSS clamping: a router rewrites the MSS option in the SYN downward — the one legitimate place the network edits an L4 header — to sidestep tunnel/PMTUD problems (common on PPPoE/VPN/overlays).

TCP vs UDP

Head-of-line blocking is the key TCP tradeoff to mention: because delivery is an ordered stream, a single lost segment blocks delivery of everything after it until it's retransmitted. This is exactly why QUIC (HTTP/3) runs over UDP with independent streams.