← Interview Prep

Data-Plane Hardware — ASICs, NOS & Fabrics

How packets are really forwarded: control vs data plane, merchant silicon (Tomahawk/Jericho/Silicon One/Trio/Spectrum/Tofino), pipelines & TCAM, shallow vs deep buffers/VOQ, SONiC/SAI white-box NOS, and InfiniBand vs RoCE.

Above ~10 Gbps a CPU can't touch every packet — forwarding happens in a dedicated ASIC at line rate, while the CPU runs only the control plane (routing protocols, management). Knowing the silicon explains why one switch has huge buffers and another has almost none, why some fabrics are lossless, and where the latency goes.

Control plane vs data plane

Control plane (CPU/RP): builds the RIB via BGP/OSPF/IS-IS, then programs the forwarding ASIC.
Data plane (ASIC/NPU): looks up the FIB and forwards every packet at line rate; TCAM does longest-prefix match and ACLs in one clock.
Packets punted to the CPU (BGP, ARP, TTL-expired, unknown) are protected by CoPP — the control plane is slow and must not be overwhelmed.

Merchant vs custom silicon

Most vendors now ship merchant chips (buy the ASIC, add software) rather than custom silicon. The families to name:

Vendor	Family	Aimed at
Broadcom	Tomahawk	High-radix DC leaf/spine — huge bandwidth (25.6/51.2 Tbps), shallow on-chip buffers, lowest cost/port.
Broadcom	Jericho / Qumran	Routing/edge — deep off-chip (HBM) buffers, VOQ, big tables; carrier & DCI.
Broadcom	Trident	Feature-rich enterprise/DC switching.
Cisco	Silicon One	One unified architecture spanning switching and routing (web-scale → enterprise).
Juniper	Trio	Programmable, deep-feature routing (MX series).
Nvidia/Mellanox	Spectrum / Quantum	Low-latency Ethernet / InfiniBand for AI/HPC.
Intel	Tofino	Fully P4-programmable pipeline (line now EOL, but the reference for programmable forwarding).
Marvell	Prestera / Teralynx	Enterprise & low-latency switching.

Nearly all of these are fabricated by TSMC. Vendors differentiate on tables, buffering, and how programmable the pipeline is.

Inside a switch ASIC

Pipeline: fixed-function (fast, rigid) vs programmable (P4 on Tofino, NPL on Broadcom) — programmability lets you add new headers/telemetry (e.g. INT) without new silicon.
Tables: TCAM for LPM/ACLs (fast, power-hungry, finite — "TCAM exhaustion" is a real DC failure mode); SRAM/hash for MAC/host tables.
SerDes & optics: the I/O; port speeds scale with lanes × per-lane rate (25G NRZ → 50/100G PAM4) — 400G/800G ports.
Cut-through vs store-and-forward: cut-through starts forwarding after the header for minimum latency (trading); store-and-forward buffers the whole frame (needed for speed changes / error checking).

Buffering: shallow vs deep

	Shallow on-chip (Tomahawk)	Deep off-chip HBM (Jericho)
Buffer	MBs shared on-die	GBs of external HBM
Absorbs	Small microbursts	Large bursts / speed mismatch (100G→10G)
Latency	Lowest, predictable	Higher under load
Model	Output-queued	VOQ (virtual output queues, avoids HOL blocking)
Best for	DC leaf/spine, HFT	WAN edge, DCI, buffering-heavy

Deep buffers aren't strictly better: they add latency and enable bufferbloat. HFT/AI-fabric designs deliberately choose shallow buffers + flow control. See Low-Latency & Trading.

Disaggregation & the NOS

White-box switch = merchant ASIC + your choice of network OS. The ASIC is abstracted by SAI (Switch Abstraction Interface) so one NOS runs on many chips.

NOS	Note
SONiC	Open-source (Microsoft-led), containerized, SAI-based — the hyperscale standard.
Cumulus (Nvidia)	Linux-native switching (`ifupdown`, FRR).
DENT, ArcOS, OcNOS, PicOS, switchdev	Other white-box / Linux `switchdev` stacks worth recognizing by name.

Lossless fabrics: InfiniBand vs RoCE

AI/HPC clusters need lossless, RDMA-capable fabrics so GPUs exchange data with minimal CPU and no drops:

InfiniBand: purpose-built, credit-based flow control (never drops), native RDMA, adaptive routing, very low latency — dominant in top HPC/AI clusters.
RoCE (RDMA over Converged Ethernet): RDMA on standard Ethernet, made lossless with PFC + ECN/DCQCN — reuses the Ethernet ecosystem but PFC tuning (head-of-line blocking, deadlocks) is the hard part.

Likely interview questions

Control plane vs data plane — what runs where, and what is CoPP protecting?
Why does a Tomahawk switch have tiny buffers and a Jericho box have gigabytes? When do you want each?
What is VOQ and what problem does it solve? (head-of-line blocking)
Cut-through vs store-and-forward — latency vs error-checking/speed-change tradeoff.
What is a programmable pipeline (P4/Tofino) good for?
What is TCAM used for, and what happens when it fills?
Why do AI clusters use InfiniBand or RoCE instead of plain TCP/Ethernet? (lossless, RDMA)
What does SONiC + SAI give you? (vendor-neutral NOS on merchant silicon)