Low-Latency & Trading Networks

How HFT shops shave nanoseconds: cut-through switching, the latency budget, kernel bypass, and how you prove it.

Trading interviews probe one core idea from many angles: in this world the metric is time, not throughput. A market-data update or an order is a tiny packet; what matters is how few nanoseconds elapse between a signal arriving on one wire and a reaction leaving on another, and how consistent that number is. Say this up front:

Bandwidth buys you capacity; latency buys you the trade. A firm will happily run a 10G link at 2% utilization if it means the switch forwards each frame 300 ns sooner and never queues. The whole design collapses to one goal: minimize per-packet latency and its jitter, end to end, and be able to measure it to the nanosecond.

Cut-through vs store-and-forward switching

Every switch has to decide when it may start transmitting a frame out the egress port relative to how much of it has arrived on ingress. That single decision is the biggest lever a switch vendor has over latency.

Store-and-forward

The classic behavior: the switch buffers the entire frame, computes the CRC and compares it against the FCS in the trailer, and only then looks up the destination and forwards. Consequences:

Latency grows with frame size — you pay the full serialization time of the frame before a single bit leaves. A 1500-byte frame on 10G takes ~1.2 µs just to clock in; the switch waits for all of it.
Corrupt and runt frames are dropped, because the FCS is verified before forwarding. Errors do not propagate.
It is mandatory when speeds differ (e.g. 1G ingress → 10G egress) or when the frame must be modified in a way that requires the whole payload.

Cut-through

The switch reads just enough of the header to make a forwarding decision — the destination MAC, the first 6 bytes after the preamble — and immediately begins streaming the frame out the egress port while the tail is still arriving. Consequences:

Near-constant, size-independent latency. Port-to-port figures are quoted in hundreds of nanoseconds (often sub-microsecond), essentially the same for a 64-byte frame and a 1500-byte frame, because you only wait for the header, not the payload.
It can forward corrupt frames. The FCS lives in the trailer, which hasn't arrived yet when forwarding starts — so a frame with a bad CRC is already partly on the wire before the switch knows. The error propagates downstream. Good switches count the CRC error (and can flag or, on some platforms, adaptively fall back to store-and-forward on a high error rate) but the bad frame is still forwarded.
Runts propagate too. A frame truncated by a collision or fault can be relayed before its (illegal) length is known.

Fragment-free — the middle ground

A compromise mode: the switch waits for the first 64 bytes before forwarding. Since Ethernet collisions only happen within the first 64 bytes (the minimum frame / slot time), any collision-induced runt is caught and dropped, while you still avoid buffering the whole frame. Largely historical now (collisions vanished with full-duplex switching) but a common interview name-drop.

Property	Store-and-forward	Fragment-free	Cut-through
Bytes read before forwarding	whole frame	first 64 B	~first 6 B (dst MAC)
Latency vs frame size	grows with size	fixed ~64 B wait	~constant, minimal
FCS checked before forward?	yes	no (only length)	no
Forwards corrupt frames?	no	collision runts caught	yes (counted, not blocked)
Speed / MTU mismatch	handles it	limited	forces S&F fallback
Typical use	enterprise, WAN edge	legacy	HFT / trading fabric

Why traders prefer cut-through

Three reasons, stated plainly:

Lowest and most predictable per-hop latency. Every hop adds only header-read time, not frame-serialization time — and it's the same on every packet, so jitter stays tiny. Predictability matters as much as the mean.
Errors are rare on a clean, short, colocated fabric. Inside a data center with good optics and few hops, CRC errors are near zero, so the downside of forwarding a bad frame almost never fires. And the endpoints (TCP, or the application on UDP multicast) will discard a corrupt payload anyway — the switch doesn't need to.
The cost of waiting is real money. Being 500 ns slower to the exchange matching engine can mean losing the queue position on a quote. The theoretical risk of relaying a corrupt frame is dwarfed by the certain cost of the extra buffering delay.

The latency budget — where the nanoseconds go

End-to-end latency is a sum of independent components. Being able to name them and say which one dominates in a given scenario is the single most-tested skill here.

Component	What it is	How to compute / rule of thumb	How to reduce
Serialization	Time to clock the frame's bits onto the wire, one at a time.	`frame_bits / link_speed`. 1500 B @ 10G ≈ 1.2 µs; 64 B @ 10G ≈ 51 ns; same 64 B @ 100G ≈ 5 ns.	Smaller frames, faster links.
Propagation	Time for the signal to physically travel the distance.	~5 µs per km in fiber (light in glass ≈ 200,000 km/s, ~2/3 of c). Distance-bound, unavoidable.	Shorter path → colocation; straighter path; air beats glass (microwave).
Switching / forwarding	Time inside the device to read the header and make the decision.	Cut-through ASIC: ~300–500 ns/hop; L1 switch: ~5 ns; store-and-forward: +full serialization.	Cut-through / L1 switches, fewer hops.
Queuing / buffering	Time a frame waits behind others for a busy egress port.	The big variable — zero when idle, unbounded under congestion/microbursts. Source of jitter.	Headroom, shallow buffers, priority for market data, avoid oversubscription.
Host / NIC / stack	Time from wire → application and back, inside the server.	Kernel stack: single-digit µs; kernel bypass: hundreds of ns; FPGA NIC: tens of ns.	Kernel bypass, busy-poll, CPU pinning, FPGA offload.

Speed-of-light-in-fiber rule of thumb: ~5 µs per kilometer, each way (~1 µs per 200 m). A 100 km round trip is ~1 ms you can never get back with any switch — which is exactly why proximity/colocation, not gear, wins the long-distance game.

The interview move: given a topology, decompose the latency and identify the dominant term. Across a data center hall, switching + host dominate and gear choice matters. Between two cities, propagation dominates and only the path (fiber vs microwave, km of route) matters — a faster switch is noise.

Why it's about small packets, not bandwidth

Market-data ticks and order messages are tiny — often 40–100 bytes of payload. A book update, a new order, a cancel: all small. This reshapes what you optimize for.

PPS > Gbps. The workload is millions of small packets per second, not a few large flows. A device's packets-per-second rate and its per-packet latency matter far more than its headline bandwidth. A 10G link can be packet-rate-bound long before it is bit-rate-bound.
Microbursts fill buffers and inject jitter. Market data is intensely bursty: an instant of quiet, then thousands of updates in a few microseconds after a print. Those bursts momentarily overrun egress and queue — adding delay and variance exactly when the market is moving and latency matters most. Deep buffers don't help; they just convert loss into latency (bufferbloat). Shallow buffers plus headroom keep jitter low.
Every per-packet feature is a tax. Anything that touches each frame — deep ACLs, NAT, heavy QoS classification, encryption, tunnel encap, store-and-forward, LAG rebalancing — adds latency per packet. On the hot path you strip the fabric down to the minimum: cut-through, no unnecessary lookups, no features that add nanoseconds.

Host, NIC, and stack optimization

Half the latency budget is often inside the server, before a byte reaches the network. The standard Linux path — interrupt → softirq → kernel stack → socket → copy to user — costs single-digit microseconds and is jittery. HFT hosts bypass most of it.

Technique	What it does	Why it helps
Kernel bypass	App talks to the NIC directly in user space: `DPDK`, Solarflare/Xilinx Onload (transparent sockets), `RDMA`/RoCE, `ef_vi`, `io_uring`/AF_XDP.	Removes syscalls, context switches, and kernel-stack traversal from the hot path — µs → hundreds of ns.
Busy-polling	A pinned core spins reading the NIC descriptor ring instead of waiting for an interrupt; disable interrupt coalescing (`ethtool -C … rx-usecs 0`).	Interrupts and coalescing trade latency for CPU efficiency — the wrong trade here. Polling eliminates interrupt-delivery latency and jitter.
Hardware timestamping	NIC stamps each packet in silicon (PHY/MAC) on RX and TX.	Nanosecond-accurate, jitter-free timing for measurement and for PTP — software clocks are too coarse and noisy.
CPU pinning / NUMA	Pin the trading thread and NIC IRQs to cores on the same NUMA node as the NIC's PCIe root; isolate cores (`isolcpus`, `nohz_full`).	Avoids cross-socket QPI/UPI hops and cache misses; stops the scheduler from migrating the hot thread.
Disable power saving	Turn off C-states/P-states, EEE (Energy-Efficient Ethernet, 802.3az), and frequency scaling; lock to max clock.	Wake-from-idle transitions add microseconds of unpredictable latency to the first packet after quiet — deadly for bursty market data.
TCP / socket tuning	`TCP_NODELAY` (kill Nagle), tuned buffers, `SO_BUSY_POLL`, huge pages, pre-faulted memory.	Nagle batches small writes — exactly the wrong behavior for tiny order messages. Remove anything that delays a small send.
Avoid the kernel on the hot path	Keep the market-data / order path entirely in user space or on the NIC; leave the kernel stack for control/admin traffic only.	The kernel is a shared, preemptible, jittery resource — you want the hot path deterministic.

# Kill interrupt coalescing and offloads that add/hide latency on the hot NIC
ethtool -C eth0 rx-usecs 0 tx-usecs 0 adaptive-rx off adaptive-tx off
ethtool --set-eee eth0 eee off          # disable Energy-Efficient Ethernet
ethtool -K eth0 gro off lro off          # no aggregation on the RX path

# Pin the NIC's IRQ and the trading thread to a core on the NIC's NUMA node
cat /sys/class/net/eth0/device/numa_node
taskset -c 4 ./trader                     # isolcpus=4 nohz_full=4 in cmdline

# Kernel: latency-oriented tuning
cpupower frequency-set -g performance     # lock max freq, no P-state ramp
# BIOS: C-states disabled, Turbo/uncore locked, hyperthreading off on hot cores

Network optimization for HFT

Ultra-low-latency switches

Purpose-built cut-through switches quote deterministic sub-microsecond port-to-port latency: Arista's 7130 / Metamako line, the low-latency Cisco Nexus models, and similar. They strip features to the minimum and forward on the destination MAC as it arrives.

Layer-1 / physical-layer switches

The extreme end: a Layer-1 switch is effectively a programmable, electronic patch panel — it cross-connects physical ports at the PHY with no packet parsing at all, giving ~5 nanosecond latency. Used for fan-out/replication of a market-data feed to many consumers, for taps, and for A/B failover — because it adds essentially nothing to the budget. The trade-off: no L2/L3 intelligence, it just moves bits.

FPGA-based NICs and switches

For the truly latency-critical path, logic runs in an FPGA on a SmartNIC (or an FPGA switch). Feed parsing, order-book building, risk checks, and even the trigger-to-order reaction happen in gates in tens of nanoseconds, never touching a CPU or the kernel. This is "tick-to-trade" in hardware.

Topology and cabling

Minimize hop count. Every switch is latency; a flat, one- or two-hop fabric beats a deep tree. Sometimes a single big L1/cut-through switch replaces a spine.
Equal-length cabling. Exchanges (and internal fabrics) deliberately cut every cross-connect to the same fiber length so that no colocated participant gets a propagation edge from a shorter run — fairness measured in nanoseconds. Internally you match lengths to control skew.
Colocation. Put the server in the same building as the matching engine. This collapses propagation delay to near zero and is the highest-leverage optimization that exists — you cannot out-engineer distance.

Timing, QoS, and feed delivery

PTP (IEEE 1588) with hardware timestamping distributes a common clock across the fabric to sub-microsecond (often nanosecond) accuracy, so that latency measurements taken at different points are comparable. It's the measurement backbone — see the next section.
QoS / strict priority for market data so that, in the rare moment of contention, the time-critical feed jumps the queue ahead of bulk/admin traffic. On a hot fabric you keep classification cheap.
Multicast SSM A/B feeds. Exchanges publish market data over UDP multicast using Source-Specific Multicast (SSM) — receivers join (S,G) and the tree is built with PIM-SSM, no shared tree or RP needed. Feeds are sent as two independent A/B streams over disjoint paths; the consumer arbitrates by sequence number, filling a gap on A from B without waiting for a retransmit. This is how you get lossless, low-latency fan-out to thousands of subscribers. See PIM-SM / SSM session for the multicast mechanics and Life of a Packet — Advanced for the multicast forwarding walkthrough.

Long-haul: why microwave beats fiber

Between distant venues (e.g. Chicago↔New York, or across Europe), the winner is microwave / millimeter-wave radio, not fiber. Two reasons, both physics:

Air is faster than glass. Radio through the atmosphere travels at ~c (300,000 km/s); light in fiber only ~200,000 km/s. That's a ~50% speed advantage per unit distance.
Straighter path. A microwave chain hops tower-to-tower in near-straight lines; fiber follows railways, roads, and rights-of-way, adding physical distance. Less distance × faster medium compounds the win.

The catch, and the reason fiber still exists on these routes: microwave is low-bandwidth (you send only the most valuable signals — top-of-book, a few symbols) and weather-sensitive (rain fade), so firms run microwave for the latency-critical trigger and fiber as the high-capacity, all-weather backup.

Measuring and proving latency

You can't optimize what you can't measure, and in this world "measure" means nanoseconds with proof. The techniques form a hierarchy of trustworthiness.

Method	Resolution	Trust	Notes
Software timestamps (app / kernel `clock_gettime`)	µs, noisy	low	Includes scheduler jitter, syscall cost, and clock error — measures your host as much as the network. Insufficient for HFT.
NIC hardware timestamps	ns	high	Stamped in the PHY/MAC on the wire, before any software touches the packet. The baseline for host-side measurement.
PTP (IEEE 1588)	sub-µs to ns	high	Synchronizes clocks across devices so timestamps taken at different points are comparable end to end. Use HW-timestamped boundary/transparent clocks.
Passive tap + capture	ns	highest	An optical/L1 tap copies the wire to a capture appliance (e.g. a timestamping FPGA) that stamps independently — ground truth, not the device under test's own claim.

Key points to make:

Software timestamps are insufficient because they conflate the thing you're measuring (network/NIC latency) with things you're not (OS scheduling, syscall latency, an unsynced or drifting system clock). The number moves for reasons that have nothing to do with the wire.
Hardware timestamps + a synchronized clock are the minimum credible setup: stamp on the wire, and put every stamp on the same PTP-disciplined timescale so a one-way delay between two boxes is meaningful.
A passive tap is the arbiter. When you need to prove a switch's port-to-port latency (or settle a dispute with a vendor/exchange), you tap ingress and egress and let an independent timestamping capture measure it, so the device isn't grading its own homework.
Report distributions, not averages: the mean hides the tail. HFT cares about the 99.9th percentile and max — the jitter — because the worst case is when the market is moving and the trade matters.

Likely follow-up questions

What's the downside of cut-through, and when would you not use it? It forwards corrupt frames and runts (FCS isn't checked before forwarding), so errors propagate; you avoid it across a lossy/long link or on any speed/MTU mismatch, which forces store-and-forward anyway.
Why isn't bandwidth the right metric here? The packets are tiny and bursty; PPS and per-packet latency bind long before bit-rate does, and a 10G link at 2% utilization can still be the fastest thing on the floor.
Latency vs jitter — which do you optimize for? Both, but jitter (variance) often matters more: a predictable 800 ns beats a "usually 400 ns but sometimes 5 µs" path, because strategies are built around the tail, not the mean.
What is bufferbloat and why is it a trap? Deep buffers absorb microbursts by queuing instead of dropping, converting loss into large, variable latency. On a trading fabric you'd rather drop (and let the app/A-B feed recover) than sit in a fat queue.
How do you cut propagation delay? You don't — physics is fixed at ~5 µs/km in fiber. You cut distance (colocation) or change medium/path (microwave through air, in a straighter line).
Where does the latency actually go on a colo hop vs a cross-city hop? Colo: switching + host dominate, so gear and kernel bypass matter. Cross-city: propagation dominates, so only the path matters and switch choice is noise.
Why hardware timestamps over software? Software stamps include scheduler and clock jitter and measure the host, not the wire; HW stamps on a PTP-synced clock, verified by a passive tap, are the only credible proof.