← Interview Prep

DC IPv6 Architecture

Designing a modern IPv6-first data-center fabric: Clos leaf-spine topology, BGP routing, unnumbered links, addressing plans, and L3 vs EVPN/VXLAN overlays.

Modern data-center networks abandoned the classic three-tier (access/aggregation/core) design with large spanning-tree L2 domains in favor of a routed Clos fabric: a regular, horizontally scalable mesh where every server-facing switch is one or two hops from every other. Building it IPv6-first turns the fabric's biggest operational pain — per-link addressing — into a non-problem via unnumbered links.

The whole design reduces to four decisions: topology (folded Clos / leaf-spine, scaled with pods and a super-spine), routing (eBGP everywhere, ECMP for width — RFC 7938), addressing (unnumbered/RFC 5549 so links carry no IP to manage), and overlay-or-not (pure L3 to the host vs EVPN/VXLAN for tenant isolation and mobility).

Clos / leaf-spine topology

A Clos network (Charles Clos, 1953) is a multistage non-blocking switching fabric. In the DC it appears as a folded Clos — the ingress and egress stages are collapsed into the same leaf switches, with spines as the middle stage. Every leaf connects to every spine; leaves never connect to leaves, spines never connect to spines.

                3-stage folded Clos (leaf-spine)

        ┌────────┐   ┌────────┐   ┌────────┐   ┌────────┐
        │ Spine1 │   │ Spine2 │   │ Spine3 │   │ Spine4 │      middle stage
        └───┬────┘   └───┬────┘   └───┬────┘   └───┬────┘
            │  ╲   ╲     │   ╲   ╱    │    ╱  ╱     │      full mesh:
            │   ╲   ╲    │    ╲ ╱     │   ╱  ╱      │      every leaf → every spine
            │    ╲   ╲   │    ╱ ╲     │  ╱  ╱       │      (wide ECMP)
        ┌───┴────┐   ┌───┴────┐   ┌───┴────┐   ┌───┴────┐
        │ Leaf1  │   │ Leaf2  │   │ Leaf3  │   │ Leaf4  │      ingress/egress
        │ (ToR)  │   │ (ToR)  │   │ (ToR)  │   │ (ToR)  │      (folded together)
        └──┬──┬──┘   └──┬──┬──┘   └──┬──┬──┘   └──┬──┬──┘
           │  │         │  │         │  │         │  │
          srv srv      srv srv      srv srv      srv srv       servers / hosts

Key properties an interviewer wants named:

• Non-blocking / predictable: with a 1:1 (or planned) oversubscription ratio, any server can talk to any other at line rate. Every leaf→leaf path is exactly two hops (leaf → spine → leaf), so latency is uniform and predictable.
• Wide ECMP: traffic between two leaves is spread across all spines. Bisection bandwidth scales with the number of spines — add a spine, add capacity, no forklift.
• Horizontal scale-out: capacity grows by adding cheap fixed-form-factor boxes rather than buying bigger chassis. Uniform hardware, uniform config, easy to automate.
• Small failure domains: losing one spine of N removes only 1/N of the inter-leaf bandwidth (graceful degradation), not a whole tier.

3-stage vs 5-stage (pods and super-spine)

A single leaf-spine tier (3-stage Clos) is bounded by spine radix (port count): the number of leaves ≤ ports on a spine. To scale past that, replicate the leaf-spine block into a pod and add a super-spine (a.k.a. spine-of-spines / fabric-plane) tier on top — a 5-stage Clos.

Why L3 to the leaf, not big L2

The defining choice of the modern fabric is pushing the L2/L3 boundary down to the leaf (ToR). Each leaf is a routed node; the fabric between leaves is pure L3.

If you need L2 adjacency between racks (VM mobility, legacy clustering that assumes a shared subnet), you reintroduce it as an overlay (EVPN/VXLAN) on top of the L3 underlay — never by widening the L2 domain in the physical fabric.

Routing design — BGP in the DC (RFC 7938)

RFC 7938 ("Use of BGP for Routing in Large-Scale Data Centers") is the canonical reference. It argues for eBGP as the single routing protocol for the fabric, over an IGP, because:

• BGP is prefix-oriented and policy-rich: easy to filter, tag (communities), and control exactly what each device advertises — essential for graceful maintenance and blast-radius control.
• BGP has a single-protocol, single-AF operational model — no separate IGP + iBGP overlay to run and troubleshoot. One protocol carries the underlay.
• eBGP's AS-path loop prevention gives a simple, robust way to prevent loops in a regular topology, and per-hop propagation limits how far a fault spreads.
• It scales to hundreds of thousands of routes and is battle-tested with mature multi-vendor support.

eBGP with per-device / per-rack ASN allocation

The standard scheme: each leaf gets its own ASN; spines within a pod share an ASN (or each gets its own); super-spines get another. Because a Clos is regular, this is templatable. Private ASNs (2-byte 64512–65534) run out fast in a big fabric, so use 4-byte private ASNs (4200000000–4294967294).

• Per-rack (shared leaf ASN) designs let two leaves in the same rack (MLAG/redundant ToR) share an ASN — but then you must handle path hiding: a route learned from a peer with your own ASN in the path is rejected by default.
• ECMP via multipath: eBGP installs only one best path by default. You must enable maximum-paths (BGP multipath) so all equal-cost paths across the spines are programmed into the FIB. Some stacks require bgp bestpath as-path multipath-relax so paths with different neighbor ASNs (but equal AS-path length) still count as ECMP — critical when each spine has a distinct ASN.

allow-as-in / path considerations

• allowas-in: needed when a device must accept a route whose AS-path already contains its own ASN — e.g. shared-ASN leaves, or a route that must transit back. Use sparingly; it defeats the natural loop protection, so bound it (allow the local AS at most once or twice).
• AS-path as a distance metric: in a clean Clos the AS-path length is the hop count, so shortest-AS-path selection naturally picks the closest exit. Keep the topology regular so this holds.
• Convergence tuning: lower BGP timers or use BFD for sub-second failure detection (don't wait for BGP hold timers). Enable fast-external-fallover so a link-down immediately tears the session.

IPv6-specific fabric: unnumbered links & RFC 5549

This is where IPv6 pays off operationally. In an IPv4 fabric every point-to-point fabric link needs a subnet, which you must plan, allocate, document, and configure on both ends — thousands of tiny subnets to manage. IPv6 removes that entirely.

Options for fabric links

BGP unnumbered

BGP unnumbered is the headline feature. Instead of configuring a neighbor by IP, you configure the neighbor by interface:

# Config intent (FRR / Cumulus / SONiC style)
neighbor swp1 interface remote-as external
neighbor swp2 interface remote-as external
address-family ipv6 unicast
  neighbor swp1 activate
  neighbor swp2 activate
  maximum-paths 64                # program all ECMP spines
exit-address-family

# What happens on the wire:
#  1. Each side enables ICMPv6 Router Advertisements on the link.
#  2. RA advertises the router's link-local (fe80::...) address.
#  3. Peer learns the neighbor's link-local from the RA (no static IP).
#  4. eBGP session forms over IPv6 link-local — no interface addressing.

• Identical config on every port: remote-as external means "whoever answers," so the same template drops onto every leaf and spine. Cabling can change without config changes.
• RFC 5549 (obsoleted/updated by RFC 8950): advertise IPv4 NLRI with an IPv6 next-hop. This is what lets a session running over an IPv6 link-local address carry IPv4 routes — so you can run a fully unnumbered IPv6 underlay and still deliver IPv4 (and IPv6) service prefixes to hosts. The router resolves the IPv6 link-local next-hop and rewrites the L2 header.

ND / NA and RA instead of ARP / DHCP-default-gateway

• Neighbor Discovery (ND) replaces ARP: Neighbor Solicitation / Neighbor Advertisement (NS/NA) resolve IPv6 → MAC, running over ICMPv6 multicast rather than broadcast.
• Router Advertisements (RA) announce the gateway and on-link prefixes; hosts learn their default router from RA (no need to configure/DHCP a default gateway). SLAAC lets hosts autoconfigure addresses from the advertised prefix.
• Security note: because ND/RA are trust-on-first-use, protect the fabric with RA-Guard and ND inspection / DHCPv6 snooping at the leaf to block rogue RAs.

The one-line pitch: an unnumbered IPv6 fabric has no link addresses to allocate, no per-neighbor BGP config, and no ARP — every switch runs the same template, and RFC 5549/8950 still lets it carry IPv4 to the hosts. That is the biggest single automation and operational simplification IPv6 brings to the DC.

Addressing plan

With links unnumbered, the addresses you actually plan are loopbacks and host/tenant segments.

• GUA vs ULA: GUA (Global Unicast, 2000::/3) is globally routable — use it for anything that talks to the internet or across sites. ULA (fc00::/7, practically fd00::/8) is site-local/private — good for the underlay loopbacks and infrastructure that must never be internet-reachable, giving you a stable internal numbering independent of your provider-assigned GUA. Many designs: ULA underlay + GUA for tenant/host prefixes.
• Aggregation: allocate loopbacks and host prefixes so each pod summarizes to a single prefix toward the super-spine — this keeps FIB sizes bounded and shrinks the failure/convergence domain. Hierarchical addressing is the whole point of doing L3 in the fabric.
• Prefix Delegation (DHCPv6-PD): hand a downstream router/host a whole prefix (e.g. delegate a /60 or /56 so it can carve its own /64s). Useful at the edge, for container hosts, or handing tenants a routable block instead of NAT.
• No NAT by default: with GUA everywhere you drop NAT (and its state/PMTUD/logging pain); reachability is controlled by routing policy and ACLs, not address translation.

Underlay vs overlay

The underlay is the physical L3 Clos that provides loopback-to-loopback reachability (BGP + ECMP). What rides on top depends on whether you need tenant isolation and L2 mobility.

Pure L3 (routed host / BGP-to-the-host)

• Every host is its own routed endpoint; the leaf routes directly to /128s or a host /64. In BGP-to-the-host, a routing daemon (FRR/BIRD) runs on the server and peers with the leaf — services announce their own anycast /128, enabling per-service ECMP and instant drain/migration.
• When it fits: cloud-native / homogeneous workloads that don't need L2 adjacency between racks, container fabrics (Calico, Cilium in native-routing mode), and web-scale services. Simplest possible fabric — no overlay state at all.

EVPN / VXLAN overlay

• VXLAN tunnels L2 (and L3) between leaf VTEPs over the L3 underlay; EVPN (BGP address family) is the control plane that distributes MAC/IP reachability — replacing flood-and-learn.
• Provides multi-tenancy (per-VRF/L3VNI routing isolation, per-VLAN L2VNI segments), L2 mobility (a VM keeps its IP/MAC when it moves across racks), and host mobility signaling (MAC moves, sequence numbers).
• Anycast gateway: every leaf owns the same gateway IP+MAC for a subnet (distributed anycast gateway), so a host's default gateway is always its local ToR — first-hop routing is optimal and survives VM moves. No HSRP/VRRP hair-pinning.

Load balancing across the fabric

A Clos only delivers its bisection bandwidth if traffic is spread evenly across the spines. That is the job of ECMP hashing.

• ECMP hashing: each switch hashes a 5-tuple (src/dst IP, L4 ports, protocol) to pick an egress spine. Per-flow hashing keeps a flow on one path (no reordering) but can hash-collide — two elephant flows landing on the same spine link while others sit idle.
• Entropy for IPv6: hashing should include the IPv6 Flow Label and, for tunneled traffic, entropy from the outer header (VXLAN uses a hashed UDP source port as entropy so the underlay can spread the encapsulated flows). Without good entropy, encapsulated traffic pins to one path.
• Flowlet / adaptive routing: split a flow at natural gaps (flowlets) and re-hash the next flowlet onto a less-loaded path — reacts to congestion without the reordering cost of per-packet spraying. Adaptive/dynamic load balancing (e.g. in modern silicon) picks the egress by real-time port load, not just a static hash.
• Per-packet spray: maximal spread but reorders packets — only viable where the endpoint (or protocol, e.g. RoCE with tuning) tolerates reordering.

Incast, microbursts, and buffers

• Incast: the classic DC pathology — many senders reply to one requester simultaneously (map-reduce, distributed storage, scatter-gather), synchronously overflowing the egress buffer of the requester's ToR port. Result: bursty loss, TCP timeouts, throughput collapse.
• Microbursts: sub-millisecond traffic spikes that overrun shallow shared buffers even when average utilization is low — invisible to per-second telemetry.
• Mitigations: shallow-buffer cut-through switches with good ECN marking (DCTCP / DCQCN for RoCE) so senders back off before drops; PFC (priority flow control) for lossless RoCE fabrics; careful buffer carving; and small-RTT-tuned congestion control. Watch buffer/queue depth and ECN-marked counts in telemetry, not just link utilization.

Operational: automation, telemetry, upgrades, failure domains

• Automation: the fabric's regularity is the point — identical templates per role (leaf/spine/ super-spine), rendered from an IPAM/source-of-truth (NetBox) and pushed via Ansible/NAPALM or a NOS with a declarative API. Unnumbered links + remote-as external mean the same config on every port, so zero-touch provisioning (ZTP) is straightforward.
• Telemetry: streaming gNMI / model-driven telemetry (not SNMP polling) for per-queue depth, ECN marks, drops, and BGP session state; correlate microbursts and incast that per-second counters hide. BFD gives fast failure signal.
• Upgrades — drain via BGP: to service a spine/leaf gracefully, drain it by making it unattractive in routing before touching it: advertise AS-path prepend, set a max-metric, lower LOCAL_PREF/MED, tag with a GRACEFUL_SHUTDOWN community (RFC 8326), or administratively shut the BGP sessions. Traffic re-hashes onto the remaining ECMP paths before you reboot — no drops. Undrain to restore.
• Failure domains: L3-to-the-leaf makes each rack an island — a leaf failure loses only its rack; a spine failure loses only 1/N of inter-leaf bandwidth. Prefix summarization at pod boundaries bounds how far a churn event propagates. Design so no single failure removes a whole tier.