← Interview Prep

Life of a Packet — Advanced

Overlays and multicast, walked field by field: MPLS L3VPN, IPv6, EVPN/VXLAN, PIM.

A continuation of the field-by-field walkthrough. Same rules: at every hop write down what changes, what stays, and why. The principle tying all five scenarios below together:

As soon as encapsulation appears (MPLS, VXLAN), you get two independent forwarding layers. The outer layer lives PE-to-PE (or VTEP-to-VTEP) and is handled by transit nodes. The inner layer doesn't change at all and only "wakes up" at the egress. A transit node physically never looks inside.

Scenario D — Carrier L3VPN: IS-IS + MPLS + MP-BGP

CE1 ── SW1 ── PE1 ══ P1 ══ P2 ══ PE2 ── SW2 ── CE2
       access  dot1q   IS-IS / MPLS / MP-BGP    dot1q  access
       VLAN100                                          VLAN100

CE1: 192.168.10.1        network behind CE2: 192.168.20.0/24
PE1: sub-if .100 = 192.168.10.254, VRF CUST-A, loopback 10.0.0.1
PE2: sub-if .100 = 192.168.20.254, VRF CUST-A, loopback 10.0.0.2
RD 65000:100, RT 65000:100

D.1 Three protocols — three distinct roles

This is the first thing to separate out loud, or the answer turns to mush:

The key idea: P routers hold not a single customer route. They switch on the outer label. That is the whole point of the model: the core does not scale with the number of customers.

D.2 Control plane, before any data

1. PE2 learns prefix 192.168.20.0/24 from CE2 in VRF CUST-A.
2. PE2 prepends the RD → the VPNv4 route 65000:100:192.168.20.0/24, attaches RT 65000:100 and assigns a VPN label (e.g. 24002).
3. MP-BGP advertises this to PE1, next-hop = 10.0.0.2 (PE2's loopback).
4. PE1 imports the route into VRF CUST-A on the RT match.
5. PE1 knows from IS-IS how to reach 10.0.0.2 and takes the transport label for it (e.g. 16002).

The RD makes the prefix unique (different customers may use the same 192.168.10.0/24), the RT controls import/export. People confuse them constantly — separate them explicitly.

D.3 Data plane per hop

CE1 → SW1 → PE1 — ordinary L3 traffic; on the trunk a tag appears:

PE1 → P1 — the main event: a push of two labels:

The S (Bottom of Stack) bit = 1 on the last label — that's how a node knows where the stack ends. Each label is exactly 4 bytes.

P1 → P2 — swap of the outer label (e.g. 16002 → another local index); the inner label is neither touched nor examined, the IP header is not inspected at all, the label TTL is decremented.

P2 → PE2 — PHP (Penultimate Hop Popping): PE2 advertised implicit-null (label 3), so the penultimate node pops the transport label. Only the VPN-labeled packet goes onto the link:

The point of PHP: PE2 shouldn't do a double lookup (pop a label, then another) — it saves an operation on the egress node.

PE2 → SW2 → CE2 — the VPN label is popped, its value points directly at the VRF (or at a specific next-hop, depending on the allocation mode), a lookup is done in VRF CUST-A, a new L2 header is built, VLAN 100 tag, TTL −1.

D.4 Summary

Note: the inner TTL decreased only twice — at the ingress and egress PE. To the customer, the core looks like a single hop.

Scenario E — IPv6-only data center

H1 ─── Leaf1 ═══ Spine ═══ Leaf2 ─── H2
2001:db8:0:10::11/64            2001:db8:0:20::22/64

Links between Leaf and Spine are either /127 or fully unnumbered (BGP over link-local).

E.1 What is fundamentally different from IPv4

Dropping the header checksum was deliberate: L2 (FCS) and L4 already compute one, and recomputing at every hop was pure wasted work.

E.2 Neighbor resolution

Instead of broadcast ARP — a Neighbor Solicitation to the solicited-node multicast address ff02::1:ff + the low 24 bits of the target:

The gain: the frame reaches essentially only the target node, not the whole segment — everyone else's NIC filters it by MAC. It's a "smart ARP."

E.3 The key next-hop property

The IPv6 next-hop is almost always a link-local address (fe80::…), even for global prefixes. Hence:

• routing works even with no global addresses on the link;
• BGP unnumbered (RFC 5549 / extension): the eBGP session comes up over the link-local discovered via RA, and carries even IPv4 prefixes with an IPv6 next-hop. Addressing p2p links disappears as a class of problems — a huge simplification for automation, worth emphasizing.

E.4 Per hop

Same logic as IPv4: L2 is rewritten, L3 doesn't change. The differences are the resolution mechanics, the absence of a checksum recompute, and that fe80::… as next-hop never appears in the packet itself — it only decides which MAC to use.

Scenario F — Data center with EVPN/VXLAN

H1 ─── Leaf1 ═══ Spine ═══ Leaf2 ─── H2
10.0.10.11/24    (underlay: loopbacks and IP only)    10.0.20.22/24
VNI 10010        VTEP1 = 10.255.0.1   VTEP2 = 10.255.0.2   VNI 10020
                 L3VNI 50000, VRF TENANT-A

We walk inter-subnet traffic (symmetric IRB) — the most instructive case.

F.1 The sender

H1 sees that 10.0.20.22 is in a different subnet → sends to the gateway MAC. The gateway is an anycast gateway: the same IP and MAC (00:00:5e:00:01:01 or vendor-specific) on every leaf. The host never notices moves and never reconfigures.

The ARP for the gateway likely won't even leave the leaf: ARP suppression — the leaf answers itself, knowing the mapping from EVPN Type-2.

F.2 On Leaf1: route first, then encapsulate

1. Frame is addressed to my router MAC → strip L2, go to L3.
2. Lookup in VRF TENANT-A: 10.0.20.0/24 learned from EVPN Type-5 (or Type-2 with IP), next-hop = VTEP2 10.255.0.2, label = L3VNI 50000.
3. Inner TTL −1 (64 → 63).
4. Build the inner Ethernet: SRC = Leaf1 router MAC, DST = Leaf2 router MAC (delivered in the EVPN route as the Router MAC extended community).
5. Encapsulate in VXLAN and send into the underlay.

F.3 Packet structure on Leaf1 → Spine

Overhead: +50 bytes (14 + 20 + 8 + 8). Hence an underlay MTU of 9216 or at least 1550.

F.4 On the Spine

The Spine is an ordinary IP router. It:

• looks only at the outer IP (10.255.0.2), does a lookup, TTL−1 on the outer header;
• does not open VXLAN, knows not a single tenant MAC or address;
• load-balances with ECMP using the UDP source port in the hash — which is exactly why entropy there is critical: without it all flows between a VTEP pair land on one uplink.

This is the central architectural point of EVPN/VXLAN: the spine holds no tenant state. Just like a P router in the MPLS scenario.

F.5 On Leaf2 and to the receiver

1. Outer IP is addressed to me → decapsulate.
2. VNI 50000 → this is an L3VNI → so routing is needed, not bridging.
3. Inner DST MAC = my router MAC → confirms the decision to route.
4. Lookup in VRF TENANT-A: 10.0.20.22 is a local host.
5. TTL −1 (63 → 62), new L2: SRC = Leaf2 anycast/router MAC, DST = MAC H2.

Result: the customer TTL decreased twice — at the ingress and egress leaf. To the host the fabric looks like a single router, no matter how many spines. A full analogy with L3VPN.

F.7 Follow-up questions

• Why UDP, if the encapsulation needs no transport? (for 5-tuple hashing on existing hardware)
• What happens on an underlay MTU mismatch? (black holes for full-size frames; outer DF and PMTUD don't reach inside)
• Symmetric vs. asymmetric IRB — which VNI is needed where.
• Why the anycast gateway removes the need for VRRP.
• What the path looks like under multihoming and what aliasing does.

Scenario G — Multicast, PIM-SM

S 10.1.1.10  ── FHR ══ R2(RP 10.255.0.9) ══ LHR ── Receiver 10.2.2.20
group G = 239.1.1.1

G.1 L2 addressing — no ARP here at all

The destination MAC is computed deterministically from the group address: 01:00:5e + the low 23 bits of the group IP.

239.1.1.1 → 01:00:5e:01:01:01

An important consequence people like to ask about: the high bit of the fourth octet is lost, so 32 different groups map to one MAC (e.g. 239.1.1.1 and 239.129.1.1). The host receives the extra traffic and drops it at the IP layer — i.e. it burns CPU.

G.2 Control plane: three phases

Phase 1 — source registration. The FHR receives the first packet from S, encapsulates it in a PIM Register (unicast, IP protocol 103) and sends it to the RP. If there are no receivers, the RP immediately replies with Register-Stop.

Phase 2 — shared tree (RPT). The receiver sends an IGMP Report to the LHR. The LHR sends a PIM (*,G) Join toward the RP (along the unicast route to the RP), and each node on the path creates a (*, G) entry with an outgoing interface list (OIL). Traffic flows S → FHR → RP → LHR → receiver.

Phase 3 — switch to the shortest path (SPT switchover). From the arriving packets the LHR learns S's real address and sends an (S,G) Join toward the source. Once the tree is built, the LHR sends an (S,G,rpt) Prune toward the RP, and the RP sends Register-Stop to the FHR. Traffic flows directly S → LHR.

G.3 Data plane per hop

Two differences from unicast to call out:

1. Replication. If the OIL has three interfaces, the router makes three copies, each with its own L2 header. Forwarding goes "from the source down the tree," not "toward the destination address."
2. RPF check. A packet is accepted only if it arrived on the interface through which the reverse path to the source lies (or to the RP — for shared-tree traffic). No match → silent drop. Asymmetric routing breaks multicast right here, and it's incident cause #1 of "the route exists but there's no traffic."

Scenario H — Multicast, PIM-SSM

S 10.1.1.10 ── R1 ══ R2 ══ LHR ── Receiver
group G = 232.1.1.1   (SSM range: 232.0.0.0/8)

H.1 The fundamental difference

The receiver knows the source address in advance (from app config, the feed spec, DNS). Therefore:

• no RP, no Register, no shared tree, no SPT switchover;
• the receiver sends an IGMPv3 Report with INCLUDE (S,G) — IGMPv2 is fundamentally insufficient here;
• the LHR immediately sends an (S,G) Join along the RPF path to the source;
• the tree is built once and is shortest from the start.

H.2 Data plane

After the tree is built, forwarding is identical to Scenario G: the same computed multicast MAC, the same unchanged source SRC IP, the same OIL replication, the same RPF check, TTL−1 per hop.

The whole difference is in the control plane, and that's exactly how to phrase it.

232.1.1.1 → the same 01:00:5e:01:01:01: the MAC overlap hasn't gone anywhere, you still have to plan group addresses carefully.

H.3 Why SSM is the standard in trading

The list to be able to rattle off:

• No RP → no single point of failure, no dependence on MSDP/anycast, no extra RP detour at start.
• Determinism: the same path from the first packet, no SPT-switchover transient and its duplicates/losses.
• Fast join: the join goes straight to the source.
• Security: a foreign source into the same group physically cannot deliver traffic — the subscription is bound to (S,G). For market data this property is worth more than it seems.
• A/B feeds map perfectly: two different (S,G) over two physically separate paths.

H.4 Side-by-side comparison

Wrap-up: two forwarding layers

Notice the recurring pattern — the best note to end on:

In both MPLS and VXLAN the same architectural principle holds: tenant state lives only at the edge, the core stays dumb and scalable. Say this as your conclusion and the walkthrough turns from a list of fields into a demonstration of architectural understanding.