← Interview Prep

ISP / Service-Provider Network Architecture

How a carrier network is built end to end — access, aggregation, core and edge — and the MPLS/BGP control plane that ties subscribers to the global Internet.

A service-provider network exists to do one thing at massive scale: take a packet from a subscriber and deliver it to anywhere on the Internet (and back), while keeping thousands of customers isolated, billed, and inside their SLA. Almost every SP interview question maps onto one of two planes, so keep them apart:

The forwarding plane is a hierarchy of tiers — access, aggregation, core, edge — each with a different job, speed and radius. The control plane is a small, stable stack: an IGP (IS-IS) for loopback reachability, MPLS/SR for label transport, and BGP (iBGP with route reflectors + MP-BGP) for both Internet routes and VPN services. The core stays dumb and fast; all intelligence lives at the edge.

The big picture — tiers end to end

Read left to right: a subscriber's traffic climbs the hierarchy, is aggregated with thousands of others, crosses a fast dumb core, and exits at the edge toward peers, transit and other customers.

  SUBSCRIBER /      ACCESS            AGGREGATION /        CORE /            EDGE /
     CPE                                  METRO           BACKBONE          PEERING
  ┌────────┐    ┌───────────┐        ┌───────────┐    ┌──────────┐     ┌────────────┐
  │ Home   │    │ OLT/GPON  │        │  Metro    │    │          │     │  PE router │──▶ L3VPN/L2VPN
  │ router │───▶│ DSLAM     │───────▶│  ring     │───▶│  P       │────▶│            │    customers
  │ (RG)   │    │ (L2 back- │        │ (EVPN/    │    │ routers  │     │            │
  └────────┘    │  haul)    │        │  VPLS)    │    │          │     │ ┌────────┐ │
  ┌────────┐    └───────────┘        └───────────┘    │  MPLS    │     │ │ Peering│ │──▶  IXP / private
  │ Business│   ┌───────────┐        ┌───────────┐    │  core    │     │ │ router │ │     peers (settlement-free)
  │  CPE    │──▶│   BNG /   │───────▶│  Agg      │───▶│  over    │────▶│ └────────┘ │
  └────────┘    │   BRAS    │        │  router   │    │  DWDM    │     │ ┌────────┐ │
  ┌────────┐    │(PPPoE/IPoE│        │ (L3, ECMP)│    │          │     │ │ Transit│ │──▶  Upstream / Tier-1
  │ Mobile │    │  subscr.  │        └───────────┘    │          │     │ │ router │ │     (paid) ──▶ INTERNET
  │ eNB/gNB│──▶ │  mgmt,QoS)│                         └──────────┘     │ └────────┘ │
  └────────┘    └───────────┘                                          └────────────┘
   L2/L3 CPE     Subscriber           Metro transport      P = "provider"    PE = "provider
                 termination          (label switching)    label-switch      edge" — services
                 + policy             + service edge       core only         live here

Access layer — where subscribers terminate

The access layer is the messiest and most operationally intense tier: it faces physical media (fibre, copper, coax, radio) and must turn an anonymous port into an authenticated, addressed, rate-limited subscriber session.

The BNG/BRAS — the subscriber edge

The Broadband Network Gateway (BNG, formerly BRAS) is the single most important box in consumer broadband. It is the first IP hop for the subscriber and the point of policy enforcement.

• Session establishment: PPPoE (a PPP session over Ethernet — explicit login, per-session state) or IPoE (DHCP-based, no PPP — simpler, common in modern fibre).
• AAA: authenticate against RADIUS (sometimes Diameter), which returns the subscriber's IP/framed-route, VRF, QoS profile and accounting attributes.
• Address assignment: hand out IPv4 (often behind CGNAT) and IPv6 (typically via DHCPv6-PD delegating a /56 or /64 prefix to the home).
• Per-subscriber QoS: hierarchical shaping/policing (HQoS) — a parent shaper per subscriber, child queues per class (voice, video, best-effort) — plus anti-spoofing (uRPF) and lawful intercept.

Fixed-access technologies

The access node's job below the BNG is usually L2 backhaul: carry each subscriber's frames (often VLAN-per-service or Q-in-Q, S-VLAN/C-VLAN) up to the BNG where the IP session terminates.

Aggregation / metro layer

Aggregation collects dozens of access nodes and funnels them toward a handful of core PoPs. Because a fibre cut anywhere must not black-hole a whole town, the metro is usually built as rings with sub-50 ms protection.

• L2 metro: Ethernet rings running ERPS (G.8032) or, increasingly, EVPN/VPLS to emulate a multipoint LAN across the ring while avoiding classic spanning-tree.
• L3 metro (seamless MPLS): push MPLS all the way to the aggregation router so the metro is just another set of P/PE nodes. This is the modern preference — one control plane end to end, labels from access to core, ECMP load-sharing, and fast reroute everywhere.
• Service placement: the aggregation router is often a distributed PE — some services (residential, small business) terminate here rather than dragging traffic to a central PoP.

Core / backbone

The core exists to move packets between metros as fast and as cheaply-per-bit as possible. Its defining property is that it holds no per-customer state — it only label-switches.

• P (provider) routers run the IGP and label transport, but carry no BGP customer routes and no VRFs. They swap MPLS labels using only the IGP-learned topology — this is what lets the core stay small in its forwarding table even as the Internet table grows past 950k+ routes.
• MPLS is the transport substrate: an outer transport label gets the packet across the core to the egress PE; an inner service/VPN label tells the egress PE which customer/VRF it belongs to.
• DWDM / optical: the physical layer multiplexes many wavelengths (100G/400G/800G coherent) onto one fibre pair. Routers plug into transponders/muxponders; long-haul spans use amplifiers and ROADMs.
• IP-over-DWDM convergence and pluggable coherent optics (400ZR) are collapsing the router and transport layers — a common "where is this going" discussion point.

Edge / PE — where services and the Internet attach

The Provider Edge (PE) is where the network earns money. Two distinct roles usually live at the edge, sometimes on separate routers:

• Service PE: terminates customer attachment circuits and holds the VRFs for L3VPN, the bridge domains for L2VPN/EVPN, and the residential subscriber routes. This is where MP-BGP service routes originate.
• Border / peering router (ASBR): speaks eBGP to other autonomous systems — IXPs, private peers, upstream transit — and injects the Internet table into the SP.

Control plane — IGP, MPLS/SR, and BGP

The SP control plane is a deliberately small, layered stack. Each layer solves exactly one problem, and the layers compose. This is the section interviewers probe hardest.

IGP — IS-IS for loopback reachability

Carriers overwhelmingly run IS-IS in the core. It runs directly on L2 (not inside IP, so it's harder to attack), scales to large single areas, is address-family agnostic (IPv4 + IPv6 in one instance), and carries the traffic-engineering / SR extensions cleanly. The IGP's only job here is to make every loopback reachable and to feed the SPF topology that MPLS/SR builds paths on — customer routes never touch the IGP. See IS-IS Theory.

MPLS transport — LDP vs Segment Routing

• LDP: the legacy label-distribution protocol. Every router advertises a label per IGP prefix; simple, but it's a separate protocol to run and keep in sync with the IGP, and TE needs RSVP-TE with per-LSP state.
• Segment Routing (SR-MPLS): the modern default. Labels (SIDs) are distributed by the IGP itself — no LDP, no RSVP state in the core. A prefix-SID identifies a loopback; an adjacency-SID a link. The ingress PE stacks SIDs to source-route the path, so the core holds no per-flow tunnel state. Enables TI-LFA for sub-50 ms protection with 100% coverage.

See Segment Routing Theory.

iBGP and Route Reflectors

iBGP requires a full mesh (iBGP learned routes aren't re-advertised to other iBGP peers to prevent loops). In a network with hundreds of PEs, a full mesh is n(n-1)/2 sessions — unmanageable. Route Reflectors (RRs) solve this: PEs peer only with a pair of RRs, which re-reflect routes. Practical patterns:

• RR redundancy: always at least two RRs per plane; PEs peer with both.
• RR clusters / hierarchy: cluster RRs by region, with a top layer of RRs peering the clusters — bounds session count and blast radius.
• Confederations: an alternative to RRs — split the AS into sub-ASes with eBGP-like rules between them. Powerful but operationally heavier; most carriers choose RRs.
• Address-family separation: often dedicated RRs per AFI (one set for VPNv4/EVPN services, another for the Internet table) so a huge Internet table doesn't bloat the service RRs.

See MP-BGP Theory.

MP-BGP services

Multiprotocol BGP carries the service routes on top of the label transport. Each VRF's routes are tagged with route targets (RTs) that control import/export between sites, and a route distinguisher (RD) makes overlapping customer prefixes unique on the wire.

Internet connectivity — peering, transit, and tiers

An SP reaches the rest of the Internet through two fundamentally different relationships:

• IXP (Internet Exchange Point): a shared L2 fabric where many networks meet and peer over one port — cheap, dense connectivity to lots of networks (often via a route server).
• PNI (Private Network Interconnect): a direct cross-connect to one big peer when volumes justify a dedicated port.
• The peering vs transit tradeoff: peering removes traffic from paid transit and shortens paths (better latency), but only reaches that peer's cone. Transit is the guaranteed catch-all. Most SPs run both and prefer peer > customer routes over transit for cost.

ISP tiers

BGP policy — the levers

Policy is how an SP steers money and traffic. The essential tools, in the order BGP evaluates them:

• Local Preference (higher wins, egress choice, AS-wide): the primary business lever — set customer > peer > transit so you always prefer the cheapest valid exit.
• AS-path prepending: make a path look longer to inbound traffic to deprefer a link.
• MED (lower wins, only between the same neighbouring AS): a hint to a peer about which of several links into you to use.
• Communities: tags on routes (e.g. 65000:100) used to trigger policy — "prepend for me", "don't export to peers", geo/relationship tagging. The glue of scalable SP policy.

Hot-potato vs cold-potato routing

• Hot-potato (default): dump traffic to the destination AS at the nearest exit (lowest IGP metric to the egress) — minimise cost on your backbone. The other network carries it far.
• Cold-potato: deliberately carry traffic across your own backbone to hand it off close to the destination (using MED or policy) — better quality, at your expense. Content networks/CDNs often do this.

Services — what the SP actually sells

Scale & resilience

Carrier networks are engineered for "five nines" and for graceful behaviour under failure. The recurring themes:

Fast convergence & load-sharing

• ECMP: hash flows across equal-cost paths for both capacity and instant failover of the remaining paths.
• FRR / TI-LFA: precompute a loop-free backup path so forwarding reroutes in <50 ms — before the IGP even reconverges. TI-LFA (with SR) gives 100% coverage of any single failure.
• BFD: sub-second liveness detection so a link/neighbour failure is noticed in milliseconds, not IGP-hello seconds.
• Redundancy: dual RRs, dual-homed subscribers/PEs, ring topologies, diverse fibre and dual PoP uplinks — no single element is a black-hole point.

Control-plane scaling

• RR hierarchy & clusters and, where used, confederations keep iBGP session counts and update propagation bounded (see the iBGP section above).
• Route/prefix filtering at the edge keeps the Internet table clean and protects RRs.

Address exhaustion & security

• CGNAT / CGN (large-scale NAT, RFC 6598 100.64.0.0/10 space): shares scarce public IPv4 across many subscribers. Adds state, logging obligations, and breaks some inbound apps — a key driver for IPv6 deployment.
• DDoS scrubbing: detect volumetric attacks (via flow telemetry), then divert the victim's traffic to scrubbing centres (often signalled with BGP FlowSpec or RTBH — remotely triggered black-holing) and return clean traffic.
• RPKI / ROV: validate that the origin AS is authorised to announce a prefix (ROAs), dropping invalid routes to fight hijacks and leaks. Increasingly mandatory at peering edges.
• uRPF, BGP max-prefix, prefix filters, GTSM/TTL security: standard edge hygiene.

Observability

• NetFlow / IPFIX / sFlow: flow records for capacity planning, peering analysis, DDoS detection and traffic engineering.
• Streaming telemetry (gNMI/gRPC) and SNMP for real-time counters; BGP Monitoring Protocol (BMP) to observe RIBs; syslog for events.