← Interview Prep

System Design — Network Monitoring (SNMP + gNMI)

Design a network monitoring system: requirements, SNMP polling vs gNMI streaming telemetry (and why both), the inventory-driven pipeline (collectors → Kafka → TSDB → alerting → Grafana), the metric model & cardinality control, symptom-based alerting, and scaling/HA/failure modes.

A common senior/system-design prompt: "design a monitoring system for a large network." The strong answer walks requirements → collection → pipeline → storage → alerting → scale → failure modes, and knows when to reach for SNMP polling vs gNMI streaming telemetry. Original, interview-focused notes.

The backbone: collectors gather metrics (SNMP poll + gNMI subscribe), a message bus buffers and fans them out, a time-series DB stores them, an alerting engine fires on symptoms, and dashboards visualize — all driven by an inventory / source of truth that says what to monitor.

Requirements first (say these)

• What: interface counters & errors, CPU/memory/temp, protocol state (BGP/OSPF up/down), optical power, queue drops — for thousands of devices, multi-vendor.
• Non-functional: low device overhead, sub-minute (ideally sub-second) resolution for critical state, high availability, bounded cardinality, retention + long-term trends, and it must survive its own outages (a monitoring system that dies in an incident is useless).

Collection: SNMP vs gNMI

	SNMP	gNMI streaming telemetry
Model	Poll (collector asks every N s)	Push (device streams on Subscribe)
Transport / encoding	UDP / BER; OIDs & MIBs	gRPC+TLS / protobuf; YANG (OpenConfig) paths
Resolution	~30–60 s (polling load caps it)	Sub-second; ON_CHANGE for state
Device cost	CPU spikes on walks; watch CoPP	Lower, steady; one session
Coverage	Universal (every device)	Newer platforms only

The realistic answer is both: gNMI where supported (fast, efficient, ON_CHANGE for BGP/interface state), SNMP everywhere else and for legacy gear. Details: gNMI & Telemetry.

Architecture

  SoT / inventory (NetBox) ──▶ what to poll / subscribe, per device

  devices ──SNMP poll──▶ SNMP collectors ─┐
          ──gNMI Sub───▶ gNMI collectors  ─┤
                          (gnmic/Telegraf/  │─▶ message bus ─▶ stream proc ─▶ TSDB ─▶ Grafana
                           snmp_exporter)   │    (Kafka)        (normalize,    (Prom/     (dashboards)
          ──dial-out────▶ (device pushes)  ─┘                    label, enrich) VictoriaM)
                                                                        │
                                                                        ▼
                                                                   alerting (Alertmanager)
                                                                        │
                                                                   Slack / PagerDuty

• Inventory-driven: the SoT generates collector targets (device, credentials, which metrics) — no hand-maintained lists; new device in NetBox → auto-monitored.
• Collectors: SNMP via snmp_exporter/Telegraf; gNMI via gnmic/Telegraf. Both normalize to a common metric model (device, interface, path → value + labels).
• Message bus (Kafka): decouples collection from storage, absorbs bursts (backpressure), and lets multiple consumers (TSDB, anomaly detection, flow correlation) read the same stream.
• Storage: a TSDB (Prometheus/VictoriaMetrics/InfluxDB) with retention + downsampling for trends; remote-write/HA for durability.
• Alerting & dashboards: Alertmanager (dedup, grouping, routing, silences) + Grafana.

Data model & cardinality

• Metrics keyed by {device, interface/path} with labels (site, role, vendor). Map OpenConfig paths and SNMP OIDs onto the same logical names so dashboards are vendor-neutral.
• Cardinality is the #1 scaling trap: per-interface × per-queue × thousands of devices explodes series count. Drop unused metrics at the collector, avoid high-cardinality labels (no per-flow), and pre-aggregate.

Alerting: symptoms, not noise

• Alert on symptoms users feel (USE-style): interface error rate climbing, BGP session down (ON_CHANGE → instant), queue drops, high CPU — not on every raw counter. See Performance / USE.
• Use rates (counters are cumulative — don't alert on absolutes), thresholds with hysteresis, and dedup/grouping so one failure isn't 500 pages.

Scale & failure modes

• Shard collectors by device group/region (each owns a slice); federate/roll up centrally. Kafka + stateless collectors scale horizontally.
• HA: redundant collectors and TSDB replicas; the pipeline must have no single point — and ideally its own failure domain (don't run monitoring on the network it monitors).
• Backpressure & gaps: Kafka buffers bursts; handle collector restarts (SNMP just resumes; gNMI re-subscribes) and mark data gaps rather than interpolating.
• Beyond metrics: add sFlow/IPFIX for flow/microburst visibility and syslog for events — same bus, different pipelines.

Likely interview questions

• SNMP vs gNMI — poll vs push, resolution, device cost; why use both?
• Draw the pipeline: collectors → bus → TSDB → alerting/dashboards, and what the SoT drives.
• Why put Kafka in the middle? (decouple, buffer, fan-out, backpressure)
• What's the cardinality problem and how do you contain it?
• How do you alert on BGP going down within a second? (gNMI ON_CHANGE)
• Why alert on rates/symptoms, not raw counters?
• How do you scale to 50k devices, and keep it HA and independent of the monitored network?
• How do SNMP OIDs and OpenConfig paths map to one dashboard? (common metric model)

Related: gNMI & Telemetry · Source of Truth · Performance / USE.