How to apply LLMs in network ops: assistive use cases, RAG grounding over configs/state, agentic diagnosis and MCP (annet-oil-mcp), a guardrailed reference architecture, and the hard limits — hallucination, prompt injection, non-determinism, and where NOT to use an LLM.
"How would you apply an LLM in network operations?" is now a standard interview question. The strong answer isn't "let AI run the network" — it's knowing where LLMs help (language and reasoning over messy network data), how to ground them (RAG + tools), and where the guardrails go (nothing touches prod unverified). This page is both interview prep and the thinking behind AI-assisted automation.
The framing that wins: LLMs are excellent at understanding and generating over network data, but networks demand correctness. So use them for assistive and read-heavy diagnostic work, ground them in real state (RAG), give them tools (MCP), and keep a human + dry-run + validation in front of any change.
| Use case | Why it fits |
|---|---|
| Explain & summarize | Summarize a config, explain a diff, translate a cryptic log or a vendor error into plain language. |
| Parse unstructured output | Turn free-form show text into structured data when no model exists — a bridge until gNMI/YANG. |
| Draft config & code | First-pass templates, a MOP, automation scaffolding, a regex — reviewed, not shipped blind. |
| Q&A over docs/configs | "Which devices run this vulnerable OS?" answered from real inventory (RAG). |
| Alert triage / diagnosis | Correlate symptoms, propose likely causes, gather evidence via tools. |
A raw LLM hallucinates IPs and invents commands because it has no ground truth. Retrieval-Augmented Generation fixes that: index your configs, operational state, tickets, and vendor docs; retrieve the relevant pieces at query time; and make the model answer from that context, with citations. "Why is BGP down on lhr1-leaf01?" becomes grounded in that device's actual config and neighbor state instead of a plausible guess.
The higher-value pattern: an agent with tools that investigates iteratively — read a device's state, query NetBox, check telemetry, form a hypothesis, gather more evidence, conclude. The model decides which tool to call; the tools do the real work and return real data.
user / alert
│
┌──▼───────────────┐ grounding ┌──────────────┐
│ LLM │◀──────────────▶│ RAG (vector │ ← configs, state,
│ (reason + plan) │ │ store) │ docs, tickets
└──┬───────────────┘ └──────────────┘
│ tool calls (MCP)
┌──▼───────────────┐ read: state / NetBox / telemetry (free)
│ network tools │ write: propose DIFF only ──▶ human approval
└──┬───────────────┘ │
│ validate + dry-run
└────────────────────────────────────────────▶ deploy pipelineGuardrails are the design: read-only by default, schema-validated outputs, diff before deploy, human-in-the-loop for writes, and full audit of what the agent did.
As the source of truth; for unverified config pushed to prod; for safety-critical real-time control. The SoT and the pipeline stay authoritative — the LLM is an assistant and a diagnostician, not the control plane.