Treat network config as code: the test pyramid (unit/pytest, SoT validation, Batfish static analysis, containerlab virtual integration, pre/post-checks), golden config & drift, and a pipeline with canary deploy + automatic rollback.
Network changes are high-blast-radius: one bad line can black-hole a data center. The mature answer is to treat config as code — Git review + automated tests + a safe, staged deploy with rollback. This is where network automation earns its keep, and a favorite senior-interview topic: "how do you make a change to 500 devices without an outage?"
The spine of it: lint → unit → render → static analysis → virtual integration → canary → post-checks → full deploy, with an automatic rollback on any failed gate. Nothing reaches prod that a pipeline hasn't tried first.
| Layer | What you test | Tools |
|---|---|---|
| Unit | Your code — parsers, filters, templates (pure functions, no devices) | pytest |
| Data / SoT validation | Intent is well-formed: schema, required fields, referential integrity, naming | pydantic / JSON Schema, yamllint, ansible-lint |
| Static config analysis | The rendered config is correct without a device — reachability, ACLs, what-if | Batfish, pyATS/Genie parse |
| Virtual integration | Apply candidate config to a real NOS image in a lab topology | containerlab, vrnetlab, GNS3/EVE-NG |
| Pre/post-checks | Operational state didn't regress (BGP up, routes present, reachability) | pyATS, Nornir, SuzieQ, Robot |
Same shape as software: many cheap unit tests at the bottom, few expensive integration tests at the top. Push validation as far left (pre-deploy) as you can.
Batfish builds a vendor-neutral model from your config text — no device needed — and answers questions about it: reachability, ACL/firewall analysis, "will these two configs route the same?" (differential), undefined references, and what-if. It's the strongest pre-merge gate: catch a bad ACL or a routing black hole before anything is deployed.
containerlab spins up a topology of real NOS containers (SR Linux, cEOS, Junos cRPD, FRR, SONiC…) in seconds. In CI you: define the topo YAML, deploy it in the runner, push the candidate config, run your assertions (pyATS/Nornir/pytest), and tear it down — a real test of the actual change on the actual software.
# pipeline sketch (GitLab CI / GitHub Actions)
stages: [lint, unit, render, analyze, integration, deploy]
lint: { script: [ "yamllint .", "ansible-lint" ] }
unit: { script: [ "pytest tests/unit" ] }
render: { script: [ "make render" ] } # SoT -> intended config
analyze: { script: [ "pytest tests/batfish" ] } # reachability/ACL, no device
integration: { script: [ "clab deploy -t lab.clab.yml", # real NOS images
"pytest tests/integration",
"clab destroy -t lab.clab.yml" ] }
deploy: # only on main, after merge
script: [ "python deploy.py --canary --post-checks --rollback-on-fail" ]
rules: [ { if: '$CI_COMMIT_BRANCH == "main"' } ]diff, NAPALM compare_config, Batfish). Remediate by re-deploying intended — or, if
the change was legitimate, reconcile it back into the SoT.The single most valuable operational pattern — and easy to describe in an interview:
serial / one PoP).commit confirmed timeout, or restore the backup) — no human in the loop for the revert.A change can "succeed" (config applied, no error) yet break the network (a session that was up is now down). Only a state diff around the change catches that. Codify the checks once and every change gets them for free — the difference between hoping and knowing.