← Interview Prep

Troubleshooting Scenarios

Physical and L1–L2 failures, counter by counter: what increments, on which side, why, and how to fix it.

Almost every "the network is broken" ticket that reaches an engineer is really an L1/L2 problem wearing an application costume. The discipline that separates a senior answer from a junior one is simple: read the interface counters, know which side they increment on, and watch the rate rather than the total. State the governing idea out loud before touching anything:

Counters are directional. Input errors are counted where the frame was received; output drops are counted where the frame was trying to leave. A CRC error means the frame arrived corrupted — so it was already damaged on the wire before this port saw it. Always compare both ends of the link; a single side never tells the whole story.

Interface counter primer

The table below is the reference every other section builds on. "Side" tells you where the counter lives: ingress (RX, the receiving port) or egress (TX, the sending port). Getting the side right is half of the diagnosis.

Memory hook: if it's corrupted on arrival, it's an input/CRC counter; if it's dropped because there was nowhere to put it on the way out, it's an output discard. Errors = something is wrong with the bits. Discards = the bits were fine, we chose (or were forced) to drop the frame.

Clearing counters — should you?

The trap that catches juniors: interface counters are cumulative since the last clear or reboot — they do not reset on their own. A port showing 8471 CRC tells you nothing until you know over what window: 8471 errors accumulated over two years of uptime is noise; 8471 in the last five minutes is a fire. Always read the "Last clearing of counters" line (Cisco) / uptime and reason about the rate, not the absolute number.

The number to trust is a rate (errors/second) or a ratio (errors ÷ total frames), not a raw total. A handful of CRCs over billions of packets is a one-off; a climbing ratio is a real fault.

Clear, or don't?

The better move: compute a rate without clearing

You almost never need to clear. Take two timestamped snapshots a known interval apart and subtract — you get the current error rate and keep all history:

show interface Gi0/1        # note CRC = N1, and the time
  ... wait 60s ...
show interface Gi0/1        # CRC = N2
  => error rate = (N2 - N1) / 60  errors/sec  (0 = not incrementing now)

Even better, let monitoring do it: SNMP/streaming-telemetry deltas, or gNMI ON_CHANGE on the error leaves (see gNMI & Telemetry) give you the rate continuously without touching anything.

If you do clear — the strategy

1. Record first. Save the full show interface output (and the "last clearing" time / uptime) before clearing, so the evidence isn't lost.
2. Clear only the affected interface, not the whole box: clear counters Gi0/1 (Cisco IOS) / clear interfaces statistics ge-0/0/1 (Junos). Note the exact timestamp.
3. Warn the team / NMS — the baseline reset will perturb dashboards and delta-based alerts.
4. Watch the increment during a controlled test (link up, traffic, a specific change) and read the fresh rate.

CRC errors

A CRC (FCS) error is the single most common L1 symptom. The frame carried a 4-byte checksum; the receiver recomputed it over the received bits and got a different value → the frame was corrupted in transit and is dropped. It increments on the ingress / receiving side, because that's the only side that can detect the mismatch.

Common causes, roughly in order of frequency

• Duplex mismatch — by far the most common on copper. One end is full-duplex, the other half-duplex; the full-duplex side records CRC/FCS and input errors. Covered in detail below.
• Bad or dirty cabling — kinked copper, damaged pairs, marginal RJ-45, or a dirty/scratched fiber connector end-face. Dirt on a fiber ferrule is the number-one cause of optical CRCs.
• Failing optics / SFP — a transceiver degrading toward end of life, producing marginal signal; swap it to confirm.
• EMI / interference — copper run near motors, fluorescent ballasts, power cabling; corrupts bits intermittently.
• Bend radius violations on fiber — light leaks out at tight bends, lowering received power until frames corrupt.
• Wrong/degrading media — the wrong cable category, an out-of-spec run length, or a bad patch panel port.

Cut-through switching caveat: a store-and-forward switch validates the FCS and drops corrupted frames, so CRC errors stop at that switch. A cut-through switch starts forwarding after reading only the destination MAC — before it has seen the FCS — so it propagates corrupted frames downstream. The result: CRC errors appear several hops away from the actual bad link. When counters show CRCs but the local cabling is clean, walk upstream toward the source. (Modern cut-through switches can auto-transition to store-and-forward when error rates rise — "adaptive cut-through" — precisely to stop this.)

What CRC counters look like

Router# show interfaces GigabitEthernet0/1
  ...
  5 minute input rate 220000 bits/sec, 41 packets/sec
     1283746 packets input, 984733120 bytes, 0 no buffer
     Received 12043 broadcasts (0 IP multicasts)
     0 runts, 0 giants, 0 throttles
     8471 input errors, 8471 CRC, 0 frame, 0 overrun, 0 ignored
     0 watchdog, 0 multicast, 0 pause input
     ...
     0 output errors, 0 collisions, 1 interface resets

Here input errors == CRC and nothing else — a clean "the bits are arriving corrupted" signature. No runts, no overruns → not congestion, not a buffer problem. Check the cable, the optics, and the duplex on both ends.

Duplex mismatch — the classic signature

This is a favorite interview question because the signature is so specific. It happens when one end auto-negotiates and the other is hard-set (or two ends disagree): a common outcome is one side at full duplex and the other at half duplex. The link comes up, small flows work, and the users complain that it "works but is painfully slow."

The tell-tale split of counters

One-line answer for the interview: "Late collisions and runts on one end, CRC/FCS and input errors on the other, with terrible throughput under load — that's a duplex mismatch until proven otherwise." The throughput collapses specifically as offered load rises, because that's when both ends try to transmit simultaneously.

How to confirm and fix

1. Look at both ends. If one shows late collisions and the peer shows CRC, you've confirmed it — no other fault produces that split.
2. Check negotiated state: show interfaces status — a mismatch often reads a-full on one side and half (no "a-") on the other, or one port shows a speed/duplex the other doesn't.
3. Fix by making both ends agree. Best practice: either autoneg on both ends, or hard-set speed and duplex on both ends. The dangerous half-state is one end auto and the other hard-coded — that's what creates the mismatch. On modern gigabit+ links, leave autoneg on both ends (gigabit essentially requires it).

Switch# show interfaces status
Port      Name    Status    Vlan   Duplex   Speed  Type
Gi1/0/5           connected 10     a-full   a-1000 10/100/1000BaseTX
Gi1/0/6           connected 10       half      100 10/100/1000BaseTX   <-- mismatch

! remedy: pin both ends identically
Switch(config)# interface Gi1/0/6
Switch(config-if)# speed 1000
Switch(config-if)# duplex full

Input vs output discards

Discards are clean frames intentionally dropped — no wire corruption. The direction tells you the whole story of why.

The quick mental test: output discards mean "too much traffic wanted to leave through this port at once." Input discards mean "the box received it fine but a rule, a policer, or a resource limit said no." Output discards climb with utilization; input discards track a policy or a punt path.

Switch# show interfaces GigabitEthernet1/0/24 | include drops|rate
  5 minute input rate 640000 bits/sec, 120 packets/sec
  5 minute output rate 986000000 bits/sec, 92000 packets/sec
  Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 148213
     <-- output drops rising while the average is under line rate => microbursts

Note the trap: the 5-minute average output rate can look comfortably below line rate while Total output drops keeps climbing. That gap is the fingerprint of a microburst — see the microburst note below.

Physical-layer & optical diagnostics

When counters point at L1, go to the transceiver's own telemetry. DOM/DDM (Digital Optical Monitoring / Digital Diagnostics Monitoring) exposes real-time Tx power, Rx power, temperature, bias current, and voltage — usually via show interfaces transceiver or show interface transceiver detail.

Reading optical power (dBm)

• Optical power is in dBm (log scale). 0 dBm = 1 mW; every -3 dB is roughly half the power. More negative = weaker.
• Each optic has a receive sensitivity floor and a Tx range in its spec. An Rx below the floor = low-light → errors/flaps. An Rx above the max = overload (too little attenuation on a short run).
• Low light usually means: dirty/scratched connector, bend-radius loss, too many patch panels, a failing far-end laser, or the wrong-distance optic (SR optic on a long run).

Switch# show interfaces TenGigabitEthernet1/1/1 transceiver detail
          Optical   Optical
          Transmit  Receive
Port      Power     Power     Temperature  Voltage  Current
          (dBm)     (dBm)     (Celsius)    (Volts)  (mA)
--------  --------  --------  -----------  -------  -------
Te1/1/1    -2.3      -18.9 --  41.2         3.28     6.4
                     ^^^^^ below Rx sensitivity (-14.4 dBm) => low-light alarm

! thresholds shown by the platform:
!   Rx power:  high alarm -1.0 / high warn -2.0 / low warn -13.9 / low alarm -14.4 dBm
!   -18.9 dBm is under the low ALARM => clean the fiber, check bend radius, reseat/replace

Other L1 faults to name

MTU / jumbo-frame mismatch

An MTU mismatch has a signature all its own: small frames pass, large frames are silently dropped. Ping with a small size works; a file transfer or anything with full-size packets stalls. Symptoms that "ping works but SSH/HTTP hangs" or "the connection establishes then freezes when data flows" often trace to MTU.

• Jumbo mismatch: one switch/interface allows 9216-byte jumbos, the next hop is at the default 1500. Full-size jumbo frames arrive as giants and are dropped; small frames sail through.
• PMTUD black hole: a router along the path needs to fragment (DF=1), so it returns ICMP "fragmentation needed" (type 3, code 4) telling the sender to lower its packet size. If a firewall filters that ICMP, the sender never learns, keeps sending too-large packets that keep getting dropped → the TCP session hangs after the handshake. This is the classic "handshake OK, bulk data dead" black hole.
• How it presents: ping succeeds, ping with a large size and DF set fails, throughput is fine for interactive traffic and zero for bulk transfers.

! prove it: small ping ok, large-with-DF fails at the mismatch point
Router# ping 10.0.30.22 size 1500 df-bit
  .....  Success rate is 0 percent (0/5)
Router# ping 10.0.30.22 size 1400 df-bit
  !!!!!  Success rate is 100 percent (5/5)
! => the path chokes somewhere between 1400 and 1500; find the small-MTU hop

Fix: make MTU consistent end-to-end (all switches in an L2 domain, both ends of a routed link), and never filter ICMP unreachables that PMTUD depends on. On tunnels, consider MSS clamping.

Other high-value cases

A grab-bag of failures worth being able to recognize on sight. Each has a distinctive fingerprint.