Monitor Debugging

Monitors · Module 13 · Page 13.5

The Engineering Problem

The module built the monitor up: it observes passively (Module 13.1), reconstructs transactions (Module 13.2), checks the protocol (Module 13.3), and broadcasts (Module 13.4). This chapter closes the module with the hardest case: when the monitor itself is wrong. A driver bug is visible on the pins — the driver drives, so its bug appears directly as wrong or X values on the interface (Module 12.6 localized driver bugs on the pins). A monitor bug is the opposite: the monitor drives nothing (Module 13.1), so a monitor bug produces no pin-level symptom whatsoever — the pins are unaffected. The bug surfaces only downstream — a scoreboard mismatch, a coverage hole, a count that's off — and, worst of all, it frames the DUT: the scoreboard accuses the design of a bug the design never committed. Teams burn days debugging an innocent DUT because the monitor lied. The problem this chapter solves is how to debug the monitor — how to tell a monitor bug from a DUT bug when the symptom points at the DUT.

Monitor debugging rests on one discriminating principle: the pins are ground truth. Because the monitor only observes — it can't affect the pins — the interface itself is the objective record of what actually happened, independent of the monitor. So when a downstream check fails, you do not trust the monitor: you go to the waveform (the pins), establish what truly occurred, and compare the monitor's reported stream against that ground truth. Wherever they diverge is the monitor bug — and the kind of divergence classifies it: a transaction on the pins but missing from the stream (missed activity), in the stream but not on the pins (phantom), present in both but with a wrong field (mis-reconstruction), or correctly observed but never delivered (broadcast gap). This chapter is that methodology: why a monitor bug is invisible and frames the DUT, the pins-as-ground-truth discriminator, the divergence catalog, and the back-to-back miss that costs a DUT team a week.

How do you debug the monitor itself — a bug invisible on the pins that surfaces only downstream and frames the innocent DUT — by treating the pins as ground truth and classifying where the monitor's reported stream diverges from the waveform?

Motivation — why a monitor bug is uniquely deceptive

A monitor bug is the most deceptive kind of testbench bug, and the reasons explain why it needs its own methodology:

• It leaves no fingerprint on the interface. The monitor drives nothing, so a monitor bug does not change the pins. There's nothing wrong to see on the interface — unlike a driver bug, which is on the pins. The bug is purely in the monitor's internal view.
• It surfaces far from its cause. The bug manifests downstream — in the scoreboard's comparison, coverage's count, a checker's result — cycles or components away from the monitor. The symptom and the cause are separated.
• It frames the DUT. The downstream symptom — "transactions dropped," "wrong data," "missing coverage" — looks exactly like a DUT bug. The natural response is to debug the DUT, which is innocent — a days-long dead end.
• It can also hide real DUT bugs. A monitor that mis-observes can make a broken DUT look fine (Module 13.1's intent-not-observation), so the testbench passes while the DUT is wrong. The monitor's error cuts both ways: false alarms and missed alarms.
• The usual instinct — trust the testbench — is exactly wrong here. Engineers trust their checkers; a monitor bug weaponizes that trust, making a correct DUT look broken. The methodology must invert the instinct: suspect the monitor first when the symptom is downstream.

The motivation, in one line: a monitor bug is invisible on the pins, surfaces downstream, and frames the innocent DUT — so it can't be debugged like a driver bug (which lives on the pins); it needs a methodology that treats the pins as ground truth and cross-examines the monitor's view against the waveform, because the one thing you cannot trust when the monitor might be wrong is the monitor.

Mental Model

Hold monitor debugging as cross-examining a witness against the security-camera footage:

The pins are the security-camera footage — the objective record of what happened. The monitor is a witness giving testimony about it. When the testimony and the footage disagree, the witness is wrong, not reality — so you debug by replaying the footage and finding where the account diverges. Picture an incident. The security camera (the interface, captured on the waveform) recorded exactly what happened — objectively, independent of anyone's account. The monitor is a witness who watched and now gives testimony (the reconstructed transaction stream) about what occurred. Most of the time the testimony matches the footage. But when a downstream decision goes wrong — the scoreboard (the judge) convicts the DUT (the defendant) — you must ask: is the testimony actually true? You replay the footage (open the waveform) and check the witness's account against it, point by point. Where they agree, fine. Where they diverge, the witness is wrong — because the footage is ground truth and the witness only watched, changing nothing. The kinds of divergence tell you how the witness erred: the footage shows an event the witness never mentioned (the witness missed it), the witness describes an event the footage doesn't show (the witness imagined it), the witness got an event's details wrong (mis-described it), or the witness gave correct testimony that never reached the judge (a delivery failure). Crucially, a driver bug is different — it's the footage showing an actor tampering with the scene; you see it in the footage itself. A monitor bug is a false account of a correct scene — the footage is clean, the testimony is wrong — and the only way to catch it is to trust the footage over the witness.

So monitor debugging is trusting the footage over the witness: the pins are ground truth (the monitor changed nothing, so they're objective), and the monitor's stream is the suspect testimony. You replay the waveform, compare the monitor's account against it, and classify the divergence to localize the bug. The fatal instinct — believe the witness (trust the checker) — is exactly what a monitor bug exploits; the methodology flips it: when the monitor might be wrong, the pins decide.

Visual Explanation — the bug is invisible on the pins, and frames the DUT

The defining picture is the divergence: the pins are correct, the monitor's view is wrong, and the scoreboard blames the DUT — the innocent-DUT structure.

The figure shows the innocent-DUT structure that defines monitor debugging. The DUT drives the interface correctly — the pins are ground truth and they are right. The monitor observes them but mis-handles what it saw — it missed a transaction, mis-built one, or dropped one — so its reported stream diverges from the pins. The scoreboard, trusting the monitor's stream (it has no other source), compares the wrong data against expected and flags a mismatch — blaming the DUT. The crucial reading is where the divergence happens versus where the symptom appears. The divergence is at the monitor — the pins (correct) and the stream (wrong) split there. But the symptom appears downstream, at the scoreboard, pointing further downstream at the DUT. The bug and the accusation are at opposite ends: the bug is in the monitor, the accusation lands on the DUT, and the correct pins sit between them, unexamined. This is why the bug is so deceptive — every arrow downstream of the monitor carries the lie, and the natural debug direction (follow the scoreboard's accusation) leads away from the cause, to the innocent DUT. The methodology's whole move is to refuse that direction: go back to the warning-colored pins — the ground truth — and compare the monitor's stream against them. The divergence reveals itself at the monitor, where the correct pins and the wrong stream part ways. The diagram is the argument for suspecting the monitor first: when the symptom is downstream and the pins are correct, the monitor — not the DUT — is where reality and the report diverged.

RTL / Simulation Perspective — the four divergence classes

When the monitor's stream diverges from the pins, the kind of divergence classifies the bug. There are four shapes, each with a distinct signature and a distinct cause. The code sketches how each arises.

// (1) MISSED: a transaction on the pins never enters the stream — wrong/too-slow sampling
@(posedge vif.clk iff vif.valid);
tr = reconstruct();                 // if reconstruct() spans cycles, a back-to-back txn is MISSED
ap.write(tr);                       //   pins show N transactions, stream has < N  → "DUT dropped them"
 
// (2) PHANTOM: a transaction in the stream that never happened on the pins — over/double sampling
@(posedge vif.clk);                 // ✗ sampling without a proper valid qualifier
ap.write(reconstruct());            //   emits on cycles with NO real transaction → stream has > N
 
// (3) MIS-RECONSTRUCTED: transaction present but a field is wrong — wrong phase (13.2)
tr.addr = vif.addr;                 // correct
tr.data = vif.data;                 // ✗ sampled at the wrong phase → stale value (right txn, wrong field)
ap.write(tr);                       //   count matches, but a FIELD diverges → "DUT corrupted data"
 
// (4) NOT-BROADCAST: observed and built correctly, but never delivered — wiring (13.4)
ap = new("ap", this);               // ✗ port created but never connected, or wrong imp
// ... reconstruct() correct, ap.write(tr) called ...   subscriber receives NOTHING → "DUT silent"
 
// The DISCRIMINATOR for all four: compare the stream against the WAVEFORM (ground truth).
//   on pins, not in stream → MISSED   |  in stream, not on pins → PHANTOM
//   in both, field differs  → MIS-RECON | correct but undelivered → NOT-BROADCAST

The code shows the four divergence classes and their root mechanisms. (1) Missed — a transaction is on the pins but never enters the stream: the sampling is wrong or too slow (e.g., reconstruct() spans cycles, so a back-to-back transaction arrives while the monitor is still busy, and is never sampled). Signature: pins show N, stream has fewer — looks like "the DUT dropped transactions." (2) Phantom — a transaction is in the stream but never happened: over- or double-sampling (e.g., sampling without a proper valid qualifier), emitting on cycles with no real transaction. Signature: stream has more than the pins — looks like "the DUT sent extras." (3) Mis-reconstructed — a transaction is present in both but a field is wrong: the wrong-phase sampling of Module 13.2 (data sampled before its phase → stale). Signature: count matches, a field diverges — looks like "the DUT corrupted data." (4) Not-broadcast — the monitor observed and built correctly but the transaction is never delivered: a wiring fault (Module 13.4 — port unconnected, wrong imp). Signature: subscriber receives nothing despite correct observation — looks like "the DUT went silent." The unifying discriminator, stated in the closing comment, is the whole methodology: compare the stream against the waveform — on pins but not in stream → missed; in stream but not on pins → phantom; in both but field differs → mis-reconstructed; correct but undelivered → not-broadcast. The shape to carry: a monitor bug is always a divergence between the stream and the pins, and the four signatures — too few, too many, wrong field, undelivered — each point at a specific cause. Classify the divergence, and you've localized the bug.

Verification Perspective — the divergence catalog and its false accusations

Each divergence class produces a characteristic downstream symptom — and each symptom looks like a DUT bug. Knowing the mapping lets you translate a false accusation back to its real monitor cause.

The figure is the translation table between monitor causes and their DUT-looking symptoms. Each monitor-bug class (left) produces a characteristic downstream symptom (middle) that accuses the DUT: missed activity → "the DUT dropped transactions" (count too low); phantom → "the DUT sent extras" (count too high); mis-reconstruction → "the DUT corrupted data" (field mismatch); not-broadcast → "the DUT went silent" (nothing received). All four are monitor bugs wearing a DUT costume. The verification insight is that the symptom alone is ambiguous — "the DUT dropped transactions" could be a real DUT bug or a monitor missing them, and the scoreboard cannot tell the difference (it only sees the stream). The disambiguator is the same for all four: compare against the pins. The warning-colored node — compare vs pins — is the single discriminator that resolves every symptom: the pins say what really happened, and that either confirms the DUT bug (the pins are wrong — a real DUT problem) or exonerates the DUT and points at the monitor (the pins are right, the stream is wrong — a monitor bug). This is the methodology's power: it turns an ambiguous, DUT-accusing symptom into a definite verdict by checking the one source the monitor couldn't corrupt — the interface itself. The practical discipline this mandates: when the scoreboard accuses the DUT, do not start in the DUT — first compare the monitor's stream against the pins. If they match, the DUT bug is real, debug the design. If they diverge, the monitor lied, and the divergence class tells you which monitor bug to fix. The figure is the map from false accusation to true cause — and the pins are the territory that settles it.

Runtime / Execution Flow — the localization procedure

The methodology is a fixed procedure: establish ground truth from the pins, compare the monitor's stream against it, classify the divergence, fix the indicated cause. The flow shows the steps.

The flow is the discipline that inverts the natural instinct. Establish ground truth (step 1): when a downstream check fails, do not start in the DUT — open the waveform and determine what actually happened on the pins, independent of the monitor. The pins are trustworthy precisely because the monitor couldn't touch them. Compare (step 2): match the monitor's reported stream against the pin activity — transaction by transaction — asking, for each, does the stream agree with the pins? Classify (step 3): where they diverge, name the class — missed (on pins, not in stream), phantom (in stream, not on pins), mis-reconstructed (in both, field differs), not-broadcast (correct but undelivered) — or, if the stream matches the pins and the pins are wrong, confirm a real DUT bug. Fix (step 4): fix the indicated cause — the monitor for a divergence (correct the sampling, the phase, the qualifier, the wiring), the DUT only if the pins themselves are wrong. The runtime insight is that this procedure always terminates in the right place because it's anchored to ground truth: every monitor bug is defined as a stream-vs-pins divergence, so comparing against the pins is guaranteed to separate monitor bugs from DUT bugs — there is no monitor bug that survives a pins comparison (the pins are what the monitor was supposed to report). The one discipline the procedure enforces — and the one engineers most often skip — is step 1: go to the pins first. Skipping it (trusting the scoreboard's accusation and diving into the DUT) is the days-long dead end. The flow is the methodology's guarantee: anchor to the pins, compare, classify, fix — and the innocent DUT is exonerated the moment you look at the interface it actually drove.

Waveform Perspective — the pins say what really happened

The discriminator is literally a waveform read: the pins show what happened, the monitor's reported stream shows what it claimed, and the divergence is visible between them. The waveform shows a missed transaction.

The waveform is the discriminator in action. The pins — ground truth — show two transactions: data=2A valid at cycle 1, then data=2B back-to-back at cycle 2. The DUT produced both, correctly, right there on the interface. But mon_emit fires only once (at cycle 2): the monitor was mon_busy reconstructing transaction #1 when #2 arrived, and missed it — so the reported stream contains one transaction where the pins show two. The crucial visual is the divergence, side by side: the pins say two, the stream (mon_emit) says one. The scoreboard, seeing one transaction where the DUT produced two, reports "the DUT dropped a transaction" — framing the DUT. But the waveform exonerates it instantly: both transactions are plainly on the pins; the DUT did nothing wrong. The divergence — two on the pins, one in the stream — classifies the bug as missed activity, a monitor bug (the observe-loop can't keep up with back-to-back traffic). Reading the waveform for monitor correctness — count the transactions on the pins, count the monitor's emits, do they match? does every pin transaction have a corresponding emit with the right fields? — is the entire discriminator. The picture to carry: the pins and the monitor's stream are two accounts of the same events, and monitor debugging is laying them side by side on the waveform — where they diverge, the monitor is wrong, and how they diverge (missing, extra, wrong-field, undelivered) names the bug. The pins never lie; the monitor can — so the waveform is the court of final appeal.

DebugLab — the week spent debugging an innocent DUT

✗ observe and reconstruct INLINE — the loop misses arrivals while busy:
  forever begin
    @(posedge clk iff valid);     // sample transaction #1
    tr = reconstruct_over_cycles(); //  spends several cycles here...
    ap.write(tr);                 //  ...transaction #2 arrived back-to-back and was MISSED
  end
  // pins: N transactions  |  stream: < N  →  scoreboard: "DUT dropped transactions"
  // DUT is INNOCENT — the waveform shows every transaction present on the pins
 
✓ DECOUPLE sampling from reconstruction — never miss an arrival:
  // thread A: capture every qualifying arrival immediately into a queue (zero-time)
  forever begin @(posedge clk iff valid); raw_q.push_back(snapshot_pins()); end
  // thread B: reconstruct from the queue off the critical path
  forever begin raw_q.get(raw); ap.write(reconstruct(raw)); end
  // now every pin transaction is captured; stream count == pin count

Common Mistakes

• Starting in the DUT when the symptom accuses it. A downstream mismatch can be a monitor bug framing the DUT; go to the pins (ground truth) before investigating the RTL.
• Trusting the monitor's stream as truth. The monitor can be wrong; only the pins are ground truth, because the monitor drives nothing and can't alter them.
• Reconstructing inline on the sampling path. Spending cycles reconstructing blocks observation; back-to-back transactions are missed under load — decouple sampling from reconstruction.
• Testing only spaced traffic. A monitor that can't keep up passes on spaced traffic and fails on back-to-back; stress-test with no idle gaps.
• Not classifying the divergence. Fixing without identifying missed vs phantom vs mis-reconstructed vs not-broadcast targets the wrong mechanism.
• Forgetting a monitor bug can hide DUT bugs too. A mis-observing monitor can make a broken DUT pass; a suspiciously perfect pass also warrants a pins comparison.

Senior Design Review Notes

Interview Insights

Exercises

1. Invert the instinct. A scoreboard reports a data mismatch accusing the DUT. State the first thing you do and why, before touching the RTL.
2. Classify from the symptom. Map each to a monitor-bug class: count too low; a field wrong with a correct bus; subscriber receives nothing; count too high.
3. Diagnose the load dependence. A failure appears only under back-to-back traffic. Explain why this points at the monitor, and the fix.
4. Design the capture. Sketch the two-thread structure that lets a monitor never miss a back-to-back arrival.

Summary

• A monitor bug is invisible on the pins — the monitor drives nothing, so its bug leaves the interface untouched and surfaces only downstream (scoreboard mismatch, coverage hole, wrong count), framing the innocent DUT.
• The discriminating principle is the pins are ground truth: because the monitor only observes, the waveform is the objective record of what actually happened, independent of the monitor — the one source a monitor bug cannot corrupt.
• The methodology: when a downstream check accuses the DUT, go to the pins first — establish ground truth from the waveform, compare the monitor's stream against it transaction by transaction, and classify the divergence.
• The four divergence classes and their DUT-looking symptoms: missed (count too low → "DUT dropped"), phantom (count too high → "DUT sent extras"), mis-reconstructed (field differs → "DUT corrupted data"), not-broadcast (nothing received → "DUT silent") — only if the pins themselves are wrong is the DUT guilty.
• The durable rule of thumb: never debug the DUT on a downstream accusation without first checking the pins — the monitor's word is not ground truth, the interface is; compare the monitor's stream against the waveform, and a divergence (missed, phantom, mis-reconstructed, not-broadcast) is a monitor bug, not a DUT bug. And build monitors that can't miss: decouple sampling from reconstruction so back-to-back traffic never slips past.

Next — Agent Architecture: the monitor, driver, and sequencer are now complete — the next module assembles them. Module 14 opens with the agent: the reusable container that bundles a driver, sequencer, and monitor into a single, configurable unit (active or passive) that encapsulates everything needed to drive and observe one interface.