Debugging Sequencers

Sequencers · Module 11 · Page 11.5

The Engineering Problem

The Sequencers module exposed the machinery — architecture (11.1), the arbitration engine (11.2), request management (11.3), scheduling (11.4) — and each chapter had its own failure mode. In practice, you don't get a labeled bug; you get a symptom: "my sequence drives nothing," "the test hangs," "responses are missing," "the interrupt was late." The hard part is mapping a symptom to its mechanism across all the things that can go wrong — and sequencer bugs are silent (a hang, a stall, a dropped response — rarely an error). Without a method, you guess; with one, you localize.

This chapter is that method: a systematic symptom-to-mechanism diagnosis for sequencers. The core move is localization: find where the flow is stuck — a hang at start_item is a grant problem; a hang at finish_item is an item_done problem; no stimulus at all is an objection or connection problem; wrong order is a scheduling problem; missing or overflowing responses is a request-management problem. Then map that location to its mechanism — connection, lock, relevance, objection, arbitration, set_id_info/get_response — using a few discriminating questions (Is the driver pulling? Does changing priority help? Is the objection raised?). And confirm with visibility tools — messages around the handshake, transaction recording, print_topology, and the sequencer's own warnings. This chapter is the methodology, the question set, and the tools.

How do you debug a sequencer systematically — localizing a symptom (a hang at start_item vs finish_item, no stimulus, wrong order, response problems) to its mechanism with discriminating questions, and confirming with the available visibility tools?

Motivation — why sequencer debugging needs a method

Sequencer bugs demand a systematic approach because they're silent, multi-cause, and symptom-ambiguous:

• The failures are silent, so there's no error to read. A starved sequence, a hung start_item, a dropped response — none throw an error (they hang, stall, or warn at most). So you can't grep for an exception; you have to reason from the symptom.
• One symptom has many causes, so you must discriminate. "No stimulus" could be a missing objection (killed), a missing connection (driver not pulling), a stuck is_relevant (excluded), a held lock (blocked), or a sequence not started on a sequencer. Guessing among five causes is slow; discriminating questions narrow it fast.
• Localization comes before mechanism, so find where first. Where the flow is stuck (start_item vs finish_item vs never-reached) halves the search before you consider why. A hang at start_item is a grant world; at finish_item, an item_done world — different mechanisms.
• The same questions discriminate repeatedly, so they're worth knowing. Does changing priority help? separates starvation (stage-2, yes) from exclusion (stage-1, no). Is the driver pulling? separates a connection bug from a sequence bug. A small question set covers most cases.
• Visibility is limited, so you instrument deliberately. There's no single "trace the sequencer" switch; you make state visible with handshake messages, transaction recording, topology, and the sequencer's warnings — chosen to confirm the localized hypothesis.

The motivation, in one line: sequencer failures are silent and multi-cause, so debugging them needs a method — localize the symptom (where it's stuck), discriminate the mechanism (with a few key questions), and confirm with deliberate visibility — rather than guessing among the many things that can go wrong.

Mental Model

Hold sequencer debugging as tracing a stuck assembly line back to the failed station:

Debugging a sequencer is tracing a stuck assembly line: the part flows through stations — order placed, order granted, part driven, completion acked, receipt returned — and a stoppage is at one station, which you find by checking where the part stopped, then why that station failed. The line has stages: a sequence places an order (start_item), the dispatcher grants it (arbitration), the driver builds the part (finish_item/driving), the station acks completion (item_done), and a receipt comes back (the response). When the line stalls, you don't tear apart every station — you walk the line and find where the part is sitting: stuck at the order desk (start_item — never granted)? at the build station (finish_item — never acked)? never even started (no objection / not on the line)? receipts piling up (response overflow)? Where it's stuck tells you which station failed, and then a targeted question tells you why that station failed (Is the dispatcher even reachable? Is it skipping this order? Did the build station forget to ack?). Then you confirm with the station's gauges (messages, recordings) before you fix it. The discipline is localize, then diagnose, then confirm — not replace random stations and hope.

So sequencer debugging is walk the line to the stoppage, then ask why that station failed: the location (which call is stuck) narrows the mechanism, a discriminating question identifies it, and visibility confirms it — a method, not a guess.

Visual Explanation — the symptom-to-mechanism decision tree

The defining tool is a decision tree: start from the symptom, branch on where it's stuck, and arrive at the mechanism.

The decision tree is the method made explicit. From the symptom, you branch on what you observe. No stimulus at all (nothing ever drives) → check the objection (was the sequence killed at time 0 because none was raised, Module 9.1?) and the connection (is the driver connected and pulling, Module 9.2?). A hang (progress stops) → localize the stuck call: a hang at start_item is a grant problem (the grant never came — no connection/non-pulling driver, a held lock/grab, is_relevant() stuck 0, or the sequence not started on a real sequencer); a hang at finish_item is an item_done problem (the driver took the item but skipped item_done, Module 9.4). Wrong order (items interleave wrongly, an atomic op broke, the interrupt was late) → scheduling (the arbitration mode/priority, a lock window, or priority inversion, Module 11.4). A response issue (a hung get_response, or "response queue overflow") → request management (a missing set_id_info → the response never routes → get_response hangs, Module 8.2; or responses never drained → the bounded queue overflows, Module 11.3). The tree's value is that it routes every symptom to a small set of mechanisms — and the first branch (what you see) plus the localization (where it's stuck) does most of the narrowing before you investigate any single mechanism. This is the spine of the methodology: observe → localize → mechanism.

RTL / Simulation Perspective — instrumenting the handshake

The first, highest-value debug step is making the handshake visible: messages around start_item/finish_item (sequence side) and get_next_item/item_done (driver side) show exactly where the flow is stuck. The code shows the instrumentation.

// ── SEQUENCE side: bracket the handshake with messages to see WHERE it stops ──
task my_seq::body();
  `uvm_info("DBG", "body() entered", UVM_LOW)              // did body even run? (objection/started?)
  req = my_item::type_id::create("req");
  `uvm_info("DBG", "calling start_item", UVM_LOW)
  start_item(req);                                          // if the next line never prints → GRANT stuck
  `uvm_info("DBG", "granted; randomize+finish_item", UVM_LOW)
  assert(req.randomize());
  finish_item(req);                                         // if the next line never prints → item_done stuck
  `uvm_info("DBG", "finish_item returned (driven)", UVM_LOW)
endtask
 
// ── DRIVER side: confirm the driver is PULLING and completing ──
task my_driver::run_phase(uvm_phase phase);
  forever begin
    `uvm_info("DBG", "calling get_next_item", UVM_LOW)      // is the driver pulling at all?
    seq_item_port.get_next_item(req);
    `uvm_info("DBG", "got item; driving", UVM_LOW)
    drive_on_pins(req);
    seq_item_port.item_done();                              // confirm item_done is reached on EVERY path
  end
endtask
 
// ── other visibility ──
//   print_topology()                  : confirm the sequencer/driver exist and are connected
//   transaction recording (8.7)       : see the scheduled item stream in the waveform
//   sequencer warnings                : "response queue overflow", "get_next_item twice", etc.
//   does changing priority/mode help? : starvation (yes) vs exclusion (no)

The instrumentation localizes the stall directly. On the sequence side, a message at body() entry answers "did the sequence even run?" (if not, it wasn't started or was killed at time 0 — objection, Module 9.1). A message before start_item and after it brackets the grant: if "calling start_item" prints but "granted" never does, the flow is stuck at the grant (Module 9.3/11.2 — connection, lock, relevance, or not-on-sequencer). A message after finish_item brackets the driving: if "granted" prints but "finish_item returned" never does, it's stuck at item_done (Module 9.4 — the driver skipped it). On the driver side, a message before get_next_item answers "is the driver pulling at all?" (if it never prints, the driver loop isn't running or isn't connected), and confirming item_done is reached on every path catches the skipped-completion bug. The other visibility (comments): print_topology confirms the components exist and are connected; transaction recording (Module 8.7) shows the scheduled item stream in the waveform; the sequencer's own warnings ("response queue overflow," "get_next_item twice without item_done") name specific failures; and the discriminating question "does changing priority/mode help?" separates starvation from exclusion (Module 11.2). The shape to carry: bracket the handshake with messages to localize where it's stuck — that single technique resolves the majority of sequencer stalls, because where almost always implies which mechanism.

Verification Perspective — the discriminating questions

The methodology turns on a few discriminating questions, each of which splits the cause space. Knowing the questions — and what each answer means — is the diagnostic core.

The discriminating questions are the fast path through the cause space. "Is the driver pulling?" — does get_next_item run (visible from the driver-side message)? No → the bug is driver/connection (the loop isn't running, or the port isn't connected, Module 9.2) — not the sequencer. Yes → the cause is sequencer-side (the driver wants items but none are granted). "Does changing priority or the arbitration mode help?" — the single most useful question for a never-granted sequence (Module 11.2): Yes, it helps → the sequence was competing and losing — starvation (stage-2, fix with priority/mode, Module 10.4); No, it does nothing → the sequence isn't a candidate at all — exclusion (stage-1: is_relevant stuck or lock-blocked, Module 11.2). This one question cleanly separates the two never-granted causes that look identical. "Did body() run?" — No → the sequence wasn't started or was killed (no objection, Module 9.1); Yes → look further along. "Is set_id_info present on the response?" — No → the response never routes → get_response hangs (Module 8.2); Yes → check draining (overflow, Module 11.3). The power of the question set is that each answer eliminates a chunk of the cause space, so a few questions pin the mechanism — far faster than investigating each possible cause in turn. The verification skill is asking the right discriminating question for the symptom: "no stimulus" → did body run? is the driver pulling?; "never granted" → does priority help?; "response problem" → set_id_info? draining?. The questions are the methodology.

Runtime / Execution Flow — the localize-discriminate-confirm loop

At run time, debugging follows a loop: localize where it's stuck, discriminate the mechanism with a question, confirm with a tool, fix — and repeat if a new symptom surfaces. The flow shows the disciplined cycle.

The loop is the disciplined alternative to guessing. Localize (step 1): bracket the handshake with messages (the RTL instrumentation) to find where the flow stopped — body() not running, stuck at start_item, stuck at finish_item, or a response problem. This narrows before you theorize. Discriminate (step 2): ask the splitting question for that location — Is the driver pulling? (connection vs sequencer), Does priority help? (starvation vs exclusion), Is set_id_info present? (response hang vs overflow) — to pin the mechanism. Confirm (step 3): use a visibility tool to verify the hypothesis before fixing — print_topology to confirm the connection, transaction recording (Module 8.7) to see the schedule (an interleave vs a lock window vs a missing sequence), or the sequencer's warnings to confirm a response-queue overflow or a "get_next_item twice." Fix (step 4): apply the mechanism-specific fix (connect the port, release the lock, raise the objection, add set_id_info, drain responses, shorten the lock window), then re-run — and if a new symptom appears, the loop restarts. The runtime value of the loop is that it's convergent: each pass narrows (localize halves it, discriminate pins it, confirm verifies it), so you reach the cause in a bounded number of steps rather than an open-ended hunt. And it's transferable: the same loop debugs any sequencer symptom, because the steps (localize/discriminate/confirm) are symptom-agnostic even though the answers differ. This loop, run with the decision tree and question set, is the whole methodology in operation.

Waveform Perspective — a debug trace localizing a stall

A debug trace makes the localization visible: the handshake messages (or recorded transactions) show exactly where the flow stopped, turning a vague "it hangs" into a precise stuck stage.

The waveform shows localization in action — the messages converting a vague hang into a precise diagnosis. The dbg trace shows the sequence's messages: "body" (entered) and "start" (calling start_item) print — and then nothing more, because start_item never returns (si_wait goes high and stays high). Already, two causes are eliminated: "body entered" means the sequence did start and wasn't killed at time 0 (not an objection bug, Module 9.1), and the absence of any message after start_item localizes the stall precisely at the grant. The drv_pull trace shows the driver is calling get_next_item (pulsing) — so the driver is connected and is pulling (not a connection bug, Module 9.2): the driver wants an item, but the sequencer isn't granting this sequence. And granted stays blank — no grant is issued to this sequence, confirming the grant is the stuck stage. So the trace has narrowed "it hangs" all the way to "stuck at start_item, the driver is pulling, but no grant is issued" — which (per the decision tree) leaves only the grant-side mechanisms: a held lock excluding it, or is_relevant() stuck 0 (Module 11.2). The discriminating question "does priority change anything?" would then split those from starvation (it won't, since it's exclusion). The picture is the methodology's payoff: a few messages turn an opaque hang into a located, mechanism-narrowed diagnosis — where it's stuck (start_item), what's ruled out (objection, connection), and what remains (lock or relevance) — at which point the fix is a targeted one, not a guess.

DebugLab — localizing a "sequence drives nothing" to its mechanism

SYMPTOM: this sequence drives nothing (others work, no error)
 
(1) LOCALIZE — bracket the handshake with messages:
    "body entered" PRINTS        → the sequence DID start (rules out: not-started, no-objection-kill)
    "calling start_item" PRINTS  → reached the handshake
    "granted" NEVER prints       → STUCK AT start_item (the grant never comes)
    ⇒ localized to a GRANT problem
 
(2) DISCRIMINATE — ask the splitting questions for a grant stall:
    Is the driver pulling?        driver prints "get_next_item" → YES (rules out: connection/driver bug)
    Is a lock held by another?    no lock/grab anywhere active   → NO  (rules out: lock-blocked)
    Is it on a real sequencer?    started via start(sqr)         → YES (rules out: not-on-sequencer)
    Does changing priority help?  NO change                      → EXCLUSION, not starvation
    ⇒ remaining cause: is_relevant() filtering it out
 
(3) CONFIRM — check is_relevant:
    the sequence's base class overrides is_relevant() to gate on a flag never set → returns 0 always
    ⇒ CONFIRMED: filtered out of every grant (Module 11.2)
 
FIX: make is_relevant() return 1 when it should run (+ wait_for_relevant); ensure the gate fires.

Common Mistakes

• Guessing among causes instead of localizing. "Drives nothing" has many causes; bracket the handshake with messages to find where it's stuck before theorizing.
• Not distinguishing starvation from exclusion. "Does changing priority help?" splits them — tuning priority for an excluded (filtered) sequence wastes time.
• Ignoring the localization signal. A hang at start_item (grant) and at finish_item (item_done) are different worlds; identify which call is stuck first.
• Overlooking the sequencer's warnings. "response queue overflow" / "get_next_item twice" name the bug; silencing them hides real failures.
• Forgetting to check the driver side. A sequencer stall is often a driver not pulling (or not completing); confirm get_next_item/item_done are running.
• Not using transaction recording for schedule bugs. Wrong-order, interleaving, and inversion are clearest in the recorded item stream — reason about the schedule visually.

Senior Design Review Notes

Interview Insights

Exercises

1. Localize the stall. Given handshake messages where "body entered" and "calling start_item" print but nothing after, state where the flow is stuck and which mechanisms remain.
2. Pick the question. For symptoms (a) never granted, (b) drives nothing, (c) get_response hangs — name the discriminating question you'd ask first.
3. Walk the tree. A sequence drives nothing; the driver is pulling, no lock is held, it's on the sequencer, and priority changes don't help. Name the remaining cause.
4. Choose the tool. For (a) a suspected connection bug, (b) a wrong-order/interleaving bug, (c) a response overflow — name the visibility tool that best confirms each.

Summary

• Debugging a sequencer is a symptom-to-mechanism method, needed because sequencer failures are silent, multi-cause, and symptom-ambiguous — so you localize, discriminate, and confirm rather than guess.
• Localize with a decision tree on where it's stuck: start_item hangs → grant (connection / lock / is_relevant / not-on-sequencer); finish_item hangs → item_done (driver skipped it); no stimulus → objection / connection; wrong order → scheduling; response issue → request management.
• Discriminate the mechanism with a few splitting questions: Is the driver pulling? (connection vs sequencer), Does changing priority help? (starvation vs exclusion), Is set_id_info present? (response hang vs overflow), Did body() run? (objection vs further) — each eliminates a chunk of the cause space.
• Confirm with visibility tools: handshake messages (localize), print_topology (connection), transaction recording (the schedule), and the sequencer's own warnings ("response queue overflow," "get_next_item twice") — verify before fixing.
• The durable rule of thumb: for any sequencer symptom, run the localize-discriminate-confirm loop — bracket the handshake with messages to find where it's stuck, ask the splitting question to pin the mechanism, and confirm with a tool — because the method converges on the cause regardless of which of the many silent failure modes it is.

Next — Driver Responsibilities: the sequencer hands items to the driver; the next module turns to the driver itself — its responsibilities, how it converts transactions into pin activity, and the role it plays as the other end of the handshake you've been debugging.