Debug-Brief Generator: Paste a Stack Trace, Get a Structured Bug Report with Root-Cause Hypotheses

The task

You're on-call for an SDK or an agent-loop service. A stack trace lands in Slack or PagerDuty and you need to write a triage brief — summary, likely root cause, blast radius, next checks — before anyone can act on it. This workflow turns raw traces into a structured, hypothesis-ranked bug report you can paste into an incident ticket.

Before AI

Manually: read the trace, guess the failing module, grep the repo, cross-reference recent PRs, then draft a bug report with reproduction steps and hypotheses. For an unfamiliar module or a nested async trace, this is 20-40 minutes of scrolling and typing before you've even opened the debugger. Frontier models have gotten sharp enough at this that a recent write-up on SWE-bench Verified reports fix rates above 80% when the model is fed the right runtime context — which is exactly what a good debug brief provides.

The workflow

Step 1 — Extract and normalize the trace

Give the model the raw trace plus any surrounding log lines. The first pass strips noise and identifies the failure signature. Keep the sample input realistic — include timestamps, thread IDs, and the top framework frames the way they'd actually appear in your logs.

You are a senior backend engineer triaging a production stack trace from a Python service that runs an LLM agent loop (tool-calls, streaming, retries).

From the raw log below:
1. Identify the exception type, the innermost application frame (ours, not library), and the outermost framework frame.
2. Strip library-internal frames unless they carry the actual failure. Keep line numbers.
3. Summarize the failure signature in one line: "<ExceptionType> in <module>:<function> during <operation>".
4. List any correlation IDs, request IDs, model names, or tool names visible in surrounding log lines.

Output as markdown with headings: **Signature**, **Cleaned Trace**, **Context IDs**.

Raw log:

2026-08-18T14:22:03.114Z INFO  [req_id=r-8f42a1] agent.loop starting turn=3 model=claude-sonnet-4 tools=[search_docs,run_sql]
2026-08-18T14:22:04.882Z DEBUG [req_id=r-8f42a1] tool_call name=run_sql args={"query":"SELECT * FROM orders WHERE tenant_id=$1"}
2026-08-18T14:22:05.017Z ERROR [req_id=r-8f42a1] Unhandled exception in agent turn
Traceback (most recent call last):
  File "/app/agentkit/loop.py", line 214, in _run_turn
    result = await self._dispatch_tool(call, ctx)
  File "/app/agentkit/loop.py", line 341, in _dispatch_tool
    return await handler(call.arguments, ctx)
  File "/app/tools/sql.py", line 88, in run_sql
    rows = await conn.fetch(query, *params)
  File "/usr/local/lib/python3.11/site-packages/asyncpg/connection.py", line 659, in fetch
    return await self._execute(query, args, 0, timeout, return_status=False)
  File "/usr/local/lib/python3.11/site-packages/asyncpg/connection.py", line 1858, in _execute
    result, _ = await self.__execute(...)
asyncpg.exceptions.PostgresSyntaxError: syntax error at or near "$1"
  QUERY: SELECT * FROM orders WHERE tenant_id=$1
  DETAIL: params=[] positional_args_count=0
2026-08-18T14:22:05.019Z WARN  [req_id=r-8f42a1] turn=3 failed, retrying (attempt 2/3)
2026-08-18T14:22:07.204Z ERROR [req_id=r-8f42a1] max retries exceeded, aborting agent loop

Step 2 — Rank root-cause hypotheses

Now ask for hypotheses, ranked by likelihood, each with the specific evidence in the trace that supports or weakens it. Forcing the model to score confidence and cite evidence is what cuts down on the confident-but-wrong guesses that plague debugging chats — the multi-bug debugging benchmark on arXiv shows models degrade fast when they can't separate the primary bug from downstream noise.

Using the cleaned trace and context above, generate 3-5 root-cause hypotheses, ranked by likelihood.

For each hypothesis, output:
- **Hypothesis** (one sentence)
- **Likelihood**: High / Medium / Low
- **Evidence for**: exact lines or values from the trace that support it
- **Evidence against**: anything in the trace that would rule it out
- **Cheapest check**: the single command, query, or file to inspect to confirm or kill this hypothesis in under 2 minutes

Rules:
- Do not invent frames or values not present in the trace.
- Prefer hypotheses about our code over library bugs unless the trace clearly implicates a library.
- If the model generating a tool call (LLM-side) is a plausible cause, treat that as a distinct hypothesis from a code-side bug.

Step 3 — Assemble the debug brief

Convert the analysis into a ticket-ready brief. This is the artifact you paste into Linear, Jira, or the incident channel.

Produce the final debug brief as markdown with exactly these sections:

**Title** — imperative, under 90 chars, includes the exception type and the module.

**Summary** — 2-3 sentences a manager can read: what broke, who's affected (if inferable), whether the loop retried.

**Reproduction** — the minimum steps or inputs implied by the trace. Say "unknown from trace alone" for anything you can't ground.

**Top hypothesis** — the highest-ranked one from Step 2, with its cheapest check.

**Alternative hypotheses** — bullet list, one line each.

**Suggested owner** — the module path most likely responsible (e.g., `app/tools/sql.py`), not a person.

**Blast radius** — best guess: single request, single tenant, single tool, or systemic. Mark as "unclear" if the trace doesn't say.

**Next 3 actions** — concrete, ordered, each doable in under 10 minutes.

Keep it under 250 words total. No filler, no restating the trace.

Gotchas

• Truncated traces mislead the model. If your logger clips at 4KB, the innermost app frame may be missing and the model will confidently pick the wrong hypothesis. Paste the full trace or explicitly note truncation.
• Multi-bug traces. When one failure triggers cascading errors, the ranking often surfaces the loudest symptom instead of the root. The arXiv multi-hop debugging work flags this exact failure mode — if the "cheapest check" for the top hypothesis comes back clean, re-run Step 2 with the top hypothesis explicitly excluded.
• LLM-side vs code-side. For agent-loop bugs, the model-generated tool arguments are as likely a culprit as your dispatch code. If Step 2 doesn't produce at least one LLM-side hypothesis for agent traces, prompt it to add one.
• Secrets in logs. Traces often carry tokens, tenant IDs, or PII. Scrub before pasting into any hosted model.
• Don't skip Step 1. Feeding the raw log straight into Step 3 gives you a plausible-sounding brief with hallucinated frames. The extract-then-analyze split is what keeps it grounded.

Time saved

Estimate: 20-30 minutes per incident, based on the manual baseline of reading, grepping, and drafting. The savings shrink when the trace is short and familiar (you'd have written the brief in 5 minutes anyway) and grow when it's a stack you don't own. This is a triage accelerator — not a fix. The debugger session still happens; it just starts with a sharper question.

What we got when we ran it

We executed every prompt above, in order, against the sample input (model: claude-sonnet-4-6). Outputs below are real and unedited apart from length truncation.

Step 1 output

Signature **`PostgresSyntaxError` in `tools/sql.py:run_sql` during parameterized query execution (missing bound parameters passed to `asyncpg.conn.fetch`)** --- ## Cleaned Trace ``` Traceback (most recent call last): # Outermost application/framework frame File "/app/agentkit/loop.py", line 214, in _run_turn result = await self._dispatch_tool(call, ctx) File "/app/agentkit/loop.py", line 341, in _dispatch_tool return await handler(call.arguments, ctx) # Innermost application frame — actual failure originates here File "/app/tools/sql.py", line 88, in run_sql rows = await conn.fetch(query, *params) # Library frame retained because it surfaces the failure detail [asyncpg internals omitted] asyncpg.exceptions.PostgresSyntaxError: syntax error at or near "$1" QUERY: SELECT * FROM orders WHERE tenant_id=$1 DETAIL: params=[] positional_args_count=0 ``` **Root cause:** `run_sql` calls `conn.fetch(query, *params)` but `params` is empty (`[]`). asyncpg interprets the bare `$1` placeholder as a syntax error when no positional arguments are supplied. The query was constructed with a placeholder but the argument-binding step either failed silently or was skipped entirely before dispatch. --- ## Context IDs | Field | Value | |---|---| | **Request ID** | `r-8f42a1` | | **Agent turn** | `3` | | **Model** | `claude-sonnet-4` | | **Tool name (failing)** | `run_sql` | | **Tool name (also registered)** | `search_docs` | | **Tool args at call time** | `{"query": "SELECT * FROM orders WHERE tenant_id=$1"}` | | **Retry state at failure** | Attempt 1 of 3 failed; retried; max retries exceeded at 14:22:07Z | --- **Recommended fix:** In `tools/sql.py`, validate that `len(params) == query.count('$')` (or equivalent placeholder count) before calling `conn.fetch`, and raise a descriptive `ToolInputError` early rather than letting asyncpg

…output truncated at 250 words.

Step 2 output

Root-Cause Hypotheses --- ### Hypothesis 1 — `run_sql` extracts `query` from LLM args but never extracts `params`, defaulting to an empty sequence **Likelihood:** High **Evidence for:** - Tool args logged at call time: `{"query": "SELECT * FROM orders WHERE tenant_id=$1"}` — only the `query` key is present; no `params`, `args`, or `bindings` key exists in the payload - `DETAIL: params=[] positional_args_count=0` confirms zero arguments reached asyncpg - `conn.fetch(query, *params)` at `sql.py:88` — if `params` was extracted as `call.arguments.get("params", [])` the splat silently passes nothing **Evidence against:** - We cannot see the argument-extraction code inside `run_sql` above line 88; it is possible params were extracted but lost through a different path **Cheapest check:** ```bash sed -n '70,90p' /app/tools/sql.py ``` Look for how `params` is populated before line 88. If you see `params = arguments.get("params", [])` with no validation, this hypothesis is confirmed. --- ### Hypothesis 2 — The LLM generated a tool call that omitted the `params` argument entirely (model-side schema non-compliance) **Likelihood:** High **Evidence for:** - Logged tool args `{"query": "SELECT * FROM orders WHERE tenant_id=$1"}` contain a `$1` placeholder but no corresponding parameter value — the model produced a parameterized query string without supplying the bound value, which is exactly the kind of structural error frontier models make when a tool schema is ambiguous or `params` is marked optional - Model is `claude-sonnet-4`; this is a plausible single-turn schema-compliance failure **Evidence against:** - If the tool schema does not define a `params` field at all, the model cannot

…output truncated at 250 words.

Step 3 output

Fix `PostgresSyntaxError` in `tools/sql.py` Caused by Unbound `$1` Parameter

---

Summary run_sql called conn.fetch with a parameterized query but an empty params list, crashing the agent on turn 3. Any request invoking run_sql with a placeholder query and no bound values will hit this. The loop retried all 3 attempts and aborted — all failed identically, confirming a deterministic bug not a transient fault.

---

Reproduction

What generated the specific `tenant_id` value to bind: unknown from trace alone.

---

Top Hypothesis run_sql never extracts a params key from call.arguments, defaulting to []. Cheapest check: sed -n '70,90p' /app/tools/sql.py

---

Alternative Hypotheses

• Model omitted params because the tool JSON schema doesn't require it
• Schema marks params optional/absent, making omission systematic across all calls
• Key-name mismatch between what model sends and what handler reads

---

Suggested Owner /app/tools/sql.py (schema definition co-owned by /app/agentkit/loop.py)

---

Blast Radius Single tool (run_sql), but systemic — any parameterized query through this tool will fail identically regardless of tenant or request.

---

Next 3 Actions