---
id: issue-fixer
type: project
doc: portfolio
url: https://auejin.com/en/projects/issue-fixer/portfolio/
lang: en
---

# Issue-Fixer — A Harness System That Absorbs a Never-Ending Production Bug Queue

**A record of 5 months of continuous production operation — design, incidents, quantified impact**

> A QA engineer files a bug in Jira; it comes back as a review-ready pull request, with no human hands between identifying the target product and environment and validating the fix in a real browser.
> **Ran continuously in production for 5 months** (unattended through nights and weekends) against the frontend monorepo of a B2B SaaS company in the 3D spatial-data domain — 924 issues in, 571 PRs out, a median of 35 minutes from detection to PR.
> This is operating data, not a benchmark. Every number was extracted from the system's own analytics DB (SQLite) and operational documents; estimation assumptions are stated in the appendix.
> Company, product, and issue identifiers have been generalized to protect internal information.

---

## 1. Executive Summary

- **What I built**: A system that **absorbs an unrelenting queue of bug tickets with no human queue attached**. Once QA files a bug in Jira, it carries the ticket to a submitted PR without human hands: issue harvesting → video analysis → browser reproduction → codebase analysis → fix planning (with self-critique) → code changes → real-browser before/after validation → commit / PR / Jira transition / Slack notification (the final merge stays behind a human review gate). It consists of a harness that orchestrates LLM CLI sessions across 22 phases, a reliability layer for failure isolation and runaway prevention, concurrent execution on a git-worktree slot pool, and a meta loop that audits its own performance and ships improvement PRs to itself.
- **Where and how long it was validated**: **5 months of continuous production operation** (launchd always-on services; unattended through nights and weekends) against the frontend monorepo of a B2B SaaS company in the 3D spatial-data domain. Not a benchmark or a demo, but an operating record against a real bug queue taking in more than 6 issues per calendar day — during which three major incidents occurred, all three converted into structures that cannot recur (§7). Long-running operation bought more than a reliability narrative: it produced **the evaluation dataset itself**, since 924 human verdicts (merged/rejected) and their follow-up commits are the ground truth for RCA accuracy and the self-improvement loop.
- **Results**: Over 5 months: 924 unique issues processed, 571 PRs generated (median 35 minutes from detection to PR), a 35.4% merge rate among fully reviewed PRs (82.4% of merges required zero human edits), and 82.9% RCA accuracy. Cumulative merge rate rose from 19.5% at the first audit to 35.4% (+15.9pp).
- **The engineering core**: The hard part was not "prompting the LLM well" — it was **turning an untrustworthy probabilistic execution unit (an LLM session) into a trustworthy system component**. A recurring theme across completion detection, failure classification, verdicts, concurrency, and self-evaluation: demote the LLM's free-form utterances into types and gates.

### How this differs from an agent that fixes code

AI that fixes bugs is common. What sets this system apart is **everything bolted on either side of the fixing** — operating the tracker so it picks its own work, establishing product context before editing, isolating issues so several run at once, refuting its own claim that the bug is fixed, and grading itself so it can improve without a human rewriting its prompts.

| A typical bug-fixing agent | Issue-Fixer | Evidence |
|---|---|---|
| A human picks the issue and spoon-feeds it as a prompt | **It operates the tracker itself** — polls Jira every 5 minutes, selects its own targets, and owns the issue lifecycle: state transitions, clarifying comments, rollback on failure | §3, §5-4 |
| Reads the repo and goes straight to editing code | **It establishes product context before touching code** — parses the QA link to identify the target app and environment, compresses the attached recording into keyframes for VLM analysis, recovers capture conditions (date, floor, view mode), and injects codebase analysis plus prior-PR history | §5-3 |
| One issue at a time (a single checkout) | **It runs them concurrently** — a git-worktree slot pool isolates each issue, 1.1 s reset between jobs, and the "hold" state that caused contention was removed so thoroughly that the schema cannot express it | §5-2 |
| An LLM saying "fixed it" is the completion signal | **The LLM's utterances are never the completion signal** — real-browser before/after refutation, verdicts as typed enums, failures as a 5-way taxonomy, completion via a marker protocol | §5-1, §5-3 |
| Improving it means a human edits the prompts | **It ships its own improvement PRs** — biweekly KPI self-audit (every number computed by deterministic code) → automated roadmap implementation → draft PR → the next cycle statistically tests the effect | §5-5 |
| Validated on benchmarks and demos | **5 months of continuous production operation** — 924 incoming issues absorbed, three major incidents converted into structures that cannot recur | §6, §7 |

---

## 2. Problem — The Intake Never Stops

What this system faced was not "one hard bug" but **an unbroken inflow**. Measured intake was 924 issues over 5 months (~150 days) — more than 6 per calendar day — and peak throughput reached 35 issues in a day (before the collapse in §7, Incident C). A developer queue is not built to absorb that rate.

| Problem | Detail |
|---|---|
| **Intake rate** | 6+ issues arriving every day — the requirement is not the ability to fix one bug well but **the throughput to drain the queue** |
| Repetitive debugging cost | Every bug consumes developer context on the reproduce → diagnose → fix → verify cycle |
| Reproduction cost | Accessing QA environments, watching attached recordings, and matching reproduction conditions takes real time |
| Lead time | Bugs wait days in developer queues — the waiting often exceeds the fixing |
| Knowledge loss | Lessons from past failed fix attempts never reach the next attempt |

The goal was never "replace developers" but to **supply review-quality fix PRs within tens of minutes, with no human queue**. Accordingly, the final gate is always human code review, and the system's success metric is defined not as its own completion rate but as **whether humans actually merged the PR** (and whether the merge needed human edits).

That the intake never stops shaped the entire design. Making throughput the goal makes concurrency mandatory (§5-2); concurrency invites shared-resource contention and cascade failures (§7, Incident B); and in always-on operation with nobody watching overnight, a single defect in failure handling amplifies into a runaway (§7, Incident A). None of these problems exist for "an agent that fixes one bug well."

---

## 3. System Architecture

```mermaid
flowchart TB
    subgraph Jira["Jira"]
        I["Bug issue (AI Review = Required)"]
    end
    subgraph Server["Server (Node.js, single leader)"]
        P["Poller (every 5 min)"]
        Q["Bull Queue (Redis)<br/>main queue concurrency 2 + dedicated browser queue 1"]
        R["Phase Runner<br/>independent LLM CLI session per phase"]
        FT["Failure ledger + circuit breaker (Redis)"]
        WP["Worktree slot pool<br/>isolated working tree per issue"]
    end
    subgraph Phases["Workflow (22 phases, excerpt)"]
        C[COLLECT] --> V[VIDEO_ANALYZE] --> RE[REPRODUCE<br/>Playwright, real browser]
        RE --> A[ANALYZE] --> PL[PLAN<br/>+ self-critique] --> F[FIX] --> VA[VALIDATE<br/>real-browser before/after]
        VA -->|on failure, up to 3x| A
        VA -->|on success| CM[COMMIT → PUBLISH]
    end
    subgraph Meta["Self-improvement meta loop"]
        DB["Analytics DB (SQLite)"] --> BR["Biweekly KPI audit report<br/>(numbers computed by code)"]
        BR --> RA["Roadmap auto-implementer<br/>→ draft PR"]
        RA -.->|after human merge| DB
    end
    I --> P --> Q --> R --> Phases
    R <--> WP
    R <--> FT
    CM --> PR["PR + Jira transition + Slack"]
    R -.events.-> DB
```

Key design decisions:

- **Independent session per phase (context isolation)**: Running the whole pipeline in one session grows the context to hundreds of thousands of tokens and accuracy collapses. Each phase spawns a fresh LLM CLI process, and the previous phase's artifacts (JSON/Markdown files) are the only context the next phase receives. Phase definitions (order, artifacts, conditional execution, whether user-input waits are allowed) live in a single JSON source of truth.
- **File schemas are the interface**: With no conversation history, artifact schemas are the contract between phases. The orchestrator verifies artifact existence after each phase and classifies omissions as retryable.
- **Verdicts demoted to enums**: Every judgment — reproduction (REPRODUCED / NOT_REPRODUCED / WORKS_AS_EXPECTED / BLOCKED), validation (FIXED / NOT_FIXED / REGRESSION / BLOCKED) — arrives as a file-based typed enum consumed by a switch, never as free text, because each verdict demands the opposite follow-up (proceed / retry / auto-close / rollback).
- **Single-executor guarantees**: A Redis leader lease (CAS) keeps the server a single instance, with the lease-renewal loop isolated in a worker thread so main-thread blocking cannot cost leadership. All git/GitHub operations run under a fail-closed bot identity (a dedicated GitHub App) — with no token, they fail loudly instead of falling back to a human's credentials.

---

## 4. Alignment with Industry Paradigms

How the system maps onto the core concepts of 2024–2026 agent engineering. (Representative sources: Anthropic's *Building Effective Agents* and *Effective Context Engineering for AI Agents*, Databricks' *What is an AI Agent Harness?*, the LangChain/Temporal durable-execution docs.)

| Industry paradigm | This system's implementation |
|---|---|
| **Agent harness** (the execution layer that turns a model into actions) | A purpose-built harness bundling the phase orchestrator, tool/artifact contracts, verification loops, guardrails, and observability. §5-1 |
| **Workflow-vs-agent spectrum** | Predictable segments run as coded workflows (deterministic source detection, branch preparation); only judgment segments are LLM phases — "autonomy only where needed" |
| **Context engineering** | Fresh context per phase + curated artifact files + conditional context injection (screenshots, prior-PR history) |
| **Durable execution** | Checkpoint-based resume (STATE file), per-failure-type retry policies, re-entry after pending-input waits |
| **Grounding / real-environment verification** | Playwright real-browser reproduction and validation, video → keyframe → VLM analysis, serving the local tree so the *modified code* is what gets verified |
| **Human-in-the-loop** | Human modes for reproduce/validate (the human only demonstrates; a VLM interprets), a pending-input protocol, draft PR + human merge gate |
| **Guardrails & reliability** | Failure ledger + circuit breaker, an error taxonomy (issue isolation vs. full stop), backpressure (defer-not-fail). §5-4 |
| **Multi-agent concurrency & isolation** | git-worktree slot pool, admission control, critical-sectioning of shared resources. §5-2 |
| **LLM-as-judge & evals** | RCA accuracy (ground truth = human follow-up behavior), LLM classification of rejection reasons (whitelisted enums), statistical testing in the KPI audit reports |
| **Self-improving systems** | The closed loop: report roadmap → automated implementation PR → regression measurement. §5-5 |

---

## 5. Technical Highlights (problem → design → trade-off)

### 5-1. The agent harness: making an LLM process's "done" and "failed" trustworthy

**Problem**: An LLM subprocess has no reliable completion or failure signal. (1) In browser phases, the dev server and Playwright live in the same process group, so the CLI finishes its work but never exits — trusting exit codes turns normal completions into timeouts. (2) "I can't fix this issue" (one issue's problem) and "the credentials were revoked" (the whole system's problem) both surface as exit code 1.

**Design**:
- **Completion via a marker protocol**: Prompts are instructed to emit a `PHASE_COMPLETE` marker, and the orchestrator parses the stream and checks for the marker **only in text the AI itself authored** — a marker literal appearing in a tool result (e.g., the agent grepping documentation) can never fire it. After observing the marker, it grants a grace period and then reclaims the whole process group with SIGTERM→SIGKILL escalation; abnormal exits on this path count as success (artifact validation is separate).
- **Failure via a 5-way taxonomy**: Retryable (missing output, timeout) / issue-permanent (→ rollback) / needs-user-input (→ a structured Jira question) / harness-fatal (→ global breaker) / shutdown, separated as types. Fatal detection matches output-text signatures (401s, exhausted credits) rather than exit codes, and is **never promoted to issue-level rollback** — promotion would let the rollback restore the re-entry condition and produce an infinite loop (§7, Incident A).
- **Model and budgets as a set**: Per-phase timeout budgets are calibrated to a specific model's turn/tool-call volume. After a CLI auto-update silently swapped the model and throughput collapsed (§7, Incident C), the model version is pinned in code and force-injected at every spawn site, and budget changes are made only from p90s of per-phase metrics (a JSONL sidecar). All timing constants live in one module whose inequalities (max phase budget < drain ceiling < queue lock < leader TTL) are asserted at boot — a bad combination of constants cannot even start the server.

**Trade-off**: The marker protocol depends on prompt discipline — teardown contracts are spelled out in the prompts, and snapshot tests catch prompt regressions. We accept "if the artifact exists, don't interrogate the process's exit" — but the reverse (exit 0 with no artifact) is always a failure, closing the false-success path.

### 5-2. Concurrency: a worktree slot pool, migrated measure-first

**Problem**: Every issue time-shared a single monorepo checkout. An exclusive lock meant raising the concurrency setting only made the second worker requeue in vain (the bottleneck was an architectural premise, not a config value), and as throughput rose, contention incidents — stash hijacks, restore failures, hold cascades — arrived in steps: of 94 observed days, contention occurred on just 6, all during throughput peaks. "No problems right now" was structurally incapable of proving safety.

**Design** (staged, with a quantitative gate at each stage):
- **Stage 1 — Measure**: Exhaustively enumerated ~30 places across 5 layers where the "there is only one checkout" premise had soaked in, tagging each for disposal (retire / replace / adapt / keep). Pinned the worst-day contention peak (12,123 deferred requeues in one day, an incident day) and per-phase p50/p90 durations from logs — the comparison baseline for judging the migration.
- **Stage 2 — Isolate**: A slot = a git worktree (objects shared) + submodules initialized with reference sharing + a copy-on-write (APFS clonefile) replica of the 3 GB node_modules. Measured: 48.2 s to provision a new slot (once per slot), **1.1 s to reset between jobs** — an order of magnitude better than the previous branch-preparation p50 of 100 s. Those measurements decided the architecture: a warm pool, not disposable clones. Concurrency stayed at 1 so slot-lifecycle bugs could be caught without concurrency noise.
- **Stage 3 — Open**: After a 12-hour soak (zero contention events), concurrency went 1→2. Effective concurrency is computed by a single function, `min(configured, pool size)`, wired together with a same-issue double-execution guard and free-slot admission control. Further expansion (2→N) is conditional on passing the baseline document's quantitative gate (p90 budget headroom).
- **Structural removal of the "hold" concept**: The old design's "quarantine a corrupted tree until a human looks" let one hold burn the entire waiting queue's failure budget (§7, Incident B). In the pool, an unresettable slot is discarded and re-provisioned (~68 s = 20 s teardown + 48 s fresh provisioning) — the lease schema has no hold field, so that incident path is unrepresentable.
- **Controlling the residual shared surfaces**: Worktree isolation is not all-or-nothing. The shared git object DB's auto-gc contention (disabled; manual idle-time gc only; no pruning that would break alternates), per-workspace build-daemon cleanup (conservative matching — better to miss a kill than kill the wrong process), self-serialization of code paths that bypass slots, and critical-sectioning of the single-port dev server (a mutex around serve-start → browser work → serve-stop). Most of these were defects only under concurrency — the queue used to serialize them for free — so they were found by **pre-emptive audits, not incidents**.

**Trade-off**: Verifiability over speed — no stage opened before the previous one was validated. The browser segment remains serial (single-port dev server), but parallelizing the non-browser segments captured most of the win first; the temporary serialization device carries built-in wait-time instrumentation that will supply the evidence for its own removal (per-slot dev servers).

### 5-3. Grounding: proving "the fix actually works" with a real browser and a VLM

**Problem**: There is a wide gap between claiming the code was fixed and the bug being fixed. The target apps are WebGL-canvas-centric, so DOM assertions are often impossible — and validating against a staging URL means judging FIXED while looking at pre-fix code: false success at the source.

**Design**:
- **Serve the local tree**: Parse the QA link to identify the target app/environment, then serve the working tree **containing the fix** (in pool mode, that issue's slot) locally to reproduce the same screen. Prompts forbid falling back to staging URLs.
- **An experiment-record schema for reproduce/validate**: Reproduction and validation artifacts share a conditions–variable–response schema, and verdicts are forced into 4-value enums. On validation failure, the failed fix is archived, the plan demoted, and the next attempt starts from a clean base (up to 3 iterations) — with the diff between reproduction and validation logs injected as context for the next try.
- **Video → keyframes → VLM**: Attached reproduction recordings are compressed into an evidence frame set by merging scene detection with user-input timestamps. Observing that the dominant reproduction-failure cause is not frame interpretation but "conditions at recording time vs. the reproduction environment," capture-context extraction (date, floor, view mode) was promoted to a first-class artifact.
- **Minimal-intervention HITL**: For bugs the AI cannot reproduce, a human demonstrates once in a browser — the page is instrumented so clicks/keys stream as ground truth, a VLM does interpretation and verdicts, and the artifact schema is identical to AI mode so it rejoins the pipeline downstream with zero changes. "Demonstrated but no verdict" is blocked from the commit path by a sentinel value plus an orchestrator guard.

**Trade-off**: Enum coercion discards nuance (compensated by a reason text field; unknown states fail conservatively). Reproduction playbooks are reused across iterations to save browser time, at the documented cost of cases where the fix changes the reproduction conditions themselves.

### 5-4. Reliability: encode blast radius in types; keep termination conditions outside the loop

**Problem**: This system's rollback resets the Jira state to exactly the poller's selection condition — **rollback is an operation that guarantees re-entry**. Any slip in failure handling becomes the automation-specific disaster class: infinite loops and cascade failures (real incidents in §7).

**Design**:
- **A durable failure ledger**: Retry counters live in Redis — not in the cache directory that rollback deletes (3 attempts per issue). Counters increment both Redis and in-memory and take the max (no failure lost across a Redis outage window); increment and TTL are wrapped in a transaction (no orphaned counters without TTL). If reading the ledger itself fails, the system closes the issue as terminal — "no grounds to declare a retry" — a fail-safe built on the premise that when the harness is sick, Redis is often sick too.
- **A global circuit breaker**: Five consecutive failures, or a single harness-fatal error, stops polling and latches with no TTL. Auto-resume with the cause unresolved is forbidden; the only release is a human-driven API path.
- **Backpressure — occupancy is not failure**: Treating shared-resource occupancy as failure locks issues into an error state that humans must revive one by one. Instead, deferred requeue — with **failure budgets separated by wait cause**: if normal contention and hold-wait share one budget, a single hold burns every waiter's budget (the arithmetic of §7, Incident B). Requeue success is only reported after confirming the replacement job exists — otherwise an issue can silently vanish from the queue.
- **Turning emergent failure into regression tests**: The runaway incident was not one module's bug but a complete cycle formed by module connections — every unit test passed while the system ran away. So the seams (spawned argv ↔ the real parser; config JQL ↔ terminal state values) and the **finite convergence of the whole cycle** (actually driving poller→worker→rollback and asserting write counts stay bounded by constants) became integration-test assets. The incident's actual log line is enshrined as a test case for the fatal path.

**Trade-off**: Manual-only breaker release is an accepted operational burden, as is the hard cap of 3 on ordinary failures (an issue terminated by a transient outage needs a human to clear its ledger) — an explicit choice that "stopping and calling a human is always cheaper than an infinite loop."

### 5-5. The self-improvement loop: code computes the numbers; the LLM only writes prose

**Problem**: An LLM reporting on its own performance games the metrics — inventing p-values, quietly shifting KPI denominators, repackaging existing features as "new." And an agent that edits its own code has structural incentives to self-grade its limitations as low-severity and to declare problems resolved without fixing them.

**Design**:
- **Ground truth for output quality is human follow-up behavior**: When a PR is merged or closed, RCA grade (CORRECT / PARTIAL / INCORRECT) is computed from the overlap between the AI's changed-file set and humans' follow-up commits. The initial formula (denominator = human files; recall) was distorted — one human lint fix could crater the grade — so it was redefined as precision (denominator = AI files) and the entire historical dataset re-scored retroactively. Unadjudicable cases are not force-scored; they go to an INDETERMINATE bucket — and the report is made to call out the resulting "100% via shrunken denominator" illusion itself.
- **The biweekly audit report**: Generated section-by-section in independent LLM sessions (the same pattern as the harness), but **LLM arithmetic is banned outright**. Only values produced by a pure-JS statistics module (Wilson CIs, Fisher's exact, BH-FDR, beta-binomial posteriors, seeded bootstrap) may be quoted, and every figure carries a hash of the KPI formula definition. KPI-gaming phrasing (denominator exclusion, formula redefinition, repackaging) is blocked by deterministic regex validators, and effect claims must cite required sample sizes — with underpowered proposals demoted to observation-only. One cycle's report actually shipped with all top-3 proposals marked "verdict deferred," and a report whose sections failed to generate remains unfabricated, publication blocked, with failure placeholders intact.
- **The closed loop — the report fixes the system**: The report's roadmap is parsed into per-initiative scope → implement → test → commit stages, ending in a draft PR. Self-leniency is blocked by separation of powers: limitation severity can only be assigned by hard-coded rules, a "resolved" claim must survive a git check that the resolving commit actually touched the limitation's files, and gate passage is decided by a script's exit code. The final gate is a draft PR plus a human merge.
- **Regression measurement built into the report order**: Each report first adjudicates "did the last proposals actually work" — implementation status via commit matching, causal effect via pre/post splits with statistical tests, verdict labels as code's exclusive property — and the outcome feeds the next roadmap as bonuses/penalties. A complete lap (report proposal → bot implementation → human merge → next report citing the effect) is preserved in the repository history.

**Trade-off**: Verdict conservatism (all small samples deferred) delays recognition of real improvements by cycles — compensated by declaring "detectable in N cycles" on the roadmap, and by a meta rule that force-promotes sample scarcity itself to a top initiative.

---

## 6. Quantitative Impact

### 6-1. System metrics (5 months, aggregated directly from the analytics DB)

The point is not any single figure but **the balance of intake and absorption**: a queue arriving at 6+ per day was taken in with no human queue, 61.8% of it returned as review-ready PRs, and the remainder filtered or terminated by automated triage.

| Metric | Value | Notes |
|---|---|---|
| Intake pressure | 924 over ~150 days | 6+ per calendar day sustained; peak throughput 35 issues/day |
| Unique issues processed | 924 | 7,839 workflow runs — includes retries and the 5,526 runaway runs of §7 Incident A |
| Auto-filtered | 176 | Non-bugs / duplicates / out of scope — automated triage |
| PRs generated | 571 (61.8%) | Detection → PR: median 35 min, p90 59 min |
| Merged | 159 of 449 fully reviewed (35.4%) | Denominator = 159 merged + 290 rejected. Excludes 122 still-open PRs — 27.8% against all 571 generated |
| Clean-merge share | 82.4% (131/159) | Human-intervention rate 17.6% |
| RCA accuracy (weighted) | 82.9% | Over 237 adjudicable cases¹ |
| Rejection breakdown | AI failures 128 / external causes 107 / unclassified 55 | AI failures: fix-caused bugs 57, wrong approach 55, wrong target 16 |

> ¹ Weighting: (CORRECT×1.0 + PARTIAL×file-overlap ratio) ÷ adjudicable cases. 237 adjudicable = 449 reviewed − 212 lacking a comparison object (no human follow-up commit or replacement PR). Clean merges count as CORRECT by definition, so this metric is not fully independent of merge success — it measures "how closely the AI's file targeting matched human judgment."

Merge-rate trajectory: 19.5% at the first audit report → 35.4% cumulative (+15.9pp). The comparison is approximate (30-day-window vs. cumulative formulas), and the path was not monotonic (a mid-period cycle at 38.5% → a dip during the incident window → recovery). The self-improvement loop shipped three improvements in this period (fix side-effect self-review, mandatory plan alternatives, paired-function contract injection), but causal attribution defers to the per-cycle statistical verdicts (§5-5) — distinguishing correlation from causation is this system's own reporting principle.

### 6-2. Time/cost model (all assumptions stated)

**Assumptions**: (A1) developer time per frontend bug: 2–4 h (mid 3 h); (A2) human review cost per AI PR: 0.25–0.5 h (mid 0.375 h), charged on all 449 reviewed PRs including rejects; (A3) the 28 merged-with-edits PRs credited at 50%; (A4) fully-loaded engineering cost $30–45/h (mid ≈ $37.5; based on mid-level frontend salaries in the operating market, converted from KRW).

| Scenario | Saved | Review cost | Net (5 months) | Dollar value |
|---|---|---|---|---|
| Conservative | 290 h | −225 h | **65 h** | ≈ $2,000 |
| Mid | 435 h | −168 h | **267 h** | ≈ $10,000 |
| Optimistic | 580 h | −112 h | **468 h** | ≈ $21,000 |

Benefits excluded from the model: lead time (days in a human queue → 35 minutes to PR), eliminated context switching, automated triage of 924 issues, unattended nights-and-weekends operation. Costs excluded: LLM API and infrastructure (no data held — honestly left blank).

> Reproducibility: every DB figure is reproduced by a single query script; the script and a per-figure query mapping are maintained as an internal appendix. No number appears in this document that cannot be queried.

---

## 7. Incidents and Structural Fixes — Three Case Studies

A production automation system's competence shows not in having no incidents, but in **what structure each incident is converted into**. All three are real; each became a core design principle.

What the three share is that **no short demo could ever have surfaced them**. The runaway loop requires running long enough for an auth token to be revoked; the shared-resource cascade requires a queue that has genuinely backed up (contention occurred on just 6 of 94 observed days, all during throughput peaks); the performance collapse requires enough elapsed time for a dependency to auto-update. Long-running production operation is not this system's bragging point — it is **the reason these designs exist**.

### Incident A — The runaway loop (July 2026)

- **Symptom**: A revoked auth token (401) failed the collection phase → rollback → rollback restored the issue to the poller's selection condition → re-entry. 5,534 rollbacks and 16,599 Jira transitions in two days (the analytics DB independently shows 5,526 runs on those two days, cross-validating the incident record).
- **Essence**: Every module was individually correct. "On failure, reset to initial state" (rollback) composed with "pick up whatever is Required" (poller) into a cycle with no fixed point — and the only candidate termination condition (the retry counter) lived **inside the directory the loop itself deletes**.
- **Structural fix**: "Termination conditions must live where the loop cannot erase them" — a Redis failure ledger + a global circuit breaker + the error taxonomy (global faults may never be promoted to issue-level isolation). Regression protection is an integration test that drives the full cycle to finite convergence, with the incident's actual log line enshrined as a test case.

### Incident B — Shared-resource cascade failure (July 2026)

- **Symptom**: One corrupted working tree's "hold" occupied the exclusive lock for 2 hours. Because hold-waits and normal contention shared one failure budget (both 7,200 s — arithmetic under which a waiter structurally cannot win), ~102 queued issues burned their budgets and **97 were condemned to an error state**. 12,123 deferred requeues in one day.
- **Essence**: A failure-isolation device (the hold) inverted into a failure amplifier (simultaneous failure of every waiter) — an interaction invisible to any single-device verification.
- **Structural fix**: Two-stage response — immediately: separate budgets per wait cause plus a bulk-recovery script (with the state-restoration order itself codified). Structurally: replace single-checkout time-sharing with the worktree slot pool, **removing the hold state from the design space entirely** (§5-2). The migration ran on staged quantitative gates: baseline → isolation → soak → open.

### Incident C — Silent performance collapse via dependency auto-update (July 2026)

- **Symptom**: An LLM CLI auto-update swapped the underlying model. The new model spent ~1.6× the turns/tool calls on the same work, pushing the planning phase past its timeout budget 50% of the time; through the amplification chain (budget overrun → timeout retries → rollback → full re-run), **throughput collapsed from 35 issues/day to 6**.
- **Essence**: "Model" and "phase budgets" were implicitly coupled — and that coupling was expressed nowhere in code.
- **Structural fix**: Pin the model version in code and force-inject it at every spawn site; recalibrate budgets only from measured p90s of phase metrics (codifying even the observation bias — "success-only records make timeout-truncated distributions look shorter than reality"); enshrine "model pin ↔ budgets are one set" in the safety rules. All timing constants moved to a single source-of-truth module whose inequalities are asserted at boot.

---

## 8. Limitations and Roadmap

**Honest limitations**:
- 64.6% of fully reviewed PRs are rejected. The AI-failure share (128) is the improvement target; external causes (107 — superseded, not-a-bug, unreproducible) are an intake-filtering problem.
- Only 53% of cases are RCA-adjudicable (no human comparison PR exists for the rest) — the 82.9% accuracy is always quoted against the adjudicable set.
- Concentrated on a single project (one monorepo) — cross-domain generalization is unproven.
- The self-reporting pipeline is imperfect too — the report for the cycle right after the major incidents failed section generation, and was left in its failed state rather than fabricated.
- The cost model is assumption-dependent and therefore presented only as ranges.

**Roadmap**: Parallelizing the browser segment (per-slot dev servers — pre-specified; launch priority being decided from exhaustive wait-time telemetry), expanding concurrency 2→N (conditional on the baseline gate), and continued reduction of AI-failure rejections (the standing target of the self-improvement loop).

---

## Appendix — Tech Stack

| Layer | Technology |
|---|---|
| Orchestration | Node.js, Bull Queue (Redis), custom phase runner (LLM CLI subprocess management) |
| LLM | Claude CLI (model version pinned), VLM-based video/screenshot analysis |
| Browser automation | Playwright (headless/headful, CDP event instrumentation, video capture) |
| Isolation / concurrency | git worktree, APFS clonefile (CoW), file leases + Redis leader lease (worker-thread isolated) |
| Observability / evals | SQLite (WAL; append-only events + denormalized views), custom statistics module (Wilson / Fisher / BH-FDR / bootstrap), automated biweekly audit reports |
| Integrations | Jira REST (polling, transitions, comments), GitHub App (batched GraphQL, fail-closed bot identity), Slack |
| Infra | macOS launchd always-on services, auto-deploy (main polling + graceful-drain restart) |

> Figures and cases in this document are drawn from production data; company-identifying information has been generalized. Architecture and code-level evidence available on request in interviews.