← Beacon Chat한국어
Full portfolio

An LLM chatbot that actively raises the confidence of its own citations — through conversation

Every number is graded before it is spoken. Counter-questions narrow the scope until "at least N" becomes "exactly N," and a sample of citations becomes the complete set.

Project codename: Beacon · Document version: 2026-08 Company, product, personal names, and internal hosts are generalized. Code symbols and figures are real.

Lead

What it is An LLM chat server (TypeScript) and a panel UI (Angular) for a reality-capture construction B2B SaaS — users ask natural-language questions about 3D-captured job sites. The model narrates only; claims about counts, absence, and failure are adjudicated by code.

How far it has been verified Three internal demo tiers (CLI → browser → full UI) run end-to-end against real dev and qa deployments with real accounts and real tokens, and passed. The final tier is feature-complete and internally verified; current work is UI/UX polish. There is no production end-user traffic yet — no figure in this document claims user scale.

Scale Server 25,283 LOC · tests 54,227 LOC (2.14 : 1, ~2,817 cases) · panel UI 10,284 LOC · 16 browser e2e specs · 116 state-transition rows (illegal transitions included) · 103 registered contract boundaries · 18 releases in 31 days.

My role Design and implementation of the server, the panel, and the verification harness. Drove two API contract changes in an external team's system I had no write access to, using measurement reports as the lever.

1. Executive Summary

On a construction site, "how many safety findings were raised on level 3 this week?" is not small talk. That number stops work and settles subcontractor invoices. A chatbot that states a wrong number with confidence is worse than none.

And the most common way an LLM chatbot gets a number wrong is not hallucination. It is a chain in which nobody lied and the chain lied anyway. Measured, in four steps:

  1. The search backend puts the true match total in its response.
  2. The intermediate MCP server drops that field while parsing, then labels the post-truncation count Total:.
  3. Our chat server silently rewrites the model's requested limit down to 10.
  4. The model sees 10 items and says "I found 10."

The user asked for 12, saw 10, and believes that is all of them. No layer acted in bad faith.

This project's answer is not to eliminate truncation but to promote it into an honest contract called a "page" — and then to go one step further: use the conversation to turn that page into the whole population.

  • Every number that is a candidate for utterance is graded before the model speaks — EXACT / LOWER_BOUND / SHOWN / UNKNOWN. The grade dictates the permitted sentence form, and UNKNOWN forbids stating a number at all.
  • Numbers that cannot be counted are given honestly as "at least N" — but the system does not stop there. The counter-question is the operator that converts a lower bound into a certainty: once a narrowed scope fits in a single page with nothing beyond it, the displayed cards go from sample to complete set and SHOWN is promoted to EXACT.
  • The same principle governs claims of absence, verdicts of failure, and permission denial. Each kind of claim has its own standing requirement.

Comparison

# Typical (ChatGPT clone · RAG tutorial · MCP tool-call demo) Beacon Ref
1 The model counts the array it received and says "I found N" Candidate numbers are graded before the model speaks, and the grade dictates the permitted phrasing. UNKNOWN forbids stating a number at all §4.1
2 Asks a clarifying question when ambiguous (a prompt rule) The counter-question is a lower-bound → certainty promotion operator. Once the narrowed scope fits one page, a sample citation set becomes the complete one and the grade rises §4.2
3 Empty tool result → "I couldn't find anything" Asserting absence requires standing. Only a payload the server accounted for as its own search result may ground "there is none." Without standing: unknown, and silence §4.3
4 Tool failure is whatever isError says Detects failures disguised as HTTP 200 by the absence of a result envelope. Error text is never classified — an upstream rewording cannot break it §4.4
5 Permissions enforced by system-prompt instruction Blocked before the call goes out; the data never arrives, so leakage is structurally impossible. The planner is deliberately blind to licensing vocabulary (vocabulary gate). 14 dedicated test files / 131 cases §4.6
6 A happy-path screenshot is the whole demo verification 116 transition rows including illegal transitions + 17 CLI reproductions + 16 browser e2e specs. The stub server the e2e suite stands on is itself contract-tested §4.7
7 Built and shipped solo Changed an external team's API contract twice by measurement report (fetch cap 30→200, cursor pagination · reachable set 16→38). Delivery deviations sealed with our own invariants §4.9

2. Problem definition

2.1 What the domain demands

Data in a reality-capture construction SaaS has three properties.

  • Large and heavily duplicated. Thousands of panoramas per project; the same spot is photographed repeatedly.
  • Semantic and structured queries are mixed. "Missing hard hat" is vector search; "how many captures on level 3 in April" is SQL.
  • The cost of a wrong answer is asymmetric. Claiming something exists when it doesn't is worse than missing it — and worst of all is wrongly saying "that's all of them." In a safety review, "no findings" is a sentence someone signs their name under.

2.2 Why that is hard

Three things must hold simultaneously.

  1. It must be honest — say "I don't know" when it doesn't, and never present a sample as the population.
  2. It must still be useful — a bot that only ever answers "I can't be sure" gets abandoned.
  3. And that must be enforced in code — writing "answer honestly" into a prompt collapses the moment the model version changes.

(1) and (2) are in tension. The entire design is a way of resolving that tension: fix honesty as a grading system, then build a path inside the conversation for raising the grade.

2.3 Constraints

  • The chat server owns no search index. Search is only possible through an MCP server owned by another team, and I have no write access to it.
  • There are two host applications (a 3D site viewer and an admin console) with completely different rendering capabilities.
  • There are four licensing tiers, and data from a tier the user does not hold must never appear in a citation, a thumbnail, or a sentence.

3. Architecture

3.1 Overall

flowchart LR
    subgraph HOST["Two host apps"]
        SV["3D site viewer"]
        PAC["Admin console"]
    end

    subgraph PANEL["Panel UI (publishable Angular lib)"]
        AD["HostAdapter · 9 methods<br/>viewer-state subscribe ↔ citation dispatch"]
        CARD["Citation cards · 11 panel states"]
    end

    subgraph SRV["Chat server (TypeScript)"]
        PEP["Capability PEP<br/>blocks BEFORE the tool call"]
        LOOP["Orchestrator loop<br/>planner ↔ responder"]
        JUDGE["Adjudication layer<br/>count-confidence · emptiness<br/>in-band-failure · subset-gate"]
        ANCH["Narration anchors<br/>code tells the model what it learned"]
    end

    subgraph EXT["External (no write access)"]
        MCP["Algorithm team's MCP server<br/>photo search · data query · 4 more"]
        LLM["Managed LLM"]
    end

    SV --> AD
    PAC --> AD
    AD -->|"view context + auth + capability hint"| LOOP
    LOOP --> PEP
    PEP -->|allow| MCP
    PEP -->|"short-circuit<br/>(data never arrives)"| ANCH
    MCP --> JUDGE
    JUDGE --> ANCH
    ANCH --> LLM
    LLM -->|"narration only"| LOOP
    JUDGE -->|"structured channel<br/>citations · counts"| CARD
    CARD -->|"click = app state transition"| SV

    classDef judge fill:#0ea5e9,color:white
    classDef gate fill:#ef4444,color:white
    class JUDGE,ANCH judge
    class PEP gate

The point is that the model sits on only one of the two arrows. Data flows adjudication layer → structured channel → cards; the model narrates alongside it.

3.2 Two planes

flowchart LR
    subgraph DATA["Data plane — deterministic"]
        R["Tool result"] --> ACC["Structured accounting<br/>returned · total · has_more · cursor"]
        ACC --> FRAME["references frame<br/>numbers only, no sentences"]
        FRAME --> UI["Cards + honest count line"]
    end
    subgraph NARR["Narration plane — the model"]
        ACC -.->|"grade verdict<br/>as an instruction"| A["narration anchor"]
        A --> M["Model: narrator<br/>numbers only from accounting"]
        M --> P["Prose"]
    end
    classDef good fill:#10b981,color:white
    class ACC,A good

Values are decided deterministically in structured fields; wording is produced dynamically by the model. The server never writes the sentence — it injects an instruction. Without this split, honesty rules pile up inside the prompt, and prompt rules have no way of being verified.

3.3 The grade ladder

flowchart LR
    U["UNKNOWN<br/>no number may be spoken"] -->|"accounting arrives"| S["SHOWN<br/>'these N…'<br/>subject separation required"]
    S -->|"narrow scope →<br/>one page + nothing beyond"| E["EXACT<br/>'exactly N'<br/>assertion permitted"]
    L["LOWER_BOUND<br/>'at least N'<br/>assertion forbidden"] -->|"narrow scope"| E
    S -.->|"population floor exists"| L
    classDef e fill:#10b981,color:white
    classDef u fill:#6b7280,color:white
    class E e
    class U u

The rightward arrows are counter-questions. This ladder is the spine of the entire document.

4. Technical highlights

Ordered by novelty × industry impact. Each is problem → design → trade-off.

4.1 Four grades of standing to state a number — the model never grades one itself

Problem

The only basis most LLM chatbots have for stating a count is the length of the array they received. Nothing decides whether that array is the whole population or the first page. And the information needed for that decision already existed — the upstream search engine sent the total, and an intermediate layer dropped it while parsing.

Worse, upstream sometimes sends a false completion signal. The data-query tool injects a row cap into the SQL it generates; when results saturate the cap, total pins to the cap value and page exhaustion reports has_more: false. Measured on a project with 7,961 panoramas: total = 3000, no more. Upstream was confidently wrong.

Design

Every candidate number is classified into four grades from structured fields alone, before the model speaks. The module header (src/mcp/count-confidence.ts) states the intent itself:

"every number that is a candidate for utterance is classified deterministically from structured fields BEFORE the model speaks, and the classification decides which phrasing the narration anchor permits. The model never grades a number itself."

Grade Condition Permitted phrasing
EXACT total below the sentinel, single-value aggregate "may be asserted ('exactly N')"
LOWER_BOUND vector search total_lower_bound, sentinel-saturated total "'at least N' only, never asserted as exact"
SHOWN the returned/displayed subset "Always spoken with the subject-separation rule ('these N'), never as a population claim"
UNKNOWN no structured accounting "No count may be uttered at all."

The false completion signal is overridden. applyLimitSentinel detects saturation and ignores upstream's has_more: false, forcing hasMore: true. The reason is in the code — "a saturated page that was exhausted reports has_more: false upstream … but rows beyond the cap may exist."

And the proof that this reclassification cannot produce a falsehood in either direction is in the comment:

"even when the true total is exactly the sentinel value, 'at least N' still holds (>= includes =)."

The spine is the subject-separation invariant. Model utterances split into two clauses that never share a subject.

  • Clause ① — subject is the whole population: "Of at least 189 matches across the project…"
  • Clause ② — subject is the returned subset: "these 38 were registered on level 3 and…"

"The results are on level 3" is forbidden (it implies the population). Only "these 38 are on level 3" is allowed.

This discipline is not confined to search results. It also governs what the user is currently looking at: the host trims its annotation list to a wire cap before sending, so that length is treated as "a SHOWN-grade number, never a total," and reaching the cap triggers a disclosure that the list may be truncated. That is the strongest evidence the principle is a rule and not a one-off patch.

The sentinel is not our constant. The code declares it — "an external contract, not our constant" — makes it env-overridable, and registers it as a contract boundary watched by a periodic sweep. That guards exactly the failure mode where upstream raises its cap and our sentinel begins quietly manufacturing false EXACTs.

Trade-off

  • More things become unsayable. UNKNOWN bans the number entirely. Against an older upstream deployment, every response degrades to UNKNOWN and count narration disappears. This leans deliberately toward saying nothing rather than saying something false — and §4.2 is what buys the loss back.
  • It couples us to an upstream contract. The sentinel mirrors an upstream prompt constant. No hard-coding, boundary registration, and periodic sweeps offset the drift risk, but the coupling itself remains.
  • The model can't summarize freely. The subject-separation invariant makes sentences somewhat wordier. That is the naturalness paid for accuracy.

4.2 The counter-question as a lower-bound → certainty operator

Problem

With §4.1 alone the bot becomes honest but useless. Answering "at least 189" every time to "how many safety findings are there?" gives the user honesty and no answer.

And this is a matter of principle, not laziness. In semantic (vector) search an exact count is unobtainable — where you cut the similarity threshold is the count, and that boundary differs per query.

Design

The product axiom:

We would rather give limited information than false information. And when all we can give is a limit, we provide a path to a number that is certain.

Two paths.

① Automatic aggregate promotion. For "how many …?" questions, the system does not count the items in a list result. It steers the planner to re-ask the data-query tool for a single-value aggregation (COUNT/SUM/AVG/MIN/MAX). Single-value aggregations are exempt from the list row cap, so the result is EXACT. That the aggregate row is uncitable is irrelevant — a number needs no citation card.

② Narrowing counter-question. On axes where aggregation cannot reach (semantic search properties), the purpose of the counter-question is redefined: shrink the population until it fits inside a fully reachable range. From the rule text:

"once a narrowed scope fits in one page with nothing more beyond it, the shown count IS exact for that scope, and you may say so."

The moment a narrowed scope returns one page with has_more = false, the cards on screen go from sample to complete set, and SHOWN is promoted to EXACT. The counter-question is not UX decoration — it is a grade-promotion operator.

Even the counter-question is bound by the discipline. The rule states "never promise counts or listings you have not retrieved." Suggestions are also grounded only in filter axes the tools actually accept (date range, scope, dimensions expressible in SQL). Proposing something unexecutable is forbidden.

The generalized principle lives in the same place:

"When a request cannot be satisfied deterministically … do not refuse outright and do not silently pick one interpretation. Ask a brief counter-question that makes the outcome deterministic. If there is exactly one way to resolve the request, just ask; if there are several, offer the options and recommend one."

The canonical case is partial entitlement: neither a blanket refusal nor a silent partial run, but "here is the feasible scope — shall I proceed?"

Measured

In two deployment probe rounds, a portfolio query produced an option-offering counter-question, identical in shape across both rounds: "I need to clarify your request … 1. Analyze your current/primary project … 2. List all your accessible projects first … Which approach would you prefer?"

Trade-off

  • More round trips. Some questions no longer resolve in one turn. What comes back instead is an answer with a higher grade.
  • Model compliance cannot be unit-tested. Tests assert only that the rule string was injected into the system prompt; actual obedience is confirmed only by deployment observation (2/2 passing). This limit cannot be removed — layering a regex judge over free text is a worse solution and is banned (I once did exactly that, misjudged a passing round as a failure, and had to correct it by reading the text).
  • Overused counter-questions are annoying. The rule pins "if you can answer deterministically, just answer" as taking precedence.

4.3 Asserting absence requires standing

Problem

"I couldn't find anything" is the easiest thing a chatbot says and the most dangerous. In a safety context, "no such findings" is a sentence someone signs.

Yet the two signals normally used to decide it are both wrong.

"0 citations ≠ 0 results." An empty citation array means not citable, not not present — tables with no deep-link target, SQL that didn't select an id, aggregate rows. The module's own docstring says the empty return happens "far more often than it returns refs" and is "the designed behaviour, not a failure." Reading absence there means telling a user who is looking at results that there are none.

"0 hits ≠ 0 results." Payloads that were never a search also count zero. Four measured shapes: a tool-level error, a permission short-circuit synthetic, a plain-prose "select a project first" reply, and a degraded empty object.

Design

readResultEmptiness(toolName, rawResult) → 'empty' | 'non_empty' | 'unknown'. The order of checks is the argument.

  1. Not one of the tools with a result contract → unknown
  2. hits/rows > 0 → non_empty ("a result with findings in it is not empty whatever else the payload is missing")
  3. Tool-level error → unknown ("A tool-level failure is not a statement about how much data exists.")
  4. Did the server account for this payload as its own search result? If yes → empty; otherwise → unknown

Step 4 is the crux. A count is sufficient grounds for non_empty only; empty may be claimed only where accounting exists.

unknown is not a computation failure. It is an actual answer.

And the right answer differs by cause. The screen looks the same; the recovery path does not.

Cause Discriminator Correct narration
Permission denial blocked before dispatch "This requires the X subscription" (structured card)
Auth failure HTTP status guard "Authentication failed"
Zero after narrowing our own record that we sent a scope "Not among the previous results. Search the whole project?"
Zero from the start no scope present "The project has no such data"

Active recovery is implemented as releasing the narrowing, not as re-running. The model cannot control scope — dispatch builds it from turn state and spreads it last, overwriting whatever the model put under the same key. Even if the model explicitly says "search everything," the outgoing call is byte-identical and still narrowed. So recovery releases narrowing at end of turn, and the instruction promises only what is achievable: "narrowing has already been released and the next search covers everything. Do not re-run inside this turn."

Trade-off

  • Against an older upstream, everything goes silent. Deployments that don't attach accounting metadata degrade every verdict to unknown and absence narration disappears. That is the intended safe direction.
  • It depends on upstream cooperation. This verdict rests on upstream accounting for its own results. Obtaining that contract is §4.9.
  • The model may still hedge on its own. The code injects instructions; it does not censor model output. Censoring would be a free-text judgment and would reproduce §4.2's limitation.

4.4 Failures disguised as HTTP 200 — detected by envelope absence

Problem

The MCP spec places tool-level failure inside the result rather than in a JSON-RPC error. Upstream goes one step further: when its upstream is down it relays that failure as an ordinary text block and leaves isError false.

Measured (4 different questions × 2 facilities, all identical): Search API error: 502 Server Error: Bad Gateway …, isError: false, no _meta on the block.

"Nothing in isError distinguishes that from an answer, so the loop handed it to the model as data and the turn answered with no citations and no basis."

The model took an error string as data and composed an answer, and no failure signal reached the user.

There was a prior case in the same class. A 401 that carried no JSON-RPC error turned the tool result into the string "{}" — and "{}" is byte-for-byte identical to a search that found nothing. The user hears "you have no data" while their session is actually expired.

Design

The verdict was moved from error wording to the structural absence of a result envelope. A real result carries four marks; any one of them clears the payload.

  1. The tool has a citation contract at all (otherwise prose is its answer, so it isn't judged)
  2. The server attached an envelope — _meta on any content block, or structured content
  3. Findings structure — a result table (even with zero rows: "a table is the server having answered with a result set") or at least one counted finding
  4. A text block opens with one of upstream's program-owned query-ran lines

The measurement that decided the design ran in the opposite direction. The deployment attaches _meta even to zero-result responses (confirmed by forcing a zero-hit query with a 1999 date range). So "zero results" cannot ground a failure verdict, and envelope absence is the honest discriminator.

The rejected alternative is documented too:

"A predicate that decided on hit units alone would ride entirely on one regex over [the upstream's] ## Image #N heading, so a heading rewrite on their side would flip EVERY photo turn to 'failed data source' … _meta survives a presentation change; the heading does not."

"THE ERROR TEXT IS NEVER CLASSIFIED. Keying the verdict on [their] wording would put it one rewording, one localization or one new upstream away from silently restoring the false-green."

Mark 4 is used only in the negative direction — matching means "not a failure," never "is a failure." If upstream rewords it, the worst outcome is the status quo, not a new misjudgment.

The error direction was tilted deliberately: "a false positive here suppresses a real answer and tells the user their data source is broken, so the predicate errs toward reading a failure as an answer rather than the reverse."

Logging does not route around the redaction chokepoint either: the diagnostic fingerprint records block count, types, and _meta key names instead of raw content.

Trade-off

  • It does not catch every disguised failure. A legitimate prose reply like "select a project first" is structurally identical to the 502. That axis has a separate defense (blocking the dispatch itself).
  • It couples to the upstream _meta contract. If upstream drops the envelope, every response could read as a failure — which is why marks 3 and 4 are retained.
  • Four code paths to maintain for one verdict. More expensive than an isError check. The judgment is that the cost is lower than "confidently wrong answers."

4.5 Firing the model as a data courier

Problem

Measured: a 10-photo result = 1,175,854 bytes, roughly 57k tokens of responder input. Ten photo descriptions run the length of a novel.

So every layer quietly truncates in self-defense. The upstream cap comment says it outright — "beyond 50 the responder node errors." Our dispatch cap was 10 for the same reason. They were two symptoms of one disease, and that silent truncation is what produced the lying chain in §1.

Design

The key insight: the channel through which the user actually sees photos is not the model's sentence — it is the structured citation frame rendering into UI cards. There is no structural reason for the model to double as a data courier.

  • Data plane: propagate upstream accounting into the citation frame as {source, displayed, total, totalIsLowerBound, hasMore}. Numbers only, no sentences.
  • Narration plane: give the model the grade verdict as an instruction.

As a side effect cost, latency, and honesty are solved by one design. A cap stops meaning "cap" and starts meaning "page size" — and a page with has_more alongside it is honest.

Prompt-cache behavior shaped the assembly order: blocks that vary per turn (view context, narrowing state) are appended last, so the stable prefix stays byte-identical across turns. Honesty instructions were added without sacrificing cache hits.

Trade-off

  • Only half of it is built. The digest step that compacts the model's input is not yet implemented. What is complete today is the instruction path plus the structured frame; resolving the input balloon itself remains (§7 roadmap).
  • It is meaningless unless the UI consumes the structured frame. Fixing only the server draws no cards. In practice there was a lag between the server emitting honest counts and the UI consuming them.
  • The frame schema grows. A larger wire contract means version-skew management on both sides.

4.6 Permission leakage blocked before the call, not filtered after it

Problem

Data from an unheld licensing tier must never surface in a citation, a thumbnail, or a sentence. The two common implementations are: instruct the prompt ("don't answer if unauthorized"), or strip the results after they arrive.

Both share a failure mode: one miss is a leak.

Design

Four filter placements were compared and the post-filter was explicitly rejected. The chosen design is a dispatch interceptor — the tool call is blocked before it goes out. The data never arrives, so leakage is structurally impossible.

Defense is doubled. Catalog scoping removes the tool from the model's view; the interceptor blocks the dispatch if it is called anyway. Both import the same map, so drift is zero.

And the planner is deliberately blinded. Planner and responder prompts must contain zero licensing vocabulary, enforced by a vocabulary gate. If the model does not know about permissions, the model has no way to leak them.

Verification status (measured). This axis has 14 dedicated test files / 131 cases, of distinct kinds:

Test kind What it pins
Illegal cells (tool × capability) combinations that must never be allowed
Drift lock Source-level assertion that the interceptor does not redefine the capability map inline — catches the moment the two paths diverge
Vocabulary gate No licensing vocabulary appears in planner/responder prompts
Audit column lock The column set of the permission-verdict audit row
e2e bridge Client hint → server verdict round trip

Explicitly not built: the capability(6) × tool(10) × hint(2) = 120-cell exhaustive matrix specified in the design has not landed. Current coverage is the combination of the 14 files above — not exhaustive. See roadmap §7.6.

Denial wording is not authored by the model either — structured copy in 5 locales and a dedicated SSE frame handle it. A path for ingesting permission verdicts as warehouse audit rows is implemented (the ingestion itself is unverified pending traffic).

Trade-off

  • The tool catalog and capability map are static. Adding an upstream tool requires updating the map. Dynamic lookup would reduce drift but make the verdict runtime-dependent and exhaustive testing impossible in principle. Verifiability was chosen — and that exhaustive verification is still outstanding debt.
  • A blinded planner may plan suboptimally. It can plan a tool it lacks rights to and burn a round trip on the block. That cost buys the removal of an entire leakage surface.
  • Denial was once spoken twice. The structured frame and the model's prose both restated the same refusal. It was fixed on two sides — server instruction replacement and UI de-duplication. Measured result: the model still restated it, and what produced a single voice for the user was the UI de-duplication. Prompt compliance is a secondary signal; the load is carried by the deterministic layer — a pattern that recurs throughout this project.

4.7 Error paths weighted equally with happy paths — 116 transition rows, CLI, and a contract-tested stub

Problem

Demo verification for LLM chatbots is usually a happy-path screenshot. But what actually angers users is the error trail — permission denial, zero results, scope rejection, expired auth, a 502 from a tool. It's hard to reproduce, so it goes untested, so it is discovered on demo day.

Design

① Illegal transitions are enumerated as data.

Target States × events Rows
SSE frame ordering 6 × 10 60
Narrowing subset gate — 20
The test harness's own state 6 × 6 36

Each row is {id, from, event, to, legal, sideEffect}. More than half carry legal: false — error paths are first-class citizens. Illegal orderings even have names (delta-before-start, plan-after-delta, unterminated, duplicate-terminal).

② The whole trail is reproducible from a CLI. Deployment round-trip UC probe (repeated K rounds), a local full-scenario script, an SSE client, a turn loop, a citation diff, cold-latency measurement, boot smoke, auth-matrix probe, upstream version preflight — 17 scripts. When a regression appears, it is reproduced with one command before a browser is opened.

③ 16 browser e2e specs — portfolio queries, narrowing, asset cards, permission-degraded, per-host mounting, locale/context wire, citation deep-link, triggers.

④ And the stub server the e2e suite stands on is itself contract-tested. This is the crux. From its header:

"Proves the substrate the deterministic corpus stands on, so a spec failure can be attributed: if THIS is red, no [e2e] spec verdict means anything."

Three checks: (1) does the dist build stream the scenario frame over real HTTP/SSE, (2) does it fail closed with 401 without a bearer — "this is what makes that forwarding load-bearing rather than decorative", (3) do the exact fields the e2e specs assert on in the DOM round-trip.

This structurally blocks mock drift — the stub diverging from reality while e2e stays green and production breaks.

Scale

Metric Value
Server source / tests 25,283 LOC / 54,227 LOC → 2.14 : 1
Test cases ~2,817 (parameterized expansions not counted; conservative)
Integration test directories 74
Panel UI 10,284 LOC / 21 specs

Trade-off

  • The techniques themselves are not new. Transition-table testing and contract testing are off-the-shelf distributed-systems engineering. What is novel is bringing them to this layer — essentially no LLM product applies them to streaming UX error trails.
  • Transition tables cost maintenance. Adding one frame adds six rows. In exchange, "is this ordering legal?" becomes a table lookup instead of a code-review argument.
  • The stub contract test requires a build artifact. It needs the dist build first, adding a CI step.

4.8 GUI context as a first-class input — while the server stays host-blind

Problem

In "safety findings on this level," "this level" is not in the conversation. It is on the screen the user is looking at in the 3D viewer. A bot that uses only conversation context cannot answer it.

At the same time there are two hosts with entirely different capabilities. The 3D viewer can move a camera; the admin console can only route. If the server starts branching per host, the server grows every time a host is added.

Design

View context was promoted to a wire contract: the project/review/team names currently open, the capture label and timestamp, the level, the panorama id, the selected annotation, and that panorama's annotation list. All optional — each host sends only what it honestly has (admin console: names only; 3D viewer: the full set) — and the server renders only what arrived.

"a host sends what it honestly has … and the server renders only what arrived"

A hostile host is assumed. Every string is length-capped and every array count-capped — "every string bounded so a hostile or buggy host cannot balloon the prompt." The prompt is treated as a trust boundary.

And the grading discipline applies here too. If the annotation list arrives at the cap, its length is a displayed subset rather than a total, so the view-context instruction discloses the possible truncation.

All host differences are absorbed by the client adapter — a 9-method contract (getFacilityKey / getAuthToken / getLicenseTierSet / getCurrentSurfaceId / getViewerSnapshot / dispatchCitation / subscribeViewerState / resolveDeepLink / getIsAdmin) plus a static capability declaration. Adding host branching to the server or the MCP envelope is forbidden by rule.

Rejecting stateful focus belongs to the same axis. The upstream MCP keeps a server-side "current project" and, when the argument is omitted, falls back to a value persisted in a store — so a stale focus left by another client can be consumed silently. We specify the arguments on every call and never take that path. Cross-project queries enforce a no-stored-focus-contamination rule with a negative boundary test.

Trade-off

  • A larger wire and more prompt cost. The view-context block costs tokens every turn. Placing it after the cache prefix mitigates but does not remove it.
  • A lying host cannot be stopped. Length caps prevent ballooning, not falsehood about content.
  • The adapter contract is fixed at nine methods. A new capability means changing the interface and both hosts following simultaneously. That rigidity is the price of the server's host-blindness.

4.9 Changing an external team's contract twice by measurement — and sealing the delivery gap

Problem

The second link of the truncation chain was not ours. The upstream MCP server is another team's asset and I have read access only. "Please include the total" simply does not happen if it isn't on their roadmap.

Design

The measurement report was the lever. Three things went into the request.

  1. Symbol-level citation — which function drops which field
  2. A live reproduction command — so they could run it themselves
  3. An internal precedent — another tool in the same folder of the same repo already follows a "Showing N of M" convention, and exactly one tool deviates from it

The third was decisive. It reframes "please do what we need" into "your own convention is unmet in one place."

Two deliveries

When What
2026-07 Accounting metadata introduced (returned / total or total_lower_bound / has_more) + the false Total: label replaced with Showing. Shipped in both Python and TypeScript
2026-08 Cursor pagination + backend fetch cap 30 → 200

Our re-measurement: reachable set for the same query 16 → 38; four pages exhausted with zero duplicates and a null final cursor.

And I caught where the delivery deviated from the request. Source measurement showed the data-query tool's cursor does not pin the SQL — it re-runs the natural-language question each page. Pages continue only while the same SQL happens to be regenerated; that is not a contractual guarantee. The flawless continuity observed in the round-trip test was an accident of that execution.

So invariants went in on our side.

  • I-a total identical across pages (including the bound style)
  • I-b page id sets disjoint
  • On violation, discard the stitch, answer from page 1 only, and disclose — "never a blended population"
  • Cursor rejection arrives in-band with isError: false, so detection is structural (a page without accounting is an unaccounted page)
  • A stitched total is never a page sum — the verified per-page total is kept and only returned accumulates
  • Four anomaly kinds enumerated + a page cap of 5 (configurable)

Trade-off

  • We are tied to an external schedule. While waiting for delivery, our side had to survive on "the most honest thing possible without knowing the total."
  • Stitching is never used for user-facing count claims. It is allowed only for id collection, capped at 5 pages. Conservative — but under a contract where the population can shift between pages, this is the only honest option.
  • Violations degrade the user experience. Discarding the stitch returns fewer results. A smaller honest answer beats a larger blended one.

4.10 Finding a dead path under a green CI, and gating the recurrence

(The incident narrative is §6.1. Here, only the structures it produced.)

Problem

The only gate approving "published → done" was merge verification. Between merge and done there was no step that executed the code. So paths that had "green tests + merged + item complete" but had never once worked accumulated — 21 items' worth.

Design

Six things were installed.

  1. Mandatory boundary-crossing tests — measured: 8 tests used the internal shape, 6 used the wire schema, 0 used both. Testing each side separately stays green forever. Now a test must feed the parser's output directly to the consumer.
  2. Contract sweep with forced invocation — opening an observation gate with a stale sweep exits 5 and refuses. There is exactly one bypass flag, and its use is stamped into the gate evidence.
  3. The sweep record carries both a commit sha and the swept paths — without paths, a gate owning a path the sweep never opened goes green on someone else's work.
  4. 103 contract boundaries registered — external-wire 41 / pinned-mirror 31 / internal-seam 15 / doc-citation 12. Because the enumeration step is non-deterministic and had in fact missed 9 boundaries once, already-named boundaries are pinned in a file as the floor for the next sweep.
  5. Unverified fan-in audit — the metric: the number of code items directly behind one observation gate = the number of items that can reach "done" on a foundation nobody has ever executed. Warning above threshold. At introduction, one terminal gate carried 51.
  6. "Answer what invokes this gate in the same commit" — applying this rule surfaced three failures the same day: a sweep tool wired into no pipeline; a pre-deploy smoke test that exited 127 for anyone with a clean install; and a module-boundary lint with rules and targets configured but zero invocations across 34 CI workflows.

As a by-product the release pipeline became three-channel immutable promotion — upper environments do not rebuild; they promote the exact artifact the previous environment actually exercised.

Trade-off

  • Gates block people. A stale sweep stops a release. Bypass is possible but leaves a trace — better than an untraceable bypass.
  • 103 boundaries are a maintenance surface. If the list only grows, sweep cost grows with it.
  • The fan-in audit warns; it does not block. Deliberately — making it blocking creates an incentive to raise the threshold and silence it.

4.11 A citation click is an app state transition, not a link

Problem

RAG citations are URLs. Clicking opens a new tab and the user loses context. For 3D site data it is worse — the meaningful target is not "that photo" but "that spot, from that viewing direction."

Design

A citation click invokes the host app's command API directly. The deep-link payload carries 4 host-scope fields, 3 BIM axes, and 3 coordinate fields (camera position, look-at point, view direction). The binding requirement is no full reload, pinned by e2e.

The episode of demoting a performance figure out of acceptance shows this axis's honesty. "Click → navigate within 500ms" was originally an acceptance criterion; it was demoted to a diagnostic ceiling. The reasoning had three steps: (a) the measured interval includes an engine camera animation whose default alone is 600ms, so even assuming zero network a single term exceeds the budget; (b) the floor of the remaining terms is a CDN, not our code; (c) the field had zero consumers in the entire app, so it had never actually been measured.

And the divergence from the customer-facing document was written down as a "known unresolved gap," not hidden.

Trade-off

  • It couples to the host. If the app's command API changes, the adapter breaks. The adapter contract localizes it, but the coupling remains.
  • It is half a feature on the admin console. No 3D viewer means routing only, which makes the coordinate fields meaningless there.
  • The coordinate resolution chain is long. Element id → model → world transform, so some axes are still unimplemented.

5. Quantitative impact

5.1 Code and verification

Metric Value
Server source 25,283 LOC / 121 files
Server tests 54,227 LOC / 349 files / ~2,817 cases → 2.14 : 1
Integration test directories 74
Panel UI 10,284 LOC / 21 specs
Browser e2e 16 specs
State-transition rows (legal + illegal) 116+
CLI reproduction scripts 17
Releases 18 in 31 days
Registered contract boundaries 103 (4 kinds)
Locales (structured copy) 5

5.2 Before → after (measured only)

Item Before After
Upstream backend fetch cap 30 200
Reachable set, same query 16 38
Count label Total: <truncated> (false) Showing N + structured accounting
Pagination zero cursor/offset params across all 6 tools cursor on both list tools
Narrowing round trip — 39 → 1 (subset + scope-applied verified; 2/2 on deployment)
Tool failure detection isError alone envelope-absence verdict (4 marks)

5.3 What is missing — and why

Of the three metric categories, one is empty, and I report it as empty.

Metric Status Reason
Cumulative users · sessions · messages None No production end-user traffic yet
Daily average / peak None Same
Tokens processed Unmeasured The single measurement (10 results ≈ 57k tokens) is a design rationale, not a load figure
Continuous uptime Not applicable Serverless execution model
Infrastructure cost Outside my access —
TTFT p50 / p95 Unmeasured No traffic
Production incidents · MTTR Not applicable No traffic

Instrumentation exists — metric emission and warehouse ingestion (tool-call rows, permission-verdict rows) are implemented. Wiring and data are different things, and the existence of wiring is not presented as a figure.

No time or cost savings are estimated either. With the above blank, even a conservative-to-optimistic range would be an assumption stacked on an assumption. "No figure without evidence" outranks "no single point estimate."

6. Incident case studies

6.1 A dead path under a green CI

Symptom. Four demo scenarios all failed on their first real run, with zero citations.

First hypothesis, rejected. I suspected a recent regression and traced commit history string by string. Result: present in all six release tags. Not a regression — it had never worked.

The most uncomfortable fact. 21 work items above that path were already marked complete, and every rule had been followed — merge verification passed 100%. This was not a rule violation. It was the outcome of following the rules.

Root cause. Two layers.

  • The only gate approving completion was merge verification, with zero execution steps between merge and done
  • Boundaries were tested from each side separately — zero tests fed the parser's output to the consumer

Structural conversion. The six items in §4.10. The point is that the response was not a bug fix but reshaping the pipeline so the same incident cannot recur.

Evidence the conversion works. Applying the new rule ("when you build a gate, answer in the same commit what invokes it") to existing assets immediately surfaced three failures — including a pre-deploy smoke test that had been exiting with code 127 and had never run for anyone doing a clean install.

What remains. The fan-in audit warns; it does not block. At introduction one terminal gate carried 51 items — meaning the same incident was already scheduled at twice the scale.

6.2 A failure disguised as a success

Symptom. Turns using a particular tool produced answers with zero citations. No failure signal reached the user.

Decomposition — three layers of cause, two of them ours.

Layer Cause
External Upstream API 502 (reproduced on 4/4 questions × 2 facilities)
Contract That failure arrives as isError: false with no envelope
Ours The only failure detector was isError === true

The measurement that decided the design ran in the opposite direction. I was heading toward "zero results means failure" — then forced a genuine zero-result response with a 1999 date range. The envelope was attached even there. So "zero results" could not ground the verdict, and envelope absence became the discriminator.

The rejected alternative is on record. Judging by a hit-unit regex would mean that the moment upstream rewrites its heading, every photo turn flips to "data source broken" — a worse failure than what we lose today.

The error direction was tilted deliberately. A false positive here erases a healthy answer and tells the user their data source is broken, so the predicate leans toward reading a failure as an answer.

A prior case in the same class. A 401 flowing through as an empty object, becoming byte-for-byte identical to a search that found nothing. Blocked on the same principle — read the transport-layer status.

6.3 A count that lied although nobody lied

Symptom. "Show me 12 photos" → 10 photos shown, with "I found 10."

Decomposition. Three truncation points; each layer produced or received the total and passed neither along. No layer acted in bad faith.

The framing determined the fix.

Do not eliminate truncation — promote it into an honest contract called a "page." A page is not a lie. Calling the page size "Total" is the lie.

Division of labor. Four items to the external team, five to us, each with symbol citations and live measurements attached. Two upstream deliveries secured (§4.9).

Our structural conversion. The four-grade taxonomy, the subject-separation invariant, the sentinel mirror, and the stitching invariants.

And the delivery deviation was absorbed too. Source measurement caught that the cursor does not pin the SQL, and our invariants sealed it — the upstream delivery was accepted, its weak point identified, and the seal placed on our side, rather than simply accepting and moving on.

7. Limits and roadmap

Kept separate from the strengths. What follows is currently true.

7.1 Latency — target not met

P95 first-token < 3.5s is the spec target, and it is not met. Measured: ~30s cold first turn, under 20s warm, 43s on some rounds. The demo guide's instruction to "throw away the first question as a warm-up" describes the present state accurately.

The cause decomposes into three: cold start, tool round trips (including a planner round), and the half-finished input compaction of §4.5. The priority is input compaction (digest). While the full upstream result enters the model's input, result volume is latency.

7.2 No production verification

Instrumentation exists, but there is no traffic to aggregate. The following are structurally blank until production opens.

  • Real user scale, cumulative sessions and messages
  • Real TTFT / error-rate distributions
  • Incident history and MTTR
  • Infrastructure cost and token unit economics

This document does not fill them in.

7.3 Model compliance is not unit-guaranteed

Honesty rules that live in the prompt layer (counter-questions, subject separation, population statements) are tested only for injection. Actual compliance is confirmed only by deployment observation. A case where the model disobeyed was measured (it restated a refusal in 2/2 rounds against an explicit instruction not to), and what saved the user-visible outcome was UI de-duplication. Prompt compliance is a secondary signal; the load is carried by the deterministic layer.

7.4 Upstream contract coupling

The sentinel constant, the envelope fields, and cursor semantics all mirror an upstream contract. Boundary registration and periodic sweeps watch for drift, but the coupling remains. In particular, the data-query cursor's failure to pin its SQL is defended by our invariants but not solved — a determinism question is outstanding with the upstream team.

7.5 Routing non-determinism

The planner sometimes routes a panorama query to the data-query tool (observed once in three rounds). Tool descriptions are upstream-owned, not our vocabulary, so a routing rule was added as a shared instruction. Whether that rule actually eliminates the miss can only be verified by post-deployment round trips.

7.6 Roadmap

Priority Item
P0 Model-input digest — feed summaries instead of full results, decoupling latency and cost from result volume
P0 "Load more" UX — the structured accounting already ships; only UI consumption and a refetch endpoint remain
P1 Safety filter implementation — input/output hooks and a production block frame
P1 Land the 120-cell exhaustive permission matrix — currently 14 files / 131 cases of partial coverage
P1 Open production instrumentation and obtain TTFT / error-rate distributions
P2 Complete the deep-link coordinate chain (element → world transform)

8. Appendix

8.1 Stack

Layer Stack
Chat server TypeScript, Node.js, serverless execution + Function URL, single SSE streaming endpoint
LLM Managed LLM (planner/responder role split, prompt caching)
Tooling MCP (JSON-RPC over HTTP), cursor pagination
State Managed sessions + session registry; AES-256-GCM at-rest encryption of conversation and feedback content (key-slot rotation, retired-key lookup)
Auth JWKS-based RS256 signature verification, dual token families, session ownership binding
Observability CloudWatch metrics, warehouse ingestion (tool calls, permission verdicts)
Panel UI Publishable Angular library, signals, host-adapter DI
Verification vitest (unit/integration), node:test (contract), Playwright (e2e), 17 CLI probes
Release Three-channel immutable promotion (dev → qa → prod) via dist-tag

8.2 Server module map (adjudication layer)

src/mcp/
  count-confidence.ts     4-grade number verdict + correction of upstream's false completion signal
  identifier-assembler.ts citation assembly + absence verdict (readResultEmptiness)
  result-identity.ts      accounting reader (carriesResultAccounting)
  in-band-failure.ts      disguised-failure detection (isInBandToolFailure) + structural fingerprint
  page-stitcher.ts        stitching invariants I-a / I-b + 4 anomaly kinds
  narration-anchor.ts     11 instruction builders — code telling the model what it learned
  interceptor.ts          permission PEP (blocks before dispatch)
  tool-capability-map.ts  static tool → capability map (shared with catalog scoping)
  envelope.ts             scope injection (spread that overwrites the model's value)
src/responder/
  subset-gate.ts          narrowing subset gate (20 transition rows)
src/frame/
  fsm.ts                  SSE frame ordering FSM (60 transition rows)
src/replay/
  harness-fsm.ts          the test harness's own FSM (36 transition rows)

8.3 Reproduction

Every code figure in this document is reproducible from the repository in this form.

# Source/test LOC ratio
git ls-tree -r --name-only <ref> <server>/src | grep '\.ts$' \
  | while read f; do git show <ref>:"$f" | wc -l; done | awk '{s+=$1} END {print s}'

# Transition-table row count
git show <ref>:<server>/src/frame/fsm.ts | grep -c "legal:"

# Contract boundary inventory
grep -oE '"kind": "[a-z-]+"' contract-boundaries.json | sort | uniq -c

Live figures (reachable set 16 → 38, lower bound 189 vs 38 unique, sentinel saturation at 3000, envelope present on zero-result responses, narrowing 39 → 1) are reproduced with the CLI probe scripts; each command and its measurement date are recorded in a separate appendix.

Short version: the summary (1–2 pages) 한국어판: the full portfolio · the summary