--- id: beacon-chat type: project doc: portfolio url: https://auejin.com/en/projects/beacon-chat/portfolio/ lang: en --- # An LLM chatbot that actively raises the confidence of its own citations — through conversation #### Every number is graded before it is spoken. Counter-questions narrow the scope until *"at least N"* becomes *"exactly N,"* and a **sample** of citations becomes the **complete set**. > **Project codename**: Beacon · **Document version**: 2026-08 > Company, product, personal names, and internal hosts are generalized. Code symbols and figures are real. --- ## Lead **What it is** An **LLM chat server (TypeScript)** and a **panel UI (Angular)** for a reality-capture construction B2B SaaS — users ask natural-language questions about 3D-captured job sites. The model **narrates only**; claims about counts, absence, and failure are **adjudicated by code**. **How far it has been verified** Three internal demo tiers (CLI → browser → full UI) run end-to-end against real dev and qa deployments with real accounts and real tokens, and passed. The final tier is feature-complete and internally verified; current work is UI/UX polish. **There is no production end-user traffic yet** — no figure in this document claims user scale. **Scale** Server 25,283 LOC · tests 54,227 LOC (**2.14 : 1**, ~2,817 cases) · panel UI 10,284 LOC · 16 browser e2e specs · 116 state-transition rows (illegal transitions included) · 103 registered contract boundaries · 18 releases in 31 days. **My role** Design and implementation of the server, the panel, and the verification harness. Drove two API contract changes in an external team's system I had no write access to, using measurement reports as the lever. --- ## 1. Executive Summary On a construction site, "how many safety findings were raised on level 3 this week?" is not small talk. That number stops work and settles subcontractor invoices. **A chatbot that states a wrong number with confidence is worse than none.** And the most common way an LLM chatbot gets a number wrong is not hallucination. It is **a chain in which nobody lied and the chain lied anyway.** Measured, in four steps: 1. The search backend puts the true match total in its response. 2. The intermediate MCP server **drops that field while parsing**, then labels the post-truncation count `Total:`. 3. Our chat server **silently rewrites** the model's requested limit down to 10. 4. The model sees 10 items and says **"I found 10."** The user asked for 12, saw 10, and believes that is all of them. No layer acted in bad faith. This project's answer is not to eliminate truncation but to **promote it into an honest contract called a "page"** — and then to go one step further: **use the conversation to turn that page into the whole population.** - Every number that is a candidate for utterance is graded **before the model speaks** — `EXACT` / `LOWER_BOUND` / `SHOWN` / `UNKNOWN`. The grade dictates the permitted sentence form, and `UNKNOWN` **forbids stating a number at all**. - Numbers that cannot be counted are given honestly as "at least N" — but the system does not stop there. **The counter-question is the operator that converts a lower bound into a certainty**: once a narrowed scope fits in a single page with nothing beyond it, the displayed cards go from *sample* to *complete set* and `SHOWN` is promoted to `EXACT`. - The same principle governs claims of absence, verdicts of failure, and permission denial. **Each kind of claim has its own standing requirement.** ### Comparison | # | Typical (ChatGPT clone · RAG tutorial · MCP tool-call demo) | Beacon | Ref | |---|---|---|---| | 1 | The model **counts the array it received** and says "I found N" | Candidate numbers are graded **before the model speaks**, and the grade dictates the permitted phrasing. `UNKNOWN` forbids stating a number at all | §4.1 | | 2 | Asks a clarifying question when ambiguous (a prompt rule) | The counter-question is a **lower-bound → certainty promotion operator**. Once the narrowed scope fits one page, a *sample* citation set becomes the *complete* one and the grade rises | §4.2 | | 3 | Empty tool result → "I couldn't find anything" | **Asserting absence requires standing.** Only a payload the server accounted for as its own search result may ground "there is none." Without standing: `unknown`, and silence | §4.3 | | 4 | Tool failure is whatever `isError` says | Detects failures **disguised as HTTP 200** by the **absence of a result envelope**. Error text is never classified — an upstream rewording cannot break it | §4.4 | | 5 | Permissions enforced by system-prompt instruction | Blocked **before the call goes out**; the data never arrives, so leakage is structurally impossible. The planner is **deliberately blind** to licensing vocabulary (vocabulary gate). 14 dedicated test files / 131 cases | §4.6 | | 6 | A happy-path screenshot is the whole demo verification | 116 transition rows **including illegal transitions** + 17 CLI reproductions + 16 browser e2e specs. The **stub server the e2e suite stands on is itself contract-tested** | §4.7 | | 7 | Built and shipped solo | Changed an external team's API contract **twice** by measurement report (fetch cap 30→200, cursor pagination · reachable set 16→38). Delivery deviations sealed with our own invariants | §4.9 | --- ## 2. Problem definition ### 2.1 What the domain demands Data in a reality-capture construction SaaS has three properties. - **Large and heavily duplicated.** Thousands of panoramas per project; the same spot is photographed repeatedly. - **Semantic and structured queries are mixed.** "Missing hard hat" is vector search; "how many captures on level 3 in April" is SQL. - **The cost of a wrong answer is asymmetric.** Claiming something exists when it doesn't is worse than missing it — and worst of all is wrongly saying *"that's all of them."* In a safety review, "no findings" is a sentence someone signs their name under. ### 2.2 Why that is hard Three things must hold simultaneously. 1. **It must be honest** — say "I don't know" when it doesn't, and never present a sample as the population. 2. **It must still be useful** — a bot that only ever answers "I can't be sure" gets abandoned. 3. **And that must be enforced in code** — writing "answer honestly" into a prompt collapses the moment the model version changes. (1) and (2) are in tension. The entire design is a way of resolving that tension: **fix honesty as a grading system, then build a path inside the conversation for raising the grade.** ### 2.3 Constraints - The chat server owns no search index. Search is only possible through an **MCP server owned by another team**, and I have no write access to it. - There are two host applications (a 3D site viewer and an admin console) with completely different rendering capabilities. - There are four licensing tiers, and data from a tier the user does not hold must **never appear in a citation, a thumbnail, or a sentence**. --- ## 3. Architecture ### 3.1 Overall ```mermaid flowchart LR subgraph HOST["Two host apps"] SV["3D site viewer"] PAC["Admin console"] end subgraph PANEL["Panel UI (publishable Angular lib)"] AD["HostAdapter · 9 methods
viewer-state subscribe ↔ citation dispatch"] CARD["Citation cards · 11 panel states"] end subgraph SRV["Chat server (TypeScript)"] PEP["Capability PEP
blocks BEFORE the tool call"] LOOP["Orchestrator loop
planner ↔ responder"] JUDGE["Adjudication layer
count-confidence · emptiness
in-band-failure · subset-gate"] ANCH["Narration anchors
code tells the model what it learned"] end subgraph EXT["External (no write access)"] MCP["Algorithm team's MCP server
photo search · data query · 4 more"] LLM["Managed LLM"] end SV --> AD PAC --> AD AD -->|"view context + auth + capability hint"| LOOP LOOP --> PEP PEP -->|allow| MCP PEP -->|"short-circuit
(data never arrives)"| ANCH MCP --> JUDGE JUDGE --> ANCH ANCH --> LLM LLM -->|"narration only"| LOOP JUDGE -->|"structured channel
citations · counts"| CARD CARD -->|"click = app state transition"| SV classDef judge fill:#0ea5e9,color:white classDef gate fill:#ef4444,color:white class JUDGE,ANCH judge class PEP gate ``` The point is that **the model sits on only one of the two arrows**. Data flows adjudication layer → structured channel → cards; the model narrates alongside it. ### 3.2 Two planes ```mermaid flowchart LR subgraph DATA["Data plane — deterministic"] R["Tool result"] --> ACC["Structured accounting
returned · total · has_more · cursor"] ACC --> FRAME["references frame
numbers only, no sentences"] FRAME --> UI["Cards + honest count line"] end subgraph NARR["Narration plane — the model"] ACC -.->|"grade verdict
as an instruction"| A["narration anchor"] A --> M["Model: narrator
numbers only from accounting"] M --> P["Prose"] end classDef good fill:#10b981,color:white class ACC,A good ``` **Values are decided deterministically in structured fields; wording is produced dynamically by the model.** The server never writes the sentence — it **injects an instruction**. Without this split, honesty rules pile up inside the prompt, and prompt rules have no way of being verified. ### 3.3 The grade ladder ```mermaid flowchart LR U["UNKNOWN
no number may be spoken"] -->|"accounting arrives"| S["SHOWN
'these N…'
subject separation required"] S -->|"narrow scope →
one page + nothing beyond"| E["EXACT
'exactly N'
assertion permitted"] L["LOWER_BOUND
'at least N'
assertion forbidden"] -->|"narrow scope"| E S -.->|"population floor exists"| L classDef e fill:#10b981,color:white classDef u fill:#6b7280,color:white class E e class U u ``` The rightward arrows are **counter-questions**. This ladder is the spine of the entire document. --- ## 4. Technical highlights Ordered by novelty × industry impact. Each is **problem → design → trade-off**. --- ### 4.1 Four grades of standing to state a number — the model never grades one itself #### Problem The only basis most LLM chatbots have for stating a count is **the length of the array they received**. Nothing decides whether that array is the whole population or the first page. And the information needed for that decision **already existed** — the upstream search engine sent the total, and an intermediate layer dropped it while parsing. Worse, upstream sometimes sends a **false completion signal.** The data-query tool injects a row cap into the SQL it generates; when results saturate the cap, `total` pins to the cap value and page exhaustion reports `has_more: false`. Measured on a project with 7,961 panoramas: `total = 3000`, no more. **Upstream was confidently wrong.** #### Design Every candidate number is classified into four grades from structured fields alone, **before the model speaks**. The module header (`src/mcp/count-confidence.ts`) states the intent itself: > *"every number that is a candidate for utterance is classified deterministically from structured fields BEFORE the model speaks, and the classification decides which phrasing the narration anchor permits. **The model never grades a number itself.**"* | Grade | Condition | Permitted phrasing | |---|---|---| | `EXACT` | `total` below the sentinel, single-value aggregate | *"may be asserted ('exactly N')"* | | `LOWER_BOUND` | vector search `total_lower_bound`, sentinel-saturated `total` | *"'at least N' only, never asserted as exact"* | | `SHOWN` | the returned/displayed subset | *"Always spoken with the subject-separation rule ('these N'), never as a population claim"* | | `UNKNOWN` | no structured accounting | *"**No count may be uttered at all.**"* | **The false completion signal is overridden.** `applyLimitSentinel` detects saturation and ignores upstream's `has_more: false`, forcing `hasMore: true`. The reason is in the code — *"a saturated page that was exhausted reports `has_more: false` upstream … but rows beyond the cap may exist."* And the **proof** that this reclassification cannot produce a falsehood in either direction is in the comment: > *"even when the true total is exactly the sentinel value, 'at least N' still holds (>= includes =)."* **The spine is the subject-separation invariant.** Model utterances split into two clauses that never share a subject. - Clause ① — subject is the **whole population**: "Of at least 189 matches across the project…" - Clause ② — subject is the **returned subset**: "**these 38** were registered on level 3 and…" "The results are on level 3" is forbidden (it implies the population). Only "these 38 are on level 3" is allowed. **This discipline is not confined to search results.** It also governs what the user is currently looking at: the host trims its annotation list to a wire cap before sending, so that length is treated as *"a SHOWN-grade number, never a total,"* and reaching the cap triggers a disclosure that the list may be truncated. **That is the strongest evidence the principle is a rule and not a one-off patch.** **The sentinel is not our constant.** The code declares it — *"an external contract, not our constant"* — makes it env-overridable, and registers it as a contract boundary watched by a periodic sweep. That guards exactly the failure mode where upstream raises its cap and our sentinel begins quietly manufacturing false `EXACT`s. #### Trade-off - **More things become unsayable.** `UNKNOWN` bans the number entirely. Against an older upstream deployment, every response degrades to `UNKNOWN` and count narration disappears. This leans deliberately toward **saying nothing rather than saying something false** — and §4.2 is what buys the loss back. - **It couples us to an upstream contract.** The sentinel mirrors an upstream prompt constant. No hard-coding, boundary registration, and periodic sweeps offset the drift risk, but **the coupling itself remains**. - **The model can't summarize freely.** The subject-separation invariant makes sentences somewhat wordier. That is the naturalness paid for accuracy. --- ### 4.2 The counter-question as a lower-bound → certainty operator #### Problem With §4.1 alone the bot becomes **honest but useless**. Answering "at least 189" every time to "how many safety findings are there?" gives the user honesty and no answer. And this is a matter of principle, not laziness. In semantic (vector) search an exact count is **unobtainable** — where you cut the similarity threshold *is* the count, and that boundary differs per query. #### Design The product axiom: > **We would rather give limited information than false information. And when all we can give is a limit, we provide a path to a number that is certain.** Two paths. **① Automatic aggregate promotion.** For "how many …?" questions, the system does not count the items in a list result. It steers the planner to re-ask the data-query tool for a **single-value aggregation** (COUNT/SUM/AVG/MIN/MAX). Single-value aggregations are **exempt** from the list row cap, so the result is `EXACT`. That the aggregate row is uncitable is irrelevant — a number needs no citation card. **② Narrowing counter-question.** On axes where aggregation cannot reach (semantic search properties), the purpose of the counter-question is redefined: **shrink the population until it fits inside a fully reachable range.** From the rule text: > *"once a narrowed scope fits in one page with nothing more beyond it, the shown count IS exact for that scope, and you may say so."* The moment a narrowed scope returns one page with `has_more = false`, the cards on screen go from **sample to complete set**, and `SHOWN` is promoted to `EXACT`. **The counter-question is not UX decoration — it is a grade-promotion operator.** Even the counter-question is bound by the discipline. The rule states *"never promise counts or listings you have not retrieved."* Suggestions are also **grounded only in filter axes the tools actually accept** (date range, scope, dimensions expressible in SQL). Proposing something unexecutable is forbidden. The generalized principle lives in the same place: > *"When a request cannot be satisfied deterministically … do not refuse outright and do not silently pick one interpretation. Ask a brief counter-question that makes the outcome deterministic. If there is exactly one way to resolve the request, just ask; if there are several, offer the options and recommend one."* The canonical case is partial entitlement: neither a blanket refusal nor a silent partial run, but **"here is the feasible scope — shall I proceed?"** #### Measured In two deployment probe rounds, a portfolio query produced an option-offering counter-question, identical in shape across both rounds: *"I need to clarify your request … 1. Analyze your current/primary project … 2. List all your accessible projects first … Which approach would you prefer?"* #### Trade-off - **More round trips.** Some questions no longer resolve in one turn. What comes back instead is an answer with a higher grade. - **Model compliance cannot be unit-tested.** Tests assert only that the rule string was injected into the system prompt; actual obedience is confirmed only by deployment observation (2/2 passing). **This limit cannot be removed** — layering a regex judge over free text is a worse solution and is banned (I once did exactly that, misjudged a passing round as a failure, and had to correct it by reading the text). - **Overused counter-questions are annoying.** The rule pins "if you can answer deterministically, just answer" as taking precedence. --- ### 4.3 Asserting absence requires standing #### Problem "I couldn't find anything" is the easiest thing a chatbot says and **the most dangerous**. In a safety context, "no such findings" is a sentence someone signs. Yet the two signals normally used to decide it are **both wrong**. **"0 citations ≠ 0 results."** An empty citation array means *not citable*, not *not present* — tables with no deep-link target, SQL that didn't select an id, aggregate rows. The module's own docstring says the empty return happens *"far more often than it returns refs"* and is *"the designed behaviour, not a failure."* Reading absence there means **telling a user who is looking at results that there are none.** **"0 hits ≠ 0 results."** Payloads that were never a search also count zero. Four measured shapes: a tool-level error, a permission short-circuit synthetic, a plain-prose "select a project first" reply, and a degraded empty object. #### Design `readResultEmptiness(toolName, rawResult) → 'empty' | 'non_empty' | 'unknown'`. The order of checks *is* the argument. 1. Not one of the tools with a result contract → `unknown` 2. hits/rows > 0 → `non_empty` (*"a result with findings in it is not empty whatever else the payload is missing"*) 3. Tool-level error → `unknown` (*"A tool-level failure is not a statement about how much data exists."*) 4. **Did the server account for this payload as its own search result?** If yes → `empty`; otherwise → `unknown` Step 4 is the crux. A count is sufficient grounds for `non_empty` only; **`empty` may be claimed only where accounting exists.** > **`unknown` is not a computation failure. It is an actual answer.** And **the right answer differs by cause.** The screen looks the same; the recovery path does not. | Cause | Discriminator | Correct narration | |---|---|---| | Permission denial | blocked before dispatch | "This requires the X subscription" (structured card) | | Auth failure | HTTP status guard | "Authentication failed" | | Zero after narrowing | **our own record** that we sent a scope | "Not among the previous results. Search the whole project?" | | Zero from the start | no scope present | "The project has no such data" | **Active recovery is implemented as releasing the narrowing, not as re-running.** The model cannot control scope — dispatch builds it from turn state and **spreads it last**, overwriting whatever the model put under the same key. Even if the model explicitly says "search everything," the outgoing call is byte-identical and still narrowed. So recovery releases narrowing at end of turn, and the instruction promises only what is achievable: *"narrowing has already been released and the next search covers everything. Do not re-run inside this turn."* #### Trade-off - **Against an older upstream, everything goes silent.** Deployments that don't attach accounting metadata degrade every verdict to `unknown` and absence narration disappears. That is the intended safe direction. - **It depends on upstream cooperation.** This verdict rests on upstream accounting for its own results. Obtaining that contract is §4.9. - **The model may still hedge on its own.** The code injects instructions; it does not censor model output. Censoring would be a free-text judgment and would reproduce §4.2's limitation. --- ### 4.4 Failures disguised as HTTP 200 — detected by envelope absence #### Problem The MCP spec places tool-level failure **inside the result** rather than in a JSON-RPC error. Upstream goes one step further: when *its* upstream is down it relays that failure as an ordinary text block and leaves `isError` false. Measured (4 different questions × 2 facilities, all identical): `Search API error: 502 Server Error: Bad Gateway …`, `isError: false`, no `_meta` on the block. > *"Nothing in `isError` distinguishes that from an answer, so the loop handed it to the model as data and the turn answered with no citations and no basis."* **The model took an error string as data and composed an answer, and no failure signal reached the user.** There was a prior case in the same class. A 401 that carried no JSON-RPC error turned the tool result into the string `"{}"` — and `"{}"` is **byte-for-byte identical to a search that found nothing**. The user hears "you have no data" while their session is actually expired. #### Design The verdict was moved from **error wording** to the **structural absence of a result envelope**. A real result carries four marks; **any one of them clears the payload.** 1. The tool has a citation contract at all (otherwise prose *is* its answer, so it isn't judged) 2. The server attached an envelope — `_meta` on any content block, or structured content 3. Findings structure — a result table (**even with zero rows**: *"a table is the server having answered with a result set"*) or at least one counted finding 4. A text block opens with one of upstream's **program-owned** query-ran lines **The measurement that decided the design ran in the opposite direction.** The deployment **attaches `_meta` even to zero-result responses** (confirmed by forcing a zero-hit query with a 1999 date range). So "zero results" cannot ground a failure verdict, and **envelope absence is the honest discriminator.** The rejected alternative is documented too: > *"A predicate that decided on hit units alone would ride entirely on one regex over [the upstream's] `## Image #N` heading, so a heading rewrite on their side would flip EVERY photo turn to 'failed data source' … **`_meta` survives a presentation change; the heading does not.**"* > *"THE ERROR TEXT IS NEVER CLASSIFIED. Keying the verdict on [their] wording would put it one rewording, one localization or one new upstream away from silently restoring the false-green."* Mark 4 is used **only in the negative direction** — matching means "not a failure," never "is a failure." If upstream rewords it, the worst outcome is **the status quo**, not a new misjudgment. **The error direction was tilted deliberately**: *"a false positive here suppresses a real answer and tells the user their data source is broken, so the predicate errs toward reading a failure as an answer rather than the reverse."* Logging does not route around the redaction chokepoint either: the diagnostic fingerprint records **block count, types, and `_meta` key names** instead of raw content. #### Trade-off - **It does not catch every disguised failure.** A legitimate prose reply like "select a project first" is structurally identical to the 502. That axis has a separate defense (blocking the dispatch itself). - **It couples to the upstream `_meta` contract.** If upstream drops the envelope, every response could read as a failure — which is why marks 3 and 4 are retained. - **Four code paths to maintain for one verdict.** More expensive than an `isError` check. The judgment is that the cost is lower than "confidently wrong answers." --- ### 4.5 Firing the model as a data courier #### Problem Measured: a 10-photo result = **1,175,854 bytes**, roughly **57k tokens** of responder input. Ten photo descriptions run the length of a novel. So **every layer quietly truncates in self-defense.** The upstream cap comment says it outright — *"beyond 50 the responder node errors."* Our dispatch cap was 10 for the same reason. **They were two symptoms of one disease**, and that silent truncation is what produced the lying chain in §1. #### Design The key insight: **the channel through which the user actually *sees* photos is not the model's sentence — it is the structured citation frame rendering into UI cards.** There is no structural reason for the model to double as a data courier. - Data plane: propagate upstream accounting into the citation frame as `{source, displayed, total, totalIsLowerBound, hasMore}`. **Numbers only, no sentences.** - Narration plane: give the model the grade verdict as an instruction. As a side effect **cost, latency, and honesty are solved by one design.** A cap stops meaning "cap" and starts meaning "page size" — and a page with `has_more` alongside it is honest. Prompt-cache behavior shaped the assembly order: blocks that vary per turn (view context, narrowing state) are appended **last**, so the stable prefix stays byte-identical across turns. Honesty instructions were added without sacrificing cache hits. #### Trade-off - **Only half of it is built.** The digest step that compacts the model's *input* is **not yet implemented**. What is complete today is the instruction path plus the structured frame; resolving the input balloon itself remains (§7 roadmap). - **It is meaningless unless the UI consumes the structured frame.** Fixing only the server draws no cards. In practice there was a lag between the server emitting honest counts and the UI consuming them. - **The frame schema grows.** A larger wire contract means version-skew management on both sides. --- ### 4.6 Permission leakage blocked before the call, not filtered after it #### Problem Data from an unheld licensing tier must never surface in a citation, a thumbnail, or a sentence. The two common implementations are: instruct the prompt ("don't answer if unauthorized"), or strip the results after they arrive. **Both share a failure mode: one miss is a leak.** #### Design Four filter placements were compared and **the post-filter was explicitly rejected**. The chosen design is a **dispatch interceptor** — the tool call is blocked **before it goes out**. The data never arrives, so **leakage is structurally impossible.** Defense is doubled. Catalog scoping removes the tool from the model's view; the interceptor blocks the dispatch if it is called anyway. Both import the same map, so drift is zero. **And the planner is deliberately blinded.** Planner and responder prompts must contain **zero** licensing vocabulary, enforced by a vocabulary gate. **If the model does not know about permissions, the model has no way to leak them.** **Verification status (measured).** This axis has **14 dedicated test files / 131 cases**, of distinct kinds: | Test kind | What it pins | |---|---| | Illegal cells | (tool × capability) combinations that must never be allowed | | **Drift lock** | Source-level assertion that the interceptor **does not redefine the capability map inline** — catches the moment the two paths diverge | | Vocabulary gate | No licensing vocabulary appears in planner/responder prompts | | Audit column lock | The column set of the permission-verdict audit row | | e2e bridge | Client hint → server verdict round trip | > **Explicitly not built**: the **capability(6) × tool(10) × hint(2) = 120-cell exhaustive matrix** specified in the design has not landed. Current coverage is the combination of the 14 files above — not exhaustive. See roadmap §7.6. Denial wording is not authored by the model either — **structured copy in 5 locales** and a dedicated SSE frame handle it. A path for ingesting permission verdicts as warehouse audit rows is implemented (the ingestion itself is unverified pending traffic). #### Trade-off - **The tool catalog and capability map are static.** Adding an upstream tool requires updating the map. Dynamic lookup would reduce drift but make the verdict runtime-dependent and exhaustive testing impossible in principle. **Verifiability was chosen — and that exhaustive verification is still outstanding debt.** - **A blinded planner may plan suboptimally.** It can plan a tool it lacks rights to and burn a round trip on the block. That cost buys the removal of an entire leakage surface. - **Denial was once spoken twice.** The structured frame and the model's prose both restated the same refusal. It was fixed on two sides — server instruction replacement and UI de-duplication. Measured result: **the model still restated it, and what produced a single voice for the user was the UI de-duplication.** Prompt compliance is a secondary signal; the load is carried by the deterministic layer — a pattern that recurs throughout this project. --- ### 4.7 Error paths weighted equally with happy paths — 116 transition rows, CLI, and a contract-tested stub #### Problem Demo verification for LLM chatbots is usually a **happy-path screenshot**. But what actually angers users is the error trail — permission denial, zero results, scope rejection, expired auth, a 502 from a tool. It's hard to reproduce, so it goes untested, so **it is discovered on demo day.** #### Design **① Illegal transitions are enumerated as data.** | Target | States × events | Rows | |---|---|---| | SSE frame ordering | 6 × 10 | **60** | | Narrowing subset gate | — | **20** | | **The test harness's own state** | 6 × 6 | **36** | Each row is `{id, from, event, to, legal, sideEffect}`. More than half carry `legal: false` — **error paths are first-class citizens.** Illegal orderings even have names (`delta-before-start`, `plan-after-delta`, `unterminated`, `duplicate-terminal`). **② The whole trail is reproducible from a CLI.** Deployment round-trip UC probe (repeated K rounds), a local full-scenario script, an SSE client, a turn loop, a citation diff, cold-latency measurement, boot smoke, auth-matrix probe, upstream version preflight — **17 scripts**. When a regression appears, it is reproduced with one command before a browser is opened. **③ 16 browser e2e specs** — portfolio queries, narrowing, asset cards, **permission-degraded**, per-host mounting, locale/context wire, citation deep-link, triggers. **④ And the stub server the e2e suite stands on is itself contract-tested.** This is the crux. From its header: > *"Proves the substrate the deterministic corpus stands on, so a spec failure can be attributed: **if THIS is red, no [e2e] spec verdict means anything.**"* Three checks: (1) does the dist build stream the scenario frame over real HTTP/SSE, (2) does it **fail closed with 401** without a bearer — *"this is what makes that forwarding load-bearing rather than decorative"*, (3) do the exact fields the e2e specs assert on in the DOM round-trip. This structurally blocks **mock drift** — the stub diverging from reality while e2e stays green and production breaks. #### Scale | Metric | Value | |---|---| | Server source / tests | 25,283 LOC / 54,227 LOC → **2.14 : 1** | | Test cases | ~2,817 (parameterized expansions not counted; conservative) | | Integration test directories | 74 | | Panel UI | 10,284 LOC / 21 specs | #### Trade-off - **The techniques themselves are not new.** Transition-table testing and contract testing are off-the-shelf distributed-systems engineering. **What is novel is bringing them to this layer** — essentially no LLM product applies them to streaming UX error trails. - **Transition tables cost maintenance.** Adding one frame adds six rows. In exchange, "is this ordering legal?" becomes a table lookup instead of a code-review argument. - **The stub contract test requires a build artifact.** It needs the dist build first, adding a CI step. --- ### 4.8 GUI context as a first-class input — while the server stays host-blind #### Problem In "safety findings on this level," **"this level" is not in the conversation.** It is on the screen the user is looking at in the 3D viewer. A bot that uses only conversation context cannot answer it. At the same time there are two hosts with entirely different capabilities. The 3D viewer can move a camera; the admin console can only route. If the server starts branching per host, the server grows every time a host is added. #### Design **View context was promoted to a wire contract**: the project/review/team names currently open, the capture label and timestamp, the level, the panorama id, the selected annotation, and that panorama's annotation list. All optional — **each host sends only what it honestly has** (admin console: names only; 3D viewer: the full set) — and the server **renders only what arrived**. ``` "a host sends what it honestly has … and the server renders only what arrived" ``` **A hostile host is assumed.** Every string is length-capped and every array count-capped — *"every string bounded so a hostile or buggy host cannot balloon the prompt."* The prompt is treated as a trust boundary. **And the grading discipline applies here too.** If the annotation list arrives at the cap, its length is a displayed subset rather than a total, so the view-context instruction **discloses the possible truncation**. **All host differences are absorbed by the client adapter** — a 9-method contract (`getFacilityKey` / `getAuthToken` / `getLicenseTierSet` / `getCurrentSurfaceId` / `getViewerSnapshot` / `dispatchCitation` / `subscribeViewerState` / `resolveDeepLink` / `getIsAdmin`) plus a static capability declaration. **Adding host branching to the server or the MCP envelope is forbidden by rule.** **Rejecting stateful focus belongs to the same axis.** The upstream MCP keeps a server-side "current project" and, when the argument is omitted, falls back to a value persisted in a store — so a stale focus left by another client can be consumed silently. We **specify the arguments on every call** and never take that path. Cross-project queries enforce a no-stored-focus-contamination rule with a negative boundary test. #### Trade-off - **A larger wire and more prompt cost.** The view-context block costs tokens every turn. Placing it after the cache prefix mitigates but does not remove it. - **A lying host cannot be stopped.** Length caps prevent ballooning, not falsehood about content. - **The adapter contract is fixed at nine methods.** A new capability means changing the interface and both hosts following simultaneously. That rigidity is the price of the server's host-blindness. --- ### 4.9 Changing an external team's contract twice by measurement — and sealing the delivery gap #### Problem The second link of the truncation chain **was not ours**. The upstream MCP server is another team's asset and I have read access only. "Please include the total" simply does not happen if it isn't on their roadmap. #### Design **The measurement report was the lever.** Three things went into the request. 1. **Symbol-level citation** — which function drops which field 2. **A live reproduction command** — so they could run it themselves 3. **An internal precedent** — *another tool in the same folder of the same repo already follows a "Showing N of M" convention, and exactly one tool deviates from it* The third was decisive. It reframes "please do what we need" into "your own convention is unmet in one place." **Two deliveries** | When | What | |---|---| | 2026-07 | Accounting metadata introduced (`returned` / `total` or `total_lower_bound` / `has_more`) + the false `Total:` label replaced with `Showing`. Shipped in both Python and TypeScript | | 2026-08 | Cursor pagination + backend fetch cap 30 → **200** | Our re-measurement: reachable set for the same query **16 → 38**; four pages exhausted with **zero duplicates** and a null final cursor. **And I caught where the delivery deviated from the request.** Source measurement showed the data-query tool's cursor **does not pin the SQL** — it re-runs the natural-language question each page. Pages continue only while the same SQL happens to be regenerated; **that is not a contractual guarantee.** The flawless continuity observed in the round-trip test was an accident of that execution. So invariants went in on our side. - **I-a** `total` identical across pages (including the bound style) - **I-b** page id sets disjoint - On violation, **discard the stitch**, answer from page 1 only, and disclose — *"never a blended population"* - Cursor rejection arrives in-band with `isError: false`, so detection is **structural** (a page without accounting is an unaccounted page) - **A stitched total is never a page sum** — the verified per-page total is kept and only `returned` accumulates - Four anomaly kinds enumerated + a page cap of 5 (configurable) #### Trade-off - **We are tied to an external schedule.** While waiting for delivery, our side had to survive on "the most honest thing possible without knowing the total." - **Stitching is never used for user-facing count claims.** It is allowed only for id collection, capped at 5 pages. Conservative — but under a contract where the population can shift between pages, this is the only honest option. - **Violations degrade the user experience.** Discarding the stitch returns fewer results. A smaller honest answer beats a larger blended one. --- ### 4.10 Finding a dead path under a green CI, and gating the recurrence *(The incident narrative is §6.1. Here, only the structures it produced.)* #### Problem The only gate approving "published → done" was **merge verification**. Between merge and done there was **no step that executed the code.** So paths that had "green tests + merged + item complete" but **had never once worked** accumulated — 21 items' worth. #### Design Six things were installed. 1. **Mandatory boundary-crossing tests** — measured: 8 tests used the internal shape, 6 used the wire schema, **0 used both**. Testing each side separately stays green forever. Now a test must feed the parser's *output* directly to the consumer. 2. **Contract sweep with forced invocation** — opening an observation gate with a stale sweep **exits 5 and refuses**. There is exactly one bypass flag, and its use is stamped into the gate evidence. 3. **The sweep record carries both a commit sha and the swept paths** — without paths, *a gate owning a path the sweep never opened goes green on someone else's work.* 4. **103 contract boundaries registered** — external-wire 41 / pinned-mirror 31 / internal-seam 15 / doc-citation 12. Because the enumeration step is non-deterministic and had in fact missed 9 boundaries once, already-named boundaries are pinned in a file as the floor for the next sweep. 5. **Unverified fan-in audit** — the metric: *the number of code items directly behind one observation gate = the number of items that can reach "done" on a foundation nobody has ever executed.* Warning above threshold. At introduction, one terminal gate carried **51**. 6. **"Answer what invokes this gate in the same commit"** — applying this rule surfaced three failures the same day: a sweep tool wired into no pipeline; a pre-deploy smoke test that **exited 127** for anyone with a clean install; and a module-boundary lint with rules and targets configured but **zero invocations across 34 CI workflows**. As a by-product the release pipeline became **three-channel immutable promotion** — upper environments do not rebuild; they promote **the exact artifact the previous environment actually exercised**. #### Trade-off - **Gates block people.** A stale sweep stops a release. Bypass is possible but leaves a trace — better than an untraceable bypass. - **103 boundaries are a maintenance surface.** If the list only grows, sweep cost grows with it. - **The fan-in audit warns; it does not block.** Deliberately — making it blocking creates an incentive to raise the threshold and silence it. --- ### 4.11 A citation click is an app state transition, not a link #### Problem RAG citations are URLs. Clicking opens a new tab and the user loses context. For 3D site data it is worse — the meaningful target is not "that photo" but **"that spot, from that viewing direction."** #### Design A citation click invokes the host app's command API directly. The deep-link payload carries 4 host-scope fields, 3 BIM axes, and **3 coordinate fields (camera position, look-at point, view direction)**. The binding requirement is **no full reload**, pinned by e2e. **The episode of demoting a performance figure out of acceptance** shows this axis's honesty. "Click → navigate within 500ms" was originally an acceptance criterion; it was demoted to a diagnostic ceiling. The reasoning had three steps: (a) the measured interval includes an engine camera animation whose default alone is 600ms, so even assuming zero network a single term exceeds the budget; (b) the floor of the remaining terms is a CDN, not our code; (c) the field had **zero consumers** in the entire app, so it had never actually been measured. And the divergence from the customer-facing document was written down as a **"known unresolved gap," not hidden**. #### Trade-off - **It couples to the host.** If the app's command API changes, the adapter breaks. The adapter contract localizes it, but the coupling remains. - **It is half a feature on the admin console.** No 3D viewer means routing only, which makes the coordinate fields meaningless there. - **The coordinate resolution chain is long.** Element id → model → world transform, so some axes are still unimplemented. --- ## 5. Quantitative impact ### 5.1 Code and verification | Metric | Value | |---|---| | Server source | 25,283 LOC / 121 files | | Server tests | 54,227 LOC / 349 files / ~2,817 cases → **2.14 : 1** | | Integration test directories | 74 | | Panel UI | 10,284 LOC / 21 specs | | Browser e2e | 16 specs | | State-transition rows (legal + illegal) | 116+ | | CLI reproduction scripts | 17 | | Releases | 18 in **31 days** | | Registered contract boundaries | 103 (4 kinds) | | Locales (structured copy) | 5 | ### 5.2 Before → after (measured only) | Item | Before | After | |---|---|---| | Upstream backend fetch cap | 30 | **200** | | Reachable set, same query | 16 | **38** | | Count label | `Total: ` (false) | `Showing N` + structured accounting | | Pagination | **zero** cursor/offset params across all 6 tools | cursor on both list tools | | Narrowing round trip | — | 39 → 1 (subset + scope-applied verified; 2/2 on deployment) | | Tool failure detection | `isError` alone | envelope-absence verdict (4 marks) | ### 5.3 What is missing — and why Of the three metric categories, **one is empty, and I report it as empty.** | Metric | Status | Reason | |---|---|---| | Cumulative users · sessions · messages | **None** | No production end-user traffic yet | | Daily average / peak | **None** | Same | | Tokens processed | **Unmeasured** | The single measurement (10 results ≈ 57k tokens) is a design rationale, not a load figure | | Continuous uptime | **Not applicable** | Serverless execution model | | Infrastructure cost | **Outside my access** | — | | TTFT p50 / p95 | **Unmeasured** | No traffic | | Production incidents · MTTR | **Not applicable** | No traffic | **Instrumentation exists** — metric emission and warehouse ingestion (tool-call rows, permission-verdict rows) are implemented. Wiring and data are different things, and **the existence of wiring is not presented as a figure.** **No time or cost savings are estimated either.** With the above blank, even a conservative-to-optimistic range would be an assumption stacked on an assumption. **"No figure without evidence" outranks "no single point estimate."** --- ## 6. Incident case studies ### 6.1 A dead path under a green CI **Symptom.** Four demo scenarios all failed on their first real run, with zero citations. **First hypothesis, rejected.** I suspected a recent regression and traced commit history string by string. Result: **present in all six release tags.** Not a regression — it had never worked. **The most uncomfortable fact.** **21 work items above that path were already marked complete**, and **every rule had been followed** — merge verification passed 100%. This was not a rule violation. It was **the outcome of following the rules.** **Root cause.** Two layers. - The only gate approving completion was merge verification, with **zero execution steps** between merge and done - Boundaries were tested from each side separately — **zero tests fed the parser's output to the consumer** **Structural conversion.** The six items in §4.10. The point is that the response was not a bug fix but **reshaping the pipeline so the same incident cannot recur.** **Evidence the conversion works.** Applying the new rule ("when you build a gate, answer in the same commit what invokes it") to existing assets **immediately surfaced three failures** — including a pre-deploy smoke test that had been exiting with code 127 and had never run for anyone doing a clean install. **What remains.** The fan-in audit warns; it does not block. At introduction one terminal gate carried 51 items — meaning the same incident was already scheduled at twice the scale. --- ### 6.2 A failure disguised as a success **Symptom.** Turns using a particular tool produced answers with zero citations. No failure signal reached the user. **Decomposition — three layers of cause, two of them ours.** | Layer | Cause | |---|---| | External | Upstream API 502 (reproduced on 4/4 questions × 2 facilities) | | Contract | That failure arrives as `isError: false` with no envelope | | Ours | The only failure detector was `isError === true` | **The measurement that decided the design ran in the opposite direction.** I was heading toward "zero results means failure" — then forced a genuine zero-result response with a 1999 date range. **The envelope was attached even there.** So "zero results" could not ground the verdict, and **envelope absence** became the discriminator. **The rejected alternative is on record.** Judging by a hit-unit regex would mean that the moment upstream rewrites its heading, **every photo turn flips to "data source broken"** — a worse failure than what we lose today. **The error direction was tilted deliberately.** A false positive here erases a healthy answer and tells the user their data source is broken, so the predicate leans toward **reading a failure as an answer.** **A prior case in the same class.** A 401 flowing through as an empty object, becoming *byte-for-byte identical to a search that found nothing*. Blocked on the same principle — read the transport-layer status. --- ### 6.3 A count that lied although nobody lied **Symptom.** "Show me 12 photos" → 10 photos shown, with **"I found 10."** **Decomposition.** Three truncation points; each layer produced or received the total and passed neither along. No layer acted in bad faith. **The framing determined the fix.** > Do not eliminate truncation — **promote it into an honest contract called a "page."** > A page is not a lie. **Calling the page size "Total" is the lie.** **Division of labor.** Four items to the external team, five to us, each with symbol citations and live measurements attached. Two upstream deliveries secured (§4.9). **Our structural conversion.** The four-grade taxonomy, the subject-separation invariant, the sentinel mirror, and the stitching invariants. **And the delivery deviation was absorbed too.** Source measurement caught that the cursor does not pin the SQL, and our invariants sealed it — **the upstream delivery was accepted, its weak point identified, and the seal placed on our side**, rather than simply accepting and moving on. --- ## 7. Limits and roadmap Kept separate from the strengths. What follows is **currently true**. ### 7.1 Latency — target not met P95 first-token < 3.5s is the spec target, and it is **not met**. Measured: ~**30s** cold first turn, under **20s** warm, **43s** on some rounds. The demo guide's instruction to *"throw away the first question as a warm-up"* describes the present state accurately. The cause decomposes into three: cold start, tool round trips (including a planner round), and the half-finished input compaction of §4.5. The priority is **input compaction (digest)**. While the full upstream result enters the model's input, result volume *is* latency. ### 7.2 No production verification Instrumentation exists, but **there is no traffic to aggregate.** The following are structurally blank until production opens. - Real user scale, cumulative sessions and messages - Real TTFT / error-rate distributions - Incident history and MTTR - Infrastructure cost and token unit economics This document **does not fill them in.** ### 7.3 Model compliance is not unit-guaranteed Honesty rules that live in the prompt layer (counter-questions, subject separation, population statements) are tested only for **injection**. Actual compliance is confirmed only by deployment observation. A case where the model **disobeyed** was measured (it restated a refusal in 2/2 rounds against an explicit instruction not to), and what saved the user-visible outcome was UI de-duplication. **Prompt compliance is a secondary signal; the load is carried by the deterministic layer.** ### 7.4 Upstream contract coupling The sentinel constant, the envelope fields, and cursor semantics all mirror an upstream contract. Boundary registration and periodic sweeps watch for drift, but **the coupling remains.** In particular, the data-query cursor's failure to pin its SQL is defended by our invariants but **not solved** — a determinism question is outstanding with the upstream team. ### 7.5 Routing non-determinism The planner sometimes routes a panorama query to the data-query tool (observed once in three rounds). Tool descriptions are upstream-owned, not our vocabulary, so a routing rule was added as a shared instruction. **Whether that rule actually eliminates the miss can only be verified by post-deployment round trips.** ### 7.6 Roadmap | Priority | Item | |---|---| | P0 | Model-input digest — feed summaries instead of full results, decoupling latency and cost from result volume | | P0 | "Load more" UX — the structured accounting already ships; only UI consumption and a refetch endpoint remain | | P1 | Safety filter implementation — input/output hooks and a production block frame | | P1 | Land the 120-cell exhaustive permission matrix — currently 14 files / 131 cases of partial coverage | | P1 | Open production instrumentation and obtain TTFT / error-rate distributions | | P2 | Complete the deep-link coordinate chain (element → world transform) | --- ## 8. Appendix ### 8.1 Stack | Layer | Stack | |---|---| | Chat server | TypeScript, Node.js, serverless execution + Function URL, single SSE streaming endpoint | | LLM | Managed LLM (planner/responder role split, prompt caching) | | Tooling | MCP (JSON-RPC over HTTP), cursor pagination | | State | Managed sessions + session registry; AES-256-GCM at-rest encryption of conversation and feedback content (key-slot rotation, retired-key lookup) | | Auth | JWKS-based RS256 signature verification, dual token families, session ownership binding | | Observability | CloudWatch metrics, warehouse ingestion (tool calls, permission verdicts) | | Panel UI | Publishable Angular library, signals, host-adapter DI | | Verification | vitest (unit/integration), node:test (contract), Playwright (e2e), 17 CLI probes | | Release | Three-channel immutable promotion (dev → qa → prod) via dist-tag | ### 8.2 Server module map (adjudication layer) ``` src/mcp/ count-confidence.ts 4-grade number verdict + correction of upstream's false completion signal identifier-assembler.ts citation assembly + absence verdict (readResultEmptiness) result-identity.ts accounting reader (carriesResultAccounting) in-band-failure.ts disguised-failure detection (isInBandToolFailure) + structural fingerprint page-stitcher.ts stitching invariants I-a / I-b + 4 anomaly kinds narration-anchor.ts 11 instruction builders — code telling the model what it learned interceptor.ts permission PEP (blocks before dispatch) tool-capability-map.ts static tool → capability map (shared with catalog scoping) envelope.ts scope injection (spread that overwrites the model's value) src/responder/ subset-gate.ts narrowing subset gate (20 transition rows) src/frame/ fsm.ts SSE frame ordering FSM (60 transition rows) src/replay/ harness-fsm.ts the test harness's own FSM (36 transition rows) ``` ### 8.3 Reproduction Every code figure in this document is reproducible from the repository in this form. ```bash # Source/test LOC ratio git ls-tree -r --name-only /src | grep '\.ts$' \ | while read f; do git show :"$f" | wc -l; done | awk '{s+=$1} END {print s}' # Transition-table row count git show :/src/frame/fsm.ts | grep -c "legal:" # Contract boundary inventory grep -oE '"kind": "[a-z-]+"' contract-boundaries.json | sort | uniq -c ``` Live figures (reachable set 16 → 38, lower bound 189 vs 38 unique, sentinel saturation at 3000, envelope present on zero-result responses, narrowing 39 → 1) are reproduced with the CLI probe scripts; each command and its measurement date are recorded in a separate appendix. --- **Short version**: the summary (1–2 pages) **한국어판**: the full portfolio · the summary