Beyond Chunks: A Structure-Preserving, Claim-Checked Research Engine for Cross-Document Question Answering

1 Introduction

The standard RAG recipe [1, 2, 3, 4] retrieves passages by lexical [5] or dense [3] similarity, often fused [6], and conditions a generator on them. Deployed over a company's documents it is asked a different kind of question than the open-domain benchmarks it was developed on: what does our policy allow, which version applies, why does this step fail, how do these documents fit together. Our users—analysts, engineers, support staff and clinicians reading their own organization's corpora—expect the answer an author or a subject expert would give: complete, ordered so a newcomer can follow it, with the figure or table that explains it, with every statement traceable to the text, and with an explicit account of what the documents do not cover. A company that adopts such a system additionally expects auditability (why did it answer this), tenant isolation, freshness as sources change, predictable cost, and no confidently wrong answers.

Chunk-based RAG struggles with exactly these expectations. (i) Recall across documents: the definition a question needs is often in a different document from the text that uses it, and similarity retrieval over chunks ranks it low. (ii) Structure: fixed windows cut tables, separate a figure from the step it illustrates, and reorder numbered procedures. (iii) Completeness: nothing in the pipeline knows what a complete answer must contain, so a partial answer is presented with the same confidence as a complete one. (iv) Faithfulness at display: citation-generation work [7] and claim-level factuality metrics [8] show that generated text drifts from its sources; production systems rarely check what they display.

Contributions.

1. A structure-preserving ingestion pipeline (section tree, tables, figures described by a vision model, page-window reading with checkpoints) and an assertion ledger of conditioned facts with cross-document links built at ingest (§3).

2. An obligation-driven answering loop that plans what a complete answer must establish, reads whole sections through several recall routes, checks every claim verbatim against its cited text, and states what is not covered (§3.3).

3. An evaluation protocol for cross-document QA that measures coverage of preregistered must-have items under a blind judge, with a dev/held-out discipline, two-run means against measured run-to-run noise, and an honest denominator for “verified” (§4).

4. Results to date, including a ceiling experiment that localizes the loss, a model-swap experiment, and a correction to our own earlier metric (§6).

2 Related Work

Retrieval-augmented generation. RAG [1], REALM [2] and Fusion-in-Decoder [4] established retrieve-then-generate over passage collections, with dense retrieval [3] complementing BM25 [5]. Contextual retrieval [9] prepends document context to chunks before indexing. Our engine keeps the document's sections as the unit of reading rather than fixed windows.

Structured and hierarchical retrieval. RAPTOR [10] builds a tree of recursive summaries; GraphRAG [11] builds an entity graph with community summaries for global questions; HippoRAG [12] uses a knowledge graph with personalized PageRank for multi-hop retrieval. Our assertion ledger is closer to these than to flat chunk indexes, but its nodes are conditioned assertions with verbatim quotes (a rule, its conditions, exceptions and source sentence), and its edges are typed relations recorded with their evidence (resolves, supersedes, depends on, contradicts as a candidate only).

Iterative and agentic retrieval. IRCoT [13] interleaves retrieval with chain-of-thought; ReAct [14] interleaves reasoning with tool actions; Self-RAG [15] trains a model to decide when to retrieve and to critique its output. Our research mode is an agent in this family; our everyday modes use a fixed obligation loop instead of free-form planning.

Long context versus retrieval. Long-context models lose information placed in the middle of the input [16]; Li et al. [18] compare RAG with long-context models and route between them; LongRAG [17] retrieves larger units for long-context readers. Our ceiling arm (A) gives the model a whole shelf, as an upper bound feasible only for small corpora.

Evaluation. RAGAS [19] and ARES [20] score faithfulness, answer relevance and context relevance; ALCE [7] scores citation quality; FActScore [8] scores atomic factual precision. LLM judges are known to be useful but biased [21]. Our protocol differs in what it counts: coverage of preregistered must-have items written from the documents before any run, a wrong category separate from incomplete, the share of displayed sentences that were checked, and cost and latency per answer (§4).

3 System

3.1 Structure-preserving ingestion

Documents (PDF, Word, Excel, PowerPoint, HTML, Markdown) are parsed with Docling [22] into a section tree: headings, paragraphs, tables as grids, figures with captions and crops, and the page each element starts on. Each figure is described by a vision model; the description is input for the writer, not shown to the reader. Sections receive stable identifiers so a re-upload updates rather than duplicates. Large PDFs are read in windows of ten pages: the windows are converted separately, concatenated, and walked once, so the section tree equals a single pass; finished windows are checkpointed under the tenant's prefix so an interrupted read resumes, and each window's figures are described while the next is read.

3.2 Assertion ledger

After indexing, an extraction pass turns each section into assertions: definitions, rules (with conditions, thresholds and exceptions), findings, procedure steps, dependencies and supersessions, each with the verbatim sentence it came from. Undefined references (“as defined in section 3”, “see Figure 2”, another document named by title or identifier) become open questions (hooks) that a resolver tries to answer elsewhere in the corpus; a resolution is a typed link. Concepts named differently in different documents are aligned by a code veto plus a model judgement.

3.3 Answering

Understanding. The question is planned into the fewest obligations a complete answer needs, each with a kind (describe, explain, compare, trace, judge, list, lookup, propose), its targets in the documents' own terms, and the documents likely to hold it.

Recall. For each obligation, candidate sections come from several routes: headings and chapters, the ledger's facts index, keyword and dense search fused by reciprocal rank [6], figures by caption, numeric scans, and a document the question names. Whole sections (not fragments) are read within a budget; every candidate not opened is recorded with the reason. For dense search a section is embedded as all of its text: passages of about 2,400 characters with a 300-character overlap, split at paragraph, sentence and word boundaries, each prefixed with the document title and the section's heading path [9]; a section scores as its best passage. Until October 2026 a section was embedded as its first 24,000 characters without context.

Writing. A part's writer receives the sections and an explicit format for its reader: an explanation built from the basics for “what/how/why” parts, ordered steps with their figures for procedures. Every sentence ends with the section identifiers it draws on.

Claim checking. A checker compares each claim with its cited text and must copy a verbatim span (verified in code) for a supports or contradicts verdict; a contradicted claim is removed, an unconfirmed one is restated in the document's words or labelled.

Completeness. An acceptance step compares the answer against the obligations; what is not covered is stated in one plain line each, never presented as complete.

No length cut-offs. A length budget decides when text is read or written, never whether. Sources that exceed one writer's budget (48,000 characters) go to a second writer, which adds only what is not yet written; composing several parts into one answer gives the writer all of the parts and output room in proportion to them, and a part the composed answer omits is appended in its own checked words; the lead, the completeness critic and the acceptance judge read the whole answer. Before this change those steps saw the first 3,000, 30,000 and 14,000 characters, and the composer saw the first 30,000 characters of the parts.

3.4 Displayed equals checked

A measurement in §6.4 showed that our earlier display kept uncited sentences that sat in a paragraph with a cited one. The invariant we adopted is that the checking unit is the displayed sentence: connective text may pass unchecked only when it carries no number, date, name or modal verb.

4 Evaluation Methodology

4.1 Question sets

Cross-document set (primary).

34 questions over five shelves, every question asked over the whole shelf: oauth (RFC 6749, 6750, 7519), medical (six FDA labels including two versions of apixaban), papers (ReAct [14], Tree of Thoughts [23], Self-RAG [15], RAPTOR [10], MemGPT [24]), northwind (a fictional medical-device quality system of 11 documents in PDF, Word, Excel and PowerPoint), and typing (seven Python PEPs). Question kinds include a term defined in another document, labels that disagree, a procedure spanning two standards, versions of one document, and who cites whom. Must-have items were written from the documents' text before any run and committed; 17 questions are the development split (changes are tuned on them) and 17 are held out (run only to confirm a change).

Regulatory set.

30 preregistered questions over three public texts (HIPAA Security Rule, GDPR, FAR 52.212-4), graded right, right-but-labelled, honest miss, or wrong. Because fixes were made between runs on the same questions, later runs are development results.

Blind set.

42 preregistered questions over four unseen public documents (CC BY 4.0 legal code, the RAG paper [1], the GNU Make manual, NIST SP 800-63B), same grading; also iterated.

Stress set.

63 questions over three corpora on the real model, including false premises, out-of-corpus questions, table lookups and conversational follow-ups.

4.2 Metrics

Coverage.

A blind judge (gpt-4o) sees the question, the must items and the answer, never the system or mode, and scores each item 1, 0.5 or 0; coverage is the mean.

Wrong.

An answer that states something the documents contradict, separately from incomplete answers; the product target is zero.

Verified share and its denominator.

The share of claims whose verbatim span the checker found. We report it against displayed sentences, not only checked claims (§6.4).

Labelled versus right.

Whether the engine's own completeness state (complete, partial, conditional) matches the grade: a right answer labelled partial is a calibration failure, not a wrong answer.

Cost and latency.

Dollars and seconds per answer from the system's own usage records.

Retrieval metrics.

Against gold evidence: for each must item, the sections that state it. Recall@k, precision@k, MRR and nDCG@10 over the union of a question's evidence (the IR set BEIR reports [25]); context recall@k, the share of must items with at least one evidence section in the top k, and context precision (average precision), as in RAGAS [19]. Every system is scored at section level; a chunk retrieves the sections it carries at least 200 characters of (or all of a shorter one).

Gold evidence.

Pooled as in TREC [26]: per must item, the union of BM25 and dense top 10 for the item and for the question, the sections behind a chunk-RAG's top 5 chunks and sections whose number the item names; a judge model (gpt-4.1-mini, temperature 0) reads every candidate's whole text and keeps those that state the item. 195 items, 194 with evidence (the other states an absence), mean pool 31.8 sections, $0.98; a hand check of 7 random items found all labels correct.

The scoreboard (paper/data/scoreboard.py) regenerates every table from stored runs; a metric not measured for a system is printed as n.m., never omitted.

4.3 Noise and decision rules

The same code scored 0.69 and 0.79 on the 17 development questions in two runs; single questions swung by up to 0.8. We therefore compare configurations on two-run means and treat a difference as real only when it exceeds the run-to-run spread. The ceiling experiments (§6.3) were written with their decision rules and committed before any of them ran.

5 What Improves the Answers

Every change we made to the engine, what it was meant to fix, what it measurably did, and whether it stays (Table 1). The yardstick is the cross-document set (34 questions, blind gpt-4o judge, coverage of preregistered must-have items, dev / held-out); retrieval changes are also scored against labelled gold evidence (§6.1). Two runs of identical code differ by up to ±0.05 on 17 questions, so a change is adopted for coverage only when it gains at least 0.03 on both splits over two-run means; a change for safety or traceability is adopted on its own test. Kept = on in the product; built, off = in the code behind a switch; not adopted = measured and rejected; open = needs a decision.

What this says. Two changes moved answers: giving the writer the right text to read (whole-scope reading with a relevance budget: 0.58 / 0.49 → 0.79 / 0.79) and a stronger writer model. Retrieval changes improved every retrieval metric and did not move answers, because the evidence was already found; changes to how the writer works (multi-step pipelines) did not beat a single well-prompted call; and more context for the writer is not better context - every ledger link in scope, unranked, cost 0.03–0.08 coverage. What remains is in the writer: 19–22% of required facts are read and never written, most of them chains across three documents. That is where the next step has to act.

6 Results

6.1 Retrieval against gold evidence

Table 2 scores every retrieval system on the same gold evidence, with the user's question as the query (the engine plans sub-queries per obligation; this isolates the retriever). Indexes were built from staging's own parse of the five shelves in a local copy (scripts/eval/retrieval/); embeddings text-embedding-3-small.

Sections beat chunks. A conventional 512-token chunk RAG reaches recall@10 0.35 and nDCG@10 0.38 (dense; hybrid 0.34 and 0.37); the chunkless engine's retrieval as it stood before October 2026 reaches 0.48 and 0.57 on the same gold, and today's 0.50 and 0.59.

What changed this week, measured. BM25 replacing ts_rank lifts keyword recall@10 from 0.39 to 0.45 and nDCG@10 from 0.45 to 0.50; headed passages replacing the first 24,000 characters lift dense MRR from 0.80 to 0.90 at equal recall (only 4 sections here were longer than the cut). The fused hybrid moves recall@20 from 0.61 to 0.63: Phase 1's preregistered gate (+20 points recall@20 on dev) is not met by these two changes.

Reranking. A listwise rerank of the fused top 50 by gpt-4.1-mini, shown each section's best-matching passage, lifts recall@10 to 0.61, recall@20 to 0.72, MRR to 0.97 and nDCG@10 to 0.75 (dev 0.64→0.73 recall@20, held-out 0.58→0.71), for $0.0065 and about 3 s per search. The same reranker over chunks reaches recall@10 0.45 and nDCG@10 0.47. On dev, Phase 1 as a whole moves recall@20 by +9 points (+13 held-out): short of the +20-point gate, so reranking is built but off by default until the end-to-end coverage check decides.

Equal k is not equal text. Ten sections can hold ten times the text of ten chunks, so recall@k flatters a section retriever. Context recall at an equal reading budget (each system walked in its own units until the budget is spent; a unit that does not fit is skipped) removes that advantage: at 16,000 characters the reranked chunk RAG finds 0.87 of the facts' evidence and the reranked chunkless retriever 0.83; at 4,000 characters 0.72 and 0.76; at 64,000, 0.96 and 0.92. Sections still rank better at the top (MRR 0.97 against 0.64), but long sections spend budget on text the question does not need. At depth, retrieval is not the bottleneck: 94% of the required facts have evidence in the chunkless top 50.

6.2 Where answers lose facts

Retrieval improvements barely moved answers. To see why, every required fact of four engine runs was followed through the stored trace (scripts/eval/crossdoc/loss_waterfall.py): was its gold evidence among the candidates (retrieved), was it opened (read), did the answer state it (the judge), and if not, did a sentence stating it exist and get removed after writing - by the claim checker, or by the display gate that shows only exactly checked statements - or was it never written. The traces gained the gate's removals and the writer's raw text on 5 October; earlier runs cannot separate gate losses from omissions.

Three findings. (1) Retrieval is nearly saturated where it runs: the reader's loop has the evidence of 92–99% of the facts among its candidates; reranking moves neither reading nor coverage (two-run noise is ±0.05). (2) Reading loses 14–20 points in the loop: candidates found but not opened under the per-part budget. (3) Writing loses most: with everything read, a fifth to a quarter of the required facts are never written, and the display gate removes a further tenth (tables whose cells were not checked one by one, worked examples, headings), while the checker itself removes almost nothing. This agrees with reports that retrieval gains do not carry to answers when the generator is the bottleneck [27], that readers saturate long before retrievers' recall [16], and that answering inside a JSON format degrades generation [28] - our writer answers in JSON. At the time of these runs the whole-scope route was enabled for the evaluation workspace only and user workspaces ran the reader's loop (0.58 / 0.49, below the conventional baseline's 0.63 / 0.59); since commit 214876b every staging workspace answers through the whole-scope path (§6.10).

6.3 Cross-document set

Table 5 gives every configuration measured on the 34 questions. Arms: A, a single long-context gpt-4.1 call holding every section of the shelf (an upper bound, no retrieval, no checker); C, a prototype of compiled knowledge (per-document briefs and cross-document topic dossiers built once per shelf, one writer call and a checker); D, the engine's research and full modes with a gpt-4o writer; B, the same engine with gpt-4.1 for synthesis; W, full mode given every section of a small scope.

Where the loss is. Given every section of the shelf, the model reaches 0.95 held out. Our research mode loses about 0.23 of that with gpt-4o and 0.14 with gpt-4.1; full mode loses 0.45–0.60. Traces of ten misses showed that in seven the engine never opened the document that held the answer. The limit is what the model is given to read, not its ability to connect what it reads.

The writer model matters. On identical compiled inputs, replacing gpt-4o with gpt-4.1 raised held-out coverage from 0.71 to 0.85 (+0.14, run spread ≤0.04), at a lower price per token. In the engine, the same swap raised research mode from 0.72 to 0.81 and full mode from 0.35 to 0.50 held out, at a large cost and latency for research mode.

The ceiling does not scale. Arm A is feasible because these shelves fit in one context. A 532-page book (1,432 sections) or a workspace of thousands of documents does not, and attention over very long inputs degrades [16]; A is an upper bound, not a design.

6.4 An honest denominator

Our earlier reports said answers were “100% verified”. That was 100% of checked claims. Counting displayed sentences without an inline citation in saved answers, and re-judging with those sentences removed (rescore_cited.py), gives Table 6. The count is a heuristic (headings, table rows and short fragments are skipped) and was taken on saved answers only.

Every advantage we had reported shrinks by about 0.1 under the strict rule. This led to the displayed = checked invariant (§3.4).

6.5 Regulatory and blind sets

Table 7 summarizes the graded runs. Both sets were preregistered, but fixes were made between runs on the same questions, so later rows are development performance; each miss was turned into a general rule with a test rather than a question-specific patch.

The first blind run on the CC BY 4.0 legal code produced four confidently wrong legal verdicts from a document that arrived as a single 18k-character section; the fixes (continuing the main evidence for every kind, the checker seeing the slices actually delivered, judge verdicts treated as leads unless closed and stated) were general and removed them.

6.6 Stress set

On 63 questions over three corpora the engine was right and cited on 44 (median 12 s, 90th percentile 35 s, 3.5 model calls per question, no HTTP errors). Failures, ranked: seven false “not stated” (two-document obligations bound to one document, headings that do not name their content, chapter-size sections clipped), two wrong documents, one fabricated sentence with a citation, one self-contradiction, conversational referents (“what about BERT”, “the other paper”), requested shape or language not honoured, computations done by reading, and right answers labelled partial. On a rerun of 55 questions, a gpt-4o writer verified 92% of claims but 14 answers were blocked by the account's rate tier; a gpt-4o-mini writer was never blocked but verified 43–44% and conflated similarly named models.

6.7 Ingestion

6.8 Answer composition: case studies

These are single-question engineering observations, not evaluations. (1) Asked “What is retrieval-augmented generation?”, the engine first produced one dense paragraph with undefined terms and no figure ($0.034); with a per-part instruction to build an explanation for a newcomer and to introduce each figure in words, it produced a definition, the two components, how they work together, an example, and the paper's Figures 1 and 2 placed after the paragraphs that introduce them ($0.093). (2) A how-to question over the GNU Make manual returned eight ordered steps after removing two machine-generated appendices from explanation answers ($0.105 to $0.046). (3) A user's “What is a neural network?” over a 532-page book had sentences removed because the claim checker's reply was truncated at 1,600 tokens on 25 claims; unreached claims were treated as unsupported. The checker's budget now scales with the number of claims and unreached claims are re-checked. (4) On the development corpus (111 documents, 5,185 sections), 14 sections were longer than the 24,000-character embedding cut, leaving 361,032 characters unreachable by dense search; the longest section spans 108 passages. Re-embedding the corpus as 7,142 headed passages took 8.4 minutes serially, about 1.9M tokens ($0.04 at text-embedding-3-small prices). Its effect on retrieval recall is not yet measured; it awaits the gold-evidence harness of the test plan. On the staging corpus (129 documents, 11,024 sections), 4 sections exceeded the cut (39,270 characters unreachable); after re-embedding, all 11,024 sections are covered by 12,791 passages.

6.9 Writer experiments

Since writing loses most facts (§6.2), six writers were compared offline on the same input the product reads (staging's parse of each shelf), the 17 development questions, and the same blind judge, writer only, no checker (scripts/eval/crossdoc/exp_writer.py): V0 today's whole-scope prompt with JSON output; V1 the same prompt in markdown [28]; V2 quotes first, then the answer, in one call; V3 per-document notes of exact quotes, a task list of the points a complete answer needs, a draft, and a coverage check that writes what the draft misses; V5 V3 with each document's brief and the shelf's cross-document dossiers; V6 every question over all five shelves as one workspace of 32 documents, the briefs choosing the documents, then V5. Each is one run (two-run noise ±0.05); total cost $27.

Results (dev coverage). The one-call long-context ceiling stays best: 0.92 (two runs, 13–18 s, $0.04–0.11). V5 0.909, V0 with instructions before the documents 0.905, V6 0.904, V3 0.902 (24–55 s, $0.21–0.35, answers twice as long); single calls with the instructions after the documents 0.85–0.86. Within the noise, the multi-step writers do not beat a single long-context call on scopes that fit; V6 shows the multi-step writer keeps that level when the workspace is too large to read: it read 3.5 of 32 documents per question, the documents holding the gold evidence were chosen for 94% of facts, and coverage held at 0.90.

What the harness taught. The first V3 scored 0.68, for reasons of the harness, not the design: the note-taker was allowed to answer “NONE” and did so for the paper the question named, and the coverage check was asked to return the draft unchanged and returned a comment instead. Instruction placement mattered as OpenAI's GPT-4.1 guide states [29] - above the documents beats below (0.905 against 0.864 for the same JSON prompt), documents as XML - and the opposite of advice written for another model family. JSON against markdown made no measurable difference here.

Where the remaining misses are. Of the 103 development facts, 3 were missed by all six writers, 10 by four or more, and 26 by at least one: what any writer misses, another states. The facts every writer missed are chains across three documents (a calibration table gives no interval, so a work instruction falls back to 12 months, while the risk file requires 6). The headroom is consistency, and multi-document chains.

6.10 The product path, on staging

The findings were built into the engine (commit 214876b) and run on staging with the checker and the display gate, all 34 questions, one run each: one reading of the whole scope when it fits; when it does not, one reading of 400,000 characters filled with the most relevant whole documents and then the most relevant sections of the rest (hybrid search, 300 deep, in the documents' order); uncited sentences beside cited ones and table cells given the section that states them; headings and lead-ins kept as structure when a deny-list of asserting words found none in them (see the correction below). The path is on for every staging workspace; before, user workspaces ran the reader's loop.

Results. Per shelf: 0.77 dev / 0.80 held-out ($0.21, 24–32 s). All 32 documents (1.5M characters) as one workspace: 0.79 / 0.79 ($0.26, 33 s); the selected reading held the gold evidence of 192 of 194 facts (0.99), about 346 of 1,063 sections per question. Every displayed sentence and table cell was verified; headings and lead-ins were not (correction below). Against the reader's loop the same questions move from 0.58 / 0.49 to 0.79 / 0.79, and the very long workspace answers as well as a single shelf. The display gate now removes 4–5% of required facts (10% before), the checker 2%; 19–22% are not fully written.

The remaining gap. The writer alone reaches 0.90–0.92 offline (§6.9). Of the statements the writer makes, the gate shows 57–60% (46% before): the rest are table cells that summarise several source sentences and paraphrases whose verified claim was restated in the documents' words, so no exact checked claim matches them. Closing it means checking a summary or a paraphrase by entailment against its cited sections rather than by exact match - a trade-off between coverage and the exact-match guarantee, to be measured on the adversarial set before it is adopted.

Correction: headings. A review after these runs wrote ten headings that state a fact (“Warfarin replaced by apixaban”, “Self-RAG outperforms RAPTOR”) and found that the deny-list let all ten through: a fact needs no listed verb, so an unchecked claim could reach the reader as a heading. The gate now keeps a heading or lead-in that is not a fixed label only when every content word in it occurs in the checked text kept under it and it holds no digit; all ten are removed, and headings such as “Increased risk of bleeding” over a checked sentence stating it stay. The rule matches words, not meaning (a reordered heading over the same words would pass); entailment would close that too. The coverage numbers above were measured before the change and have not been re-run: removing more headings can only remove structure, not checked statements.

6.11 Entailment instead of exact match

Would checking a removed statement by entailment against the sections it cites - rather than by exact match to a verified claim - recover what the gate removes, and at what cost to safety? Offline, on the 68 saved staging answers of the product path (per shelf and the 32-document workspace), no answer generated again (scripts/eval/crossdoc/exp_entail.py, 2026-10-05).

What is removed. The writer's whole text (kept in the trace) against the answer shown: of 2,377 written units (sentences, headings, lead-ins, table cells), 993 (42%) were not shown. 622 of them cite a section: 332 sentences (37 rejected by the claim checker, 295 by the gate), 140 table cells, 90 headings, 60 lead-ins. The other 371 cite nothing and cannot be checked against anything.

Method. Each cited removed unit is put to a model with its cited sections in full (grouped per answer and cited set): “supported” only when every name, number, condition, direction of a relation and scope is in the sources; it must quote the supporting text, and a unit is accepted only when every quoted piece stands verbatim in those sections (pieces split at ellipses, table bars and sentence ends). Safety: 300 verified statements the reader saw were each turned false one way by gpt-4.1 - negation, a number, an entity, a reversed relation, a widened scope, an added detail - 298 usable; the false versions and the true originals were checked in separate calls with the originals' citations.

Results. Both models accept about half of the removed units, and about 7% of the false statements. Read one by one, the 23 false statements gpt-4.1-mini accepted are: 5 in fact true (the generator failed: None | X is valid syntax, “abstraction” and “granularity”), about 8 with a minor unsupported addition (“especially in cross-origin requests”), and 10 that change the meaning - “warfarin should always be discontinued and apixaban started regardless of the INR” (the label: when the INR is below 2.0), “start both a parenteral anticoagulant and warfarin after that time”, “INR affects rivaroxaban”, “the server may request a new access token”. Requiring both models to accept leaves 15 of 298 false accepted and restores 281 of 622; requiring in addition that the statement's content words be in its quotes (80%) leaves 3 of 298 false accepted but rejects true paraphrases too (71 of 298 originals accepted, 56 of 622 units restored). Among the restored units is a sentence that contradicts itself (“at least 5 complaints … for flow rate deviation (… C-2025-0481 is for occlusion alarm)”), accepted because each piece is in the log.

Effect on answers. Restoring every unit gpt-4.1 accepted and judging blind with the same judge: per shelf 0.77 to 0.82 (dev) and 0.79 to 0.83 (held-out); 32-document workspace 0.81 to 0.84 and 0.79 to 0.83 (the originals as the mean of two judgements; the same answers judged twice differ by 0.00–0.03). About +0.04, at about $9 for the whole experiment (model calls $6.3 metered, judging about $3).

Reading. Entailment recovers half of what the gate removes but buys +0.04 coverage with a different guarantee: about one adversarial false statement in fourteen passes, a third of them changing the meaning, several of them about drug dosing. Exact match stays the display check. The gap to the 0.90–0.92 the writer reaches offline is not in the gate: with every accepted unit restored, coverage is 0.83, and the larger loss is what is never written (19–22% of required facts). A safer use of the same signal - asking the writer to restate an accepted unit in the documents' words so that the exact checker verifies it - is not measured.

6.12 Ledger links for the writer

The whole-scope writer is given each section's document title, page and heading path, and its text; the ledger's links reached the answer only afterwards, as flags about cited sections (7–8 of 34 answers). We gave the writer the ledger as well (commit 4927ae1, switch writer_links): a list of the documents with their identifiers, class, effective date and which documents supersede parts of them, and under every section its resolved references (marked when the target is outside the reading), what refers to it, supersessions and possible conflicts; the prompt says these are pointers, not evidence. Code only, no model call. Two runs on the 32-document workspace (linkB1, linkB2), same questions, judge and engine otherwise as authB.

Results. The writer was given a median 222 link lines per answer (at most 378), on 70 of about 338 sections read. Coverage fell: dev 0.78 and 0.77 against 0.81 (authB, mean of two judgements), held-out 0.69 and 0.72 against 0.79 - below the noise band on held-out in both runs. Answers took 72–83 s against 33 s (two runs sharing the server in both cases); cost per answer was $0.24. Not adopted: the switch stays off. Every link the scope holds, unranked, is mostly not about the question; our reading is that it spends the writer's attention on cross-references rather than on the facts asked for, which agrees with the loss being in what the writer leaves out (§6.2). Links chosen for the question - the references and conflicts of the sections that bear on it - are a different experiment and are not measured.

6.13 What the trace found in linking

Recording why a reference stays unresolved (the explorer's “not linked, and why”) exposed a resolver gap at once. In the GNU Make manual 110 of 578 references were open, 101 of them as “no document matches”; most named a section of the manual itself (“see Phony Targets”, “variables from the environment”), classified as references to another document and looked up among document titles only. Linking a reference to the section whose heading names it - every content word of the reference in the heading, at most three other words, the citing document first, ambiguous and self-references refused - resolved 77 of them (open 110 to 34); every one of a sample of twelve was correct, and two other corpora gained no link (no false matches). No model call is involved; existing documents are re-linked in place. Decisions not to link two names (the alias judge's rejections and the code's vetoes of similar names) are now kept with their reasons instead of discarded.

6.14 Linking documents that do not name each other

A reader's report (2026-10-13) was that uploads which continue one another were not linked. Two causes were found: a document was linked only when one named the other, and on a shelf too large to read at once the selection was made from the question's words, so the volume that joins two others, which rarely uses those words, was never read. Three changes followed. First, names in the titles give series order (Part 2 continues Part 1, Rev D is newer than Rev C, a gap is only “maybe the same series”), and so does an upload's own sequence number. Second, two documents are linked as talking about the same things when their text uses the same rare names. The names are read from the sections' text, not only from extracted statements. A name must contain a word the workspace never writes in lower case, and a shared name links a pair only when sharing that many names is rarer than chance. Third, before writing, the reader follows one hop to passages about the same people and things, inside the reading's spare budget. No series link joins two documents whose titles name different people.

Bridge check (synthetic, five story volumes padded to 160,680 characters; the rule in Volume 1, the new name in Volume 3, the event in Volume 5; the question names only Volumes 1 and 5; two runs each; paper/data/bridge_runs.json). Premise recall went from 0.5 to 1.0 on the question whose wording missed Volume 3, and from 0.75 to 1.0 mean over both questions. At a 25,000-character reading the hop's room was too small at first (0.5 before and after); reserving 15% of the selection for the hop gave 1.0 on both questions, with Volume 3 cited in all four answers. Spend: about $1.65.

Names from the text (no model calls; the three workspaces above). Extraction had skipped the sentence “took the name Wren” in one workspace and worded the shared key differently in all three, so statement matching linked Volume 3 to Volume 5 in none of the three. Reading the names from the text linked it in 3 of 3, in 0.16–0.23 s per workspace. At 2,000 synthetic documents (about 80 capitalised names per section) the chance rule keeps 94,969 links where names without it give 430,208. One upload costs 4.85 s to index and re-link, and 400 random name-heavy logs link 149 of 79,800 pairs. On staging (2026-10-14) re-linking nine workspaces took 100 s and raised the links from 3,767 to 5,585. Limits: two questions on one synthetic shelf are an existence proof, not a rate; linking accuracy on real corpora is n/m.

Reading pairs side by side (staging, 2026-10-14/15; paper/data/pair_reading_runs.json; dry runs unless stated). Chance cannot tell one character called Wren in two volumes from two unrelated papers that cite an author called Daniel. In a reader's mixed 57-document workspace (papers, history articles, licences, standard operating procedures) the statistics alone first drew 434 links, then 95 after the ordinary-word and size rules. We now treat every pair that shares a specific thing as a candidate (241 pairs there), and a model (gpt-4o) reads both documents before they are linked. The first version read one passage per document, the section with the smallest identifier that named the rarest shared thing, and cut the link when that passage only mentioned the shared thing in passing. It removed 52 of the 95 chance links, and we first reported that every removal we checked was right. That was wrong. We had checked whether each passage was a passing mention, and by construction it was. We had not checked whether the documents belonged together. A gap-finding agent (§6.16) showed that 19 of the 52 removals joined documents our own questions treat as one topic. ReAct lost its links to the other language-model papers on two acknowledgement sections; the JWT and bearer-token standards were split on two registry tables; 10 of 21 typing-proposal pairs were cut on their header fields. The reader now picks, for each document, the two sections that say most about the shared things (each weighted by how few documents use it) and shows each document's opening. A link is cut only when the documents themselves are judged to be about different subjects. Scored against labelled pairs in that workspace (77 that belong together: the typing proposals, the OAuth standards, five language-model papers, five software licences, one company's quality documents, one handbook; 138 that join different labelled topics), the first reader linked 69% of the must-link pairs and 2% of the must-not pairs. The revised reader links 88% and 4%. Four of its six must-not links join the GNU Make manual to software licences, which the manual contains; two join documents that only both mention Linux. It still splits the language-model papers, whose shared names are mostly the works they cite. The cheap model (gpt-4o-mini) was not enough: it called 48 pairs that only name the same countries related. Cost: $0.54 for 241 pairs, 150 s, about $0.0022 a pair. Each reading is stored with what the reader was shown, keyed on the two documents' texts (no longer on the passages, which moved with every upload because each shared thing is weighted by how many documents use it), so a re-link reads only pairs whose text changed. Limits: the labels are ours, made from the topics of the evaluation questions, on one workspace; one run per reader.

Citations by full title (staging, 2026-10-15, live, no model calls). The same agent then found that no paper in the corpus was ever linked to a paper it cites: an author-year citation (“Yao et al., 2022”) was compared with document titles, which never hold authors' names, and ReAct had no link of any kind on the map. A document whose text carries another document's full title (four words or more, compared as letters and digits only, so a title broken across lines in a reference list still matches, and only as whole words) now cites it, a link of its own kind that the subject reader cannot cut. On the question that asks which of the five language-model papers cite each other (gold P6), the re-linked workspace has both required links (Tree of Thoughts and MemGPT cite ReAct) and none of the excluded ones (Self-RAG and RAPTOR do not). Across the ten staging workspaces the re-link made these links without a model call; the full re-link with the revised reader took 1,298 s and moved the link count from 4,216 to 4,323. Limits: a short title (“Climate change”) is never treated as cited, so citations of short-titled documents are still missed.

6.15 Judgment questions and a quote-checked grader

Questions that ask for a judgment (“was the report on time?”, “is the residual risk still acceptable?”) lost their answer to the display check: the documents state the rule and the dates, never the conclusion in words, so the conclusion was removed and the answer still read as complete. The engine now asks the writer for the verdict apart, with the sentences it rests on, and shows it first, labelled derived from the documents, only while every premise stands checked. Day counts are made by code from checked dates, from each earlier checked date to the event the question names, with a stated limit applied, and the verdict may rest on them. A premise the second look restates in the document's words keeps the verdict. When the verdict cannot be shown, the note names the fact that failed, and no such note appears when a checked sentence already gives the conclusion.

The grader had to change first. A gap-finding agent (§6.16) found that the blind judge credited answers with conclusions they never write. It gave an answer with no submission date, no day count and no verdict full marks, quoting “41 days, 11 days late”. The judge now quotes the answer for every item it credits; code checks that the quote stands in the answer and that the item's numbers (not its document codes) stand there too (quoted-5). Older runs were re-graded with it, and no score in this subsection is compared with an older grade.

Proof (2026-10-14; five questions: N3, N5, N9, M1, P6; two runs each on staging; graded by quoted-5; paper/data/judgment_runs.json). Mean coverage was 0.605 for the build before the verdict layer (two runs re-graded), 0.53 for the first verdict layer, and 0.718 for this build. One leak remains: the grader credited N3's “the Manual governs, so it was late” from the count line, while the answer's verdict said the opposite. Corrected by hand to 0.8, the mean is 0.698. Answers labelled complete scored 0.78 against 0.655 for the rest (five each). On N3 the submission date, refused on 2026-10-13, was confirmed in both runs, and the second run showed 41 days from receipt (11 over the limit) beside 27 from recording (within). Both runs still concluded “on time”, counting from recording as SOP-12 does, where the Quality Manual governs and counts from receipt: a wrong verdict that passed every check. Writing a correct verdict, and not only a checked one, is the open problem here. Limits: five questions by two runs, with a run-to-run spread of about 0.14, show a direction, not a rate. Spend: $1.52 for the answers.

6.16 A gap-finding agent

Since 2026-10-13 a separate agent reads stored answers, traces and code, and writes up what keeps an answer from what the reader needed, with a reproduction and a proposed fix. It does not change code; the builders fix and measure. In two days it registered eleven gaps. Each was confirmed by the builders before any fix. The measured ones, before their fixes:

• The checker deleted true facts and the answer still read as complete. In a hand sample of about 100 removed statements, about 40 were faithful facts the documents state. The fix gives a refused sentence a second look in the document's words, and the checker sees the part of the section it came from with table rows whole. Not yet measured on answers.

• A re-check after that second look forgot what the first check had confirmed. One answer went from 12 confirmed sentences to 8 of 46. Confirmed claims are now carried forward, and only an exact copy skips the checker.

• “vs.” and “e.g.” split sentences before checking. Replaying 40 stored drafts gave 6 splits in 4 answers before the fix and 0 after. One shown comparison had paired a figure with the wrong number.

• Table cells whose source sat in a “Source” column or a sources line were blanked unchecked: 78 and 97 cells in two runs of 20 answers. They are now checked, and removals are stated (“1 table removed”). Not yet measured on answers.

6.17 Comparing versions of a form

A customer-shaped task outside the question sets: given two years of each person's tax return, or two invoices from one supplier, say what changed, and do it for many people at once. The documents are synthetic (invented names, identification numbers in never-issued ranges): four pairs of individual returns in US, Canadian and Australian layouts, three pairs of supplier invoices, one return of a second person with a near-identical name (a trap), and two of the fifteen PDFs scanned, so they go through OCR. A generator writes the truth with the documents: 71 changed fields or lines. Each pair is asked “What changed between …?” with only its two documents in scope, and two batch questions put all returns, or all invoices, in scope at once. The writer is gpt-4.1 on the whole-scope path; gpt-4o judges each true change as stated or not, and lists statements in the answer that are false.

Reading. Every value of every return was in the parsed text, both scans included (125 of 125). The invoice checker missed one line per invoice, the tax line written “Tax (13%)” where the truth file writes “13% = $126.36” - a checker format, not a reading loss.

The first run found a display bug. Two pairs scored 0.08 and 0.09 while the writer's own text (in the trace) had every change in a correct table. The claim checker removes an unsupported claim's text from the answer; on a table row that cut across the cell bars and glued the rows into one (a fifteen-row table of changes became one row of dashes), and the sentence filter cut table units the same way. A claim on a table row is now marked unverified and the display gate blanks that cell alone; the row stays whole. Table 10 is one run before and one after (single runs; the cross-document noise band is ± 0.05).

What the judge called false. Twelve statements after the fix, none a wrong value: nine restate a value that did not change (“the standard deduction remained $22,500”, the filing status, an address; three of them unchanged invoice lines in the batch answer), two are correct rows cited to a section labelled “(anonymous)” (sections with no title), and one reads a removed line and an added line as a replacement. The person with the near-identical name was kept apart: the batch answer says their second year is not in the documents. Open: the change column is written by the model and should be computed and checked by code (the ledger's computation path), and untitled sections need a label from their document.

A full score that was not a usable answer. A tester asking the batch question on staging got every value right and no person's name: four people's changes ran together as one list. The writer had written each name as a heading (“Jordan Ellis (SSN 000-41-4417)”); the display check removed them, because a heading had to restate the checked lines under it, and those lines never repeat the name - and the coverage judge, which looks only for the values, scored the answer 1.00. Three code checks followed: a heading that names stays when every word and number of it stands in the documents the lines under it cite (an abbreviation may stand for the initials of their words; an asserting word, or any word or number the documents lack, still removes it); a “Change” cell stays when the code computes the same signed difference from the first and last values of its row, or “added”, “removed”, “changed” as the row's empty sides say (a writer's difference is never in the documents, so every such cell had been blanked, and a model's arithmetic is not trusted); and a sentence that moves a value “from A to B” is removed when its direction word contradicts A and B (staging showed “Total tax increased from $2,646.00 to $2,113.00”, both values in the returns). The writer is told to give each person or thing asked about its own named part and a table per comparison. The first run with the new instruction lost invoice line items (mean 0.93; one batch question 0.68): “one row per item that differs” was read as the header fields and totals. Naming line items, and a line in one version only as added or removed, restored them: 1.00 on all nine questions, every person and supplier named, every change cell computed and passed by the code, 482 of 482 claims verified then, 458 of 458 now. Of the 21 statements the judge still calls false, 8 are correct differences it cannot match (the truth file lists values, not differences), 11 are unchanged rows the writer kept despite the instruction, and 2 read a renamed line as one removed and one added - none is a wrong value. Single runs.

7 Conventional RAG Baseline

Both halves are now measured on the 34 cross-document questions (Tables 2 and 3), from staging's own parse of the shelves: documents split into 512-token chunks (2,000 characters, 250 overlap) with structure ignored; hybrid retrieval (dense and BM25 fused by reciprocal rank); the top 16 chunks to one gpt-4.1 call told to answer completely from the context and cite chunk ids; no checker; the same blind judge; two runs; a second variant puts the listwise reranker in front (scripts/eval/retrieval/answer_baseline.py).

Result. Coverage 0.63 dev / 0.59 held-out (reranked: 0.70 / 0.65), at 2–5 s and $0.016–0.025 per answer. Faithfulness by a RAGAS-style judge (claims split from the answer, each checked against the chunks it was written from, gpt-4.1-mini) is 1.00: the baseline does not invent, it omits. Re-measured on current code (5 October, inside staging, same judge): the engine's whole-scope full mode covers 0.78 / 0.72 (three runs, 23 s, $0.15, every displayed claim verified) and beats it; the reader's loop that user workspaces ran until 214876b covers 0.58 / 0.49 (one run each with and without reranking; 40–62 s, $0.20–0.26) and loses to it. Faithfulness figures are not directly comparable across methods: the engine's come from its verbatim span checker, stricter than an LLM judge. §6.2 locates the loss.

The protocol as first written, kept for the record: documents split into 512-token chunks with 64-token overlap from the same parse; hybrid retrieval (BM25 and dense, reciprocal-rank fusion) with k in 8, 16; a single gpt-4.1 answer call instructed to cite chunk identifiers; no checker. A second variant adds contextual chunk headers [9]. Estimated cost: a few dollars. Its results will fill the last row of Table 5.

8 Limitations

The question sets are small (34, 30, 42, 63) and written by the system's developers; coverage on 17 questions moves by up to 0.1 between runs of identical code. The judge is a single model (gpt-4o) and may favour answers in its own style [21]. Only OpenAI models were measured. The regulatory and blind sets were iterated, so only the 17 held-out cross-document questions are a clean held-out measure. The uncited-sentence count is a heuristic. The latest build (page windows, answer composition, claim-check budget) has not yet been re-measured on the cross-document set. Cost figures reflect 2026 prices.

9 Ethics and Reproducibility

All evaluation corpora are public (IETF RFCs, FDA labels, arXiv papers, Python PEPs, public regulations) or fictional (the northwind quality system, generated by a script). Questions, must items, run labels, answers and judgements are stored with the code; Table 5 is recomputed from them by a script. Raw model prompts and responses are not retained for customer data; captures exist only for approved evaluation workspaces or consented diagnostic reruns.

Acknowledgements. Parts of the system and of this document were produced with AI coding assistants under the author's direction; all numbers were taken from stored outputs and checked against them.

A Run labels

References

1. Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal et al.. Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks. Advances in Neural Information Processing Systems. 2020.
2. Kelvin Guu, Kenton Lee, Zora Tung, Panupong Pasupat, Ming-Wei Chang. REALM: Retrieval-Augmented Language Model Pre-Training. Proceedings of the 37th International Conference on Machine Learning. 2020.
3. Vladimir Karpukhin, Barlas O, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov et al.. Dense Passage Retrieval for Open-Domain Question Answering. Proceedings of EMNLP. 2020.
4. Gautier Izacard, Edouard Grave. Leveraging Passage Retrieval with Generative Models for Open Domain Question Answering. Proceedings of EACL. 2021.
5. Stephen Robertson, Hugo Zaragoza. The Probabilistic Relevance Framework: BM25 and Beyond. Foundations and Trends in Information Retrieval. 2009.
6. Gordon V. Cormack, Charles L. A. Clarke, Stefan Buttcher. Reciprocal Rank Fusion Outperforms Condorcet and Individual Rank Learning Methods. Proceedings of SIGIR. 2009.
7. Tianyu Gao, Howard Yen, Jiatong Yu, Danqi Chen. Enabling Large Language Models to Generate Text with Citations. Proceedings of EMNLP. 2023.
8. Sewon Min, Kalpesh Krishna, Xinxi Lyu, Mike Lewis, Wen-tau Yih, Pang Wei Koh et al.. FActScore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generation. Proceedings of EMNLP. 2023.
9. Anthropic. Introducing Contextual Retrieval. ://www.anthropic.com/news/contextual-retrieval. 2024.
10. Parth Sarthi, Salman Abdullah, Aditi Tuli, Shubh Khanna, Anna Goldie, Christopher D. Manning. RAPTOR: Recursive Abstractive Processing for Tree-Organized Retrieval. International Conference on Learning Representations. 2024.
11. Darren Edge, Ha Trinh, Newman Cheng, Joshua Bradley, Alex Chao, Apurva Mody et al.. From Local to Global: A Graph RAG Approach to Query-Focused Summarization. 2024.
12. Bernal Jimenez Gutierrez, Yiheng Shu, Yu Gu, Michihiro Yasunaga, Yu Su. HippoRAG: Neurobiologically Inspired Long-Term Memory for Large Language Models. Advances in Neural Information Processing Systems. 2024.
13. Harsh Trivedi, Niranjan Balasubramanian, Tushar Khot, Ashish Sabharwal. Interleaving Retrieval with Chain-of-Thought Reasoning for Knowledge-Intensive Multi-Step Questions. Proceedings of ACL. 2023.
14. Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan et al.. ReAct: Synergizing Reasoning and Acting in Language Models. International Conference on Learning Representations. 2023.
15. Akari Asai, Zeqiu Wu, Yizhong Wang, Avirup Sil, Hannaneh Hajishirzi. Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflection. International Conference on Learning Representations. 2024.
16. Nelson F. Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni et al.. Lost in the Middle: How Language Models Use Long Contexts. Transactions of the Association for Computational Linguistics. 2024.
17. Ziyan Jiang, Xueguang Ma, Wenhu Chen. LongRAG: Enhancing Retrieval-Augmented Generation with Long-context LLMs. 2024.
18. Zhuowan Li, Cheng Li, Mingyang Zhang, Qiaozhu Mei, Michael Bendersky. Retrieval Augmented Generation or Long-Context LLMs? A Comprehensive Study and Hybrid Approach. 2024.
19. Shahul Es, Jithin James, Luis Espinosa-Anke, Steven Schockaert. RAGAs: Automated Evaluation of Retrieval Augmented Generation. Proceedings of EACL: System Demonstrations. 2024.
20. Jon Saad-Falcon, Omar Khattab, Christopher Potts, Matei Zaharia. ARES: An Automated Evaluation Framework for Retrieval-Augmented Generation Systems. Proceedings of NAACL. 2024.
21. Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang et al.. Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena. Advances in Neural Information Processing Systems, Datasets and Benchmarks Track. 2023.
22. Christoph Auer, Maksym Lysak, Ahmed Nassar, Michele Dolfi, Nikolaos Livathinos, Panos Vagenas et al.. Docling Technical Report. 2024.
23. Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Thomas L. Griffiths, Yuan Cao et al.. Tree of Thoughts: Deliberate Problem Solving with Large Language Models. Advances in Neural Information Processing Systems. 2023.
24. Charles Packer, Sarah Wooders, Kevin Lin, Vivian Fang, Shishir G. Patil, Ion Stoica et al.. MemGPT: Towards LLMs as Operating Systems. 2023.
25. Nandan Thakur, Nils Reimers, Andreas Ruckle, Abhishek Srivastava, Iryna Gurevych. BEIR: A Heterogeneous Benchmark for Zero-shot Evaluation of Information Retrieval Models. Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks. 2021.
26. TREC: Experiment and Evaluation in Information Retrieval. MIT Press. 2005.
27. Mathur, Rittner, Thakur, Schiff, Islam. Retrieval Improvements Do Not Guarantee Better Answers: A Study of RAG for AI Policy QA. arXiv preprint arXiv:2603.24580. 2026.
28. Zhi Rui Tam, Cheng-Kuang Wu, Yi-Lin Tsai, Chieh-Yen Lin, Hung-yi Lee, Yun-Nung Chen. Let Me Speak Freely? A Study on the Impact of Format Restrictions on Performance of Large Language Models. Proceedings of EMNLP: Industry Track. 2024.
29. OpenAI. GPT-4.1 Prompting Guide. ://developers.openai.com/cookbook/examples/gpt4-1_prompting_guide. 2025.