# Proofdesk

> A RAG that refuses to guess: every answer cited to the exact passage, or an honest refusal instead.
>
> https://pravda.systems/work/proofdesk · 2026-07-12

**Problem.** Retrieval chatbots answer whether or not the documents support them, and a wrong answer looks exactly like a right one. Anyone who has to stand behind the answer cannot tell the two apart.

**Solution.** A fail-closed RAG that cites every sentence to the exact passage, has a separate cross-vendor verifier check each claim against the evidence, and refuses rather than ship anything unproven.

**Outcome.** 89.58 percent grounding on a 60-question golden set with every shipped answer grounded, 100 percent correct refusal of 12 unanswerable questions, and a measured 10.42 percent false-refusal cost, which is the price of never guessing.

## Key metrics

- **89.58%** Grounding rate (golden set v3) — 43 of 48 answerable questions on a 60-question golden set, and every answer that shipped was grounded in the document it cited (43/43). This replaces an earlier 91.67% from runs #24/#25: those predate the entitlement gate and the eval runner was silently unauthenticated behind it, so run #35 is the first one that actually measured production. _(verified: eval run #35, eval_runs table, golden_sha 91b9bdb0, 2026-07-17; app-proofdesk docs/METRICS.md + DECISIONS.md 2026-07-17)_
- **100%** Refusal correctness — All 12 unanswerable questions refused. The fail-closed floor never let an unanswerable question through. _(verified: eval run #35, refusalCorrectness 12/12; app-proofdesk docs/METRICS.md)_
- **10.42%** False-refusal rate — The measured cost of refusing to guess. Of 48 answerable questions, 5 were declined: 4 genuinely too conservative, 1 a transient fail-closed refusal when the pipeline was unavailable, which is the system working as designed. _(verified: eval run #35, per-question refusalReason decomposition; app-proofdesk docs/METRICS.md)_
- **342 docs, 4,400 passages** Corpus indexed — Documents read, chunked, and embedded into the live retrieval index. _(verified: GET /api/stats {documents:342, passages:4400}, live 2026-07-12; cross-checked in app-proofdesk docs/GO-PUBLIC-CHECKLIST.md)_
- **7 / 8** Multi-hop grounding — Cross-document questions that plain vector search would miss, answered by a one-hop graph expansion over the corpus. Above the 6/8 bar. _(verified: eval run #35, the multi-hop subset; app-proofdesk docs/METRICS.md 2026-07-17)_

## The problem

Retrieval chatbots guess. Ask one about your own contracts or policies and it will produce a fluent answer whether or not the documents support it. The wrong answer looks exactly like the right one. For anyone who has to stand behind the answer, a lawyer, an ops lead, a support desk, that is worse than no answer, because there is no way to tell the two apart. The number that matters is not how often the tool answers. It is how often it is wrong when it sounds sure.

## What I built

Proofdesk is a RAG that refuses to guess. It answers only from the documents you give it, and every sentence is cited to the exact passage it came from. Before an answer is allowed out, a separate verifier from a different vendor checks each claim against the retrieved evidence. If any claim is not supported, the answer never ships. The desk returns an honest refusal instead. When the verifier itself is unreachable, the desk still refuses rather than fall back to an unchecked answer. Failing closed is the whole product, not a safety feature bolted on the side.

## How it works

The pipeline runs embed, retrieve, draft, verify, gate, store. A question is embedded with OpenAI text-embedding-3-large and matched against 4,400 passages in Neon Postgres. A one-hop graph expansion pulls in connected documents, a newer version or a referenced contract, that plain vector search would miss. Claude Opus drafts an answer with inline citations. Gemini, a different vendor, verifies each claim against the evidence and returns a per-claim verdict. A hard gate blocks the answer unless every claim is proven, then stores the full run: the passages found, the draft, each verdict, the models, and the timings. Every number the site reports traces back to one of those stored runs.

## Results

On a 60-question golden set, grounding measured 89.58 percent, and every answer that shipped was grounded in the document it cited. All 12 unanswerable questions were refused. The cost of that discipline is measured too: 10.42 percent of answerable questions were declined, four of them genuinely over-cautious and one a fail-closed refusal during a pipeline outage, which is the system doing its job rather than failing at it. Every refusal fails closed, never a wrong answer dressed as a right one.

An earlier version of this page reported 91.67 percent, and the correction is worth stating rather than quietly swapping. That figure came from two eval runs that predate the product's entitlement gate, and the eval runner turned out to have been silently unauthenticated behind it, so for five days no run was measuring production at all, while the scoreboard looked fine. The number moved because the measurement started working, which is the more useful thing to know about it. The demo corpus is Meridian, a fictional company whose 337-document back-file I generated so the desk could be tested on realistic contracts, amendments, and policies without touching any client data. Stating that the corpus is invented is part of the honesty the product is built on. A live map draws all 342 documents and shows which ones nobody cites and where two documents disagree.

## Tech

- **Languages:** TypeScript, Python, SQL
- **Frameworks:** Next.js, React
- **Infrastructure:** Vercel, Neon Postgres (pgvector), Queued PC-hosted ask-runner
- **AI/ML:** Claude Opus (answer drafting), Gemini and GPT-4.1 (cross-vendor claim verifier), OpenAI text-embedding-3-large (retrieval, 1024-dim), Claude Sonnet (graph entity extraction)

## Skills demonstrated

- Fail-closed RAG architecture (embed, retrieve, draft, verify, gate, store)
- Cross-vendor claim-level verification (Claude drafts, Gemini checks)
- Passage-exact citation with full evidence storage
- Golden-set evaluation with grounding and refusal decomposition
- One-hop knowledge-graph retrieval expansion
- pgvector retrieval on Neon Postgres at 4,400-passage scale
- Queued PC-hosted inference runner behind a Vercel app
- Honest metrics methodology, every number traced to a stored run