# Proofdesk

> A knowledge desk that checks every answer against exact passages, or returns an honest refusal instead.
>
> https://pravda.systems/work/proofdesk · 2026-08-25

**Problem.** Retrieval chatbots answer whether or not the documents support them, and a wrong answer looks exactly like a right one. Anyone who has to stand behind the answer cannot tell the two apart.

**Solution.** A fail-closed knowledge desk that returns exact stored passages with every shipped answer, has a separate cross-vendor verifier check each extracted claim against the evidence, and refuses rather than ship anything unproven.

**Outcome.** 89.58 percent grounding on the latest valid stored 60-question evaluation, 100 percent correct refusal, and a separate 997-document synthetic scale tenant where one changed file refreshed 3 of 997 document relationships and an independent full check matched all 2,991 links.

## Key metrics

- **89.58%** Grounding rate (golden set v3) — 43 of 48 answerable questions on the latest valid 60-question production run. Every answer that shipped was grounded in the document it cited (43/43). _(verified: eval run #35, eval_runs table, golden_sha 91b9bdb0, 2026-07-17, app-proofdesk docs/METRICS.md)_
- **100%** Refusal correctness — All 12 unanswerable questions refused. The fail-closed floor never let an unanswerable question through. _(verified: eval run #35, refusalCorrectness 12/12, app-proofdesk docs/METRICS.md)_
- **10.42%** False-refusal rate — The measured cost of refusing to guess: 5 of 48 answerable questions declined, including one fail-closed pipeline outage. _(verified: eval run #35, per-question refusalReason decomposition, app-proofdesk docs/METRICS.md)_
- **997 docs · 1.266m words** Synthetic scale tenant — A separate authenticated public-safe tenant stores 9,798 passages and 1,265,816 words. All 997 documents read back searchable, connected, and organized. The public example remains a separate 342-document corpus. _(verified: project 262 SQL + authenticated production status, 2026-08-25, app-proofdesk docs/proof-run-2026-08-24/stage31/knowledge-update-status.json)_
- **3 / 997 relationships** Bounded one-document refresh — A reversible one-document edit refreshed three affected document relationships, then an independent full check matched every one of 2,991 links. _(verified: jobs 1383/1385 + full SQL oracle, app-proofdesk docs/proof-run-2026-08-24/stage31/bounded-neighbor.json)_

## The problem

Retrieval chatbots guess. Ask one about contracts or policies and it can produce a fluent answer
whether or not the documents support it. The wrong answer looks exactly like the right one. For a
lawyer, operations lead, or support desk that must stand behind the result, that is worse than no
answer. The important question is not how often the tool answers. It is whether an answer is allowed
out when its evidence does not hold.

## What I built

Proofdesk answers only from the documents a project owns. Each shipped answer includes the exact
stored passages used as evidence. A verifier from a different vendor checks each extracted claim
against all retrieved evidence, then a
deterministic gate blocks the whole answer unless every claim is supported. If retrieval, drafting,
verification, or storage fails, the desk returns an honest refusal. It never falls back to unchecked
generation.

The product around that gate is real too: invited projects, owner-funded budgets, document upload,
PDF citations, proof receipts, sharing, and a living-corpus console. Owners can add, replace, or
remove documents, answer setup questions, inspect actual usage, see duplicate or contradiction
flags, and read whether saved passages, document connections, and the broader knowledge view are
current. Removed material stops serving new answers while old answer records stay honest.

## How it works

Vercel authenticates and queues each question. An on-demand Linux Node runner executes embed,
retrieve, bounded LOCAL or GLOBAL expansion, draft, verify, gate, and store. Embedding is pinned per
project. The base search reads active passage vectors. Graph links add directly related documents.
GLOBAL routing can reserve one existing evidence slot for a real passage selected through a
community summary. The summary is only a routing hint. It cannot enter evidence, citations, or the
proof pack.

A separate Python worker ingests documents and maintains extraction markers, exact document
centroids, neighbor edges, communities, and quality flags. Updates are content-aware. One changed
document does not force a blind full rebuild: current centroid identity and graph generations make
staleness explicit. In acceptance, an independent full query matched the bounded result before and
after restore. The console reads those persisted facts without starting work or changing whether
Ask is available.

## Results

The latest valid stored v3 evaluation, run #35, measured 89.58 percent grounding across 48
answerable questions. Every answer that shipped was grounded. All 12 unanswerable questions refused,
for 100 percent refusal correctness. Five answerable questions refused too. The 10.42 percent
false-refusal rate is the measured cost of choosing no answer over an unsupported one.

The public example remains 342 documents and 4,400 passages. Separately, an authenticated
public-safe synthetic tenant proves the storage and maintenance substrate at 997 documents, 9,798
passages, and 1,265,816 stored words. All 997 documents read back searchable, connected, and
organized. A reversible one-document edit recomputed one centroid and 3 of 997 affected source
neighborhoods, while an independent full query matched all 2,991 edges before and after restore.

The official 73-question LOCAL/GLOBAL pair is still withheld because the owner-funded OpenAI credit
needed for that run is exhausted. A 13-question preflight improved GLOBAL, but it is not promoted to
the headline metric. The scale tenant is generated fiction, not client data, and the missing paired
claim stays missing until a stored run earns it.

## Tech

- **Languages:** TypeScript, Python, SQL
- **Frameworks:** Next.js, React
- **Infrastructure:** Vercel, Neon Postgres (pgvector), On-demand Linux Node and Python runners
- **AI/ML:** Claude Opus (answer drafting), Gemini and GPT-4.1 (cross-vendor claim verification), Project-pinned OpenAI and Gemini embeddings (1024 dimensions), Claude Sonnet (graph extraction, quality checks, and community summaries)

## Skills demonstrated

- Fail-closed RAG architecture (embed, retrieve, expand, draft, verify, gate, store)
- Cross-vendor claim-level verification
- Passage-exact citations with immutable evidence records
- Golden-set evaluation with grounding and refusal decomposition
- Graph and community-routed retrieval with real passages as the only evidence
- Incremental corpus lifecycle and exact bounded neighbor reconciliation
- Living-corpus console with quality flags and current knowledge status
- Honest metrics methodology, every number traced to a stored run or proof artifact