CiteVyn
“Can I trust this answer — and trace every claim to its source?”
The judge that grades its answers is built not to flatter. Each answer is scored by a panel — the same model asked with three differently worded rubrics at temperature zero, so every score is reproducible, and the median is taken; a fourth, skeptic judge argues why the answer is wrong, and its score applies as a floor, because one low vote can never move a three-member median. A run where fewer than 90% of cases produce a usable score fails outright, rather than passing on a flattering average over the lucky survivors.
GateA 15-case retrieval suite on the candidate index gates promotion; any override is audited
Committed visual baseline, citevyn at df8cfc3. Both citations are real documentation URLs.
- Not claimed
- Corpus scale · adoption · SLA
What it does, and what it refuses
Works now
- Refuses when no source supports the answer, rather than guessing
- A 15-case retrieval suite runs against the candidate index and refuses promotion below a 0.95 pass rate
- Every promotion writes an audit entry — on the clean path and the forced one alike
- Intent is routed and labelled on screen: usage, exact lookup, how-to, out of scope
Known gaps
- Retrieval surfaces in-domain chunks on 5 of 19 near-miss refusal cases. The answer layer refused every one
- 53 golden case files exist on disk; 52 ran in the recorded report
- An operator can force promotion past a failing rate — the override is written to the audit trail with the measured rate
- Cost controls layer 4 is in-process only;
make budgetis planned, not implemented
Not claimed
- Corpus scale
- adoption
- SLA
Swap the embedding model without re-indexing and cosine distance quietly stops meaning anything: same dimensions, wrong space, confident citations to the wrong page. A dimension check cannot see it. So each index is stamped with the model that built it, and a stamp that disagrees drops the vector arm to zero rather than rank against the wrong space. An unstamped legacy index is let through on purpose.
Discussion
Comments need JavaScript and a GitHub account — the same account system the site's own code already lives in. Without either, read or start the thread directly on GitHub.