WorkPramaanam
Pramaanam
A citation verifier for Indian Supreme Court judgments. It checks citations and quotes against 38,397 judgments.
100 of 100
invented quotes caught by Pramaanam. One AI model (gemini‑3.1‑flash‑lite) let 35 of the same 100 through.
The problem
A citation makes two claims that can be checked: the case exists, and the judgment says what is quoted. Text written by a language model can carry citations and quotes that read well and are wrong.
Asking a second model to check does not settle it. Given 100 invented quotes, gemini-3.1-flash-lite accepted 35 as real. Pramaanam checks against the judgments themselves, and accepted 0.
How it works
Find the case
Each citation is matched to a judgment in a corpus of 38,397 Indian Supreme Court judgments. The case has to exist and the name has to match.
Check the quote
Quoted words must appear in the judgment at the cited paragraph. The check is deterministic and runs against the judgment text, not against what a model remembers.
Flag later history
Headnotes are used to flag cases that were later overruled, so a real citation to bad law is not waved through.
Optional LLM reading
An opt-in reading agent can look at harder cases. A guard in the code means it can never mark a citation VERIFIED. Only the deterministic checks can do that.
The results
| Check | Sample | Result |
|---|---|---|
| Held-out test: citation matched to the right case | 667 | 96.1% |
| Held-out test: real quotes verified | 275 | 272/275 |
| Held-out test: altered quotes that passed | 1,166 | 1/1,166 |
| Invented quotes accepted by gemini-3.1-flash-lite | 100 | 35/100 |
| Invented quotes accepted by Pramaanam | 100 | 0/100 |
Invented quotes: model against checker
Fabricated quote rejectedFabricated quote accepted as real
Altered quotes
Altered quote caughtAltered quote passed
Real quotes
Real quote verifiedReal quote not verified
One mark per quote. Marks are grouped by outcome; their position carries no other meaning.
The three real quotes not verified went to review, because a later judgment's headnote marks the case as overruled or per incuriam. The quotes themselves were verbatim. The thresholds were fixed on a development set before the held-out run. The comparison with gemini-3.1-flash-lite is one run of a small free model on 100 invented quotes from the development set, so a stronger model may do better.