A vision LLM had to read exact numbers off scanned German paperwork. A benchmark of 107 hand-verified questions scored 94% (101/107). Because it ran as an elluminate experiment with reasoning-level grading, every failure came with the judge’s reasoning attached, which pointed at image resolution, not prompting. Three experiments later: 107/107. Here is how that loop ran.
Where AI projects stall
Most AI projects survive the demo. They die on the stretch between a promising prototype and a system you’d trust to set a customer’s bill: organisations scrap 46% of AI proofs-of-concept before they reach production (S&P Global, 2025).
In our experience the missing piece is rarely model capability. It’s evidence: nobody can say, with numbers, whether the system is good enough, so it either ships on a hunch or doesn’t ship at all. elluminate exists to produce that evidence. What follows is one concrete example of how that looks in practice.
The problem: numbers that have to be exactly right
A German statutory health insurer needed to extract exact figures from the documents members upload: tax assessments, pension notices, salary slips, and a mix of online and handwritten paper questionnaires. The case that drove this work is contribution calculation: for self-paying members the monthly contribution (Beitrag) is income-based, so a number read off the page directly sets what someone pays. The capability is broader than that one case, though. The document mix grows over time, so the evaluation is built to generalise across what members submit, not to fit a single form or field.
This is visual question answering with a strict, exact-match bar: the answer is a specific number on a specific page, and “close” is wrong. A model that reads 18.450 as 18.150 sets the wrong contribution for a real member. Scanned forms make it harder: faint print, dense tables, and handwriting that takes people a second look.
There’s a second reason the bar is 100% and not 94%: the extracted figure is only the front door. Behind it runs the insurer’s contribution logic (income types, contribution-base floors and ceilings, family co-insurance, a thicket of special rules), and that machinery is where real bugs live. As long as a few percent of readings are flaky, every wrong contribution is ambiguous: a misread page and a bug in the calculation look identical from the outside. Pinning extraction at a verified 100% collapses that ambiguity. Once the inputs are trustworthy, any error left in a final figure is a genuine programming error to track down, not noise leaking in from a blurry scan.
The system under test was the customer’s production setup as deployed: a self-hosted, open-weights vision-language model (Qwen 3.5 family), fed through the same image pipeline members’ documents go through.
The test set: real documents, exact answers
The benchmark is built from 21 real member documents, anonymized. These are actual submissions spanning the categories above, not synthetic mock-ups. Synthetic paperwork is too clean: it wouldn’t surface the artifacts the production system actually has to cope with.
From those documents we wrote 107 questions, each a targeted numeric extraction (the assessed income on a tax assessment, the monthly amount on a pension notice, a single figure from a salary slip). The questions and their reference answers were drafted by a separate strong model (Claude) reading each document, then verified by hand against the source, so the answer key is trustworthy rather than circular. Each question becomes a test case: one row carrying the document, the question, and the reference answer to grade against. In elluminate a set of those cases is a collection.
Step one: define “good” once
An evaluation is only as trustworthy as its definition of correct. The most durable way to pin that down is a set of plain yes/no questions a domain expert can write and own, no code required. In elluminate that set is a criterion set. We wrote one deliberately format-tolerant criterion: the figures feed a structured pipeline, so a missing ”€”, a different thousands separator, or extra words must not count as a failure. Only the number matters.
# criteria are immutable once created — the name is versioned, so every
# past experiment keeps the exact definition of "good" it ran against
criterion_set, _ = client.get_or_create_criterion_set(
name="Document Parsing Correctness v2",
defaults={"criteria": [
"Does the answer contain the numeric value {{reference_answer}}? "
"Only the number must match — formatting is irrelevant: a missing € "
"sign, different thousands separators, a missing trailing ',00' (zero "
"cents), or extra explanatory text explicitly do NOT count as errors."
]},
)(The production prompt and criterion are German; shown here in English for readability.) Because the criterion stays fixed, a later change in score reflects a change in the system, not in the test.
Step two: score the production outputs
With “good” defined, the rest is mechanical. We ran every document through the production path (image pipeline, vision model, prompt) and uploaded the answers to elluminate to score against the criterion. add_responses registers the outputs; rate_responses grades each one and, in DETAILED mode, hands back the judge’s reasoning:
responses = experiment.add_responses(
responses=model_answers, # generated externally, exactly as in prod
template_variables=test_cases,
metadata=metadata,
)
ratings = experiment.rate_responses(responses, rating_mode=RatingMode.DETAILED)
for resp, rating_set in zip(responses, ratings):
passed = all(r.rating == RatingValue.YES for r in rating_set)
# DETAILED mode -> each rating also carries the judge's reasoningRunning the real pipeline end to end matters: the score reflects what a member would actually get, including the image downscaling that, as we’ll see, decides several answers. And because the generations are stored, re-grading is free: refine the criterion or compare runs in seconds, with no new inference.
94%, and elluminate tells you why
The first run, mirroring production at 896 px images, scored 101/107 (94%). Respectable for an untuned open-weights model, but not good enough here: each of the six misses is a member billed on the wrong income.
A bare “94%” is where teams either declare victory or start guessing. Because the run used RatingMode.DETAILED, every failing answer carried the judge’s reasoning, and the six failures sorted quickly:
- 4× handwriting: handwritten dates and amounts misread (a “1” read as “4”).
- 1× fine print: a small printed digit read as 3 instead of 5.
- 1× empty answer: the model returned nothing at all.
The same handwritten date at two resolutions. Downscaled to production’s 896 px, the looped German 1 blurs into a 4 (model read 03.04.26). The zoom pass re-renders just that page at 2048 px and reads it correctly (03.01.26). Synthetic sample, not a real member document.
The score told us where we stood. The reasoning told us what to do next.
Five of the six are small, hard-to-see regions of the page, which left two candidate explanations. Either the pixels weren’t there to begin with (an image-pipeline problem) or the model saw the strokes fine and merely applied the wrong reading convention, which a prompt could fix. The second isn’t far-fetched: Germans write the digit 1 with a long upstroke, and on a rough scan that shape is easily taken for a 4, exactly the misreading in our failure list. We tested both hypotheses.
Iterating, one experiment at a time
Attempt 1: prompt engineering (it didn’t work)
We tried seven system-prompt strategies on the single hardest case: spelling out German digit conventions, transcribe-first, stroke-counting, self-critique. The best got it right 1 time in 5. So the convention wasn’t the problem. The model simply couldn’t see the strokes well enough.
Attempt 2: resolution (the real lever)
If the pixels aren’t there, no instruction adds them back. Production downscaled every page to 896 px on the longest edge, and a vision encoder pools small regions away at that size. So we swept resolution: same questions, same criterion, only the image size changed between runs.
| Image resolution | Pass rate |
|---|---|
| 896 px (production) | 94% · 101/107 |
| 1536 px | 98% · 105/107 |
| Two-pass zoom + retries | 100% · 107/107 |
The last handwritten date only became legible at 2048 px.
Bumping to 1536 px lifted the score to 105/107 (98%). One stubborn handwritten date needed a full 2048 px before it read correctly, and did so 5 runs out of 5, confirming a resolution limit rather than luck.
Attempt 3: a two-pass “zoom” read
Rendering every page at 2048 px is expensive, and most pages don’t need it, so the second pass is gated. Pass one runs cheap and low-resolution across all pages; alongside each answer, the model itself reports where the value sits and, as a confidence score, how sure it is that it read the number cleanly. High confidence (clean printed numbers) and we keep the answer and stop. Low confidence, which is exactly what handwriting and fine print produce, and we re-render just that page at 2048 px and read it again.
The self-reported score doesn’t separate right from wrong on its own, but it reliably drops below the threshold for anything that isn’t crisp print, which makes it a dependable “look closer” trigger and keeps the expensive pass off the pages that don’t need it. The re-read sends the whole page, not a tight crop: a slightly-off crop can land on a neighbouring table cell and make the model read the wrong number. Full-page zoom only adds pixels; it can’t hide the answer.
Closing the last gap to 100%
That left the lone empty answer. The production model is a “thinking” model with a variable reasoning budget; occasionally it spent the whole budget reasoning and emitted nothing. The fix is as unglamorous as it sounds: retry with a higher token budget, plus a transport-level retry for flaky connections. Re-running the experiment: 107/107.
One clean run can’t separate “fixed” from “got lucky”, though: this failure was stochastic to begin with. That’s where cheap repetition matters: the questions and the criterion are already in place, so re-running the experiment costs a few minutes of inference and nothing else. The 2048 px result earlier was pinned down the same way, across five runs. From a 94% start, it took three experiments to reach a verified 107/107. That 100% is honest about its scope: it means the 107 questions in the benchmark are solved, not that every document a member could submit will read perfectly. So the set doesn’t stay frozen at 107. Production keeps surfacing harder cases (a messier handwriting sample, a form layout the benchmark never saw), and each one is folded back in as a new question, keeping the bar at 100% against a benchmark that steadily grows. That is how the generalisation gap closes: every failure found in the field turns into a permanent test the system can’t silently regress on.
Why the loop worked
None of the fixes were clever: more pixels, a second look at hard pages, a retry. The real work was knowing which fix to make and proving it worked, and that part elluminate makes routine:
- Experts define “good”: a format-tolerant yes/no criterion, owned by the people who know the domain, not buried in code.
- Evidence, fast: reasoning-level grading turned six failures into a diagnosis; saved generations made re-grading instant.
- Apples to apples: the same questions and the same criterion across every run, so each change (a prompt, a resolution, a second pass) was an isolated, comparable experiment.
A detail that’s easy to miss: while iterating, we probed candidate fixes on just the failing cases (the quickest way to see whether a fix helps at all), but no score counted until the full 107 questions were re-run, because a fix can also break cases that used to pass. Raising the resolution is hard to imagine backfiring, but the two-pass setup changed the prompt for every page, since each answer now comes with a location and a confidence, and that could plausibly have unsettled answers that were already correct. Re-running the full benchmark after each change is what ruled that out, at no cost beyond the inference itself.
The benchmark also outlives the project: the collection and the criterion stay in elluminate, so re-scoring a swapped model or a new document type takes minutes, and every future change starts from a known baseline.
Each step produced a number we could defend, and the saved experiments double as the audit trail an EU AI Act review expects. For a regulated workflow that handles member income data, that trail is a precondition for shipping at all.
Customer and documents anonymised; example values are illustrative, not real figures. The system under test is a self-hosted open-weights vision-language model, evaluated as deployed.
This is one approach to turning “we think it works” into “we know it works.” The same methodology applies to any AI system where you need to compare performance with precision: model upgrades, prompt changes, pipeline modifications.
Interested in running structured evaluations for your own AI systems?
