Problem: our recognition pipeline goes forward and forgets pages. Multi-date lab tables, page breaks, mixed document types → data lost. We needed a way to preserve all information from medical documents.
This diagram shows the thinking evolution from "save raw text" to the current fact-based extraction with FHIR-mapped types.
PDF (visual):
┌────────────────┬──────────┬──────────┬───────┬───────────┐
│ Parameter │ 15.03.26 │ 10.01.26 │ Units │ Range │
├────────────────┼──────────┼──────────┼───────┼───────────┤
│ Hemoglobin │ 145 │ 138 │ g/L │ 130-160 │
│ ESR │ 12 │ 18 ↑ │ mm/hr │ 2-15 │
│ Cholesterol │ 4.9 │ 6.2 ↑ │ mmol/L│ <5.2 │
│ Target (ESC):│ │ │ │ │
│ Very high │ │ │ │ <4.0 │
│ High risk │ │ │ │ <5.0 │
└────────────────┴──────────┴──────────┴───────┴───────────┘
Vasya proposed: save all pages to SQL (raw text + images, numbered by page), then launch agent "spiders" that come back and solve specific problems (dates, ranges, consistency).
Two poles discussed:
- Pole A: Strict workflow (steps, cheap, fast, but fragile)
- Pole B: Agent with tools (flexible, but expensive and slow)
Decision: need a combination, not either extreme.
Parameter 15.03.26 10.01.26 Units Range
Hemoglobin 145 138 g/L 130-160
ESR 12 18 mm/hr 2-15
Cholesterol 4.9 6.2 mmol/L <5.2
Target (ESC):
Very high <4.0
High risk <5.0
Lost: which value belongs to which date, risk tiers belong to Cholesterol. No structure, no columns, no spatial info.
| Parameter | 15.03.26 | 10.01.26 | Units | Range |
|-------------|----------|----------|--------|---------|
| Hemoglobin | 145 | 138 | g/L | 130-160 |
| ESR | 12 | 18 ↑ | mm/hr | 2-15 |
| Cholesterol | 4.9 | 6.2 ↑ | mmol/L | <5.2 |
Target (ESC): Very high <4.0, High risk <5.0Better: columns preserved, dates visible. Lost: risk tiers detached from Cholesterol; card layouts, mixed formats would break.
<table>
<tr><th>Parameter</th><th>15.03.26</th><th>10.01.26</th>...</tr>
<tr><td>Hemoglobin</td><td>145</td><td>138</td>...</tr>
...
</table>
<div class="footnote">Target (ESC): Very high <4.0...</div>More structure than markdown. But very verbose — 3 params = 500+ chars of HTML tags. Huge token cost. Risk tiers still detached. Not tried in practice.
{
"tables": [{
"columns": ["Parameter", "15.03.26", "10.01.26", "Units", "Range"],
"rows": [
["Hemoglobin", "145", "138", "g/L", "130-160"],
["ESR", "12", "18", "mm/hr", "2-15"],
["Cholesterol", "4.9", "6.2", "mmol/L", "<5.2"]
]
}]
}Structure preserved. But: risk tiers lost (not a table row), non-table content lost. Not everything in medical documents is a table — cards, narratives, mixed layouts exist.
{"type": "lab_result", "name": "Hemoglobin", "value": "145", "unit": "g/L", "range": "130-160"}
{"type": "lab_result", "name": "Cholesterol", "value": "4.9", "unit": "mmol/L", "range": "<5.2"}
{"type": "metadata", "key": "esc_very_high_risk", "value": "<4.0 mmol/L"}
{"type": "metadata", "key": "esc_high_risk", "value": "<5.0 mmol/L"}Result: only 1 date extracted (latest), historical values (138, 18) lost! Risk tiers dumped into metadata with random keys, detached from Cholesterol. Metadata became a catch-all.
Key number: 33% recall on multi-date (extracted 20/60 params — only latest date).
Realized: we already have fhir-services that extracts 7 types (conditions, medications, allergies, procedures, family_history, symptoms, observations). Medical document content types are known and standardized as FHIR resources. No need to invent our own taxonomy.
{
"type": "observation",
"name": "Hemoglobin",
"value": "145",
"unit": "g/L",
"range": "130-160",
"date": "2026-03-15",
"notes": [],
"category": "laboratory",
"sample_material": "BLOOD",
"page": 1
}
{
"type": "observation",
"name": "Hemoglobin",
"value": "138",
"unit": "g/L",
"range": "130-160",
"date": "2026-01-10",
"notes": [],
"category": "laboratory",
"sample_material": "BLOOD",
"page": 1
}
{
"type": "observation",
"name": "Cholesterol",
"value": "6.2",
"unit": "mmol/L",
"range": "<5.2",
"date": "2026-01-10",
"notes": [
{"type": "risk_classification", "text": "ESC targets: Very high <4.0, High <5.0 mmol/L"}
],
"category": "laboratory",
"sample_material": "BLOOD",
"page": 1
}8 fact types (FHIR-mapped): observation (6 categories: lab, imaging, vital-signs, procedure, exam, social-history), condition, medication, allergy, procedure, family_history, symptom, metadata.
What was added:
dateper observation → multi-date supportnotes[{type, text}] → footnotes, risk classifications, method notesalt_name→ bilingual documentsanyOfstrict schema → type safety
| Metric | Value |
|---|---|
| Recall on 17 GT cases | 92%+ (remaining gaps = GT errors, not LLM errors) |
| Value accuracy | 98.3% |
| Languages tested | EN, DE, FR, HE, RU, KO (6+) |
| Max dates from single doc | 11 dates (18-page longitudinal report, 12 years of history) |
| Max facts from single doc | 308 facts (27-page Korean comprehensive checkup) |
| Page-break handling | ✅ Notes split across pages collected correctly |
| Processing | Single-pass GPT-5.2, no agents, no SQL, no multi-pass |
Old prompt: "extract test parameters" → 33% on multi-date
New prompt: "extract all facts" + schema → 100% on multi-date
Same model (GPT-5.2). Same single pass. Same API call.
Problem was in PROMPT + SCHEMA, not in PIPELINE.
No SQL needed. No agent spiders needed. No page saving needed. The spatial context stays in the IMAGE — LLM sees the PDF and knows what relates to what.
apps/benchmark/scripts/extract-facts.ts— extraction script with full schemaapps/benchmark/scripts/evaluate-gt.ts— evaluation against ground truthapps/benchmark/patients/real-prod/— 12+ real production PDFs with extraction results- Linear: BG-1059