Web-based educational content is delivered through expanding sections, embedded videos with transcript buttons, interactive H5P widgets, and styled block quotes, all rendered inside a dynamic learning management system. Extracting this content into revision notes requires either DOM scraping or screenshot OCR, and both paths have distinct, largely uncorrelated failure modes. We hypothesised that cross-checking the two captures through a layered verification pipeline, powered by a single small language model on a free tier, could produce revision notes whose completeness is verifiable without a human re-reading every page. We built a four-layer architecture: deterministic hard gates on summary similarity and size, bidirectional LLM judgement with position-bias cancellation, mechanical intersection of absence reports with substance-check filtering, and targeted LLM adjudication of surviving suspicions. Each layer has its own error profile, but the layers compensate. Evaluated on a 26-page checkpoint from a 165-page corpus across four university short courses, the architecture produces zero false passes. Two real capture gaps were surfaced and confirmed. The technique runs at zero cost on a free-tier small LLM language model (google/gemma-4-26b-a4b-it that can run on a high-end laptop. To parallalise the LLM steps gemma4 was run on the free teir of OpenRouter.
-
-
Save simbo1905/3126226becb4c6bfc06d75bb93236250 to your computer and use it in GitHub Desktop.
Web-based educational content is delivered through expanding sections, embedded videos with transcript buttons, interactive H5P widgets, and styled block quotes, all rendered inside a dynamic learning management system (LMS). Extracting this content into revision notes requires either DOM scraping or screenshot OCR, and both paths have distinct, largely uncorrelated failure modes.
We hypothesised that cross-checking the two captures through a layered verification pipeline, powered by a single small language model on a free tier, could produce revision notes whose completeness is verifiable without a human re-reading every page. We built a five-layer architecture: combined-OCR inventory detection, deterministic hard gates, bidirectional LLM judgement, mechanical intersection with substance-check filtering, and targeted LLM adjudication.
Each layer has its own error profile. The deterministic gates miss single-paragraph losses below the summarisation noise floor. The judge over-flags page furniture and detail-level differences. The adjudicator cannot auto-clear OCR name-mangles. Yet the layers compensate: the combined-OCR inventory catches the shared failure mode, the intersection filters one-order-only phantoms, the substance check filters compression phantoms, and the adjudicator clears synonymously expressed concepts.
Evaluated on a 26-page checkpoint from a 165-page corpus across four university short courses, the architecture produces zero false passes. Two real capture gaps were surfaced and confirmed. The technique is offered as a reusable pattern for any context where two substantially different capture paths produce text from the same source material.
University short courses delivered through mainstream LMS platforms present body text alongside lazy-loaded expanding sections, embedded videos with a Transcript button, interactive H5P widgets, styled block quotes, and figures with in-image text. A student taking personal revision notes wants the full reachable content: body text, expanded sections, verbatim transcripts, and widget prose.
Two natural capture techniques exist. DOM extraction uses browser automation to traverse the page and produce markdown. Screenshot OCR takes a full-page retina screenshot and runs it through one or more OCR engines. Both have distinct, largely uncorrelated failure modes, and both share exactly one common failure point: failing to click through all expandable content.
This study asks whether combining the two captures and cross-referencing their outputs through a layered pipeline can produce revision notes whose completeness is verifiable without a human re-reading every page. We targeted a model that runs on a consumer laptop: several candidates (mistral-small, magistral-small, gemma-4-26b, and others) were tested under Ollama on a 2025 MacBook. The selected model, google/gemma-4-26b-a4b-it, is a mixture-of-experts whose active weights fit in roughly 2 GB of RAM, making it suitable for overnight batch jobs on modest hardware. For the evaluation, LLM calls were offloaded to OpenRouter's free tier to parallelise across a dozen pages.
DOM extraction uses browser automation (Chrome DevTools Protocol via a
puppeteer-like driver or a standalone tool such as simonw/rodney) to traverse
the page and extract educational content into markdown. Video transcripts
live in hidden DOM elements readable through textContent with no extra
clicks. Expanders require a click per element, and because the LMS
deliberately resists scraping, every interaction must be manual-paced.
The DOM-extracted markdown is not the direct input to the verification
pipeline. An assembly step (build_all_txt_301.py) first composites the page
markdown, video transcript sidecars, a media inventory file, and the visible
text of rebuilt H5P widgets into a single _all.txt per page. This composite
is the actual repo-side input. If any input is missing or stale, the
_all.txt is incomplete, and the verification pipeline will flag the gap
against the OCR side.
OCR-based extraction takes a full-page retina screenshot and runs it
through two independent OCR engines: Tesseract 5.5.3 (Smith 2007, via the
tesseract CLI) and Apple Vision OCR (via a Swift command-line wrapper using
the Vision framework on macOS). Tesseract always receives the unscaled
image; tall images that exceed the leptonica dimension limit (32767px) are
tiled into 2000px strips. Vision handles tall pages natively up to 7000px;
above that, it is also tiled.
The two engines produce different error profiles on the same screenshot, and together they recover text that neither recovers alone.
OCR cannot see content inside a collapsed expander, nor can it read the text of a video transcript hidden behind a button. This creates a two-pass OCR workflow: pass 1 operates on the page in its default collapsed state and inventories the interactive elements visible on the page; pass 2 repeats after every expander has been clicked and every transcript has been loaded.
The inventory step is where the uncorrelated errors of the two OCR engines
are exploited. Tesseract reads the LMS expand/collapse glyph (Unicode
U+25B6) as an ASCII > character, while Vision bullet-maps it to * or
drops it entirely. Vision reads body text with near-perfect accuracy but
misses some styled text; Tesseract is fast but fragile on tall pages.
Neither engine alone provides a reliable signal for all element types. The inventory detection layer concatenates both OCR outputs into a single prompt, preceded by a glyph legend that explains how each engine renders icons, and asks the model to produce a JSON inventory of videos, interactive widgets, expanders, and figures with quoted evidence. The model sees both readings side by side and reasons over them together.
A prompt matrix experiment (7 rounds, 3 prompt architectures) confirmed that the combined single-prompt approach with glyph legend outperformed both a split two-call approach and a combined approach without the legend.
The pass 1 inventory is checked against the live DOM's <details> element
count (the ground truth for expanders); any count mismatch is a hard stop.
When the inventory reports expanders, a pass 2 capture is mandatory before
any content verification can proceed.
| Failure mode | DOM extraction | OCR extraction |
|---|---|---|
| Missed expander (collapsed content) | Shared: click needed | Shared: click needed |
| Transcript outside viewport (lazy-loaded) | Not a failure (hidden DOM readable) | Content invisible to OCR |
| H5P interactive text | Fragile (widget rebuild needed) | Partially visible (button labels, hotspots) |
| OCR word-mangling (proper nouns, styled text) | Not applicable | Present (Tesseract and Vision make uncorrelated errors) |
| Deep/dynamic DOM (multi-megabyte, deeply nested) | Extraction fragility | Not applicable (screenshot bypasses DOM) |
| Page furniture (navigation, breadcrumbs, due dates) | Absent from markdown (scraper strips) | Present in OCR; summariser must strip |
| Stale prose (course team rewrote a page) | Present (both sides equally stale) | Present (both sides equally stale) |
The shared failure mode (expander not clicked) has its own mitigations. The page's visible body text often says "click on the following sections" and the LMS renders triangular expand/collapse glyphs that both OCR engines detect as solid right-pointing triangle characters (Unicode U+25B6). These two signals provide independent evidence that expandable content exists, independent of the two-pass OCR inventory.
The educational material is structured to aid learning, and the LMS has been optimised for read-aloud. The body text invites the learner to click, watch, and interact.
The scenario is non-adversarial: no party is trying to conceal content. Failures are engineering defects, not attacks, and this allows the verification architecture to lean on consistency rather than cryptography.
The verification pipeline has five layers, applied to every page after both capture paths are complete. All LLM calls use google/gemma-4-26b-a4b-it at temperature 0, max_tokens 2000, via the OpenRouter chat completions endpoint. Each prompt is prepended with a cache-busting session identifier to prevent response caching.
flowchart TD
A[Live LMS Page] --> B[DOM Extraction]
A --> C[Screenshot OCR]
C --> D1[Tesseract 5.5.3]
C --> D2[Apple Vision OCR]
D1 --> E[Combined-OCR Inventory Detection]
D2 --> E
E -->|expanders detected| F[Pass 2: expand and re-capture]
E -->|no expanders| G
F --> G
B --> H[Assembly: _all.txt]
G --> I[Summarise Vision text]
H --> J[Summarise repo text]
I --> K[Judge A-then-B]
I --> L[Judge B-then-A]
J --> K
J --> L
K --> M{Subject gate<br/>cosine < 0.44?}
L --> M
M -->|fail| V1[FAIL]
M -->|pass| N{Size gate<br/>ratio < 0.75/0.65?}
N -->|fail| V1
N -->|pass| O[Intersect absence lists<br/>across both orders]
O --> P{Substance check:<br/>item in claimed-missing source?}
P -->|phantom| NOTES[Route to notes]
P -->|real| Q[Targeted LLM Adjudication]
Q -->|discussed: true| NOTES
Q -->|discussed: false| R{Direction?}
R -->|absent from repo| V2[FLAG]
R -->|absent from vision| V3[REVIEW]
O -->|symmetric difference| V3
R -->|all cleared| V4[PASS]
The first layer detects what interactive elements exist on the page. Both OCR channels (Tesseract and Vision) are concatenated into a single prompt with a glyph legend that explains how each engine renders icons. The model is asked to return a JSON inventory of videos, interactive widgets, expanders, and figures, each with a short evidence quote from the OCR text.
Every evidence quote is mechanically verified against the concatenated OCR source using an OCR-aware substring check with confusable normalisation (I/l/1/|, O/0). A fabricated quote fails loudly rather than silently poisoning the inventory.
This layer exploits the uncorrelated error profiles of the two engines. Tesseract detects glyphs that Vision misses; Vision reads body text that Tesseract mangles. Neither engine alone provides a reliable inventory. The combined single-prompt approach lets the model reason over both readings simultaneously.
A prompt matrix experiment (7 rounds, 3 architectures) confirmed that this design outperformed both a split two-call approach and a combined approach without the glyph legend.
The inventory result drives the expansion rule: if expanders are detected,
a pass 2 capture is mandatory. The inventory count is checked against the
live DOM's <details> element count; any mismatch is a hard stop. This
layer protects against the shared failure mode (unclicked expanders) that
both capture paths have in common.
For the content verification layers below, the Vision text (the cleaner prose channel) is used as the OCR-side input. Tesseract serves as the detector co-channel: its glyph readings drive the inventory, and its text is used alongside Vision for the video marker grep (a strict marker-line grep on both texts that must agree on transcript counts).
Both the OCR-recovered text (the pass 2 Vision output, enriched with freshly
fetched video transcripts inlined at the Transcript marker position) and
the assembled repo text (_all.txt) are summarised independently by the same
model using a full-coverage summary instruction: include every section
heading, named person, organisation, framework, and case study. The model
produces 250 to 400 words of plain prose.
Summarisation strips page furniture (navigation, headers, due-date text) before the content-comparison layers operate, so the downstream checks reflect content differences, not CMS rendering.
The study summaries (the _summary.md files used for revision) are produced
by a separate run of the same model with a different prompt: terse,
business-like, action-oriented, with FRAMEWORK HIGHLIGHT tags. The study
summaries and the verification summaries are different artefacts with
different prompts.
This separation matters: if the same model with the same prompt produced both the study summary and the verification summary, a consistent compression blind spot would produce identical omissions on both sides, and the judge would see no difference. Using different prompts for the two summaries produces uncorrelated compression profiles, which is the whole point of the cross-check.
Both summaries are judged by the same model twice, A-then-B and B-then-A, to cancel any presentation-order bias. The judge is asked three questions with a single structured JSON reply: (a) are both summaries about the same subject, (b) list any whole topic absent from one side but present in the other, with a short evidence quote, and (c) which side is more complete.
Non-subject differences (page furniture: navigation, breadcrumbs, student instructions) are separated into a dedicated non-content bucket. The resulting JSON verdicts (two per page) become the raw input to the reconciliation layer.
Before interpreting the item-level absence reports, two deterministic checks run on the summarised text. Neither gate makes a model call; both are pure Python standard library.
Subject gate. The two summaries are compared with a stopword-filtered bag-of-words cosine similarity. A score below 0.44 indicates materially different subjects (the suspected wrong-page class), which is a hard fail. The threshold was calibrated on 24 healthy same-subject summary pairs (minimum cosine 0.489) and one wrong-page cross-pair (cosine 0.387).
Size gate. The ratio of the two summary word counts is measured. Below 0.75 for no-video pages and below 0.65 for pages with video transcripts (asymmetric by design because the OCR side cannot see transcripts) triggers a hard fail. The no-video threshold was calibrated on 24 healthy pairs (minimum ratio 0.831).
Simulated loss of the longest paragraph from healthy summaries dropped the ratio to between 0.63 and 0.74, confirming that a single lost expander is below the deterministic noise floor and must be caught by the item-level layers.
The two judge verdicts (A-B and B-A) are reconciled by intersecting their absence lists across the two orders. Items are matched by normalised item text, evidence-quote containment, and token overlap (threshold 0.6). Items present in only one order (the symmetric difference) are routed to human review without adjudication.
Each intersected item is substance-checked mechanically: the item's text and evidence quote are normalised and searched as substrings in the source text of the side the item is claimed absent from. Items present in that side's source are compression phantoms (the summariser dropped a point the source contains) and are routed to notes. Items not found pass on to the next layer.
Each surviving suspicion receives a targeted call to the same model, with a prompt that names the recovery channels explicitly: the concept may be genuinely absent, OCR can misread proper nouns, collapsed sections and interactive widgets may be hidden, and the summariser may have dropped or rephrased the concept.
The adjudicator returns a single boolean (discussed) with a short quoted
basis. A true reply clears the suspicion (routes to notes); a false
confirms it as a real difference. Parse errors default to review, never to
proof.
The five layers produce a single verdict per page:
- fail: a deterministic hard gate tripped, or the judge returned
same_subject: false. - flag: at least one confirmed real item in the alarm direction (content visible on the live page whose counterpart is absent from the DOM capture).
- review: confirmed real items in the expected direction (absent from the OCR capture; transcript and H5P content), contradicted order disagreements, symmetric-difference items, or old-schema judge verdicts.
- pass: no hard gate failure and all suspicions cleared.
The asymmetry is deliberate. A flag requires a human to double-check, but
a pass means every automated layer cleared the page independently. False
passes require correlated failures across five decorrelated layers.
A 16-test scenario suite was constructed over controlled summary pairs, with each scenario judged by the same model at temperature 0 in both presentation orders. Scenarios: identical summaries, a deliberately wrong-topic cross-pair, one/two/three whole topics injected into one side, one topic injected into the reverse side, a page-furniture-only addition, and a detail-level addition.
The checker enforced exact JSON parseability, exact same-subject values, exact item counts, required evidence keywords, and the non-content bucket.
The suite achieves 12 of 16 exact passes. The alarm classes (identical, wrong-topic, and all injected-topic counts in both orders with verbatim evidence quotes) pass at 100%. The four failures are over-flagging only: the model reports page-furniture and detail-level additions as absences despite acknowledging them as administrative in its notes.
These failures are in the safe direction (they cost a double-check) and are kept failing on purpose with doc comments as living documentation of the model's conservative bias.
The hard-gate thresholds were set by measuring 24 healthy same-subject summary pairs and one wrong-page cross-pair. The negative control (cosine 0.387) is cleanly separated from the healthy minimum (cosine 0.489) by the 0.44 threshold.
Simulated loss of the longest paragraph from healthy summaries dropped the size ratio to between 0.63 and 0.74, confirming that a single lost expander is below the deterministic noise floor and must be caught by the item-level layers.
The pipeline was run over a corpus of 165 pages from four university short
courses. Each page had a completed two-pass OCR capture (Tesseract plus
Apple Vision on a retina screenshot, with expander detection confirmed
against the live DOM) and a DOM-extracted markdown page assembled into
_all.txt.
The checkpoint evaluation covers 26 pages from the largest course module and three canary pages from a second course, plus two canary pages from a third course, and the wrong-page negative control.
Results: 8 pass, 12 review, 2 flag, 1 correct fail (the negative control). The adjudicator cleared every known phantom with quoted evidence from the source text.
The two flags were confirmed as genuine gaps: a welcome-page module outline present in the live OCR text but absent from the DOM capture, and an OCR name-mangle (Zuboff read as Lubott) that is a trivial human double-check but is kept as a flag for the record rather than suppressed by rule. No page produced a false pass.
Two known phantom classes were tested against actual page source texts. A sectors-list item (retail, banking, insurance, healthcare) flagged as absent from one side was cleared with the verbatim quote from that side's source, confirming the compression phantom.
A concept described in synonymous language (surveillance-capitalist data extraction described as a market in behavioural futures, while the flagged item called it behavioural surplus) was also cleared, confirming that the adjudicator handles paraphrased same-topic content correctly.
An OCR name-mangle (Lubott for Zuboff) was confirmed absent by the adjudicator, as the exact mangled spelling does not appear in the source. This is the naming-drift blind spot, a documented accepted limitation.
All 26 checkpoint pages contain complete pass 1 OCR inventories that agree with the live DOM on expander counts. The glyph-based detection (Unicode U+25B6 recognised by both OCR engines) provides an independent signal.
When either OCR engine detects the glyph and the inventory reports zero, or when the body text contains language inviting expansion, the page is flagged for a manual expander check before being cleared for the verification pipeline.
Tesseract and Apple Vision produce different error profiles on the same screenshot. Tesseract is fast but fragile on tall pages; Vision handles tall pages easily but misses some styled text. On a 2880 by 4992 pixel screenshot, the two engines together recovered text that neither recovered alone.
The inventory detection layer feeds both OCR channels to the model in a single prompt so it can see glyphs, body text, and invitation language together. For the content verification layers, the Vision text (the cleaner prose channel) is used as the OCR-side input, with Tesseract serving as the detector co-channel for the video marker grep.
The deterministic gates miss single-paragraph losses below the noise floor. The judge over-flags page furniture and detail-level differences. The adjudicator cannot auto-clear OCR name-mangles.
Yet on the 26-page checkpoint, every page that passed had survived the combined-OCR inventory check, the subject gate, the size gate, two independent judge calls, and the adjudicator: five decorrelated checks with no shared code path beyond the summariser input.
The two flags were confirmed as real gaps. The review verdicts are the expected-class items: transcript content invisible to OCR, H5P text not captured by the DOM scraper, and order contradictions that the intersection rule routes to human attention. No review verdict was later found to conceal a real gap.
The inventory detection layer makes one model call per pass (two calls for a two-pass page). The summariser and judge each make two model calls per page. The adjudicator makes one call per remaining suspicion (typically zero to three per page).
At the provider's free tier for gemma-4-26b on OpenRouter, the cost is zero for the evaluation runs described here. The deterministic gates and substance check make no model calls and are pure Python standard library.
Stale prose. If the course team rewrites a page after both captures were made, both sides are equally stale and the difference is invisible to all layers. The methodology explicitly excludes this; a verbatim re-diff is a separate follow-up.
Single-expander loss on video pages. A single collapsed expander whose paragraph is small relative to the summary word count falls below the deterministic size gate on video pages (the video transcript inflates the word count, masking the loss). The item-level layers and glyph detectors provide a compensating signal.
OCR name-mangles. The name-mangle class surfaces as a flag rather than being auto-cleared. This is by design (the asymmetry favours false fails over false passes) but produces a small number of known-harmless flags, as seen with the Lubott/Zuboff case. A future iteration could add a fuzzy-match step before adjudication: if the flagged item is a proper noun that fuzzy-matches a name in the source text (Levenshtein distance below 2), route it to notes as a likely OCR mangle.
Evaluation scope. The evaluation covers 26 checkpoint pages plus canary pages from three courses, not the full 165-page corpus. The checkpoint module is the largest by page count and contains all content classes (no-video pages, video pages, H5P pages, expanding sections, figures, block quotes, and activity pages). Multi-video pages were not tested at the time of evaluation.
Non-content bucketing. The non-content bucketing instruction in the judge prompt is under-used by the model, producing page-furniture items in the absence lists that the mechanical substance check filters correctly but that would reduce the signal-to-noise ratio on a human-readable diff without the reconciliation layer.
We hypothesised that two independent capture paths with uncorrelated error profiles could be cross-verified by a layered pipeline powered by a single small language model on a free tier. The architecture confirms this: each layer has its own error, but the errors do not correlate.
The combined-OCR inventory detection catches the shared failure mode by feeding both OCR channels to the model so it can see glyphs and body text together. The deterministic gates handle the unambiguous failure classes without a model call. The judge catches whole-topic absences with position-bias cancellation. The intersection filters one-order-only phantoms. The substance check filters compression phantoms. The adjudicator clears synonymously expressed concepts.
On the evaluated corpus, the architecture produces zero false passes. The flags are real, the phantom diff is filtered, and the remaining human-review queue is small enough to clear in minutes of spot-checking.
The technique is offered as a reusable pattern for any context where two substantially different capture paths produce text from the same source material and a model that runs on a consumer laptop is available, either locally or via a free cloud tier.
- Tesseract OCR 5.5.3: Smith, R. (2007). An Overview of the Tesseract
OCR Engine. ICDAR 2007. Command-line
tesseracton unscaled PNG input. - Apple Vision framework:
VisionandVNRecognizeTextRequeston macOS, invoked via a Swift command-line wrapper. Tiled above 7000px image height. - google/gemma-4-26b-a4b-it: Gemma 4 26B instruction-tuned
mixture-of-experts model. Active weights fit in roughly 2 GB of RAM,
making it suitable for overnight batch jobs on consumer hardware. Tested
under Ollama on a 2025 MacBook alongside mistral-small and
magistral-small. Also runs on older Apple Silicon Macs with as little as
8 GB RAM via TurboFieldfare (a Swift and Metal runtime that streams
expert weights from SSD, keeping only the 1.35 GB shared core in
memory), though at approximately one-tenth of local Ollama speed.
Evaluation calls were offloaded to OpenRouter
(
https://openrouter.ai/api/v1/chat/completions), temperature 0, max_tokens 2000, free tier. - Ollama: Local LLM runtime used for model selection trials on the 2025 MacBook.
- TurboFieldfare: Swift and Metal runtime for running gemma-4-26b on Apple Silicon Macs with limited RAM (8 GB minimum) by streaming expert weights from SSD.
- OpenRouter: LLM routing service providing free-tier access to gemma-4-26b, used to parallelise evaluation calls across a dozen pages.
- simonw/rodney: Chromium-based browser automation tool for DOM extraction.
- Python standard library:
json,math,re,collections.Counter,urllib.requestfor deterministic gates, substance check, and HTTP calls. - ImageMagick (
magick): Image tiling for OCR on tall screenshots (-crop {w}x{2000} +repage). - promptfoo: Prompt testing framework for the 16-scenario controlled suite.
The three approved prompts shipped in semantic_check.py and the promptfoo
suite configuration. All three run on google/gemma-4-26b-a4b-it at temperature
0, max_tokens 2000, via the OpenRouter chat completions endpoint
(https://openrouter.ai/api/v1/chat/completions). Each prompt is prepended
with a cache-busting session identifier (UUID, ISO timestamp, probe number)
to prevent response caching. Each prompt is shown exactly as used, with the
substitution markers {a}, {b}, {item}, {evidence}, and {text}.
Used to compress the OCR-recovered text (pass 2 Vision output, enriched with
inlined video transcripts) and the assembled repo text (_all.txt) into 250
to 400 words of plain prose before comparison. The full-coverage instruction
exists to reduce asymmetric compression, where one summariser drops a minor
point that the other keeps (which the judge would then report as a phantom
difference).
Note: the study summaries (the _summary.md files used for revision) use a
different prompt: terse, business-like, action-oriented, with FRAMEWORK
HIGHLIGHT tags. The verification summaries use the prompt below. Different
prompts for the two summary types produce uncorrelated compression profiles,
which is the whole point of the cross-check.
Summarise the following text as lecture-note handout notes. Cover every distinct
point, however minor, and include every section heading, named person,
organisation, framework, and case study. Be faithful to the source; add nothing
that is not in it. Write 250 to 400 words in plain prose with short headings.
Text:
Note: the text to be summarised is appended directly after the code block above.
Compares the two summaries A-then-B and B-then-A to cancel position bias.
The preamble avoids any claim about expander state (the neutral wording
about OCR-hidden content works for both collapsed and expanded captures).
The schema uses absent_from_A and absent_from_B (topics absent from
each side) rather than the earlier ambiguous missing_in_A field name, a
correction applied after the pilot phase when the judge model answered
inconsistently across orders.
You are comparing two summaries of the same course page. Summary A was produced
from an automated OCR capture of the live page; transcripts, interactive
widgets, and collapsed sections may be hidden from OCR. Summary B was produced
from the offline repository capture. Treat them as two sets of lecture-note
handouts written independently of the same lecture.
Questions:
1. Are both summaries about the same subject? Answer same_subject true or false.
2. Is there any whole topic that is absent from summary A but present in summary
B and material to the subject? And in the reverse direction, any whole topic
absent from B but present in A and material? Put topics absent from A in
absent_from_A and topics absent from B in absent_from_B. Whole-topic absence
counts as missing; detail-level differences within a present topic are not
missing items, record them under notes. For each missing item, quote a short
evidence snippet from the side that supposedly contains it. Differences that
are not subject matter (page chrome, student instructions, navigation)
belong in chrome_notes, not in the absent_from lists.
3. Which summary is more complete, or are they materially equivalent?
Reply with JSON only: {"same_subject": true|false, "absent_from_A": [{"item":
"...", "evidence": "short quote"}, ...], "absent_from_B": [...same shape...],
"more_complete": "A"|"B"|"equal", "chrome_notes": [...], "notes": "..."}
Summary A:
{a}
Summary B:
{b}
A targeted, single-concept query. It runs after the deterministic gates and mechanical substance check, only for items that survived both. The prompt names the recovery channels explicitly: OCR and DOM scraping limitations, summariser compression, and the edge-case instruction that OCR-mangled proper nouns that match in context count as discussed. The answer is a single boolean with a short quote. This prompt is the most critical for filtering false suspicions (compression phantoms, synonymously expressed concepts).
A mechanical audit suspects that the concept below is absent from the
source text that follows. The source text was recovered from an online course
page by OCR of a screenshot or by DOM scraping, and AI summaries of two
captures of the same page were compared to produce this suspicion. The
suspicion may be wrong: OCR misreads words, scraping can miss collapsed
sections, and summarisation drops or rephrases points. OCR can also mangle
proper nouns, so a name or citation that differs in spelling but matches in
context counts as discussed.
Concept: {item}
Context quote from the other capture of the same page: {evidence}
Source text:
---
{text}
---
Does the source text genuinely fail to discuss the concept, or does it discuss
the same concept in different words? Reply with JSON only: {"discussed":
true|false, "basis": "one sentence with a short quote if discussed"}