Extract & dedup slide screenshots from a YouTube video, then OCR to Markdown — playbook + lessons learned

yt-slide-extraction-playbook.md

Extracting & Deduping Slide Screenshots From a YouTube Talk

A playbook + lessons learned from pulling a Keynote deck out of a long YouTube video. The slides lived in a small picture-in-picture box in the top-right corner. Goal: get one image per distinct slide, then OCR the deck to Markdown. We only cared about a 15-minute window (05:05:00 → 05:20:00).

---

Pipeline at a glance

1. yt-dlp — download just the 15-minute section (not the whole multi-hour file).
2. ffmpeg — every 10 s, grab a frame, cropped to the PIP region, in one pass.
3. Perceptual hash (dHash) — collapse consecutive identical frames → one per slide.
4. gpt-5.4-mini (OpenAI Responses API) — OCR each unique slide → Markdown.

YouTube URL
  └─ yt-dlp --download-sections  →  section.mp4   (15 min, video-only)
       └─ ffmpeg fps=1/10 + crop  →  cropped/pip_NNN.jpg   (90 frames)
            └─ dHash dedup        →  unique/pip_NNN.jpg    (~36 slides)
                 └─ Responses API  →  slides.md

---

Commands

# 1) Download ONLY the window you need. --force-keyframes-at-cuts makes frame 0
#    land exactly on START, so no offset math later. Video-only = no audio merge.
yt-dlp --download-sections "*05:05:00-05:20:00" --force-keyframes-at-cuts \
  -f "bestvideo[ext=mp4]/bestvideo/best" \
  -o "section.%(ext)s" "https://www.youtube.com/watch?v=VIDEO_ID"

# 2) One frame every 10s, cropped to the PIP, in a single filter chain.
#    crop=W:H:x:y  where  x = source_width - W - right_pad,  y = top_pad
ffmpeg -hide_banner -loglevel warning -i section.mp4 \
  -vf "fps=1/10,crop=1440:800:450:30" -q:v 2 cropped/pip_%03d.jpg

# 3) Dedup by perceptual hash (script below).
uv run --with pillow python dedup.py cropped unique 10

# 4) OCR the unique set to Markdown (script below).
OPENAI_API_KEY=... python ocr_slides.py unique slides.md

---

Lessons learned

1. Confirm the time format before you download

We first read 5:05 as 5 minutes; it was 5 hours 5 minutes (05:05:00). On a multi-hour video, grabbing the wrong window wastes real time. Confirm mm:ss vs hh:mm:ss up front — and when a wrong guess is expensive, ask the one sharp question instead of guessing.

2. Download sections, not whole videos

--download-sections "*START-END" pulls only the window you need. --force-keyframes-at-cuts re-encodes just the cut boundaries so the clip starts exactly on the requested timestamp (frame 0 = your target time, no offset math). -f bestvideo skips the audio merge for speed when you only need frames. Ignore the spammy Late SEI is not implemented warnings during the cut.

3. Combine frame-grab + crop in one ffmpeg pass

-vf "fps=1/10,crop=1440:800:450:30" samples every 10 s and crops in one filter chain. Cropping at extraction time avoids decoding/compressing frames twice. Compute the offset from the source: x = width - crop_w - right_pad.

4. File hashes don't dedup JPEGs — perceptual hashes do

Two screenshots of the same slide are not byte-identical: each frame is encoded independently, so compression noise differs. md5/sha of the bytes fails. Use a perceptual hash (dHash: 8×8 grayscale → 64-bit fingerprint) and dedup by Hamming distance. ~15 lines of Python, robust to noise, no CV stack.

5. Inspect the distance distribution before picking a threshold

Don't guess the Hamming cutoff. Print the sorted consecutive distances and find the gap. Cleanly bimodal (0–6 same-slide, 15+ new-slide) → drop the threshold in the gap. Continuous with a fuzzy middle band (ours had an ambiguous 8–13 zone) → the threshold is a judgment call; show the tradeoff (we tabulated 7 / 10 / 14 → 41 / 36 / 32 kept) and pick deliberately.

6. "Keep first vs last of a run" changes what you capture

"Keep a frame if it differs from the last kept frame" collapses consecutive duplicates while preserving a slide that reappears later. But keeping the first of a run can snapshot a partial animation build; for transcription you often want the last (fully-built) state. Choose deliberately.

7. The agent's image-Read may not show you the pixels

In some harnesses, reading a local image returns a CDN URL instead of embedding the pixels into context. You then can't OCR it yourself — you need a separate vision call. Check how your environment surfaces images before planning a "just read them" step.

8. Shared vision endpoints rate-limit hard under bursts

A shared/cloud vision MCP tool 429'd almost immediately on parallel calls. For batch OCR, call a vision API directly (OpenAI Responses API with input_image base64 data URLs) from one script with a small thread pool and retries. You own the rate, it runs in parallel, and a mini model is plenty for OCR.

9. Pick the right detail level and estimate cost

For OCR use detail:"high" (or original). A 1440×800 slide is ~1125 patches — under the 1536-patch budget for mini models, so it's read at full fidelity. With gpt-5.4-mini's 1.62× multiplier that's ~1.8 k tokens/image; 36 slides ≈ 65 k tokens — cents.

10. Check for secrets early; keep your artifacts

The OpenAI key wasn't in the agent shell's env (the SDK was configured elsewhere). Verify secret presence before scripting a 36-call batch. And keep intermediate folders (section.mp4, cropped/, unique/) — every later stage is reproducible from them, and integrity checks (re-counting frames) catch drift (we had a frame silently go missing between runs).

---

dedup.py — perceptual-hash dedup

# ABOUTME: Dedups cropped Keynote PIP frames by perceptual hash (dHash).
# ABOUTME: Keeps one frame per visually-distinct slide; copies keeps to out dir.
import sys, glob, os, shutil
from PIL import Image

SRC_DIR = sys.argv[1]
OUT_DIR = sys.argv[2]
THRESH = int(sys.argv[3]) if len(sys.argv) > 3 else 10
HASH_SIZE = 8  # 8x8 -> 64-bit fingerprint


def dhash(path, size=HASH_SIZE):
    img = Image.open(path).convert("L").resize((size + 1, size))
    px = list(img.getdata())
    bits = 0
    for r in range(size):
        for c in range(size):
            left = px[r * (size + 1) + c]
            right = px[r * (size + 1) + c + 1]
            bits = (bits << 1) | (1 if left > right else 0)
    return bits


def hamming(a, b):
    return bin(a ^ b).count("1")


files = sorted(glob.glob(os.path.join(SRC_DIR, "pip_*.jpg")))
hashes = [(f, dhash(f)) for f in files]

# Consecutive distance distribution (helps confirm the threshold sits in the gap).
dists = [hamming(hashes[i][1], hashes[i - 1][1]) for i in range(1, len(hashes))]

os.makedirs(OUT_DIR, exist_ok=True)
kept, dropped = [], []
last = None
for f, h in hashes:
    if last is None or hamming(h, last) > THRESH:
        shutil.copy2(f, OUT_DIR)
        kept.append(os.path.basename(f))
        last = h
    else:
        dropped.append(os.path.basename(f))

print(f"threshold(hamming)={THRESH}  total={len(files)}  kept={len(kept)}  dropped={len(dropped)}")
if dists:
    sd = sorted(dists)
    print(f"consecutive distance: min={sd[0]} median={sd[len(sd)//2]} max={sd[-1]}")
    print("non-zero consecutive distances:", [d for d in sd if d > 0])
print("KEPT   :", ", ".join(kept))
print("DROPPED:", ", ".join(dropped))

ocr_slides.py — batch OCR via OpenAI Responses API

# ABOUTME: OCR-transcribes deduped slides via the OpenAI Responses API.
# ABOUTME: Sends each frame as base64 to gpt-5.4-mini (parallel), writes md in order.
import os, sys, glob, base64, json, time, concurrent.futures, urllib.request, urllib.error

DIR = sys.argv[1]
OUT = sys.argv[2]
MODEL = os.environ.get("OCR_MODEL", "gpt-5.4-mini")
KEY = os.environ.get("OPENAI_API_KEY")
MAX_WORKERS = int(os.environ.get("MAX_WORKERS", "6"))
URL = "https://api.openai.com/v1/responses"

PROMPT = (
    "Transcribe ALL text on this slide exactly as written. Preserve structure "
    "(title, headings, bullets, sub-points, captions, labels, footnotes). "
    "Render tables as markdown. Output ONLY the transcription, no commentary."
)

if not KEY:
    sys.exit("OPENAI_API_KEY not set in environment")


def ts(n):
    s = 5 * 3600 + 5 * 60 + (n - 1) * 10  # window starts at 05:05:00
    return f"{s // 3600}:{(s % 3600) // 60:02d}:{s % 60:02d}"


def extract_text(obj):
    for item in obj.get("output", []):
        if item.get("type") == "message":
            for part in item.get("content", []):
                if part.get("type") == "output_text":
                    return part.get("text", "")
    return "[unparseable: " + json.dumps(obj)[:200] + "]"


def call(path, retries=5):
    with open(path, "rb") as f:
        b64 = base64.b64encode(f.read()).decode()
    payload = {
        "model": MODEL,
        "input": [{"role": "user", "content": [
            {"type": "input_text", "text": PROMPT},
            {"type": "input_image", "image_url": f"data:image/jpeg;base64,{b64}", "detail": "high"},
        ]}],
    }
    data = json.dumps(payload).encode()
    for attempt in range(retries):
        req = urllib.request.Request(URL, data=data, headers={
            "Content-Type": "application/json",
            "Authorization": f"Bearer {KEY}",
        })
        try:
            with urllib.request.urlopen(req, timeout=180) as resp:
                return extract_text(json.loads(resp.read()))
        except urllib.error.HTTPError as e:
            body = e.read().decode(errors="replace")[:200]
            if e.code in (429, 500, 502, 503) and attempt < retries - 1:
                time.sleep(2 ** attempt)
                continue
            return f"[ERROR {e.code}: {body}]"
        except Exception as e:
            if attempt < retries - 1:
                time.sleep(2 ** attempt)
                continue
            return f"[ERROR: {e}]"


def main():
    files = sorted(glob.glob(os.path.join(DIR, "pip_*.jpg")))
    results = {}
    with concurrent.futures.ThreadPoolExecutor(max_workers=MAX_WORKERS) as ex:
        futs = {ex.submit(call, p): p for p in files}
        for i, fut in enumerate(concurrent.futures.as_completed(futs), 1):
            name = os.path.basename(futs[fut])
            results[name] = fut.result()
            print(f"[{i}/{len(files)}] {name} ok", flush=True)
    with open(OUT, "w") as f:
        f.write("# Slide Deck — Transcription\n\n---\n\n")
        for idx, p in enumerate(files, 1):
            name = os.path.basename(p)
            n = int(name[4:7])
            f.write(f"## Slide {idx} — `{ts(n)}` (`{name}`)\n\n")
            f.write(results.get(name, "[missing]").strip() + "\n\n---\n\n")
    print("wrote", OUT, "with", len(files), "slides")


if __name__ == "__main__":
    main()

---

Tooling

• yt-dlp, ffmpeg, ffprobe — extract & crop.
• Python stdlib + Pillow (via uv run --with pillow) — perceptual hashing.
• OpenAI Responses API, gpt-5.4-mini, detail:"high" — OCR.
• gh CLI — publish this gist.

Generated from a real extraction session. Names/IDs generalized.