Runnable scope for dotnet/extensions #7516 — MEDI chunk metadata propagation (typed provenance vs loose-dict copy)

00-problem-baseline.cs

// 00-problem-baseline.cs
// SPIKE 0 — Prove or KILL hypothesis H1.
//
// H1: With MEDI PR #7516's current first-wins metadata propagation, a chunk that spans
//     multiple source pages produces a WRONG/INCOMPLETE source citation, AND the typed
//     PageNumber (the field that actually matters for citation) is dropped entirely.
//
// JTBD under test: "Given a retrieved chunk, what page(s) of the source does it cite?"
//   This is the #1 production-RAG requirement (legal/medical/financial attribution).
//
// Run: /home/luquinta/.dotnet/dotnet run 00-problem-baseline.cs
//
// Faithful standalone analogs of the real MEDI types (kept inline so the spike is fast and
// avoids the cross-platform repo's build gremlins). Mapping to the real abstractions:
//   Element  -> Microsoft.Extensions.DataIngestion.IngestionDocumentElement
//               (typed `int? PageNumber`, `IDictionary<string,object?> Metadata`)
//   Chunk    -> Microsoft.Extensions.DataIngestion.IngestionChunk<T>
//               (`IDictionary<string,object> Metadata`)
//   The first-wins merge below mirrors #7516's ElementsChunker.AccumulateMetadata (TryAdd).

using System;
using System.Collections.Generic;
using System.Linq;

// ---- MEDI analogs -----------------------------------------------------------------------

// Mirrors IngestionDocumentElement: typed PageNumber + a loose nullable metadata dict.
sealed class Element
{
    public string Text { get; init; } = "";
    public int? PageNumber { get; init; }                          // TYPED signal (the citation field)
    public Dictionary<string, object?> Metadata { get; } = new();  // loose parse-time bag
}

// Mirrors IngestionChunk<T>: non-null metadata dict. No typed provenance today.
sealed class Chunk
{
    public string Content { get; init; } = "";
    public Dictionary<string, object> Metadata { get; } = new();
}

// ---- #7516 behavior: ElementsChunker.AccumulateMetadata (first-wins) ---------------------

static class Pr7516Chunker
{
    // Merge a run of elements into ONE chunk, replicating #7516:
    //  - copies element.Metadata (loose dict) into the chunk via TryAdd  => FIRST-WINS on collision
    //  - IGNORES the typed PageNumber entirely
    //  - skips null values
    public static Chunk Merge(IEnumerable<Element> elements)
    {
        var els = elements.ToList();
        var chunk = new Chunk { Content = string.Join(" ", els.Select(e => e.Text)) };

        foreach (var el in els)
        {
            foreach (var kv in el.Metadata)
            {
                if (kv.Value is null) continue;          // skip nulls (as #7516 does)
                if (!chunk.Metadata.ContainsKey(kv.Key))  // TryAdd == first-wins
                    chunk.Metadata[kv.Key] = kv.Value!;
            }
            // NOTE: el.PageNumber (the typed citation field) is never propagated. <-- the defect
        }
        return chunk;
    }
}

// ---- Scenario: a contract whose clause spans pages 1, 2, 3 -------------------------------

static class Program
{
    static void Main()
    {
        // A "Limitation of Liability" clause flows across three pages. A semantic chunker keeps
        // the clause together as ONE chunk -> that chunk legitimately spans pages 1-3.
        // Each element carries the typed PageNumber (the real signal). Some readers ALSO stuff a
        // loose "page" key into the dict; we include it to show first-wins lossiness on the dict too.
        var elements = new List<Element>
        {
            new() { Text = "8. LIMITATION OF LIABILITY. In no event shall either party",
                    PageNumber = 1, Metadata = { ["page"] = 1, ["bbox"] = "72,90,540,120" } },
            new() { Text = "be liable for any indirect, incidental, or consequential damages",
                    PageNumber = 2, Metadata = { ["page"] = 2, ["bbox"] = "72,90,540,118" } },
            new() { Text = "arising out of or related to this Agreement, even if advised.",
                    PageNumber = 3, Metadata = { ["page"] = 3, ["bbox"] = "72,90,540,116" } },
        };

        int[] truePages = elements.Select(e => e.PageNumber!.Value).Distinct().OrderBy(p => p).ToArray();

        var chunk = Pr7516Chunker.Merge(elements);

        // ---- The JTBD: cite the source page(s) for this retrieved chunk --------------------
        // A consumer can only look at what the chunk actually carries: chunk.Metadata.
        int[] citedPages = ExtractCitedPages(chunk);

        Console.WriteLine("=== SPIKE 0: baseline (#7516 first-wins) — citation JTBD ===\n");
        Console.WriteLine($"Chunk content : \"{chunk.Content}\"");
        Console.WriteLine($"True source pages (from typed PageNumber): [{string.Join(", ", truePages)}]");
        Console.WriteLine($"Chunk metadata carried       : {{ {string.Join(", ", chunk.Metadata.Select(kv => $"{kv.Key}={kv.Value}"))} }}");
        Console.WriteLine($"Typed page provenance on chunk: (none — chunk has no typed page field)");
        Console.WriteLine($"Pages a consumer CAN cite    : [{string.Join(", ", citedPages)}]\n");

        bool citationCorrect = citedPages.SequenceEqual(truePages);
        bool typedPageDropped = true; // chunk model has no typed page field, and #7516 never set one

        Console.WriteLine("---- VERDICT ----");
        Console.WriteLine($"Citation correct & complete : {(citationCorrect ? "YES ✅" : "NO ❌")}  " +
                          $"(cited [{string.Join(",", citedPages)}] vs true [{string.Join(",", truePages)}])");
        Console.WriteLine($"Typed PageNumber propagated  : {(typedPageDropped ? "NO ❌ (dropped)" : "yes")}");
        Console.WriteLine($"Loose-dict page is lossy     : first-wins kept page={chunk.Metadata.GetValueOrDefault("page")}, dropped {truePages.Length - 1} of {truePages.Length}");
        Console.WriteLine();
        Console.WriteLine(citationCorrect
            ? "H1 FALSIFIED — citation already works. STOP; recommend shrinking/closing #7516."
            : "H1 CONFIRMED — first-wins breaks cross-page citation and drops the typed page field.\n" +
              "             -> a minimal, citation-critical fix is warranted (see spike 01).");
    }

    // A realistic consumer's best effort: read whatever page signal the chunk carries.
    // With #7516 that's only the loose "page" key (first-wins), so it sees a single page.
    static int[] ExtractCitedPages(Chunk chunk)
    {
        if (chunk.Metadata.TryGetValue("page", out var p) && p is int page)
            return new[] { page };
        if (chunk.Metadata.TryGetValue("page_number", out var p2) && p2 is int page2)
            return new[] { page2 };
        return Array.Empty<int>();
    }
}

01-collect-typed-provenance.cs

// 01-collect-typed-provenance.cs
// SPIKE 1 — The MINIMAL fix (Docling-lite: collect typed provenance, derive ranges).
//
// H1 was CONFIRMED by spike 00 (first-wins cites only page 1 of a clause spanning 1-3).
// This spike tests the SMALLEST fix that the prior art validates:
//   - propagate the TYPED page provenance (every framework models position as a typed field:
//     LlamaIndex start_char_idx, Haystack split_idx_start, Docling prov.page_no, Unstructured page_number)
//   - COLLECT, don't overwrite (Docling's collect-all: chunk carries the list; page range is DERIVED)
//   - do NOT touch the loose parse dict, do NOT introduce a standardized-key vocabulary
//
// Claim to validate: this restores correct/lossless citation with a TINY API-surface delta
// and WITHOUT the dict bloat that Adam objects to (we stop copying the loose parse dict).
//
// Run: /home/luquinta/.dotnet/dotnet run 01-collect-typed-provenance.cs

using System;
using System.Collections.Generic;
using System.Linq;

// ---- MEDI analogs (same as spike 00) ----------------------------------------------------

sealed class Element
{
    public string Text { get; init; } = "";
    public int? PageNumber { get; init; }
    public Dictionary<string, object?> Metadata { get; } = new();
}

// The ONLY new type: a typed provenance record. Docling models this as ProvenanceItem
// (page_no, bbox, charspan). We keep the minimal citation-critical shape; bbox is optional.
readonly record struct SourceProvenance(int PageNumber, string? BoundingBox = null);

sealed class Chunk
{
    public string Content { get; init; } = "";
    public Dictionary<string, object> Metadata { get; } = new();

    // NEW (the whole API-surface delta): a typed, collected provenance list on the chunk.
    public List<SourceProvenance> Provenance { get; } = new();

    // DERIVED projections — not stored, computed from the list (Docling pattern: caller derives range).
    public IReadOnlyList<int> Pages => Provenance.Select(p => p.PageNumber).Distinct().OrderBy(p => p).ToList();
    public (int Min, int Max)? PageRange => Provenance.Count == 0 ? null
        : (Provenance.Min(p => p.PageNumber), Provenance.Max(p => p.PageNumber));
}

// ---- The minimal-fix chunker ------------------------------------------------------------

static class CollectChunker
{
    // Merge a run of elements into ONE chunk:
    //  - COLLECT each element's typed PageNumber into chunk.Provenance (lossless)
    //  - do NOT copy the loose parse dict at all (no bloat, no first-wins, no PII columns)
    public static Chunk Merge(IEnumerable<Element> elements)
    {
        var els = elements.ToList();
        var chunk = new Chunk { Content = string.Join(" ", els.Select(e => e.Text)) };

        foreach (var el in els)
        {
            if (el.PageNumber is int page)
            {
                var bbox = el.Metadata.TryGetValue("bbox", out var b) ? b as string : null;
                chunk.Provenance.Add(new SourceProvenance(page, bbox));
            }
        }
        return chunk;
    }
}

static class Program
{
    static void Main()
    {
        var elements = new List<Element>
        {
            new() { Text = "8. LIMITATION OF LIABILITY. In no event shall either party",
                    PageNumber = 1, Metadata = { ["page"] = 1, ["bbox"] = "72,90,540,120" } },
            new() { Text = "be liable for any indirect, incidental, or consequential damages",
                    PageNumber = 2, Metadata = { ["page"] = 2, ["bbox"] = "72,90,540,118" } },
            new() { Text = "arising out of or related to this Agreement, even if advised.",
                    PageNumber = 3, Metadata = { ["page"] = 3, ["bbox"] = "72,90,540,116" } },
        };

        int[] truePages = elements.Select(e => e.PageNumber!.Value).Distinct().OrderBy(p => p).ToArray();

        var chunk = CollectChunker.Merge(elements);
        int[] citedPages = chunk.Pages.ToArray();   // consumer reads the TYPED provenance

        Console.WriteLine("=== SPIKE 1: minimal fix (collect typed provenance) — citation JTBD ===\n");
        Console.WriteLine($"Chunk content : \"{chunk.Content}\"");
        Console.WriteLine($"True source pages            : [{string.Join(", ", truePages)}]");
        Console.WriteLine($"Typed provenance on chunk    : [{string.Join(", ", chunk.Provenance.Select(p => $"p{p.PageNumber}"))}]");
        Console.WriteLine($"Derived PageRange            : {(chunk.PageRange is {} r ? $"{r.Min}-{r.Max}" : "(none)")}");
        Console.WriteLine($"Pages a consumer CAN cite    : [{string.Join(", ", citedPages)}]");
        Console.WriteLine($"Loose parse dict copied      : NO (chunk.Metadata count = {chunk.Metadata.Count}) — no bloat, no PII columns\n");

        bool citationCorrect = citedPages.SequenceEqual(truePages);

        Console.WriteLine("---- VERDICT ----");
        Console.WriteLine($"Citation correct & complete  : {(citationCorrect ? "YES ✅" : "NO ❌")}  " +
                          $"(cited [{string.Join(",", citedPages)}] vs true [{string.Join(",", truePages)}])");
        Console.WriteLine($"Typed PageNumber propagated   : YES ✅ (collected, lossless)");
        Console.WriteLine($"Standardized-key vocab needed : NO ✅ (collect-all derives range; no key registry)");
        Console.WriteLine($"Loose-dict bloat / PII risk   : ELIMINATED ✅ (we stopped copying the parse dict)");
        Console.WriteLine($"API surface added             : 1 typed list (`Provenance`) + 2 derived projections");
        Console.WriteLine();
        Console.WriteLine("RESULT: minimal fix restores the JTBD AND addresses Adam's persist concern in one move —");
        Console.WriteLine("        it is LESS propagation than #7516 does today (drops the loose-dict copy), not more.");
    }
}

02-embed-persist-policy.cs

// 02-embed-persist-policy.cs
// SPIKE 2 — Does the elaborate embed/persist POLICY add clarity, or is it busy work?
//
// The prior art's other big lever (Spring AI excludedEmbedMetadataKeys, LlamaIndex
// excluded_embed/llm_metadata_keys + MetadataMode, Docling excluded_embed class-var) controls
// WHICH metadata gets concatenated into the text that is sent to the embedding model.
// Motivation: page numbers / bboxes are SEMANTIC NOISE — embedding them pollutes the vector.
//
// The question Luis asked: do we need to build that subsystem in MEDI, or is it busy work?
//
// This spike compares two worlds:
//   (A) DICT-STUFFING world  — provenance lives in the loose Metadata dict, the embedder
//       concatenates metadata into the embed text => you NEED an exclusion list to keep page
//       noise out of the vector. (This is the world Spring AI / LlamaIndex live in.)
//   (B) TYPED-PROVENANCE world (spike 01) — provenance is a typed field, NOT part of the
//       content text. You embed chunk.Content; provenance is simply never in the embed text.
//       => the exclusion subsystem is UNNECESSARY for provenance. It's free.
//
// Run: /home/luquinta/.dotnet/dotnet run 02-embed-persist-policy.cs

using System;
using System.Collections.Generic;
using System.Linq;

readonly record struct SourceProvenance(int PageNumber, string? BoundingBox = null);

sealed class Chunk
{
    public string Content { get; init; } = "";
    public Dictionary<string, object> Metadata { get; } = new();   // domain metadata (dict-stuffing world)
    public List<SourceProvenance> Provenance { get; } = new();     // typed (spike-01 world)
    public IReadOnlyList<int> Pages => Provenance.Select(p => p.PageNumber).Distinct().OrderBy(p => p).ToList();
}

static class Program
{
    static void Main()
    {
        // Same cross-page clause chunk. It carries BOTH representations so we can contrast them.
        var chunk = new Chunk { Content =
            "8. LIMITATION OF LIABILITY. In no event shall either party be liable for any " +
            "indirect, incidental, or consequential damages arising out of or related to this Agreement." };
        // dict-stuffing representation:
        chunk.Metadata["page"] = 1;
        chunk.Metadata["bbox"] = "72,90,540,120";
        chunk.Metadata["section"] = "Limitation of Liability";   // a USEFUL domain key (good to embed)
        // typed representation:
        chunk.Provenance.AddRange(new[] { new SourceProvenance(1), new SourceProvenance(2), new SourceProvenance(3) });

        Console.WriteLine("=== SPIKE 2: embed/persist policy — clarity vs. busy work ===\n");

        // ---- World A: dict-stuffing => embedder concatenates metadata into embed text -------
        // Naive embedder (Spring AI / Haystack style): prepend metadata values to the content.
        string embedTextNaive = string.Join("\n", chunk.Metadata.Select(kv => $"{kv.Key}: {kv.Value}")) + "\n" + chunk.Content;
        // To remove the page/bbox NOISE you must configure an exclusion list (the subsystem):
        var excludedEmbedKeys = new HashSet<string> { "page", "bbox" };   // <-- Spring AI excludedEmbedMetadataKeys
        string embedTextWithPolicy = string.Join("\n",
            chunk.Metadata.Where(kv => !excludedEmbedKeys.Contains(kv.Key)).Select(kv => $"{kv.Key}: {kv.Value}"))
            + "\n" + chunk.Content;

        Console.WriteLine("[World A] dict-stuffing + embedder concatenates metadata:");
        Console.WriteLine($"  embed text WITHOUT policy : \"{Truncate(embedTextNaive)}\"  <- page/bbox noise pollutes vector ❌");
        Console.WriteLine($"  embed text WITH exclusion : \"{Truncate(embedTextWithPolicy)}\"  <- needs the exclusion subsystem to fix");
        Console.WriteLine($"  => requires: per-key exclusion lists + MetadataMode plumbing on every embedder. SURFACE.\n");

        // ---- World B: typed provenance => provenance is simply not in the embed text --------
        // The embedder embeds chunk.Content. Provenance is a typed field, never concatenated.
        string embedTextTyped = chunk.Content;
        // (If you WANT the useful domain key 'section' in the vector, you opt it IN explicitly —
        //  opt-in for the few good keys is simpler than opt-out for the many noisy ones.)
        string embedTextTypedPlusSection = $"Section: {chunk.Metadata["section"]}\n{chunk.Content}";

        Console.WriteLine("[World B] typed provenance (spike 01):");
        Console.WriteLine($"  embed text (default)      : \"{Truncate(embedTextTyped)}\"  <- no page noise, NOTHING to exclude ✅");
        Console.WriteLine($"  page still persisted/citeable: pages=[{string.Join(",", chunk.Pages)}] (typed field, not embedded) ✅");
        Console.WriteLine($"  optional opt-in enrichment: \"{Truncate(embedTextTypedPlusSection)}\"  <- add the few GOOD keys explicitly\n");

        Console.WriteLine("---- VERDICT ----");
        Console.WriteLine("Is the embed-exclusion SUBSYSTEM needed in MEDI?  NO ❌ — for PROVENANCE it's busy work.");
        Console.WriteLine("  Reason: typed provenance is never in the embed text, so there is nothing to exclude.");
        Console.WriteLine("  The exclusion subsystem only earns its keep in the dict-stuffing world MEDI should avoid.");
        Console.WriteLine("Persist vs. embed is STILL cleanly separated ✅ — but for free, structurally:");
        Console.WriteLine("  persisted/citeable = typed Provenance; embedded = Content (+ explicit opt-in enrichment).");
        Console.WriteLine();
        Console.WriteLine("RECOMMENDATION: take the typed-provenance spine (spike 01). DEFER the excluded*MetadataKeys /");
        Console.WriteLine("MetadataMode machinery — it solves a problem typed fields don't have. Revisit only if a");
        Console.WriteLine("concrete need to embed-filter DOMAIN (dict) metadata appears.");
    }

    static string Truncate(string s, int n = 70) => s.Replace("\n", " | ").Length <= n
        ? s.Replace("\n", " | ") : s.Replace("\n", " | ").Substring(0, n) + "…";
}

README.md

MEDI chunk metadata propagation: runnable scope for dotnet/extensions #7516

Three self-contained .NET file-based apps that make the proposed scope of dotnet/extensions#7516 runnable and checkable in under a minute. No packages, no repo checkout: each file inlines faithful analogs of the real MEDI types (IngestionDocumentElement with typed PageNumber, IngestionChunk with a loose Metadata dict) so the behavior under discussion is the only variable.

Requires the .NET 10 SDK (file-based apps).

dotnet run 00-problem-baseline.cs        # reproduce the problem
dotnet run 01-collect-typed-provenance.cs # the minimal fix
dotnet run 02-embed-persist-policy.cs     # why the embed-exclusion subsystem is unnecessary

What each app shows

• 00-problem-baseline.cs: the current first-wins (TryAdd) propagation. A chunk spanning source pages 1-3 cites only [1], and the typed PageNumber (the field citation actually needs) is dropped. This is the job that breaks.
• 01-collect-typed-provenance.cs: the minimal fix (Docling-style collect-all): propagate the typed PageNumber per source element into a small typed provenance list; derive Pages/PageRange; stop copying the loose parse dict. Citation becomes lossless ([1,2,3]), no key vocabulary needed, and it is less propagation than the PR does today.
• 02-embed-persist-policy.cs: compares dict-stuffing (needs an excluded*MetadataKeys / MetadataMode exclusion subsystem to keep page noise out of the vector) against typed provenance (provenance is never in the embed text, so there is nothing to exclude). Conclusion: the exclusion machinery is busy work for provenance once it is a typed field.

Prior art surveyed

LlamaIndex (start_char_idx, excluded_embed_metadata_keys, MetadataMode), Haystack (split_idx_start), Docling (prov.page_no, collect-all provenance, excluded_embed), Unstructured (per-field ConsolidationStrategy), Spring AI (excludedEmbedMetadataKeys). Two multi-element patterns exist: Unstructured's per-field strategy (needs a standardized key vocabulary) and Docling's collect-all (needs none). These spikes take the Docling-style path.