LightRAG dates, tagging, metadata

001 - Base summary.md

Adding queryable tags and date ranges

My goal is to allow filtering by tag and by date range. To do that we allow documents to be submitted with an optional array of tags, an optional start time and an optional end time. Tags and start end dates propagate by taking the union of the documents values to knowledge base entries and relations.

Filtering by tag while doing vector lookup can very likely be supported by: Qdrant, Milvus, PostgreSQL, MongoDB, OpenSearch. Others will need to overfetch and filter.

Querying is modified to support only returning chunks and knowledge base entries and relationships that have overlap, are contained or are a superset of the query daterange. Queries without required / forbidden tags and date ranges return the same data as they do now.

The full plan also adds a json metadata field that's not queryable and is transparantly returned when querying but it's not needed for this change.

002 - Scoping.md

Adding scopes

It is often useful to be able to query a dataset while explicitly not taking into account certain information. An example is querying a knowledge base about a book based on what was known in which chapter.

My proposal tags documents as described above but allows configuring tag prefixes as scopes. Knowledge graph generation happens as normal but when a knowledge entry or relation gains an addition that is tagged with a scope it is also added to the entry with the same name in that scope (created if it doesn't yet exist).

Querying with a scoped in the required tags then only considers the knowledge base entries and relations for that scope.

For example if you have two scope prefixes (known-in-chapter: and language:) and you have a two chapter book in english and spanish you would tag:

Upload	Scope tags
Chapter 1, English	known-in-chapter:01, known-in-chapter:02, language:en
Chapter 1, Spanish	known-in-chapter:01, known-in-chapter:02, language:es
Chapter 2, English	known-in-chapter:02, language:en
Chapter 2, Spanish	known-in-chapter:02, language:es

Querying with required language:es would give you only the conclusions you could take from the spanish book. All chapters would be included in the knowledge graph results.

Querying with required known-in-chapter:01 would only give you knowledge graph items with conclusions you could take from the first chapter of both languages.

003 - AI implementation plan tags.md

name	Document metadata filters
overview	Canonical merged plan — upload metadata, nested DateFilter + tag filters, KG union-interval denormalization with kg_union_skip_open_bounds (skip missing bounds instead of ±∞), opt-in vector pushdown (default off), evidence-level KG filtering with optional apply_forbidden_tags_to_kg (default off), relationship dates/tags from relation_chunks evidence, and full WebUI.
todos	id content status models-storage Add DocumentMetadata model, user_* keys on carry-over AND directive whitelists, DocStatusResponse mapping, API limit constants pending id content status upload-plumbing Wire metadata through upload/text/texts, apipeline_enqueue_documents; ainsert v1 shared metadata only (optional metadata applies to all docs in call) pending id content status query-params Add DateFilter, tag filters (normalize at validation), apply_forbidden_tags_to_kg to QueryRequest/QueryParam and REST routes; Ollama chat path out of scope v1 pending id content status filter-engine Implement query_filters.py (has_active_filters fix, orphan-chunk rule), doc/chunk KG post-filter, recompute hooks on all entity_chunks/relation_chunks writers, answer-cache filter hashing pending id content status vector-pushdown Opt-in chunk + optional KG pushdown, kg_union_skip_open_bounds config, per-backend query() pushdown pending id content status webui Upload metadata form, DateFilter controls with presets, apply_forbidden_tags_to_kg toggle, document list display (tags, dates, meta.href link), API types, i18n, Bun tests pending id content status tests API, pipeline (carry-over + FAILED→PENDING retry), query-filter, KG recompute, pushdown, WebUI tests; run targeted subsets pending
isProject	false

Document Metadata and Query Filtering

Scope

In scope

• Upload metadata on all three ingest paths: POST /documents/upload, /text, /texts
• Query filters on POST /query, /query/stream, /query/data: nested date_filter, required_tags, required_tags_mode (and | or), forbidden_tags, apply_forbidden_tags_to_kg
• SDK/pipeline plumbing via apipeline_enqueue_documents
• Persistence through pipeline status transitions and API readback via DocStatusResponse
• Opt-in vector-backend pushdown for chunks_vdb and (when enabled) entities_vdb / relationships_vdb (default off)
• KG union-interval denormalization of dates/tags on entities and relations, recomputed on merge/purge
• kg_union_skip_open_bounds config (default true) — missing contributor bounds skip union min/max instead of widening to ±∞
• KG query filtering — evidence-level post-filter (default); optional union pushdown as coarse gate
• Per-backend schema/migration notes and compatibility when pushdown is disabled or schema not upgraded
• Full WebUI — upload metadata form, query filter controls with archive presets, document list display, API client types, i18n, Bun unit tests

Out of scope (v1)

• Query filtering on user_meta / meta — roundtrip only
• PATCH /documents/{id}/metadata and post-upload metadata edits (see Re-upload policy below)
• Metadata on scan-discovered files without upload metadata
• starts_in / ends_in range modes
• Ollama-compatible chat query path (ollama_api.py builds QueryParam separately) — no date_filter / tag filters in v1; REST /query* only
• Per-document metadata on ainsert list input (shared blob only — see SDK metadata)
• Backfill/migration of filter fields on pre-existing chunks or KG rows (see Existing corpora)

Storage contract

Store user metadata on doc_status.metadata under dedicated, namespaced keys:

Key	Type	Notes
user_start_date	ISO-8601 string with offset or absent	Optional document effective start; timezone-aware; offset preserved at rest
user_end_date	ISO-8601 string with offset or absent	Optional document effective end; timezone-aware; offset preserved at rest
user_tags	list[str]	Normalized lowercase at write time; may be empty
user_meta	dict	Arbitrary JSON object; roundtrip only — never queryable

Upload without dates is valid — zero, one, or both bounds may be present. When a bound is present, it must be timezone-aware.

Timezone policy (all surfaces)

Principle: all datetimes are timezone-aware on input and in storage. Do not convert to UTC for persistence — future calendar dates in a named offset (e.g. legislation effective 2030-03-30T00:00:00+01:00) cannot be reliably re-projected through UTC when DST rules change. Comparisons use instant semantics at query time only.

Surface	Rule
Upload API (DocumentMetadata)	Non-null dates must be timezone-aware; reject naive → 422
Query API (DateRangeFilter)	Same
Storage (user_* keys)	ISO-8601 strings with offset preserved (Z / +00:00 allowed; +01:00 etc. kept as submitted). Never store naive. Never astimezone(UTC) before write.
SDK (ainsert / apipeline_enqueue_documents)	Optional shared metadata; datetimes must be aware; strings must include offset or Z
WebUI	Date pickers emit timezone-aware ISO with explicit offset (operator's chosen zone or Z — not naive, not offset-stripped)
KG denormalized fields (filter_start_date / filter_end_date)	Same offset-preserving storage; union min/max picks the contributing doc's stored string for the winning bound
Interval / ordering checks	Compare aware datetimes by instant (timestamp() or equivalent); no UTC conversion required for storage

No implicit local timezone anywhere in the pipeline.

Add all four user_* keys to both whitelists in utils_pipeline.py:

Whitelist	Purpose
_DOC_STATUS_METADATA_CARRY_OVER_KEYS	Survives ordinary status transitions (PARSING → PROCESSING → PROCESSED/FAILED, etc.)
_DOC_STATUS_METADATA_DIRECTIVE_KEYS	Survives FAILED→PENDING reset and interrupted-doc normalization via doc_status_reset_metadata()

Carry-over alone is insufficient. Reset keeps only directive keys; without them, user metadata is wiped on manual retry.

Regression test (required): upload with metadata → force FAILED → manual retry (/documents/reprocess_failed or scan retry) → user_start_date / user_end_date / user_tags / user_meta still present on doc_status.metadata and DocStatusResponse.

Expose in DocStatusResponse as top-level start_date, end_date, tags, meta.

flowchart LR
    upload[Upload endpoints] --> enqueue[apipeline_enqueue_documents]
    enqueue --> docStatus[doc_status.metadata user_* keys]
    query[Query endpoints] --> dateFilter[DateFilter predicate]
    dateFilter --> pushdown{pushdown active?}
    pushdown -->|yes| vdbFilter[chunks_vdb.query]
    pushdown -->|no| postFilter[post-retrieval filter]
    vdbFilter --> postFilter
    postFilter --> operate[operate.py]

Loading

Upload API design

Shared Pydantic model

class DocumentMetadata(BaseModel):
    start_date: datetime | None = None
    end_date: datetime | None = None
    tags: list[str] = Field(default_factory=list)
    meta: dict[str, Any] = Field(default_factory=dict)

Validation:

• start_date <= end_date when both present — compare by instant (both must be timezone-aware)
• Timezone-aware only: every non-null start_date and end_date must be timezone-aware — reject naive datetimes with HTTP 422 ("Date must include a timezone offset or Z"). Naive values are never passed through or stored.
• Accept ISO-8601 with explicit offset or Z (e.g. 2020-01-15T00:00:00Z, 2030-03-30T00:00:00+01:00); document in API docs
• Serialize for storage via serialize_aware_datetime(dt) — ISO-8601 with offset preserved; optional cosmetic normalize Z ↔ +00:00 only when offset is actually UTC. Do not astimezone(UTC) for persistence.
• Tags: normalize at write via shared normalize_tags() (see Tag normalization); enforce limits below
• Tag / meta limits — frozen constants in constants.py (see API limits below)
• Storage: persist user_* dates as offset-preserving ISO-8601 strings. Reject naive before any write.

API limits (constants.py)

Freeze before implementation; validate on upload and query:

Constant	Value	Applies to
MAX_DOCUMENT_TAGS	64	user_tags per document
MAX_DOCUMENT_TAG_CHARS	128	each tag after normalization
MAX_QUERY_FILTER_TAGS	32	required_tags and forbidden_tags each
MAX_DOCUMENT_META_BYTES	262_144 (256 KiB)	serialized user_meta JSON at upload only
MAX_DOCUMENT_META_DEPTH	8	nested object depth in user_meta

Validate user_meta size on upload (serialized UTF-8 byte length); query path does not accept meta.

Tag normalization

normalize_tags(raw: list[str]) -> list[str] — shared upload and query (required_tags / forbidden_tags).

Per-tag pipeline (accepts full Unicode input):

1. Strip leading/trailing whitespace; skip empty inputs
2. Unicode NFKC → casefold
3. Map to allowed charset [a-z0-9:_-] only: decompose and drop combining marks (Latin accents → base letters); any remaining disallowed rune → _
4. Collapse repeated _; strip leading/trailing _
5. Reject tag if empty after normalization or if length > MAX_DOCUMENT_TAG_CHARS
6. Dedupe preserving first-seen order; cap count at MAX_DOCUMENT_TAGS / MAX_QUERY_FILTER_TAGS

Final stored/filter tags must match ^[a-z0-9:_-]+$. Examples: Patent → patent; Årsrapport → arsrapport; channel/ENG → channel_eng.

Re-upload and duplicate ingest policy (explicit)

• Re-uploading the same filename is a duplicate under existing dedup rules — it does not update metadata on the canonical processed document.
• Duplicate FAILED stubs (dup-*): upload metadata is written to the stub row that receives the enqueue (the duplicate attempt), not merged onto metadata.original_doc_id. API readback shows metadata on whichever row was written — operators see tags/dates on the FAILED duplicate stub for review, while the canonical processed doc keeps its prior metadata (often none). Filtering always uses full_doc_id → that row's metadata — duplicate-stub metadata does not scope retrieval for the original document's chunks.
• Metadata cannot change after ingest in v1 without a future PATCH /documents/{id}/metadata API (listed under Future optimizations).
• Operators who need corrected dates/tags on the canonical doc must delete and re-ingest with new metadata, or wait for PATCH.

Test: enqueue duplicate with metadata → assert metadata on dup-* FAILED stub; canonical doc-* unchanged; filtered query uses canonical doc's metadata only.

Endpoint changes

Endpoint	Metadata input
POST /documents/text	Optional metadata: DocumentMetadata
POST /documents/texts	Optional shared metadata (all texts in batch)
POST /documents/upload	Optional multipart metadata JSON string

Thread through pipeline_index_texts, pipeline_enqueue_file, apipeline_enqueue_documents.

SDK metadata (ainsert / apipeline_enqueue_documents)

Entry point	v1 metadata semantics
POST /documents/text	Optional per-request metadata
POST /documents/texts	One shared metadata for all texts in the batch
POST /documents/upload	One shared metadata for all files in the multipart batch
apipeline_enqueue_documents(..., document_metadata=...)	Optional shared metadata applied to every document in the batch
LightRAG.ainsert(..., metadata=...)	Optional shared metadata for all strings in input (single or list) — mirrors /texts; no per-item metadata map in v1

To attach different metadata per document from the SDK, call apipeline_enqueue_documents once per document (or use REST /text per doc). Do not extend ainsert with list[DocumentMetadata] in v1.

Query API design

Nested DateFilter (replaces flat daterange + incomplete_dates_policy)

Three orthogonal knobs: range relation, document shape, undated OR-semantics.

class DateRangeFilter(BaseModel):
    start: datetime | None = None  # absent => -∞ for interval math
    end: datetime | None = None      # absent => +∞ for interval math
    # model_validator: non-null start/end must be timezone-aware; start <= end by instant; serialize with offset preserved

class DateRangeMode(str, Enum):
    OVERLAP = "overlap"
    CONTAINED_BY = "contained_by"
    CONTAINS = "contains"

class DocumentDateShape(str, Enum):
    ANY = "any"
    DATED = "dated"
    BOUNDED = "bounded"
    HAS_START = "has_start"
    HAS_END = "has_end"

class UndatedMatch(str, Enum):
    EXCLUDE = "exclude"
    INCLUDE_OR = "include_or"
    ONLY = "only"

class RequiredTagsMode(str, Enum):
    AND = "and"   # default — document must contain all listed tags
    OR = "or"     # document must contain at least one listed tag

class DateFilter(BaseModel):
    range: DateRangeFilter | None = None
    range_mode: DateRangeMode = DateRangeMode.OVERLAP
    shape: DocumentDateShape = DocumentDateShape.ANY
    undated: UndatedMatch = UndatedMatch.EXCLUDE

class QueryRequest(BaseModel):
    ...
    date_filter: DateFilter | None = None
    required_tags: list[str] = Field(default_factory=list)
    required_tags_mode: RequiredTagsMode = RequiredTagsMode.AND
    forbidden_tags: list[str] = Field(default_factory=list)
    apply_forbidden_tags_to_kg: bool = False

Query range validation (on DateRangeFilter, shared with upload via helper in query_filters.py e.g. validate_date_pair(start, end)):

• Every non-null bound must be timezone-aware — reject naive with 422 (same rule as upload)
• start <= end when both present (compare by instant)
• Interval predicates compare aware bounds by instant; stored document ranges are offset-preserving ISO strings parsed back to aware datetimes

Add matching types to QueryParam (enums in lightrag/query_filters.py).

apply_forbidden_tags_to_kg (default off)

Scope	required_tags	forbidden_tags when apply_forbidden_tags_to_kg=False	forbidden_tags when apply_forbidden_tags_to_kg=True
Documents / chunks	always	applied	applied
Entities / relations (union tags + evidence filter)	via eligible docs; union gate uses required_tags_mode	not applied to KG union tag sets	applied to filter_tags on entity/relation payloads and evidence filter

Rationale: KG objects union tags across all contributing documents, so they rapidly accumulate the full corpus tag vocabulary. Applying forbidden_tags to KG by default would drop most entities/relations in mixed corpora (e.g. forbid newspaper → entity also mentioned in a tagged newspaper doc loses the whole node). required_tags always applies to document eligibility (and thus to evidence-level KG filtering). Opt in when operators want union-tag hygiene (e.g. forbid draft on KG hits).

Include apply_forbidden_tags_to_kg in compute_args_hash when filters active.

Document classification

Each document falls into one of four shape classes (derived from user_start_date / user_end_date):

Class	Condition
undated	neither bound
start_only	start only
end_only	end only
complete	both bounds

Effective range for interval math (after shape gate passes): missing start → −∞, missing end → +∞.

shape gate

DocumentDateShape	Eligible shape classes
any	all four
dated	start_only, end_only, complete
bounded	complete only
has_start	start_only, complete
has_end	end_only, complete

range_mode predicates (on effective doc range vs query range)

Let Q = query range, D = doc effective range (with ±∞ for open bounds).

DateRangeMode	Predicate
overlap	D_start <= Q_end AND D_end >= Q_start
contained_by	D_start >= Q_start AND D_end <= Q_end
contains	D_start <= Q_start AND D_end >= Q_end

Point-in-time ("in force on date D"): set range.start = range.end = D; overlap and contains agree for typical open-ended law/patent docs.

undated OR-semantics

Evaluated after computing base = shape_matches(doc) AND range_matches(doc, range_mode):

UndatedMatch	Result
exclude	base only (fully undated docs fail unless shape=any and overlap with unbounded — they don't; so undated excluded)
include_or	base OR doc is undated — only fully undated, not partial docs outside range
only	doc is undated (ignore range, range_mode, shape)

Filter semantics — tags

required_tags_mode controls how required_tags is interpreted when the list is non-empty. forbidden_tags is unchanged (must contain none of the listed tags).

required_tags_mode	Predicate (when required_tags non-empty)
and (default)	required_tags ⊆ doc.user_tags — must contain all
or	required_tags ∩ doc.user_tags ≠ ∅ — must contain at least one

required_tags	forbidden_tags	Effect
omitted or []	omitted or []	no tag filter — all docs match (subject to date filter); required_tags_mode ignored
non-empty	empty	apply required_tags_mode (and or or)
empty	non-empty	must contain none of the forbidden tags
non-empty	non-empty	required predicate (per mode) and no forbidden tags

Omitting required_tags or passing [] are equivalent. Default required_tags_mode: "and" preserves prior AND-only behavior.

Query-side tag normalization: apply normalize_tags() to required_tags and forbidden_tags during validation (same charset rules as upload).

Examples:

• required_tags: ["patent", "legal"], required_tags_mode: "and" — doc must have both tags
• required_tags: ["newspaper", "letter"], required_tags_mode: "or" — doc with either tag matches (archive discovery across formats)

Include required_tags_mode in filter cache hash when filters active (see LLM answer cache).

has_active_filters gate

Decision: extend date_active — shape and undated are first-class filters, not silent no-ops when range is absent.

def has_active_filters(query_param) -> bool:
    df = query_param.date_filter
    if df is None:
        date_active = False
    else:
        date_active = (
            df.range is not None
            or df.shape != DocumentDateShape.ANY
            or df.undated != UndatedMatch.EXCLUDE
        )
    tag_active = bool(query_param.required_tags) or bool(query_param.forbidden_tags)
    return date_active or tag_active

When date_filter.range is None: skip interval / range_mode predicates (range component matches all docs); shape and undated still apply. Examples:

date_filter	Effect
shape: bounded only	Only docs with both date bounds
undated: only only	Only fully undated docs
undated: include_or only	All fully undated docs (no dated docs unless range also set)
shape: dated + range: …	Dated docs matching range OR per undated

Empty date_filter: {} (all defaults) → inactive (date_active false).

forbidden_tags alone activates filtering (document scope). KG forbidden-tag enforcement only runs when both forbidden_tags non-empty and apply_forbidden_tags_to_kg=True.

Archive-oriented presets (WebUI + SDK helpers)

Canonical preset table — sync design/document-metadata-query-filters.md § WebUI presets to match before WebUI work:

Preset	range_mode	shape	undated	Use case
Broad retrieval	overlap	any	exclude	Default RAG
Archive discovery	overlap	any	include_or	Newspapers, letters: period + uncatalogued
Legislation / in force	contains	has_start	exclude	Law/treaty/patent valid for whole query period
Activity in period	overlap	has_start	exclude	Patents, trials, news touching period
Wholly within period	contained_by	has_start	exclude	Parliament sessions, events wholly inside window
Strict catalog	overlap	bounded	exclude	Annual reports, complete FY ranges
Uncatalogued only	(ignored)	(ignored)	only	Admin / cleanup — only fully undated docs

Advanced UI still exposes range_mode, shape, and undated individually.

Example — patents active throughout 2010–2015:

{
  "date_filter": {
    "range": { "start": "2010-01-01T00:00:00Z", "end": "2015-12-31T23:59:59Z" },
    "range_mode": "contains",
    "shape": "has_start",
    "undated": "exclude"
  },
  "required_tags": ["patent"]
}

Example — 1930s OR undated letters (tag OR across formats):

{
  "date_filter": {
    "range": { "start": "1930-01-01T00:00:00Z", "end": "1939-12-31T23:59:59Z" },
    "range_mode": "overlap",
    "shape": "any",
    "undated": "include_or"
  },
  "required_tags": ["newspaper", "letter"],
  "required_tags_mode": "or"
}

Compatibility invariant

When has_active_filters is false, query behavior is bit-for-bit identical to today — regardless of LIGHTRAG_VECTOR_METADATA_FILTERS:

• No metadata_filter on BaseVectorStorage.query()
• No oversampling on FAISS / NanoVectorDB
• No doc_status scans or post-retrieval pruning
• No filter fields in compute_args_hash

Query-time filtering implementation

Shared filter helpers (lightrag/query_filters.py)

• Enums: DateRangeMode, DocumentDateShape, UndatedMatch, RequiredTagsMode

• Models: DateRangeFilter, DateFilter, VectorQueryFilter (normalized query-side filter bundle)

• classify_doc_dates(metadata) -> DocDateClass

• parse_stored_datetime(iso: str) -> datetime — parse stored string to timezone-aware datetime

• serialize_aware_datetime(dt: datetime) -> str — persist with offset preserved (reject naive)

• effective_doc_range(metadata) -> (start, end) with ±∞ (parse stored ISO strings to aware datetimes; open bounds unchanged)

• normalize_tags(tags) -> list[str] — shared upload + query; output charset [a-z0-9:_-] only (see Tag normalization)

• validate_date_pair(start, end) -> None — reject naive, enforce ordering by instant, no UTC conversion for storage; raises ValueError for Pydantic validators

• document_matches_date_filter(metadata, date_filter) -> bool — when range is None, range predicates are skipped; shape + undated still apply

• document_matches_filter(metadata, vector_filter) -> bool — date + tags (respects required_tags_mode)

• resolve_eligible_doc_ids(doc_status, vector_filter) -> set[str]

• build_eligible_chunk_ids(doc_status, eligible_doc_ids) -> set[str]

• build_vector_query_filter(query_param) -> VectorQueryFilter | None

• has_active_filters(query_param) -> bool

• build_backend_filter_expr(filter) -> ... for pushdown backends

• build_filter_cache_hash(query_param) -> str | None — stable hash of all active filter fields; None when has_active_filters is false

Chunks without doc_status metadata (v1 rule)

Post-retrieval filtering resolves text_chunks[chunk_id].full_doc_id → doc_status row. Several paths do not guarantee a row with user_* metadata:

Path	Notes
ainsert_custom_kg	Writes text_chunks / chunks_vdb; may use full_doc_id with no doc_status row (rebuild_vdb.py documents this)
Scan-discovered files	No upload metadata (out of scope to add at scan)
Orphan / legacy chunks	full_doc_id missing from doc_status

v1 rule: if full_doc_id has no doc_status row, or the row exists but has no user_* date/tag fields, treat the chunk's document as undated and untagged for filtering:

• Included when undated is include_or or only, or when no date/tag filters are active
• Excluded by shape != any, tag filters, and date range predicates (undated docs fail shape/range unless undated allows them)
• KG evidence from such chunks follows the same rule via eligible_chunk_ids

Test: ainsert_custom_kg chunk + filtered query (strict date/tags) → custom-KG evidence excluded; undated: include_or may include it.

LLM answer cache (correctness)

Today operate.py answer-cache compute_args_hash for kg_query (~L4641) and naive_query (~L6651) does not include retrieval filter fields — filtered and unfiltered queries with the same text could share a cache entry (bug).

Required: when has_active_filters(query_param), append a stable filter fingerprint to both answer-cache hashes via build_filter_cache_hash():

• All date_filter fields (range, range_mode, shape, undated) when date-active
• required_tags, required_tags_mode, forbidden_tags, apply_forbidden_tags_to_kg when tag-active

When filters inactive, omit filter fields (preserve today's cache behavior).

Keyword-extraction cache (cache_type="keywords" in extract_keywords, ~L4898): stay filter-agnostic — keywords depend on query text/mode, not retrieval scope.

Chunk filter-field stamping

When enable_vector_metadata_filters=true, stamp denormalized filter fields on chunk rows at pipeline chunk upsert — immediately before chunks_vdb.upsert(chunks) (~line 5403 in the processing path that builds the chunks dict).

Hook: stamp_chunk_filter_metadata(chunks, doc_status_metadata) -> None (in query_filters.py or utils_pipeline.py) copies from the parent doc's user_* metadata into each chunk payload:

• filter_start_date, filter_end_date, filter_tags (normalized tag list)

Called only when pushdown is enabled; post-retrieval filtering still reads doc_status directly when pushdown is off.

Two-tier strategy (chunks)

Tier	When	What
Pushdown	filters active AND enable_vector_metadata_filters=true AND supports_metadata_filters AND chunks_vdb	query(..., metadata_filter=filter)
Oversample + in-Python	filters active AND no native pushdown AND chunks_vdb	top_k * factor only when metadata_filter is not None; prune in Python
Post-retrieval	filters active	eligible doc/chunk ID filter in operate.py

Include build_filter_cache_hash() in answer-cache paths when has_active_filters (see LLM answer cache).

Knowledge graph filtering and denormalization

KG objects (entities, relations) are shared across documents. Filtering uses two layers: denormalized summaries (optional pushdown) and evidence-level truth (always when filters active).

flowchart TD
    merge[merge_nodes_and_edges / purge] --> recompute[recompute_kg_filter_metadata]
    recompute --> graph[graph node/edge fields]
    recompute --> vdb[entities_vdb / relationships_vdb payload]
    query[Filtered query] --> eligible[eligible_doc_ids + chunk_ids]
    eligible --> evidence[Evidence filter via entity_chunks / relation_chunks]
    evidence --> prune[Prune KG-attached text to eligible chunks]
    query --> pushdown{KG pushdown enabled?}
    pushdown -->|yes| unionGate[Union interval + tag union coarse gate]
    unionGate --> evidence
    pushdown -->|no| evidence

Loading

Evidence-level filter (always when filters active)

After entity/relation vector search in _perform_kg_search:

1. Resolve eligible_doc_ids / eligible_chunk_ids from doc_status (same as chunks).
2. Keep entity/relation if any chunk in entity_chunks / relation_chunks (authoritative; fallback truncated graph source_id) intersects eligible_chunk_ids.
3. Prune evidence: when resolving KG-attached text (_find_related_text_unit_from_*), only use eligible chunks — citations stay in-scope even if the node description summarizes broader corpus.
4. Optional oversample on entities_vdb / relationships_vdb when filters active (same pattern as FAISS chunks).

For range_mode: contains / contained_by, union pushdown alone is unsafe; evidence filter is required (default path).

Union-interval denormalization (entities and relations)

Policy (fixed — not implementer choice):

Store	When filter_* fields are written
Graph nodes / edges	Always — recompute on every merge/purge/custom-KG upsert (cheap; supports evidence filter bookkeeping and purge recompute even when pushdown is off)
entities_vdb / relationships_vdb payloads	Only when enable_vector_metadata_filters=true (pushdown coarse gate)

Stamp summary fields:

Field	Computation
filter_start_date	min of contributor doc starts by instant; store the winning contributor's ISO string (offset preserved)
filter_end_date	max of contributor doc ends by instant; store the winning contributor's ISO string (offset preserved)
filter_tags	set union of contributor user_tags
has_undated_evidence	true if any contributor doc has no date bounds
filter_doc_count	optional; count of distinct contributing full_doc_id

Contributor set: for each entity/relation, walk entity_chunks / relation_chunks → chunk IDs → text_chunks[chunk_id].full_doc_id → load doc_status metadata (missing row → undated/untagged per orphan rule). Do not use endpoint entities' union for relation dates.

Recompute hook: recompute_kg_filter_metadata(entity_or_relation_key, chunk_ids, doc_status, text_chunks) — must run after any write that changes entity_chunks / relation_chunks or graph membership.

KG filter_* recompute triggers (complete)

Write path	Module	Recompute
Extraction merge / upsert	merge_nodes_and_edges	All touched entities + relations
Document purge / resume	_purge_kg_contributions	Surviving entities + relations
Custom KG insert	ainsert_custom_kg	Touched entities + relations (often undated — no doc metadata)
Manual entity create	acreate_entity	New entity (undated until evidence docs exist)
Manual entity edit / merge	aedit_entity	Edited entity + affected relations
Entity rename	aedit_entity rename branch	New name: recompute; old name: drop graph filter_* when node removed
Manual relation create/edit	utils_graph.py relation CRUD	Affected edge
Relation delete	graph delete paths	Remove edge filter_*

Manual graph CRUD without document evidence: entities/relations may have entity_chunks / relation_chunks pointing at custom-KG or orphan chunks — recompute still runs; result is undated / untagged until evidence links to docs with user_* metadata. No manual date/tag API on graph nodes in v1.

Extend meta_fields on entities_vdb / relationships_vdb when pushdown enabled (graph properties always maintained separately).

Existing corpora (no backfill)

• No migration/backfill for chunks or KG rows ingested before this feature.
• Post-retrieval filters read live doc_status.metadata — docs without user_* keys behave as undated / untagged for filter purposes.
• Enabling pushdown on an existing corpus: old chunk/KG vector rows lack filter_* payload fields until those documents are reprocessed (or corpus cleared and re-ingested). Until then, pushdown may treat them as empty filter fields (undated, no tags) — document this in operator docs.
• Do not add a one-time sweep job in v1.

Configuration: kg_union_skip_open_bounds (default true)

Env / vector_db_storage_cls_kwargs / LightRAG global config:

kg_union_skip_open_bounds: bool = True  # DEFAULT ON — do not let missing bounds widen union to ±∞

Contributor bound	kg_union_skip_open_bounds=true (default)	false (legacy permissive)
Missing start	Skip for min(start) — does not widen to −∞	Treated as −∞
Missing end	Skip for max(end) — does not widen to +∞	Treated as +∞
Fully undated doc	Sets has_undated_evidence=true only	Same
No contributor has any date	Both filter_start_date and filter_end_date null (undated KG object)	Same

When true (recommended for archives):

Contributor doc bound	Effect on union min/max
Missing start	Skip for min(start) — do not treat as −∞
Missing end	Skip for max(end) — do not treat as +∞
Fully undated doc	Contributes to has_undated_evidence=true only; does not move interval
At least one doc with start	filter_start_date = min(starts present)
At least one doc with end	filter_end_date = max(ends present)
No contributor has any date bound	filter_start_date and filter_end_date remain null (KG object is undated for interval purposes; date_filter.undated / shape gates apply)
Contributors have starts but none have end	filter_end_date stays null (open end = +∞ for query interval math on the summary row, same as single doc with no end)

Query interval math on the summary row: null filter_end_date after union still means +∞ for range predicates when the aggregate is "open ended" (no contributor supplied an end). Null on both bounds → undated KG object; date_filter.shape / undated gates apply.

When kg_union_skip_open_bounds=false (legacy permissive): missing start → −∞, missing end → +∞ in union (one open-ended law doc widens entire entity to eternity).

Document in env.example.

Union pushdown vs evidence (KG)

Stage	Role
Union interval + tag union on VDB	Coarse candidate gate when pushdown on; same DateFilter + required_tags (per required_tags_mode); forbidden_tags only if apply_forbidden_tags_to_kg
Evidence ∩ eligible_chunk_ids	Authoritative keep/drop for entities/relations
Pruned chunk text	Authoritative citations

For range_mode: overlap, union-only may suffice as gate; for contains / contained_by, always run evidence filter after search.

Relationship dates and tags (how they are set)

Relations do not inherit dates/tags from endpoint entities. They are derived only from relation evidence:

flowchart LR
    relChunks[relation_chunks edge key] --> chunkIds[chunk IDs]
    chunkIds --> fullDoc[text_chunks.full_doc_id]
    fullDoc --> docMeta[doc_status user_start_date / user_end_date / user_tags]
    docMeta --> union[Union min/max dates + tag set + has_undated_evidence]
    union --> graphEdge[graph edge properties]
    union --> relVdb[relationships_vdb payload]

Loading

Steps:

1. After extraction/merge, read relation_chunks[canonical_edge_id] (or src|tgt key per storage) → list of chunk IDs.
2. Map each chunk → full_doc_id via text_chunks.
3. Load each doc's user_start_date, user_end_date, user_tags from doc_status.metadata.
4. Apply same union rules as entities (kg_union_skip_open_bounds, tag union, has_undated_evidence).
5. Write to graph edge properties (new fields parallel to entity node) and relationships_vdb payload on upsert; recompute on purge.

Example: edge "Company —[licensed]→ Patent" with chunks from 2010 patent doc (tags patent) and 2015 amendment (tags patent, legal):

• filter_start_date = min(2010, 2015)
• filter_end_date = per union rules
• filter_tags = {patent, legal}

If amendment chunk is purged, recompute from surviving chunks only.

Entities use entity_chunks[entity_name] with identical aggregation — no separate manual date API on graph CRUD in v1; dates/tags on KG objects are always derived from document metadata of evidence docs.

apply_forbidden_tags_to_kg in KG predicate

def kg_union_tags_match(
    filter_tags, required, forbidden, apply_forbidden_to_kg, required_mode=RequiredTagsMode.AND
) -> bool:
    if required:
        if required_mode == RequiredTagsMode.AND:
            if not required.issubset(filter_tags):
                return False
        else:  # OR
            if not required.intersection(filter_tags):
                return False
    if apply_forbidden_to_kg and forbidden and filter_tags.intersection(forbidden):
        return False
    return True

Evidence-level filter ignores union filter_tags for forbidden unless apply_forbidden_tags_to_kg; it only requires eligible chunks (documents already passed doc-level forbidden_tags).

WebUI

API types (lightrag_webui/src/api/lightrag.ts)

export type DateRangeMode = 'overlap' | 'contained_by' | 'contains'
export type DocumentDateShape = 'any' | 'dated' | 'bounded' | 'has_start' | 'has_end'
export type UndatedMatch = 'exclude' | 'include_or' | 'only'
export type RequiredTagsMode = 'and' | 'or'

export type DateFilter = {
  range?: { start?: string | null; end?: string | null }
  range_mode?: DateRangeMode       // default overlap
  shape?: DocumentDateShape        // default any
  undated?: UndatedMatch           // default exclude
}

// QueryRequest
date_filter?: DateFilter | null
required_tags?: string[]
required_tags_mode?: RequiredTagsMode  // default 'and'
forbidden_tags?: string[]
apply_forbidden_tags_to_kg?: boolean  // default false

Upload UI (UploadDocumentsDialog.tsx)

Collapsible optional metadata: start date, end date, tags (chips), meta (JSON). Shared across batch.

• Client-side: start <= end when both set; serialize as timezone-aware ISO with explicit offset (never naive)
• Surface API 422 for naive dates or ordering violations via existing toast pattern

Query UI (QuerySettings.tsx)

Document filters collapsible group:

Control	Maps to
Preset dropdown	sets range_mode + shape + undated (see preset table)
Date range start / end	date_filter.range
Match mode (advanced)	date_filter.range_mode — "Touches period" / "Wholly within" / "Covers entire period"
Document dates (advanced)	date_filter.shape
Undated handling (advanced)	date_filter.undated
Required / forbidden tags	required_tags, forbidden_tags
Required tag match (advanced)	required_tags_mode — "All tags (AND)" / "Any tag (OR)"; default AND
Apply forbidden tags to KG (advanced)	apply_forbidden_tags_to_kg — default off; tooltip explains union-tag accumulation

Defaults in settings.ts: no date_filter, empty tag lists, required_tags_mode: 'and', apply_forbidden_tags_to_kg: false. Omit date_filter key entirely when inactive.

• Query date range pickers: emit timezone-aware ISO with explicit offset (never naive)

Document list UI (DocumentManager.tsx)

Tags column, date range column, details dialog for start_date / end_date / tags / meta.

meta.href external link (display-only)

meta.href is a display / navigation convenience only — never queryable, never denormalized to chunks or KG, never shown in filter UI. Do not document it alongside tags or dates as a filter dimension.

When doc.meta?.href is a non-empty string and passes URL validation, show a link icon beside the document name in the table row and again in DocumentStatusDetailsDialog.

Rule	Detail
Source	Top-level DocStatusResponse.meta.href only (not nested paths, not internal metadata keys)
Visibility	Render link control only when href is valid; no placeholder when absent
URL policy	Allow http: and https: only after trim; reject javascript:, data:, protocol-relative //, and other schemes
Navigation	<a href={href} target="_blank" rel="noopener noreferrer"> — icon button or anchor with aria-label from i18n
Tooltip	Full URL on hover (truncated display in table if needed)
Click isolation	stopPropagation on link click so row selection / details toggle is not triggered

Shared helper (new lightrag_webui/src/features/documentMetadata.ts or lib/documentMetadata.ts):

export function getDocumentMetaHref(meta: Record<string, unknown> | undefined): string | null

Used by table row and details dialog; unit-tested for valid https, invalid schemes, non-string href, empty string.

No backend change — href is a conventional meta key documented in upload examples; roundtrip via existing DocStatusResponse.meta.

i18n

Preset labels, range mode labels, shape labels, undated option tooltips, and documentPanel.documentManager.metaHref / metaHrefAriaLabel ("Open source link") in all locale files.

Vector pushdown configuration (opt-in, default off)

LIGHTRAG_VECTOR_METADATA_FILTERS=false / enable_vector_metadata_filters: bool = False.

LIGHTRAG_KG_UNION_SKIP_OPEN_BOUNDS=true / kg_union_skip_open_bounds: bool = True (see KG section).

Denormalized fields:

Chunks: filter_start_date, filter_end_date, filter_tags — stamped at pipeline chunk upsert when pushdown enabled (see Chunk filter-field stamping)

Entities / relations: graph filter_* always; vector payload fields + has_undated_evidence only when pushdown enabled

Extended BaseVectorStorage.query(..., metadata_filter=None) — metadata_filter=None preserves existing behavior.

Pushdown predicates implement document_matches_date_filter for chunks; for KG payloads use denormalized summary interval + kg_union_skip_open_bounds semantics + tag union gate (required_tags_mode + apply_forbidden_tags_to_kg aware). Backend tag clauses: and → all terms must match payload filter_tags; or → at least one term matches (e.g. Qdrant should / OpenSearch should minimum_should_match=1).

Per-backend notes

All pushdown backends: predicates only when metadata_filter is not None. Apply to chunks_vdb; when enable_vector_metadata_filters and backend supports it, same filter expr on entities_vdb / relationships_vdb as coarse gate (evidence filter still runs in operate.py).

Backend	Pushdown	No-filter behavior
Qdrant	bool filter: shape branches + range_mode comparison + tag terms + undated OR	unchanged
Milvus	boolean filter expr	unchanged
Postgres	SQL WHERE with shape/range_mode/undated branches	unchanged
MongoDB	$match after $vectorSearch	unchanged
OpenSearch	bool filter wrapping kNN	unchanged
FAISS	oversample + Python prune only when metadata_filter is not None	unchanged
NanoVectorDB	same as FAISS	unchanged
NoopVectorDB	N/A	unchanged

Tests (targeted subsets)

Area	Cases
tests/api/routes/test_document_metadata.py	reject naive; offset preserved on roundtrip; tag charset normalization; duplicate stub vs canonical metadata readback
tests/pipeline/test_document_user_metadata.py	persistence; carry-over; directive keys survive FAILED→PENDING retry; normalize_tags
tests/operate/test_query_metadata_filters.py	has_active_filters (shape-only, include_or without range); orphan/custom-KG chunks; tag AND/OR; KG evidence; answer-cache hash differs filtered vs unfiltered
tests/operate/test_kg_filter_metadata_recompute.py	merge/purge/custom-KG; acreate_entity / aedit_entity rename; manual graph CRUD
tests/kg/<backend>_impl/test_metadata_filter_pushdown.py	per-backend predicate correctness
lightrag_webui/src/api/lightrag.test.ts	date_filter serialization; preset → payload
lightrag_webui/src/features/documentMetadata.test.ts (or lib/)	getDocumentMetaHref accepts https, rejects javascript: / non-string / empty

Key files

Backend: document_routes.py, query_routes.py, query_filters.py (new), utils_pipeline.py, constants.py, base.py, operate.py (filter + answer-cache hash), utils_graph.py (KG recompute on CRUD), pipeline.py, lightrag.py, kg/*_impl.py, env.example

WebUI: lightrag.ts, QuerySettings.tsx, UploadDocumentsDialog.tsx, DocumentManager.tsx, settings.ts, locales/*.json

Future optimizations

• starts_in / ends_in range modes
• kg_evidence_policy query enum (strict_drop vs prune_evidence)
• Chunk-first inverted KG retrieval
• PATCH /documents/{id}/metadata with KG metadata recompute

004 - AI implementation plan scopes.md

name	KG scope tag materialization
overview	Extend the document metadata filters foundation with configurable scope tag prefixes (e.g. `tenant:`), materializing separate KG entity/relation variants per scope tag plus a null `tenant_tag` aggregate, and selecting the active partition from `required_tags` (no separate tenant_tag param). WebUI graph view gets a dedicated scope selector.
todos	id content status prereq-metadata Complete document metadata filters plan (user_tags on doc_status, upload plumbing, normalize_tags, required_tags on QueryRequest) pending id content status scope-helpers-config Add kg_scopes helpers (incl. resolve_kg_scope_from_required_tags), scope_tag_prefixes + enable_kg_scope_materialization config pending id content status merge-routing Route merge_nodes_and_edges fragments to aggregate + per-scope variants; stamp tenant_tag on graph/VDB/tracking pending id content status purge-recompute Recompute/delete scoped variants on purge; handle empty scoped nodes/edges pending id content status query-scope-from-tags Derive KG partition from required_tags in operate.py query paths; filter entities_vdb/relationships_vdb and graph traversal pending id content status graph-api Extend /graphs with required_tags; scope-aware get_knowledge_graph; expose tenant_tag on graph CRUD/merge pending id content status webui-docs-tests Graph view scope selector, shared required_tags wiring, design doc update, targeted operate/api/WebUI tests pending
isProject	false

KG scope tag materialization

Prerequisite

This builds on the pending document metadata filters plan: user_tags on doc_status.metadata, upload plumbing, and normalize_tags(). Scope materialization reads document tags at merge time; it does not replace date/tag query filters — it adds write-time KG partitioning for configured scope prefixes.

Goal

For configured scope prefixes (e.g. tenant:), materialize multiple KG entries per logical name:

Variant	tenant_tag	Evidence
Aggregate	null	All contributing documents (cross-scope union)
Scoped	tenant:1	Only chunks from docs whose user_tags include tenant:1
Scoped	tenant:2	Only chunks from docs tagged tenant:2

Example: entity IKEA mentioned in tenant:1 and tenant:2 docs yields three graph nodes / VDB records sharing display name IKEA but isolated descriptions, source_id, and chunk tracking.

Confirmed semantics:

• Relations are scoped the same way; edges connect only endpoints with the same tenant_tag.
• Documents without any scope-prefixed tag contribute only to the aggregate (tenant_tag=null); they are invisible to tenant-scoped queries.

flowchart TD
    doc[Doc user_tags] --> classify{Has scope tags?}
    classify -->|tenant:1| scoped1[Merge into IKEA tenant:1]
    classify -->|none| aggOnly[Merge into IKEA null only]
    classify -->|tenant:1| agg[Merge into IKEA null aggregate]
    scoped1 --> graph1[Graph node scoped key]
    agg --> graph0[Graph node aggregate key]
    query["Query required_tags tenant:1"] --> resolve[resolve_kg_scope_from_required_tags]
    resolve --> vdbFilter[VDB filter tenant_tag]
    vdbFilter --> graph1

Loading

Design decisions

1. Identity model (three fields)

Field	Role	Example
entity_name	Canonical display / extraction name	IKEA
tenant_tag	Scope tag value or null for aggregate	tenant:1
graph_node_id	Graph storage key + VDB lookup key	IKEA (null) or IKEA\x1ftenant:1

Use a dedicated delimiter (\x1f, unlikely in entity names) in new helpers in lightrag/query_filters.py (or new lightrag/kg_scopes.py):

def parse_scope_tag(user_tags: list[str], prefixes: list[str]) -> list[str]: ...
def make_scoped_entity_id(entity_name: str, tenant_tag: str | None) -> str: ...
def split_scoped_entity_id(graph_node_id: str) -> tuple[str, str | None]: ...
def make_entity_vdb_id(entity_name: str, tenant_tag: str | None) -> str:
    return compute_mdhash_id(f"{entity_name}\0{tenant_tag or ''}", prefix="ent-")

Relations mirror this: scoped endpoints + scoped relation_chunks keys via make_relation_chunk_key(scoped_src, scoped_tgt).

2. Configuration

Add to lightrag/constants.py / env.example / LightRAG constructor:

Setting	Default	Purpose
scope_tag_prefixes	[] (disabled)	e.g. ["tenant:"] — tags matching any prefix are scope tags
enable_kg_scope_materialization	false	Master switch; when off, behavior is identical to today

Env: LIGHTRAG_SCOPE_TAG_PREFIXES=tenant: (comma-separated), LIGHTRAG_KG_SCOPE_MATERIALIZATION=true.

v1 rule: one variant per matching scope tag on the document (not Cartesian products across multiple prefix families). A doc tagged tenant:1 and region:eu with both prefixes configured creates separate variants per tag, not a combined key.

3. Merge-time routing (lightrag/operate.py)

merge_nodes_and_edges already processes one document per call (doc_id present). At the start of merge:

1. Load user_tags from doc_status for doc_id.
2. scope_tags = parse_scope_tag(user_tags, scope_tag_prefixes).
3. target_scopes = [None] + scope_tags (always aggregate; add scoped targets only when doc has scope tags).

For each entity fragment collected in all_nodes:

• Call _merge_nodes_then_upsert once per tenant_tag in target_scopes, passing tenant_tag and filtering nodes_data to fragments whose source_id chunk belongs to this merge doc (already true for per-doc merge).
• Scoped merges only ingest chunks from this doc; aggregate merge ingests all chunks from this doc.

For each edge (src, tgt):

• Same target_scopes loop; resolve scoped_src = make_scoped_entity_id(src, tag), scoped_tgt = make_scoped_entity_id(tgt, tag).
• _merge_edges_then_upsert(scoped_src, scoped_tgt, ...) with tenant_tag.

Extend _merge_nodes_then_upsert / _merge_edges_then_upsert signatures:

• graph_node_id param (defaults to entity_name when scoping disabled).
• Stamp tenant_tag on graph node/edge properties and VDB payloads.
• entity_chunks_storage / relation_chunks_storage keys = graph_node_id (not bare entity_name).
• VDB upsert uses make_entity_vdb_id(entity_name, tenant_tag).

LLM description merge runs independently per scope variant — scoped IKEA descriptions summarize only tenant:1 evidence.

4. Recompute on purge (lightrag/operate.py + lightrag/lightrag.py)

After _purge_kg_contributions removes a document's chunks:

1. For each affected canonical entity/relation name (from anchors):
   • Enumerate surviving chunk IDs per scope variant by re-walking entity_chunks / relation_chunks keys that split_scoped_entity_id maps back to the canonical name.
   • If a scoped variant has zero surviving chunks: delete graph node/edge, VDB record, and tracking row.
   • Else: rebuild description via existing rebuild path (rebuild_knowledge_from_chunks scoped) or lighter chunk-list trim + description re-merge.

Hook: recompute_scoped_kg_variants(canonical_entity, doc_status, text_chunks, ...) called from purge completion and optionally from the existing recompute_kg_filter_metadata path in the metadata-filters plan.

Anchors unchanged: full_entities / full_relations keep canonical extraction names (unscoped); purge discovers all scope variants by scanning tracking keys with prefix make_scoped_entity_id(name, "") or indexed reverse map.

5. Query-time selection via required_tags (no tenant_tag param)

Reuse the existing required_tags field from the document metadata filters plan — do not add a separate tenant_tag on QueryParam / QueryRequest. Scope selection is derived at runtime by splitting required_tags into scope tags vs ordinary filter tags.

Add to lightrag/kg_scopes.py:

@dataclass(frozen=True)
class ResolvedKgScope:
    kg_scope_tag: str | None          # None => aggregate partition
    doc_filter_tags: list[str]        # non-scope tags left for doc eligibility

def resolve_kg_scope_from_required_tags(
    required_tags: list[str],
    scope_prefixes: list[str],
) -> ResolvedKgScope:
    """Partition required_tags into at most one KG scope tag + remaining doc filters."""

Resolution rules (v1):

required_tags scope subset	KG partition	Doc filter tags
none	aggregate (tenant_tag=null)	all of required_tags
exactly one tag matching a scope prefix (e.g. tenant:1)	that scoped variant	remaining non-scope tags (e.g. patent)
more than one scope-prefixed tag	422 — "At most one scope tag allowed in required_tags for KG retrieval"	—

Examples:

• required_tags: [] → aggregate KG + no doc tag filter
• required_tags: ["tenant:1"] → scoped KG and doc filter requires tenant:1 (consistent)
• required_tags: ["tenant:1", "patent"] → scoped KG for tenant:1; docs must also have patent
• required_tags: ["patent"] → aggregate KG; docs must have patent

required_tags_mode (and / or) applies only to doc-level eligibility (unchanged from metadata-filters plan). Scope resolution runs before doc matching and is independent of AND/OR — there is at most one KG partition.

Do not combine forbidden_tags with scope-prefixed tags that also appear in required_tags. Scope tags in required_tags select a KG partition; putting related scope tags in forbidden_tags (e.g. required_tags: ["known-in-chapter:03"] + forbidden_tags: ["known-in-chapter:04"]) creates contradictory doc-filter semantics when uploads carry forward-propagating markers — results become hard to reason about. Reserve forbidden_tags for non-scope facets (draft, author-notes, type:appendix, …). For single-chapter doc isolation alongside a scope checkpoint, add non-scope upload tags (e.g. chapter:03) and require those in required_tags next to the single scope tag.

Changes in retrieval (lightrag/operate.py):

1. At query entry: resolved = resolve_kg_scope_from_required_tags(query_param.required_tags, scope_prefixes).
2. Pass resolved.kg_scope_tag into _get_node_data / _get_edge_data for VDB + graph partition filter.
3. Pass resolved.doc_filter_tags (plus required_tags_mode) into existing doc/chunk eligibility logic.
4. _find_related_text_unit_from_*: chunk resolution uses scoped entity_chunks row (authoritative).

When enable_kg_scope_materialization=false or scope_tag_prefixes empty: resolve_kg_scope_from_required_tags is a no-op (all tags stay doc filters; KG uses today's single-partition behavior).

Graph subgraph API (graph_routes.py, lightrag.py):

Extend GET /graphs with optional required_tags (same normalization/limits as query):

@router.get("/graphs")
async def get_knowledge_graph(
    label: str,
    max_depth: int = 3,
    max_nodes: int = 1000,
    required_tags: list[str] = Query(default=[]),
    required_tags_mode: RequiredTagsMode = RequiredTagsMode.AND,
):

• Resolve scope from required_tags the same way as query.
• Resolve seed label IKEA → graph_node_id = make_scoped_entity_id("IKEA", kg_scope_tag).
• BFS traversal only follows edges whose endpoints share the same tenant_tag (scoped subgraph is self-contained).
• Node labels returned to WebUI use canonical entity_name, not internal graph_node_id.

Optional helper endpoint for the scope selector dropdown:

GET /graph/scope/list → distinct non-null tenant_tag values present in the graph (or derived from doc_status scope tags). Returns ["tenant:1", "tenant:2", ...].

6. Graph API / manual edits (lightrag/utils_graph.py)

• create_entity, edit_entity, create_relation, edit_relation, amerge_entities: add optional tenant_tag (default null).
• amerge_entities refuses cross-tenant_tag merges.
• REST graph routes: expose tenant_tag on read; accept on write.
• List/export endpoints: return tenant_tag alongside entity_name.

7. Storage / backend notes

• Graph: node id = graph_node_id; properties include entity_name, tenant_tag.
• entity_chunks / relation_chunks KV: keyed by graph_node_id.
• entities_vdb / relationships_vdb: extend meta_fields in lightrag/lightrag.py with tenant_tag, graph_node_id (if distinct from hashed id).
• No backfill in v1 (same policy as metadata filters): existing nodes without tenant_tag treated as aggregate; reprocess to create scoped variants.

8. WebUI

Graph view scope selector (primary UX)

Add a Scope control to the knowledge-graph panel — not buried in retrieval query settings. Suggested placement: GraphViewer.tsx toolbar area or Settings.tsx popover (alongside depth / max nodes).

Control	Behavior
Scope dropdown	Options: All (aggregate) + scope tags from GET /graph/scope/list
Selection	Updates `graphScopeTag: string
Graph fetch	useLightragGraph.tsx passes required_tags: graphScopeTag ? [graphScopeTag] : [] to queryGraphs
Refetch trigger	Include graphScopeTag in the fetch signature alongside queryLabel, graphQueryMaxDepth, graphMaxNodes

• All (aggregate): required_tags=[] → aggregate KG partition; graph shows cross-scope union view.
• tenant:1: required_tags=["tenant:1"] → scoped partition + doc tag filter (harmless for graph-only fetch).

Extend queryGraphs(label, maxDepth, maxNodes, requiredTags?) to append required_tags query params. Types in lightrag.ts.

i18n keys under graphPanel.scope.* (label, aggregate option, tooltip explaining scoped vs all).

Retrieval query panel

Reuse the same required_tags chips UI from the metadata-filters plan — no separate scope field. When a user adds tenant:1 to required tags on a query, KG retrieval automatically uses the scoped partition via resolve_kg_scope_from_required_tags. Tooltip note: scope-prefixed tags (per server config) also select the KG partition.

Optional: when graph scope selector changes, offer to sync retrieval required_tags (off by default) — not required for v1.

Upload UI: document tags already support tenant:1 via metadata form — no new upload fields; document scope tag prefixes in operator docs.

Example: spoiler-aware book knowledge (known-in-chapter:)

A novel is ingested one chapter per upload (10 chapters). Configure scope materialization with:

LIGHTRAG_SCOPE_TAG_PREFIXES=known-in-chapter:
LIGHTRAG_KG_SCOPE_MATERIALIZATION=true

Upload tagging (forward-propagating “still known” markers)

Facts introduced early stay known for the rest of the book. Tag chapter N with every checkpoint from N through 10 — the first chapter carries all markers; the last carries only its own:

Chapter upload	tags (abbreviated)
Chapter 1	known-in-chapter:01 … known-in-chapter:10 (all ten)
Chapter 2	known-in-chapter:02 … known-in-chapter:10
Chapter 3	known-in-chapter:03 … known-in-chapter:10
…	…
Chapter 10	known-in-chapter:10 only

Add a non-scope chapter:NN tag on each upload (not in scope_tag_prefixes) to identify that chapter’s document alone:

Chapter upload	non-scope tag
Chapter 3	chapter:03

Chapter 1 introduces Elena. Chapter 5 reveals Marcus as the villain. Chapter 10 adds the final twist about Elena’s origin.

At merge time, each chapter document writes into:

• the aggregate partition (tenant_tag=null) — full book, all spoilers;
• one scoped partition per known-in-chapter: tag on that document.

So Elena exists as:

Partition	What it contains
null (aggregate)	All chapters — full spoilers
known-in-chapter:03	Merges from chapter uploads 1–3 (each carries tag 03)
known-in-chapter:05	Merges from chapter uploads 1–5
known-in-chapter:10	Merges from all chapter uploads (every upload carries 10)

A scoped partition is “reader knowledge as of the end of chapter N”: every upload that still carries known-in-chapter:0N contributed to it.

Querying “what the reader knows after chapter 3”

{
  "query": "Who is Elena and what do we know about her?",
  "required_tags": ["known-in-chapter:03"]
}

Layer	Effect
KG scope (known-in-chapter:03)	Entity/relation search uses the chapter-3 scoped variants
Doc filter	Chapter uploads 1, 2, and 3 match (each includes tag 03); chapters 4+ do not
Evidence prune	Chunks from chapters 1–3 only

Result: Elena’s description reflects chapters 1–3; Marcus (chapter 5) and the chapter-10 twist are absent.

Graph view: set scope selector to known-in-chapter:03.

Querying “only what happens in chapter 3” (single chapter text)

Pair the scope checkpoint with a non-scope chapter tag:

{
  "required_tags": ["known-in-chapter:03", "chapter:03"]
}

KG scope stays known-in-chapter:03; doc filter narrows to the chapter 3 upload only.

Querying “full book / all spoilers”

{
  "required_tags": []
}

Aggregate KG partition; no tag gate. Use for editor or “I’ve read the whole book” mode.

Querying “only revelations in chapter 10”

{
  "required_tags": ["known-in-chapter:10", "chapter:10"]
}

KG scope known-in-chapter:10; doc filter = chapter 10 upload only.

WebUI graph scope selector

Dropdown options: All (full book), known-in-chapter:01 … known-in-chapter:10 from GET /graph/scope/list. Selecting known-in-chapter:05 fetches /graphs?...&required_tags=known-in-chapter:05 — characters and relations as the reader knows them after chapter 5, without a separate query field.

flowchart LR
    subgraph uploads [Chapter uploads]
        c1["Ch1 tags: 01..10"]
        c3["Ch3 tags: 03..10"]
        c10["Ch10 tags: 10 only"]
    end
    subgraph partitions [KG partitions for Elena]
        p3[known-in-chapter:03]
        p5[known-in-chapter:05]
        p10[known-in-chapter:10]
        agg[aggregate null]
    end
    c1 --> p3
    c1 --> p5
    c1 --> p10
    c1 --> agg
    c3 --> p3
    c3 --> p5
    c3 --> p10
    c3 --> agg
    c10 --> p10
    c10 --> agg
    q3["Query: known-in-chapter:03"] --> p3
    qfull["Query: no tags aggregate"] --> agg

Loading

Operator tips:

• One upload per chapter; keep known-in-chapter: markers chapter-level, not per entity.
• Use non-scope chapter:NN when you need a single chapter’s document, not cumulative reader knowledge.
• Never put scope-prefixed tags in forbidden_tags while also requiring a related scope tag in required_tags — the forward-propagating tag sets make that especially confusing; use chapter:NN for per-chapter doc boundaries instead.

Key files to change

Area	Files
Scope helpers	New lightrag/kg_scopes.py or extend lightrag/query_filters.py
Config	lightrag/constants.py, lightrag/lightrag.py, env.example
Merge / purge	lightrag/operate.py, lightrag/lightrag.py
Graph CRUD	lightrag/utils_graph.py, graph REST routes
Query API	lightrag/api/routers/query_routes.py — uses existing required_tags on QueryParam
Graph API	lightrag/api/routers/graph_routes.py — /graphs?required_tags=, GET /graph/scope/list
VDB pushdown	Per-backend query() metadata filter on tenant_tag payload field (reuse metadata-filters infra when present; post-filter fallback otherwise)
WebUI graph	GraphViewer.tsx, Settings.tsx, useLightragGraph.tsx, settings.ts
Tests	tests/operate/test_kg_scope_materialization.py, tests/operate/test_kg_scope_purge.py, tests/api/routes/test_query_scope_from_required_tags.py, tests/api/routes/test_graph_scope.py
Docs	design/document-metadata-query-filters.md § scope tags

Test plan (targeted)

Run after implementation:

./scripts/test.sh tests/operate/test_kg_scope_materialization.py tests/operate/test_kg_scope_purge.py
./scripts/test.sh tests/api/routes/test_query_scope_from_required_tags.py tests/api/routes/test_graph_scope.py

Core cases:

1. Merge routing — doc tenant:1 creates aggregate + tenant:1 variants; descriptions differ when second doc is tenant:2.
2. Unscoped doc — contributes only to aggregate; required_tags: ["tenant:1"] returns no IKEA if IKEA only from unscoped doc.
3. required_tags split — ["tenant:1", "patent"] scopes KG to tenant:1 and filters docs to those with patent.
4. Multiple scope tags — ["tenant:1", "tenant:2"] → 422.
5. Scoped relations — edge exists under tenant:1 scoped endpoints; absent when querying wrong tenant.
6. Purge — removing tenant:1 doc trims scoped variant; deletes variant when empty; aggregate retains other tenants' evidence.
7. Graph scope — /graphs?label=IKEA&required_tags=tenant:1 returns scoped subgraph; aggregate when required_tags empty.
8. Disabled / no prefixes — bit-for-bit parity with current merge + query.
9. VDB id stability — make_entity_vdb_id deterministic; re-upsert same scope updates in place.
10. Book chapters — forward-propagating known-in-chapter: tags; required_tags: ["known-in-chapter:03"] matches chapters 1–3; single-chapter query pairs chapter:03.

Implementation order

1. Land document metadata filters (user_tags on doc_status) — hard dependency.
2. Scope helpers + config (kg_scopes.py, constants, env).
3. Merge routing in _merge_nodes_then_upsert / _merge_edges_then_upsert + merge_nodes_and_edges.
4. Purge recompute + empty-variant deletion.
5. resolve_kg_scope_from_required_tags + query retrieval filters.
6. Graph API: /graphs required_tags, scope-aware traversal, /graph/scope/list, tenant_tag on CRUD.
7. WebUI graph scope selector + API wiring.
8. Tests.

Risks and mitigations

Risk	Mitigation
Storage multiplication (N tenants × M entities)	Opt-in config; scope prefixes should be low-cardinality (tenant: not person:)
Graph API confusion (same display name, many nodes)	Always return tenant_tag; document in API docs
Merge cost (1 + len(scope_tags) merges per fragment)	Acceptable for low scope-tag count per doc; batch per scope in one merge call
Custom-KG / ainsert_custom_kg	Require explicit tenant_tag on insert or default aggregate only

005 - AI generated usecases.md

Document metadata and query filters — design use cases

This document captures why user-controlled document metadata and query-time filters matter for archival and knowledge-work domains, and what requirements each scenario imposes on upload and query APIs.

Ingest (upload): start_date, end_date, tags, meta
Query (retrieval scoping): date_filter, required_tags, forbidden_tags only — meta is never a query input

Policy: The meta JSON object is roundtrip-only. It is stored at ingest and returned on document status / detail responses. It is not indexed for retrieval, not accepted on QueryRequest, and will not gain a query API (no JSON path, key match, or numeric filters). To scope search, use tags or date_filter.

Problem statement

LightRAG today scopes retrieval by vector similarity and knowledge-graph structure, not by document-level attributes. Archival corpora—legislation, patents, parliamentary records, newspapers, correspondence—are organized around time, subject tags, and catalog payloads. Users need to:

1. Attach temporal bounds, filterable tags, and roundtrip meta when documents enter the system.
2. Restrict retrieval using dates and tags only, without changing the core RAG modes (local, global, hybrid, mix, naive).

Poor date semantics (overlap-only, undifferentiated “partial dates”) fail legal and legislative workflows where how an interval relates to a query range is as important as whether they touch at all.

---

Upload metadata

Fields

Field	Type	Required	Purpose
start_date	ISO-8601 datetime	No	Effective start (enacted, filed, published, session opened, …)
end_date	ISO-8601 datetime	No	Effective end (repealed, expired, adjourned, …)
tags	string[]	No	Broad category facets for query filtering (channel, jurisdiction, doc type, …). Keep cardinality low — see below
meta	JSON object	No	Structured catalog fields for roundtrip only — stored at ingest, returned on read; never used for query filtering

Tags vs meta — when to use which

Tags participate in every filtered query (doc_status scan, post-retrieval pruning, and optional vector pushdown). Large tag sets per document and long required_tags lists slow filtering — tags are for broad, stable categories you routinely scope on (e.g. channel:engineering, jurisdiction:uk), not a mirror of every entity in the document.

Concern	Prefer tags	Prefer meta
Filter retrieval (required_tags / forbidden_tags)	Yes — sparingly, broad facets only	Never — meta is not queryable
Human-readable document list / WebUI details	Short summary tags only (optional)	Yes (rich structure)
Stable machine ids (UUID, DOI, message id)	No — high cardinality hurts query performance	Yes (roundtrip to client / export)
Nested or numeric fields	No	Natural JSON
Long participant / keyword lists (every @mention, every IPC code)	No — do not tag every value	Full list in meta; tag only coarse facets you query often (e.g. one channel:…, type:transcript)
A few principals you often filter on (lead author, primary channel, jurisdiction)	Yes — small, curated set	Duplicate in meta for display if needed

Practical rule: use tags only for broad categories that should narrow retrieval. Put exhaustive lists (all participants, all message ids, all citation fields) in meta. If a field must affect search, promote one coarse tag (e.g. assignee:acme-corp), not every related identifier.

meta is roundtrip-only, permanently. There is no query API on meta. High-cardinality data that must never slow queries belongs in meta, not in tags.

Tag naming convention (recommended): lowercase facet:value prefixes; low facet cardinality (tens of distinct values per facet across the corpus, not thousands per document):

channel:engineering
type:transcript
jurisdiction:uk
lang:en

Use person:… (or similar) only for a few named principals you regularly filter on — not every participant in a large thread. Full rosters live in meta.participants.

Normalization at ingest: Unicode NFKC + casefold → charset [a-z0-9:_-] only (see plan). Same normalization on query required_tags / forbidden_tags. Enforce per-document tag count and per-tag length limits at the API boundary.

Tag use cases

Tags are the only non-date dimension for query scoping. Prefer few, broad tags per document; combine with date_filter rather than encoding fine detail as dozens of tags.

Chat and messaging archives

Why interesting: Transcripts are sliced by time window and channel; participants are the primary access path (“what did we decide in #engineering when Ada was involved?”). Vector search alone pulls semantically similar text from other channels and dates.

Upload example

{
  "metadata": {
    "start_date": "2024-03-15T14:00:00Z",
    "end_date": "2024-03-15T15:30:00Z",
    "tags": [
      "channel:engineering",
      "type:transcript",
      "source:slack-export"
    ],
    "meta": {
      "thread_id": "1700000000.000100",
      "message_count": 847,
      "export_batch": "slack-2024-q1",
      "participants": [
        { "id": "U01ADA", "display_name": "Ada Lovelace" },
        { "id": "U01CHARLES", "display_name": "Charles Babbage" }
      ],
      "key_participants": ["ada-lovelace", "charles-babbage"]
    }
  }
}

Tag discipline: one channel, doc type, and source — not every participant. The full participant list is in meta. Add person:ada-lovelace to tags only for documents where Ada is a primary subject and you expect frequent required_tags queries on her; otherwise rely on vector search within the channel-scoped corpus.

Optional narrow tagging when a principal is query-critical:

"tags": ["channel:engineering", "type:transcript", "person:ada-lovelace"]

meta holds stable ids and display structures for the WebUI; filtering uses a small tags set (channel:…, type:…, occasional person:…).

Query examples

Question	Filters
What was discussed about the database migration in #engineering?	required_tags: ["channel:engineering"]
Summarize Ada’s points during the March 15 incident call	required_tags: ["person:ada-lovelace"], date_filter.range that afternoon, shape: has_start
Threads with Ada and Charles both present	required_tags: ["person:ada-lovelace", "person:charles-babbage"]
Everything except #random and #social	forbidden_tags: ["channel:random", "channel:social"]
Incident channel only, exclude bot noise	required_tags: ["channel:incident-response"], forbidden_tags: ["person:status-bot"]

---

Parliamentary and committee records (tags)

Tags: chamber:commons, committee:finance, lang:en, type:hansard, topic:budget.

Query example: “Opposition statements on the budget bill”

{
  "required_tags": ["committee:finance", "topic:budget"],
  "date_filter": {
    "range": { "start": "2024-03-01", "end": "2024-03-31" },
    "range_mode": "overlap",
    "shape": "has_start"
  }
}

---

Legislation and case law (tags)

Tags: jurisdiction:uk, code:ukpga, subject:employment, court:ewca-civ.

Query example: UK employment statutes, exclude repealed marker tag

{
  "required_tags": ["jurisdiction:uk", "subject:employment"],
  "forbidden_tags": ["status:repealed"]
}

---

Patents and IP (tags)

Tags: ipc:h01l, assignee:acme-corp, family:us-12345678.

Query example: Semiconductor portfolio in filing window

{
  "required_tags": ["ipc:h01l", "assignee:acme-corp"],
  "date_filter": {
    "range": { "start": "2010-01-01", "end": "2015-12-31" },
    "range_mode": "overlap",
    "shape": "has_start"
  }
}

---

News and media monitoring (tags)

Tags: outlet:nyt, section:business, lang:en, geo:us.

Query example: Business-section coverage, exclude opinion desk

{
  "required_tags": ["section:business"],
  "forbidden_tags": ["desk:opinion"]
}

---

Internal knowledge base / runbooks (tags)

Tags: team:platform, env:production, type:runbook, service:payments.

Query example: Production payments runbooks only

{
  "required_tags": ["type:runbook", "service:payments", "env:production"]
}

---

meta use cases (roundtrip only)

These scenarios explain why meta exists despite not being queryable. Clients ingest rich structures once and read them back on document status / detail APIs without a separate catalog database.

Chat export provenance

Return thread_id, message_count, and participant objects so the WebUI can link to the source export or show avatars. Retrieval is still scoped by tags and date_filter; meta is not consulted during search.

Legal and bibliographic citation

{
  "meta": {
    "citation": "UK Public General Act 1996 c. 18",
    "eli": "http://www.legislation.gov.uk/ukpga/1996/18",
    "version_date": "2024-01-01"
  }
}

Displayed in document details and exported reports; users filter with tags like jurisdiction:uk and dates, not ELI URLs in meta.

Patent and docket identifiers

{
  "meta": {
    "publication_number": "US1234567B2",
    "application_number": "US16/123456",
    "priority_claims": ["EP20190123456"]
  }
}

Filter portfolio with required_tags: ["assignee:acme-corp"]; show official numbers from meta in the UI.

Financial filing metadata

{
  "meta": {
    "cik": "0000320193",
    "accession": "0000320193-24-000001",
    "fiscal_year": 2019,
    "form": "10-K"
  }
}

FY period filtering uses date_filter + tags: ["form:10-k"]; SEC accession stays in meta for deep links.

Scientific dataset descriptors

{
  "meta": {
    "doi": "10.1234/example",
    "sample_size": 1200,
    "instruments": ["spectrometer-a", "spectrometer-b"]
  }
}

Site/parameter filtering uses tags; DOI and instrument list are display-only roundtrip.

Design constraint for integrators

If a field must narrow retrieval, add a single coarse tag (e.g. form:10-k) at ingest — not every field from meta. Do not assume a future meta query language will exist.

Domain use cases

Historic law and legislation

Corpus characteristics

• start_date ≈ enacted / promulgated; end_date ≈ repealed / superseded.
• Many acts still in force → start only, no end.
• Tags: jurisdiction, code title, subject matter.

Interesting queries

Question	Date filter sketch
Material touching the interwar period	overlap + any
Acts in force for all of 1920	contains + has_start
Statute in force on a specific day	Point range + overlap or contains + has_start
1930s legal context plus undated fragments	overlap + include_or
Only fully cataloged intervals	bounded + exclude undated

Requirements

• Open-ended end (+∞) for in-force acts.
• contains for “valid throughout” (stricter than overlap).
• has_start to drop undated and end-only catalog noise.
• Tag filters for jurisdiction and subject.

---

Patents and intellectual property

Corpus characteristics

• start_date ≈ filing / priority; end_date ≈ expiry.
• Expiry often missing in metadata; validity may still be inferred as open-ended until expiry is recorded.
• Tags: IPC class, assignee, family id.

Interesting queries

Question	Date filter sketch
Patents active at any point in 2010–2015	overlap + has_start
Patents covering the entire 2010–2015 window	contains + has_start
Filing activity in a year (broad)	overlap + has_start
Portfolio tagged semiconductor in period	+ required_tags

Requirements

• has_start (filing date is the reliable anchor).
• contains vs overlap distinction for clearance / landscape vs mention retrieval.
• Forbidden tags to exclude deprecated or withdrawn classifications.

---

Parliamentary session transcriptions

Corpus characteristics

• Often one session date or a short span (start + end).
• Tags: chamber, committee, language, document type (transcript, hansard).

Interesting queries

Question	Date filter sketch
Debates during Q1 2024	overlap + has_start
Session wholly within a sitting period	contained_by + has_start
Budget debates in March	Narrow range + required_tags

Requirements

• has_start (single sitting date satisfies shape).
• contained_by for “session inside term” (precision).
• Tag filters for chamber and topic.

---

Treaties and international agreements

Corpus characteristics

• Signed / ratified start; termination or withdrawal end; many open-ended.
• Similar temporal logic to legislation.

Interesting queries

Question	Date filter sketch
Agreements in force during a crisis window	overlap + has_start
Treaty valid for entire negotiation period	contains + has_start

Requirements

• Same as legislation: has_start, contains, open end.

---

Newspapers and serials

Corpus characteristics

• Usually issue date (effectively start-only or start == end).
• Digitized reels may lack per-issue dates → undated batches.

Interesting queries

Question	Date filter sketch
Issues from the 1930s	overlap + dated or has_start
1930s coverage plus undated reel scans	overlap + include_or
Only precisely dated issues	dated + exclude undated

Requirements

• include_or for discovery over incomplete digitization.
• dated / has_start when catalog quality matters.

---

Letters, manuscripts, and personal archives

Corpus characteristics

• High proportion of undated items.
• When dated, often a single letter date.

Interesting queries

Question	Date filter sketch
Correspondence mentioning X (no date filter)	No date_filter
1880–1900 or undated for manual review	overlap + include_or
Strictly dated 19th-century letters	overlap + dated

Requirements

• include_or is essential for mixed-quality archives.
• Default broad search often has no date filter at all.

---

Financial and annual reports

Corpus characteristics

• Fiscal periods with both bounds well defined.
• Tags: entity, report type, GAAP framework.

Interesting queries

Question	Date filter sketch
FY2019 reports only	overlap or contained_by + bounded
Strict complete fiscal intervals	bounded + exclude undated

Requirements

• bounded shape for catalog strictness.
• contained_by when the fiscal year must fall entirely inside a reporting window.

---

Court opinions and case law

Corpus characteristics

• Decision date (point-like).
• Tags: court, jurisdiction, matter type.

Interesting queries

Question	Date filter sketch
Opinions from 2015–2020	overlap + has_start
Jurisdiction-scoped search	+ required_tags

Requirements

• Point ranges work with overlap + has_start.
• Less need for contains unless querying multi-year precedential “window”.

---

Environmental monitoring and scientific reports

Corpus characteristics

• Monitoring period start/end (usually bounded).
• Tags: site, parameter, standard.

Interesting queries

Question	Date filter sketch
Reports covering a contamination event window	overlap or contains
Studies wholly inside a study period	contained_by + bounded

Requirements

• bounded when both period bounds are always recorded.

---

Clinical trials and regulatory filings

Corpus characteristics

• Enrollment start; completion often missing for ongoing trials.
• Tags: phase, condition, sponsor.

Interesting queries

Question	Date filter sketch
Trials active during a approval review period	overlap + has_start
Trials running through entire calendar year	contains + has_start

Requirements

• Same open-end pattern as patents and law.

---

WebUI presets (suggested)

Presets map to DateFilter fields so users choose intent, not raw enums:

Preset	range_mode	shape	undated	Domains
Broad retrieval	overlap	any	exclude	Default RAG
Archive discovery	overlap	any	include_or	Letters, newspapers
Legislation / in force	contains	has_start	exclude	Law, treaties
Activity in period	overlap	has_start	exclude	Patents, trials, news
Wholly within period	contained_by	has_start	exclude	Sessions, events
Strict catalog	overlap	bounded	exclude	Financial, scientific periods
Uncatalogued only	—	—	only	Admin / cleanup

Advanced UI exposes range_mode, shape, and undated individually.

---

Implementation requirements (summary)

Upload path

• Optional dates, tags, meta on all ingest endpoints.
• Dates: timezone-aware only; store with offset preserved (no UTC conversion at persistence — future calendar dates cannot be reliably re-projected through UTC when DST rules change). Comparisons use instant semantics at query time.
• All four user_* keys on both _DOC_STATUS_METADATA_CARRY_OVER_KEYS and _DOC_STATUS_METADATA_DIRECTIVE_KEYS (retry reset uses directive whitelist only).
• Denormalize filter_start_date, filter_end_date, filter_tags onto chunk vectors only when vector metadata pushdown is enabled (LIGHTRAG_VECTOR_METADATA_FILTERS, default off), at pipeline chunks_vdb.upsert time.

Query path

• Single document_matches_filter() implementing shape, range mode (skipped when range is None), undated OR, and tag predicates.
• has_active_filters: date active when range set, shape != any, or undated != exclude (not only range / undated=only).
• Chunks whose full_doc_id has no doc_status row → undated/untagged; excluded unless undated allows.
• Normalize query tags to [a-z0-9:_-] (same as upload).
• Answer-cache hashes in kg_query and naive_query must include filter fingerprint when active; keyword-extraction cache stays filter-agnostic.
• Vector pushdown when enabled; no filter machinery when inactive.

Out of scope (v1)

• Any query API for meta — roundtrip only.
• starts_in / ends_in range modes.
• Post-upload metadata edit (PATCH); re-upload same filename = duplicate, not metadata update.
• Backfill of filter fields on pre-existing chunks/KG rows.

Known modeling limit

One (start_date, end_date) pair per document cannot represent multiple roles (e.g. patent filing vs priority vs expiry). Secondary dates belong in meta for display/export or must be promoted to tags / dates if they need to affect retrieval.