Run GraphRAG Toolkit RDF

datalog.md

Writing Datalog Rules for RDF Triple Stores — Field Guide

Audience: an LLM writing rules that will actually be loaded into a store. Target engine: RDFox (.dlog rule files). Rule 0: Datalog is not SPARQL. Stop writing CONSTRUCT. It won't load.

---

1. Mental model (read once, then stop guessing)

• A rule is HEAD :- BODY . — "if BODY matches, materialise HEAD".
• Atoms are triple patterns: [?s, ?p, ?o]. Variables start with ?. Constants are IRIs or literals.
• Rules fire until fixpoint. Recursion is fine. Recursion through negation or aggregation is not.
• The store adds triples; it never deletes them via rules. Want "delete"? You want a different tool.
• No existentials: you cannot invent a fresh blank node in the head unless you build an IRI with SKOLEM.

2. Syntax cheat sheet (RDFox)

# Comments start with a hash.
@prefix :     <http://example.org/> .
@prefix rdf:  <http://www.w3.org/1999/02/22-rdf-syntax-ns#> .
@prefix rdfs: <http://www.w3.org/2000/01/rdf-schema#> .
@prefix xsd:  <http://www.w3.org/2001/XMLSchema#> .

# Basic rule. Head atoms comma-separated; body atoms comma-separated. Ends with a full stop.
[?x, rdf:type, :Adult] :- [?x, rdf:type, :Person], [?x, :age, ?a], FILTER(?a >= 18) .

# Multiple heads.
[?y, :parentOf, ?x], [?x, rdf:type, :Child] :- [?x, :childOf, ?y] .

# Recursion (transitive closure).
[?x, :ancestorOf, ?z] :- [?x, :parentOf, ?z] .
[?x, :ancestorOf, ?z] :- [?x, :parentOf, ?y], [?y, :ancestorOf, ?z] .

# Negation — variable ?x must be bound by a positive atom first.
[?x, rdf:type, :Orphan] :- [?x, rdf:type, :Person], NOT EXISTS ?y IN ([?x, :childOf, ?y]) .
# Shorthand when every variable in the negated atom is already bound:
[?x, rdf:type, :Unemployed] :- [?x, rdf:type, :Person], NOT [?x, :worksFor, :ACME] .

# BIND — compute a value; the new variable is then usable in the head.
[?x, :fullName, ?n] :- [?x, :first, ?f], [?x, :last, ?l], BIND(CONCAT(?f, " ", ?l) AS ?n) .

# Aggregation — GROUP BY variables after ON; result bound with AS.
[?p, :childCount, ?n] :- AGGREGATE([?p, :parentOf, ?c] ON ?p BIND COUNT(?c) AS ?n) .

# Skolemised node when you must "create" an entity.
[?e, rdf:type, :Marriage], [?e, :spouse, ?a], [?e, :spouse, ?b]
    :- [?a, :marriedTo, ?b], FILTER(STR(?a) < STR(?b)), BIND(SKOLEM("Marriage", ?a, ?b) AS ?e) .

Literals: "Tom", "42"^^xsd:integer or just 42, "2026-09-04"^^xsd:date, true. Built-ins mirror SPARQL 1.1 functions (STR, CONCAT, REGEX, YEAR, NOW, ABS, COALESCE…).

3. The process — follow it in order

Step 1 — Look at the data before writing a single rule. Do not guess predicate IRIs. Query them:

SELECT ?p (COUNT(*) AS ?n) WHERE { ?s ?p ?o } GROUP BY ?p ORDER BY DESC(?n)
SELECT DISTINCT ?c WHERE { ?s a ?c }
SELECT ?s ?p ?o WHERE { ?s ?p ?o } LIMIT 20

Note datatypes ("18" vs 18 vs "18"^^xsd:int — three different things to a triple store).

Step 2 — Write the inference in English, one line each. "Every Person with age ≥ 18 is an Adult." If you can't write the sentence, you can't write the rule.

Step 3 — Translate one sentence → one rule. Head = the conclusion. Body = the conditions. Name it with a comment.

Step 4 — Safety check (the store will reject unsafe rules). Every variable in the head, in a FILTER, in BIND arguments, or in a negated atom must appear in a positive body atom (or be introduced by a BIND/AGGREGATE result). [?x, :p, ?y] :- [?x, :q, ?z] . is rejected — ?y is unbound.

Step 5 — Stratification check. Draw the dependency graph of predicates. If predicate A depends negatively (via NOT) or via AGGREGATE on B, and B depends (any path) back on A: rejected. Fix by splitting into a base predicate and a derived one.

Step 6 — Build a 10-triple test file with the expected output written down first.

@prefix : <http://example.org/> .
:alice a :Person ; :age 34 ; :parentOf :bob .
:bob   a :Person ; :age 12 .
# Expected after reasoning: :alice a :Adult . :bob NOT a :Adult . :alice :childCount 1 .

Step 7 — Load, materialise, verify, count.

# RDFox CLI (shell)
dstore create test
active test
import test.ttl
import rules.dlog
# verify

SELECT ?x WHERE { ?x a :Adult }                          # expect exactly :alice
SELECT (COUNT(*) AS ?n) WHERE { ?s ?p ?o }               # sanity: did it explode?

If a count is suspiciously huge, you built a Cartesian product or a reflexive closure. See §5.

Step 8 — Only then scale to real data.

4. Naive-model mistakes → corrections

What the naive model writes	Why it fails	Write instead
CONSTRUCT { ?x a :Adult } WHERE { … }	That's SPARQL	[?x, rdf:type, :Adult] :- … .
?x a :Adult :- ?x :age ?a, ?a >= 18 .	Bare comparison isn't an atom	…, FILTER(?a >= 18) .
[?x, :p, ?y] :- [?x, :q, ?z] .	?y unbound (unsafe)	Bind ?y in the body or use BIND/SKOLEM
[?x, :p, _:b] :- …	No blank nodes in heads	BIND(SKOLEM("p", ?x) AS ?b)
[?x, :age, 18] when data has "18"	Datatype mismatch: no match	Check the data; use "18" or cast with BIND(xsd:integer(?s) AS ?i)
NOT [?x, :p, ?y] with ?y unbound	Unsafe negation	NOT EXISTS ?y IN ([?x, :p, ?y])
[?x, :knows, ?y] :- [?x, rdf:type, :Person], [?y, rdf:type, :Person] .	Cartesian product; O(n²) triples	Join on a shared variable; you almost certainly meant something else
[?x, :ancestorOf, ?x] :- … (or forgetting to exclude ?x=?y)	Reflexive junk	Add FILTER(?x != ?y) where needed
Missing final . / missing @prefix	Parse error	Every rule ends with .; declare every prefix
Using owl:sameAs, rdfs:subClassOf and expecting magic	Store won't apply OWL unless told	Write the rules or enable the store's OWL/RDFS axiomatisation explicitly
Rule A negates B, B derives from A	Not stratifiable	Split predicates; negate only base/lower-stratum facts
Generating rdfs:subClassOf closure by hand for every class	Verbose, brittle	One generic rule: [?x, rdf:type, ?d] :- [?x, rdf:type, ?c], [?c, rdfs:subClassOf, ?d] .

5. Performance sanity (five lines, no excuses)

1. Every body atom shares a variable with another — otherwise it's a Cartesian product.
2. Put constants and selective patterns in the body; don't derive [?s, ?p, ?o] catch-alls.
3. Avoid FILTER(REGEX…) on large joins; pre-derive a flag predicate.
4. Transitive closure of a dense relation is quadratic in triples. Say so before writing it.
5. After loading, SELECT (COUNT(*) …) per derived predicate. Expected ≈ actual, or debug.

6. Debugging

• Rule rejected → read the error: "unsafe" (§3 Step 4), "not stratified" (Step 5), "parse" (missing ./prefix/bracket).
• Rule accepted, zero results → IRIs or datatypes don't match the data. Re-run the Step 1 queries.
• Too many results → §5 line 1 or a missing FILTER(?x != ?y).
• Wrong results → materialise one atom of the body at a time as a temp predicate and look.
• RDFox: explain on a derived triple shows which rule fired.

7. RDFox specifics worth knowing

• Turtle-style atoms also parse: ?x rdf:type :Adult :- ?x :age ?a, FILTER(?a >= 18) . — pick one style per file.
• a is accepted as shorthand for rdf:type in rule atoms; fine, but be consistent.
• Deleting a fact (delete) retracts its consequences incrementally — you never "re-run" reasoning.
• Rules are stored per data store; import twice = duplicate rule warning, not double firing.
• explain <triple> in the shell shows the derivation tree. Use it before theorising.
• Named graphs: atoms become :g(?s, ?p, ?o); the default graph is [?s, ?p, ?o].
• Equality reasoning (owl:sameAs) needs the store created with equality noUNA/UNA; otherwise owl:sameAs is just a predicate.

8. Pre-flight checklist (run before emitting rules)

• Every prefix used is declared.
• Every rule ends with ..
• Every head/filter/negation variable is bound by a positive body atom or BIND.
• No blank nodes in heads.
• No negation/aggregation cycles.
• Predicate IRIs and datatypes verified against the actual data, not imagined.
• No body atom is disconnected from the rest.
• Test triples + expected output exist and were checked.

grtk_run.sh

docker run -d --name graphrag \
  --add-host host.docker.internal:host-gateway \
  -p 8888:8888 \
  -e AWS_REGION=eu-central-1 -e AWS_PROFILE=default \
  -e GRAPH_STORE='sparql+http://admin:admin@host.docker.internal:12110/datastores/my-ds/sparql' \
  -v ~/.aws:/home/jovyan/.aws:ro \
  ortecha/graphrag-toolkit-rdf:latest

structured-chunker.md

Building a structure-preserving DOCX chunker

Goal: carry a DOCX's heading hierarchy into the lexical graph as lg:parent edges between __Chunk__ nodes, at ingest time.

Why no backend change is needed

ChunkGraphBuilder already translates LlamaIndex PARENT/CHILD/PREVIOUS/NEXT node relationships into __PARENT__/etc. edges, and _link_chunks maps those to lg:parent / lg:child / lg:chunkPrevious / lg:next — all four declared in the shipped ontology. SentenceSplitter simply never sets them. So: write a NodeParser, pass it as IndexingConfig(chunking=[parser]). Nothing in the toolkit or the SPARQL contrib changes — the only other piece you write is a ChunkNodeBuilder subclass, and only if you want chunk properties (see below).

Files

docx_hierarchy_parser.py	the NodeParser — sectioning, packing, collapse rule
section_properties.py	ChunkNodeBuilder subclass promoting metadata to lg: properties
ingest.py	two-phase ingest; --dry-run costs nothing, --reuse-extraction rebuilds free
test_docx_hierarchy_parser.py	offline suite — no network, no LLM, no store
verify.sparql + verify.py	post-ingest checks

Non-negotiables

Rule	Why
Set PARENT only, never CHILD	builder reads info.node_id while iterating relationships.items(); LlamaIndex's list-valued CHILD → AttributeError
Put the heading breadcrumb in node.text	extraction reads node.text only, never metadata — and it keeps content-hashed chunk ids unique
Chunk metadata does not reach the graph by default	ChunkNodeBuilder replaces it wholesale. To promote a field, see Chunk properties below
Emit a document's chunks in one _parse_nodes call	IdRewriter remaps relationship ids only within a single call; across calls PARENT ids dangle and the graph silently materialises bare __Chunk__ nodes
Set file_path metadata yourself	DocxReaderProvider sets only file_name
Adding any metadata key re-keys every chunk	IdRewriter._new_node_id hashes (source_id, text, metadata); a new field changes all ids, so a re-ingest adds chunks rather than merging. Land metadata changes before ingesting, or use a fresh datastore
Leave include_prev_next_rel at True	it produces the lg:next reading-order chain and does not touch PARENT

Structure

One anchor chunk per section; PARENT = the nearest enclosing section's anchor (walk up on skipped levels, e.g. H1→H3). Overflow chunks attach to their own section's anchor. A heading-only section still emits an anchor, or its descendants dangle. Pre-heading content becomes a parentless preamble chunk.

Section numbers

Parse a leading label off the heading into section_number ('2.1', 'a'), section_number_path ('2.1.a' — absolute, composed through ancestors), and heading_text (label stripped). Keep heading and the breadcrumb intact so node.text and chunk ids don't move. Metadata alone is not queryable — see Chunk properties to land these as lg:sectionNumber.

Be strict, and anchor at start. A bare integer with no punctuation is not a section number ("2026 Roadmap"), and neither is a word followed by a digit — "NODE 0 — The Cleaner", "Step 1 — …", "Priority 1 — …" all appear in the sample and must be rejected. Accept only: dotted 2.1, trailing-period 1., paren 1) / a), roman IV..

Dotted labels are already absolute — 2.1 under 2 is 2.1, not 2.2.1.

Word auto-numbering puts the label in numbering.xml, not the run text: detect w:numPr on the heading and flag it rather than silently reporting an unnumbered document. Resolving it properly (abstract numbering, overrides, restarts) is a separate job.

Packing

Content is built from atoms — paragraph, run of consecutive bulleted items, table — packed until the next would exceed budget. Never split mid-atom.

• Walk document.element.body (CT_P/CT_Tbl), not .paragraphs, which drops tables.
• List items = List Paragraph style plus a w:numPr; the marker lives in numbering.xml, so render your own - .
• Budget must subtract breadcrumb tokens, with a floor so a long breadcrumb can't starve the chunk.
• An atom bigger than the budget must be intercepted before the packing loop, or it falls through and is emitted over budget (real bug: a 2120-token chunk against a 256 limit). Lists break at item boundaries, tables at rows, paragraphs at sentences.

Tables

• Render as GitHub pipe rows, one string per row, so oversized tables break at rows.
• Dedupe horizontally merged cells by cell._tc identity — row.cells yields a merged cell once per spanned column, duplicating its text. Identity distinguishes a real merge from two neighbours that read the same. The row ends up narrower than the header; pipe tables have no colspan.
• Escape | in cell text.
• A row that alone exceeds budget can't stay a pipe row: re-render it vertically as column: value so each value keeps its label, then sentence-split.

Chunk properties (lg:sectionNumber and friends)

Both ends of the channel already exist: ChunkGraphBuilder emits chunk.<key> = params.<key> for every entry in metadata['chunk']['metadata'], and the SPARQL _chunk handler renders any such parameter as lg:<key>. Only the middle is missing — nothing populates that dict per chunk. (GraphRAGConfig.chunk_external_properties copies from source metadata, so it gives every chunk the same value — useless for anything that varies per chunk.)

Subclass ChunkNodeBuilder, call super().build_nodes(), then fill metadata['chunk']['metadata'] by matching chunk['chunkId'] back to the input node id. Skip None rather than writing an empty value. Property names become RDF local names verbatim, so keep them alphanumeric.

Inject via index.build(docs, builders=[...]) — not extract_and_build(), which forwards its kwargs to the extraction pipeline too. builders replaces the default set wholesale, so list all four (Source, Chunk, Topic, Statement) and rebuild the constructor args BuildPipeline would normally supply.

That forces a two-phase run — which is worth having anyway: extraction is the expensive half, so cache it (FileBasedDocs as an extract() handler, then iterate it into build()) and every later graph-shape change is free.

Collapsing machine-generated sections

A leaf section whose paragraphs are templated stays one chunk. Signal: distinct opening shapes (first 12 chars, digit runs → #) ÷ paragraph count. Threshold 0.4 — the sample's log dump scores 0.11, every prose section 0.67–1.00. Length and terminal punctuation do not separate them (a real prose section has 0% terminal punctuation, 46-char median line).

Gate on the section having already produced ≥3 chunks (run the cheap packing first), and veto when it has subsections. Collapsed chunks are exempt from max_chunk_tokens up to a collapse_max_tokens ceiling; past that, split at that coarser budget.

Worth it: uncollapsed, console output produced 38% of every statement in the graph; collapsed, 11% — with the prose graph unchanged.

Environment

• AWS_REGION is ignored — a profile's own region outranks it. Use AWS_DEFAULT_REGION.
• Contrib graph stores need GraphStoreFactory.register(SPARQLGraphStoreFactory); without it sparql+http:// is "Unrecognized graph store info".
• LexicalGraphIndex.get_stats() issues openCypher MATCH — it does not work on the SPARQL backend. Query with SPARQL.
• Run scripts from the repo root: inside .../lexical_graph/, its logging.py shadows the stdlib.
• Vector store dummy:// skips embeddings entirely — the graph is identical to a pgvector run. Use it for graph-only work.

Run and verify

python -m pytest docx-chunker/ -q            # offline
python docx-chunker/ingest.py --dry-run      # tree + invariants, no LLM, no writes
python docx-chunker/ingest.py --datastore X  # extract (cached) then build
python docx-chunker/verify.py --datastore X

Before ingest (free): no PARENT pointing outside the emitted set; all chunk texts unique (ids are content hashes — duplicates silently merge into one node); no chunk over budget except deliberately collapsed ones, which are capped at collapse_max_tokens instead. Test the IdRewriter round-trip and id determinism across runs.

After ingest: lg:parent count == chunks − roots; depth matches heading depth; no lg:parent target lacking lg:value; one lg:next chain start.

Sizing

statements = 0.617 × tokens^0.618 (R² 0.85). The exponent generalises — it describes how extraction behaves as chunks grow. The coefficient does not — it encodes claim density of one author's technical prose. ~77 statements per 1k tokens at a long-form 230-token chunk. Facts ≈ 1.48 × statements.