A pattern for building persistent knowledge bases using LLMs.
This is an idea file, designed to be copied into your own LLM agent (OpenAI Codex, Claude Code, OpenCode, Pi, etc.). Its purpose is to communicate the architecture, while leaving implementation details to the agent and the user.
Most people's experience with LLMs and documents looks like RAG: you upload a collection of files, the LLM retrieves relevant chunks at query time, and generates an answer. This works, but the LLM rediscovers knowledge from scratch on every question. Nothing accumulates.
The idea here is different.
Instead of retrieving from raw documents on every query, the system incrementally compiles knowledge into a persistent semantic model.
Raw sources are processed only when they change. The extracted knowledge is normalized, merged, cross-referenced, and stored as a canonical Knowledge Intermediate Representation (KIR).
KIR is the semantic source of truth.
User-facing knowledge systems—Obsidian, Logseq, MkDocs, Docusaurus, search indexes, graph databases, or future tools—are reconciled from KIR rather than directly generated from raw documents.
This separates knowledge compilation from knowledge presentation.
The result is a system that accumulates knowledge over time instead of repeatedly rediscovering it.
The architecture consists of four independent layers.
Raw Sources
↓
Document Compiler
↓
Document IR
↓
Knowledge Compiler
↓
Knowledge IR (Canonical)
↓
Knowledge Reconciler
↓
Obsidian / Logseq / MkDocs / Search / Graph / ...
Each layer has a single responsibility.
The curated collection of immutable source documents.
Examples include:
- articles
- books
- papers
- meeting notes
- PDFs
- HTML
- Markdown
- images
- transcripts
Raw sources are never modified.
They are the immutable source of truth.
The Document Compiler transforms one raw source into one Document IR.
Its responsibilities include:
- parsing
- metadata extraction
- section extraction
- glossary detection
- entity extraction
- concept extraction
- provenance tracking
Document IR represents exactly one source document.
It never merges information across documents.
The Knowledge Compiler merges multiple Document IR artifacts into a canonical semantic representation called Knowledge Intermediate Representation (KIR).
Responsibilities include:
- canonical concepts
- aliases
- semantic relations
- taxonomy
- provenance
- conflict detection
- semantic normalization
Knowledge IR is independent of any particular knowledge management system.
It contains semantic knowledge, not presentation.
KIR is the stable interface of the system.
Knowledge Reconciler projects the canonical Knowledge IR into one or more user-facing workspaces.
Examples include:
- Obsidian
- Logseq
- MkDocs
- Docusaurus
- Markdown
- Graph databases
- Search indexes
This layer understands the conventions of a particular system.
It is responsible for:
- page generation
- updating existing pages
- preserving manual edits
- maintaining links
- workspace-specific organization
Reconciliation performs no semantic reasoning.
All semantic decisions have already been made by the Knowledge Compiler.
Without KIR, the LLM is responsible for simultaneously:
- understanding documents
- extracting concepts
- resolving aliases
- maintaining semantic consistency
- updating wiki pages
- preserving workspace structure
These concerns become tightly coupled.
Introducing KIR separates them into two independent systems:
Knowledge Compiler
↓
Knowledge IR
↓
Knowledge Reconciler
This provides several advantages.
Knowledge exists independently from any wiki implementation.
Changing from Obsidian to Logseq does not require recompiling raw sources.
The same semantic knowledge can be synchronized into multiple targets simultaneously.
For example:
Knowledge IR
├──► Obsidian
├──► Logseq
├──► MkDocs
├──► Neo4j
├──► Search Index
└──► LLM Context
The Knowledge Compiler can evolve without changing synchronization logic.
Likewise, new synchronization targets can be added without modifying the compiler.
Given identical:
- raw sources
- compiler version
- prompt version
- schema version
the compiler produces identical Knowledge IR.
The semantic model is reproducible and versionable.
Adding a new source performs the following pipeline:
Raw Source
↓
Document Compiler
↓
Document IR
↓
Knowledge Compiler
↓
Knowledge IR
↓
Knowledge Reconciler
Only the affected documents and concepts are recompiled.
The workspace is synchronized incrementally.
Questions may be answered from:
- Knowledge IR
- synchronized workspaces
- both together
Depending on the application.
Answers generated during exploration can themselves become new raw sources and enter the compilation pipeline.
Knowledge continuously accumulates.
The compiler continuously validates the semantic model.
Examples include:
- duplicate concepts
- conflicting definitions
- taxonomy conflicts
- orphan concepts
- circular relations
- missing provenance
Workspace-specific validation (broken links, orphan pages, missing indexes, etc.) belongs to the reconciler layer.
Maintaining a knowledge base is not difficult because reading is difficult.
It is difficult because semantic consistency is difficult.
Humans gradually stop maintaining links, summaries, definitions, and cross-references.
Traditional wiki systems mix semantic knowledge with presentation.
KIR separates these concerns.
The Knowledge Compiler performs semantic reasoning once.
Knowledge Reconciler projects that semantic model into any desired workspace.
As a result:
- knowledge accumulates instead of being repeatedly rediscovered;
- semantic reasoning is centralized and reproducible;
- presentation becomes replaceable;
- knowledge survives changes in tools and workflows.
The wiki is no longer the knowledge base.
The wiki is one possible view of the knowledge base.