Com­pile your Know­ledge Base into a LLM Wiki

There is doc­u­ment­a­tion. Somewhere.

Teams accu­mu­late know­ledge faster than they organ­ize it, and most of it is hard to find when someone actu­ally needs it. At a data-plat­form cus­tomer, the plat­form team keeps a solid know­ledge base for the developers who build data products on it. But that cov­ers the plat­form itself. Ship­ping a data product also means man­aging AD groups, work­ing with the Git­Ops stack, and con­sult­ing the enter­prise archi­tec­ture cata­log. Those docs live else­where: other Con­flu­ence spaces, Ser­vi­ceNow art­icles, the READMEs of ref­er­ence pro­jects. Cross-ref­er­ences are often miss­ing, espe­cially across tools. So find­ing the right inform­a­tion is hard, espe­cially dur­ing onboard­ing — and when search fails, the default is a ticket to the plat­form ops team. The prob­lem does­n’t fade with exper­i­ence: even senior developers struggle to keep up. None of this shows up as a budget line, but the cost is paid on every onboard­ing and every change.

Why today’s tool­ing falls short

The recent wave of LLM tool­ing prom­ised a fix. Retrieval-aug­men­ted gen­er­a­tion (RAG) embeds chunks of the docs and retrieves them at query time. The Model Con­text Pro­tocol (MCP) lets the model fetch live pages on demand. Both leave syn­thesis to query time. RAG returns chunks without con­text, because it starts from scratch on every query. MCP solves a dif­fer­ent prob­lem: fetch­ing live data, not com­pil­ing it.

What neither does is pre­pare the know­ledge in advance: com­pile it once, cite every claim, link related ideas expli­citly. Andrej Karpathy named that pat­tern an LLM Wiki in April 2026. The rest of this post is what an LLM Wiki is, where it earns its keep and where it breaks, and how we served one from Snow­flake — mirrored by a pub­lic scaf­fold you can clone and run.

Com­pil­a­tion beats re-synthesis

Karpathy’s fram­ing is a soft­ware-engin­eer­ing ana­logy: com­pil­a­tion beats re-syn­thesis. A com­piler turns source into a bin­ary once; you do not re-com­pile on every run. An LLM Wiki applies that to know­ledge. The LLM reads the raw sources and syn­thes­izes them into cur­ated Mark­down pages, link­ing related con­cepts and sur­fa­cing gaps and con­tra­dic­tions. From then on, you query the wiki — not the raw sources.

A work­ing wiki has three lay­ers. Raw sources — the scattered tools from above — sit in folders the LLM only reads. The wiki is a par­al­lel folder of Mark­down pages, one per concept, with expli­cit [[wikilinks]] between related top­ics. Because the LLM owns the wiki entirely, it cre­ates pages, updates them, and records where each claim came from. The schema — a single AGENTS.md file — tells the LLM how to struc­ture the wiki and what each page should contain.

Con­cretely, that is three places on disk:

knowledge-wiki/
├── AGENTS.md                    # the schema: how the agent maintains the wiki (you own this)
├── raw/                         # immutable copies of every source; the agent only reads here
│   ├── confluence/
│   ├── servicenow/
│   └── repos/
└── wiki/                        # the curated knowledge base; the agent owns it
    ├── index.md                 # the hub every page links back to
    ├── concepts/
    ├── decisions/
    ├── runbooks/
    └── sources/                 # one record per source: its URL, date, and the pages it touched

Keep­ing it cur­rent takes four oper­a­tions: ingest­ing new sources into pages, answer­ing ques­tions from the wiki alone, lint­ing for stale or orphaned con­tent, and refresh­ing pages after sources change.

What a cur­ated wiki page looks like

Every wiki page opens with a front­mat­ter block: tags, a status field, and a last_updated date. Concept pages use a matur­ity status — stub, growing, mature — that advances as other pages cite them. Decision pages use their own vocab­u­lary: proposed, accepted, superseded. The lint oper­a­tion reports drift auto­mat­ic­ally. For example, if a mature page drops below its cita­tion threshold after a refresh, lint bumps it back to growing.

Take one page — decisions/use-hidden-partitioning, from a tiny, pub­lic example wiki com­piled from two Apache Ice­berg docs (par­ti­tion­ing and par­ti­tion evol­u­tion). It con­tains three [[wikilinks]] to related pages, plus two source cita­tions at the bot­tom, so every claim is trace­able. Each source cita­tion links to a source page — a wiki record car­ry­ing the ori­ginal URL, the inges­tion date, and the list of wiki pages updated from it. Fol­low the link to read the ori­ginal; re-run the ingest to pick up changes. This decision record cap­tures why hid­den par­ti­tion­ing beats the Hive-style approach: the prob­lem con­text, the rationale from the Ice­berg docs, and the tradeoffs accepted.

The raw Apache docs behind it are a single flat page: no internal links, no metadata, no record of why the design was chosen. The cur­ated record keeps just the choice, the con­sequences, and the sources.

Here is that page, lightly abridged:

---
tags: [decision]
status: accepted
decided_on: 2013-01-01
last_updated: 2026-05-27
---

# Use Hidden Partitioning Over Hive-Style Partitioning

## Context
Early big-data table formats implemented partitioning as user-visible columns.
Three classes of bugs were endemic: silent data corruption from incorrect
partition value formats, silent full-table scans when readers omitted the
partition filter, and query breakage when the partition scheme changed.

## Decision
Iceberg tracks partition specs inside the table's [[Table Metadata]], not in
the user-facing schema. Writers supply data for logical columns only.
Readers write predicates against logical columns; the engine translates them
internally. This is referred to as [[Hidden Partitioning]].

## Sources
- [[sources/iceberg-docs-partitioning]]
- [[sources/iceberg-docs-evolution]]
- [[sources/iceberg-docs-reliability]]

Scaled up, a wiki forms the graph this post opened with: each dot a page, each line a [[wikilink]]. What was scattered across sep­ar­ate doc­u­ment­a­tion pages is now cross-ref­er­enced and trace­able in one place.

RAG, MCP, and the wiki — com­ple­ment­ary, not competing

The obvi­ous objec­tion is that this is just RAG with extra steps. Those extra steps are the point. At serving time the wiki is indeed retrieval — a Snow­flake search ser­vice, later in this post — but it retrieves over com­piled, cited pages rather than raw chunks. All three approaches dif­fer less in how they retrieve than in what they retrieve from.

What each approach retrieves over

RAG oper­ates on raw chunks at query time, so source attri­bu­tion stays weak — a doc­u­ment ref­er­ence, not a per-state­ment cita­tion. MCP fetches live pages, but the answer lacks syn­thesis. The LLM Wiki sits between the two: know­ledge is com­piled in advance, so the wiki bakes in cross-ref­er­ences and cita­tions, and fresh­ness becomes incre­mental rather than per-query.

RAG still wins at mil­lions of doc­u­ments, hard latency require­ments, and fully unstable con­tent. The wiki wins at focused, stable know­ledge bases — team wikis, research domains, gov­erned know­ledge sur­faces. Its edge over raw-page RAG is the per-state­ment attri­bu­tion in the table below: every claim traces to a source, which makes answers auditable.

Which tool for which job

MCP and the LLM Wiki are com­ple­ment­ary, not com­pet­ing. MCP is for live oper­a­tions: open a ticket, fetch the cur­rent incid­ent. The LLM Wiki is for stable know­ledge: explain a concept, or trace a decision back to its source. dbt Labs, for instance, recently added tools to its MCP server that let an LLM pull from dbt’s doc­u­ment­a­tion on demand. That works well when a query is a lookup — fetch the right canon­ical page and you are done. The wiki is the bet­ter fit when answer­ing takes real syn­thesis — recon­cil­ing sev­eral pages into one coher­ent answer. The decid­ing factor is how much syn­thesis a query needs, not how many sources you have.

So the wiki and RAG are not rivals at serving time. At scale the wiki is served via retrieval; every retrieved unit is simply already a syn­thesis with its cita­tions attached. An informal test put the com­piled cor­pus ahead on ques­tions span­ning sev­eral sources, and roughly even with raw pages on single-page look­ups — the serving sec­tion has the detail.

Where it falls short

Syn­thesis can lose detail: Com­pil­ing many pages into one means leav­ing things out. Answers come from the com­piled wiki, not the raw sources, so a detail dropped dur­ing syn­thesis is gone at query time — and, like any LLM step, syn­thesis can intro­duce errors of its own. Per-state­ment cita­tions make a page audit­able after the fact, and lint catches some drift, but neither pre­vents a lossy or wrong page up front. Spot-check­ing com­piled pages against their sources stays necessary.

It only pays when syn­thesis does: The wiki’s value is the syn­thesis it bakes in. If your users’ ques­tions are point look­ups — fetch one page, read one value — that syn­thesis goes unused, and raw-page RAG or MCP-on-docs answers just as well. The wiki pays off when answer­ing means pulling sev­eral pages together, not when it means find­ing one.

Stay­ing cur­rent has a cost: The wiki is com­piled, not live. Keep­ing it cur­rent means recom­pil­ing, and recom­pil­ing often enough to track a fast-mov­ing source costs real time and tokens. In fast-chan­ging domains or live incid­ents it will lag. The fix is to split the work: the wiki for stable know­ledge, MCP for live operations.

These lim­its point the same way: the wiki is worth the effort when know­ledge is stable and answer­ing takes syn­thesis. Once it exists, the next ques­tion is access — a query­able inter­face for end users, a dev tool for engineers.

Serving the wiki from Snowflake

The wiki lives in Git. A cod­ing agent with the repo checked out can already query it — that is the simplest way to use one. But that asks every­one to clone the repo and know how to point an agent at it, and it does noth­ing for non-tech­nical users. To reach every­one, the wiki has to be served some­where: a chat inter­face for non-tech­nical read­ers, and a hos­ted end­point that engin­eers and their cod­ing agents can call without each clon­ing the repo. Snow­flake offers the tools for that, and is what we used — the wiki sits next to the gov­erned data, and search, agents, and access con­trol come built in. It is one instance of the broader Snow­flake-as-GenAI-plat­form story.

The pipeline

Get­ting the wiki into Snow­flake is the easy part. Snow­flake’s Git integ­ra­tion can pull the repo dir­ectly; in our test we instead pub­lished the Mark­down to a cloud-stor­age stage, to work around access lim­its in the envir­on­ment. Either way, a sched­uled job loads the pages and indexes them for Cor­tex Search, re-pro­cessing only what changed.

The step that needs care is chunking. Cor­tex Search embeds a text column for semantic search but trun­cates any­thing past its embed­ding win­dow — about 512 tokens, roughly 385 words — so a whole page indexed as one row would lose its tail. The job there­fore splits each page into smal­ler chunks. Split naively, though, and a chunk loses its page’s front­mat­ter and source links, and the cita­tion trail breaks. So the job car­ries each page’s metadata and cita­tions into every chunk — a retrieved chunk can still name the page and source it came from.

The same approach is mirrored by a pub­lic scaf­fold of SQL scripts you can read end to end: the input tables and a sync pro­ced­ure that parses each page’s front­mat­ter and chunks it, the Cor­tex Search ser­vice, helper UDFs that resolve [[wikilinks]], and the Cor­tex Agent that answers over them. Clone cortex-kb-scaffold, point it at a stage of Mark­down, and you get the pipeline described here.

Reach­ing the agent

On top of the search ser­vice sits a single Cor­tex Agent — the thing users and tools actu­ally talk to. It answers from the wiki, fol­lows [[wikilinks]], and returns the source cita­tions attached to each chunk. That one agent serves every­one two ways: non-tech­nical users chat with it in CoWork (formerly Snow­flake Intel­li­gence) inside Snowsight, and engin­eers reach it from Cor­tex Code (since rebranded to CoCo, Snow­flake’s cod­ing agent) or any app over its REST API — a setup teams already run over raw sources alone, no wiki in between. It is the same agent and the same cita­tions either way, with no sep­ar­ate iden­tity layer. And because it is all Snow­flake objects, the same roles that gate the rest of the plat­form gov­ern access — no bolted-on per­mis­sion layer for the wiki.

Serving it this way also makes the com­pil­a­tion step easy to test. An informal com­par­ison ran the same Cor­tex Search ser­vice over a wiki com­piled from part of a real know­ledge base, then over that source’s raw pages. On ques­tions whose answer spans sev­eral sources, the com­piled cor­pus answered with fewer tokens, lower latency, and com­par­able cor­rect­ness. On plain single-page look­ups the two were close, some­times with a slight edge to the raw pages.

When to use it, and how to start

The pat­tern fits a spe­cific shape: stable know­ledge where answer­ing a ques­tion takes real syn­thesis, onboard­ing is a recur­ring cost, and answers have to trace back to a source. If your team sits between a RAG bot that returns scattered chunks and a ticket queue, a cur­ated wiki can fill the gap. It does not fit every case. For point-lookup ques­tions — or a single canon­ical source you can hand an agent dir­ectly — raw-page RAG or the dbt-style MCP-on-docs pat­tern is sim­pler and just as good. For very large, real-time, or fast-mov­ing con­tent, RAG or MCP wins.

Karpathy’s gist is the ori­ginal concept note — enough to build one from scratch. If you would rather not start from a blank folder, I keep a wiki­seed tem­plate: the file lay­out shown earlier, the AGENTS.md schema that tells the agent how to main­tain the wiki, and the skills that run the loop. No CLI and no build step — just Mark­down and an AI cod­ing agent. Get­ting star­ted takes two steps:

1. Get the files. Use the Use this tem­plate but­ton (or git clone) to cre­ate a folder you name.
2. Ini­tial­ize. Open it in Claude Code, Git­Hub Copi­lot, or Cor­tex Code and type initialize the wiki. It asks for your wiki’s name, domain, and cat­egor­ies, then clears the example pages so you start clean.

From there the loop is three plain-lan­guage trig­gers: Ingest <url> pulls a source into raw/ and syn­thes­izes pages from it, Query: <question> answers from the wiki with cita­tions, and Lint the wiki flags stale or orphaned pages. Point it at your own docs, and the wiki fills in.