Heart of Gold
Most retrieval systems return the passages closest to the query and stop. Heart of Gold returns those too, as a floor with a stated guarantee, and then adds a small surplus of passages the query did not ask for, chosen because a statistical certificate says they are connected to the floor in a way the corpus supports. The idea comes from beyond-accuracy work in recommender systems, where serendipity is measured rather than hoped for, applied to retrieval-augmented generation. It runs locally and is exposed to Claude agents as an MCP server with seven tools.
Corpora
| Stanford Encyclopedia of Philosophy | Mindscape | |
|---|---|---|
| Entries | 1,795 | 436 episodes |
| Chunks | 85,263 | 20,842 |
| Graph | 19,537 citation edges, 95.8% resolved, one connected component | timestamps and speakers per chunk |
The floor
A query is embedded twice (once as a query, once as a document, because the two spaces differ) and a brute-force cosine pass returns a pool of 376 chunks. A lexical channel adds any chunk containing the query’s rare terms, so names and jargon are never lost to the embedding. A cross-encoder reranker (Qwen3-Reranker-0.6B) reads every candidate against the query and returns a probability; the top 28 are the floor. Under an interactive deadline the reranker scores the dense pool first and the lexical additions second, and the floor is labeled partial when the budget ran out, with the count. Nothing is thresholded; the caller sees the probabilities.
| Rerank budget | Floor kept (28 chunks) | Top 10 kept | Wall time (median) |
|---|---|---|---|
| 4 s | 0.54 | 0.53 | 6.0 s |
| 6 s | 0.69 | 0.73 | 8.1 s |
| 8 s | 0.75 | 0.79 | 8.4 s |
| 12 s | 0.99 | 0.99 | 14.0 s |
The surplus
Chunk embeddings pass through a TopK sparse autoencoder (17,281 features, k = 28) that gives each chunk a small set of active concepts. Pairs of documents that share a concept are candidates for a connection. A candidate is certified only when a three-part PMI test says the co-activation is stronger than the corpus’s own citation graph would predict, with the citation graph as the null model. A placebo arm runs the same mining against a scrambled reference and refuses a pass when the placebo lights up; it has refused passes, and those runs are recorded as findings rather than adjusted. The client that consumes the surplus is built so the synthesizing model never learns which passage was the certified one, enforced by the graph’s topology rather than by a prompt.
Operations
Background warm-up with explicit readiness states, a latency budget per call with labeled fallbacks, and an append-only log of every query with the floor version it received. The Mindscape search page and the AMA checker are two clients of the same server.
Stack
Python, sentence-transformers, PyTorch on Apple silicon, SQLite, FastMCP; a LangGraph client in the companion project.