Heart of Gold

Heart of Gold is a search engine that returns what you asked for and then, sometimes, one passage you did not ask for. It first returns a ranked set of passages, which I call the floor, and can then add a passage that passes a statistical test of its connection to that set. Passing the test is a reason to look at the connection, not proof that it is useful, so the page shows the evidence and leaves that call to the reader. It runs locally with MCP tools for Claude and Codex; the public page (opens in a new tab) puts the same retrieval in a browser.

In a businessInternal search has to handle exact terms and loosely worded questions, and it has to stay usable when one component is slow. This engine combines keyword and semantic search with reranking, time budgets, and fallbacks.

Technical skills: Python · PyTorch · sentence-transformers · SQLite · FastMCP · Hybrid retrieval / reranking

My contribution

I designed how additional connections are selected while keeping the relevant results, and built the engine. I recorded my own judgments of the worked example separately from the model panel’s labels.

Result and limit

The SEP demo shows the retrieved passages and the evidence for additional connections. The system can select these connections, but I haven’t yet collected enough reader judgments to say how often they help.

A connection worth reading

A question about inherited claims to restitution brought back passages on reparations, plus an unexpected link to biological inheritance systems. Both entries ask what passes to descendants and on what grounds. I found that connection useful. The same query also returned two connections I judged to be artifacts, including one that confused Black feminist philosophy with black holes.

The worked example includes all three suggestions. Only one of the 129 pairs set aside for human calibration has an author verdict so far, so human precision is not yet established.

Technical methods and evaluation

Jump to the relevant passages, additional connections, or operations.

Corpora

The live Mindscape search index includes transcripts through 2026-09-21. The September 2026 refresh preserves the original model and certified additional connections while adding new episodes to relevance search; it does not rerun the original evaluations or certify new connections.

The floor

“Chunks” are short passages cut from source documents. An embedding represents a passage or question as a list of numbers so the system can compare their meanings. “Lexical” search matches words; reranking uses a second model to reassess the initial candidates against the question.

The system embeds a query both as a question and as a document, since the embedding model treats those two inputs differently. A cosine search produces a pool of 376 chunks. A lexical search adds candidates containing rare query terms, which helps when the embedding search misses names or technical vocabulary. Qwen3-Reranker-0.6B scores the candidates against the question, and the top 28 form the floor. If the time budget expires before all candidates have been scored, the response is marked partial and includes the number scored.

The budget was measured on the Mindscape instance (20,842 chunks; twelve blind queries against the fully reranked floor). The SEP instance has four times the chunks and a larger lexical channel, so its pools run larger. A 12 s budget scored 376 of 468 candidates on one query, in 13.9 s, so the 12 s row does not carry over. Both public pages show the cosine floor first and swap in the reranked result when it arrives.

The surplus

A TopK sparse autoencoder (17,281 features, k = 28) identifies features shared by chunks from different documents. Pairs of documents sharing a feature can be considered for the surplus. The test compares the strength of that connection with pairs with comparable overall similarity, while the citation graph excludes connections already linked by the encyclopedia’s editors. A placebo comparison checks whether matched random pairs also pass. The write-up explains the conditions and the failed versions. In the research client, the model writes its answer before it can learn which passages were added through this procedure.

Operations

The server reports when its models are ready and labels responses that use a fallback or run out of time. It also logs each query and the version of the floor returned. The Mindscape search page and private AMA workflow use this server; the public AMA checker uses its own question index. Ask the Stanford Encyclopedia of Philosophy runs the Ask Mindscape answer-generation workflow over the encyclopedia instance.

Stack

Python, sentence-transformers, PyTorch on Apple silicon, SQLite, FastMCP; a LangGraph client in the companion project.

My role

I designed a system that keeps its relevant search results while adding connections selected by a separate test, and built it. I also judged the worked example myself, including the two suggestions I rejected, and recorded those judgments separately from the model panel’s labels.

Where this could be useful

This approach could support research across a technical library, with source passages available for inspection. Whether the additional connections help would need to be tested with the people doing that research. These are potential applications of an independent project.

If you are hiring, the experience page says which business problem this project shares its mechanics with.