Has this been asked on Mindscape?
Every month Sean Carroll answers listener questions in an AMA episode, and the show notes preserve who asked what. That makes the archive its own labeled dataset. This page harvests every question-and-answer pair from the 63 AMA episodes, embeds the questions, and lets a listener check a new question against all of them before submitting it. A match shows the original asker, the episode, the timestamp where Sean reads the question, and how close the match is. There is no language model in the loop; the page is an index and a similarity, which is why it answers in about 150 ms and never invents an answer.
Evaluation
The corpus supplies its own test set. A harvested question is, by construction, already answered, and its source episode is known.
| Query set | n | Recall@1 | Recall@5 | Median rank | Worst rank |
|---|---|---|---|---|---|
| Verbatim question | 200 | 1.00 | 1.00 | 1 | 1 |
| Paraphrased question | 50 | 0.98 | 1.00 | 1 | 2 |
The separation between a question and its nearest other question (median cosine 0.635, 90th percentile 0.74) leaves room for the “close variant” band the page reports between an exact hit and a miss.
The fuller version
The public page is the index-only tier of a larger tool. The private version runs a LangGraph workflow over Heart of Gold that retrieves the passage where Sean answers, extracts the precedent, and classifies the new question as already answered, a close variant, or new. On 20 harvested questions it never saw the source of, retrieval found the source AMA every time, the extracted precedent was the right one 85% of the time, and the status was correct in every case, at about 73 seconds per question. The public page trades that depth for speed and for having no model that could confabulate.
Stack
FastAPI, Qwen3-Embedding-0.6B, a precomputed index; served from a Mac under launchd through a named Cloudflare tunnel.