Mindscape Explorer
Mindscape Explorer treats eight years of podcast transcripts as one corpus and asks what the show’s breadth looks like. The site has six doors. Search finds the moment in the archive closest to a question, with a timestamp and a link. Recommend gives three lists per episode, the obvious one, a serendipity list, and a deliberate “least likely from here”. Explore holds the topic models, the episode map, the guests, and the timeline. Findings states each result with its numbers and its caveat. The build log tells the story in order; this page holds the numbers.
Corpus
| Measure | Value |
|---|---|
| Posts fetched from the WordPress API | 444 |
| Episodes with a transcript | 439 |
| Transcripts parsed into labeled utterances | 436 (3 were published without speaker labels and carry a flag) |
| Transcript words | 7.39 million |
| Date range | July 2018 to September 2026 |
| Requests to the site per full run | 20 to 30, three seconds apart, every response cached |
Topic model
TF-IDF over one- to three-word terms, factored by NMF. The number of topics is swept and selected by the one-standard-error rule on a combined coherence-and-stability score, which gives an anchor of 50; the two other resolutions are 27 and 92, the anchor divided and multiplied by τ ≈ 1.839. Nesting between resolutions is inspected with a similarity matrix and a best-parent table rather than assumed.
Check against the host’s own tags
The topics were fit without seeing Sean’s hand-assigned subject tags. One classifier per tag, tuned on inner folds and scored by nested cross-validation, 17 tags, format tags excluded.
| Features | Model | Macro AP | ROC-AUC |
|---|---|---|---|
| Topic loadings, three resolutions concatenated | Gradient boosting | 0.649 | 0.913 |
| Full TF-IDF matrix (baseline) | Logistic regression | 0.549 | 0.848 |
| Chance | 0.078 | 0.500 |
Physics (0.90), Biology (0.89), Astronomy (0.88), Economics (0.84), and Environment (0.81) have vocabularies of their own. Humanity (0.42), World (0.38), and Psychology (0.37) barely beat chance; those tags record an editorial judgment that word counts do not carry.
Recommender
Relevance is topical affinity (the Bhattacharyya coefficient between topic distributions, averaged over the three resolutions). Metrics follow Kaminskas and Bridge (2016); intervals are 95% percentile bootstraps over 439 seed episodes, 2,000 resamples. Coverage and Gini are census quantities and carry no interval.
| List | Intra-list diversity | Novelty | Catalogue coverage | Gini |
|---|---|---|---|---|
| More like this | 0.49 (0.48–0.50) | 0.64 (0.63–0.65) | 99.5% | 0.23 |
| Serendipity | 0.82 (0.81–0.82) | 0.64 (0.64–0.65) | 99.8% | 0.31 |
| Least likely from here | 1.00 | 0.82 (0.82–0.82) | 36.7% | 0.91 |
Serendipity minus more-like-this, paired on the same seeds, is +0.33 (0.32–0.34) in diversity and +0.001 (−0.007–0.008) in novelty. The serendipity list is twice as varied at no cost in novelty.
Episode routes
Each transcript is cut into tenths, projected onto the fixed topic basis, and measured as a walk (sum of Hellinger steps). Intervals are 95% bootstraps over episodes; p-values are two-sided permutation tests on the difference of medians.
| Format | n | Median route | 95% interval |
|---|---|---|---|
| Solo lecture | 28 | 3.12 | 2.66–3.90 |
| Interview | 347 | 3.54 | 3.38–3.74 |
| AMA | 64 | 6.44 | 6.34–6.63 |
Solo minus interview is −0.42 (−0.90–+0.44), p = 0.22. AMA minus interview is +2.91 (2.66–3.16), p < 0.001. Solos wander as much as interviews do; the AMAs are the expeditions.
Second representation
33,522 passages and 49,760 interview turns embedded with Qwen3-Embedding-0.6B. The Mantel correlation between the lexical and semantic episode geometries is 0.644 (p = 0.0005, 2,000 permutations); 46% of an episode’s ten nearest neighbours are shared between the two. In embedding space the three formats walk similar distances (medians 2.14, 2.19, 2.36; rank correlation with the lexical routes 0.04), so an AMA changes its words far more than its meaning.
Search by meaning
The Search page runs Heart of Gold’s relevance floor over 20,842 chunks of about 376 tokens. The reranker scores about 32 chunks a second, so the interactive budget was measured rather than picked, on twelve blind queries against the fully reranked floor.
| Rerank budget | Floor kept (28 chunks) | Top 10 kept | Wall time (median) |
|---|---|---|---|
| 4 s | 0.54 | 0.53 | 6.0 s |
| 6 s | 0.69 | 0.73 | 8.1 s |
| 8 s | 0.75 | 0.79 | 8.4 s |
| 12 s | 0.99 | 0.99 | 14.0 s |
The service runs at 12 s. The page shows the cosine-only floor after about a second and swaps in the reranked list when it arrives. When the machine is asleep the same box falls back to a BM25 keyword index over the same passages, and says so.
Stack
Python, scikit-learn, LightGBM, sentence-transformers, UMAP, matplotlib; a make-style DAG runner with fingerprinted stages; one HTML file plus a data file and lazily loaded charts, served from Cloudflare with a Worker route for search; the search service on a Mac under launchd through a Cloudflare tunnel, with a D1 FTS5 fallback.