What is an advantage worth?

What is an idol worth on Survivor? On a recent Rob Has a Podcast, Rob Cesternino asked how often each player would win if a season were replayed from the merge a thousand times. Drew Basile wanted to know how many extra wins each advantage is worth. This project builds that simulator and runs both experiments, and it borrows chess piece values to compare the advantages. The demo (opens in a new tab) shows the advantage values and lets a visitor play any season out from the merge in the browser.

In a businessSimulation answers questions no historical table can, like what one more resource is worth to the person who holds it. This project randomizes the treatment inside the simulation, compares it with an exact baseline, and checks the design with a placebo before reading any estimate.

Technical skills: Python · scikit-learn · Simulation / Monte Carlo · Experimental design · JavaScript / Web Workers · Model validation

My contribution

I wrote the plan before fitting, built a model of how each castaway votes, a game engine that rebuilds every model input from simulated history and is checked against all 77 real seasons, the advantage experiment, and a browser port that matches the Python to fifteen decimal places.

Result and limit

A perfectly timed idol multiplies its holder's chance of winning by 2.25 from the merge; used the way players used them, by 1.34. A vote steal is worth about 1.7 extra votes, and a nullifier almost nothing. Winner odds from the merge beat a random guess only slightly, and the page says so.

What an advantage is worth

In each simulated season, one castaway still in the game is picked at random and handed the advantage. Without one, a castaway picked at random wins exactly one season in n, where n is the number left, because the win chances sum to one. So the value of an advantage is the holder’s win chance minus 1/n, and the design needs no separate control runs. Every completed US season with a merge was simulated 400 times per advantage, from the merge and again with eight left, with models fitted without that season.

The baseline at the merge is 9.3%, with 11.1 castaways left on average. Intervals bootstrapped over seasons are about ±0.5 points. Perfect timing plays the advantage the first time it would save its holder, which assumes the holder can see the votes, so it is an upper bound. Blind timing plays it at one council picked at random. The show’s rule for the idol uses how often real idol holders survived when they were the main target, 29% of the time.

Timing is most of an idol’s value. Played perfectly, an idol more than doubles its holder’s chance. Played the way real players played idols, it raises the chance by about a third. Players captured about 25% of the idol’s potential gain. With eight left, playing an idol at a random council does as well as real play (×1.22 against ×1.23).

A vote steal is worth about 2.2 extra votes at the merge and 2.7 with eight left. A steal moves two votes, one off the holder’s pile and one onto their choice. An extra vote moves one.

The nullifier is close to worthless to its holder. It only matters on a night when someone else plays an idol for the person the holder voted for.

Advantages get a castaway to the end more than they win the game. The gain in reaching the final tribal council is 1.8 to 3.8 times the gain in winning, because the jury vote is close to a coin toss in this simulator.

Who gains most

Each run also records who held the advantage and where they stood when they got it: how many allies they had left, whether their original tribe was outnumbered, whether they got votes at the last tribal council, and the era. The placebo runs give every castaway’s own chance with nothing in play. So each holder can be compared with themselves, and the gain can be split by situation. The groups were fixed in the plan before the runs.

At the merge, the holder’s spot barely matters. A perfectly timed idol adds 11 to 12 wins in 100 whether its holder has no allies left or three or more, and whether their original tribe is outnumbered or not. Played the way real players play idols, it adds 2 to 3 in every group.

With eight left, allies start to matter. A perfectly timed idol adds 7 wins in 100 for a holder with no allies left and 10 for one with allies. A vote steal adds 3 against 4 to 5. An idol saves its holder once, and a holder with no allies still has to survive the votes that come after it.

The idol is worth less in the new era with eight left. A perfectly timed idol adds 7 wins in 100 from Season 41 on, against 10 before. That rests on the ten new-era seasons, and the runs don’t say why. At the merge the two eras are close, 11 against 12.

Favourites gain a little more than underdogs. At the merge a perfectly timed idol adds 14 wins in 100 for a castaway the simulator already favoured and 11 for an underdog. This split uses the same runs to say who is a favourite, so it is descriptive.

Before the merge, being outnumbered changes little. Drew’s example is an extra vote that is useless on a four-to-one tribe and valuable at a swap tribal. A third set of runs starts at the first tribal council on each season’s last pre-merge tribes, after the last swap, because the simulator does not model swaps. There a perfectly timed idol adds 11 wins in 100 and more than doubles its holder’s chance, from 7 in 100. A perfectly timed extra vote adds about 2 wins in 100 whether its holder is outnumbered on their tribe by another original tribe or not. Perfect timing spends it only on a night it saves the holder, so the four-to-one nights when it could not help never count against it. A vote steal helps an outnumbered holder a little more, 4.5 wins in 100 against 3.4, within the margin for luck.

How the simulator works

A vote model, not a boot model. The earlier Who goes home? models predict who leaves a tribal council. A simulator needs the step underneath, whom each castaway writes down. Votes give about seven times as many observations as boots (9,200 first-round votes against 1,160 boots), and they keep targeting apart from the idol. Players vote for an idol holder about as often as chance predicts (10.5% of votes against 11.1% of candidates). So an idol’s value comes from cancelling votes, which a rule can supply.

The model reads the voting record. For each voter and candidate it uses who has voted with whom, who has voted against whom, the voter’s allies’ last targets, immunity wins, idols held and the council’s make-up. It is gradient boosting trained on all 77 seasons from five countries and tested on US seasons it never saw. Votes are drawn for everyone at once, and a fitted setting controls how tightly blocs vote together. The most votes go home, and an idol held by the main target cancels the votes against them at the rate real idols did.

It predicts boots better than the earlier models. Turned into boot probabilities, the vote model picks the castaway who leaves 31.5% of the time, against 27.8% for the earlier model on the same gameplay records and 16.8% for a random guess. Its log loss is 1.761 against 1.786, a gain of 0.025 with a 95% interval of 0.002 to 0.047 across seasons.

A simple rule does much worse. A fair test of a vote model is a rule anyone could apply. Each voter writes down whoever their allies have voted for most, and the most votes go home. That rule names the boot 21.7% of the time, about the same as “whoever has received the most votes so far” (21.6%). The adopted vote model names it 30.5% of the time, on seasons it never saw, and a random guess 16.8%.

It counts a planned split as one alliance. When a majority splits its votes to beat an idol, both halves count as voting together and as voting with the majority, using survivoR’s record of 84 planned splits. That leaves the results across all seasons where they were and improves the last ten seasons, where splits are common (log loss per vote 1.649 against 1.657). Knowing who has just lost an ally added nothing on top.

The rest of the season comes from smaller models. One picks which tribe loses immunity before the merge, another the individual immunity winner after it, and a third each juror’s vote. Castaways find idols at the rate of their era. One boot per council, no quits or returns, and reward challenges are not simulated.

The engine is checked against the data. Replaying all 77 real seasons, the simulator reproduces every input the vote model was trained on, across 57,743 rows of voter and candidate. The browser version runs the same code, and a test checks that it computes the same inputs and vote probabilities as the Python to fifteen decimal places.

How good is it?

It is a weak oracle for who wins. Simulating each past season from its merge, the simulator gives the real winner a 10.1% chance on average, against 9.1% for a random guess. It ranks the real winner first in 12% of seasons, against 9% by chance, and in its top three in 31%, against 27%.

The merge vote and the jury are the hard parts. At each season’s first merged tribal council, the vote model does worse than a random guess (log loss 2.24 against 2.18), because that vote is where alliances break and re-form. Given the real finalists and everything before the final tribal, the jury model makes the real winner its favourite in 46% of seasons.

The advantage values hold up anyway. Each one averages 20,000 runs over 50 seasons against an exact baseline, so a season the simulator reads badly adds noise but does not tilt the estimate. The placebo hands out nothing, so it changes nothing in the code and its result is luck alone. It returns +0.4 points at the merge and −0.3 with eight left.

The replay tool compares like with like. On the demo page, each replay runs twice, once with the advantage and once without, on the same random draws at every challenge, vote and jury. The two runs differ only where the advantage changed something. That makes the range on each gap 1.4 to 2.5 times narrower than two separate batches would give.

Can the edit pick the winner?

Sean Falconer, working with Amanda Rabinowitz and Angie Caunce, once forecast the winner week by week from personality traits that fans coded for each player. The same question can now be put to Gamebot’s caption features, which cover 50 seasons. After each episode, a model ranks the players still in the game using only the episodes aired so far, and each season is scored by a model that never saw it.

The edit picks the winner more often than a guess, though not by much. The name-based edit features put the eventual winner first at 19% of forecast points, against 11% for a guess, and at 27% against 16% in the last third of a season. The winner ends the season as the edit’s favourite in 17 of 50 seasons. The edit’s winner is talked about often and described as trusted, is named by the host, and is rarely named as a target.

The voting record says little about who wins. Bonato and Malik ranked players by their place in the network of votes. Ranking by closeness in that network, votes received and shared targets does no better than a guess, and neither does the wider record of immunity wins and correct votes. Adding the edit to the record helps, which matches the simulator’s trouble naming winners from gameplay alone.

The edit works because the editors know the ending, so these numbers describe reading the show, not the game. One caveat comes from the plan itself. Its grid of penalty settings proved too narrow, and the comparison with a guess passes only after the grid was widened, a change logged after the first run.

What else was tested

The edit predicts tonight’s vote and little else. Gamebot’s caption features record who is named as tonight’s target, who is mentioned, and who appears in the “previously on” recap. Measured at the moment of the vote, they lift the pick of tonight’s boot from 32.1% to 38.9% and the top three from 62.8% to 83.7%. The same features from earlier episodes add nothing before an episode airs (a change in log loss of −0.005, interval −0.015 to +0.003). Speaker-attributed features, such as airtime, confessional topics and conversation partners, add nothing on top of the name-based ones, and neither does the music under each line.

Demographics add nothing to the vote model. Occupation and personality type each left the model flat or worse, and race and ethnicity were reported and never used, a rule set before fitting.

What this does not show

The values are averages over every US season, including those from before advantages existed, so each one says what an advantage would do if dropped into a season. Other players react to a holder only as much as the historical data shows, which is barely. The data records who held an advantage, not who knew about it, so a known idol and a secret idol look the same to the simulator. A test on the episode subtitles marked when another castaway is heard knowing about a holder’s idol. Holders known on screen drew the votes the model expects. Holders nobody was shown knowing about drew about a quarter fewer, though that could also mean the edit left them out of the story. Adding the label did not sharpen the vote predictions, so the simulator leaves it out. The simulator knows the voting and challenge record and nothing about relationships the record does not capture. A season’s odds are a description of 1,000 simulated seasons, not a forecast about a person.

Data: the survivoR tables through Gamebot, my warehouse of Survivor statistics. Every plan was written into the specification before the models were fitted, with later changes logged beside it.