"Jeopardy!" Sabermetrics

America's Favorite Trivia Game

“Jeopardy!” is America's longest-running quiz show.  Originally airing 1964-1975, its Alex-Trebek-hosted 1984 reboot has become a staple of syndicated TV, airing over seven thousand episodes during the past thirty-three years.  Its regular audience may be modestly sized, but its fans are devoted, and its staying power has made it a ubiquitous reference point for trivia geeks.

Compared to other quiz shows, "Jeopardy!" is question-dense.  There are thirty questions in both the Jeopardy! and Double Jeopardy! rounds, as well as a single Final Jeopardy! question.  That makes the show's archive an excellent subject for large-scale data analysis, whether for fans looking to study the show's history or players looking to prepare for an appearance on the show.  Luckily, the "Jeopardy!" fan community has created the J! Archive, which contains almost every question ever aired and extensive data on how the contestants answered.  After scraping data from over 5,000 episodes, I created a 330,000-question dataset, beginning with the 1983 pilot and continuing up to the February 7th, 2017 show.  In what follows, I'll adapt some concepts from sports statistics to unveil secrets for maximizing performance.

"Jeopardy!" Is a Game of Skill—Until the Final Round

How important are the three rounds to determining the outcome of a game?  The dollar values are doubled in the second round, and in Final Jeopardy!, players can wager everything they’ve earned on a single question.  If you play badly in the first round, how easy is it to catch up in the second?  If you've dominated the first two rounds, how likely are you to be sunk by the high-stakes final question?

To answer those questions, we can use the concept of “Win Probability Added,” a baseball statistic describing how each event in a game affects the contestants' chances to win.  Each player begins “Jeopardy!” with a 33.3% chance of winning; by the end, someone has reached a 100% chance of winning.  Therefore, the winner has to add 66.6 percentage points to his or her win probability over the course of the game.  In a game where each round was equally and independently important, the average winner would add 22.2 percentage points of win probability every round.  Is that how "Jeopardy!" works?

It’s not far off.  Being in first place at the end of each round steadily increases a player’s chances of winning, with no round unimportant or all-important.  There’s something strange about these numbers, though.  The first round appears more important than the second—but how can that be, when the second round’s questions are worth twice as much as the first?

It's because the first two rounds are not independent variables.  "Jeopardy!" is a game of skill, so the change in win probability during the first round reflects not only who's in the best monetary position, but (to an extent) the relative talents of the three players.  In other words, their first-round scores predicts their likelihood of doing well in the second round.*  The  correlation isn't enormous: first-round score only accounts for less than 20% of the variance in the second-round score.  There's plenty of room to stage a comeback in Double Jeopardy!  Still, it's enough to make the first round tell viewers (and players) a little more about who's going to win than the second.

The round that tells us most about the final score, though, is Final Jeopardy, even though two rounds of information about player talent have been already baked into the odds.  That's because, as the regression above might make clear, its result is independent from the rest of the game.

Here are two blunt charts to show how arbitrary Final Jeopardy! is.  First, after thirty years, contestants have gotten the Final Jeopardy! question wrong almost exactly as often as they've gotten it right.  Second, having constructed a simple nearest-neighborhood classifier to try to predict whether a player would correctly answer the final question based on performance over the first two rounds, I found it could not do much better than 50%—that is, player scores tell you no more about who will get the last question correct than randomly guessing would.  Final Jeopardy! is a coin flip.

Start With the Hard Clues (but Hope You Don't Get a Daily Double)

"Jeopardy!" players seeking to maximize their earnings might wonder which questions offer the best return.  As the questions become worth more money, they become harder.  Which can be expected to pay out best—the easy, sure-thing ones or the high-value, challenging ones?

To answer that question, I calculated an expected-value payout for each value of question.  For every question in the set, I used the actual buzz outcomes to total the net dollar outcome and divided it by number of buzzes.  For example, if a $400 question provoked two wrong buzzes and one correct one, the average value was -$166 (i.e., ($400 -$400 -$400)/3).  To questions on which no one buzzed, I assigned a $0 value.  Finally, I averaged the results of questions from the same round and face value to determine expected values.

The high-value questions provide the best payouts.  They are, granted, harder than the easy ones.  Note that questions in the same round do not increase payout in exact proportion to increased value (e.g., the first round's $1000 question does not pay 25% more than the $800), and that Double Jeopardy! questions are harder than their equivalents in Jeopardy! (e.g., the $800 question in Double Jeopardy! is worth less than twice as much as the $400 question in Jeopardy!, though slightly more than the $800 Jeopardy! question). Still, face value increases faster than difficulty.

Theoretically, that means players should prioritize the higher-value questions.  However, they seem to do exactly the opposite: the most difficult questions are the least asked.  Standard “Jeopardy!” contestant practice is to start with the easy clues and work toward the harder ones.  That means, though, that if time runs out before the board is cleared, the highest-value questions remain unasked.  That is only a desirable result for a leading player who is playing defense late in the game to protect a lead.  Otherwise, everyone should be trying to maximize their prize winnings.  "Jeopardy!" players would do well to keep that in mind.

The only danger to high-value clues, are the Daily Doubles.  Players usually seem excited to land a Daily Double, but Daily Doubles bring bad luck as often as good.  While players can wager as much as they like on Daily Doubles, the fact that they are forced to answer (and face a possible penalty) causes payouts to plummet when the clues get harder.

On average, contestants wager about $1450 on the first round's Daily Double (with only a very a slight upward tick for wagers on the harder clues compared to the easier) and $2800 on second-round Daily Doubles.  Since Daily Doubles are roughly as difficult as other clues of their face value, it makes sense that payouts on the easy clues are much higher than for regular questions.   The reverse effect occurs for difficult clues, though.  Payouts steadily decrease as face value increases, because wrong responses cause players to take large penalties.  When landing on a Daily Double on the very hardest clues, players should keep in mind that only slightly more than 50% of Daily Doubles at the $2000 level in Double Jeopardy! are answered correctly.

"What is Australia?"

Like most shows produced by a permanent staff following a corporate vision, “Jeopardy!” hews to a central set of topics.  In fact, over 28,000 clues in its history have shared the same 300 correct responses.  A prospective “Jeopardy!” player should study these to prepare for an appearance.  Beyond their direct importance, these responses can also provide a manageable subset on which to study the show's question proclivities in more depth.

Astoundingly, the thirty-two most common correct “Jeopardy!” responses are place names.  "What is Australia?" is the most popular answer-in-the-form-of-a-question of all time, with "What is Canada?" (the most popular Final Jeopardy! response) close behind.  The three most common answer-types, in fact, are all different categories of place, accounting for 153 of the top 300 answers.  50 different U.S. states (including Puerto Rico, but excluding North Dakota) are represented.  Historical figures are also a recurring inspiration, especially American presidents.  Twenty of the forty-five presidents are in the top 300, accounting for about 2,000 correct responses.

However, common correct responses don't necessarily tell us what to study, because a place name can be the answer to almost any kind of question.  "What is Australia?" has been the correct responses to clues dealing with the Great Sandy Desert, biologist Elizabeth Blackburn, and the America's Cup.  Looking at the most common categories show us in what context these answers appear.  The show returns to its favorite categories even more frequently than its favorite answers: just 100 categories account for nearly 10% of all those ever played.

Geography, it turns out, is the favorite category subject, but it does not dominate in the way we might have expected from the frequency of place names in the answer set.  One important group of categories that might not have been obvious from the top 300 answers is word games.  An aspiring “Jeopardy!” player needs not only memorize all the state capitals, but should practice calling to mind ten-letter words and before-and-after puns.

Conclusion

A great deal more information can still be mined from the "Jeopardy!" archive, revealing further strategic advantages that aspiring players could leverage when preparing to go on the show.  In the end, of course, the outcome of an individual game of "Jeopardy!" will usually owe most to long-term acquired knowledge and luck.  The best player won't always win (except, perhaps, if that player is IBM's Watson).  Only the most dominant players are able to string together more than a few consecutive wins.  That balance of luck and skill is what keeps "Jeopardy!" entertaining.

*Note: The regression incorporates information from every episode of "Jeopardy!" in the archive.  That makes the regression equation slightly misleading, because the dollar-values of the question doubled in November 2001.  The true y-intercept for games prior to that date is lower and for games afterward is higher.  Since the proportions between the two rounds remained the same, I wanted to use the whole set for the overall regression.