There are two big winners in the U.S. this morning, and the happiest one is possibly not the guy with graying hair who gets to keep running the country. Instead, Nate Silver, the nation’s 34-year old Delphic oracle can look on his mighty empire of polling data, and smile: he nailed it. Again.
The author of The New York Times–hosted blog FiveThirtyEight had correctly predicted the electoral lay of the land when all around him political pundits pronounced the race too close to call or maddeningly inconclusive.
Adored or despised during the campaign, depending on how you felt about his robust certainty that President Obama would serve a second term; fingered as an Obama-loving ideologue who fudged the data by stacking the algorithmic scales; and then derided by MSNBC’s Joe Scarborough as a joke for failing to call the election a toss-up when it was—clearly! —a toss-up, Silver ended up kicking sand in the eyes of all the political jocks rooting against a political future where bloodless number crunching would replace bloody partisan opining.
And while his very public $2,000 bet on Twitter with Scarborough that math would beat intuition struck the Times as indecorous (and viewed by some in the media as reckless), it may well be the easiest money Silver ever made. You always bet on the math, especially when you know the other guy doesn’t really know math the way you know math.
As America voted yesterday, I checked in with Steve Ross, known to generations of Columbia journalism students as the school’s impassioned but often lonely data maven. “The only things that could derail [Silver’s] prediction are a systematic flaw in polling methodology across all states and almost all pollsters—or dirty tricks on the part of election officials in multiple states, “ Ross said via email.
Even though polling in 2012 was probably less precise than four years ago due to the proliferation of cellphones and number carrying, while chance—as the confluence of factors in Superstorm Sandy demonstrated—is always waiting in the statistical wings to upscuttle the best-quantified predictions, the fact is that the polls that tried the hardest to assess a broad sample of people, as Silver himself noted during a valedictory interview on the Times, all kept converging.
“When you consider multiple polls and you ‘trend’ the same pollster’s results over time, you build confidence that the reported results are ‘real,’ said Ross. “Silver has calculated that confidence as betting odds. The calculations are his own—and not strictly speaking irrefutable—but they are based on good math and real polling data. Scarborough’s position [was] based on wishful thinking and all pundits’ desires to keep the pot boiling until the final results are in. I'll take math over wishful thinking any day, even if it isn’t perfect math.”
But in some ways, it’s slightly unfair to crown Silver the nation’s psephologist-in-chief; after all, he panned his gold from the work of numerous pollsters who also supplied the raw data for other prognosticators to make accurate predictions. University of Virginia veteran Larry Sabato, for instance, also forecast a robust Obama victory on the quaintly analogue-sounding “Sabato’s Crystal Ball,” noting that “for a challenger to defeat an incumbent, the fates must be with the challenger again and again.” The betting agencies were not wrong in their odds.
In the end, the election just wasn’t that surprising, which of course, is possibly why so many chose to keep seeing the possibility of a surprise at the end of the tunnel. And that is why one of the most telling points to come out of Nate versus the naysayers was nicely encapsulated on a tweet last night from @zachrodgers: “It’s great that NYT, via@Fivethirtyeight, and not Google or some emachine learning platform is leading the quant takeover.”
Clearly, hiring Silver has been the smartest move in the Times’s digital strategy; his blog is an election-time traffic sponge. But clearly, too, conventional horserace journalism, as the boys and girls on the political bus now know it, is increasingly meaningless. Big data has just kicked them in their anecdotal ledes—and all their talk of momentum or inconclusiveness can now be simply and quickly settled with an algorithm.
Sure, the public isn’t going to abandon narrative just yet, and any reporter who can find the time and space to write of presidential campaigns as, say Gay Talese wrote of boxing, will be rewarded by a grateful audience and real influence. But just as with Daniel Craig’s James Bond, it’s the new and nerdier Q—who’s really going to be running the show from here on in.
But just as Silver is a harbinger of this world, before long he too may eventually be its victim. Sooner or later Big Data is going to transform or destroy traditional polling along with the journalists who rely on it. If companies can pay to know why someone will buy one soap over another, why are we going to stick with phoning people to ask them who they are likely to vote for? There are so many other behavioral attributes waiting to be identified, mined, and analyzed. And the cost of doing this is falling.
Today, politics and journalism belong to people like Nate Silver; but tomorrow belongs to the candidate who pours his money, not into polling, but into data analytics. The “quant takeover” has only just begun.
