An election forecast that’s 50-50 is not “giving up”
41 comments
·March 8, 2025bitshiftfaced
culi
I'm not sure how to verify your comment since 538 was cut by ABC a month or 2 ago. But Nate Silver's pollster rating methodology is pretty much the same as 538's was during his tenure there and can be found here: https://www.natesilver.net/p/pollster-ratings-silver-bulleti...
It actually explicitly looks for statistical evidence of "herding" (e.g. not publishing poll results that might go against the grain) and penalized those pollsters.
In both rating systems, polls that had a long history of going against the grain and being correct, like Ann Seltzer's Iowa poll, were weight very heavily. Seltzer went heavily against the grain 3 elections in a row and was almost completely correct the first 2 times. This year she was off by a massive margin (ultimately costing her her career). Polls that go heavily against the grain but DON'T have a polling history simply aren't weighted heavily in general.
bitshiftfaced
> I'm not sure how to verify your comment
Here's how 538 explains how they factor in bias into their grading:
> Think about this another way. If most polls in a race overestimate the Democratic candidate by 10 points in a given election, but Pollster C's surveys overestimate Republicans by 5, there may be something off about the way Pollster C does its polls even if its accuracy is higher. We wouldn't necessarily expect it to keep outperforming other pollsters in subsequent elections since the direction of polling bias bounces around unpredictably from election to election.
- https://abcnews.go.com/538/best-pollsters-america/story?id=1...
null
alphazard
Highly recommend this video by NNT about why prediction markets tend towards 50-50.
https://www.youtube.com/watch?v=YRvPF__du9w
Prediction markets are usually implemented as binary options. Like vanilla options, their price depends not just on the most likely outcome, but the whole distribution. When uncertainty increases (imagine squishing a mound of clay), you end up pushing lots of probability mass (clay) to the other side of the bet and the expectation of the payoff (used to make a price) tends towards 1/2.
moduspol
Agreed with the points in OP. Though we did have the story that came out shortly after the election that apparently internal polling for the Harris campaign never showed her ahead [1].
Obviously it says in the article that they did eventually fight it to a dead heat, which is in-line with a 50-50 forecast, but I do wonder what, if anything, failed such that this key detail was never reported on publicly until after the election.
As the article notes, public polls started appearing in late September showing Harris ahead, which they never saw internally. Are internal polls just that much better than public ones? Is the media just incentivized to report on the most interesting outcome (a dead heat)?
[1] https://www.usatoday.com/story/news/politics/elections/2024/...
t-3
> As the article notes, public polls started appearing in late September showing Harris ahead, which they never saw internally. Are internal polls just that much better than public ones? Is the media just incentivized to report on the most interesting outcome (a dead heat)?
Intentional bias for motivational purposes is a thing - it doesn't make any logical sense, but people want to vote for the winner.
HKH2
Well, trying to have 'safe' opinions is not completely illogical. It's a survival strategy.
roenxi
Don't rule out Harris' internal polling just being bad and the public polling was superior. We're looking at a team whos message was "our opponent is a racist, homophobic, sexist, rapist, felonic avatar of Hitler" and then saw him gain margin which is quite remarkable. What did voters have to see in Harris' team for that to happen? They're clearly doing something wrong. If the team managed that I wouldn't assume their pollsters were a lone beacon of competence.
It seems quite plausible they hired bad pollsters and their internal data was off.
delichon
There's a pretty clear test of whether a pollster is about reporting or influencing: the partisanship of their errors. Neutral pollsters have differences with the results randomly distributed across parties. Propagandists have errors that favor their patrons. This essay leans on the magnitude of the errors, but that's less probative than their distribution.
Does any poll aggregator order by randomness of polling error?
culi
This mixes up accuracy with precision and 538 had written at length about this.
There's a big difference between pollster bias correction vs rating.
There are many pollsters that are pretty consistently +2D/R but are reliably off from the final result in that direction. These polls are actually extremely valuable once you make the correction in your model. Meanwhile, polls that can be off from the final result by a large amount but average out to about correct should not be trusted. This is a yellow flag for herding
A pollster can have an A+ rating while having a very consistent bias in one direction or another. The rating is meant to capture consistency/honesty of methodology more than result
kazinator
[delayed]
jdoliner
I don't know about the framing of "giving up." But I think anyone who's been following election models since the original 538 in 2008 has probably gotten the feeling that they have less alpha in them than they did back then. I think there's some obvious reasons for this that the forecasters would probably agree with.
The biggest one seems to be a case of Goodhart's Law, leading to herding. Pollsters care a lot now about what their rating is in forecasting models, so they're reluctant to publish outlier results, those outlier results are very valuable for the models but are likely to get a pollster punished in the ratings next cycle.
Lots of changes to polling methods have been made due to polls underestimating Trump. Polls have become like mini models unto themselves. Due to their inability to poll a representative slice of the population they try to correct by adjusting their results to compensate for the difference between who they've polled and the likely makeup of the electorate. This makes sense in theory, but of course introduces a whole bunch of new variables that need to be tuned correctly.
On top of all this is the fact that the process is very high stakes and emotional with pollsters and modellers alike bringing their own political biases and only being able to resist pressure from political factions so much.
The analogy I kept coming back to watching election models during this last cycle was that it looked like an ML model that didn't have the data it needed to make good predictions and so was making the safest prediction it could make given what it did have. Basically getting stuck in this local minima at 50-50 that was least likely to be off by a lot.
rachofsunshine
Even if polling had been exactly right, you wouldn't have been that confident in the outcome.
In my unsophisticated toy model, plugging in the exact actual result as the polling average (but not telling it how the actual vote went) spits out 66% R-34% D. Clearly one side favored, but hardly a guarantee. Because the result was close, and even highly accurate data in a close result yields an uncertain forecast.
Remember that asteroid a month ago? We knew what its position would be seven years in the future with a precision of a few hours. But because the position was very close to an impact, even that high precision was not enough to rule out an impact.
pessimizer
> Due to their inability to poll a representative slice of the population they try to correct by adjusting their results to compensate for the difference between who they've polled and the likely makeup of the electorate.
This is where polling becomes race science.
> This makes sense in theory
It does not make sense in theory. It is a necessity for the profession, but all justifications for it are specious. Polls only get it right when everybody is getting it right. What they offer is false precision and justification for the current narratives.
It's similar to AI in that way. It's also similar to the mythical prediction markets that polls have been compared to lately, the "mythical" here meaning with no insiders involved. On issues where there are no real insiders, like close elections, the prediction markets are simply a lagging indicator of what pundits said in the paper this morning. That goofy Iowa poll swung them so hard that I though Seltzer should have been investigated for whatever the prediction market equivalent to securities fraud is.
It might be more accurate in light of the OP to say that polls get it right when everybody is getting it right, and when everybody isn't sure what's going to happen, polling accuracy is around 50/50.
The best book to read about polling is The Full Facts Book of Cold Reading by Ian Rowland. It also tells you how to write defenses like this, which are part of the con.
genewitch
okay, now change "election prediction models" to "Climate models" and see if you feel like downvoting me merely for pointing out the (slight?) hypocrisy in "excusing" every other model we humans use for being "inaccurate" or "not having the full details" or the "whole slice of"...
when none of the models tend to agree... and the IPCC literature published however often they do it is hung upon the framework of models.
sojournerc
All models are wrong, but some are useful.
Climate modeling is way messier than the media portrays, yet even optimistic models show drastic change.
I'm not in the catastrophy camp, but it's worth preparing for climate change regardless of origin. It's good for humanity to be resilient to a hostile planet.
bryanlarsen
Nate Silver also had an article saying that the most likely scenario would be a blowout. There was a 25% chance that Trump would win all 7 battleground state, a 15% chance Harris would win all 7. No other permutation of the 7 states was anywhere close.
chicagobob
True, but the word "blowout" in this case is just a crazy side-effect of our weird electoral college system.
Everyone knows that in all the swing states (except Arizona), the final vote margin was just a few percent, and that was well within the MOE for all the "50-50" polling in each of those states.
No one seriously believes that any President has had a blowout election since maybe Obama in 2008 or Bush in 2004, but the media sure loves the word "blowout".
nightski
So basically, if you ignore how the entire system works then it wasn't a blowout lol. I'm guessing the media was taking into account that we indeed use an electoral college system so that is all that matters.
DangitBobby
I think "blowout" to some (most? vast majority?) without more context implies that the voting citizens strongly preferred a candidate. So people pushback against the clickbait word being used to drive engagement.
bombcar
If they're not completely independent variables, that should be the expected outcome (e.g., they all trend the same direction).
And if they're not that close, they're not battleground.
fullshark
They are where the campaign money and attention is going, what about battleground implies independence? Whoever wins them will win the election.
culi
Trump won the popular vote by 1.5%. That's the 8th closest election in all of US history.
Maybe he meant an EC blowout. But that's easier to predict. Most polling had almost all the swing states as extremely close. The outcome was likely to swing in the same direction across all swing states so an EC blowout is likely
wakawaka28
As I recall none of the big polls the mainstream media was pushing projected Trump to win. They even refused to call the election until the wee hours of the morning, when the results were pretty clear. Just like in 2016, the polls were intended to deceive people into thinking there was no hope for an alternative candidate.
culi
Major decisions desks (AP, Edison, and Fox News; plus a recent newcomer called DDHQ) have made about 3 incorrect calls in their entire history (they were house races, not senate or presidential calls). An incorrect call is a HUGE deal and they have an extremely high bar for certainty. Even for Obama, who's wins were basically a landslide compared to the past 3 elections, calls weren't made until around midnight. Historically it's even common for presidential candidates to concede even before decision desks have made an official call (another norm Trump has challenged).
Anyways Decision Desks are a very different topic than pollsters and have no overlap.
I don't know what "mainstream media" you were consuming but as a bettor, I definitely didn't get the sense that the race was locked in. All major polls had Biden trailing Trump. Harris got a big boost but that only pushed her to being neck and neck with Trump. We can check the Wayback Machine to verify your claim
* NYTimes frontpage the day before the election says "turnout is a focus in a closely divided race" https://web.archive.org/web/20241105011526/https://www.nytim...
* Vox frontpage says "5 reasons Harris could win and 4 reasons Trump could" and "Should it really be this hard to beat Trump?" https://web.archive.org/web/20241106000611/https://www.vox.c...
* CNN frontpage does not include any particular characterization of chances https://web.archive.org/web/20241105000156/https://edition.c...
A cursory look at other "mainstream" media sources will easily verify that the race was clearly represented as a very close tossup.
randomNumber7
Tell me you don't know the difference between bias and variance. The longest article/comment wins.
null
lyu07282
At the end of the day it was a 312 vs. 226 in the Electoral College. Seems a bit odd that this is supposed to be impossible to be predicted with any useful amount of certaintly. But perhaps that says more about the nature of the Electoral College than it says about pollsters.
https://en.wikipedia.org/wiki/Efforts_to_reform_the_United_S...
culi
Almost every major model had this outcome as very likely.
There was an inordinate amount of very close swing states in 2024. Whichever way the swing was gonna be was likely to apply across the board.
If you're going by popular vote, then this was actually the 8th closest election in all of US history.
stevenwoo
Not that I'm claiming this for this particular discussion, but a lot of people in favor of what is happening in the USA want to justify what's going on by claiming this was some sort of overwhelming political sweep in this country instead of being a close race at all levels (roughly handful of votes in House and Senate to achieve parity). Who would have picked a Arab American protest vote as major factor in swing states in 2024 when asked in 2020?
wakawaka28
They have to claim it was hard to predict, because they deliberately led people to believe that the establishment-favored candidate was going to win. They knew better the whole time but hoped to help their candidate by making the situation look hopeless for the true popular choice.
jmyeet
It is insufficient to make a prediction like 50-50 in a US election unless you also can explain turnout. Anyone can throw out numbers like "51-49" or "55-45" but how did you get to those numbers? How are different demographics voting? How is there turnout changing? I've seen people laud their own accuracy in 2020 while being off by about 30 million in predicted turnout.
The way different demographics vote in US elections doesn't change much from election to election. What does change is turnout and turnout is a function of many things: enthusiasm for the candidate, voter suppression, ease of access to voting and so on.
2020 was unprecedented because of the pandemic. We greatly expanded early voting and mail-in ballots, which greatly increased participation.
A perfect example of this is Arizona. In 2020, Native Americans were crucial to flipping the state to Biden. Arizona state lawmakers responded to this by essentially punishing them and making it way more difficult to vote. Voter ID requirements, birth certificates and even having a physical address are all impediments to people who were born on and/or reside on reservations. There were fewer voting places and voting options. A rural voting place might randomly close early too after being an hours long drive.
Some looked at this and said Native Americans in Arizona swung hard to Trump. No, they were simply largely prevented from voting such that the only Native Americans who could reliably vote were more affluent and thus more likely to be Trump voters.
My point is: what polling model captured this prior to the 2024 election? I guarantee you it's none.
What really happened in 2024 was:
1. Biden voters swung to the couch in the millions;
2. Trump basically didn't lose white women, despite the abortion issue; and
3. Trump activated a previously low-propensity voter demographic: angry, young, terminally online white males, basically the Andrew Tate and 4chan crowd.
Any model has a difficulty with low-propensity demographics. Did any model capture this? I think it only started to become apparent with early voting exit polls.
I don't think the 2024 polls were particularly accurate. I do think they threw their hands up and simply converged to 50-50. Small differences in turnout predictions for different demographics can massively impact the result.
An accurate model can output a 50-50 prediction. Sure, no problem there. But there is a human bias that does tend to make 50% more likely in these cases. It is the maximum percentage you can assign to the uncomfortable possibility without it being higher than the comfortable possibility.
538 systematically magnified this kind of bias when they decided to rate polls, not based on their absolute error, but based on how close their bias was relative to other polls' biases.(https://x.com/andrei__roman/status/1854328028480115144) This down-weighted pollsters like Atlas Intel who would've otherwise improved 538's forecast.