An election forecast that’s 50-50 is not “giving up”

I'm not sure how to verify your comment since 538 was cut by ABC a month or 2 ago. But Nate Silver's pollster rating methodology is pretty much the same as 538's was during his tenure there and can be found here: https://www.natesilver.net/p/pollster-ratings-silver-bulleti...

It actually explicitly looks for statistical evidence of "herding" (e.g. not publishing poll results that might go against the grain) and penalized those pollsters.

In both rating systems, polls that had a long history of going against the grain and being correct, like Ann Seltzer's Iowa poll, were weight very heavily. Seltzer went heavily against the grain 3 elections in a row and was almost completely correct the first 2 times. This year she was off by a massive margin (ultimately costing her her career). Polls that go heavily against the grain but DON'T have a polling history simply aren't weighted heavily in general.

It isn't just human bias that makes this problematic. Their definition of success is that a prediction of 70% should be right about 70/100 times.

Give a model a big reward for "success" under this criteria and it will have a tendency to converge on the long term average. As long as the long term average stays the same, no one will be able to say the estimate was wrong.

If you've ever seen the YT videos from aspiring data scientists learning sequence prediction, you've inevitably seen a variation from people trying to predict stock prices. "Look how low the loss is!" - Sure, but it gets the direction wrong, and just consistently predicts a close value to the previous day's close. But the model is technically correct, just useless.

A 50-50 model for an election might not be giving up and might not be wrong, but it is often useless.

> As the article notes, public polls started appearing in late September showing Harris ahead, which they never saw internally. Are internal polls just that much better than public ones? Is the media just incentivized to report on the most interesting outcome (a dead heat)?

Intentional bias for motivational purposes is a thing - it doesn't make any logical sense, but people want to vote for the winner.

Don't rule out Harris' internal polling just being bad and the public polling was superior. We're looking at a team whos message was "our opponent is a racist, homophobic, sexist, rapist, felonic avatar of Hitler" and then saw him gain margin which is quite remarkable. What did voters have to see in Harris' team for that to happen? They're clearly doing something wrong. If the team managed that I wouldn't assume their pollsters were a lone beacon of competence.

It seems quite plausible they hired bad pollsters and their internal data was off.

That is interesting. Presenting public poll results is not really about poll results, but about shaping opinion. The struggle with the media in general is between reporting on the reality that they see vs the reality they want to create. It's an infantile make-belief: "If I put my gloves on, it will start snowing". But, take a large outfit like WP or NYT and it could sort of work that way, too!

But, this also doesn't disagree with the original article that there was a large margin of uncertainty. In that case, a newspaper had a choice of mentioning the uncertainty and emphasizing it, or presenting the 3% in favor or their candidate. I think they went with the latter.

Also, when Bezos blocked WP from endorsing Harris, I always wondered if he had his own better polling results, and he was pretty sure Trump would win, or he just gambled. It's like there are these secret reliable set of poll results to which billionaires get access too, while the public gets the watered down version. Going by the article at hand, perhaps him or his team could better read the results and didn't drink the kool-aid.

This mixes up accuracy with precision and 538 had written at length about this.

There's a big difference between pollster bias correction vs rating.

There are many pollsters that are pretty consistently +2D/R but are reliably off from the final result in that direction. These polls are actually extremely valuable once you make the correction in your model. Meanwhile, polls that can be off from the final result by a large amount but average out to about correct should not be trusted. This is a yellow flag for herding

A pollster can have an A+ rating while having a very consistent bias in one direction or another. The rating is meant to capture consistency/honesty of methodology more than result

Except the methodology biases the response: phone-only vs. in-person for example.

This doesn't account for poll bias that biases both parties, of course. Not all polling is designed to wage parties against each other: some is designed to pit parties against populace.

50-50 forecasts can be massive depending on your priors. If someone (who I trusted) told me there was a 50-50 chance I would get shot tomorrow if I went outside, you couldn’t pay me enough to go leave the house.

Suppose it rains on 50% of days. One forecaster gives a 50% chance of rain every day. Another gives a 90% chance of rain/not rain and is "wrong" 10% of of time. Both stations are giving you accurate information from a probability perspective (when station A says a 50% chance, you know there's a 50% chance of rain tomorrow, when station B says 90%, you know there's a 90% chance of rain tomorrow) but the 50% chance station is less useful as the number is lower. They have effectively given up in the same way pollsters have given up.

Weddings are unusually locked in dates. Canceling a hotel a few days out is generally free and 50% odds of rain can easily be high enough to postpone a weekend trip. 10% odds however may be worth the gamble even if both possibilities exist.

With elections you need to take both a short term and long term perspective. True 50/50 odds mean doubling down at the last minute is likely more valuable vs saving that money for the next election. I think the pollsters got things wildly wrong with that estimate because their underlying assumptions where off, but that’s a different issue.

The utility of consulting weather forecasts prior to events like weddings lies in the non-trivial possibility that it will give you a clear indication of what will happen.

The problem is that there are two 50-50 odds definitions.

First one, is where you have no information about the event what so ever.

Second is where you have information, and it turns out that the outcome is still 50-50.

While the use of that information in planning (pricing things, betting, investing) is identical, they fundamentally represent 2 different sets of information.

[dead]

You’re welcome to bet against my prediction market about whether the quarter I flip will land on mars.

Or on the two markets for “New pope by tomorrow” and “new pope by 2030”, I’m sure those should both be 50% yes.

My experience is that the vast majority of prediction markets do not tend towards 50/50 because reality isn’t 100% uncertain.

It's the same as bookies, right? They are often seen as predicting the odds of an event happening, but that's actually the odds they estimate so that the bets are roughly divided 50/50, to limit their exposure, which is pretty different and depends on their market. English bookies will always overrate the chance of England winning at football, because English people will disproportionately place a bet for their team to win.

> When uncertainty increases (imagine squishing a mound of clay), you end up pushing lots of probability mass (clay) to the other side of the bet and the expectation of the payoff (used to make a price) tends towards 1/2.

This doesn't make any sense. If you think the chances of someone of getting elected is very high (eg. Putin getting re-elected), nobody will be buying shares in him losing. True, the shares of "putin loses" is dirt cheap, but that doesn't mean much if he has a high chance of getting elected.

Lets says I have a coin. I have no data on it. I do not know if it will come up heads or tails or if it is a fair coin.

I throw the coin N number of times and determine the coin is fair. I still do not know which side the coin will come up.

But this isn't the same as having no data on the coin. I've determined the coin is fair by gathering data, which otherwise I would not know. Often this added piece of information is crucial to making good decisions and not at all equivalent to the first case.

For instance, I might want to bet you that the next 10 throws in a row will all be heads. If you know the coin is fair, this is a good bet - but without data on the coin, it could be a trap.

Having no data at all is better than having the wrong information.

If you were previously pretty sure one side would win you gain information from a study that says it's 50-50.

If you already though it was even then the new study is pointless.

I think the lack of signal from the electorate does represent an entropic state and is likely because available information channels are saturated with noise.

Interesting. If you think about it, that's how the stock market works. It's not how much you think a stock is worth; it's really how much you think everyone else thinks the stock is worth.

Was this a new development in polling?

Predict 100 fair coin flips..

My result didn't match yours. The model isn't useful?

That's not how polls work though. A poll shows how a sample of people say they'll vote. An ideal poll would have the same proportions as the final popular vote count.

I came away from 2016 wondering if any of the electoral outcome stuff is worth doing given how much it gets misunderstood.

“They said Hillary had a 90% but Trump won! They’re idiots!”

No! They said Trump had a 10% chance and he won. It was a predicted outcome!

Even if polling had been exactly right, you wouldn't have been that confident in the outcome.

In my unsophisticated toy model, plugging in the exact actual result as the polling average (but not telling it how the actual vote went) spits out 66% R-34% D. Clearly one side favored, but hardly a guarantee. Because the result was close, and even highly accurate data in a close result yields an uncertain forecast.

Remember that asteroid a month ago? We knew what its position would be seven years in the future with a precision of a few hours. But because the position was very close to an impact, even that high precision was not enough to rule out an impact.

> Due to their inability to poll a representative slice of the population they try to correct by adjusting their results to compensate for the difference between who they've polled and the likely makeup of the electorate.

This is where polling becomes race science.

> This makes sense in theory

It does not make sense in theory. It is a necessity for the profession, but all justifications for it are specious. Polls only get it right when everybody is getting it right. What they offer is false precision and justification for the current narratives.

It's similar to AI in that way. It's also similar to the mythical prediction markets that polls have been compared to lately, the "mythical" here meaning with no insiders involved. On issues where there are no real insiders, like close elections, the prediction markets are simply a lagging indicator of what pundits said in the paper this morning. That goofy Iowa poll swung them so hard that I thought Seltzer should have been investigated for whatever the prediction market equivalent to securities fraud is.

It might be more accurate in light of the OP to say that polls get it right when everybody is getting it right, and when everybody isn't sure what's going to happen, polling accuracy is around 50/50.

The best book to read about polling is The Full Facts Book of Cold Reading by Ian Rowland. It also tells you how to write defenses like this, which are part of the con.

-----

edit:

Section headings from "The Win-Win Game" from TFFBoCR, which teaches 10+1 ways how to make failures seem like successes:

> 1. Persist, wonder and let it linger.

> (Phase A: The psychic persists with the official statement and tries to encourage at least partial agreement. B: she acts puzzled, and invites the client to share the blame for the 'discrepancy.' C: she leaves the discrepancy unresolved, in case the client finds a match later on.)

> 2. I am right, but you have forgotten.

> 3. I am right but you do not know.

> 4. I am right but nobody knows.

> 5. I am right, but it's embarrassing.

> 6. I am wrong now, but I will be right soon.

> 7. I am wrong, but it doesn't matter.

> 8. I am wrong in fact, but right emotionally.

> 9. I am wrong in fact, but right within [the] system.

> 10. Wrong small print, right headline.

> [+1]. Accept, apologise, and move on.

> (In this way the psychic cuts her losses and moves on. She leaves the problem behind, where it will be quickly forgotten, and at the same time she comes across as extremely honest.)

Very much worth reading for entrepreneurs looking for investment or any other confidence men. Rowland even tried to brand it a few years ago in Cold Reading For Business as the "CRFB" system.

okay, now change "election prediction models" to "Climate models" and see if you feel like downvoting me merely for pointing out the (slight?) hypocrisy in "excusing" every other model we humans use for being "inaccurate" or "not having the full details" or the "whole slice of"...

when none of the models tend to agree... and the IPCC literature published however often they do it is hung upon the framework of models.

But the surveys were pretty accurate this election. So I’m not sure why you say they are useless.