Alignment is not free: How model upgrades can silence your confidence signals
38 comments
·May 6, 2025Centigonal
mlin4589
Good question! We do know from OpenAI's system card from GPT-4 that the post-trained RLHF model is significantly less calibrated compared to the pre-trained model, so it's a matter of speculation that something similar is occurring. However, it's more of a hunch more than anything. I would be curious if it's possible to reproduce this behavior, or the impact of distillation on calibration.
Disclaimer: I wrote this blog post.
Workaccount2
Wouldn't it be something if AI parlance crept into common parlance...
itchyjunk
Could you please elaborate what less or more calibrated means here? Thanks!
Scene_Cast2
For binary labels: you take a slice of labeled data. The mean of the ML model prediction on this data is different from the mean of the label. In practice, often a synonym for "loss is worse / could be better".
Not sure if that's what the GP meant, I only worked with binary labels stuff.
behnamoh
there's evidence that alignment also significantly reduces model creativity: https://arxiv.org/abs/2406.05587
it’s it similar to humans. when restricted in terms of what they can or cannot say, they become more conservative and cannot really express all sorts of ideas.
Alex_001
That paper is a great pointer — the creativity vs. alignment trade-off feels a lot like the "risk-aversion" effect in humans under censorship or heavy supervision. It makes me wonder: as we push models to be more aligned, are we inherently narrowing their output distribution to safer, more average responses?
And if so, where’s the balance? Could we someday see dual-mode models — one for safety-critical tasks, and another more "raw" mode for creative or exploratory use, gated by context or user trust levels?
gamman
Maybe this maps to some human structures that manage control-creativity tardeoff through hierarchy?
I feel that companies with top-down management would have more agency and perhaps creativity towards (but not at) the top, and the implementation would be delegated to bottom layers with increasing levels of specification and restriction.
If this translates, we might have multiple layers with varied specialization and control, and hopefully some feedback mechanisms about feasibility.
Since some hierarchies are familiar to us from real-life, we might prefer these to start with.
It can be hard to find humans that are very creative but also able to integrate consistently and reliably (in a domain). Maybe a model doing both well would also be hard to build compared to stacking few different ones on top of each other with delegation.
I know it's already being done by dividing tasks between multiple steps and models / contexts in order to improve efficiency, but having explicit strong differences of creativity between layers sounds new to me.
pjc50
In humans this corresponds to "psychological safety": https://en.wikipedia.org/wiki/Psychological_safety
> is the belief that one will not be punished or humiliated for speaking up with ideas, questions, concerns, or mistakes
Maybe you can do that, but not on a model you're exposing to customers or the public internet.
malfist
How are you defining "creativity" in context with a statistical model?
exe34
> it’s it similar to humans. when restricted in terms of what they can or cannot say, they become more conservative and cannot really express all sorts of ideas.
This reminds me of the time when I was a child, and my parents decreed that all communications would henceforth happen in English. I became selectively mute. I responded yes/no, and had nothing further to add and ventured no further information. The decree lasted about a week.
qwertytyyuu
People use llm as part of their high precision systems? That’s worrying
sega_sai
Can we have models also return a probability, reflecting how accurate the statements it made is ?
jsnider3
You can ask a model to give you probability estimates of its confidence, but none of the frontier models were trained to be good at giving probability estimates to my knowledge.
cyanydeez
Sure, but then you need probability stats on the probability stats.
sega_sai
I am not sure what you mean. The idea is that the network should return the text, and a confidence expressed as probability. When trained, the log-score should be optimized. (i'm not sure it would actually work given how the training is structured, but something like this would be useful)
redman25
It's not that simple how would the model know when it knows? Removing hallucination has to be a post-training thing because you need to test the model against what it actually knows first in order to provide training examples of what it knows and doesn't know and how to respond in those circumstances.
erwin-co
Why not make a completely raw uncensored LLM? Seems it would be more "intelligent".
qwertytyyuu
Before rlhf, it’s much harder to use, remember the difference between gtp3 and chat gpt. The fine tuning for chat made it easier to use
null
khafra
"LLM whisperer" folks will confidently claim that base models are substantially smarter than fine-tuned chat models; with qualitative differences in capabilities. But you have to be an LLM whisperer to get useful work out of a base model, since they're not SFT'ed, RLHF'ed, or RLAIF'ed into actually wanting to help you.
andai
How can I learn more about this?
Is it like in the early GPT-3 days, when you had to give it a bunch of examples and hope it catches the pattern?
im3w1l
Back in those days I would either create a little scene with a knowledgeable person and someone with a question. Or I would start writing a monologue and generate a continuation for it.
msp26
Brand safety. Journalists would write articles about the models being 'dangerous'.
teruakohatu
In theory that sounds great, but most LLM providers are trying to produce useful models that ultimately will be widely used and make them money.
A model that is more correct but swears and insults the user won't sell. Likewise a model that gives criminal advice is likely to open the company up to lawsuits in certain countries.
A raw LLM might perform better on a benchmark but it will not sell well.
andai
Disgusted by ChatGPT's flattery and willingness to go along with my half-baked nonsense, I created an anti-ChatGPT, which is unfriendly and pushes back on nonsense as hard as possible.
All my friends hate it, except one guy. I used it for a few days, but it was exhausting.
I figured out the actual use cases I was using it for, and created specialized personas that work better for each one. (Project planning, debugging mental models, etc.)
I now mostly use a "softer" persona that's prompted to point out cognitive distortions. At some point I realized, I've built a therapist. Hahaha.
alganet
What kinds of contents do you want them to produce that they currently do not?
simion314
>What kinds of contents do you want them to produce that they currently do not?
OpenAI models refuse to translate or do any transformation for some traditional, popular stories because of violence, the story was about a bad wolf eating some young goats that did not listen the advice from their mother.
So now try to give me a prompt that works with any text and that convinces the AI that is ok in fiction to have violence or bad guys/animals that get punished.
Now I am also considering if it censors the bible where some pretend good God kills young chilren with ugly illnesses to punish the adults, or for this book they made excaptions.
Mountain_Skies
>alignment
Amazing how this Orwellian spin on propaganda has been so quickly embraced.
qwertytyyuu
It supposed to mean getting the ai to share our values so it doesn’t do things we don’t like in pursuit of what we tell it to do. Not necessarily political alignment
rusk
Upgrade scripts it is so. plus ca change
Very interesting! The one thing I don't understand is how the author made the jump from "we lost the confidence signal in the move to 4.1-mini" and "this is because of the alignment/steerability improvements."
Previous OpenAI models were instruct-tuned or otherwise aligned, and the author even mentions that model distillation might be destroying the entropy signal. How did they pinpoint alignment as the cause?