Sprinkling self-doubt on ChatGPT

We've been building out our agent [0], and we've found this to be the case.

We actually dialed it back a bunch, because it feels _terrible_. Yes, you get more correct answers, but it's more akin to giving the agent anxiety. Especially with agents that have access to tools, they'll burn enormous amounts of time on tool calls, trying to get enough information to overcome a motivation that's essentially burned into its identity.

(We saw one conversation where it just browsed social media instead of looking at the code for like 5 minutes, which ... you know, I get it.)

It's been much more effective to make uncertainty or further exploration be part of the agents success criteria.

- BAD: "Critique your own thoughts" -> leads to the agent trying really hard to get it right, but still not willing to actually be wrong

- GOOD: "Expose where your thoughts are unsupported or could benefit from further information" -> leads to the agent producing high-quality results, with loose ends that the user can choose to incorporate, ignore, or correct.

That prompt, combined with dialing up the thinking (either with API or prompt tuning) works much better, because it's sidestepping the training and tuning that's implicitly encouraged it to sound correct at all times.

[0] https://tern.sh, code migration AI

I had things like

* Do not use emotional reinforcement, not even robotic one (e.g., "Excellent," "Perfect," "Unfortunately", "Acknowledged").

* Do not use metaphors or hyperbole (e.g., "smoking gun," "major turning point").

in it. When they upgraded to GPT 5, Advanced Voice embedded into every answer things like:

---

Me: "I want to make a pizza tonight. What's the best way to start?"

It: "Sure, I'll stick to your guidelines and leave out the emotional reinforcements. If you want to make a pizza tonight, I'd first briefly explain how to go about it and why."

Me: "What should I drink with it?"

It: "To put it simply: Just drink whatever you like with it. Water, juice, or maybe a simple soft drink are classic options. It's really up to you, and I'll stick to your guidelines of saying it simply and without any frills. If you need anything else, just let me know.

Understood, I've noted that. I'll just stick to that and give you the information directly and without frills in the future. If you need anything else, just ask."

---

Absolutely useless. The next day my subscription expired, and when I talked with it again, it behaved like it used to prior to GPT 5: good.

I'm now big into Claude.

Also, before you ask: The chat about the pizza was just to test if it would behave that same way in every conversation.

This sounds like how I think.

But for me, it often results in situations where I think much harder and longer than others but fail to act.

I learned to sometimes act instead of thinking more, because by acting I gain information I could not have learned by thinking.

Perhaps this human insight can be applied to working with LLMs. Perhaps not :)

I don't know how anyone can trust these things at all.

What I want: a balanced, nuanced, deep engagement with a topic I don't already have mastered that may need to challenge my intuition or require of me some effort to research and correct my conception.

When I ask it to explain something I already understand quite well, which there is no broad public consensus about or the public consensus is known to be based in misconception, I will tend to get the public consensus view that there's no clear answer or provided an answer based in the misconception.

When I make it at all clear about what my conception is, I'll get a confirmation or reinforcement of my conception.

If I play a bit as a character who believes the opposite of my conception, unless my conception has a very, very clear basis in public consensus, I will get a confirmation or reinforcement of the opposite of my conception.

Why should I trust them in fields I'm not an expert in, given this? They want to please and serve rather than challenge you or inform you. Even when you ask them to be blunt and factual they do the theater of those things and not the substance. Their basis in Human linguistic outcome dooms them to pretend to be human which means people-pleasing or social-consensus-finding goals over truth-finding goals.

This is unrelated but occurred to me as I was reading it:

It would be really amusing or entertaining or inspiring or something to see what the best possible outcome from an LLM-style chat session would be. E.g. get a bunch of brilliant humans together, ask them ChatGPT-style questions, but use their collective resources to craft perfect responses to everything (I mean, slowly, over days or months, of course). LLMs are useful sometimes, sure, but sometimes it feels like people think their LLM is being really useful because they have low standards for how good the answer could be.

like... we're trying to make LLMs really good at doing human stuff. But humans are also really bad at what LLMs do: always having experts available as resources that give really good, directed, thoughtful answers to questions on any subject and come up with sophisticated plans to execute on goals. Businesses kinda do this but it's all twisted up; you never get to know if you're doing anything 'right' or in the best known way; mostly everyone is just making everything up as they go in the context of what they happen to know. It would be nice for once to see what the best possible human effort on a task or question would look like.

e.g. you ask it a math question and it teaches you the answer perfectly and describes it in the most lucid possible way. Maybe it's a subject no one really teaches right and the answer involves reformulating it in a new presentation that is easier to understand, and it expresses itself with just the right amount of confidence, nothing bizarre. Then you ask it for advice and it gives really, really good human-to-human interpersonal advice, takes into account everything you say and really sees you as a person and understand what you're going through but also has an eye towards pushing you to be better instead of just validating what you think you need. Then you ask it to make a website and what you get is a brilliant piece of original software plus it factors out some of the functionality into well-documented, well-tested open source software with a good plan for improvement. Then you ask it to fix local politics in your city and it realizes it can't be fixed without running for office so it puts together a team and a platform and finds a good candidate and starts a campaign...

In my use it feels as though this should not be done in advance or in the same prompt, even with reasoning models. It'd be better to make a "double check" MCP that calls your prompt, asks whether anything should be amended in that reply or use as is, amends if needed, then gives answer.

What you do not want to do is reposition your context into under-informed persona space, so leave the second-guessing out of the initial context. Instead use it as its own judge. (Doesn't have to be, but could also be an alt model.)

> you are a highly critical thinker and this is tempered by your self-doubt: you absolutely hate being wrong but you live in constant fear of it

Is there any work happening to model these kinds of emotional responses at a “lower level” than prompts?

I see work around like councils of experts and find myself wondering if some of those “experts” should actually be attempting to model things like survival pressure or the need to socially belong that many would normally consider to be non-rational behaviors.

I turned this on and for most "general" use cases I found it useful, I also observed a downward biased in a family of "quantatitative estimation" tasks, so just remember when you have this kind of stuff turned on (always beware of mutating global state!)

Ah, ChatGPT’s hidden INTP mode. We’ll finally get the right theory for ASI, but it will provide no clues on how to actually implement it in a timely manner.

This article is so sparce with any details it's basically useless.

Does telling the AI to "just be correct" essentially work? I have no idea after this article because there no details at all related to what changed the type of prompts etc

I've been experimenting with explicit directions/instructions in Claude AI. I have no real evidence that the responses are any better than without these instructions but anecdotally, I haven't been having the "No that's wrong. Sorry you're quite right, here's the correct info. No wrong again." and so on conversation half as much as without them.

FWIW here's what I use to make the thing more bearable: "After forming a response, check it again and if any contradictions or errors found, fix the issues and then check that answer again and so on. Also I'd prefer you to search the internet rather than tell me to and how to. When you find contradictory information in search results, do additional searches to resolve the contradiction before responding. But also, try to use logic and reasoning to clearly state what you think it should be. If you make any factual claims, double-check them against the search results before sending. When checking your response, specifically look for logical contradictions between different parts of your answer. Don't keep saying phrases in praise of my intelligence. Things like "you nailed it!" and "that's actually very perceptive of you". What the kids call 'glazing'; yeah don't do that"

Using the arbitrary to confuse the wetware as to where arbitrary settings reside (in arbitrariness). Note to self- save money for the analog NI bots around the corner (that compose specifics, and then resort to symbols/metaphors for output).