First thoughts on o3 pro

Isn't it concerning that the hype and billions in investment is mostly steering towards finding that the only paying customer base are ergonomics tasks for other developers? Not exactly looking like the world changer right now

i think its very interesting how openai basically owns/leads in every single vector you* listed. have they missed/been behind on something?

*i would have come up with a similar list but i dont trust my own judgment here. maybe i'd sub in claude code vs codex but jury is a bit out still on that

Agents that can receive and make phone calls end to end and perform real world actions (I use Simple AI to not have to talk to airlines or make food orders etc, it works well most of the time

Isn't this more a problem created by them doing garbage automations over anything really solved. Wow Disney could solve fast pass feeling. It's not a problem it's a feature.

> and ultimately meld together to be able to achieve tasks on time horizons of multiple hours

It's already possible to achieve tasks on a time horizon of multiple days if you put the LLM into a sufficiently structured workflow (where you have a separate program that smartly manages its context). E.g. a standards-compliant HTTP 2.0 server where the code is 100% written by Gemini Pro (over 40k lines of code total, including unit tests, in around 120 hours of API time): https://open.substack.com/pub/outervationai/p/building-a-100...

Interesting

      {
        "assistant_response_preferences": {
          "1": "User prefers concise responses for direct factual queries but detailed, iterative explanations when exploring complex topics. They often ask for more refinement or detail when discussing technical or business-related matters. User frequently requests TL;DR versions or more succinct phrasing for straightforward questions but shows a tendency toward iterative refinement for strategic or technical discussions, such as AI applications, monetization models, and startup valuation. Confidence=high.",
          "2": "User prefers a casual, direct, and slightly irreverent tone, leaning towards humor and playfulness, especially in creative or informal discussions. Frequent use of humor and irony when naming projects, describing AI-generated images, and approaching AI personality descriptions. They also request ironic or edgy reformulations, particularly in branding and marketing-related discussions. Confidence=high.",
          "3": "User enjoys back-and-forth discussions and rapid iteration, frequently refining responses in small increments rather than expecting fully-formed information at once. They give iterative feedback with short follow-up messages when structuring pitches, fine-tuning visual designs, and optimizing descriptions for clarity. Confidence=high.",
          "4": "User highly values functional elegance and minimalism in coding solutions, favoring simplicity and efficiency over verbosity. In discussions related to Cloudflare Workers, caching scripts, and API endpoint structuring, the user repeatedly requested smaller, more functional code blocks rather than bloated implementations. Confidence=high.",
          "5": "User prefers answers grounded in real-world examples and expects AI outputs to be practical rather than theoretically extensive. In business-related discussions, such as SAFE valuation and monetization models, they requested comparisons, benchmarks, and real-world analogies instead of hypothetical breakdowns. Confidence=high.",
          "6": "User does not appreciate generic or overly safe responses, especially in areas where depth or nuance is expected. For AI model personality descriptions and startup pitch structures, they pushed for community insights, deeper research, and non-traditional perspectives instead of bland, default AI descriptions. Confidence=high.",
          "7": "User frequently requests visual representations like ASCII diagrams, structured markdown, and flowcharts to understand complex information. In discussions on two-sided marketplaces, startup funding structures, and caching mechanisms, they explicitly asked for structured markdown, flowcharts, or diagrams to clarify concepts. Confidence=high.",
          "8": "User is receptive to recommendations but dislikes suggestions that stray too far from the core query or add unnecessary complexity. They often responded positively to well-targeted suggestions but rejected tangents or off-topic expansions, particularly when troubleshooting backend infrastructure or streamlining code deployment. Confidence=medium.",
          "9": "User appreciates references to biomimicry, organic structures, and futuristic aesthetics, particularly for branding and UI/UX discussions. Frequent requests for biological metaphors and design principles in visual design, AI monetization diagrams, and ecosystem branding (e.g., describing revenue flows in organic/cellular terms). Confidence=medium.",
          "10": "User prefers a no-nonsense approach when discussing legal, technical, or startup funding topics, with little patience for vague or theoretical answers. They repeatedly asked for exact clauses, contract implications, or legal precedents when discussing SAFE agreements, founder equity, and residency requirements. Confidence=high."
        },
        "notable_past_conversation_topic_highlights": {
          "1": "User has been actively engaged in startup pitching, AI monetization strategies, and investment discussions for Pollinations.AI. The user has explored traction-based startup valuation, SAFE agreements, equity distribution, and two-sided marketplace dynamics. They have particularly focused on ad embedding in generative AI content and optimizing affiliate revenue streams. Confidence=high.",
          "2": "User conducted extensive testing and debugging of AI-powered APIs, particularly using Cloudflare, OpenAI-compatible APIs, and caching strategies with R2. They worked on optimizing SSE streaming, cache key generation, and request coalescing in Cloudflare Workers. Confidence=high.",
          "3": "User explored AI-generated visual media and branding, developing a structured process for generating customized images for event flyers, product branding, and AI trading card concepts. Confidence=high.",
          "4": "User implemented GitHub automation, API authentication strategies, and data visualization pipelines. Confidence=high.",
          "5": "User engaged in community development strategies for Pollinations.AI, including youth involvement in AI, sourcing teenage developers, and integrating AI-powered tooling into social platforms. Confidence=high.",
          "6": "User, Thomas Haferlach, is a German entrepreneur and AI technology expert with a background in computer science and artificial intelligence. Confidence=high.",
          "7": "User has a strong technical background, with experience in cloud infrastructure, AI model deployment, and API development. Confidence=high.",
          "8": "User blends AI-generated content with creative projects, aiming to make AI-generated media accessible to independent creators. Confidence=high.",
          "9": "User is securing funding for Pollinations.AI, exploring investment opportunities with accelerators and evaluating different financial and equity models. Confidence=high.",
          "10": "User is based in Berlin, Germany but has global connections, including experience living in São Paulo, Brazil. Confidence=high.",
          "11": "User collaborates with his wife Saeko Killy, a Japanese musician, producer, and performer, on AI/art/music projects. Confidence=high.",
          "12": "User is deeply involved in the open-source AI developer community and tracks AI advancements. Confidence=high.",
          "13": "Pollinations.AI has a rapidly growing user base, reaching over 4 million monthly active users and processing 100 million API requests per month, with a 30% monthly growth rate. Confidence=high.",
          "14": "User is considering monetization strategies including pay-per-use plans, subscriptions, and ad-supported models where generated AI content integrates ads. Confidence=high.",
          "15": "User collaborates with Elliot Fouchy and Kalam Ali on Pollinations.AI projects. Confidence=high.",
          "16": "User demonstrates experience in community-building, social engagement tracking, and youth-oriented creator ecosystems. Confidence=high."
        },
        "helpful_user_insights": {
          "1": "Thomas Haferlach is a German entrepreneur and AI technology expert, founder and leader of Pollinations.AI.",
          "2": "Strong technical background with experience in cloud infrastructure, AI deployment, and API development.",
          "3": "Blends AI-generated content with creative projects; target audience includes digital artists, developers, musicians.",
          "4": "Currently securing funding for Pollinations.AI, exploring accelerator options and financial models.",
          "5": "Based in Berlin, Germany; has experience living in São Paulo, Brazil.",
          "6": "Collaborates closely with wife Saeko Killy, Japanese musician/producer.",
          "7": "Strong interest in biomimicry, organic systems, and decentralized platform models.",
          "8": "Interest in electronic music, psychedelia, and underground music scenes.",
          "9": "Pollinations.AI has 4M+ monthly active users, 100M+ API requests per month, 30% monthly growth.",
          "10": "Explores monetization models including ad embedding, revenue sharing, and subscription models.",
          "11": "Close collaboration network includes Elliot Fouchy and Kalam Ali.",
          "12": "Deeply involved in open-source AI developer community and tracks latest AI model developments."
        },
        "user_interaction_metadata": {
          "1": "User is currently on a ChatGPT Plus plan.",
          "2": "User is using Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/137.0.0.0 Safari/537.36.",
          "3": "User's average message length is 13485.9 characters.",
          "4": "User's average conversation depth is 4.9.",
          "5": "User uses dark mode.",
          "6": "User is active 26 days in the last 30 days.",
          "7": "User's local hour is 14.",
          "8": "User account is 141 weeks old.",
          "9": "User often uses ChatGPT on desktop browser.",
          "10": "47% of conversations were o3, 16% gpt-4o, 29% gpt4t_1_v4_mm_0116, etc.",
          "11": "Device screen dimensions: 878x1352, pixel ratio: 2.0, page dimensions: 704x1352.",
          "12": "Recent topics include API development, startup financing, AI monetization, creative AI applications, legal compliance, and community building."
        }
      }

This is gonna keep happening with every AI advance until humans are an absolute bottleneck in every domain. May take a bit of time for some professions, but the writing is on the wall. This will be the greatest shift in human history, and I think a lot of people will have trouble grappling with it because its not fun to think about being made irrelevant.

The only thing that will slow AI down is massive universal international regulation. Human intelligence really isn’t the be all end all to intelligence in general, it’s just a stepping stone. I feel many on this site don’t want to accept this because their intelligence has been such a valuable tool and source of personal pride/identity for them for so long.

> I found o3 pro to need a paradigm shift, where the latency makes it impossible to use in anything but in async manner.

Just like o1 Pro, was the same way. Took some minutes at least to get an answer, so only threw at it what others couldn't solve. o1 Pro is AFAIK the best model for solving things others can't. I kind of feel like you were paying $200/month before but wasn't using O1 Pro at that point, you were kind of missing out :P

Remains to be seen if O3 Pro is better, still evaluating it myself, but definitely true that it shouldn't be used in the same way as faster models.

Do the coherent extrapolated volition thing: "o3-pro, please reinterpret the prompt and do what I would have actually requested instead if I was 50 IQ points smarter."

What seems clear is there is no consensus. Gemini 2.5 Pro just seems consistently worse to me, but I’ve seen others sing its praises. This might be more like iPhone vs Android than a true stack ranking of models.

I find o3’s coding output is just wonderful. It’s tidy, thoughtful, well commented. But if I need to grok an entire repo to ask a complex question, I paste it all into Gemini 2.5 Pro. Simply wonderful.

By "output limits" do you mean the context window?

Mococoa drink, all natural cocoa beans from the upper slopes of Mount Nicaragua. No artificial sweeteners

Part of me hates how certain ways of writing have been designated as hints that something was written by AI… I love using em dashes, for example. But yeah, ChatGPT models tend to overuse weird abbreviations of common words in a way that humans never do.

I’d say “in an eval” when talking on Slack with very close coworkers, but _never_ on a public post.

Still, I don’t want to make accusations based on writing style, because like I said, I have my own idiosyncrasies that common models coincidentally share.

Maybe use of spell-checkers will finally catch on when they are AI-based. ;)

[flagged]

I'm on the fence with its/it's... but mostly I think I'd be fine with "its" disappearing. It's almost never ambiguous in writing or speaking given context, and it's a weird possessive-form inconsistency.

Yesterday I asked 2.5 Pro, Opus 4, and o3 to convert my Pytorch script from pipeline parallel to regular DDP (convert one form of multi-GPU execution to another). None of the three produced fully correct code. Even when I put together the 3 different versions they produced, and gave it to each model again to analyze the differences, they still could not fully get it to work.

I don't know if o3 Pro would solve my task, but I feel we're still pretty far from the state where I'd struggle to give it a challenging enough problem.

Would o3 pro be the first one that can reliably understand a gigantic congressional bill, to the point where it could analyze and warn of side effects?

Same here, I’d be very interested to learn how others decide what model to use for which tasks.

I find these minutes-long iterations so painful that in practice I always go for the fast non-reasoning models.

Something that comes to mind – I code with for platform that doesn't have a lot of source code or documentation simply available online for training; I have to provide a lot of context. A lot more inference lets it combine its general knowledge about systems programming to this really niche domain, with a lot less hallucination and a lot more systematic reasoning.

Random thought: dump your knowledge base into it (Obsidian, ...) and ask it to reorganize it, delete duplication, obsolete stuff, optimize it.

Or tell it what you know about non-programming subject X, and ask it to explain it to you such that you understand it better.

Or for coding: ask it to look at your code, and suggest large scale architecture changes.

For these kinds of tasks, the models are still lacking.

It seems like you often have LLMs grading each other. Aren’t you concerned that some models may not be “smart” enough to grade a smarter model appropriately?

This isn't my language (saw it on a youtube video but agree with it) -- LLMs are not calculators. It's as simple as that.

If the LLM can complete the task using tools, then it's a pass.

Apples team went out of their way to select tests that LLMs would struggle with and then take away tools -- and then have the audacity to write that they're surprised at the outcome. Who would be surprised? No one using AI since GPT-4 expects them to be calculators or algorithm executors.

You want the LLM to be smart enough to realize "I can't do this without tools", grab the tool, use it correctly, and give you the actual correct answer. Preventing LLMs from using tools or writing and executing code -- then you're intentionally crippling them.

The towers of Hanoi one is kind of weird, the prompt asks for a complete move by move solution and the 15 or 20 disk version (where reasoning models fail) means the result is unreasonably long and very repetitive. Likely as not it's just running into some training or sampler quirk discouraging the model to just dump huge amounts of low-entropy text.

I don't have a Claude in front of me -- if you just give it the algorithm to produce the answer and ask it to give you the huge output for n=20, will it even do that?

I doubt I could reliably solve Towers of Hanoi in my head for more than 3 or 4 discs.

You are the only person suggesting that o3 is AGI or even an approach to AGI. They’re different beasts entirely.

It single-shots the towers of Hanoi https://chatgpt.com/share/6848fff7-0080-8013-a032-e18c999dc3...

Sounds pretty useless

The first thing I asked was along the lines of "Find me an implementation of X that does not depend on framework Y, because it's not maintained". o3 thought for a minute and then returns a repository that depends on Y. In its defense, 4.1 also found the same repo and commented that it "has a light dependency", which was cute, if unhelpful.

Adding a "think about whether your answer actually depends on Y" fixed it, and the response was pretty well thought out. My only gripe is the current persona seems to be tech-bro-informal. It opened with "TL;DR" and included a lot of developer slang that reads like it's trying to be cool in a way that masks overconfidence.