OpenAI Codex Review

You guys are doing great work, codex too, keep at it.

Can it be used to fix bugs? Because the ChatGPT web app is full of them and I don't think they are getting fixed. Pasting big amounts of text freezing the tab is one of them.

> incorrect, its an o3 finetune.

This is Open AI's fault (and literally every AI company is guilty of the same horrid naming schemes). Codex was an old model based on GPT-3, but then they reused the same name for both their Codex CLI and this Codex tool...

I mean, just look at the updates to their own blog post, I can see why people are confused.

https://openai.com/index/openai-codex/

The particularly integration pain point to me is about network access, that prohibits several banal tasks to be offloaded to codex:

1. Cannot git fetch and sync with upstream, fixing any integration bugs; 2. Cannot pull in new library as dependency and do integration evaluations.

Besides that, cannot apt install in the setup script is annoying (they blocked the domain to prevent apt install I believe).

The agent itself is a bit meh, often opt-to git grep rather than reading all the source code to get contextual understanding (from what the UI has shown).

How do you find it compares to Claude Code?

A.I. Assist is probably the ultimate low-code platform. Will it be long before software engineers are replaced?

Even content changes can require deliberate thought. Any system of decent size is probably going to have upstream/downstream dependencies - adding a field might require other systems to account for it. I guess I can see small CSS changes, but how does the user know when the change is small or "small"?

People will learn about accessibility, multi-platform (mobile/desktop) and many other gotchas real quick.

This almost seems like this is a funnel to force people to become software engineers.

Me too :)

I feel it will get there in short order..but for the time being I feel that we'll be doing some combination of scattershot smaller & maintenance tasks across Codex while continuing to build and do serious refactoring in an IDE...

In my testing with it anything that requires a bit of critical thought gets completely lost. It's about on par with a bad junior engineer at this point.

For instance I ask it to make a change and as part of the output it makes a bunch of value on the class nullable to get rid of compiler warnings.

This technically "works" in the sense that it made the change I asked for and the code compiles but it's clearly incorrect in the sense that we've lost data integrity. And there's a bunch of other examples like that I could give.

If you just let it run loose on a codebase without close supervision you'll devolve into a mess of technical debt pretty quickly.

It is most definitely going to be the unemployment line. When in the history of productivity gains, has it translated to more time for people to do other things that are not work? It always translates to more profits for shareholders and bigger pay for executive class, followed by more work for half the workers to fill up the time opened up by the said productivity gains, and unemployment for the other half.

Think we've got a long time yet for that. We're going to be writing code a lot faster but getting these things to 90-95% on such a wide variety of tasks is going to be a monumental effort, the first 60-70% on anything is always much easier than the last 5-10%.

Also there's a matter of taste, as commented above, the best way to use these is going to be running multiple runs at once (that's going to be super expensive right now so we'll need inference improvements on today's SOTA models to make this something we can reasonably do on every task). Then somebody needs to pick which run made the best code, and even then you're going to want code review probably from a human if it's written by machine.

Trusting the machine and just vibe coding stuff is fine for small projects or maybe even smaller features, but for a codebase that's going to be around for a while I expect we're going to want a lot of human involvement in the architecture. AI can help us explore different paths faster, but humans need to be driving it still for quite some time - whether that's by encoding their taste into other models or by manually reviewing stuff, either way it's going to take maintenance work.

In the near-term, I expect engineering teams to start looking for how to leverage background agents more. New engineering flows need to be built around these and I am bearish on the current status quo of just outsource everything to the beefiest models and hope they can one-shot it. Reviewing a bunch of AI code is also terrible and we have to find a better way of doing that.

I expect since we're going to be stuck on figuring out background agents for a while that teams will start to get in the weeds and view these agents as critical infra that needs to be designed and maintained in-house. For most companies, foundation labs will just be an API call, not hosting the agents themselves. There's a lot that can be done with agents that hasn't been explored much at all yet, we're still super early here and that's going to be where a lot of new engineering infra work comes from in the next 3-5 years.

Well, the optimistic take is that if something gets cheaper to produce (e.g. code), demand for it actually increases.

Now you could argue that any non technical person could just oversee the agents instead. Possibly. Though in my experience, humans like to have other humans they trust oversee and understand important stuff for them.

I guess maybe the analogy is we as software devs are all horses.

With Codex and Claude Code, these model agents are cars.

Some of horses will become drivers of cars and some of us will no longer be needed to pull wagons and will be out of a job.

Is that the proper framing?

Maybe, maybe that's FUD...I can't predict the future.