GPT 4.5 level for 1% of the price

Do the benchmarks reflect real-world usability? My feeling is that the benchmark result numbers stop working above 75%.

In a real problem you may need to get 100 things right in a chain which means a 99% chance of getting each single one correct results in only 37% change of getting the correct end result. But creating a diverse test that can correctly identify 99% correct results in complex domains sounds very hard since the answers are often nuanced in details where correctness is hard to define and determine. From working in complex domains as a human, it often is not very clear if something is right or wrong or in a somewhat undefined and underexplored grey area. Yet we have to operate in those areas and then over many iterations converge on a result that works.

Not sure how such complex domains should be benchmarked and how we objectively would compare the results.

It doesn't really matter what nationality or ethnicity you are, but if you communicate with the model in Chinese you might get better results from this model.

Then again, if they've misrepresented the strength of the model overall, there might be some other shenanigans with their results. The fact that their results show their model is worse than GPT-4.5 on 2 Chinese language benchmarks, while it's so much stronger on some of the others, is a bit weird.

> VC has been investing in AI as-if it were a winner takes all market, but it has been obvious that isn’t the case.

I really don't see how this was supposed to go, and I've never heard an explanation.

I don't see any kind of coherent vision from any of these types.

Most normal folks (i.e not SV/HN types that seem to desire to replace their marketable programming skills with LLM output) really don't "want" LLMs in any real sense.

Sure, people use them like a search engine, kids cheat on homework with them etc, but there's not this overwhelming universal desire for them like there was for, say iPhones.

I never once have heard any sort of proposed roadmap for how LLMs were supposed to work as a product.

They were just going to get, uh "really good" and take everyone's office job or something?*

Normal F500 organizations that are obviously a target for LLM use (via hyperscaler sales) are still yet to see a clear path to "revolutionizing" their workforce or whatever via LLMs-it's just not there. Costs are too high, there's no obvious use case, "hallucinations" are a real impediment etc.

I'll add many of the public usecases for this (i.e those a hyperscaler would blog about as a sales promo) are seriously weak ("we reduced onboarding time by 20% with $MODEL")

I would really like to hear a proposal for how this is all supposed to come together. Does anyone have a concrete plan for the future for all this stuff?

*I'll note, this is NOT the way to sell a product to the masses, either.

Addendum: I'm not an "LLM hater" by any means. I pay for GH Copilot, and have been running local LLMs since it's been a thing (granted with limited hardware, and limited quality)--I intend to wait a bit and buy better hardware with one motivating factor being running local LLMs in a year or so when the open-source offerings stabilize"

can I say something.

This is one of the most finest and most accurate things that I have read in a long time.

This really could be a blog post which I encourage you to make! (I would prefer github pages but if you really want , I have a domain name on cloudflare and I am more than willing to host the static page of such blog on my own domain name for absolutely free (lets go , cloudflare!)

Its just facts. Pure facts. "" Generally, I’ve found that almost no founders or friends I speak with have any vision for the future anymore. They care only about making money and do not care how. It’s a spectacular collapse of vision and purpose—these people have always existed but it feels incredibly pervasive now. ""

Why did I read it in a monotonous way as if a student from the future understands the current scenario. I felt as if it was the same level of sadness in my heart as that when you listen to some video which has raining background and he reads the dark comedy (something like burialgoods oats shitposting but this time more serious and real!)

Currently saving this on wayback machine just for this comment. Internet needs to preserve this comment , no matter what.

There used to be significant alignment between engineers, founders and certain VCs: lots of excitement around building software that genuinely made things better/easier/cheaper. Each group naturally wanted a different thing out of this arrangement but each camp was on the same page.

Now I feel like everything is more top-down. The tech sector feels less like market capitalism and more like something being centrally planned: we all must chase trends that come from various industry thought-leaders. And it all must be done a very specific way (this in particular is why Chinese companies are likely going to disrupt the AI market: they’re free from this burden)

Founder risk has been nil for a long time either because they pay themselves six figures out the gate or because the job market has been hot enough that they can market utter failure to get another job.

There’s a lot of opportunity to make low cost software that out competes big tech just because it doesn’t demand 10000x returns on every if statement.

I’d encourage Europeans to start replacing American software vendors with small teams today. You won’t become the next American oligarch but you’ll be able to clean up millions from the incompetent Americans.

VCs are happy to throw money at something if they believe they can corner a market. It just doesn't have to do much with reality. Until of course, it becomes self fulfilling. But in this case it seems like it's not going to happen because nobody has a moat.

> I’ve found that almost no founders or friends I speak with have any vision for the future anymore.

I think in general there is a feeling that the time to get your bag is rapidly shrinking.

Once everything is built by these things there will be no reason to create anything as the platform owners (big tech) will be able to take everything for themselves and no longer have to share 70% with those pesky creators/small business/startups etc.

Possible, but if you look at the graduate students and lecturers behind many of these IPs you will find they are Chinese (or Russians or Iranians).

This is the paradox in those who are championing barring Chinese students from the US to prevent them from stealing IP, they don't see that at least 50% of this IP is generated by students from China, in a way they will be handing the CCP a gift by incentivising those students to remain in China.

>They’ve stolen unimaginable amounts of IP and will continue to do so.

All AI models are built on the back of massive amounts of "IP stealing". Either we consider IP to be valid and then all western companies in this space are just as bad, or we go with the direction the western companies are claiming and then China is not doing anything wrong.

All developing markets "steal" until they've caught up with the competition. Just look at the US and how they "stole" innovations and tech from Europe.

Would you say the west “stole” the IP for paper , ice cream, tea and noodle? Weird notion.

These sour grapes comments are so goofy, and honestly a little racist. The millions of Chinese engineers working out in China are extremely talented, and to downplay their achievements like this and to chalk them all up as thieves is ridiculous. They have the skills, the man power, and the vision, and they’re eating the West’s lunch regardless of your feelings on how fair it is.

It is entirely irrelevant who pioneered the tech. This is why no one gives a crap about xerox anymore.

Dismissing Chinese tech is foolish. They are tech leaders in many areas and moving to new ones every day. Solar, Nuclear, Batteries, EVs, Drones, Robotics etc. They have no one to copy in those fields because they have left the rest of the world behind.

>When it comes to hardware who pioneered all those technologies?

Not sure that matters anymore in the new world order.

Frontier tokens are largely fungible now. The details of how they came about doesn't make them any less useful.

This is the old way of doing it, and probably the way the US is going to go with, at the detriment of its own population. - I would posit that since we are talking about digital goods, there is a better way:

Require open source / open weights of any company that used data to it doesn't own to train its models. If chinese companies do not comply, their copyright becomes void in the US, and these models are very easy to copy. Treat advances in architecture as a utility, and let the utilization of those architectures be the market for companies to compete in.

That one weird trick. x4.

Of course, none of this will prevent china from producing technology that's clearly as impressive, if not more so.

A copyright exemption would just put them at the level of deepseek officially, but they've been working around that anyway in practice. I'm not sure that change would make any difference.

This is all protectionistic measures. Just let it implode, and let companies with better technology take over.

Why should people involved in some hyped company deserve all this "socialism for the rich" from the state?

Correct answer, never think about the future in terms of linear extrapolations. It's a non-linear differential equation with lots of variables and expect complex feedback loops. Systems react to change.

Jevons paradox: https://en.wikipedia.org/wiki/Jevons_paradox

you are assuming that cost is stopping from ppl using these technologies.

These things are not actually useful. They hyper optimzed it for coding usecase but it still sucks balls at it.

> I think you're witnessing it rather getting back in touch with reality than collapsing.

You're witnessing the USA tech scene getting in back to reality. Software engineers in other western countries looked at the salaries the Tech scene was paying in the USA, and scratched their heads.

Its a collapse from fictional reality to real reality , but a collapse nonetheless.

Sometimes reality acts more weird than fiction itself. I have just now decided to call this "fictional reality"

Like yesterday when I realized that nuclear bombs weren't that far away from the creation of chemical resonance & they happened after world war I and I think , just really 5-6 years before nuclear bombs but still!

It actually gave me a lot of hope because I felt that a lot of people were focusing on AI , so I can use AI (sometimes , if I want) to focus on a passion project that I want , to maybe earn some money.

I have also thought of creating AI projects but that too for fun. I don't know two shits but I just want to know what the hype is about from a theoretical standpoint.

What user experience are you talking about? Chatbot? Or software in general? Cause Tiktok beats Facebook out of water. Chatbot for English communities sure, I also prefer Claude over Deepseek in terms of project support and UI. But this is because they are focusing on Chinese communities, Doubao has much better features that is used by Chinese. It's not really comparable even if all US chatbots were accessible in China. Once LLM tech slows, I am sure Chinese chatbots would beat the American ones in terms of user experience.

So far. China has been focused on becoming a world's factory for 30 years. They started moving up the food chain fairly recently.

Give it another generation and if China will not walk off the ledge with either government or societal issues (which, granted, is where they are slowly going IMO) they will own the UX and design as well. My 2c.

A lot of consumer tech with very competitive UX is coming out of China. They are also getting very strongly into e.g. web frontend tech. I see no reason why the west would have any special advantage in this.

Chinese hardware docs are very scarce if you don't know mandarin and don't have Chinese connections.

So to compete on software West might encounter unexpected difficulties. You need good platform docs to develop good software.

Probably not a problem for big companies.

Software is more trivial than hardware. That’s why you see bootcamps for software but not for hardware. China can easily eclipse the US on the software front. And they have.

The current western lifestyle of dirt cheap foreign goods exists only in a world of relative poverty and underdevelopment.

It's not about patriotism. Many people outside the US, myself included, see a problem with authoritarian superpowers per se. Although now that the US is rapidly drifting towards authoritarianism, that just seems like an inevitable future to prepare for.

I think (hope) most folks care less about the “attack on patriotic identity” and are more concerned that what is essentially a dictatorship is rising in power significantly. History has shown dictatorships rarely end well for the general populace and the rest of the world.

Democracy has its flaws, but one of the features that most people prefer is that it can significantly change how it looks and operates to reflect the will of its people without violence.

Like 95% of the planet, I'm not American. Like 82% of the planet, I'm not Chinese.

BYD being better than Tesla isn't a matter of patriotism in most of the world. DeepSeek and Baidu can spend as long as they want playing musical chairs/rap battles with Anthropic and OpenAI, it makes no odds to me which wins.

America and China both have politics that have no reason to care for people like me, nor people like my friends, that they are for different reasons and differ in penalties for being an out-group doesn't matter when I'm a foreigner to both, when my antecedent are who the 13 Colonies rebelled against and more recent antecedent forced unwanted opium sales on China.

As a European I can say that I like this development because prices go down and models get better and OpenAI has no monopol anymore.

It has nothing to do with just giving up and going 'Wellp, I guess China wins.'

China and the US are obviously very different culturally in just about every way possible. This difference makes for great competition. Someone in another topic mentioned something that seemed pretty insightful to me - in that where LLM companies failed in the US was in basically becoming clones of each other, whereas DeepSeek (and now perhaps Baidu) were going in a different way, and that way turned out to be better.

US companies will inevitably copy these strategies, one way or the other, as will Chinese companies copy what ends up working well from the US (see their latest rockets looking more than a little inspired by Starship). And the true competitiveness ensures in the end that the main people who will win will not be whichever guy ended up founding an AI company first, but you and I. It's how capitalism is supposed to work - companies beat themselves down into a race to the bottom, and society reaps the rewards. It only gets really messed up when there's no "real" competition, which is an increasingly frequent state of affairs. But that definitely will not be the case here.

Expect the same thing from India in the future as well. Their economy is advancing rapidly, and soon enough we're going to have another 1.4 billion people able to fully utilize the outliers such a population entails to similarly drive things forward in their own unique way. It's a great future for the world as a whole.

Given the framing of "most people" and "patriotic": China's got 1bn+ people.

don't know why you're getting down voted because it's true. we should work together with the new world superpower instead of fight it.

and don't start on some dictator BS. the US does/has done as many, if not more, bad things as china.

Surely, this is as inevitable as not being able to use Wechat as an American.

The models aren't what worry me anyway. China is going to kick our ass when it comes to AI integration into society and the economy.

Imagine the difficulties faced by America vs China in integrating AI into healthcare.

We are just too worried about winning this AI model sporting event even though the entire concept is flawed and doomed to failure. We actually have to figure out how to use these models for more than how many Rs are in strawberry. That appears to be the actual hard part.

Of course, none of this is helped by having wasted an entire generation of some of America's best minds on javascript programming for obscene profit.

"Free" does not mean "available to everyone."

Good point. But pushing it as a product with that knowledge still puts it in a weird spot for me.