Results of "Humanity's Last Exam" benchmark published

You shouldn't engage in a CAL, a regular lawsuit from anyone wronged will be cheaper and way more painful for them.

If you're in the US, consider small claims court. It's a small sum of money, you won't need to pay a lawyer, they'll probably not even show up.

Scale AI's whole business model is wage theft. I don't mean to be insensitive, but out of all the Scale AI experiences I've heard about, yours is the least egregious. It's a dystopian, shitty company.

I know someone who had 5+ questions accepted after the deadline, as he thought (as was represented on the website) that they would still be eligible for prizes. The lack of clarity is shameful; the minimum that can be done now is complete transparency of the ranking, etc.

Hi everyone, this is Long Phan from CAIS. I noticed this thread and wanted to provide you with our perspective on the contest.

The goal was to involve experts from a wide range of fields and disciplines in the development of frontier AI — especially people who might not normally have the chance to participate in this industry. To that end, we consider the contest a great success.

I’m happy to report that we received tens of thousands of submissions, many of them highly competitive. Our participants really rose to the occasion. It’s true that we extended a grace period for submissions, and the intention here was to make the project accessible to the broadest possible group of people. At the same time, the reality is that the vast majority of our prize-winners submitted their questions within the initial deadline.

We appreciate your contributions to Humanity’s Last Exam, and we hope you’ll take pride in your efforts to push this fledgling technology forward.

It feels that they preferred giving 500$ to many people than to many times 500$ to few people. I also got only 500$ to a question that wasn't my best (I got ~8 questions accepted)

Out of curiosity, do you know if there's a public list of the "top 550 submissions"? Is it ordered as in the code base?

"Intelligence" itself is very ill-defined and we've never been able to measure it properly, IQ is rife with issues.

At some point, you just have to be pragmatic and measure the questions you want the AI to be good at answering, rather than trying to measure intelligence in general.

In that sense, I see this as one more benchmark that collects questions that we want/expect AI to be good at, is not good at yet and have been underrepresented at previous benchmarks. That's obviously valuable, there's nothing "magical" about it. Although it is reasonable to be annoyed at the "Humanity's Last Exam" naming, of course they must have missed plenty of edge-cases like everyone else and it is very arrogant to claim it will be the "Last" one.

The things that are missing are what stops us from having useful agents so far: Agency, judgement, sense of time, long horizon planning, not being gullible. I kinda feel like some amount of ego is necessary to get a model to behave like that.

I agree that many aspects of intelligence—and of the lack of intelligence—are not being measured by such benchmarks. One issue is they are only examining problems that have right answers.

One of the most powerful uses of LLMs for me, at least, is brainstorming: having them suggest possible avenues for me to pursue with specific projects I am working on. If I give Claude or ChatGPT or Gemini enough context about my problems, they usually come up with useful suggestions—sometimes amazingly well. Are they better at that than the smartest human? I don't know. How do you quantify the quality of an idea? But those ideas often seem really, really good to me.

Another difficult-to-measure capability is interaction. Back-and-forth conversations with models don't always go well, but when they work they frequently blow me away. But those successes are dependent partly on the model, partly on me, and partly on how the conversation happens to unfold. Again, that success or failure doesn't seem measurable with benchmarks that require objectively right answers.

ARC-AGI is a benchmark with no language that could plausibly be solved by primitive humans, assuming only intelligence.

I think the concept you're dancing around the edges of is the nature of what parts of "intelligence" are driven by:

1. Language and how interrelated it is to our ability to transfer knowledge and experience, as well as its role in structuring our internal thinking. I haven't seen any academic research on the matter, but there are more and less concrete instances of this throughout history. This Wikipedia article about the history of Algebra is a great example of how 2000 years of evolution led to a formulation of the same concepts, but with a reduced cognitive load that 10-12 years olds learn today as a matter of course. (https://en.wikipedia.org/wiki/History_of_algebra#Stages_of_a...).

2. Knowledge, transferred through language, education, and culture. Calculus in the early 1600's is a great example, without it and subsequent developments, probably 80% of the college/post-grad math/science/physics education wouldn't even exist. The stuff we teach our 18 year olds today required the 1600s' greatest minds to figure out.

3. The capacity of our human wetware.

It's hard to treat #3 in isolation because our modern concept of intelligence is inextricably tied to #1 and #2. Also it's hard to place where "critical thinking" and "creativity" enter the picture, since they both rely heavily on all three aspects above.

>He would likely be more intelligent than a toddler

I think you are falling into the trap of "we have technology and are therefore smarter." I would expect an average Roman senator could formulate far better speeches off the top of his head than 99% of modern people, and also in excess of anything an LLM is capable of. And that's supposed to be an LLM's specialty, there's no comparison when it comes to organizing actual projects like construction or campaigns.

This is true but that's because it's gotten hard to do much else. LLMs are eating up everything else that don't require long horizon planning or multimodality.

If you created a new benchmark today that didn't lean on the things I've mentioned or esoteric/super specialized domain knowledge (that would actually require some sort of super-human performance to ace) like this or Frontier Math, LLMs would probably do pretty well.

I'm curious why you are confident they would be more intelligent than a modern toddler?

I largely empathize with your point. But, as I can recognize there are some out there far better at problem solving than I am, I am growing ok with the idea that intelligence can be measured. Not to a single number, most likely, but to a variety of different aspects.

Similarly, I'd imagine that a human from 2000 years ago is probably more hardy than one from the modern age. If only because of selection effects at play.

Obviously, you can't extrapolate a straight line between either measurement and expect it to continue in either direction. But I don't know why you couldn't build up a measurement for it?

(And it should go without saying that you shouldn't be judging worth using this sort of measurement.)

> We will ask them how much do they like your plan. The more they like it the higher your score

Here's my evil-AI response:

"Kill all of your enemies, and all their descendants and friends, and salt the land".

I still have like 95% of the A4 left for other good plans.

In other words, we should ask it to give us "the answer to life the universe and everything". :)

Having read hitchhiker's guide as a child in the '90s, that asking this question to a machine (even as a joke) is not far-fetched shocks me.

Honestly, I thought space travel to the Moon and maybe Mars would be common before this level of advances in artificial intelligence.

Turns out gravity was harder to solve than intelligence.

You could go even simpler than that.

"Where should I go for dinner?"

Does it know what questions to ask? Does it know to ask questions at all? Where does one even start with such a question? These are things easily knowable to a human, but an AI would likely just just if you like Italian food or something

Even simpler, ask it to reason through getting out of an escape room.

It does feel a bit like the early days of AI:

"We want to make computers do what smart people do. What do smart people do? They play chess! Once we've solved that, everything else will be easier."

It has been remarkable how much of the "easier" stuff they've made progress on -- like natural language and images. But after a huge quantum improvement, it doesn't seem very good at adapting to a lot of the things we really need them for.

Isn't this a terrible question to measure intelligence? It looks like it's testing niche domain knowledge along the lines of:

> What color is the ball hidden behind the flowerpot in my neighbor's backyard?

Maybe you can reason towards the answer if you only have a deep knowledge of bird anatomy and not Apodiformes anatomy, and that's the intelligence part?

Yes, indeed. And I wonder what this type of question has to do with intelligence. Think of the 10 most intelligent people you know. How many of them know the answer to this?

This is testing “knowledge”, not intelligence. And with access to most of the knowledge in the world and basically infinite memory, that’s not very exciting for an AI.

Good point. I wouldn’t expect a human to need the last sentence.

A grade is mostly meaningless if you don't know how it was calculated, so no one would "rely" on it. If nothing else, you need to know the grading methodology after the test.

It's the same problem with cheating students. Once the test questions are known, they have a very short lifespan before cheaters can make them worthless. Tests have to be refreshed.

I can answer 2 of them quite quickly with pen and paper (compsci, physics) and one that I had to look up some definitions on wikipedia (maths) so I am certain there are people who can do more than 5.

The computer science one seems weirdly easy compared to the rest, it's multiple choice and it is very easy to get it by process of elimination even if you don't understand how to actually do the problem.

The public dataset is available on HF here: https://huggingface.co/datasets/cais/hle

As the main website notes: "The dataset consists of 3,000 challenging questions across over a hundred subjects. We publicly release these questions, while maintaining a private test set of held out questions to assess model overfitting."

You can just view the dataset on hugging face