Is Winter Coming? (2024)

Related similar thing when I sent my dog's recent bloodwork to an LLM, including dates, tests, and values. The model suggested that an advancement in her kidney values (all still within normal range) were likely evidence of chronic kidney disease in its early stage. Naturally this caused some concern for my wife.

But, I work in healthcare and have enough knowledge of health to know that CKD almost certainly could not advance fast enough to be the cause of the kidney value changes in the labs that were only 6 weeks apart. I asked the LLM if that's the best explanation for these values given they're only 6 weeks apart, and it adjusted its answer to say CKD is likely not the explanation as progression would happen typically over 6+ months to a year at this stage, and more likely explanations were nephrotoxins (recent NSAID use), temporary dehydration, or recent infection.

We then spoke to our vet who confirmed that CKD would be unlikely to explain a shift in values like this between two tests that were just 6 weeks apart.

That would almost certainly throw off someone with less knowledge about this, however. If the tests were 4-6 months apart, CKD could explain the change. It's not an implausible explanation, but it skipped over a critical piece of information (the time between tests) before originally coming to that answer.

I very often also get better programming results than less experienced engineers, even though I'm not remotely doing any kind of "prompt engineering".

Also how you ask matters a lot. Sometimes it just wants to make you happy with whatever answer, if you go along without skepticism it will definitely make garbage.

Fun story: at a previous job a Product Manager made someone work a full week on a QR-Code standard that doesn't exist, except in ChatGPT's mind. It produced test cases and examples, but since nobody had a way to test

When it was sent to a bank in Sweden to test, the customer was just "wait this feature doesn't exist in Sweden" and a heated discussion ensued until the PM admitted using ChatGPT to create the requirements.

This same phenomenon is true for classic search engines as well. Whenever I am becoming informed on a new topic my first searches are always very naive and targeted just at discovering the relevant jargon that will let me make better searches. It turns out that many disciplines contain analogue concepts, just with different words being used to describe them. Understanding the domain specific language used is more than half the battle.

This anecdote corroborates my theory that it will still be critical to become an expert in your field. Everyone is treating AI like it's a zero-sum game with regards to jobs being "lost" to AI, but the reality is that the best results will come from experts in the field who have the vocabulary and knowledge to get the best answers.

My fear is that people treat AI like an oracle when they should be treating it just like any other human being.

For the sake of discussion I want to play devil's advocate concerning your point

> It was an interesting result to me because it shows that experts in a field are not only more likely to recognize when a model is giving incorrect answers but they're also more likely to get correct answers because they are able to tap into a set of weights that are populated by text that knew what it was talking about. Lay people trying to use an LLM to understand an unfamiliar field are vulnerable to accidentally tapping into the "amateur" weights and ending up with an answer learned from random Reddit threads or SEO marketing blog posts, whereas experts can use jargon correctly in order to tap into answers learned from other experts.

Couldn't it be the case that people who (in this case recognizable to the AI by their choice of wording) are knowledgeable in the topic need different advise than people who know less about the topic?

To give one specific examples from finance: if you know a lot about finance, getting some deep analysis and advice about what is the best way to trade some exotic options is likely sound advice. On the other hand, for people who are not deeply into finance the best advice is likely rather "don't do it!".

I had a similar experience. I did some back of the envelope math and my wife suggested I run it through ChatGPT. After actually doing the math, I felt a lot better about my understanding of the problem and I just... don't trust an LLM to understand algebra. Yeah, they're awesome most of the time, but I don't want to trust it with something important, be wrong, and then have to explain to someone that I trusted the opinion of a couple of big matrices, over my own knowledge and experience, on a high school word problem.

LLM users are highly susceptible to the confirmation bias, by putting their expectations into the prompt.

Maybe try a 2-step approach: first ask the LLM to translate your question into expert-language, then ask that question :)

As someone with this very style (my own blog posts often rise into the 5k word range) and also from a technical background, I can at least explain my motivations for length: absolute domination of the argument.

In professional settings, brevity is often mistaken for inexperience or a weak position. As the thinking goes, a competent engineer should be able to defend every position they take like a PhD candidate defending their dissertation. At the same time, however, excess verbosity is viewed as distinctly “cold” and “engineer” in tone, and frowned upon by non-technical folks in my experience; they wanted an answer, not an explainer.

The problem is that each of us have the data points of what succeeds in convincing others: the longer argument, every single time. Thus we use it in our own writing because we want to convince the imagined reader (as well as ourselves) that our position is correct, or at the very least, sound. In doing so we write lengthy posts, while often doing research to validate our positions with charts, screenshots, Wikipedia articles, news sources, etc. It’s as much about convincing ourselves as it is other readers, hence why we go for longer posts based on real world experiences.

One plot twist subjective to me: my lengthy posts are also about quelling my brain, in a very real sense. It is the reader, and if I do not get everything out of my head about that topic and onto “paper”, it will continue to dwell and gnaw on the missed points in perpetuity. Thus, 5k posts about things like the inefficiency of hate in Capital or a Systems Analysis of American Hegemony, just so I can have peace and quiet in my own head by getting it completely out of said head.

Ironically this is one of the best use cases I’ve found for AI tools at the moment.

Summarize and critique this argument in a series of bullet points.

More seriously though, I think there is a lack of rigorous thinking about AI specifically and technology in general. And hence you get a lot of these rambling thought-style posts which are no doubt by intelligent people with something compelling to say, but without any fundamental method for analyzing those thoughts.

Which is why I really recommend taking a course in symbolic logic or analytic philosophy, if you are able to. You’ll quickly learn how to communicate your ideas in a straightforward, no nonsense manner.

The problem is, unless you expand every point then some jerk on HN will nit-pick your "logical fallacies".

I find myself writing longer and more defensively because lots of people don't understand nuance or subtext. Forget hyperbole or humour - lots of technical readers lack the ability to understand them.

Finally, editing is hard work. Revising and refining a document often takes several times longer than writing the first draft.

Strategery is the most extreme example of this. He has interesting things to say but boy does he love the act of actually _saying_ them.

I wish it was a skill easy to learn but it's not.

I mean it's their blog, the writing is just as much for them as it is for you.

To have a good mental model for modern AI agents you have to understand both the LLM and the other stuff that's built up around it. OP is correct about the behavior of LLMs, and that is valuable information to keep in mind. Then you layer on top of that an understanding that some implementations of agents will sometimes automatically feed search results into context, if you ask them to or are paying for an advanced tier or whatever the extra qualifications are for your particular tool.

If you skip this two-part understanding then you run the risk of missing when the agent decided not to do a search for some reason and is therefore entirely dependent on statistical probability in the training data. I've personally seen people without this mental model take an LLM at its word when it was wrong because they'd gotten used to it looking things up for them.

Defending TFA a little ... the rest of the article builds up context around the word "Completely", so that the single example of "Highway zipping" is not just what is being discussed.

"Completely" here should be expanded to include all the unique and unforseen circumstances a driver might encounter, such as a policeman directing traffic manually or any other "soft" situation that is not well represented in training.

Not to mention the somewhat extreme amount of apriori and continuous mapping that goes into operating a fleet of AVs. That is hardly to be considered "Completely autonomous".

This isn't just pedantry, the disconnect between a technical person's deep understanding and a common user's everyday experience is pretty much what the article hinges on. Try taking a Waymo from SF to NYC. This seems like something a "Completely autonomous" car should be able to do given a layperson's understanding of "Completely", without the experts' long list of caveats.

waymo is pretty amazing but it's still short of complete full self driving, and it's not at all clear that it can close the gap. similary, current LLMs are really remarkable. but it's not at all clear that they will make the leap into "real" intelligence.

> For example, while researching new monitors today, I told Gemini...

You told an agent, not just an LLM.

> And this is already not hype, it's reality anywhere Waymo operates.

Some beg to differ; see e.g. https://www.youtube.com/watch?v=040ejWnFkj0 .

> For example, while researching new monitors today, I told Gemini to compare $NEW_MODEL_1 with $NEW_MODEL_2.

But this feature was a staple of most online shops that sell monitors and a bunch of "review" sites. You don't need a highly complex system to compare 2 monitors, you need a spreadsheet.

Waymo only works because it's geofenced — that's a massive barrier to "completely autonomous" (or level 5 automation)

> what LLMs do is string together words in a statistically highly probable manner.

I guess the article fails to admit that when you have billions of connected points in a vector space, "stringing together" is not simply "stringing together". I'm not a fanboy but somehow GPT/attention based logic is capable of parsing input and data then remodeling it in depths that are surprising.

Waymo operates on highly controlled and mapped environments. Can they handle Rome or Mumbai?

They absolutely can if you prompt them to. You can even add it to your system prompt for it to happen in every new conversation!

> Let's just say that people once thought that Big Tech was once invincible, until it wasn't.

Half the world runs on Big Tech. Some of them have cash reserves bigger than the GDP of sizeable countries. They lead in R&D investment: https://www.rdworldonline.com/top-15-rd-spenders-of-2024/

> Obviously those exposed in the AI hype will tell you that there is no winter.

Go look at how much money was spent on AI R&D in the last AI 'summers' (and winters). Pennies compared to the billions and billions of dollars the private and public sector is throwing at it right now.

Will some investments turn out to be a waste of time and money? Yes.

Will investment be reduced to a fraction of what it is today? Hell no.

The music stops when humans are economically obsolete.

The point isn't that progress is not happening but that it's slowing down. You get more of the same, smaller memory footprint, faster responses, less hallucinations, etc. Significant progress would be another deep seek kinda of breakthrough, near 0% hallucination rate, performing like current models with less than half of their dataset, epistemological self awareness (i.e. I am not sure of the correctness of the answer I just gave you, ability to override assumptions from the training dataset, etc).

We are just getting cars of different shapes and colours, with built-in speakers and radio. Not exactly progress

That's also one of those things that would probably confuse LLMs as readily as it confuses North Americans (for much the same reason - training).

Spring 2024 for me was from the 1st of September to the 30th of November.

o1 was not a major advancement