AI Responses May Include Mistakes

We lost that battle back when we collectively decided that sales and marketing is respectable work.

Complain to it enough times, remain resilient and you’ll eventually figure it out (that’s a wild card though). Or find someone who has and take their word for it (except you can’t because they’re probably indistinguishable from the ‘bot’ now according to the contradictory narrative). Iterate. Spiral. No one should have to go through that though. Be merciful.

> Normally this would never fly, but nowadays it's kind of OK?

We started down this path ever since obvious bugs were reframed as "hallucinations".

I've recently started wondering what the long term impacts of AI slop is going to be. Will people get so sick of the sub-par quality that there will be a widespread backlash, and a renewed focus on handmade or artisinal products made by hand? Or will we go the other way where everyone will accept the status-quo and everything will just get shittier, and we will just have multiple cycles of AI slop trained on AI slop?

Suddenly? That's the level of quality that is standard in all software projects I've ever seen since I've started working in IT.

Enshittification is all around us and is unstoppable. Because we have deadlines to hit and goals to shows we reached to the VP. We broke everything and the software is just half working? Come on that's an issue for the support and ops teams. On to the next beautiful feature we can put on marketing slides!

And how quickly the bar is being lowered

And suddenly this type of quality is becoming "normal" and acceptable now?

The notion that "computers are never wrong" has been engrained in society for at least a century now, starting with scifi, and spreading to the rest of culture.

It's an idea that has caused more harm than good.

When were search results 100% fact checked and accurate??

This is a very very good point. If this was happening with different queries that we never used, or a new type of questions/queries than I would have some patience. But it happens exactly with the formulations that were giving you the best results in the SERP before!

That was true before AI too (I know, I did such searches myself). Google results has been drowning in slop for over a decade now - it was just human-generated slop, aka. content marketing and SEO stuff.

I'm not defending the AI feature here, just trying to frame the problem: the lies and hallucinations were already there, but nobody cared because apparently people don't mind being constantly lied to by other people.

Ad supported search has been awful for a few years now, just buy a Kagi subscription and you'll be like me: horrified but mildly amused with a dash of "oh that explains a lot" when people complain about Google

I'm a member of a few car groups on Facebook and the misinformation coming from Google is infuriating, because people treat it as gospel and then you have to explain to them that the AI slop they were shown as the top result in Google is not - in fact - correct.

As a simple example - someone googled "how to reset sensus system in Volvo xc60" and Google told them to hold the button under the infotainment screen for 20 seconds and they came to the group confused why it doesn't work. And it doesn't work because that's not the way to do it, but Google told them so, so of course it must be true.

I think it's just much easier for an LLM to learn how to be convincing than it is to actually be accurate. It just has to convince RLHF trainers that it's right, not actually be right. And the first one is a general skill that can be learned and applied to anything.

https://arxiv.org/html/2409.12822v1

It's the unfounded promises that this will be solved because the tech will only get better that really upset me. Because sure, it will get better, I'm pretty certain of that. They'll have additional capabilities, they'll have access to more-recent data, etc. But "better" does not necessarily equate to "will fix the lying problem." That's a problem that is BAKED INTO the technology, and requires some kind of different approach to solve -- you can't just keep making a hammer bigger and bigger in the hopes that one day it'll turn into a screwdriver.

Before LLMs really took off, we were in the middle of an "AI winter", where there just weren't any promising techs, at least none with sufficient funding attached to them. And it's WORSE now. LLMs have sucked all the air out of the room, and all of the funding out of other avenues of research. Technologies that were "10-20" years away now might be 30-40, because there's fewer people researching them, with less money, and they might even be completely different people trying to restart the research after the old ones got recruited away to work on LLMs!

> eg i ask for a source, it gives me a quote

It should give you both - the quote should be attributed to where it was found. That's, generally, what people mean when they ask or search for "a source" of some claim.

As for general point - using LLMs as "better search" doesn't really look like those Google quick AI answers. It looks like what Perplexity does, or what o3 in ChatGPT does when asked a question or given a problem to solve. I recommend checking out the latter; it's not perfect, but good enough to be my default for nontrivial searches, and more importantly, it shows how "LLMs for search" should work to be useful.

> LLMs as a replacement for search

Some people expect LLMs as part of a better "search".

LLMs should be integrated to search, as a natural application: search results can heavily depend on happy phrasing, search engines work through sparse keywords, and LLMs allow to use structured natural language (not "foo bar baz" but "Which foo did a bar baz?" - which should be resistant to terms variation and exclude different semantics related to those otherwise sparse terms).

But it has to be done properly - understand the question, find material, verify the material, produce a draft reply, verify the draft vis-a-vis the material, maybe iterate...

Its good to be shown direction. When I only have a vauge idea of what I want, AI usually helps me frame it into searchable terms I had no clue existed.

I find LLMs are often better for X vs Y questions where search results were already choked by content farm chaff. Or at least LLMs present more concise answers, surrounded by fewer ads and less padding. Still have to double check the claims of course.

Maybe that's because we're conditioned by the UX of search.

But another thing I find even more surprising is that, at least initially, many expected that the LLMs would give them access to some form of higher truth.

Half my browser extensions have sole purpose of removing shit from sites I visit.

HN is like a unicorn that havent made me block a single thing yet.

We can't expect the vast majority of regular users to have any of that skill.

What is this going to lead to? fascinating times.

What’s the word?

I switched to DDG over seven years ago and just realized it had been that long when I read your comment. Google started wasting my time and I had to shift.

fyi, you can remove any and all “ai” assistant bs etc from DDG if you use the noai subdomain (in case you wanna avoid their stuff, although it’s much less prominent anyway) https://noai.duckduckgo.com/

I'm so lazy, I have chat bots do all kinds of complex calculations for me. I even use it as a stock screener and the poor thing just suffers, burning fuck tons of electricity.

> Let's use the word machine for this number problem!

You know, that's a thought process that makes internal sense.

You have someone who's terrible at math. They want something else to do math for them.

Will they prefer to use a calculator, or a natural language interface?

How do you use a calculator without knowing what you're doing?

That's just plain wrong, and I'm a smoker. LLMs won't affect the ones around you, unless you engage with them in some way. Sit next to me while I smoke and you'll be affected by passive smoking regardless of how much you engage or not. Not really a accurate comparison :)

Wish I could edit, but I was referring to the bill splitting math specifically here. I didn't mean to quote the rest.

When it comes to facts that actually matter, people need to know to verify the output.

Yes, more popups, retention metrics be damned. Even 2 years since ChatGPT, many people still think it's omniscent which is what's causing trouble.

Unfortunately, getting bit in cases such as publishing misinformation or false legal citations waste everyone time, not just their own.

That always struck me as pretty overblown, given that before map apps, people got lost all the goddam time. It was a rare trip with any complexity that a human map reader wouldn’t make a mistake or two.

LLMs aren’t competing with perfect, they are competing with websites that may or may not be full of errors, or asking someone that may or may not know what they are talking about.

Yeah, but we're all used to having software integrated into our lives now. And we all now how shitty and broken software often is...

Apple maps currently insists that there's a hotel and restaurant across the street from me.

According to the address on the business website that Apple Maps itself links to, the business is 432 km away from me.

Which local models are you using, that do not output loop garbage at temperature 0?

What do you get at very low temperature values instead of 0?

Although I broadly agree, I wouldn't go quite as far as where you say:

> All the "no true Scotsman" stuff about what it's doing behind the scenes is irrelevant, because we have no idea what human brains are doing behind the scenes either.

Computers and transistors have a massive speed advantage over biological brains and synapses — literally, not metaphorically, the same ratio as the speed difference between how far you walk in a day and continental drift, with your brain being continental draft — which means they have the possibility of reading the entire Internet in a few weeks to months to learn what they know, and not the few tens to hundreds of millenia it would take a human.

Unfortunately, the method by which they acquire information and knowledge, is sufficiently inefficient that they actually need to read the entire Internet to reach the skill level of someone who has only just graduated.

This means I'm quite happy to *simultaneously* call them extremely useful, even "artificial general intelligence", and yet also agree with anyone who calls them "very very stupid".

If we actually knew how our brains did this inteligence thing, we could probably make AI genuinely smart as well as absurdly fast.

Their point wasn’t that it’s not useful. It’s that it isn’t artificial intelligence like the masses consider the term.

You wouldn’t say Intellisense isn’t useful but you also wouldn’t call it “AI”. And what LLMs are like is basically Intellisense on steroids (probably more like a cocktail of speed and acid, but you get my point)

Historically, there's been some discussion about that:

https://en.wikipedia.org/wiki/Chinese_room

It matters if we are making a distinction between essence and output.

On the output side, it functionally doesn’t really have a difference. At least in terms of more abstract things like writing code. Although I would argue that the output AI makes still doesn’t match the complexity and nuance of an individual human being, though, and may never do so, simply because the AI is simulating embodiment and existing in the world. It might need to simulate an Earth equivalent to truly simulate a human’s personal output.

In the essence side, it’s much more of a clear distinction. We have numerous ways of determining if a thing is human or not - biology, for one. It would take some serious sci-fi until we get to the point where an android is indistinguishable from a human on the cellular level.

LLMs can't write code.

They don't have capacity to understand logical or temporal relationships, which is the core competency of coding.

They can form syntactically valid strings in a formal language, which isn't the same thing as coding.

How could you think this hard and be so far off. Google is in a hyper strong position here and I don’t even like them.

They can refine grounded results over time and begin serving up increasingly well reasoned results over time as Models improve cost effectively. Then that drives better vectors for ads.

Like what about this is hard to understand?

Most folks just want confirmation. They don't want to have their views/opinions changed. LLM are good at trying to give folks what they're looking for.

Repent.

You are not there to "love what gives you the kicks". That's a kind of love that should not exit the bedroom (better, the bathroom).

> Most people use visual body cues to determine whether someone is lacking in confidence in their answer

Do they?

And a rational interlocutor, besides conversational shortcuts, replies in the form "The best estimations from sources like S0, S1 and S2, publish a value between V1 and V2".

The one that appears when you search with Google.