Claude can now search the web

https://danstroot.imgix.net/assets/blog/img/comic2.png?auto=...

The longer I've been in the workforce, the more I realize most humans actually kind of suck at their jobs. LLMs being more human like is the opposite of what I want.

Splitting hairs, but LLMs themselves don’t search.

LLMs themselves don’t choose the top X.

That’s all regular flows written by humans run via tool calls after the intent of your message has been funneled into one of a few pre-defined intents.

It would probably be really great for web searching llms to let you calibrate how they should look for info by letting you do a small demonstration of how you would pick options yourself, then storing that preference feedback in your profile’s system prompt somehow.

Here though they're not replacing a random person, they're replacing _you_ (doing the search yourself). _You_ wouldn't look at the top X hits then assume it's the correct answer.

Bold of you to assume that most people even bother googling simple questions

Be careful what you call AI, you might just get what you wish for...

Degenerative AI ?

Grok's DeepSearch and DeeperSearch are also pretty good, and you can look at their stream of thought to see how it reaches its results.

Not sure how OpenAIs version works, but grok's approach is to do multiple rounds of searches, each round more specific and informed by previous results

They're probably doing RAG on a huge chunk of the internet, i.e. they built their own task-specific search engine.

I'm glad you mentioned this. I asked Deep Research to lay out a tax strategy in a foreign country and it cited a ton of great research I hadn't yet found.

Yeah, Kagi's search results are so much better than Google's, it defies comparison.

It's neat but I've found the value kinda variable. It seems heavily influenced by whatever the first few hits are for a query based on your question, so if it's the kind of question that can be answered with a simple search it works well. But of course those are the kinds of questions where you need it the least.

I find myself much more often using their "Quick Answer" feature, which shows a brief LLM answer above the results themselves. Makes it easier to see where it's getting things from and whether I need to try the question a different way.

Has anyone compared Perplexity with Kagi Assistant?

I am always looking for Perplexity alternatives. I already pay for Kagi and would be happy to upgrade to the ultimate plan if it truly can replace Perplexity.

Does it just start a search or does the chat continue with the results? Would be cool to continue the chat with result, which were filtered acording to the blacklist.

I have a subscription, please could I ask how you do this? I only know of the append ? Feature.

I have in my prompt for it to always use search, no matter what, and I get pretty decent results. Of course, I also question most of its answers, forcing it to prove to me that its answer is correct.

Just takes some prompt tweaking, redos, and followups.

It's like having a really smart human skim the first page of Google and give me its take, and then I can ask it to do more searches to corroborate what it said.

Try their Deep Research or grok's DeepSearch. Both do many searches and read many articles over a couple of minutes

That is interesting. I have often been amazed at how good it is at picking up when to search vs use its weights. My biggest problem with ChatGPT is the horrendous glitchyness.

> Bard […] it provided two references. Which turned out to be AI generated web pages.

Welcome to the Habsburg Internet.

People outside the US obviously don't exist, therefore the statement is correct.

Many of OpenAI's announcements seem to be timed almost perfectly as responses to other events in the industry or market. I think Sam just likes to keep the company in the news and the cultural zeitgeist, and he doesn't really care if what he's announcing is ready to scale to users yet or not.

> Web search is available now in feature preview for all paid Claude users in the United States.

It is for all paid users, something OpenAI is slow on. I pay for both and I often forget to try OpenAI's new things because they roll out so slow. Sometimes it's same-day but they are all over the map in how long it takes to roll out.

Why? I am not aware of what's wrong with them.

Install MCP plugin and call a search engine of your choice.

If you’re unhappy about something, try to first think of a solution before expressing your discontent.

Exactly. An LLM is not a conventional search engine and shouldn't be prompted as if it were one. The difference between "Rust crate to access Postgres with Arrow support" and "What would a hypothetical Rust crate to access Postgres with Arrow support look like?" isn't that profound from the perspective of a language model. You'll get an answer, but it's entirely possible that you'll get the answer to a question that isn't the one you thought you were asking.

Some people aren't very good at using tools. You can usually identify them without much difficulty, because they're the ones blaming the tools.

It's absolutely how LLMs should work, and IME they do. Why write a full question if a search phrase works just as well? Everything in "Could you recommend xyz to me?" except "xyz" is redundant and only useful when you talk to actual humans with actual social norms to observe. (Sure, there used to be a time when LLMs would give better answers if you were polite to them, but I doubt that matters anymore.) Indeed I've been thinking of codifying this by adding a system prompt that says something like "If the user makes a query that looks like a search phrase, phrase your response non-conversationally as well".

Totally agree here. I tried the following and had a very different experience:

"Answer as if you're a senior software engineer giving advice to a less experienced software engineer. I'm looking for a Rust crate to access PostgreSQL with Apache Arrow support. How should I proceed? What are the pluses and minuses of my various options?"

“Prompting” is kind of a myth honestly.

Think about it, how much marginal influence does it really have if you say OP’s version vs a fully formed sentence? The keywords are what gets it in the area.

It's funny because many people type full sentence questions into search engines too. It's usually a sign of being older and/or not very experienced with computers. One thing about geeks like me is we will always figure out what the bare minimum is (at least for work, I hope everyone has at least a few things they enjoy and don't try to optimise).

Well, compare it to the really good answer from Grok (https://x.com/i/grok/share/MMGiwgwSlEhGP6BJzKdtYQaXD) for the same prompt. Also, framing as a question still pointed to the non-existent postgres-arrow with Claude.

That's primarily how i do, though it depends on the search ofc. I use Kagi, though.

I've not yet found much value in the LLM itself. Facts/math/etc are too likely incorrect, i need them to make some attempt at hydrating real information into the response. And linking sources.

Often times I come up with a prompt, then stick the prompt in an LLM to enhance / identify what I’ve left out, then finally actually execute the prompt.

Cite things from ID based specs. You’re facing a skill issue. The reason most people don’t see it as such is because an LLM doesn’t just “fail to run” here. If this was code you wrote in a compiled language, would you post and say the language infuriates you because it won’t compile your syntax errors? As this kind of dev style becomes prevalent and output expectation adjust, work performance review won’t care that you’re mad. So my advice is:

1. Treat it like regular software dev where you define tasks with ID prefixes for everything, acceptance criteria, exceptions. Ask LLM to reference them in code right before impl code

2. “Debug” by asking the LLM to self reflect on its decision making process that caused the issue - this can give you useful heuristics o use later to further reduce the issues you mentioned.

“It” happening is a result of your lack of time investment into systematically addressing this.

_You_ should have learned this by now. Complain less, learn more.

Yeah I have to manually hold it down every time I talk. I have a lot of pauses and simply would not be able to interface with that without that option. It’s why I essentially can’t use Gemini voice mode

I assume it would be possible to make yourself with the OpenAI api together with a locally run voice model to only detect the activation word. There might be of the shelf solutions for this, but I am not aware of any.

The AI isn't "reading the web" though, they are reading the top hits on the search results, and are free-riding on the access that Google/Bing gets in order to provide actual user traffic to their sites. Many webmasters specifically opt their pages out of being in the search results (via robots.txt and/or "noindex" directives) when they believe the cost/benefit of the bot traffic isn't worth the user traffic they may get from being in the search results.

One of my websites that gets a decent amount of traffic has pretty close to a 1-1 ratio of Googlebot accesses compared to real user traffic referred from Google. As a webmaster I'm happy with this and continue to allow Google to access the site.

If ChatGPT is giving my website a ratio of 100 bot accesses (or more) compared to 1 actual user sent to my site, I very much should have to right to decline their access.

>You can now use Claude to search the internet to provide more up-to-date and relevant responses.

It's a search engine. You 'ask it to read the web' just like you asked Google to, except Google used to actually give the website traffic.

I appreciate the concept of an AI User-agent, but without a business model that pays for the content creation, this is just going to lead to the death of anonymously accessible content.

You could make this justification for a lot of unapproved bot activity.

Many if not most websites are paid for by eyeballs not by get requests. A bot is a bot is a bot. Respect robots.txt or expect to have your IPs banned.

How can you be so sure? Processors love locality, so they fetch the data around the requested address. Intel even used to give names to that.

So, similarly, LLM companies can see this as a signal to crawl to whole site to add to their training sets and learn from it, if the same URL is hit for a couple of times in a relatively short time period.

> This isn't a vacuum charged with crawling the web, it's an adhoc GET request.

Doesn't matter. The robots-exclusion-standard is not just about webcrawlers. A `robots.txt` can list arbitrary UserAgents.

Of course, an AI with automated websearch could ignore that, as can webcrawlers.

If they chose do that, then at some point, some server admins might, (again, same as with non-compliant webcrawlers), use more drastic measures to reduce the load, by simply blocking these accesses.

For that reason alone, it will pay off to comply with established standards in the long run.

No thank you, when I define a robots.txt file I expect all automated systems to respect it.

Someday I’ll have enough “karma” to downvote things like this.

The agent should respect robots.txt no matter who is using the Robot.

While robots.txt is not there to directly prevent automated requests, it does prevent crawling which is needed for search indices.

Without recursive crawling, it will not possible for a engine to know what are valid urls[1]. They will otherwise either have to brute-force say HEAD calls for all/common string combinations and see if they return 404s or more realistically have to crawl the site to "discover" pages.

The issue of summarizing specific a URL on demand is a different problem[2] and not related to issue at hand of search tools doing crawling at scale and depriving all traffic

Robots.txt does absolutely apply to LLMs engines and search engines equally. All types of engines create indices of some nature (RAG, Inverted Index whatever) by crawling, sometimes LLM enginers have been very aggressive without respecting robots.txt limits, as many webmasters have reported over the last couple of years.

---

[1] Unless published in sitemap.xml of course.

[2] You need to have the unique URL to ask the llm to summarize in the first place, which means you likely visited the page already, while someone sharing a link with you and a tool automatically summarizing the page deprives the webmaster of impressions and thus ad revenue or sales.

This is common usage pattern in messaging apps from Slack to iMessages and been so for a decade or more, also in news aggregators to social media sites, and webmasters have managed to live with this one way or another already.

The ones I noticed in my logfiles behave impeccably: retrieve robots.txt every week or so and act on it.

(I noticed Claude, OpenAI and a couple of others whose names were less familiar to me.)

Apparently they use smart appliances to scrape websites from residential accounts.

https://github.com/ai-robots-txt/ai.robots.txt

Bluesky / ATProto has a proposal for User Intents for data. More semantics than robots.txt, but equally unenforceable. Usage with AI is one of the intents to be signaled by users

https://github.com/bluesky-social/proposals/tree/main/0008-u...

Small difference. Its called llms.txt

https://llmstxt.org/

If they don't comply with robots.txt, why would they comply with anything else?

Every automated crawler follows human-driven actions.

It must form the search index somehow. That is prior the human action. Simply it would not find the page at all if it respects.

So, do you mean LLMs are human-like and conscious?

I thought they were just machine code running on part GPU and part CPU.

if a human triggers the web crawlers by pressing a button, should they ignore robots.txt?

In practice, robots.txt is to control which pages appear in Google results, which is respected as a matter of courtesy, not legality. It doesn't prevent proxies etc. from accessing your sites.

In what circles is it a joke? Google bots seem to respect it on my sites according to logs.

I have replaced all robots.txt rules with simple WAF rules, which are cheaper to maintain than dealing with offending bots.

I’ve been experimenting with different LLM + search combos too, but results have been mixed. One thing I’m particularly interested in is improving retrieval for both images and videos. Right now, most tools seem to rely heavily on metadata or simple embeddings, but I wonder if there’s a better way to handle complex visual queries. Have you tried anything for video search as well, or are you mainly focused on images? Also, what kinds of queries have you tested?

I notice I have been using the Google AI summary more and more for quick things.

I had subscribed to Perplexity for a month to use their deep research. I think it ran out earlier this week but I am really missing it Saturday morning here.

That thing is awesome. Sonnet 3.7 is more in the middle of this to me. It can help me understand all the things I found from my deep research requests.

I am surprised the hype is not more for Sonnet 3.7 honestly.

Pretty much. Claude from their announcements seems to me at least to be about SWE's and coding at the moment. Personally while I understand their decision I find it a bit limiting, and just a little targeted against the SWE profession. If all AI does is disrupt SWE's but not really add new products and/or new possibilities; then it feels IMO like a bit of a waste and is quite uneven in its society disruption.

At least in my circle SWE's are either excited or completely fearful of the new technology; and every other profession feels like it is just hype and hasn't really changed anything. They've tried it sure; but it didn't really have the data to help with even simpler domain's than SWE. Anecdotally I've had the comment from people around me - my easy {insert job here} will last longer than your tech job from many people I know from both white and blue collar workers. Its definitely reduced the respect for SWE's in general at least where I'm located.

I would like to see improvements in people's quality of life and new possibilities/frontiers from the technology, not just "more efficiencies" and disruption. It feels like there's a lack of imagination with the tech.

I would say no. While I pay for chatgpt Claude and perplexity monthly (I don't know why anymore) my wife does not use any at all. She has around 5-10 things she uses on the smartphone, and if she needs something new there is still google.

I on the other side reduced my googling by 95%

Have you actually done any kind of study on the utility the 'average user' has received, or is this just guessing?

That's because the attention mechanism was designed for Seq2Seq models (i.e. translation in its most general form).

Any other use of it is a case of "I have a hammer, so that's a nail".

That's a great way to use it!