Cursor IDE support hallucinates lockout policy, causes user cancellations

Validity is not soundness. Wonder why people are just beginning to realize what logicians have been studying for more than a century. This goes to show that most programming was never based on logic but vibes. People have been vibe coding with themselves before AI became prominent.

For this to be true everyone must be logically on the same page. They must share the same axioms. Everyone must be operating off the same data and must not make mistakes or have bias evaluating it. Otherwise inevitably sometimes people will arrive at conflicting truths.

In reality it’s messy and not possible with 100% certainty to discern falsehoods and truthoods. Our scientific method does a pretty good job. But it’s not perfect.

You can’t retcon reality and say “well retrospectively we know what happened and one side was just wrong”. That’s called history. It’s not useful or practical working definition of truth when trying to evaluate your possible actions (individually, communally, socially, etc) and make a decision in the moment.

I don’t think it’s accurate to say that we want to redefine truth. I think more accurately truth has inconvenient limitations and it’s arguably really nice most of the time to ignore them.

As a digression, the definition of knowledge as justified true belief runs into the Gettier problems:

    > Smith [...] has a justified belief that "Jones owns a Ford". Smith 
    > therefore (justifiably) concludes [...] that "Jones owns a Ford, or Brown 
    > is in Barcelona", even though Smith has no information whatsoever about 
    > the location of Brown. In fact, Jones does not own a Ford, but by sheer 
    > coincidence, Brown really is in Barcelona. Again, Smith had a belief that
    > was true and justified, but not knowledge.

   > Imagine that we are seeking water on a hot day. We suddenly see water, or so we 
   > think. In fact, we are not seeing water but a mirage, but when we reach the 
   > spot, we are lucky and find water right there under a rock. Can we say that we 
   > had genuine knowledge of water? The answer seems to be negative, for we were 
   > just lucky. 

I don’t think it’s so clear cut… Even the most adamant “facts are immutable” person can agree that we’ve had trouble “fact checking” social media objectively. Fluoride is healthy, meta analysis of the facts reveals fluoride may be unhealthy. The truth of the matter is by and large what’s socially cohesive for doctors’ and dentists’ narrative, that “fluoride is fine any argument to the contrary—even the published meta-analysis—is politically motivated nonsense”.

> Sounds very human

well yes, of course it does, that article goes out of its way to anthropomorphize LLMs, while providing very little substance

Isn't the point of computers to have machines that improve on default human weaknesses, not just reproduce them at scale?

Exactly, most of us behave in almost the same as AI does. We finally have a mirror to reflect upon.

The other very human thing to do is invent disciplines of thought so that we don't just constantly spew bullshit all the time. For example you could have a discipline about "pursuit of facts" which means that before you say something you mentally check yourself and make sure it's actually factually correct. This is how large portions of the populace avoid walking around spewing made up facts and bullshit. In our rush to anthropomorphize ML systems we often forget that there are a lot of disciplines that humans are painstakingly taught from birth and those disciplines often give rise to behaviors that the ML-based system is incapable of like saying "I don't know the answer to that" or "I think that might be an unanswerable question."

In a way, the main problem with LLMs isn't that they are wrong sometimes. We humans are used to that. We encounter people who are professionally wrong all the time. Politicians, con-men, scammers, even people who are just honestly wrong. We have evaluation metrics for those things. Those metrics are flawed because there are humans on the other end intelligently gaming those too, but generally speaking we're all at least trying.

LLMs don't fit those signals properly. They always sound like an intelligent person who knows what they are talking about, even when spewing absolute garbage. Even very intelligent people, even very intelligent people in the field of AI research are routinely bamboozled by the sheer swaggering confidence these models convey in their own results.

My personal opinion is that any AI researcher who was shocked by the paper lynguist mentioned ought to be ashamed of themselves and their credulity. That was all obvious to me; I couldn't have told you the exact mechanism the arithmetic was being performed (though what is was doing was well in the realm of what I would have expected from a linguistic AI trying to do math), but the fact that its chain of reasoning bore no particular resemblance to how it drew its conclusions was always obvious. A neural net has no introspection on itself. It doesn't have any idea "why" it is doing what it is doing. It can't. There's no mechanism for that to even exist. We humans are not directly introspecting our own neural nets, we're building models of our own behavior and then consulting the models, and anyone with any practice doing that should be well aware of how those models can still completely fail to predict reality!

Does that mean the chain of reasoning is "false"? How do we account for it improving performance on certain tasks then? No. It means that it is occurring at a higher level and a different level. It is quite like humans imputing reasons to their gut impulses. With training, combining gut impulses with careful reasoning is actually a very, very potent way to solve problems. The reasoning system needs training or it flies around like an unconstrained fire hose uncontrollably spraying everything around, but brought under control it is the most powerful system we know. But the models should always have been read as providing a rationalization rather than an explanation of something they couldn't possibly have been explaining. I'm also not convinced the models have that "training" either, nor is it obvious to me how to give it to them.

(You can't just prompt it into a human, it's going to be more complicated than just telling a model to "be carefully rational". Intensive and careful RHLF is a bare minimum, but finding humans who can get it right will itself be a challenge, and it's possible that what we're looking for simply doesn't exist in the bias-set of the LLM technology, which is my base case at this point.)

I think it works until it doesn't. The nature of technical debt of this kind means you can sort of coast on things until the complexity of the system reaches such a level that it's effectively painted into a corner, and nothing but a massive teardown will do as a fix.

Starting with "On" is cooler in philosophical tradition, though, starting in classical and medieval times, e.g. On Interpretation, On the Heavens, etc by Aristotle, De Veritate, De Malo, etc. by Aquinas. Capital is actually "Das Kapital", too

> You're not supposed to trust the tool, you're supposed to review and rework the code before submitting for external review.

It sounds like the guys in this article should not have trusted AI to go fully open loop on their customer support system. That should be well understood by all "customers" of AI. You can't trust it to do anything correctly without human feedback/review and human quality control.

> You're not supposed to trust the tool

This is just an incredible statement. I can't think of another development tool we'd say this about. I'm not saying you're wrong, or that it's wrong to have tools we can't just, just... wow... what a sea change.

1) Once you get it to output something you like, do you check all the lines it changed? Is there a threshold after which you just... hope?

2) No matter what the learning curve, you're using a statistical tool that outputs in probabilities. If that's fine for your workflow/company, go for it. It's just not what a lot of developers are okay with.

Of course it's a spectrum with the AI deniers in one corner and the vibe coders in the other. I personally won't be relying 100% on a tool and letting my own critical thinking atrophy, which seems to be happening, considering recent studies posted here.

> You're not supposed to trust the tool, you're supposed to review and rework the code before submitting for external review.

Then it's not a useful tool, and I will decline to waste time on it.

> But I need to iterate a few times so that the code looks like what I want.

The LLM too. You can get a pretty big improvement by telling the LLM to "iterate 4 times on whichever code I want you to generate, but only show me the final iteration, and then continue as expected".

I personally just inject the request for 4 iterations into the system prompt.

If i dont trust my tool, i would never use it, or use something else better

> You're not supposed to trust the tool, you're supposed to review and rework the code before submitting for external review.

The vibe coding guy said to forget the code exists and give in to vibes, letting the AI 'take care' of things. Review and rework sounds more like 'work' and less like 'vibe'.

/s

It's wild how people write without version control... Maybe I'm missing something.

Thankfully modern source control doesn't reuse user-supplied filenames for its internals. In the dark ages, I destroyed more than one checkout using commands of the form:

  find -name '*somepattern*' -exec clobbering command ...

It's not a big deal in the sense that it's easily reversed, but it is a big deal in that it means the tool is unpredictably unhelpful. Of the properties that good tools in my workflow possess, "unpredictably unhelpful" does not make the top 100.

When a tool starts confidently inserting random wrong code into my 100% correct code, there's not much more I need to see to know it's not a tool for me. That's less like a tool and more like a vandal. That's not something I need in my toolbox, and I'm certainly not going to replace my other tools with it.

https://dwheeler.com/essays/apple-goto-fail.html

This is not the first time that Apple has released a terrible UX that very few users liked, and it certainly wont be the last.

I don’t necessarily agree with the post you’re responding to, but what I will give Apple credit for is making their AI offering unobtrusive.

I tried it, found it unwanted and promptly shut it off. I have not had to think about it again.

Contrast that with Microsoft Windows, or Google - both shoehorning their AI offering into as many facets of their products as possible, not only forcing their use, but in most cases actively degrading the functionality of the product in favor of this required AI functionality.

    > Apple made an out of character misstep by releasing a terrible UX to everyone

Remember „You are a bad user, I am a good bing“? Apple is just slower in fixing and improving things.

Fast!? They were two years slow and still fell face flat, and then rolled back the software

Apple made a huge mistake by keeping their commitment to "local first" in the age of AI.

The models and devices just aren't quite there yet.

Once Google gets its shit together and starts deploying (cloud--based) AI features to Android devices en masse, Apple is going to have a really big problem on their hands.

Most users say that they want privacy, but if privacy comes in the way of features or UX, they choose the latter. Successful privacy-respecting companies (Apple, Signal) usually understand this, it's why they're successful, but I think Apple definitely chose the wrong tradeoff here.

> we're going to have to accept that everything everywhere else is just going to be awful. Hallucinations in your doctor's notes, legal rulings, in your coffee and laundry and everything else that hasn't yet been IoT-ified.

I installed a logitech mouse driver (sigh) the other day, and in addition to being obtrusive and horrible to use, it jams an LLM into the UI, for some reason.

AI has reached crapware status in record time.

> Hallucinations in your doctor's notes, legal rulings, in your coffee

"OK Replicator, make me one espresso with creamer"

"Making one espresso with LSD"

"all that's left to jump on at the moment is AI" -> No, it's the effective applications of AI. It's unprecedented.

I was in the VC space for a while previously, most pitch decks claimed to be using AI: But doing even the briefest of DD - it was generally BS. Now it's real.

With respect to everything being awful: One might say that's always been the case. However, now there's a chance (and requirement) to build in place safeguards/checks/evals and massively improve both speed and quality of services through AI.

Don't judge for the problems: Look at the exponential curve, think about how to solve the problems. Otherwise, you will get left behind.

They had an opportunity to actually adapt, to embrace getting rapid feedback/iterating: But they are not equipped for it culturally. Major lost opportunity as it could have been a driver of internal change.

I'm certain they'll get it right soon enough though. People were writing off Google in terms of AI until this year.. and oh how attitudes have changed.

They also have text thread and email summaries. I still think it counts as a slow rollout.

It’s not just you. iOS auto correct has gotten damn near malicious. E seen it insert entire words out of nowhere

I don't know Scala. I asked cursor to create a tutorial for me to learn Scala. It created two files for me, Basic.scala and Advanced.scala. The second one didn't compile and no matter how often I tried to paste the error logs into the chat, it couldn't fix the actual error and just made up something different.

Yeah, everyone wanted a thinking machine, but the best we can do right now is a dreaming machine... And dreams don't have to make sense.

Developer on LSD is likely to hallucinate less in terms of how weird the LLM hallucinations are sometimes. Besides I know people, not myself, who fare very well on LSD and particularly when micro dosing Adderal style

Granted the Scala language is much more complex than Go. To produce something useful it must be capable of an equivalent of parsing the AST.

> Yes, most people who have an incentive in pushing AI say that hallucinations aren't a problem, since humans aren't correct all the time.

Humans often make factual errors, but there's a difference between having a process to validate claims against external reality, and occasionally getting it wrong, and having no such process, with all output being the product of internal statistical inference.

The LLM is engaging in the same process in all cases. We're only calling it a "hallucination" when its output isn't consistent with our external expectations, but if we regard "hallucination" as referring to any situation where the output for a wholly endogenous process is mistaken for externally validated information, then LLMs are only ever hallucinating, and are just designed in such a way that what they hallucinate has a greater than chance likelihood of representing some external reality.

> Yes, most people who have an incentive in pushing AI say that hallucinations aren't a problem, since humans aren't correct all the time.

We have legal and social mechanisms in place for the way humans are incorrect. LLMs are incorrect in new ways that our legal and social systems are less prepared to handle.

If a support human lies about a change to policy, the human is fired and management communicates about the rogue actor, the unchanged policy, and how the issue has been handled.

How do you address an AI doing the same thing without removing the AI from your support system?

Absolutely, but at least you have some lines of defence while with real world info you have nothing. And the most offending stuff like importing a package that doesn't exist or using a function that doesn't exist does get caught and can be auto fixed.

Yes and it irritates the hell out of me. Cursor support is garbage, but issues with billing and other things are so much worse.

The team I work with it took nearly 3 months to get basic questions answered correctly when it came to a sales contract. They never gave our Sec team acceptable answers around privacy and security.

I've always wondered how Reddit can make money from these companies. I agree they are literally everywhere, even in non-company specific but generic subreddits where if it's big enough you might have multiple shadow marketing firms competing to push their products (e.g. AI, movies, food, porn etc).

Reddit is free to play for marketing firms. Perhaps they could add extra statistics, analytics, promotions for these commercial users.

Isn't the real scary thing here that the AI agent is empowered to control your life?

You're imagining that if you get the answer you want from the AI you hang up the phone, and if you don't you're imagining a human will pick up and have the political power and will to overrule the AI. I think it's more realistic is the way things have played out here: nobody took or had any responsibility because "they made the AI responsible" and second-guessing that choice isn't second-guessing the AI, it's second-guessing the human leaders who decreed that human support had no value. This with the result that the humans would let the AI go about as far as setting fire to the building before some kind of human element imbued with any real accountability steps in. The evidence of this is all the ignored requests presented here.

When admitting fault with your a PR hat on after pissing off a decent(?) number of your paying customers, you're supposed to fully fall on your own sword, not assign blame to factors outside of your control.

Instead of saying "race condition that appears on very slow internet connections", you might say "race condition caused by real-world network latencies that our in-office testing didn't reveal" or some shit.

I don't think you are being transparent.

Like it all sounds like a business decision (limiting 1 device to 1 sub) which is actually that was confirmed both by the actual tech limitation (logging out users from their other devices) and your own support.

Blaming the AI, then the user connection, and then some odd race conditions seem unnecessary. You can just say it was a bad business decision, roll it back, and go on with your day. Instead of treating multiple paying users badly.

It's weirdly common at young startups. Interviewed at two places in the past few weeks where cursor was required. Funnily enough the reason for hiring was because they needed people to fix things ...

Why is Ultralytics yolo famous for it?

These bros are getting high on their own supply. I vibe, I code, but I don't do VibeOps. We aren't ready.

VibeSupport bots, how well did that work out for Canada Air?

https://thehill.com/business/4476307-air-canada-must-pay-ref...

"It's evolving, but backwards."

I worry that they don't understand the limitations of their own product.

100% for first line support is very common. It was common years ago before ChatGPT and ChatGPT made it so much better than before.

In my products, if a user has payed me, their support tickets get high priority, and I get notified immediately.

Other tickets get replied within the day.

I am also running it by myself; I wonder why big companies with 50+ employees like cursor cheaps out with support.

You're giving the game away when you talk about the joy LLMs are robbing from you. I think we all intuit why people don't like the idea of big parts of their jobs being automated away! But that's not an argument on the merits. Our entire field is premised on automating people's jobs away, so it's always a little rich to hear programmers kvetching about it being done to them.

The parts worth thinking about you still think about. The parts that you’ve done a million times before you delegate so you can spend better and greater effort on the parts worth thinking about.

I'm confused when people say that LLMs take away the fun or creativity of programming. LLMs are only really good at the tedious parts.

For me it's typically wrong not in a fundamental way but a trivial way like bad import paths or function calls, like if I forgot to give it relevant context.

And yet the time it takes me to use the LLM and correct its output is usually faster than not using it at all.

Over time I've developed a good sense for what tasks it succeeds at (or is only trivially wrong) and what tasks it's just not up for.

Terrible analogy. I don't commit jank. If the LLM comes out with nonsense, I'll fix it first.

> Do you double check that the LLM hasn't magically recreated someone else's copyrighted code?

I frankly do not care, and I expect LLMs to become such ubiquitous table-stakes that I don't think anyone will really care in the long run.

Add "Without compromising quality then"!

In my context, the scale is small. It just passed the threshold where a naive implementation would be just fine.

> What would happen if the "AI" and web search didn't return anything? Would you have stuck with your implementation?

I was fairly certain there must exist some type of algorithm exactly for this purpose. I would have been flabbergasted if I couldn’t find something on the web. But it that failed, I would have asked friends and cracked open the algorithms textbooks.

> I'm not against use of "AI". But this increasing refusal of those who aspire to work in specialist domains like software development to systematically learn things is not great. That's just compounding on an already diminished capacity to process information skillfully.

I understand what you mean, and agree with you. I can also assure you that that is not how I use it.

There is a time and a place for everything. Software development is often about compromise and often it isn’t feasible to work out a solution from foundational principles and a comprehensive understanding of the domain.

Many developers use libraries effectively without knowing every time consideration of O(n) comes into play.

Competently implemented, in the right context, LLMs can be an effective form of abstraction.

Extremely likely, yes. In this case, since it was an unknown unknown at the time, the LLM nicely explaining that this class of algorithms exists was nice, then I could immediately switch to Wikipedia to learn more (and be sure of the underlying information)

I think of LLMs as an autocomplete of the web plus hallucinations. Sometimes it’s faster to use the LLM initially rather than scour through a bunch of sites first.

Nope. And I don’t need to. That is the beauty of abstractions and information hiding.

Just read the docs and assume the library works as promised.

To clarify, the LLM did not tell me about the specific library I used. I found it the old fashioned way.

I think it's best if we all keep the hours from ~10pm to the morning sacred. Even if we are all up coding, the _reason_ I'm up coding at that hour is because no one is pinging me

Computers don't need AI help to have trouble finding the printer, lol.

The definition of code quality is irrelevant to my argument as both human and AI written code are held to the same standard by the same measure (however arbitrary that measure is). 100 units of something vs 99 units of something is a 1 unit difference regardless of what the unit is.

By the time the AI is actually writing code, I've already had it do a robust architecture evaluation and review which it documents in a development plan. I review that development plan just like I'd review another engineers dev plan. It's pretty hard for it to write objectively bad code after that step.

Also, my day to day work is in an existing code base. Nearly every feature I build has existing patterns or reference code. LLMs do extremely well when you tell them "Build X feature. [some class] provides a similar implementation. Review that before starting." If I think something needs to be DRY'd up or refactored, I ask it to do that.

Code that you can understand and fix later, is acceptable quality per my definition.

Either way, LLMs are actually high up the quality spectrum as they generate a very consistent style of code for everyone. Which gives it uniformity, that is good when other developers have to read and troubleshoot code.

No, I get to spend 2 hours working with LLMs, and then spend the rest of the day doing whatever I please. Repeat.

One reason is lazy one liners are allowed and have high cost/benefit for shitposters and attract many upvotes, so this gets voted highly setting the tone for flamewars in the thread.

It’s miles better on HN. Most bad responses are penalized. The culture is upvoting things that are contributing. I frequently upvote responses that disagree with me. Oftentimes I learn something from it.

think about the demographic that would join a cursor subreddit. Basically 90% superfans. Go against the majority opinion and you'll be nuked

I think it's a combination of oppositional defiant disorder and insufficient moderation.

Paid Copilot is pretty crap too to be honest. I don’t know why it’s so bad.

Hmm. I double checked and I have privacy mode enabled, so I don't think that's the root cause. I also removed all but the bare essential extensions (only the theme I'm using and the core language support extensions for typescript and python).

I prefer Roo, but they're largely the same right now. They each have some features the other doesn't.

They mention an user taking an action will be billed. I guess even sending a message or reacting with an emoji would count as taking an action ? Even logging in ?

History shows that metered plans are extremely unpopular whether it be cell phone service, video, music etc.

> but 1 credit could be 200k+ context each

There is a thread on Cursor forums where the context is around 20K to 30K tokens.

Yes and no.

In a corporate environment, compliance needs are far more important than some trivial cost.

Embrace, extend, extinguish.