Thoughts on a Month with Devin

Thanks - this all makes sense - I still don't feel like this would constitute a massive productivity boost in most cases, since it's not fixing time consuming major issues. But I can see how it's nice to have.

Is that true? I'd like to think my commits are less burdensome to review than a fresh out of boot camp junior dev especially if all that's being done is fixing linter issues. Perhaps there's a small benefit, but doesn't seem like a major productivity boost.

> I have the automatic suggestions turned off. I use a keybind to activate it when I want it.

I didn't realise you could do that. Might give it another go.

> Copilot uses "all of github" via training, and your current project in the context.

The current project context is the bit I didn't think it had. Thanks!

Cursor requires you to use their specific IDE though, doesn't it? With Copilot I don't have to switch contexts as it lives in my Jetbrains IDE.

I think a lot of people who are enabling copilot in vs code (like I did a few days ago), are experiencing "suggested autocomplete as I type" for the first time where before there was no grey text below what I am writing personally.

It is a huge distraction, especially if it changes as I write more. I turned it off almost immediately.

I deeply regret turning on copilot in vscode. It (M$) immediately weaseled into so many places and settings. I'm still trying to scaled it back. Super annoying and distracting. I'd prefer a much more opt in for each feature than what they did.

I think it's qualitatively different this time.

Unlike with offshoring, this is a technological solution, which understandably is received more enthusiastically on HN. I get it. It's interesting as tech! And it's achieved remarkable things. But unlike with offshoring (which is a people thing) or magical NOCODE/CASE/etc "solutions", it seems the consensus is that AI coding assistants will eventually get there. At least a portion of even HN seems to think so. And some are cheering!

The coping mechanism seems to be "it won't happen to me" or "my knowledge is too specialized" but I think this will become increasingly false. And even if your knoweldge is too specialized to be replaced by AI, most engineers aren't like that. "Well, become more specialized" is unrealistic advice, and in any case, the employment pool will shrink.

PS: I am offhsoring (in a way). I'm not based in the US but I work remotely for a US company.

> Ok then why does nearly every company make people write code for interviews or do take home programming projects?

For the same reason they put leetcode problems to "test" an applicants skill. Or have them write mergesort on a chalkboard by hand. It gives them a warm fuzzy feeling in the tummy because now they can say "we did something to check they are competent". Why, you ask? Well it's mostly impossible to come up with a test to verify a competency you don't have yourself. Imagine you can't distinguish red and green, are not aware of it, but want to hire people who can. That's their situation, but they cannot admit it - because it would be clear evidence that they are no good fit for their current role. Use this information responsibly ;)

> Why do people list programming languages on their resumes if it's "least important"?

You put the programming languages in there alongside the HR-soothing stuff because you hope that an actual software person gets to see your resume and gives you an extra vote for being a good match. Notice that most guides recommend a relatively small amount of technical content vs. lots of "using my awesomeness i managed to blafoo the dingleberries in a more efficient manner to earn the company a higher bottom line"

If you don't want to be a software developer that's fine. But your questions point me towards the conclusion that you don't know a lot of things about software development in the first place which doesn't speak for your ability to estimate how easy it will be to automate it using LLMs.

> If anything, CS degrees have proven time and time again they aren't translatable into software development (which is why there's an entire degree field called Software Engineering emerging).

Emerging? I graduated in 2006 with a BEng in Software Engineering.

The difference between it and the BSc CompSci degree I started in, was that optional modules became mandatory — including an industrial placement year (paid internship).

> Software engineer{ing,s} (project construction, integrations, scaling, organizational/external factors, etc) will stick around for along time.

My gut disagrees, because LLMs are at about the same level in those things as they are in low level coding: not yet replacing humans in project level tasks any more than they do in coding tasks, but also being OK assistants for both coding and project domains. I have no reason to think either has less or more opportunity for self-training, so I expect progress to track for the foreseeable future.

(That said, the foreseeable future in this case is 1-2 years).

> the CS concepts are very easy for LLMs to recall

They're easy to recall, but you have to know what to recall in the first place. Or even know enough of the territory to realise there's something to recall. Without enough background, you'll get a whole set of amazing tools that you have no idea what to do with.

For example, you may be able to write a long description of your problem with some ideas how to steer the AI to give you possible solutions. And the AI may figure out what the problem is and that the hyperloglog is something that could be useful to you. And you may have the awesome programming skills to implement that. But that's a lot of maybes. It would be much faster/easier if you knew about hyperloglog ahead of time and just asked for the implementation or library recommendation.

Or even if you don't know about the actual solution, you'd have enough of CS vocabulary to ask: "how do I get a fast, approximate distinct count from a multiset". It would take a long imprecise description to get the same thing for a coder with no theory background.

I'm not convinced an LLM is really "recalling" any CS concepts when they try to solve a problem. IMHO, we're lucky if it matches the pattern of the request against the pattern of a solution and the two are actually related. I'm no expert but I don't think there's any reason to think that an LLM is taking a CS concept and applying it to something novel in order to get a solution. If they were, I believe their success rate would be much higher.

In many places where someone might reach for something they remember from their CS coursework, there's often an open-source library or tool doing much the same thing. Understanding how these libraries and tools function is certainly valuable but, much of the time, people can get by with only a vague hunch; indeed, this is why they exist! IMHO, I would be happier with the LLM assistant if it picked reliable library code rather than writing up a sketchy solution of its own.

I'm also familiar with this idea people who have managed to be successful in the field without a CS degree are more capable. In my opinion, this is hogwash. I think if we take a step back, we'll see that people graduating from established, top-tier CS programs are looking for higher pay than those who have come from a less expensive and (very often) business focused program. To be fair, people from each of these backgrounds has their strengths; in many organizations a developer who has done two semesters of accounting is a real benefit, in others the ability to turn a research paper into the prototype of a new product is going to be important.

Years of experience often washes out much of these differences. People who have started from business oriented education programs may end up taking a handful of CS courses as they navigate their career, likewise many people with a CS background end up accruing business centered skills.

In my opinion, people start out their education at a place that they can afford, a place that is familiar to them, often a place that they feel comfortable. Someone's economic background (and of their family) plays a big role on what kind of educational program they choose when they are on the cusp of adulthood. Smart and talented people can always learn what they need to learn, regardless of their starting point.

I think honestly the meme that non-CS degree engineers are most capable is selection bias.

If they had taken a CS degree they would likely be just as, of not more capable.

To self-learn the topics you need to make good software takes an immense amount of effort and although the data and material is out there, takes a lot of work to figure out.

I'm only recently starting to pick up on "magic" patterns that are actually extremely simple to understand given the right base knowledge... I can gain tons of insights from talks givem in the early 2010s but if I watched them without the correct practical experience and foundational knowledge it is the same as the title to a HN post this week[1], gibberish.

With the correct time playing with the foundational patterns and learning some of the backing knowledge it unlocks amazing patterns in my mind and makes the magic seem simple. A great example, CSP[2]. I've known about and used the actor model before, which I first discovered when I found Erlang, but now with CSP I could ask the question "Why should actors be heavy?", you can put an actor into a light-weight task and spawn tons of them and build a tree of connections. Stuff like oneTBB flow graph[3]now makes sense and looks like a beautiful pattern with some really interesting ideas that can be implemented in more general computing than the high performance computing it was designed for. It seems niche but golang is built on those foundations, and the true power of concurrency in golang comes from embracing that. It fundamentally changes the way I want to structure and layout code and I feel like a good CS course can get you there quicker...

Unfortunately a good CS course probably wouldn't accelerate the average CS grads understanding of that but can get someone dedicated and hungry there much much quicker. Someone fresh out of a JS bootcamp is maybe a decade away from that if they ever even want to search for that knowledge.

1. https://news.ycombinator.com/item?id=42711751

2. https://en.m.wikipedia.org/wiki/Communicating_sequential_pro...

3. https://oneapi-spec.uxlfoundation.org/specifications/oneapi/...

he’s certainly sure, but lord knows if it is

So when people say these things are like junior developers, they really mean that they’re like the worst _stereotype_ of junior developers, then?

How would it decide if it needs a reality check? Would the thing checking have the same limitations?

Maybe. It's pretty weird and I'm still thinking about it.

You can't throw junior engineers working on an issue under the bus when they clearly can't do that. Or at least it takes some effort. In return you may coach them and hope they eventually improves.

Devin does look like junior engineers, but I've learned to just click "Terminate Session" immediately after I spotted that it was doing something hopeless. I've managed to get some real work done out of it, without much effort on my side (just check what it's doing every 10~15 minutes and type a few lines or restart session).

It stands for Monte Carlo search tree.

Ie. Better outputs from models, not external tooling and prompt engineering.

https://github.com/zz1358m/MCTS-AHD-master

> You earn the right to take on bigger challenges and no one in this space has earned the right yet including us.

Not entirely. We're in interesting times where products with better models can suddenly leapfrog and displace even current upstarts. Cursor won over Copilot from leveraging Claude Sonnet 3.5. They didn't "earn the right".

Improvements with models will help those with the existing infrastructure that can benefit from it. I'm not saying Devin will win when that time comes, but a similar product might find their space quickly.

I can relate, I have been genuinely amazed more than once by how it could "understand" some very complex code nobody dared to touch like you mention.

Can Aider sit inside VS Code, understand what files I have open, and use them as context? Their docs lead me to say no, that they are an inline chat/completion experience