Peer Programming with LLMs, for Senior+ Engineers

> The space is moving so fast that, if I wrote down my workflows and workarounds just two months ago, so much of it would be stale today.

There is also the problem that none of these workflows were validated or verified. Everyone is free to go on social media or personal blogs and advertise their snake oil. Thus in a scenario where these workflows are found to be lacking, the perceived staleness might actually be ineffectiveness beyond self promotion.

Very important comment. My workflow changed dramatically with the increased capabilities of these tools.

Yeah. I have Claude 4 correcting code that Claude 3.5 wrote a few months ago.

My most common use is similar: when I'm working on problems in a somewhat unfamiliar domain finding out what their "terms of art" are. The chances that I've just come up with a completely new concept are pretty low so it's just a matter of helping me formulate my questions in the language of the existing body of knowledge.

You should go a step further and integrate search (tavili, searxng, etc) into your flow. You'll get better results, and you can refine sources and gradually build a scored list of trusted sources.

Lately I look for opportunities to have the LLM do some easy small work while I go work on some harder task in parallel.

I've also tried asking the LLM to come up with a proposed solution while I work on my own implementation at the same time.

LLMs can also be much faster if a task requires some repetitive work. When I recognize a task like that, I try coding the first version and then ask the LLM to follow my pattern for the other areas where I need to repeat the work.

(Disclaimer: I built and sell a product around that workflow)

It often is, if you pick the right tasks (and more tasks fall into that bucket every few weeks).

You can get a simple but fully-working app out of a single prompt, though quality varies widely unless you’re very specific.

Once you have a codebase, agent output quality comes down to architecture and tests.

If you have a scalable architecture with well-separated concerns, a solid integration test harness with examples, and good documentation (features, stack, procedures, design constraints), then getting the exact change you want is a matter of how well you can articulate what you want.

One more asterisk, the development environment has to support the agent: like a human, agents work well with compiler feedback, and better with testing tools and documentation/internet access (yes my agents have these).

I use CheepCode to work on itself, but I am still building up the test library and preview environments to de-risk merging non-trivial PRs that I haven’t pulled down and run locally. I also use it to work on other apps that I'm building, and since those are far more self-contained / easier to test, I get much better results there.

If you want to put less effort into describing what you want, have a chat with an AI to generate tickets. Then paste those tickets into Linear and let CheepCode agents rip through them. I’ve got tooling in the works that will make that much easier, but I can only be in so many places at once as a bootstrapped founder :-)

> is it really more efficient to have an LLM generate code, then review that code, fix errors and spend some time to fully understand it?

The answer is always "it depends". There are some drudge work tasks that are brilliantly done by LLMs, such as generating unit tests or documentation. Often the first attempt is not great, but iterating over them is so fast that you can regenerate everything from scratch a dozen times before you spend as much time as you would do if you wrote them yourself.

It also depends on what scope you're working on. Small iterations have better results than grand redesigns.

Context is also critical. if your codebase is neatly organized with squeaky clean code then LLMs generate better recommendations. If your codebase is a mess with inconsistent styles in spaghetti code then your prompts tend to generate more of the same.

Depends on the code, but often yes. The less you care about that specific result, the more efficient it is. One-off tools under 2k lines where you can easily verify the result? Why would I not generate that and save time for more interesting stuff?

I frequently use it to assemble the boiler plate tedious stuff. E.g. add a new test file and wire up all the build rules etc, then implement tests x y z. Or the other day I had to create an input form in an Angular app and I just asked for the 7 or 8 fields I wanted and it did it and wired it all up for me (both template and Typescript). It can do those sorts of things in a few seconds and saves me maybe only 15-20 minutes but the mental relief of me not having to do it is great, even if the time saving is relatively low.

The other thing is sometimes just writing out method signatures with all the right types and conversions/casts etc between types/interfaces/classes etc when I can't be bothered to do all the look ups myself ("Create a method here called foo that accepts a Bar instance and converts it to Baz. Return type should be Quux - add a conversion from Baz to a new Quux instance before the final return - use the builder pattern and map the constant magic-strings in blah.ts to appropriate enum values in Quux." Etc etc and then I write the logic in the middle) Again not a huge time saving but it mentally lightens the load and keeps you concentrating on the problem rather than the minutia

If you can identify blocks of code you need to write that are easy to define reasonably well, easy to review/verify that it is written correctly but still burdensome to actually write, LLMs are your new best friend. I don't know about how other people think/write, but I seem to have a lot of that kind of stuff on my table. The difficult part to outsource to LLMs is how to connect these easy blocks, but luckily thats the part I find fun in coding, not so much writing the boring simple stuff.

In many cases yes. In many cases no. Overall, it can save you a lot of time even accounting for time wasted when you give it more than it can handle and you give up and do it all yourself.

Yes, but the bar for skepticism is higher than that, because LLMs also compile code and catch errors, and generate and run tests; compile errors and assertion failures are just more prompts to an LLM agent.

Looks like an elevated vibe coding method for UI development. Does this work for non-web/UI development?

This is showing the workflow of your tool quite well, but would be way more convincing & impressive if you had actually fixed the bug and linked to the merged PR.

Yes, although those who are not senior engineers will not preemptively see the value in the documented approaches. One has to be a senior to preemptively appreciate the value in them.

It seems like we need to use forceful language with these things now. I've had copilot censor everything I asked it. Finally I had to to say, "listen you cracked up piece of shit, help me generate a uuid matcher. "

I really want the LLM to do the opposite. To tell me that’s the stupidest fucking thing it’s ever seen. They’re surprisingly bad at that though.

That’s what the recent Copilot feature on GitHub can do. You assign it tasks and it comes back with a PR. You could also assign it to review a PR.

Well put. That’s my challenge too - losing the mental model of my entire codebase. Sometimes it feels like the time I saved using an LLM I then give right back when reassembling the mental model.