Promising results from DeepSeek R1 for code

Glad to hear you’re finding aider useful!

There’s a faq entry about how these stats are computed [0]. Basically using git blame, since aider is tightly integrated with git.

The faq links to the script that computes the stats. It’s not designed to be used on any repo, but you (or aider) could adapt it.

You’re not the first to ask for these stats about your own repo, so I may generalize it at some point.

[0] https://aider.chat/docs/faq.html#how-are-the-aider-wrote-xx-...

Aider has a command to add files to the prompt. For files that are not added, it uses tree-sitter to extract a high-level summary. So for a `.env`, it will mention to the LLM the fact that the file exists, but not what is in it. If the model thinks it needs to see that file, it can request it, at which point you receive a prompt asking whether it's okay to make that file available.

It's a very slick workflow.

You can use an .aiderignore file to ensure aider doesn't use certain files/dirs/etc. It conforms to the .gitignore spec.

[dead]

Every metric has limitations, but git blame line counts seem pretty uncontroversial.

Typical aider changes are not like autocompleting braces or reformatting code. You tell aider what to do in natural language, like a pair programmer. It then modifies one or more files to accomplish that task.

Here's a recent small aider commit, for flavor.

  -# load these from aider/resources/model-settings.yml
  -# use the proper packaging way to locate that file
  -# ai!
  +import importlib.resources
  +
  +# Load model settings from package resource
  MODEL_SETTINGS = []
  +with importlib.resources.open_text("aider.resources", "model-settings.yml") as f:
  +    model_settings_list = yaml.safe_load(f)
  +    for model_settings_dict in model_settings_list:
  +        MODEL_SETTINGS.append(ModelSettings(**model_settings_dict))
  

That's pretty reaching though if you're comparing an AI to a formatter. Presumably 70% of a new Aider release isn't formatting.

"The stats are computed by doing something like git blame on the repo, and counting up who wrote all the new lines of code in each release. Only lines in source code files are counted, not documentation or prompt files."

The problem is it's very expensive. More expensive than Claude.

Only various distillations are available for most people’s hardware, and they’re quite obviously not as good as actual R1 in my testing.

Yesterday, I had it think for 194 seconds. At some point near the end, it said "This is getting frustrating!"

i spent a months salary on these eggs and can no longer afford to cook them :(

The eggs cost more than the laptop...

EXO is also great for running the 6bit deepseek, plus it’s super handy to serve from all your devices simultaneously. If your dev team all has M3 Max 48gb machines, sharing the compute lets you all run bigger models and your tools can point at your local API endpoint to keep configs simple.

Our enterprise internal IT has a low friction way to request a Mac Studio (192GB) for our team and it’s a wonderful central EXO endpoint. (Life saver when we’re generally GPU poor)

Can you link to the model you’re talking about? I can’t find the exact one using your description. Thanks!

If you are specifically running it for coding, I'm satisfied with using it via continue.dev in VS Code. You can download a bunch of models with ollama, configure them into continue, and then there is a drop-down to switch models. I find myself swapping to smaller models for syntax reminders, and larger models for beefier questions.

I only use it for chatting about the code - while this setup also lets the AI edit your code, I don't find the code good enough to risk it. I get more value from reading the thought process, evaluating it, and the cherry picking which bits of its code I really want.

In any case, if that sounds like the experience you want and you already run ollama, you would just need to install the continue.dev VS Code extension, and then go to its settings to configure which models you want in the drop-down.

This model is listed on ollama. The 20GB one is this one: https://ollama.com/library/deepseek-r1:32b-qwen-distill-q4_K...

Search for a GGUF on Hugging Face and look for a "use this model" menu, then click the Ollama option and it should give you something to copy and paste that looks like this:

  ollama run hf.co/MaziyarPanahi/Mistral-7B-Instruct-v0.3-GGUF:IQ1_M

   ollama run deepseek-r1:32b

https://ollama.com/library/deepseek-r1

I'm using it inside of LM Studio (https://lmstudio.ai), which has a "Discovery" tab where you can download models.

A lot of the niceness about DeepSeek-R1's usage in coding is that you can see the thought process, which (IME) has been more useful than the final answer.

It may well be that o1's chain of thought reasoning trace is also quite good. But they hide it as a trade secret and supposedly ban users for trying to access it, so it's hard to know.

I would say worry? Yes. Panic? No.

It's... good. Even the qwen/llama distills are good. I've been running the Llama-70b-distill and it's good enough that it mostly replaces my chatgpt plus plan (not pro - plus).

I think if anything - One of my big takeaways is that OpenAI shot themselves in the foot, big time, by not exposing the COT for the O1 Pro models. I find the <think></think> section of the DeepSeek models to often be more helpful than the actual answer.

For work that's treating the AI as collaborative rather than "employee replacement" the COT output is really valuable. It was a bad move for them to completely hide it from users, especially because they make the user sit there waiting while it generates anyways.

Deepseek is a big deal but we should be happy not worried that our tools are improving.

As far as realizing the prophecy of AI as told by its proponents and investors goes, probably not. LLMs still have not magically transcended their obvious limitations.

However this has huge implications when it comes to the feasibility and spread of the technology, and further implications with regards to economy and geopolitics now that confidence in the American AI sector has been hit and people and organizations internationally have somewhere else to look for.

edit: That being said, this is the first time I've seen a LLM do a better job than even a senior expert could do, and even if it's on small scope/in a limited context, it's becoming clear that developers are going to have to adopt this tech in order to stay competitive.

There are two things. First, deepseek v3 and r1 are both amazing models.

Second, the fact that deepseek was able to pull this off with such modest resources is an indication that there is no moat, and you might wake up tomorrow and find an even better model from a company you have never heard of.

Yeah, it is definitely a big deal.

I expect it will be a net positive: they proved that you can both train and run inference against powerful models for way less compute than people had previously expected - and they published enough details that other AI labs are already starting to replicate their results.

I think this will mean cheaper, faster, and better models.

This FAQ about it is very good: https://stratechery.com/2025/deepseek-faq/

End result is on par with o1 preview, which is ironically more intelligent than o1, but the intermediate tokens are actually useful. I've got it running locally last night and out of 50 questions so far I've gotten the answer in the chain of thought in more than half.

Today it is. Tomorrow everyone will look at it like Wish or Temu.

What do you use it at home for?

deepseek fine-tuned qwen32b with data generated by deepseek671b

Keep in mind, two of the functions were translated, and the third was created from scratch. Quoting from the FAQ on the Gist [1]:

Q: "It only does conversion ARM NEON --> WASM SIMD, or it can invent new WASM SIMD code from scratch?"

A: "It can do both. For qX_0 I asked it to convert, and for qX_K I asked it to invent new code."

* [1]: https://gist.github.com/ngxson/307140d24d80748bd683b396ba13b...

Porting well written code if you know the target language well is pretty fun and fast in my experience. Often when there are library, API, or language feature differences, these are better considered outside of most work it would take to fully describe the entire context to a model is what has happened in my experience, however.

That is what I am struggling to understand about the hype. I regularly use them to generate new simd. Other than a few edge cases (issues around handling of nan values, order of argument for corresponding ops, availability of new avx512f intrinsics), they are pretty good at converting. The names of very intrinsics are very similar from simd to another. The very self-explanatory nature of the intrinsics names and having similar apis from simd to another makes this somewhat expected result given what they can already accomplish.

Deepseek r1 is not exactly better than the alternatives. It is, however, open as in open weight and requires much less resources. This is what’s disruptive about it.

Interesting that both de-novo and porting seems to have worked.

I do not understand why GGML is written this way, though. So much duplication, one variant per instruction set. Our Gemma.cpp only requires a single backend written using Highway's portable intrinsics, and last I checked for decode on SKX+Zen4, is also faster.

This depends pretty heavily on the repo.

> That's why I'm not worried. There is already SO MUCH more demand for code than we're able to keep up with. Show me a company that doesn't have a backlog a mile long where most of the internal conversations are about how to prioritize what to build next.

I worry about junior developers. It will be a while before vocational programming courses retool to teach this new way of writing code, and these are going to be testing times for so many of them. If you ask me why this will take time, my argument is that effectively wielding an LLM for coding requires broad knowledge. For example, if you're writing web apps, you need to be able to spot say security issues. And various other best practices, depending on what you're making.

It's a difficult problem to solve, requiring new sets of books, courses etc.

We have already entered a new paradigm of software development, where small teams build software for themselves to solve their own problems rather than making software to sell to people. I think selling software will get harder in the future unless it comes with special affordances.

> There is already SO MUCH more demand for code than we're able to keep up with. Show me a company that doesn't have a backlog a mile long where most of the internal conversations are about how to prioritize what to build next.

This is viewing things too narrowly I think. Why do we even need most of our current software tools aside from allowing people to execute a specific task? AI won't need VSCode. If AI can short circuit the need for most, if not nearly all enterprise software, then I wouldn't expect software demand to increase.

Demand for intelligent systems will certainly increase. And I think many people are hopeful that you'll still need humans to manage them but I think that hope is misplaced. These things are already approaching human level intellect, if not exceeding it, in most domains. Viewed through that lens, human intervention will hamper these systems and make them less effective. The rise of chess engines are the perfect example of this. Allow a human to pair with stockfish and override stockfish's favored move at will. This combination will lose every single game to a stockfish-only opponent.

It's interesting. Maybe I'm in the bigtech bubble, but to me it looks like there isn't enough work for everyone already. Good projects are few and far between. Most of our effort is keeping the lights on for the stuff built over the last 15-20 years. We're really out of big product ideas.

The big fear shouldn't be on loss of jobs, it should be the inevitable attack on wages. Wage will track inversely to proximity as a commodity status.

Even the discussion around AI partially replacing coders is a direction towards commoditization.

Dev effort isn't always the bottleneck. It's often stakeholders ironing out the ambiguities, conflicting requirements, QA, ops, troubleshooting, etc.

Maybe devs will be replaced with QA, or become glorified QA themselves.

That's the naiveity of software engineers. They can't see their limitations and think everything is just a technical problem.

No, work is never the core problem. Backlog of bug fixes/enhancements is rarely what determines the headcount. What matters is the business need. If the product sells and there is no/little competition, the company has very little incentive to improve their products, especially hiring people to do the work. You'd be thankful if a company does not layoff people in teams working on mature products. In fact, the opposite has been happening, for quite a while. There are so many examples out there that I don't need to name them.

I’m more bearish about LLMs but even in the extreme optimist case this is why I’m not that concerned. Every project I’m on is triaged as the one that needs the most help right now. A world when dozen projects don’t need to be left on the cutting room floor so one can live is a very exciting place.

> If anything we're the last generation that knows how to debug and work through issues.

I suspect that comment might soon feel like saying "not too worried about assembly line robots, we're the only ones who know how to screw on the lug nuts when they pop off"

Yep, and we still need COBOL programmers too. Your job as a technologist is to keep up with technology and use the best tools for the job to increase efficiency. If you don’t do this you will be left behind or you will be relegated to an esoteric job no one wants.

A fair amount has been written on how to debug things, so it's not like the next generation can't learn it by also asking the AI (maybe learn it more slowly if 'learning with AI' is found to be slower)

Sure, but let's still appreciate how awesome it is that this very difficult (for a human) PR is now essentially self-serve.

How much hardware efficiency have we left on the the table all these years because people don't like to think about optimal use of cache lines, array alignment, SIMD, etc. I bet we could double or triple the speeds of all our computers.

I am working on something even deeper. I have been working on a platform for personal data collection. Basically a server and an agent on your devices that records keystrokes, websites visited, active windows etc.

The idea is that I gather this data now and it may become useful in the future. Imagine getting a "helper AI" that still keeps your essence, opinions and behavior. That's what I'm hoping for with this.

We are getting closer and closer to that. For a while llm assistants were not all that useful on larger projects because they had limited context. That context has increased a lot over the last 6 months. Some tools will even analysis your entire codebase and use that in responses.

It is frustrating that any smaller tool or api seem to stump llms currently but it seems like context is the main thing that is missing and that is increasing more and more.

The idea is that you give the libraries and APIs as context with your prompt.

Can confirm anecdotally. Even R1 (the full, official version with web search enabled) crashes out hard on my personal Rust benchmark - it refers to multiple items (methods, constants) that don't exist and fails to import basic necessary traits like io::Read. Embarrassing, and does little to challenge my belief that these models will never reliably advance beyond boilerplate.

(My particular test is to ask for an ICMP BPF that does some simple constant comparisons. Correctly implemented, this only takes 6 sock_filters.)

And then your manager is wondering if you're a software engineer why you can't debug it

`git blame` comes in handy

"This broke. Here is the error behavior, here are diagnostics, here is the code. Help me dig in and figure this out."

Did you happen to try the same thing on Deepseek R1 on https://chat.deepseek.com/ ?

Yes! I've done something like this here in my repo. This was nice while lasted (Deepseek is practically useless through the API since yesterday)

https://github.com/bodo-run/yek/blob/main/.github/workflows/...

https://github.com/bodo-run/yek/blob/main/scripts/ai-loop.sh

Using askds https://github.com/bodo-run/askds

How do you know you've got a bug, to tell the AI to fix it?

Where companies depend on code for business critical applications? Forever.

When your AI-managed codebase breaks, who are you going to ask to fix it? The AI?

Couldn't agree more. Every time these systems get better, there are dozens of comments to the effect of "ya but...[insert something ai isn't great at yet]".

It's a bit maddening to see this happening on a forum full of tech-literate folks.

Ultimately, I think to stay relevant in software development, we are going to have accept that our role in the process could evolve to humans essentially never writing code. Take that one step further and humans may not even be reviewing code.

I am not sure if accepting that is enough to guarantee job security. But I am fairly sure that those who do accept this eventuality will be more relevant for longer than those who prefer to hide behind their "I'm irreplaceable because I'm human" attitude.

If your first instinct is to pick these systems apart and look for things that they aren't doing perfectly, then you aren't seeing the big picture.

Quite the contrary, really. We've been seeing "success stories" with AI translating function calls for years now, it just doesn't get any attention or make any headlines because it's so simple. SIMD optimization is pretty much the lowest-hanging fruit of modern computation; a middle schooler could write working SIMD code if they understood the problem.

There's certainly a bit of irony in the PR, but the code itself is not complex enough to warrant any further hysteria. If you've written SIMD by hand you're probably well familiar with the fact that it's more drudgery than thought work.