Tools: Code Is All You Need

Directionally I think this is right. Most LLM usage at scale tends to be filling the gaps between two hardened interfaces. The reliability comes not from the LLM inference and generation but the interfaces themselves only allowing certain configuration to work with them.

LLM output is often coerced back into something more deterministic such as types, or DB primary keys. The value of the LLM is determined by how well your existing code and tools model the data, logic, and actions of your domain.

In some ways I view LLMs today a bit like 3D printers, both in terms of hype and in terms of utility. They excel at quickly connecting parts similar to rapid prototyping with 3d printing parts. For reliability and scale you want either the LLM or an engineer to replace the printed/inferred connector with something durable and deterministic (metal/code) that is cheap and fast to run at scale.

Additionally, there was a minute during the 3D printer Gardner hype cycle where there were notions that we would all just print substantial amounts of consumer goods when the reality is the high utility use case are much more narrow. There is a corollary here to LLM usage. While LLMs are extremely useful we cannot rely on LLMs to generate or infer our entire operational reality or even engage meaningfully with it without some sort of pre-existing digital modeling as an anchor.

Something I've realized about LLM tool use is that it means that if you can reduce a problem to something that can be solved by an LLM in a sandbox using tools in a loop, you can brute force that problem.

The job then becomes identifying those problems and figuring out how to configure a sandbox for them, what tools to provide and how to define the success criteria for the model.

That still takes significant skill and experience, but it's at a higher level than chewing through that problem using trial and error by hand.

My assembly Mandelbrot experiment was the thing that made this click for me: https://simonwillison.net/2025/Jul/2/mandelbrot-in-x86-assem...

I have the feeling that's not really MCP specifically VS other ways, but it is pretty simply: at the current state of AI, to have a human in the loop is much better. LLMs are great at certain tasks but they often get trapped into local minima, if you do the back and forth via the web interface of LLMs, ask it to write a program, look at it, provide hints to improve it, test it, ..., you get much better results and you don't cut yourself out to find a 10k lines of code mess that could be 400 lines of clear code. That's the current state of affairs, but of course many will try very hard to replace programmers that is currently not possible. What it is possible is to accelerate the work of a programmer several times (but they must be good both at programming and LLM usage), or take a smart person that has a relatively low skill in some technology, and thanks to LLM make this person productive in this field without the long training otherwise needed. And many other things. But "agentic coding" right now does not work well. This will change, but right now the real gain is to use the LLM as a colleague.

It is not MCP: it is autonomous agents that don't get feedbacks from smart humans.

> try completing a GitHub task with the GitHub MCP, then repeat it with the gh CLI tool. You'll almost certainly find the latter uses context far more efficiently and you get to your intended results quicker.

This is spot on. I have a "devops" folder with a CLAUDE.md with bash commands for common tasks (e.g. find prod / staging logs with this integration ID).

When I complete a novel task (e.g. count all the rows that were synced from stripe to duckdb) I tell Claude to update CLAUDE.md with the example. The next time I ask a similar question, Claude one-shots it.

This is the first few lines of the CLAUDE.md

    This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.

    ## Purpose
    This devops folder is dedicated to Google Cloud Platform (GCP) operations, focusing on:
    - Google Cloud Composer (Airflow) DAG management and monitoring
    - Google Cloud Logging queries and analysis
    - Kubernetes cluster management (GKE)
    - Cloud Run service debugging

    ## Common DevOps Commands

    ### Google Cloud Composer
    ```bash
    # View Composer environment details
    gcloud composer environments describe meltano --location us-central1 --project definite-some-id

    # List DAGs in the environment
    gcloud composer environments storage dags list --environment meltano --location us-central1 --project definite-some-id

    # View DAG runs
    gcloud composer environments run meltano --location us-central1 dags list

    # Check Airflow logs
    gcloud logging read 'resource.type="cloud_composer_environment" AND resource.labels.environment_name="meltano"' --project definite-some-id --limit 50

My absolute favorite use of MCP so far is Bruce Hauman's clojure-mcp. In short, it gives the LLM (a) a bash tool, (b) a persistent Clojure REPL, and (c) structural editing tools.

The effect is that it's far more efficient at editing Clojure code than any purely string-diff-based approach, and if you write a good test suite it can rapidly iterate back and forth just editing files, reloading them, and then re-running the test suite at the REPL -- just like I would. It's pretty incredible to watch.

The problem I see with MCP is very simple. It's using JSON as the format and that's nowhere as expressive as a programming language.

Consider a python function signature

list_containers(show_stopped: bool = False, name_pattern: Optional[str] = None, sort: Literal["size", "name", "started_at"] = "name"). It doesn't even need docs

Now convert this to JSON schema which is 4x larger input already.

And when generating output, the LLM will generate almost 2x more tokens too, because JSON. Easier to get confused.

And consider that the flow of calling python functions and using their output to call other tools etc... is seen 1000x more times in their fine tuning data, whereas JSON tool calling flows are rare and practically only exist in instruction tuning phase. Then I am sure instruction tuning also contains even more complex code examples where model has to execute complex logic.

Then theres the whole issue of composition. To my knowledge there's no way LLM can do this in one response.

    vehicle = call_func_1()
    if vehicle.type == "car":
      details = lookup_car(vehicle.reg_no)
    else if vehicle.type == "motorcycle":
      details = lookup_motorcycle(vehicle.reg_ni)

Regarding the Playwright example: I had the same experience this week attempting to build an agent first by using the Playwright MCP server, realizing it was slow, token-inefficient, and flaky, and rewriting with direct Playwright calls.

MCP servers might be fun to get an idea for what's possible, and good for one-off mashups, but API calls are generally more efficient and stable, when you know what you want.

Here's the agent I ended up writing: https://github.com/pamelafox/personal-linkedin-agent

Demo: https://www.youtube.com/live/ue8D7Hi4nGs

> It demands too much context.

This is solved trivially by having default initial prompts. All majors tools like Claude Code or Gemini CLI have ways to set them up.

> You pass all your tools to an LLM and ask it to filter it down based on the task at hand. So far, there hasn't been much better approaches proposed.

Why is a "better" approach needed? If modern LLMs can properly figure it out? It's not like LLMs don't keep getting better with larger and larger context length. I never had a problem with an LLM struggling to use the appropriate MCP function on it's own.

> But you run into three problems: cost, speed, and general reliability

- cost: They keep getting cheaper and cheaper. It's ridiculously inexpensive for what those tools provide.

- speed: That seem extremely short sighted. No one is sitting idle looking at Claude Code in their terminal. And you can have more than one working on unrelated topics. That defeats the purpose. No matter how long it takes the time spent is purely bonus. You don't have to spend time in the loop when asking well defined tasks.

- reliability: Seem very prompt correlated ATM. I guess some people don't know what to ask which is the main issue.

Having LLMS being able to complete tedious tasks involving so many external tools at once is simply amazing thanks to MCP. Anecdotal but just today it did a task flawlessly involving: Notion pages, Linear Ticket, git, GitHub PR, GitHub CI logs. Being in the loop was just submitting one review on the PR. All the while I was busy doing something else. And for what, ~1$?

More appropriately: the terminal is all you need.

I have used MCP daily for a few months. I'm now down to a single MCP server: terminal (iTerm2). I have OpenAPI specs on hand if I ever need to provide them, but honestly shell commands and curl get you pretty damn far.

I think the GitHub CLI example isn't entirely fair to MCP. Yes, GitHub's CLI is extensively documented online, so of course LLMs will excel at generating code for well-known tools. But MCP shines in different scenarios.

Consider internal company tools or niche APIs with minimal online documentation. Sure, you could dump all the documentation into context for code generation, but that often requires more context than interacting with an MCP tool. More importantly, generated code for unfamiliar APIs is prone to errors so you'd need robust testing and retry mechanisms built in to the process.

With MCP, if the tools are properly designed and receive correct inputs, they work reliably. The LLM doesn't need to figure out API intricacies, authentication flows, or handle edge cases - that's already handled by the MCP server.

So I agree MCP for GitHub is probably overkill but there are many legitimate use cases where pre-built MCP tools make more sense than asking an LLM to reverse-engineer poorly documented or proprietary systems from scratch.

    > So maybe we need to look at ways to find a better abstraction for what MCP is great at, and code generation. For that that we might need to build better sandboxes and maybe start looking at how we can expose APIs in ways that allow an agent to do some sort of fan out / fan in for inference. Effectively we want to do as much in generated code as we can, but then use the magic of LLMs after bulk code execution to judge what we did.

Give it an objective and the LLM writes its own code and uses the tool iteratively to accomplish the task (as long as you can interact with it via a REST API).

For well known APIs, it does a fine job generating REST API calls.

You can pretty much do anything with this.

https://github.com/CharlieDigital/runjs

Yup, I can't help but think that a lot of the bad thinking comes from trying to avoid the following fact: LLMs are only good where your output does not need to be precise and/or verifiably "perfect," which is kind of the opposite of how code has worked, or has tried to work, in the past.

Right now I got it for: DRAFTS of prose things -- and the only real killer in my opinion, autotagging thousands of old bookmarks. But again, that's just to have cool stuff to go back and peruse, not something that must be correc.t