A PM's Guide to AI Agent Architecture

I really don't understand how people given access to a pile of tools and data sources and unleash them on customers. It's horrible UX in my experience and at times worse than a phone tree.

My view is that you need to transition slowly and carefully to AI first customer support.

1. Know the scope of problems an AI can solve with high probability. Related prompt: "You can ONLY help with the following issues."

2. Escalate to a human immediately if its out of scope: "If you cannot help, escalate to a human immediately by CCing bob@smallbiz.co"

3. Have an "unlocked agent" that your customer service person can use to answer a question and evaluate how well the agent performs in helping. Use this to drive your development roadmap.

4. If the "unlocked agent" becomes good at solving a problem, add that to the in-scope solutions.

Finally, you should probably have some way to test existing conversations when you make changes. (It's on my TODO list)

I've implemented this for a few small businesses, and the process is so seamless that no one has suspected interaction with an AI. For one client, there's not even a visible escalation step: they get pinged on their phone and take over the chat!

I MVP'd one of these (a basic sequence of LLM customer support 'agents') at my last job, I guess spring 2024. So much has changed since then!

'Routing through increasingly specialised agents' was my approach, and the only thing that would've done the job (in MVP form) at the time. There weren't many models that would fit our (v good) CS & Product teams' dataset of "probable queries from customers" into a single context window.

I never personally got my MVP beyond sitting with it beside the customer support inbox, talking to customers. And AFAIK it never moved beyond that after I left.

Nor should it have been, probably - there are (wild, & mostly ineffable) trade-offs that you make the moment you stop actually talking to users at the very moment they get in touch. I don't remember ever making a trade-off like that where it was worthwhile.

I _do_ remember it as perhaps the most worthwhile time I ever spent doing product-y work.

I say that because: To consider a customer support query type that might be 0.005% of all queries received by the CS team, even my trash MVP had to walk a path down a pretty intricate tree of agents and possible query types.

So - if you believe that 'solving the problems users have with your product' = 'making a better product'. then talking to an LLM that was an advocate for a tiny subset of users, and knew very intimately the details of their issue with your product, that felt really good. It felt like it was a very pure version of what _I_ should be to devs, as any kind of interface between them and our users.

It was very hard to stay a believer in the idea of a 'PM' after seeing that, at least. As a person who preferred to just let people get on with things.

I enjoyed the linked post; it's really interesting to see how far things have come. I'm surprised nobody has built 'talk to your customers at scale', yet - this feels like a far more interesting problem than 'avoid talking to your customers at scale'.

I'm also not surprised, I guess, since it's an incredibly bespoke job to do properly, I imagine, for most products.

Nice framing for PMs, but technically it is way too rosy. MCP is real but still full of low utility services and security issues, so “skills as plug-ins” is not production ready. A2A protocols were only just announced this year (Google, etc.) and actual inter-agent interoperability is still research grade, with debugging across agents being a nightmare. Orchestration layers (skills, workflows, multi-agent) look clean in diagrams but turn into brittle state machines under load. LLM “confidence scores” are basically uncalibrated logits dressed up as probabilities.

In short: nice industry roadmap, but we are nowhere near robust, trustworthy multi-agent systems yet.

> Confidence calibration: When your agent says it's 60% confident, it should be right about 60% of the time. Not 90%, not 30%. Actual 60%.

With current technology (LLM), how can an agent ever be sure about its confidence?

I'm typically pretty critical of PM oriented pieces, but I found this to be a decent overview of how to reason about building these systems from first principles + some of the non-tech pain points + how to address them.

As an engineer, I like this framework but can think of approximately zero PMs who could use it to build a product.

What does the PM title even mean at this point? It's a bit surprising to see a deep dive into technical architecture - though there is massive value in understanding what's involved - as a PM responsibility, this is more in TPM (technical program manager) land which is a different job.

In my book they ideally focus on understanding scope, user needs and how to measure success, while implementation details such as orchestration strategies, evaluation and making sure your system delivers the capabilities you want in general, are engineering responsibilities.

[flagged]

Stop trying to treat these things as more than they are. Stop trying to be clever. These models are the single most complex things ever created by humans; the summation of decades of research, trillions in capex, and the untold countless hours of thousands of people smarter than you and I. You will not meaningfully add to their capabilities with some hacked together reasoning workflows. Work within the confines of what they can actually do; anything else is complete delusion.