Prompting LLMs is not engineering

This article is just gatekeeping the word "engineer" for no reason. When I was developing an agent recently I picked a model (thus fixing most of the "uncertainties" mentioned) and started to construct the system prompt, from simpler to more complex, sometimes back to simpler, discovering that to this particular model the name choice means a lot, observing how often I was or wasn't getting desired results, etc etc. I think it would be challenging to find a definition of the word "engineer" that wouldn't include this process, even if it, obviously, wasn't as complex as some other engineering projects.

This guy should read Bruno Latour, it would make him understand that engineering and more generally science is also a belief system, based on some way of validating the knowledge that can be (or not) what occidental people call logical. But it's another ethnocentrism. See symetry proof of azandé for more insights

My theory is that, when done properly, it’s much closer to science than engineering.

And by “done properly,” i mean done in a regimented way with evals to verify that a wide range of inputs produce the desired outputs.

Prompting is much closer to discovering the properties of an already existing system than building something using engineering methods.

I have no problem taking issue with using the term "engineering" for this, but this reads like OP does not believe that you can get better results based on the way you structure your prompt and what information you include. That is wild to me and should be obvious to anyone who spends more than a few hours using LLMs.

For what it's worth, I think about the "engineering" in "Prompt/Context Engineering" almost more akin to how it's used in "Social Engineering". You are influencing the model to produce a desirable result.

Prompting is more art than engineering. But understanding the basics of LLMs, which are engineered systems, helps you get better results. I think of it like being a good interviewer: you’re not controlling the conversation, but you know how to guide it. A good prompter, like a good interviewer, knows how to ask the kind of question that gets a meaningful answer. I think of LLMs like a giant library and my task is to steer it to the right sections for a good answer.

Prompting "ChatGPT" is very different than prompting LLMs. For the latter, whether you want to use the E word or not, every one of the unknowns the author cites either don't exist or are greatly reduced when you're using LLMs to construct something purpose-built.

Anyone working in tech should take a few hours, download LangFlow and choose a small model in ollama and just run one of the template flows and peek at their prompt setups. Maybe the author just chose a poor title, there's so much writing like this that is the equivalent of saying "Drills can't make good holes" when what you mean is the $15 harbor freight special hooked up to a janky power source and unsteady hand.

There are plenty of engineering disciplines that need to have statistical approaches to solving problems. Prompt engineering, should be approached by creating benchmark datasets and defining measures of efficacy and reliability.

More generally, IMHO engineering is use of physical and informational tools to build a solution. Tools may have unpredictability and reliability concerns. Its the job of an engineer to utilize the power of these tools while overcoming the reliability issues that might be present.

Example: Semiconductor manufacturing involves shooting gases at piece of silicon. There are all sorts of random scattering, distributional anomalies, and patterning problems that arise largely at random. I think you would still call it engineering.

I don’t mind people criticising AI.

It’s bad in loads of ways.

But it also has some good bits.

To call it all snake oil is just as much bullshit as to claim it will solve all humanity’s problems.

If a founder hires an engineer who develops a product, the founder is not magically an engineer. He hired the work out.

This isn’t complicated but people struggle to understand it for some weird reason.

If I have someone photograph my wedding, and then sit at my computer, once the pictures arrive to me, distribute the photographs, I’m not suddenly magically a photographer.

I could go on and on. Using an LLM to write a biology paper does not make that person a biologist.

Typing out a request to my friend to write me a poem about chickens with red mittens doesn’t magically somehow make me a poet.

It wild that so many struggle with this basic ass concept.

Coding in general isn't engineering. Software engineering is a relatively boring administrative exercise and unless you just want to maximize your salary you should probably do everything you can to avoid a "software engineering" role.

Doesn't matter that much.

Management thinks they can do our jobs by replacing us with a shell scrip... LLM prompt.

We're always moving up and down the tech ladder, solving problems we thought were solved, and, we fix them. But some prompt is supposedly going to replace understanding of your infra?! Hardly.

But you'll save money for this quarter. Past that is definitely up for discussion.... With your shareholders.

This seems very obvious to literally any one that is half competent at actual engineering.