Skip to content(if available)orjump to list(if available)

If we had the best product engineering organization, what would it look like?

abc-1

I always feel like these manager types have drunk deeply from the koolaid for some reason. It’s a lot of words and processes that they usually cargo culted from somewhere else. A lot of it seems to boil down to “don’t be an idiot” and “actually care about your work”. They always have this air of superiority because they’re high up on the org chart. Like CEOs who think they’re the chosen ones, when plenty of people could do it just fine. I laugh hard when people like Zuck say software devs will be replaced by AI, not realizing an AI CEO probably wouldn’t have burnt 30 billion on a terrible metaverse flop.

wodenokoto

Would it have bought instagram and whatsapp? Would it have identified major social media trends in competitors that couldn't be bought and outcompeted them at their own game? Would it have suggested developing your own ad platform or just should banner space per cpm?

There is a lot to not like about Meta and Zuckerberg, but saying he's a bad business man is a little silly. Metaverse was a wrong and expensive move, but it was a wrong move they could afford.

Xcelerate

Exactly. I’ve never understood this criticism. A good CEO (in terms of the business) is able to make significant, groundbreaking decisions but is also able to reverse course quickly if those decisions aren’t working out (anyone remember the Facebook phone)?

Unless the CEO is psychic, they’re necessarily going to make a lot of bad or wrong decisions. The key is being able to recover quickly and move on to the next thing. A bad CEO makes no big decisions for fear of being wrong.

When FB bought Instagram for $1B, there were a lot of talk show hosts riffing on Zuckerberg for making one of the stupidest business decisions of all time. A lot of executives who got to their position by corporate ladder climbing have personalities that would be terrified of that sort of widespread criticism. They would never make the kinds of decisions that might possibly put them in the unenviable position of being made fun of on national television.

mempko

The assumption that hierarchical organizations are inevitable blinds us to more effective ways of organizing. When we look at companies today, their feudal-like structure means CEO decisions naturally have outsized impact - but this is a product of the system, not inherent necessity.

Take Meta's acquisitions of Instagram and WhatsApp under Zuckerberg. While these proved strategically valuable, framing them as evidence of unique CEO insight misses a crucial point: Many others in the organization likely would have made similar choices given the same position and information. The success stems more from the concentrated decision-making power than from individual brilliance. What's fascinating is how we conflate organizational structure with individual capability. When good outcomes emerge from hierarchical systems, we rush to credit the person at the top rather than examining how the structure itself shapes and amplifies their decisions. This creates a self-reinforcing cycle: hierarchical success is used to justify more hierarchy.

But imagine if we distributed decision-making power more broadly, tapping into the collective intelligence and diverse perspectives of entire organizations. Research on collective intelligence and successful worker cooperatives suggests groups often make better decisions than individuals, especially on complex issues. Companies like Valve and Morning Star have demonstrated that flat organizations can be both innovative and profitable. The real opportunity lies in reimagining organizational structures that harness our full human potential - not just that of a select few at the top. By questioning our assumptions about hierarchy, we open ourselves to discovering more dynamic, equitable, and effective ways of working together.

ryandrake

I think questions about whether a CEO is "good" or not are kind of impossible to answer, because you can't test the answers. All we can observe is the current reality and the decisions they've already made. There is no way to observe the other alternate universes where Zucc didn't buy Instagram, and/or the universes where he bought something else. Did the company do better or worse in those universes? No way to know.

Also, sometimes people say things like "Only [ceo's name] could run [company]. Look at their results!" This is just survivor bias. Who could know whether someone else could or couldn't run Facebook (or Tesla, or whatever)? Who can say with certainty that out of the 8+ billion people on the planet, only one particular guy could run the company, and that particular guy happened to be the guy who indeed ran it? What an improbable coincidence!

miki123211

IMO metaverse was a bet.

It's perfectly fine to take a bet that you're not 100% convinced will pay off (professional poker players and traders understand this on a very deep level), as long as the potential upside is massively larger than the downside.

Zuck understood that Meta could take the hit if the metaverse bet didn't pay off, but that they'd be massively better off if it did, and they had their own platform. Apple's blatantly anticompetitive behavior around ATT was a prime example of what happens when your business is reliant on "platform overlords."

I'm not fully convinced that the metaverse era is over, though. If they can get Orion costs down and put something of that quality into serial production, I think they still have a chance there.

Zanfa

> IMO metaverse was a bet.

It was FOMO. They had no vision, they had no plan, it was clear that it was only a thing because it was a buzzword at the time. Just like every other company stuffing crypto-adjacent things everywhere. It might have been a bet, but it was obvious that it was a really terrible one.

wavemode

No. "Bet" implies some sort of clear vision or value proposition. Metaverse was a dream.

It's okay to dream. It's not okay to burn billions of dollars on dreams with no proof of concept or business plan. Set aside the question of whether or not it's a bad idea - that's just plain bad execution.

short_sells_poo

I wholeheartedly agree with your thoughts, with perhaps the exception of the CEO. Or rather, the CEO needs to be someone who will be followed by the rest of the organization. Their decision making could be replaced by an AI, probably more readily than the specialist engineers in fact, but it comes back to humans being tribal. Would the middle management follow an AI overlord? Would the engineers buy into the AIs decisions?

The CEO's job is to decide on the future direction of the company, and then convince both the owners and the employees that this is a good direction. The first part is easy to replace with AI, the second part isn't. At least not for now.

I guess taken to the limit, the CEO will become replacable the moment that the employees have been replaced with AIs, because that that point there's nobody left to lead really. One could actually argue that this is tantamount to the CEO cutting off the branch they are sitting on. After all, once all the employees are AIs, what's to stop the shareholders from saying: "Hold it right there Steve/Jeff/Mark/etc, why are we paying you big bucks? You can be replaced with an AI that will make much better decisions, and there are no employees left to lead anyway."

robertlagrant

> I guess taken to the limit, the CEO will become replacable the moment that the employees have been replaced with AIs, because that that point there's nobody left to lead really.

This seems very software-centric. You can do this today - e.g. Red Bull famously outsources basically everything but marketing[0], so they already have very few employees. However they do have a lot of suppliers, and that all needs managing.

[0] Their marketing is either simple TV ads or incredibly complex stuntwork and extreme sports.

Sevii

I appreciate that they have a programming philosophy that they want people at the company to adopt. A common problem I see at companies that don't have onboarding is that people join the team with assumptions from previous jobs but you never level set them with the company. So 12 months down the line the new guy wants to change the process and you have to repeat the same discussions about what agile means for the nth time.

Amazon does a good job of training new hire on the 'Amazon way'. Amazon does 6 pagers, they do design docs. Amazon does SOA. Amazon does not use relational databases. Everything has an API. Because of the 'Amazon way' and the training they do new team members understand at least some of the context and expectations.

Is it the best way? Probably not but no one knows what the best way is anyway. At least they have a way. Saves a lot of effort compared to every new hire relitigating the process and architecture.

saghm

> Amazon does a good job of training new hire on the 'Amazon way'. Amazon does 6 pagers, they do design docs. Amazon does SOA. Amazon does not use relational databases. Everything has an API. Because of the 'Amazon way' and the training they do new team members understand at least some of the context and expectations.

As a counterpoint, a huge part of Amazon's culture (or at least, AWS's) in my experience was the emphasis on operations and the fact that they didn't have any separation between SREs/on-call engineers from the people who implement their services, and at least for me as someone who had never been on-call before in any meaningful capacity (due to my previous job working on a libraries rather than services), the training for it was basically non-existent on the two teams I spent time on. The "training" I did receive essentially consisted of being put on the rotation once to shadow, where I was able to sort of see what the actual on-call person did but didn't really have any explanation for how to know how to do them other than being told to read the runbooks, which were not really written in a way that was easy to understand for me as someone who was so new to learning all of the internal AWS tooling and ops in general. The next time I came up on the rotation, I was expected to be able to manage on my own, which essentially meant that literally no matter what occurred, I ended up having to escalate because I wasn't knowledgeable enough to fix literally anything within a timeframe that would have been reasonable.

xwolfi

Which is the only way to learn tbh, you can receive as much positive reinforcement imaginable, nothing prepares you for a large scale incident like living through one, building the connections you need to solve it, getting the shame of your life, and losing sleep over your failure.

saghm

I'm not really sure what you mean by "positive reinforcement", but I don't think it's possible to disagree more with this sentiment. "building the connections you need to solve it, getting the shame of your life, and losing sleep over your failure" isn't a strategy for teaching for something; it's a coping mechanism for someone trying to brute force their way through something that they weren't adequately trained for.

Most people seem to think it's fine for companies to offload the entirety of the burden of learning to individual employees, and maybe I'm an outlier in this regard, but to me, this seems more like a cop out to avoid trying to actually solve the problem at the cost of the employee's emotional health. I'm not surprised that companies default to this, but it's also not surprising that burnout is so common in our industry when this is considered the "best" or "only" way to do things.

pyrale

Nothing teaches you to swim like being thrown in the middle of the atlantic.

Sevii

My team didn't add new members to the oncall rotation for about 6 months to ameliorate this issue. But starting oncall at first is rough and even with months of context on our systems people usually take a few rotations before they really figure it out. We expected new members to have to escalate.

saghm

I don't think this really ameliorates the issue much; it just pushes the problem down the line. IMO this is a big part of why people transfer internally to different teams so much at Amazon, and that masks the problem even further. If by six months of on-call rotation you expect someone to be self-sufficient, but they only start after six months on the team, they'll have been on the team a year by that point, and people either transferring or leaving the company after a year on average isn't going to be immediately obvious as a problem, but if people start a few weeks in, you're going to have 5-6 months of noticing that there are issues when that person is on-call, and if that happens more than a few times, the trend will be noticeable.

wesselbindt

> they didn't have any separation between SREs/on-call engineers from the people who implement their services

I.e., treat DevOps as a way of working, rather than a role meaning something akin to "Ops person who knows terraform, or k8s, or Ansible etc".

infomaniac

> Amazon does not use relational databases

This is false, at least in my very thin exposure to the company: I interviewed for a team last year which was maintaining EC2 SSH keys using MySQL.

mnahkies

I watched an interesting talk from the RDS team about how they dogfood RDS the other day https://www.pgevents.ca/events/pgconfdev2024/schedule/sessio...

chupasaurus

If it was using MyISAM relations are in question.

9rx

SQL, and therefore MySQL by extension, isn't relational.

snapcaster

He typed smugly, confident that everyone reading this will appreciate his pedantry that totally contributed to the conversation

rrr_oh_man

> Amazon does not use relational databases

Huh?

vamega

Relational databases are not the preferred storage mechanism at Amazon. If a team wants to use an OLTP relational database it’s possible that it will be a decision they will need to defend at any kind of design review.

Of course there are relational databases running OLTP workloads, but it’s far away from the norm. There was a program a while ago to shift many RDBMS systems onto something else.

emmelaich

So they do joins in code rather than SQL? Wouldn't that risk hiding scaling problems?

deskglass

Friends say they typically use Dynamo and that using a relational database requires approval from a vp (because of scaling concerns).

hyperliner

Related: Amazon kicked out Oracle from the company.

Somewhere along the same timeline, the operational recommendation for teams was to not use relational databases.

https://www.theregister.com/2019/10/16/amazon_ditches_oracle...

zug_zug

I don't usually love this type of post, but this one is the exception - very valuable.

Just the section on how to rewrite alone communicates something incredibly valuable that grumpy-engineers like myself have great trouble getting others to understand.

I don't have my mind made-up on XP, I've never worked at a place that actually supported collaboration (often workers spoke different first languages, vastly different experience levels, had minimal social graces, were uncomfortable asking questions), but I think it could exist with great effort and would have a lot of upsides.

devin

It is likely a bit of nostalgia on my part, but one of my first gigs had an owner that was focused on XP being the way we worked, and given how junior the team was overall, I think it produced excellent results and made for a fun, lively atmosphere.

switch007

> We’re an inverted organization. That means that tactical decisions are made by the people who are doing the work, not managers

I've worked at various places where this was supposedly the system.

Guess who had budgets, hiring powers, went to leadership offsites? Yeah not the ICs. It usually just means the C level will smile and nod while "listening" to your feedback instead of ignoring you completely

Has anyone worked at a true inverted company where centuries of classical power structures are thrown out the window?

I feel it can never be properly implemented unless in eg a cooperative

Rygian

The "Valve Handbook for New Employees" was an interesting read. I still don't know how much is fact and how much is fiction, but I liked some of the ideas there.

Maxious

>"We sort of had to collectively admit we were wrong on the premise that you will be happiest if you work on something you personally want to work on the most," Valve developer Robin Walker tells Keighley, soundly rejecting the ethos that Valve has publicly carried as a torch for some time. The studio used a new Half-Life [Alyx, 2020] project as a way to focus the entire studio—even though, as we've previously reported, that project began life with more modest expectations in terms of length and content.

https://arstechnica.com/gaming/2020/07/valve-secrets-spill-o...

InitialLastName

> Guess who had budgets, hiring powers, went to leadership offsites?

To be clear, those are strategic decisions "what are our goals and how do we allocate resources to achieve them". Tactical decisions are "what specific actions do we take to use available resources to achieve goals".

pjm331

To be fair - it says “tactical” decisions - not all decisions.

lubujackson

What an amazing article on "de-FAANGing" the perverse org/incentive structure of most startup/tech places. Would love to see more of this type of leadership in the real world.

Sevii

I like how he says he doesn't need FAANG level people. Then his next paragraph describes working at FAANG.

"We’re an inverted organization. That means that tactical decisions are made by the people who are doing the work, not managers. (In theory, anyway, we’re not perfect.) So we’re looking for people who have peer leadership skills, who are great at teamwork, who will take ownership and make decisions on their own."

holografix

Exactly right. People who have “leadership” skills are the ones that pay attention to their own leadership and are manage up more than anything else.

They usually repackage people’s work around them into their own, take ownership and defend loudly their territory (project ownership) and methodically build relationships with leadership. Having “leadership skills” and being good a team work are often orthogonal to each other.

khazhoux

> People who have “leadership” skills are the ones that pay attention to their own leadership and are manage up more than anything else.

No, it's actually people who can put together a technical plan and drive its execution with other engineers, or who can clarify complex problems esp. when there are conflicting opinions, or who can see problems before they become disasters and organize the right group of people to take care of it...

There are many examples of leadership, which have nothing to do with the sour view of managing-up or taking credit for others' work.

noirbot

I mean, that's the sour way to put it. As someone kinda stuck in that sort of position now, I own a project and have been able to do very little of the work because I'm spending 90% of my time making sure leadership actually makes decisions so I can get the decisions my team needs in order to proceed. I'm spending hours in meetings with other teams to get them to prioritize our dependencies and data access needs.

If you're not lucky enough to have management that's exactly technically aligned to your project someone has to be managing up and paying attention to leadership or else expectations will be totally off from reality.

numbsafari

> FAANG level people

"FAANG" isn't a "level", it's just a cluster.

adastra22

The recent book The NVIDIA Way is about that org’s culture that prevent FAANG incentives from creeping in to destroy productivity.

zeroonetwothree

There’s a weird disconnect because on the one hand I agree you can’t measure productivity and on the other hand we all know that some engineers are vastly more productive than others. So what gives?

Trasmatta

We all "know" that, but there are also some engineers that only give a very strong illusion of being more productive.

Maybe engineer #1 is constantly pushing up code. In the time it takes them to merge 15 PRs, engineer #2 opens only 1 - but maybe they thought really deeply about the problem, and their approach actually saves the team hundreds or thousands of hours of future development work vs how engineer #1 would have solved the problem.

Part of what makes this so hard to measure is the long tail effects of development decisions. (Incidentally, that's also a source of burnout for me - the constant mental overhead of worrying about the long term implications of what I'm doing, and particularly how they effect other people. It's very challenging.)

ebiester

The problem is that the vast majority of code will not have long term implications so long as it reaches a minimum of design, performance, and does its job without bugs. Consistency of patterns is more important than the optimal pattern for most decisions.

There are some core areas of the application that are much more important, but they are often the earliest data structures and built before the problem is known. You will not know how your code will change, so make it as consistent as possible with the rest of the system until you know more.

llm_trw

The engineers are only productive because they have the support structure in place.

The most productive fpga engineer I ever hired was so hopeless with git that I had to hire a second software engineer to babysit him.

After I left both of them got fired and the product they were ahead of schedule on when I left had slipped 2 years behind before it finally got cancelled three years later.

skeeter2020

I have an incredibly productive staff developer. Not only does he work a lot, he also produces, and it's very high quality. He also does a relatively poor job of upskilling his teammates, and is a little rough when mentoring. This is not intentional (i.e. he's not a jerk).

Overall I don't know if, in the context of a staff developer, he's vastly more productive than say, another dev I have who produces less but levels-up his team better than almost anybody I've ever seen.

ggregoryarms

Maybe that other dev has a unique ability you should reward. Sound awesome. Focus on that.

jprete

Those are two different concepts hiding in similar words. You can't [numerically or precisely] measure productivity, but some engineers are vastly more productive [such that you can easily tell the difference without a formal measurement].

fshafique

Gut feeling uses all your internal predispositions and biases.

skeeter2020

you don't need to rely on gut feeling and risk bias. You can stop looking for the "productivity metric" and instead bet on <some> measure, then track the change over time. It's the only thing that's ever worked for me.

Sevii

You can measure productivity with correlated metrics. The issue has always been that the metrics which are easy to track don't line up incentives with the actual business goals. A group of 10 people who write 200k loc per year are probably more productive than a group of 10 people who write 10k loc per year. If you took those metrics and then did an investigation of the people in your company writing 10k loc you might find that they are slackers or that they write assembly.

The issue is when metrics are used to stack rank teams with no thinking put into it. You can't treat correlated metrics like direct metrics. A logger might be evaluated based on how many trees he cut down in a day. There is no comparable way to pay software engineers piecemeal.

Metrics are good, but people want to use them without thinking or taking context into consideration.

bigs

Or you may find they write higher quality code - less bugs, more performant code, or so on.

Salgat

It's an extremely complex mixture of many factors (which can vary wildly between two different productive engineers), and trying to make that into some magical formula ends up creating a system that can be gamed to superficially appear productive to managers.

throw5959

You can measure productivity by measuring the success, but that's kinda useless for day to day software engineering management.

ChrisMarshallNY

I tend to go by results, and for me, "results" means shipped* code that is used and accepted by end users**, can be maintained and extended***, and doesn't generate trouble tickets.

* MVP doesn't count.

** Can include users inside the organization.

*** It's OK if it requires senior-level ongoing support. I think expecting it to be maintained by monkeys is a bad idea.

pinkmuffinere

To me, "MVP doesn't count" feels like a crazy take -- in many roles, the _only_ ask is to produce a series of different MVP's. I guess maybe the definition of "MVP" is a bit squishy, and these people-who-ship-MVPs themselves make MVP-MVP's, which shouldn't count as shipped?

skeeter2020

saying "MVP doesn't count" implies that you throw it away and then right "the perfect system" at some point. If you've ever had an MVP land you know that's not how it happens.

Trasmatta

How do you define success? If a product bombs, is that because of the engineering or the product design?

throw5959

I don't think it's possible to answer generally. Track what matters for your business.

skeeter2020

if it's successful, it's because of sales. If it fails, engineering didn't build the right thing / was too slow - it really doesn't matter.

daz0007

a weird disconnect... of any true innovation or even reality... such vague objectional blandness...

"They'd beg to work for us" - what the f8ck.... if they were the best they would not beg anyone how degrading...They would be there for a mission or wanting to improve something about themseves or other parts of the world.

There's nothing here apart from Agile coach wanting to get some more work.

1984 was released in 1949, if anyone thinks these words / values really mean what is writen wow. People, Internal Quality, Lovability, Visibility, Agility, Profitability...

theideaofcoffee

This was a really great read, lots of insight and things to think about.

But it's also depressing to see how good things could be and how poorly (IME) most orgs are run now. I know I've seen the exact 180-degree opposite of almost everything mentioned here: no team leadership or empowered people, no clear path to the next level for those interested, lack of communication, no emphasis on internal quality, overall pathological product choices (or lack thereof) and on and on. I'd kill to be part of an org that puts this much thought into everything.

paulcole

> Everybody wants the best people in the business.

A fundamental mistaken belief.

Who wants to pay for the very best people when the 97,000th best person will do? Also how can you decide who the best people are when you can’t even measure their productivity?

gwern

OP is an example of how AI-generated images are usually clutter. Not only do the images not add anything meaningful to the text, and arbitrary parts of the images could be deleted or randomized without affecting the reader's understanding, most of them could be randomly shuffled without anyone noticing. (Which makes them worse then clipart/stockart: if an article swapped the 'hacker hoodie' stockart with the 'neural net brain circuit' stockart, some readers would at least briefly be confused.)

blululu

Came here to say exactly this. The ”author” didn’t even bother to use a decent quality image generator. The first image I saw maxed out my AI slop filter and I stopped reading. Made me wonder how much of the article was written by ChatGPT.

drcwpl

Agree wholeheartedly, it was unfortunate that the author did not use a good image generator, or even correctly prompt Dall-E to get better images, his take on using AI then became rather flimsy. I gave up at this point!

null

[deleted]

magic_smoke_ee

First, it would be worker-owned co-op with very little turnover and intense competition for the few roles that get filled.

mrbluecoat

+1 for Extreme Programming. I've been a fan from the beginning when Agile was all the rage and my recommendations for XP were met with blank stares.

Trasmatta

I'm glad that it works for some people, but I did not like the forced pair programming in XP at all. And I found adherents to XP were even more cult like than Agile teams.

Salgat

Does XP and pair programming actually require two people to be simultaneously working together at the same time? My understanding is that this includes one person who codes while another person looks at the results and reviews them afterward. The two are still working closely together and exchanging feedback, just at different points in the process in an iterative loop.

NomDePlum

My understanding is that original meaning was that pair programming requires the pair work together at the same desk and machine.

With the ability to share screens/IDEs remotely the need to be at the same desk may have shifted, but working together is intrinsic to pair programming I believe.

The original text went into some detail about making the desk work for 2 people, and having screwdrivers available to do so, which for some reason always amused me.

p_l

No, the point is that both "driver" and "navigator" (as some pair programming referred to the roles) are looking at the coffee simultaneously, just one has the keyboard at the time.

This is extended to "mob" programming where you have whole team of "navigators" and one person at keyboard.

skeeter2020

resist the oxymoron of agile zealot - the first rule is do what works for YOU

mft_

A great post, well worth reading. The principles in the section on 'people' are applicable to any organisation in any industry.

I especially liked the simple 'career ladder' example, for a) focussing on mostly on behaviour rather than knowledge, and b) for being simple to use and track progress with. (I've never seen anything like it in any of the large organisations I've worked in to date.)