Skip to content(if available)orjump to list(if available)

In praise of “normal” engineers

tikhonj

> The smallest unit of software ownership and delivery is the engineering team.

I see where this is coming from, but it's also pretty sad. In my experience, it tends to create environments where engineers are second-class citizens compared to managers or product: we're just responsible for "delivery", but can't independently make any real decisions beyond a tiny scope. Our timespan of discretion becomes measured in days or weeks, while, on the best teams I've seen, it's measured in months, quarters or years.

It's counterintuitive, but you absolutely can have real individual owernship for engineers without creating single points of failure or brittle systems. It's a matter of having other sources of slack, encouraging quality work, and giving people a lot of room to fluidly adjust what they're working on when. Folks still have real ownership and can make longer-term decisions on their own, but they also collaborate in ad hoc ways and share tacit knowledge, so that somebody else can jump in, help out or even take over in a pinch. I'm being a bit vague, but all I can say is that I've seen this before, and I'll know it when I see it again.

In practice, the model I saw did end up with more rewriting than a normal team—but we were still way more productive overall, even accounting for the rewrites! Turns out that rewriting systems incrementally, in self-contained chunks, is an amazing way to both evolve the design and to build up instutitional knowledge and capacity. It looks like waste, but it's actually slack that is crucial to making your system as a whole more flexible, adaptable and resilient. (In fact, that's true of a lot of the "waste" top-down management systems try to reduce—I'm incresingly convinced that anybody trying to optimize software team "utilization" is actively sabotaging the work!)

dkarl

People outside of engineering can't give engineers immediate credit for long-term decisions. They don't have the competency to know what to reward.

I'd go further and say that even within engineering, people outside of a team can't give immediate rewards for work whose long-term value is internal to the team, for the same reason: they don't know if the work you're doing is actually valuable or is just superficially similar to the kind of work that often does have long-term value.

If you're confident enough in your long-term decisions, and how you spend slack time, then you should be fine with being rewarded for delivery, since you expect your long-term work to pay off in future delivery and future rewards.

maxsilver

> They don't have the competency to know what to reward.

I'd even take this a bit further, and say this is basically an argument that the engineering manager, engineering/dept VP, and CTO all need to be engineers or past-engineers themselves, so they actually do have enough competency to know what to reward.

tikhonj

the best manager I ever had was a VP who came from a banking background—the key thing was not that he had written a lot of production software himself, but that he had seen what great software looks like (apparently he worked closely with the core Slang/SecDB guys), and was willing to trust engineers who could build similar styles of tools

on the other hand, the recent skip-level I had that got me to quit in six months was an engineer himself, but had no real opinion on code quality or anything more than a superficial, process-oriented understanding of the dynamics on the team :(

moral of the story: taste is necessary; direct technical experience is mostly necessary, but nowhere near sufficient

scarface_74

The purpose of the engineer is to add business value by either reducing costs or increasing revenue. You reward employees for results.

EPWN3D

Honestly, I don't even think that does the trick anymore. Once you're on the management track, you get infected with the same MBA bullshit as the rest of management, and your brain rewires itself around that culture and those incentives.

I've worked for VPs who were former engineers, and they eventually lost their appreciation for the artistic side of the discipline and got obsessed with features, demoes, burn down charts, etc.

If you want to solve this problem, you need to keep the professional manager class out of your org. Because once they get a foothold, they fuck everything.

spimmy

just made this exact same point myself, lower in this thread. (author here)

matthewkayin

I think that rewrites are an important part of how software is written, and it's an important part of being "agile", in the sense that you can go in and write a prototype that's coded very simply without much regard for long-term architecture, knowing that requirements likely will change and that you likely won't get the design right on the first go anyways.

Coding is like writing, in the sense that it's often faster to write a sloppy first draft followed by a better second draft than it is to agonize over getting the first draft right on the first go. The first draft is generative. Its purpose is not to be good but instead to let you get something built quickly and to let you explore the problem, so that you know what edge cases you'll need to account for in your final architecture.

But this still of working will never get through management because the moment you show them a working product, they'll tell you to ship it and won't give you a chance to rewrite.

I think the best way to solve this is to flatten the hierarchy. Get rid of the notion of managers who rule over engineers and give ownership of the code back to the engineers. Have the engineers and product "owners" make decisions together in a democratic fashion.

athoscouto

Agreed that you can have real individual ownership. Not only that, I think that is the only way to be really "productive".

But I think that is beside the point.

Individuals are not fungible, but team members are - or at least can be, depending on how you structure your teams.

And as your org grows, you want predictability on a team level. Skipping a bunch of reasoning steps, this means having somewhat fungible team members, to give you redundancy.

The engineering parallel here is the tradeoff between resilience and efficiency. You can make a system more reliable by adding redundancy. You make a system more efficient by removing redundancy.

bsoles

I blame Agile and the like for fucking up individual ownership by treating every engineer and every task interchangeable. You can't build expertise by working on a different type of task every sprint...

hnthrow90348765

>we're just responsible for "delivery"

Ownership has never gotten me anything but more headache. I'm just here to put the things on the pages.

We've got to charge for additional responsibility. Manager/executive pay scales with how many people they're responsible for, no sense in not giving developers that too.

brettgriffin

The driving factor behind how much ownership engineers say they want versus what they actually want largely comes down to how the company handles product failure accountability.

In smaller teams at earlier ventures, there is a lot more leniency here. As the team and/or product space matures, failure begins to demand accountability.

I worked with a team last year whose engineers essentially had free reign to build the product they wanted. The product sucked, had virtually no adoption, but everyone was happy.

Another team I work closely with is in a razor-thin margin and established product space. I haven't seen a single engineer chime in on a product decision and I don't think they'd like to make those decisions for more pay.

xyzzy_plugh

I used to share this mindset, and I still agree that individual ownership is possible for engineers. Unfortunately many, many engineers simply do not want it. I would reckon most if not all engineers are comfortable with ownership at the team boundary, but many simply do not care beyond that. It's just a day job.

Individual ownership at the individual engineer boundary can breed distrust within a team or org, but often alienates team members who like their job but aren't trying to lead, at least with respect to what ownership entails. In this blended environment someone almost always ends up without agency. Sometimes no one gets agency. Who wants that?

It's surprisingly simple and effective by comparison to give a team agency and ownership, usually in part because of the dynamic of having a manager or lead to begin with.

Simply put, there are too many modes of failure at the individual level for software ownership to settle there comfortably: staffing changes, job security, career growth are the obvious ones, but the dysfunction (even in otherwise healthy orgs, there's always some amount) will find the shortest path to complete the circuit here.

I like to think of it like a gearbox. If you only have one gear, and you break it, or wear out all the teeth, then you don't get to go. If you have many gears, well, the ride may be uncomfortable at times, but you'll keep moving.

almosthere

I personally think _most_ people should treat their jobs as a _day_ job - unless they have actual ownership in the company (beyond what would be a 50-100k payout at option time)

thePhytochemist

I think this is key when people talk about "ownership". Actually owning a product means that if it fails you're holding the bag, if it succeeds you take the profits. And you have full control over it. Unless a company actually wants to do this I wish they wouldn't use that english word.

Trying to hire an employee and tell this story that they "own" the product is just silly. It's like companies that try to describe themselves as a family - just kind of a weird and incorrect use of a real word that has other meaning.

ghaff

I certainly did a ton of traveling/speaking/meeting with customers/sometimes late night calls in different time zones/etc. but I still treated it as colloquially a "day job," albeit not really a 9-5 one.

eikenberry

In my experience this is mostly big-company vs. small company cultural differences due almost strictly to size and scaling. Small companies work best when individuals have ownership and large companies with team based ownership. They attract culturally like-minded people.

tikhonj

The highest-agency, highest-ownership team I worked on was at Target of all places. (To be fair, it was not a typical team for the company!) The VP who made that work learned that style of leadership from a decade in Strats at Goldman. Both are pretty big as companies go!

On the flip side, I've seen early-stage startups and scale-ups where engineers did not have real ownership. It's easy to get into a situation where an individual engineer "owns" a specific part of a startup... but can't make any real decisions on it because the founders want to dictate work at a week-to-week level or something.

It's a function of culture, not scale.

spimmy

i'm struggling to see how what you are saying you value is any different from what i am saying i value (author here).

tikhonj

possibly we're saying the same things in different ways, or maybe it's just hard to pin the difference down in a words

what do you mean by "the smallest unit of software ownership and delivery is the engineering team" in practice?

what's the largest scope of work some engineer can do entirely on their own recognizance?

bevr1337

> what's the largest scope of work some engineer can do entirely on their own recognizance?

None. Everything we develop was built on someone else's work. Even when our collaborator is not physically in the room with us, the work is still a collective endeavor.

spimmy

well obviously there's a ton of variance here. but in the type of engineering i'm most familiar with, any sizable amount of production services or surface layer being owned by a single person is a bad thing.

individuals can get sick, go on vacation, etc. having it be owned by a team creates resiliency from a people perspective.

dkkergoog

[dead]

rebeccaskinner

I’ve been lucky enough to work on a number of teams in my career that are full of brilliant high performing people who are also kind, empathetic, and focused working together to deliver something to customers. These days I’m lucky enough to lead a team like that.

Counter to the article, in my experience these are the hardest teams to manage well because organizations typically aren’t set up to deal with them. Larger companies tend to lean into standardization and making things accessible to average engineers in ways that make high performing teams less effective and often demotivated.

I think this is an overly cynical approach that assumes that you can’t invest and grow people into exceptional engineers. In my experience you can if you are willing to invest in it, and the long term benefit of having more high performing engineers who aren’t being restricted from doing good work outweighs the cost of training and growing people.

tikhonj

I definitely agree that the best teams have cultures that make even normal engineers incredibly effective compared to the status quo. Managers definitely put too much stock in hiring and "engineering quality" compared to culture, trust, systems and processes.

> Any asshole can build an org where the most experienced, brilliant engineers in the world can build product and make progress.

But I have to wonder: if "any asshole" can build orgs like that, why don't they? I know of only a handful of orgs that actually manage to build strong teams of really strong engineers, and they're almost exclusively either trading firms or specialized/research-oriented teams. What's stopping everyone else?

I guess this comes down to the initial point in the article: what do we even mean by "productivity"? (Or, the word I'd prefer: "effectiveness"?) There are lots of orgs where only experienced folks can succeed because they're the best at working around and tolerating disfunction. I've seen performance review processes that—intentionally or unintentionally—selected not for some general technical strength/taste/etc, but for the willingness, ability and stamina to put up with management dysfunction. But, to me, that is very much not what I have in mind when I think "top engineer".

bluefirebrand

> But I have to wonder: if "any asshole" can build orgs like that, why don't they

Well the supply of extremely talented engineers isn't limitless so you wind up competing for talent with companies much larger than you building much cooler stuff or the same stuff but paying more

MontyCarloHall

> But I have to wonder: if "any asshole" can build orgs like that, why don't they?

Other posters have said money, but it’s also a function of time—how long can a company afford to wait to find the perfect “unicorn” engineer who excels in every aspect of the product but may take a long time to hire, versus hiring someone whose expertise only covers some of the product’s needs but can be found immediately?

Certain companies benefit disproportionately from unicorns, especially those in interdisciplinary fields. For instance, in quantitative finance, a single individual who’s an excellent 1. systems programmer, 2. mathematician, and 3. financial market expert will contribute a lot more than a team of three specialists in each of those domains. But it’s a lot faster to hire that team of three versus finding the rare individual who can supplant a whole team. This is also true in less exotic fields—it’s rare to find someone who’s truly a “full stack” web developer, with a deep understanding of networking protocols and Linux system administration and (cloud based) distributed systems and databases and caching services and frontend development & cetera. But the company that can afford the money and time to find these people will make a much better product than the company that cannot.

(Whether the product actually needs the quality commensurate with all that engineering firepower is an entirely other question.)

spimmy

because there is, by definition, an extremely limited supply of the "best engineers in the world". if you can afford to pay top-10% salaries, and attract and retain top-10% engineers, more power to you.

maybe not literally "any asshole" can do it -- but it certainly asks more of your leaders to craft sociotechnical systems oriented towards learning and enablement. and i don't think that's a bad thing.

sorokod

Beyond budget constraints, those brilliant engineers may not be good team players.

Their brilliance may be in the way of finding the necessary compromises and doing the required but not intellectually challenging work.

tikhonj

I dunno, almost all the brilliant engineers I've known have also been great team players and mentors. Not all, but I'd say it's pretty correlated.

The reason brilliant assholes stand out is one of those statistical paradoxes whose name I've forgotten: somebody who's an asshole has to be brilliant to succeed, while team players can get pretty far with a wide range of skill levels.

MontyCarloHall

You are thinking of Berkson's Paradox! [0]

[0] https://en.wikipedia.org/wiki/Berkson%27s_paradox

spimmy

ha! ok, i never thought of it quite like that.

burningChrome

>> I know of only a handful of orgs that actually manage to build strong teams of really strong engineers

The best teams I've been on work like a team in sports.

You have the guys who are really good, really sharp developers. They know all the ins and outs of the framework, but are insanely specialized with one thing, but do the majority of the heavy lifting. Then you have the mid-range guys who are more of the JOT guys. They know UI/UX, accessibility, front-end dev and some back-end stuff. Then you have all the entry level rubes. The guys you can give them something to do and they'll figure it out. They're usually learning as they go, but can handle tasks with some direction and hand holding. As the project runs on, they get more comfortable with the processes and tasks so they need less direction and hand holding.

Building teams is all about finding a good mix of people who compliment each other. Too many of the really sharp devs and they'll be arguing over everything. Get too many mid-range or entry level guys and it will slow down the whole project. You also have to have devs who are comfortable with their skill level and know what they're expected to be doing. Too many times I've been on teams where the mid-range guys start bumping heads with the senior devs. Lunch becomes a rage fest over what we should be doing better. They think they should a lead dev, not the guy they don't like.

The last thing is your senior/lead devs have to have the right attitude too. I've been around some insanely sharp lead devs, but they're complete assholes. They know everything, you know nothing. You use shit frameworks, they're always on the cutting edge and your an idiot because you like Angular not something bougie like Svelt.

The key in all of this is finding the chemistry that works. When you get that chemistry, you can capture lightening in a bottle and really build some amazing stuff. When it works, its the coolest thing. People are dialed in, they're enthusiastic about what they're doing. They're willing to work longer hours to make sure the product we're building is incredible. The team is happy, delivering and working on faster sprints and things just feel effortless.

ChrisMarshallNY

> if "any asshole" can build orgs like that, why don't they?

The biggest reason, is that most first-line and middle managers suck. They can't create and maintain a productive environment.

The cure is/was to pay heaps of money. People will put up with almost any kind of hell, for good pay.

AlotOfReading

There are large parts of this I really like, but it's hard to overstate just how much I disagree with the idea that "The only meaningful measure of productivity is impact to the business". The natural result of this view is a focus on quantifiable changes and short term thinking. The greatest value of experienced engineers is in avoiding landmines that will destroy a project or even a company. It's difficult to quantify things that never happen, or communicate the value in avoiding them in terms that don't sound wildly hyperbolic.

Retric

“Impact on the business” isn’t inherently a short term viewpoint.

Maximum impact is a long term proposition.

AlotOfReading

It's not inherently a short term viewpoint, that's just the typical and natural result of adopting this view. Measuring long term impacts is hard. Attributing them is even harder.

Here's a real scenario. I worked for a company that evaluated by impact. They had a cellular modem with known issues in the field. A replacement was designed to fix those issues, but couldn't be backwards compatible. The cost to immediately upgrade the field units was high, so deployment was delayed to a cheaper, later time. One way of looking at this is that the decision saved millions of dollars and that argument was made. After the evaluation and before the deployment, a set of older field units failed in such a way that it made headlines across the country, which would have been prevented by the new units.

So, was the impact of those decisions negative the whole time in an unknowable way? Did their impact become negative as soon as the incident occurred? If the incident was still possible but hadn't occurred, would the impact be different?

People aren't good at evaluating things that haven't happened yet, so they'll tend to focus on the things they can see immediately in front of them. That incentivizes engineers to build things that have immediate short term impacts and discount long tail risks which can't be reliably evaluated.

tikhonj

100%. I've taken to using intentionally fuzzy phrases to describe productivity/impact/effectiveness/whatever. Like, "in a holistic accounting, person X did a great job". It's not necessarily—in fact, necessarily not—easily quantifiable or legible, it requires some human insight and judgement, but so does any complex, creative endeavor. Trying to reduce management to metrics is inherently short-sighted.

esafak

The company should be constantly striving to better quantify the unknown.

spimmy

idk, avoiding landmines seems like pretty significant business impact

tikhonj

it's real impact, but it requires folks to be able to confidently talk about counterfactuals—"we would have wasted X days if not for..."—which I've found to be really hard at most places

the exception was places where leadership already thought in the same terms about software quality/etc, which meant I didn't have to do much convincing :P

how would you build teams or structures to support that sort of holistic thinking about software?

spimmy

i think it has to start with having engineering managers and directors who are deeply technical themselves. the idea that you can split up this work into "managers do the people stuff" and "engineers do the technical stuff" is bananas. it's all sociotechnical work.

this is why i advocate the engineer/manager pendulum so strongly. we get better results when management has strong tech skills (and staff+ engineers have organizational skills as well).

nonameiguess

I'm not sure I totally agree with measuring value by avoiding landmines or anything at all related to project management, but it definitely bugs me to see everything reduced to business impact. There are plenty of things in life that matter to individuals, to humanity, to the entire world at large, that have nothing to do with selling products for money. When I think of the engineers I revere the most, I don't think of titans of post-2001 Silicon Valley so much as John von Neumann, Robert Oppenheimer, Nikola Tesla, Leonardo DaVinci, whoever the hell built the Roman aqueducts and Egyptian pyramids, Babylonians and Mesoamericans who figured out how to predict eclipses.

Did these people have a business impact? I guess Tesla made Westinghouse a lot of money at one point, but that seems far from the most distinguishing thing that made him great at what he did. If anything, he was mediocre at business.

Even if we want to look at current titans of the computing industry, I admire the work done by orgs like Nvidia or humans like Geoff Hinton, but they also just got lucky that what they were doing for completely different reasons ended up benefiting so tremendously from the galaxy-scale data harvesting that has been going on due to the Internet becoming primarily ad-monetized, which they didn't know was going to happen. How many equally great engineers toiled in obscurity on dead ends but did equally great work? Doug Lenat was just as great an AI engineer, if not better, than Geoff Hinton. History just went one way and not the other, due to factors completely outside of the control of either of them.

narag

Some good points, bad logic. The fact that you can't effectively measure something doesn't mean that something doesn't exist.

Individual productivity exists.

Maybe it's easier to measure groups' productivity? Probably.

"Business impact"? I don't think so, that later concept seems much more arbitrary. But feel free to look for the keys under the lamplight. If you choose that metrics, you're not going to retain many extra productive people anyway.

The old problem: judging the work of an expert is very difficult if you lack comparable expertise. I can give you advice, but I can't make you smart to accept it. How could you tell if I'm a genius or an overconfident asshole?

tikhonj

the real problem is not that we can't measure productivity, but that we can't even fully define an abstract productivity metric—what does "productivity" even mean?

we can come up with some notion that talks about the net effect of an individual (the "wins about replacement" of programmers), but that tells us absolutely nothing about how any given individual achieves that; hell, the net effect of any given individual is presumably a function of the entire context and the rest of the org, not just of the individual!

alternatively we can try to define more direct notions of "productivity" even if we can't measure them, but those notions end up varied, multidimensional and, again, painfully context-specific—it's absolutely a useful thing to think about, but it does not let us pin down what a "top 1%" engineer "really" is, or even if that's a meaningful notion

sailfast

A real genius can often explain their work without being an asshole, I think. So it’s fairly straightforward to tell the difference.

null

[deleted]

acedTrex

> The best engineering orgs are the ones where normal engineers can do great work

I quite like this. It is absolutely true as well, not all teams can be 100% rockstars. How well do you enable the average dev to be good, maybe not GREAT but good and reliable.

lizardking

Most great engineers were once good engineers. A healthy engineering organization helps its engineers on this path.

spimmy

yes! love this. and many great engineers will go back to being "good engineers" as they pick up new skills. rinse and repeat++

JohnMakin

> Any asshole can build an org where the most experienced, brilliant engineers in the world can build product and make progress. That is not hard. And putting all the spotlight on individual ability has a way of letting your leaders off the hook for doing their jobs. It is a HUGE competitive advantage if you can build sociotechnical systems where less experienced engineers can convert their effort and energy into product and business momentum.

This hits close to home for me recently. I don't profess to be a 10x engineer, but I found myself on a team with a few people with much less experience and competence than I am normally accustomed to. I started getting every single ticket with any kind of complexity, because I could get it done, and some people on my team couldn't. I could have continued this way, contributing a massive percentage of the output and claimed to be superior to everyone - but the reality is, for me anyway, this is exhausting (and feels really unfair). Plus I am a little lazy (as I believe all good sysadmins/sre's/ops guys are). I want my team to help me. So what I did was work extra for a few weeks and wrote a ton of abstractions over the most complex stuff we do (I dont write software, I write IAC, I'm aware this is a common pattern in software engineering) so that the less knowledgeable engineers could do the work I'd been doing much of. It freed my time up to work on more interesting problems. This was the first time in my career I had to do this without anyone already ordering me to.

I've been on teams where there was someone like me and everyone else was running around behind them frantically trying to keep up or cleaning up the inevitable tech debt that accumulates. It's miserable and really inefficient.

zkmon

At my company, I always prefer hiring average engineers with great attitude. Too many high-credential people with bad attitude were placed on high pedestals due to their perception-creating skills, and became immune to questioning. Once they bag that sweet perception from the boss, they also start bully around. Bosses would usually dismiss any data that goes against their perceptions. Mostly because going by perception is lot easier than looking into the data.

Tade0

> Are you using golang, python, COBOL, lisp, perl, React, or brainfuck?

For years now I had this feeling that people confuse React with the entire frontend ecosystem, but dismissed it thinking that surely they're aware that there's a whole world of non-React frontend out there?

tasuki

> The smallest unit of software ownership and delivery is the engineering team. It doesn’t matter how fast an individual engineer can write software, what matters is how fast the team can collectively write, test, review, ship, maintain, refactor, extend, architect, and revise the software that they own.

Yes, and there are often teams of one. I am currently in such a team. Even though I work on a different problem than other "teams", I think it's reasonably easy to eyeball the relative productivity (not in my favour, in my current circumstances; though, to the credit of my superiors, they're being very patient with me).

ericvolp12

> Make it easy to do the right thing and hard to do the wrong thing.

This is basically the mantra of every platform team I've worked on. Your goal is to make the easy and obvious solution to engineers' problems the "right" one for the sustainability of software and reliability of services.

Make it easy to ship things that are reliable and manage distributed state well and can scale well and engineers will build better muscle memory for building software in that shape and your whole org will benefit.

This will never stop being true.

simonw

Coincidentally, Charity Majors wrote one of my favorite essays about that too: https://charity.wtf/2018/12/02/software-sprawl-the-golden-pa...

> Assemble a small council of trusted senior engineers.

> Task them with creating a recommended list of default components for developers to use when building out new services. This will be your Golden Path, the path of convergence (and the path of least resistance).

> Tell all your engineers that going forward, the Golden Path will be fully supported by the org. Upgrades, patches, security fixes; backups, monitoring, build pipeline; deploy tooling, artifact versioning, development environment, even tier 1 on call support. Pave the path with gold. Nobody HAS to use these components … but if they don’t, they’re on their own. They will have to support it themselves.

paxys

There are plenty of good "normal" engineers whose abilities top out at following direction from management, implementing a spec (albeit really well), adding to an existing architecture etc. I'd wager the vast majority of the industry falls into this category. Yes, it's very important for an org to ensure that these kinds of engineers are successful, because they are the workhorses of your company and without them nothing will get done.

Ultimately though you can't have a workforce just of these engineers. Someone has to lead. Someone has to tell management what to build. Someone has to invent new tech from scratch.

"10x engineer" is a bullshit LinkedIn thoughtfluencer term that has unfortunately caught on, but everyone who has worked in the industry for more than a day knows that there is a hierarchy in the tech org, and the ones on top are more valuable than the rest.

commandersaki

10x engineer in the original sense to me is simply someone that 10x-es your company in some way. Maybe Avie Tevanian who brought Apple OSX/etc. or Andy Rubin who brought Google Android.

ysofunny

> Someone has to invent new tech from scratch.

uhm, as a research university or private company like a startup??