Estimates are difficult for developers and product owners

And I'd add that the need for them is a sign they aren't worth doing.

As you say, worthwhile software is usually novel. And to justify our expense, it needs to be valuable. So to decide whether a project is worth doing, we're looking at some sort of estimate of return on investment.

That estimate will also, at least implicitly, have a range. That range is determined by both the I and the R. If you don't have a precise estimate of return, making your estimate of investment more precise doesn't help anything. And I've never seen an estimate of return both precise and accurate; business is even less certain than software.

In my opinion, effort put into careful estimates is almost always better put into early, iterative delivery and product management that maximizes the information gained. Shipping early and often buys much clearer information on both I and R than you can ever get in a conference room.

Of course all of this only matters if running an effective business is more important than managerial soap opera and office politics. Those often require estimates in much the same way they're required from Star Trek's engineers: so the people with main character syndrome have something to dramatically ignore or override to prove their dominance over the NPCs and material reality.

From another thought-experiment-y perspective:

Say you have problem A to solve. Then either one of those is true:

1) it has been solved before, ergo by virtue of software having a zero cost of copying (contrary to, say, a nail, a car, or a bridge), so there is no actual problem to be solved.

2) it hasn't been solved before, ergo it is a new problem, and thus at any moment you may turn a stone and discover something that was not foreseen (whether they are rabbits, yaks, bikesheds, dragons, or what have you eldritch horrors) and thus of unknown cost.

Any task that cannot be obviously fit into one or the other can nonetheless be split into an assembly of both.

Thus any attempt at estimates is as futile as gambling to win, tasks are only ever done when they're done, and "successful estimators" are kings of retconning.

It's all make-believe.

I am in a project where we have to give estimates in hours and days.

Needless to say we always underestimate. Or overestimate. Best case we use the underestimated task as buffer for the more complex ones.

And it has been years.

Giving estimations based on complexity would at least give a clear picture.

I honestly don’t know what the PO and TL gains with this absurd obscenity.

Another reason is that figuring out what the software to be written should actually do, and how it should work, is work that is part of the project and the time it will take needs to be estimated.

As well as the actual development work that will result, which isn't known yet at the time of estimation.

"... anybody with any brains has already left town..." -- Bob Dylan

Anybody with any business experience has already isolated themselves from the certainty of software project failure, where "failure" is a euphemism for "late." So it doesn't matter if software can't be estimated.

This can be nerve-wracking to a beginner, but one gets used to it over time.

Very insightful. This is also my go-to argument for why software engineering is real engineering.

That's the point: how can you tell WHEN you are going to reach a place you've never been before traveling an uncertain path?

Making mistakes over and over again. And adding a lot of time buffers just to be safe

I have noticed the same for a lot of long-running software projects. The estimate is created at the start an never revised.

Projects can and will fail or run late; but heck; a 6-months projects cannot found late after 5 months and 29 days; things must be discovered early, so that the most important issues can be addressed.

Exactly. And I’ve found this is incredibly hard to get people to do, developers especially. No one wants to say a task is going over / taking longer than expected but will gladly run over. It’s hard for pms to resource if estimates are not updated as scope and complexity changes.

Also doesn’t help when estimates become due dates.

Which if you try to do - those agile people will kill you for it.

They wrangle a number out of you which goes into an user story estimate, which feeds into a Gantt chart they use to make their pretty powerpoints they present to upper management to say that the feature will make it into the Q4 release.

If you move this number around the whole estimation will crumble, not that it wont in real life but you deprave them of two things - an illusion of control and somebody to blame when things go south.

This is why stuff like Jira is so polarizing for many developers. It is an additional translation tax on me, the developer, to have to go constantly keep this tracking tool thing up to date so that the product owners/managers stay off my nuts. The burden of effort is placed on me to make their tool functional that they can have what they need to provide roll up status. This tool virtually never benefits me, it's always a time consuming sidequest that breaks focus and scatters important information all over a kanban board where tickets disappear and makes it really hard to cohesively work as a unit to get anything done. Like many orgs we've had PM's and stuff try to mandate expectations on when those updates should be provided, somehow completely oblivious to the human nature of development: you should never expect that a developer can simply hop in and out of moments of deep focus and suddenly provide a bunch of updates regularly when it's convenient for you, as if that has zero impact on their work. It takes a toll, full stop. I won't say no to it, but I do expect PM's to know that style of PM'ing is effectively asking me to make your job easier by populating your tool with updates that are useful to you and not me because I'm well aware you aren't involved enough to assess where things are yourself.

We've gone through so many cycles of how to use Jira well at my org where these frustrations are shared and we try a different approach, and we're finally starting to converge on the idea that this has historically been a little too lopsided requiring too much tax on the developer doing the actual work. We agreed on a new approach that has actually been pretty awesome: the product owners or managers that are trying to direct a body of work must be a little more in the trenches with us to have an overall understanding of where the different pieces are in a moving body of work. We don't expect them to understand the nitty gritty work in the trenches, but at the same time no more 30,000 foot view product managers who just ask for status updates at EOD. _Everyone_, developers included, is responsible for keeping documentation up to date as we go. So we have central working-bodies of information to find details without having to cruise thru 100+ jira tickets to find details we're looking for. The expectation is that they're engaged enough with development whether in chat or on meetings that if they were blindsided by an executive asking for an update, they could speak to it with some authority over at the water cooler without having to go check Jira. This has really helped weed out the lazy product owners/managers, has forced them to thoughtfully consider their meeting schedules, and has placed the exceptional ones in the pod of work being done and really added a lot of velocity and together-ness about the things we're pushing along.

This approach we're using now was born out of some hurt feelings from projects that didn't go so well & we had to have some real restrospective convos where everyone aired out their beef. Those are good convos to have, I think a lot of teams would find that people aren't deceptively trying to screw you over. Being encouraged to level set human-to-human genuinely is one of the greatest parts of working where I work. Always walk away from those types of chats learning valuable things: for the most part our product owners really do care. Not just about their career aspirations but also about _us_ nerdy and sometimes socially maladjusted developers. They look forward to working with us, and they want to make this as easy as possible for themselves but also for the developers. In the past they spent a lot of time in planning phases trying to scaffold out a project in Jira and attaching timelines to it so that their needs are met to provide predictable timelines to their bosses... but also with the hope that by plainly outlining work it sort of 2 in 1 satisfies our needs and will make development timelines a breeze. We've had to ask them to cede rigidity on that latter part because even the software architects admit the work being done is often a moving target. And when that target moves, maybe you realized you need to add a pivotal software solution to the stack, you can sometimes throw like 45 planned tickets into the dumpster. New ship dates need to be assessed. This was our reality check that we were all collectively shit at adapting to the dynamic nature of any given project. Now, our product owners decided that the expectation they have of their own role is that they understand this dynamic and are prepared and willing to make the case about why the shipping timeline must change. So there's actually a pain point solved here: don't break your back doing so much up front work to try and guess/capture what the body of work might look like only for it all to possibly get thrown away, involve architecture a bit more in the planning phases, but most importantly let's engage throughout the project and we'll try our best to have shared ownership/interest in making sure where we are in the project is broadly understood by everyone involved.

We're currently in phases of implementing a major platform right now and it's just night and day better and dare I say fun. We're still keeping Jira up to date, but the product owners and PMs are more or less managing this as it is a tool they find useful. Removing the "can you update this ticket please" 24/7 has forced them to be a little more involved and have the right chats, but also makes us developers happier to jump in and get it updated on our own volition because we also want to help them have an easier time. If my PM pings me and says "hey I'm looking at this ticket that's stuck in blocked, I just wanted to make sure we got an update from so-and-so about provisioning this credential so I can follow up if needed" I will likely automagically jump in and be like "still stuck, but let me update that ticket for you there's a couple key details I want to make sure our there for you before you reach out". There's an inherent "we're both interested in seeing this ticket through" here that doesn't strike a nerve with either party. Pretty much everyone involved both developers & non developers has a really solid read on where anything's at and we're all just talking a lot more. And for developers I find that it's really good, even if you've got them committed in one narrow body of work, to understand the larger pieces in motion. When they're in tune with the broader orchestration of a projects timeline, they tend to weigh in during unsuspecting moments that might tie seemingly unrelated pieces together. They might be assigned to work on x, but in a group chat about y they notice y has a dependency on x and they'll speak up and call out the need to test that they both work together. We've had a lot of great callouts materialize like this, and on a human-psyche level I think it snowballs & avalanches encouraging developer participation in a way that is really meaningful for PMs. It's interesting that Jira & the expectation that predicting development time in an arena of uncertainty was previously in the way of forming the group dynamics we have now. Jira, despite just being a tool, can really amplify a lot of bad outcomes when it's used by those who aren't fit to be near development, it sort of devolves into a two dimensional behind-schedule tracker that detrimentally impacts how team members on the ground communicate with each other.

And since we're talking a lot more there's just like... way more memes. And memes are important in any development timeline. We prioritize laughing through the pain together.

Sorry, but how does Kanban come into this?

From your description, SCRUM, could work just as equally.

Don't get me wrong, I'm a fan of Kanban, it's awesome for visually presenting where bottlenecks are for tasks, but estimations aren't a feature of that.

But SCRUM maybe, where people are having a sprint planning meeting maybe more what you're thinking?

Another reason kanban doesn't work for large projects is because you have to coordinate your cycles with multiple dependencies teams roadmaps and releases.

> technically it is irrational

Only if your company operates in a vacuum, without investors or customers

> E = Equilibristic estimate. This is based on success as 50% likely such as for critical chains and simulations.

I've found giving probabalistic estimates to be hopeless to effectively communicate, even if you assume the possible outcomes are normally distributed, which they aren't.

Three-point or PERT estimates are in the same vein, but are just an old and established business process concept and not a trademarked "project" from a (defunct?) consultancy.

https://en.wikipedia.org/wiki/Three-point_estimation

https://projectmanagementacademy.net/resources/blog/a-three-...

And yeah, they can be useful when you can must put numbers to things and can decompose the work into familiar enough tasks that your estimation points are informed and plausible. Unfortunately, in a field often chasing originality/innovation, with tools that turn over far too often, that can be a near-impossible criteria to meet.

This sounds like that if you don't trust a fortune teller, you can mitigate it by going to 4 different fortune tellers, and then somehow combining their predictions into your much more certain future.

So now you have 4 wrong estimates to work with :) To have some predictability you should have small features. That's the only thing that can give you strong estimates. No industry has solved giving strong and correct estimates - it's in the name, it's estimation!

One interesting angle for me is that I am seldom given complete specs or requirements when asked for an estimate. Of course you ask questions to try to determine key information that has not been specified but often the answers are not available or fully reliable.

So any estimate has to include uncertainty about _the scope of the work itself_ as well as the uncertainties involved in delivering the work.

The natural follow on question when you present a range as the answer to an estimate is: what would help you narrow this range? Sometimes it is "find out this thing about the state of the world" (how long will external team take to do their bit) but sometimes it is "provide better specs".

This is what I do but I don't try to make it complicated with too many numbers. "2 weeks but there's a 10% chance something bad happens and it takes longer".

I have no problem if they just hear the "2 weeks" part. If they come complaining in 3 weeks I just say "we hit that 10%".

The other important thing is to update estimates. Update people as soon as you realise you hit the 10%. Or in a better case, in a week I might be able to say it's now "1% chance of taking more than a week".

> The numbers actually have agreed-upon meaning

Theoretically, yes, but some managers go blank when given a hard concept like a probability distribution.

What's wild about software is we don't do 1/10th as much planning as "real" engineering projects do, and also find ways to create problems akin to the ones they run into from interfacing with an unpredictable real-world environment, by inventing ways for all our platforms and libraries and such to be insanely unstable.

If you're not subject to the batshit insanity of the broader software market, and you crank the planning up a little closer to what "real" engineering does, software delivery gets extremely predictable. See: NASA's Space Shuttle software development processes.

(There are actual, not self-inflicted, problems with software development that "real" engineering doesn't see, though—for one thing, you can't on a whim completely restructure a building that's already been built, and for another, you generally don't have to design real-world engineering projects to defend against the case of intentional sabotage by someone who's just e.g. using the bathroom—it may happen, but it's rare and short of wacky Mission Impossible type plans, busting a water pipe in the bathroom isn't going to get you access to super-secret documents or anything like that)

Physical real-world projects include a buffer for this, called "contingencies" or "change orders" so that if a requirement changes or they discover something like previously unknown site geology that will require changes to the foundation they can absorb it. Based on a large history of similar projects their estimates are usually pretty good but occasionally they will run over.