How I Think About Software Estimates, Based On Reality

A few thoughts on estimating. I had a conversation with someone yesterday who asked me how I worked with the engineers on software estimates. My answer shocked him, I think. I wanted to expand here on what was a throwaway conversation:

My Favorite Story About Estimates

First, let’s start with this story about the Sydney Opera House, as told by Nicholas Nassim Taleb in The Black Swan. You should know that construction is incredibly well-understood and for some types of projects (tract homes in a real estate development, for example), builders can repeatedly complete them within 5% of the estimated time.

Sydney Opera House
Software estimates are hard, likely to be just as far off as the estimates were for the construction of the Sydney Opera House.

The Sydney Opera House, started in 1959, was scheduled to be completed in 1963 for $7M (Aus). Actual construction took nearly four times the original estimate – it actually finished in 1973 (10 years late!) – and it cost more than 12 times the original budget at $104M (Aus). And of course, the Opera House was only 1/3 of the original project.

If builders can be that far off, simply because it’s never been done before, why should we think that we should be able to estimate software, which always by definition has never been done before?

Interesting Things Are Hard/Impossible To Estimate

There is a fundamental disconnect between estimates and interesting things. By “interesting” I mean innovative, creating value for a customer, never been done before, new inventions. Interesting things are unpredictable. User stories are estimatable, therefore not interesting.

Software Estimates Don’t Follow A Normal or Standard Distribution (I.e. Not A Bell Curve)

Poisson vs Normal Distribution Software estimates are not a standard distribution. They are a really screwed up distribution where the likely value is way the heck out there beyond the value you think it should be. (And very occasionally, extremely rarely, things go a lot faster than you expect.)

Timeboxes Work Much Better For Managing Interesting Things, Especially In Software

I prefer timeboxes, and for interesting things, we get done what we get done in the timebox. The art of product management is figuring out what to do in the timebox.

Note: this works much better in software than in construction. Buildings have to obey the laws of physics, but software doesn’t. There is no such thing as a Minimum Viable Product in construction – you can’t build a fancy roof until you build the structure to support it. But you can do that in software. There’s a lot of software out there that is essentially fancy roofs floating in the air.

Failure Is Necessary For Learning, Unpredictable, and Not Estimatable

Think about failure, which is so important in innovation. Failure is, of course, immune to estimates by definition.

For example, let’s assume I can get a decent estimate for doing something interesting (which we know I can’t, but hang on). Then we do it, and finish version 1. It only takes twice as long as we estimated! (That’s a great result.)

Unfortunately, given reality, version 1 is not that good, can’t be released, and has to be done again. It was a failure, but it was a productive failure. We learned a lot. We didn’t get the feature to market when we expected to, but if we’d put version 1 into the market, it would have been bad in oh so many ways.

Do it again

So we start doing it again, and mostly we have to start from scratch, but we did learn some things in version 1.

We also realize we can get a little bit of version 2 out to early adopters. It’s definitely not a full feature. Customers have to do manual work to get the value, but they are willing because it’s so useful to them.

And with the cut-down version 2 we learn some valuable new insights via customer feedback. This causes us to cancel the rest of version 2, and move directly to version 3. Versions 1 and 2 are sunk costs, and they are PAINFUL. But because we did them, we have version 3, and it’s beautiful. And it only took us four times as long to get the feature out as originally estimated, which is actually a pretty good result.

What Are Your Thoughts About Estimating Interesting Things?

The title of this post might have been a little misleading. I suspect I may have created a firestorm. I can’t wait to hear what you think!

(See my follow up thoughts in the next post.)

About the author

Your host and author, Nils Davis, is a long-time product manager, consultant, trainer, and coach. He is the author of The Secret Product Manager Handbook, many blog posts, a series of video trainings on product management, and the occasional grilled pizza.

You might also like

  • "Perhaps the main reason we were behind schedule and over budget was because budgets and schedules are based on previous experience with similar projects. We really didn't know how much it would cost to build or how long it would take." — Tom Kelly, Grumman, on building the lunar module for the Apollo program

    Estimates are terribly difficult, particularly when the work is so different from anything your team has done before. That's why dev teams now use RELATIVE estimates–"this" is harder than "that"–instead of time and costs estimates.

    • Exactly right. Think about how Edison might have estimated how long it would take to get a working light bulb to market. Would he have even tried? Or did he just keep working at it, noticing if he was making progress generally (despite thousands of failures on the way), until he was done? (Or until he had the MVP for the lightbulb?)

  • I love the versions 1, 2, and 3. I once worked with a developer who said, "I always do 3 versions of the code. I can do 3 versions by planning for it, or 3 versions by not planning for it and being surprised and behind. I would rather plan for 3 versions." This was back in the '80s.

    He went on to say that he could design one version, code one version, fix one version. He kind of like that, but he didn't like all the fixing. He preferred prototyping one version (his words), throwing that one away, designing one version, coding one version. In other words, learning from his first effort, using that learning to design the "right" version, and then coding it. He wrote quite clean code, and he never gave me an estimate longer than a month.

    He almost always gave me interim milestones of two weeks or less. I was the project or program manager, which was why he gave me estimates. Sometimes he said, "I don't know, but I will know something in two weeks.

    • Johanna – That sounds like a great technique, but one that would be very hard to sell to an executive stakeholder (even though it's reality-based). How did you deal with that?

  • stimates are terribly difficult, particularly when the work is so different from anything your team has done before. That's why dev teams now use RELATIVE estimates–"this" is harder than "that"–instead of time and costs estimates.

  • ▪︎ Before making ANY assessment of ANY project from Sydney Opera House to website development for the neighborhood bowling league, you must find the Root Cause of the cost, schedule, and technical performance shot falls
    ▪︎ The only estimate that is unpredictable comes from Ontological Uncertainty. Aleatory and Epistemic uncertainties ARE predictable. This starts with defining the needed Accuracy and Precision of the estimate. This may not be obtainable because of cost or time, but that’s not that same as unpredictable
    ▪︎ Timeboxes do not remove the uncertainties, they just show the unfavorable variance faster and on periodic boundaries
    ▪︎ Estimating “failure” is the very basis of risk management. “Risk Management of Project Management for Adults” – Tim Lister. Read how risk management is done, and you’ll see forecasting failure is at the heart of good risk management.

    Here are some resources for learning how to estimate agile projects that have been applied in a wide variety of domains

    • Glen – Can you explain a bit more about the different types of uncertainty, and how they impact estimates? And I would love to hear more details about how to effectively estimate projects that have a lot of Ontological Uncertainty (which seems to apply to most interest product development projects, imho).

      Although, despite the amount of writing out there about aleatory, epistemic, and ontological uncertainty, I don’t know if those are the best ways to think about the types of failures Talib talks about. As I mention, the Sydney Opera House example comes from one of his talks (from about the time of The Black Swan). And his point wasn’t about a logical problem, per se, it was about a human cognitive bias. Assuredly, that bias is related to ontological uncertainty – we don’t know what we don’t know – but our ability to imagine the impact of ontological uncertainty leaves us terribly optimistic. His point was that we need better ways that traditional risk analysis to help mitigate risks where the unknowns are massive.

  • {"email":"Email address invalid","url":"Website address invalid","required":"Required field missing"}