You Don't Have Time Not to Test

This struck a nerve.

I've been on both sides of this war: the test evangelist fighting for coverage and the pragmatist shipping to beat a deadline. After 20+ years in software, the truth is painfully obvious: testing is the greatest productivity hack that everyone keeps "postponing until next sprint."

The author gets the psychology exactly right. We overestimate the initial cost and drastically undervalue the compound returns. What they call "Time Technical Debt" is the perfect description for that sinking feeling when you're working on a mature codebase with spotty test coverage.

The most insightful point is how testing fundamentally changes your design for the better. When you have to make something testable, you're forced to:

- Think about clear interfaces

- Handle edge cases explicitly

- Create clean separation of concerns

- Build proper startup/shutdown sequences

These aren't "testing best practices," they're just good engineering. Testing is simply the pressure that forces you to do it right.

My experience: if your system is hard to test, it's probably hard to reason about, hard to maintain, and hard to extend. The difficulty in testing is a symptom, not the disease.

At my last company, we built a graph of when outages occurred versus test coverage by service. The correlation was so obvious it became our most effective tool for convincing management to allocate time for testing.

If the way you write automated tests is slowing you down rather than speeding you up, you need to rethink the way you are writing tests.

In my opinion tests are the ultimate productivity hack for building software. In exchange for a relatively small up-front cost (if you know what you are doing) they let you move so much faster! You can make changes with confidence that each change won't break your existing functionality, without having to spend hours on manual testing for every change you want to ship.

This goes for smaller personal projects too. I gave a talk about this a few years ago, originally titled "Massively increase your productivity on personal projects with comprehensive documentation and automated tests" but then renamed (after I'd pitched the talk but before I presented it) to "Coping strategies for the serial project hoarder". Video, slides and notes here: https://simonwillison.net/2022/Nov/26/productivity/

So I joined this company as a lead of a team of 4 developers. I found a medium size Java codebase with not a single test. Instead of unit tests, we had a list of scenarios, and at the end of each sprint a QA guy would manually go through each scenario and verify the software.

One of my first moves was to ask the developers to write tests for their code. I got terrible pushback, specifically from one of them that said the dreaded "we don't have time to write tests".

This went back and forth a few times until it got escalated to the lead of the whole project, who sided with the developer: writing tests takes too much time and we don't have enough.

Two days. It took me two days of my own time to go through a completely unfamiliar codebase and get a reasonable code coverage with unit tests.

The benefits were immense and immediate. For starters, we caught a bunch of regressions before a new release that might have taken days of manual testing by QA. We were also able to ship faster and with more confidence.

As OP say, if you don't have time to test, you certainly don't have time not to test.

During my several decades of translating businesses logic into code I never found a test that does not pay back the time spent to write it many times over, they are truly the super-power of the trade mainly because the can be completely automatized. Because of this automatic nature from a business perspective their marginal cost becomes almost zero (pretty much regardless of the initial cost and maintenance cost), the reason for this is that the value function of the business logic embedded in code, the reason for the code to even exist (i.e. the thing that gives a value to the piece of software), needs to account in the long run for the risk components, risks generated by defects or poorly aligned specifications, security risk, reputation risk, obsolescence, performance, etc. The way to manage risk at the business level is by somehow hedging and this hedge is precisely testing, again and again, hopefully automatically. Testing is then not a cost, it's precisely the instrument that allows to stop paying over and over for those risks and therefore in the end capture more value and reduce the variance of the outcomes.

People also mentioned side projects and testing. What I found personally is that tests are the best way to also document/encode ideas for later so that when you pause something for months because... life, getting back to a project becomes easier. I found very useful for instance to add some tests that would just fail for some feature easy to implement just to have an easy entry point back into the project.

Tests can be good. Tests can be bad.

  - If you are writing a heavy algorithmic code, then write unit tests.
  - If you are writing glue code joining services, then write integration tests.
  - Integrating with third-party services means you prefer circuit-breakers and production monitoring over tests.

> “We don’t have time to [write automated] test[s],” stated the principal engineer… This was in response to my candid observation that test coverage was low, existing tests didn’t always run locally for mysterious reasons, integration tests were non-existent, and pull requests could merge without passing the few tests that did exist.

Given the state of the existing automated tests, perhaps the Principal Engineer is correct… Why would the new automated tests be any better?

I'd love to hear how people write tests for ML. When I'm doing a greenfield project with a new model, a lot of issues are very statistical, e.g. incorrect downsampling - the model will run and train, just less optimally than normal.

I can't put optimality bounds because I don't know how well the model _should_ train, and when it doesn't train, that's not necessarily because of an incorrect implementation. And, actually training a model for a test is quite resource and time heavy.

Tests are a superpower with hardware development. It can be difficult to find the best boundaries to mock, but once you do it's amazing how much easier it is to develop! I find that tests are a great way to get the software into states that are hard to recreate in real life, such as faults as well

Sometimes you're writing code because you want to see what happens. Richard P. Gabriel wrote a wonderful, moving essay about this recently, "I Throw Itching Powder at Tulips": https://www.dreamsongs.com/Files/Tulips.pdf

> Software engineering is not what I do when I program. I am programming. I write software as part of doing science. I use software as a machine or instrument to explore how the mind / brain might work. Not when the mind’s thinking—that’s old AI. I mean when people are creating.

I find that it's pretty common when I'm, say, tweaking gameplay in a game, that I have no trouble making the code behave the way I expect, but impossible to predict ahead of time how I will feel about different possible behaviors. I have to try them on to see how I feel in them. Adding tests to that feedback loop doesn't help; it just slows it down, dramatically slowing down the process of learning and exploration.

On the other hand, when I am writing code to meet someone else's needs rather than as a form of self-expression or exploration, I usually want the code to behave in a specific way, at least in certain situations, and to keep behaving that way as I make other changes to it in the future, unless I change my mind. In those cases I think it's very rare for automated tests to slow me down instead of speeding me up.

In languages with expressive static type systems like OCaml, the crossover happens a little later, because the static type system already constrains the program's behavior a lot. But automated tests still become very valuable very soon when I want a specific behavior.

I've also, unfortunately, worked on projects where the test suite we had was worse than nothing. Invariably it wasn't because we were throwing itching powder at tulips so that the optimal test suite would be empty; it was because we'd done a bad job writing the tests.

I remember asking folks for tests for their stuff, and they didn't want to do it because it would take too much time and effort. And I said "It doesn't have to be perfect, just do a simple sanity check".

It's possible people on the opposite ends of the perfectionism scale might view tests differently.

The perfectionists want 100% test coverage which takes a serious commitment of time and effort, while the fast-and-loose folks don't want to bother with tests at all.

I wonder if the best way to test is to have the fast-and-loose people make fast-and-loose tests, and then let the perfectionists iterate on the tests as time goes on.

Depends on your toolset. For some kind of code, testing is the solution and not a problem. For React before version 17 between the test code never being able to know if your callbacks have settled and react-testing-library insisting that you aria-everything or screw up your code with test id’s, writing a test is hours of wasted time now and more hours wasted gradually over the next year because your tests already take 70 sec to run. React always knew what was queued to run and why it took them 17 releases to expose it to test runners is beyond me.

Context matters.

If you are a startup without customers/users then writing tests is a bad idea - you need to focus everything on getting working code and users. You may be throwing your code away because you still don't even know what your users really want.

On the other hand if you are writing a banking system then sure, testing is not negotiable.

"Should we be writing tests?" cannot be answered without also knowing what the software is, what the company does and the context in which the software is being written.