Time Spent on Hardening

Given his experience, I'm surprised that the author is surprised that companies don't know how much time they spend on hardening. Nobody gets paid to do that unless necessary for compliance; companies prefer to build features, and don't track this stuff. Don't even think about asking them to quantify the benefit of hardening.

https://www.wiley.com/en-us/How+to+Measure+Anything+in+Cyber...

Imagine result types vs exceptions.

With result types you're expected to consider failure at every step. It's time consuming but when you finally compile successfully and run it you are surprised it ran successfully the first time. But if you're being honest you probably wrote at least some code to handle cases that will never happen.

On the other end of the extreme imagine you only write the happy path and can raise exceptions but you don't catch a single one. Your code compiles successfully very early but every time you run it you get an uncaught exception error with a stack trace letting you know you missed something. You kinda just code, run, get the exception, fix it, run it again, get the next exception, etc in a loop until you're satisfied it runs well enough.

The advantage of this approach is you only spend time handling the sad paths that definitely happen. The disadvantage is if you ship this code your customers might find a sad path case you missed, six months later, and it's much more expensive to fix by now.

What trade-off do you make? The way I see it, it depends on how expensive failure is, how hard it will be to come back to this months or years later, and how easy it is for you to deploy a fix.

This metric is typically tracked internally and probably wouldn't be as public because it could indicate how "buggy" a product is. An easy way to measure this is time spent taking incidents from open -> mitigated -> resolved and treating that as time spent * engineers for amount of impact.

The tricky part would be measuring time spent on hardening and making the business decision on how to quantify product features vs reliability (which I know is a false tradeoff because of time spent fixing bugs but still applies at a hand wavy level)

Also, depending on the system, time spent on hardening is many times happening concurrently with some other tasks.

Maybe you trigger a load test, or run a soaking test or whatever, while that runs you do something else, pause and check results, metrics, logs, whatever.

If something is funky, you may fix something and try again, get back to your other task and so on.

It's messy, and keeping track of that would add significant cognitive load for little gain.

Code that directly affects revenue (e.g. licensed entitlement enforcement) is hardened, quietly and iteratively, based on failure and attacker.

The time spend on hardening software is always zero or very close to that unless the company makes that hardening a selling point of the product they make.

In the world of VC powered growth race to bigger and bigger chunk of market seems to be the only thing that matters. You don't optimize your software, you throw money at the problem and get more VMs from your cloud provider. You don't work on fault tolerance, you add a retry on FE. You don't carefully plan and implement security, you create a bug bounty.

It sucks and I hate it.