Goodhart's law isn't as useful as you might think (2023)

I want to block some time to grok the WBR and XMR charts that Cedric is passionate about (for good reason).

I might be wrong but I feel like WBR treats variation (looking at the measure and saying "it has changed") as a trigger point for investigation rather than conclusion.

In that case, lets say you do something silly and measure lines of code committed. Lets also say you told everyone and it will factor into a perforance review and the company is know for stack ranking.

You introduce the LOC measure. All employees watch it like a hawk. While working they add useless blocks of code an so on.

LOC commited goes up and looks significant on XMR.

Option 1: grab champagne, pay exec bonus, congratulate yourself.

Option 2: investigate

Option 2 is better of course. But it is such a mindset shift. Option 2 lets you see if goodhart happened or not. It lets you actually learn.

I can confirm this. We've standardized Goodhart's law creating a 90-day rotation requirement for KPIs. We found that managers would reuse the same performance indicators with minor variations and put them on sticky notes to make them easier to target.

Just a side note that this usage isn't really the application Goodhart had in mind. Suppose you're running a central bank and you see a variable that can be used to predict inflation. If you're doing your job as a central banker optimally, you'll prevent inflation whenever that variable moves, and then no matter what happens to the variable, due to central bank policy, inflation is always at the target plus some random quantity and the predictive power disappears.

As "Goodhart's law" is used here, in contrast, the focus is on side effects of a policy. The goal in this situation is not to make the target useless, as it is if you're doing central bank policy correctly.

This doesn't feel well elucidated, but I've been thinking about Goodhart's law in other area's of life -- e.g. Owning a home is cool and can enable some cool things. However, when home ownership becomes the goal, it's becomes easy to disregard a lot of life giving things in pursuit of owning a home.

This seems to pop up in a lot of areas and I find myself asking is X thing a thing I really desire or is it something that is a natural side effect of some other processes.

This is all well and good, but unfortunately depends on the people pushing for the metric/system to give a shit about what the metric is supposed to improve. There are still far too many that prefer to slap 1 or 2 careless metrics on an entire team, optimize until they're promoted, then leave the company worse off.

There is a very good essay in the first comment by "Roger" dated Jan-2023, reproduced below. Skip the primary essay and work from this:

"I really appreciated this piece, as designing good metrics is a problem I think about in my day job a lot. My approach to thinking about this is similar in a lot of ways, but my thought process for getting there is different enough that I wanted to throw it out there as food for thought.

One school of thought 9https://www.simplilearn.com/tutorials/itil-tutorial/measurem...) I have trained in is that metrics are useful to people in 4 ways:

    1. Direct activities to achieve goals
    2. Intervene in trends that are having negative impacts
    3. Justify that a particular course of action is warranted
    4. Validate that a decision that was made was warranted

The implication is that the goal is not to find the “right” metrics, but to instead find the most useful metrics to support the decisions that are most critical at the moment. After all, once you pick a metric, 1 of 3 things will happen:

    1. The metric will improve until it reaches a point where you are not improving it anymore, at which point it provides no more new information.
    2. The metric doesn’t improve at all, which means you’ve picked something you aren’t capable of influencing and is therefore useless.
    3. The metric gets worse, which means there is feedback that swamps whatever you are doing to improve it.

One other important point to make is that for many people, the purpose of metrics is not to make things better. It is instead to show that they are doing a good job and that to persuade others to do what they want. Metrics that show this are useful, and those that don’t are not. In this case, of course, a metric may indeed be useful “forever” if it serves these ends. The implication is that some level of psychological safety is needed for metric use to be more aligned with supporting the mission and less aligned with making people look good."

Seems like the headline should be:

Is Goodhart's Law as useful as you think?