We investigated Amsterdam's attempt to build a 'fair' fraud detection model

What nobody seems to talk about is that their resulting models are basically garbage. If you look at the last provided confusion matrix, their model is right in about 2/3 of cases when it makes a positive prediction. The actual positives are about 60%. So, any improvement is marginal at best and a far cry from ~90% accuracy you would expect from a model in such a high-stakes scenario. They could have thrown a half of cases out at random and had about the same reduction in case load without introducing any bias into the process.

A part of me wonders how do such unsatisfactory projects get greenlighted and kept at all. I can imagine a higher-level official who knows almost nothing of machine learning get bamboozled by a bright-eyed university graduate who has only worked on toy projects before. After some time, everyone is afraid to admit that the project has basically failed, but it gets pushed through anyway to save face. There needs to be a process to get rid of such crap fast and without blame, just so that powers that be don't mount unreasonable opposition to removing something that is provably garbage and a waste of tax money.

> But the model designers were aware that features could be correlated with demographic groups in a way that would make them proxies.

There's a huge problem with people trying to use umbrella usage to predict flooding. Some people are trying to develop a computer model that uses rainfall instead, but watchdog groups have raised concerns that rainfall may be used as a proxy for umbrella usage.

(It seems rather strange to expect a statistical model trained for accuracy to infer and indirect through a shadow variable that makes it less accurate, simply because it's something easy for humans to observe directly and then use as a lossy shortcut or to promote alternate goals that aren't part of the labels being trained for or whatever.)

> These are two sets of unavoidable tradeoffs: focusing on one fairness definition can lead to worse outcomes on others. Similarly, focusing on one group can lead to worse performance for other groups. In evaluating its model, the city made a choice to focus on false positives and on reducing ethnicity/nationality based disparities. Precisely because the reweighting procedure made some gains in this direction, the model did worse on other dimensions.

Nice to see an investigation that's serious enough to acknowledge this.

Congrats Amsterdam: they funded a worthy and feasible project; put appropriate ethical guardrails in place; iterated scientifically; then didn’t deploy when they couldn’t achieve a result that satisfied their guardrails. We need more of this in the world.

> A more concerning limitation is that when the city re-ran parts of its analysis, it did not fully replicate its own data and results. For example, the city was unable to replicate its train and test split. Furthermore, the data related to the model after reweighting is not identical to what the city published in its bias report and although the results are substantively the same, the differences cannot be explained by mere rounding errors.

Very well written, but that last part id concerning and point to one part: did they hire interns? How cone they do not have systems? It just cast a big doubt on the whole experiment.

A big part of the difficulty of such an attempt is that we don't know the ground truth. A model is fair or unbiased if its performance is equally good for all groups. Meaning e.g. if 90% of cases of Arabs committing fraud are flagged as fraud, then 90% of cases of Danish people committing fraud should be flagged as fraud. The paper agrees on this.

The issue is that we don't know how many Danish commit fraud, and we don't know how many Arabs commit fraud, because we don't trust the old process to be unbiased. So how are we supposed to judge if the new model is unbiased? This seems fundamentally impossible without improving our ground truth in some way.

The project presented here instead tries to do some mental gymnastics to define a version of "fair" that doesn't require that better ground truth. They were able to evaluate their results on the false-positive rate by investigating the flagged cases, but they were completely in the dark about the false-negative rate.

In the end, the new model was just as biased, but in the other direction, and performance was simply worse:

> In addition to the reappearance of biases, the model’s performance in the pilot also deteriorated. Crucially, the model was meant to lead to fewer investigations and more rejections. What happened instead was mostly an increase in investigations , while the likelihood to find investigation worthy applications barely changed in comparison to the analogue process. In late November 2023, the city announced that it would shelve the pilot.

Key point:

The model is considered fair if its performance is equal across these groups.

One can immediately see why this is problematic, easily by considering equivalent example in less controversial (i.e. emotionally charged) situations.

Should basketball performance be equal across racial, or sex groups? How about marathon performance?

It’s not unusual that relevant features are correlated with protected features. In the specific example above, being an immigrant is likely correlated with not knowing the local language, therefore being underemployed and hence more likely to apply for benefits.

Amsterdam reduced bias by one measure (False Positive Share) and bias increased by another measure (False Discovery Rate). This isn’t a failure of implementation; it’s a mathematical reality that you often can’t satisfy multiple fairness criteria simultaneously.

Training on past human decisions inevitably bakes in existing biases.

I have a growing feeling that the only way to be fair in these situations is to be completely random.

Why is there so much focus on "fair" even when reality isn't?

Not all misdeeds are equally likely to be detected. What matter is minimizing the false positives and false negatives. But it sounds like they don't even have a base truth to be comparing it against, making the whole thing an exercise in bureaucracy.

The article talks a lot about fairness metrics but never mentions whether the system actually catches fraud.

Without figures for true positives, recall, or financial recoveries, its effectiveness remains completely in the dark.

In short: great for moral grandstanding in the comments section, but zero evidence that taxpayer money or investigative time was ever saved.

Does anyone know what they mean by reweighing demographics? Are they penalizing incorrect classifications more heavily for those demographics, or making sure that each demographic is equally represented, or something else? Putting aside the model's degraded performance, I think it's fair to try and make sure the model is performing well for all demographics.