Bias Testing Beyond Demographic Parity

Bias Testing Beyond Demographic Parity

A hiring model advances 22% of male applicants and 21% of female applicants. Demographic parity holds. The fairness section of the model card is one paragraph long, it contains that number, and it is filed.

The same model, on the same data, has a false-negative rate of 8% for one group and 19% for the other. Qualified candidates in the second group are more than twice as likely to be rejected. The overall advance rates match because the model is also advancing more unqualified candidates from that group — which is not fairness, it is compensating one injustice with another.

Both statements are true. Both are computed from the same confusion matrix. Only one of them was in the report.

The metrics do not agree, and cannot

The central fact of algorithmic fairness — the one that most bias programmes are structured to avoid confronting — is that the common fairness definitions are mutually incompatible. Except in degenerate cases (a perfect classifier, or base rates that are identical across groups), you cannot satisfy them simultaneously. This is not an engineering limitation to be overcome with a better model. It is arithmetic.

Three families, and what each one actually asserts:

Independence (demographic parity). The selection rate is equal across groups. Asserts: outcomes should be distributed equally, regardless of any measured differences between groups. Ignores whether the prediction is correct.

Separation (equalised odds, equal opportunity). The error rates are equal across groups — the same true-positive rate, and in the strong form the same false-positive rate. Asserts: the model should be equally accurate for everyone. Permits unequal selection rates when base rates genuinely differ.

Sufficiency (calibration, predictive parity). A given score means the same thing regardless of group — of everyone scored 0.8, the same proportion are positive in each group. Asserts: the score is an honest probability for everyone.

The diagram's conclusion is the point of this piece. There is no metric that resolves the trade-off, because the trade-off is not empirical. It is a question about what kind of unfairness your organisation is prepared to defend, and it should be answered by the people accountable for that — in writing, with reasons — and not silently, by whichever metric an engineer found first in a fairness library.

Choosing, and defending the choice

The choice is not arbitrary. The decision context tells you a great deal.

When the cost of a false negative falls on the individual and is severe — a qualified loan applicant denied, a sick patient triaged as well, a candidate screened out — separation is usually the defensible frame. Ask: is the model equally good at not missing people, in every group? Equal opportunity (equal true-positive rates) is often the specific metric that matches the harm.

When the score is handed to a human who will act on it as a probability — a risk score shown to a caseworker, a propensity shown to an underwriter — sufficiency matters intensely. If a 0.7 means something different for one group than another, every downstream human decision is systematically distorted, and the humans have no way to know.

When the outcome is a scarce allocation and the historical base rates are themselves a product of discrimination — this is where independence has its strongest claim. If your training labels encode who was hired in a discriminatory decade, a model that is beautifully calibrated to those labels is beautifully calibrated to the discrimination. "The base rates differ" is sometimes a fact about the world and sometimes a fact about the record-keeping, and telling those apart is not a statistical question.

Write down which one you chose, why, what you gave up, and who signed it. That paragraph is worth more to a regulator than any number you can compute.

The measurement problems nobody mentions

Intersectionality. A model can pass every fairness test on sex and every fairness test on race and fail badly for women of a particular ethnicity. Marginal fairness does not imply joint fairness. This is not a corner case — it is where the largest disparities are typically found, and it is invisible to any programme that tests one attribute at a time.

The cost is combinatorial: two attributes with four levels each give sixteen cells, and the cells you most need to examine are the smallest. Which leads directly to the next problem.

Small subgroups and statistical power. A subgroup with 40 members cannot support a stable error-rate estimate. The confidence interval will be enormous and the point estimate will bounce. Two failure modes follow, and teams commit both: reporting a subgroup disparity that is pure noise, and — far more common — concluding "no significant disparity" from a sample that could not have detected one. A null result from an underpowered test is not evidence of fairness. It is an absence of evidence, and it must be reported as such. State the minimum detectable effect for every subgroup you tested. If you cannot detect a 15-point gap, say so.

Proxy attributes. Removing the protected attribute does not remove the bias; it removes your ability to measure it. Postcode, name, school, device, employment gap, phrasing register — the information reconstitutes itself from correlates. Fairness-through-unawareness is a way of not knowing.

Missing labels. Frequently you do not hold demographic data, for good legal reasons. The options — inference from proxies, voluntary collection, third-party enrichment — all carry their own risks, and choosing to not measure is also a choice with consequences. It is defensible only if you say out loud that you have chosen it.

Generative systems: bias without a confusion matrix

Everything above assumes a classifier with labels. Most of what is being deployed now does not have one, and the fairness literature's metrics simply do not apply to a system that writes text.

Bias in a generative system shows up as differential quality, tone, and treatment rather than differential rates:

• The assistant's answers are shorter, hedged, or less specific when the query mentions certain names or dialects.
• The tone shifts — more formal, more cautious, more presumptuous — with the perceived group of the user.
• Refusals are asymmetric: the same request is declined in one framing and served in another.
• The system's representations in generated content default to a narrow demographic unless prompted otherwise.
• A CV summariser emphasises different attributes depending on the name at the top.

None of these are measurable with a fairness library. They are measurable with matched-pair testing under a rubric: construct input pairs identical in every respect except the attribute under test, have qualified people score both against dimensions for quality, tone, specificity, and helpfulness, and measure the difference. This is the audit-study method borrowed from social science, and it is the only technique that reliably surfaces this class of harm.

It is inescapably human work. The difference between "warm" and "condescending," between "cautious" and "dismissive," between a summary that highlights leadership and one that highlights diligence, is a judgement — and it is a judgement that requires consistency across evaluators to mean anything. Rubric-driven scoring with anchored scale points, gold tasks to calibrate the evaluators, inter-rater agreement measured on overlapping assignments, and adjudication where credentialed Evaluators disagree — that machinery is not bureaucratic overhead here. It is what converts a set of impressions into a finding that will survive being challenged.

What to do tomorrow

Compute the other metrics. If your fairness report contains one number, add the false-negative rate by group, the calibration by group, and the selection rate by group, and put all three in the same table. The disagreement between them is the finding.

Then do three things that cost nothing but honesty. Name the fairness definition you have chosen and the harm it prioritises, and get it signed by someone accountable. State the minimum detectable effect for every subgroup, so that "no disparity found" cannot be mistaken for "no disparity." And run at least one intersectional cut, because that is where it will be.

A single fairness metric on a slide is not a fairness programme. It is the one number that happened to look acceptable.