Bias and Fairness: First Choose the Relationship to Protect, Then Compute Group Differences

A common intuition is: as long as the number of men and women in the training set is equal, the system will naturally be fair. This intuition does not hold up. Balanced sample sizes cannot address the discrimination carried by the hiring history itself, nor the measurement error in performance labels, differences in how job advertisements reach different groups, and unequal opportunity distribution after hiring. Equal male-to-female ratios at the data level only give the model an equal number of samples from the two groups; it cannot erase the inequalities already written into real-world decisions behind those samples.

A more accurate understanding is that bias is not a single bad parameter inside the model that can be unscrewed separately, but a causal chain that runs through the entire system. It may arise from problem definition—the objective originally chosen to be optimized itself excludes a certain kind of harm; from sampling—some groups are systematically underestimated or overestimated in the data; from labels—labels record past decisions rather than true ability; from features—feature encodings hide proxy variables for sensitive attributes; from the objective function—the loss function treats errors for different groups the same or differently; from thresholds—the same score is cut off by different thresholds for different groups; from the interface—display order and copy affect how users interact with the system; and finally from feedback after deployment—the decisions made by the system change the distribution of subsequent data, which in turn reinforces the original bias.

Therefore, before calculating any metrics, you must first spell out three things: who the affected people are, what decisions the system actually makes, and what benefits and harms each type of error brings. Missing a qualified candidate and wrongly admitting an unqualified candidate have different costs; the candidate's perspective and the employer's perspective are also different. Folding these two types of errors into a single number will obscure the differences that truly need to be weighed.

This determines the essence of fairness analysis: it is half normative choice and half empirical measurement. Normative choice answers "what relationship we hope to protect and at what cost we are willing to do so"; empirical measurement answers "to what extent this relationship is violated in real data". No single metric can replace value judgment. Metrics can only quantify the value judgment already made, but cannot make that judgment for us.

Focusing only on overall metrics such as overall accuracy can obscure the very different experiences of two groups. Let’s illustrate with screening results for two groups of 100 candidates each.

Group A has 40 truly qualified candidates and 60 unqualified. The system passed 32 of the qualified candidates while mistakenly passing 12 unqualified ones. Therefore, Group A’s true positive rate TPR_A = 32/40 = 80%, false positive rate FPR_A = 12/60 = 20%, and pass rate is 44%.

Group B has only 20 truly qualified candidates and 80 unqualified. The system passed 12 qualified candidates and mistakenly passed 8 unqualified ones. Group B’s true positive rate TPR_B = 12/20 = 60%, false positive rate FPR_B = 8/80 = 10%, and pass rate is 20%.

Looking at the two groups side by side, the differences immediately emerge. In terms of opportunity, more qualified candidates in Group B are missed: only 60% of qualified candidates are let through, compared with 80% in Group A. This gap can be measured by the equal opportunity gap:

Equal opportunity gap = TPR_A − TPR_B = 0.80 − 0.60 = 0.20

In terms of error rates, the direction is opposite: Group A’s false positive rate is 20%, while Group B’s is only 10%, meaning that the proportion of unqualified candidates mistakenly accepted is higher in Group A. The pass rates are also very different: 44% for Group A and 20% for Group B.

The two groups’ respective "unfairness" points to different subjects. Group A bears more cost from false acceptances, while Group B bears more cost from missed detections. Faced with this table, asking "which group is fairer" has no single answer; you must first state what you prioritize protecting: preserving everyone’s opportunity so that true positive rates are consistent across groups; or making error rates consistent across groups; or requiring that individual risk scores themselves have comparable meaning. Choosing different protected objects yields different fairness definitions and different answers.

The table in the previous section shows that different protected subjects correspond to different fairness definitions. There are three common ones, plus one calibration requirement related to the meaning of risk scores; each protects one relationship and ignores other things.

Demographic parity requires that P(pass | group) be similar across groups, i.e., that pass rates for different groups are roughly the same. It protects the outcome of resource allocation: regardless of background, the proportion allowed by the system to proceed to the next stage should be similar. It is suitable for focusing on quotas and opportunity allocation at the outcome level. Its cost is that it looks only at outcomes, not qualifications—it ignores real differences in qualifications between groups and also ignores whether the labels themselves are legitimate. If the qualification rates of two groups are different to begin with, forcing pass rates to be equal means sacrificing one side.

Equal opportunity requires that the true positive rate (TPR) be similar across groups, i.e., the proportion of qualified individuals who receive the opportunity should be similar. It protects the dimension of opportunity fairness that "qualified people are not missed." It is suitable for focusing on whether qualified candidates receive the same pass. Its cost is that it does not constrain the false positive rate (FPR): as long as the proportion of qualified individuals who are passed is equal, the system can admit many unqualified individuals in one group and not do so in another.

Equalized odds is stricter, requiring both TPR and FPR to be similar across groups; that is, both types of errors—missing qualified individuals and admitting unqualified individuals—must be treated equally. It is suitable for scenarios where both types of errors matter. Its cost is that the constraint is stronger; it may require different thresholds for different groups, or sacrificing overall accuracy to satisfy.

Within-group calibration turns to the meaning of the risk score itself: the same score should correspond to the same true rate, and whatever the score is, it should imply a similar actual success rate. It is suitable for scenarios that focus on whether risk scores are credible and comparable across groups. Its cost is that when base rates differ across groups, it is difficult to simultaneously satisfy equal error rates; calibration and equalized error rates pull against each other.

These definitions are not interchangeable equivalents; they are trade-offs aimed at different harms. Therefore, the choice of metric must be determined before deployment through joint discussion among business stakeholders, domain experts, and affected communities, rather than after results are obtained by looking back at a pile of numbers and picking the most favorable one as the conclusion.

Putting the previous definitions together runs into a structural constraint: they usually cannot all be satisfied simultaneously. The reason lies in differences in base rates.

Suppose the actual positive rates of two groups are different to begin with, and the scores are calibrated within each group. Calibration means that the same score should correspond to the same actual success rate across groups. Thus, among people who receive high scores in group A, the true positive proportion must reflect group A's base rate; among people who receive high scores in group B, the true positive proportion must reflect group B's base rate. Because the two groups have different base rates, the positive proportion among high-scoring people naturally differs. If we then require both TPR and FPR to be equal across groups, we pull the score distribution and predicted values toward conflicting constraints: one side requires scores to faithfully reflect each group's actual rate, while the other side requires the admitted and rejected proportions to be consistent across groups.

This conflict disappears only when predictions are completely accurate or the two groups have exactly the same base rate. When predictions are perfect, no errors occur and both error rates are zero, so they are naturally equal; when base rates are the same, calibrated scores are naturally aligned, and equal error rates no longer conflict with base rates. In reality, neither condition holds, so the conflict almost always exists.

This does not mean "fairness cannot be achieved"; rather, it shows that different definitions express different moral relations: some are concerned with resource allocation outcomes, some with not missing qualified people, and some with whether risk scores are trustworthy. Each is valid on its own, but they cannot all be maximized at the same time. Teams must therefore publicly explain their priorities, which residual harm they accept, and how they plan to remedy it, rather than invoking a mathematical theorem and using "not simultaneously attainable" as an excuse to avoid making a choice.

For beginners, the easiest point to confuse is this: not simultaneously attainable does not mean you can just pick any metric arbitrarily. On the contrary, precisely because there is real tension among the definitions, value trade-offs must be put on the table and subjected to scrutiny by the affected groups. Mathematical proofs constrain which wishes can be realized at the same time, but they cannot decide for you whom to protect.

If you only debias the training set, after deployment the gap often comes back. The reason is that fairness is not a property that exists only inside a dataset; it is an end-to-end property: from problem definition to outcome feedback, every stage can create or amplify group disparities.

Drawing the whole chain (Figure 1) shows a loop rather than a straight line. The starting point is problem definition—what optimization objective was originally chosen, what counts as success, and what counts as harm; this step already writes the situations of certain groups into the direction the system is meant to pursue. Next is data and labels—whether sampling covers everyone, and whether labels record true ability or historical decisions. Further down is model scores—what the model learns from these data, and whether scores are comparable across groups. Then threshold and human decision—how the same score is cut off, and whether human review corrects or amplifies bias. Finally there is outcome feedback—the decisions the system makes change who gets opportunities and who is recorded as successful, and these outcomes flow back into the next round of training data and labels, thereby reshaping the problem definition.

This loop explains why single-point fixes fail. Suppose you debias the training set at the data stage and align model scores between two groups; but as long as the threshold still exerts different effects on different groups, or human reviewers make secondary judgments with their own biases, or those who gain opportunities after deployment in turn dominate the new labels, then the gap will reappear at some later stage. Conversely, if you only adjust the threshold without governing the data and labels, upstream problems will likewise flow down along the chain.

Therefore, viewing fairness as an end-to-end property means that no stage should be individually exempted. The real question to answer is not "Which stage introduced bias?" but "In this loop, will each stage amplify or narrow the gap, and who is continuously monitoring the loop itself?"

If one uses a label of "previously hired and high-performing" to train a model, what the model actually learns may not be "who is competent" but may be "who received opportunities in the past." The key here lies in how the label itself is generated.

People who were not hired have no performance labels at all; they are missing from the training data. The observed outcomes have therefore already been filtered by the old process: those rejected by the old system, regardless of their true ability, never appear as positive examples. Even for those who were hired, performance scores are not a clean objective signal; they are affected by multiple factors such as the direct manager's evaluation style, the amount of opportunity in the assigned role, and whether the person stayed in the job. Training on such labels, the model learns who was historically allowed to enter and who received good evaluations inside, rather than who truly has the ability to perform.

It is likewise unrealistic to expect that deleting the gender field will solve the problem. Features such as school, residential address, and career interruptions can all act as proxy variables for the sensitive attribute, and the model will use these proxies to reconstruct patterns highly correlated with gender; bias simply enters the model through a different path.

Therefore, the label itself must become an object of scrutiny. Examine its generation process: who recorded the label, and under what conditions; examine its missingness mechanism: which people are systematically excluded from the label; examine its measurement consistency: whether the same label means the same thing in different groups. When a single label is unreliable, use multiple outcomes to cross-validate, use blinded human review to reduce evaluator bias, or draw on natural experiments to obtain information closer to true ability. And when a legitimate label really cannot be obtained, the right response is not to push ahead with high-stakes screening, but to narrow the system's use, or repurpose it for resource support—for example, providing training or help to more people—rather than using it to decide who can be excluded.

A small average gap between men and women does not mean that all women are treated fairly. Average metrics fold people in different circumstances into a single number, and intersectional group analysis is precisely about unfolding that number again.

When gender intersects with age, region, disability, language, and other factors, a group that is very small but suffers severe errors may emerge: older women, women in certain regions who use minority languages, and women simultaneously facing the double impact of disability and gender. These people are diluted in the average of "all women"; looking at any single dimension alone does not reveal them, and only when dimensions are layered together does the failure become visible.

But slicing is not a case of the finer the better. Overly fine slicing brings three consequences: each cell has insufficient samples, and estimates are extremely unstable; confidence intervals become extremely wide, and any conclusion loses meaning; small samples also increase privacy risk, because a cell may contain only a single-digit number of people, making it easier to reverse-infer individuals' information.

The right approach is to pre-select important intersections before evaluation, based on harm mechanisms—which combinations of dimensions may bear structural burdens in reality—rather than exhaustively enumerating every group afterward and then picking out statistically significant results. The "significance" obtained from after-the-fact exhaustive enumeration is often noise.

When reporting, be clear about the criteria: how many samples are behind each number, how wide the confidence intervals are, and what the worst group performance is. For groups whose samples are too small for frequency statistics to give stable conclusions, use hierarchical estimation to borrow information from neighboring groups, use qualitative research to supplement mechanisms that quantitative methods cannot see, and use human review as a backstop, rather than presenting a set of noisy rankings as definite conclusions.

Bias mitigation can intervene at four different levels, each corresponding to a different lever and each with its own side effects.

The data layer acts at the source: improve coverage so that omitted groups enter the sample; improve labels by replacing annotations that record historical decisions rather than true ability; improve sampling so that each group's representativeness in the training data is closer to the true distribution. The learning layer works on the training objective: weight different samples or add constraints to the loss function to guide the model toward a fairer convergence. The post-processing layer does not touch the model itself but adjusts thresholds after output so that final decisions satisfy pre-selected fairness conditions. The process layer steps outside the algorithm: change the use of the system, provide appeal channels, or guarantee a minimum level of resources for affected groups.

These four levels are not equivalent safe options. Resampling can only change the distribution of the sample; it cannot fix errors in the labels themselves—if the labels are wrong from the start, no amount of repeated sampling can correct them. Threshold adjustment directly affects the calibration relationship between scores and the true rate; a score that was originally well calibrated may become distorted as a result. And once complex constraints are added in the learning layer, it becomes increasingly difficult for maintainers to explain why changes occur between versions, and interpretability and auditability decline accordingly.

Therefore, in terms of selection order, priority should be given to fixing root causes and process-layer problems, and then choose verifiable technical measures. No intervention is done once and for all; after introduction, five dimensions must be measured simultaneously: whether overall utility remains within an acceptable range, whether the target fairness metric improves, whether the performance of non-target groups is harmed, whether calibration is disrupted, and whether stability across versions is controllable.

There is another easily overlooked constraint: protected attributes are sometimes precisely what is necessary for measuring bias and correcting it; otherwise you cannot calculate where the gaps are at all. Using these attributes is reasonable, but they must be governed under privacy protection and access permissions, rather than letting anyone read and link them at will.

Adding a "recruiter confirmation" button to the system may appear to add another layer of human oversight, when in fact the disparity may increase instead. The reason is that human review is not a neutral filter; it will reintroduce bias in its own way.

Reviewers anchor on the score given by the model: once they see a high-confidence ranking, they tend to accept it directly rather than make an independent judgment. People also tend to review only the few samples flagged by the model, while most other decisions that are let through are barely checked. More insidiously, reviewers may use different override standards for different groups—for a low-scoring person from one group they are more willing to offer a second chance, while for another group they strictly follow the score. And when the interface hides uncertainty and displays scores as a series of pseudo-precise rankings, this automation bias is amplified further, because humans cannot see the places where the model itself lacks confidence.

For human-machine oversight to actually work, the direction needs to be reversed: give reviewers evidence relevant to the decision rather than a single isolated score; limit the system's use, specifying which decisions may be suggested by machines and which must be made independently by people; provide an independent second evaluation channel so that judgment is not held hostage by a single score; and require recording the reason for every override so that the review process itself can be audited after the fact.

Evaluation must also measure each link in the decision chain separately, rather than looking only at offline model scores. Look separately at model recommendations, final human decisions, the rate at which humans adopt model recommendations, override rate, processing time, and the outcomes that ultimately fall on different groups. Fairness conclusions must be aimed at the entire final decision chain, not the offline model that is frozen after training—because what actually acts on people is the combined result of model recommendations plus human review.

After the system goes live, an evaluation set built from old data quickly becomes obsolete. The jobs themselves change, the applicant population changes, and recruiter behavior changes; these three together alter the system's input distribution and the base rates of different groups. At the same time, true performance labels often appear only months later, and they are visible only for people who are hired—rejected people never generate performance records. As a result, an evaluation set that accurately reflected the current state at launch gradually becomes an outdated snapshot over time, and fairness conclusions drawn from it are no longer trustworthy.

Therefore, monitoring should target signals that can reveal problems earlier, rather than waiting until all performance labels are complete before looking back. The following need to be continuously observed: whether input coverage shows a sudden change for some group; whether the score distribution drifts; whether the pass rates, override rates, and appeal rates are abnormal for each group; the pattern of missing feedback—which people systematically have no labels; and how long the labels themselves are delayed. For these signals, the correct approach is to set investigation thresholds: once a threshold is crossed, trigger manual investigation, rather than letting the monitoring system automatically declare "discrimination exists".

When comparing a new version with an old one, the comparison must be made under a common sample and a common policy; otherwise, the difference mixes in sample changes and policy changes, rather than the effect of the version itself. Once a gap is found, the first response should not be to immediately blame the model or algorithm, but to first check whether the data pipeline has errors, whether group coding is accurate, and whether the denominator uses a consistent statistical definition. After ruling out these basic issues, then decide whether to roll back, limit the scope of use, or re-evaluate. If the order is reversed, you will waste a rollback that could have been avoided on a pipeline failure.

When reporting fairness results, a common confusion is: which matters more—a ratio of 0.8 or a difference of 2 percentage points? The answer is that both matter, but they answer different questions.

Ratios are extremely sensitive to low base rates. When the event itself is rare, a tiny absolute fluctuation can make the ratio jump sharply, so ratios can amplify differences but are also easily dominated by small-sample noise. Differences show the actual scale—how many people are truly treated differently. Both must be accompanied by their respective denominators, confidence intervals, and the severity of the harm in order to be interpreted correctly. Looking only at ratios can misjudge severity in low-base-rate scenarios; looking only at differences can underestimate relative unfairness when base sizes are vastly different.

A complete fairness evaluation must report at least these quantities: selection rate, TPR, FPR, PPV (positive predictive value), calibration, the performance of the worst-off group, results for key intersectional slices, and the stability of these metrics over time. It must also explain how labels are derived and how groups are defined; otherwise the numbers themselves lack traceable semantics.

When selecting thresholds, a dangerous tendency is to repeatedly tune on the test set until the numbers look good. This is equivalent to making the test set play the role of the training set, and subsequent evaluation conclusions naturally lose meaning. The correct practice is to retain an independent validation set, and any threshold changes must be approved through a governance process, not done privately by engineers in the lab.

Finally, compressing "fair or not" into a single pass/fail value obscures what really matters: different fairness metrics are themselves in conflict, and affected parties' perspectives also differ. A single-value fairness score seems convenient but actually erases all the trade-offs that need to be seen.

Linking the preceding steps yields a complete chain from the problem straight through to verifiable practice, with each step resting on the conclusions of the previous section.

Step one: Define who is affected, the decisions the system makes, and which harms are unacceptable. Without this step, all subsequent metrics lack an anchor. Step two: Review historical processes, how labels are generated, and the selection mechanism of the old process to determine whether the data records true ability or past decisions. Step three: Build confusion matrices with denominators for each group, not just a single overall number. Step four: Choose which fairness relationship to protect—opportunity, error rates, or the meaning of risk scores—based on the actual costs of the two types of errors. Step five: Check whether there are conflicts among the selected definitions and how large the uncertainty is. Step six: Intervene at the layer closest to the root cause, prioritizing fixing data and processes rather than adjusting thresholds first. Step seven: Evaluate the entire final human-machine decision chain, and break out intersectional groups separately. Step eight: Provide an appeal channel and continuously monitor feedback loops, because deployed data will in turn reshape the next round of training.

To make each step verifiable, fix conditions and observe evidence separately at four layers: input, mechanism, output, and falsification.

At the input layer, fix the same batch of samples, the same preprocessing, and permission boundaries; observe input hashes, slice labels, and the reasons samples were rejected to ensure the starting point of every comparison is consistent.

At the mechanism layer, change only one core variable at a time while locking all other configurations; observe where the key intermediate state first deviates from expectations—this locates which link in the chain the problem actually occurs at.

At the output layer, fix the same acceptance rules and resource budget; observe stratified differences in quality, cost, latency, and failure rate across different groups, and confirm that improvements are not obtained at the expense of diluting a group's cost in the overall metric.

At the falsification layer, keep a control group that does not enable the target mechanism, and check whether benefits replicate stably across samples and across random seeds. If they hold only for a particular batch of samples or a particular seed, the claimed effect is more likely a coincidence than the mechanism itself working.

The core of the entire chain is always to first choose the relationship to protect, then compute group differences. Metrics are quantitative tools for value judgments, not substitutes for value judgment; verifiability ensures that this judgment stands up to scrutiny rather than remaining at the level of slogans.