Two systems can achieve the same overall accuracy and still affect people very differently. One might make occasional errors across many situations. Another might perform well for most users while repeatedly failing for a smaller group. That difference is a reason to examine the distribution of errors, not simply celebrate an average.
NIST's work on harmful bias describes a problem extending beyond training data. Institutional practices, human decisions, measurement choices, and computational processes can all contribute. This wider view matters because replacing a dataset will not necessarily correct a poorly defined task or an unfair process around the model.
Consider a hypothetical application that ranks requests for a community service. Its historical data records who previously received support. If past access depended partly on how easily people could complete forms, predicting historical acceptance may reproduce that pattern. The system would be learning a record of previous decisions, which is not automatically the same thing as measuring present need.
A useful first question is therefore: what does the target label actually represent? Another is: whose experience is missing from the data? Teams should also consider whether an input is a reasonable basis for the decision. An available field is not automatically an appropriate field, even if it improves a numerical score.
Evaluation should connect errors to consequences. A missed eligible request and an incorrectly accepted request may impose different burdens on different people. Looking at these outcomes separately can reveal problems that disappear in a combined metric. Any comparison should also acknowledge small sample sizes and uncertainty instead of treating a handful of cases as a stable conclusion.
Technical analysis needs a practical route to correction. People affected by an automated decision should have an understandable way to question it, and staff need authority to investigate. In this illustrative service, a regular review of appealed cases could expose problems in both the model and the application process. Managing bias is an ongoing responsibility shared by the people who define, build, and operate the system.
Reference: NIST, Identifying and Managing Harmful Bias in AI: https://www.nist.gov/artificial-intelligence/ai-fundamental-research-managing-ai-bias