How to run an AI bias assessment: fairness testing, metrics and documentation

An AI bias assessment checks whether an AI system treats groups of people differently in ways you cannot justify, and records what you decided to do about it. Bias in AI is rarely deliberate. It comes from historical data that reflects past decisions, from groups being under-represented in training data, from proxies that stand in for protected attributes, and from how people use the outputs. An assessment exists to find it, measure it and decide what to do about it, with a record that shows the reasoning. This guide gives a method that works across sectors and jurisdictions.

Why the rules ask for it

Bias testing appears in law and standards in different forms.

• New York City Local Law 144 requires employers and employment agencies using automated employment decision tools to have an independent bias audit no more than a year before use, to publish a summary of the results, and to notify candidates. The audit reports selection rates and impact ratios by sex, race and ethnicity, and their intersections.
• The EU AI Act requires providers of high-risk systems to examine training, validation and testing data for possible biases that could affect health, safety or fundamental rights or lead to discrimination, and to take measures to detect, prevent and mitigate them. The 2026 amendments moved the legal basis for processing special category data for bias detection into a new Article 4a, extended it to deployers (organisations that use a high-risk system rather than build it), and kept strict conditions.
• Colorado replaced its 2024 AI Act with SB 26-189, effective 1 January 2027, which focuses on notice, explanation, correction and human review for automated decisions rather than the earlier algorithmic discrimination impact assessments. Anti-discrimination law still applies regardless.
• NIST published Special Publication 1270 on identifying and managing bias in AI, which describes systemic, statistical and human sources of bias, and the AI RMF treats fairness with harmful bias managed as a characteristic of trustworthy AI.
• ISO/IEC 42001 asks organisations to assess the impacts of AI systems on individuals and groups, and ISO/IEC TR 24027 addresses bias in AI systems and decision-making.

Step 1: Define the decision and the harm

Start from what the system does. A model that ranks job applicants, one that sets credit limits and one that drafts marketing copy carry very different fairness questions. Write down three things.

• The decision or output, and how it is used.
• Who is affected, and whether they can see, question or appeal the outcome.
• What unfair treatment would look like, such as being screened out, being charged more, receiving worse service or being flagged for investigation more often.

This framing decides the metric. Fairness has no single definition that fits every decision.

Step 2: Name the groups and get the data

List the groups that could be disadvantaged. Protected attributes under anti-discrimination law are the floor, such as sex, race, ethnicity, age and disability. Add groups specific to the setting, such as people whose first language is not the system’s language, people with thin credit files or people in regional areas.

Then work out how you will measure outcomes for those groups. This is often the hardest part. Many organisations do not hold protected attributes, and collecting them raises its own legal questions. Options include voluntary self-identification, a separate testing dataset collected with consent, or statistical inference used only for testing. Take advice for your jurisdiction before choosing.

Look at intersections. A system can look fair for women and fair for older applicants while treating older women badly. Local Law 144 requires intersectional reporting for this reason.

Step 3: Choose the fairness metrics

The common metrics measure different things.

• Selection rate and impact ratio. The share of each group receiving the favourable outcome, and each group’s rate as a ratio of the highest. Suits screening and allocation. The four-fifths rule flags a ratio below 0.8 as a signal of adverse impact.
• Equal opportunity. Equal true positive rates across groups, meaning qualified people in each group are equally likely to be selected. Suits decisions where missing a deserving person is the main harm.
• Equalised odds. Equal true positive and false positive rates. Suits risk scoring where both kinds of error cause harm.
• Predictive parity and calibration. A given score means the same likelihood of the outcome in every group. Suits scores that people act on directly.

These cannot all hold together when base rates differ between groups. A credit model calibrated for every group will generally have different error rates across them. Choose the metric that matches the harm you defined in step 1, set the threshold before testing, and record why.

Step 4: Test the system

Run the system on data that represents the people it will actually see. Measure the chosen metrics for each group and each intersection with enough data to be meaningful. Report confidence intervals or sample sizes, because small groups produce noisy numbers.

Where you find a difference, look for the cause. Common ones are under-representation in training data, labels that encode past human decisions, features that act as proxies, and thresholds set on the whole population. Generative systems need a different approach. Test them with matched prompts that vary only the group signal, such as a name or pronoun, and compare the outputs.

Step 5: Mitigate and accept

Mitigation can act at several points.

• Data. Rebalance, collect more data for under-represented groups, or remove features that act as proxies.
• Model. Use training techniques that constrain the chosen fairness metric.
• Threshold. Adjust decision thresholds, where the law allows it.
• Process. Add human review for borderline cases, offer an appeal route, or limit the system to uses where the difference does not matter.

Every mitigation has a cost, usually in accuracy or simplicity. The accountable owner should accept the residual position in writing, knowing the metric, the threshold, the result and the trade-off.

Step 6: Monitor in production

A model that passed testing can drift into bias as the population, the data or how people use it changes. Re-run the metrics on live outcomes on a schedule, and whenever the model, data or threshold changes. Watch appeal and complaint patterns by group, which can reveal problems the metrics miss.

What to document

A bias assessment record should let a stranger follow the reasoning.

• The decision, the affected people and the harm defined.
• The groups considered, the data used and how group information was obtained.
• The metrics chosen, why, and the thresholds set before testing.
• The results by group and intersection, with sample sizes.
• The causes found, the mitigations applied and their effect.
• The residual position and who accepted it, with the date and the system version.
• The monitoring plan and when the next review falls due.

Where Aicura fits

Aicura holds the assessment record that the testing sits inside. Its risk assessments treat bias and fairness as one of their risk domains, and each bias risk item carries a written rationale and suggested controls that land in Risks and Controls, where each control is tracked as applicable, implemented and tested. Every assessment runs against a fixed, saved version of the system’s AI Register record, so the record shows exactly which version of the system was assessed. At Pro, impact assessments add who is affected and the rights involved, and results from drift checks run inside your own environment appear on the system’s Testing tab for step 6.

Aicura does not compute fairness metrics on your data or certify a system as unbiased. The analysis, the choice of metric and the acceptance stay with your people.