AI Control Library / CF-08: Testing & Robustness
Designation: Core | Control types represented: Operational and Technical
Control requirement
Test AI behavior for relevant bias and fairness concerns using justified populations, metrics, scenarios and uncertainty analysis. Let results influence deployment, remediation and ongoing evaluation.
Why this matters
Overall accuracy can hide different burdens of error or access. Fairness testing needs to examine the actual decision and evaluation coverage rather than infer fairness from missing sensitive fields.
How the control types work together
Operational controls define scope, authority and the actions required when conditions are not met. Technical controls produce evidence or enforce the relevant boundaries. Both must apply to the same deployment and operating conditions.
When it applies
AI uses with material people-related effects, including generated content and human decisions influenced by model outputs.
Define the assessed boundary, dependencies and evidence limitations. Tailor depth and review frequency to potential impact, and document why a particular safeguard or exclusion is appropriate to the actual deployment.
Ownership
The assessment lead coordinates domain, statistical and privacy expertise; the business owner approves treatment and use conditions.
Implementation guidance
- Define relevant harms, groups and intersections, selecting measures and scenarios for stated reasons.
- Use reviewed data collection and appropriate evaluation samples, recording missingness, small groups and label limitations.
- Evaluate outcomes and errors with uncertainty and qualitative context, separating observed disparities from conclusions about cause.
- Retest proposed corrections and preserve limitations and release conditions, with monitoring for changed populations.
How to verify it
Use representative records and authorized controlled tests. Identify the relevant version, operating conditions and expected result before testing, and retain enough evidence to support the conclusion.
| Test | Expected result |
|---|---|
| Inspect an aggregate pass | Relevant subgroup effects and limits were evaluated. |
| Review a small-sample result | Uncertainty does not become an unsupported fairness claim. |
| Evaluate a correction | Relevant outcomes improve without an unexamined new disparity. |
Record a finding when required results cannot be demonstrated or practice differs materially from the approved conditions. Missing evidence limits the conclusion; it should not be treated as a successful test.
Evidence to retain
- Fairness test scope and rationale
- Reviewed datasets and methods
- Results and uncertainty
- Treatment and release decisions
Protect evidence according to its sensitivity, access needs and retention rules. Avoid unnecessary copies of personal information, secrets or confidential material, while preserving the records needed for authorized review.
If the control fails
Escalate material unfair effects or insufficient consequential-use evidence to business and assessment authorities. Narrow or hold affected use while safeguards or further evaluation are established.
Example in practice
Hypothetical example: A screening model’s overall score improves, but one relevant group experiences more false rejections. The release decision addresses that error burden instead of relying on the headline score.
Related controls
- CF02-003: Demographic Impact Assessment
- CF02-007: Bias Metrics Definition & Monitoring
- CF02-008: Bias Remediation Process
Framework mappings
- NIST AI RMF (MEASURE 2.11)
- EU AI Act (Art. 10(2)(f))
- ISO 42001 (Cl. 8.2)
Review and maintenance
Review at the interval approved by the control owner and after relevant changes, incidents or failed assessments. Update affected procedures and configuration, address evidence gaps and verify corrective action before closing findings.
Version 1.0