AI Control Library / CF-08: Testing & Robustness
Designation: Core | Control types represented: Operational
Control requirement
Establish an AI testing and robustness policy defining required evaluation, acceptance criteria, independence, safe test conditions, evidence and reassessment across the lifecycle. Require findings to influence release and continued-use decisions.
Why this matters
A successful demonstration cannot establish reliability across relevant conditions. Testing needs a clear purpose and decision consequence rather than a collection of favorable examples.
How the control types work together
Operational controls establish the assigned responsibilities, review processes and response decisions described here. Their effectiveness depends on performing the work and using evidence to guide action, rather than approving a document alone.
When it applies
AI acquisition, development, adaptation, deployment and material changes, including hosted services and agents.
Define the assessed boundary, dependencies and evidence limitations. Tailor depth and review frequency to potential impact, and document why a particular safeguard or exclusion is appropriate to the actual deployment.
Ownership
The AI assurance owner maintains policy; system and domain owners define risk-based criteria and release authorities apply them.
Implementation guidance
- Define evaluation categories from intended use, harms, security boundaries and uncertainty, including failure and misuse conditions.
- Specify representative data, protected holdouts, repeat testing where variability matters and qualified review.
- Establish acceptance and blocking criteria before tests, with controlled exceptions and safe testing authorization.
- Require retained versions, results, limitations and retesting after material changes or failed operation.
How to verify it
Use representative records and authorized controlled tests. Identify the relevant version, operating conditions and expected result before testing, and retain enough evidence to support the conclusion.
| Test | Expected result |
|---|---|
| Trace a release decision | Required evaluations and criteria preceded approval. |
| Inspect a failed critical case | The result affects release or a documented authorized restriction. |
| Review a materially changed system | Relevant evaluation is repeated rather than inherited automatically. |
Record a finding when required results cannot be demonstrated or practice differs materially from the approved conditions. Missing evidence limits the conclusion; it should not be treated as a successful test.
Evidence to retain
- Testing policy and criteria
- Versioned plans and datasets
- Evaluation and release records
- Exceptions and reassessment
Protect evidence according to its sensitivity, access needs and retention rules. Avoid unnecessary copies of personal information, secrets or confidential material, while preserving the records needed for authorized review.
If the control fails
Escalate missing evaluation or unmet criteria to the release and risk authorities. Hold or narrow consequential use until suitable evidence supports the decision.
Example in practice
Hypothetical example: A new model performs well in a sales demo but fails a required confidential-data test. Policy blocks release until the failure is addressed and retested.
Related controls
- CF01-010: AI Change Quality Testing & Approval
- CF08-010: AI Reliability & Performance Benchmarking
- CF08-013: AI Output Reliability & Hallucination Testing
Framework mappings
- CSA AICM (TVM-01)
- NIST AI RMF (MEASURE 1.1)
- EU AI Act (Art. 9(6))
- ISO 42001 (Cl. 8.2)
- MITRE ATLAS
Review and maintenance
Review at the interval approved by the control owner and after relevant changes, incidents or failed assessments. Update affected procedures and configuration, address evidence gaps and verify corrective action before closing findings.
Version 1.0