AI Control Library / CF-08: Testing & Robustness
Designation: Core | Control types represented: Operational and Technical
Control requirement
Benchmark AI reliability and performance against defined use-specific acceptance criteria, using representative workloads, controlled versions and credible measurement. Report variability, limits and regressions as well as aggregate results.
Why this matters
A public benchmark or best-case latency does not establish suitability for an organization’s workflow. Evaluation must reflect the task, environment and consequences of failure.
How the control types work together
Operational controls define scope, authority and the actions required when conditions are not met. Technical controls produce evidence or enforce the relevant boundaries. Both must apply to the same deployment and operating conditions.
When it applies
Model and application selection, release and material changes, including load, availability and task-quality evaluation.
Define the assessed boundary, dependencies and evidence limitations. Tailor depth and review frequency to potential impact, and document why a particular safeguard or exclusion is appropriate to the actual deployment.
Ownership
The system owner sets criteria with domain and performance specialists; qualified reviewers assess results.
Implementation guidance
- Define task success, harmful failures, latency, capacity and availability needs before measurement.
- Use representative protected evaluation sets and workload conditions, controlling leakage and recording configuration and dependencies.
- Measure relevant variability and failure modes, including stress and degraded dependencies, without discarding unfavorable runs.
- Compare candidates and approved baselines, recording tradeoffs, limits and the resulting release conditions.
How to verify it
Use representative records and authorized controlled tests. Identify the relevant version, operating conditions and expected result before testing, and retain enough evidence to support the conclusion.
| Test | Expected result |
|---|---|
| Reproduce a reported result | Task, dataset, configuration and measurement support the claim. |
| Apply representative peak load | Quality and service behavior meet approved conditions or expose a recorded gap. |
| Inspect comparison claims | Candidates were assessed under sufficiently comparable conditions. |
Record a finding when required results cannot be demonstrated or practice differs materially from the approved conditions. Missing evidence limits the conclusion; it should not be treated as a successful test.
Evidence to retain
- Benchmark plan and criteria
- Versioned workloads and configuration
- Results and variability
- Comparison and acceptance decisions
Protect evidence according to its sensitivity, access needs and retention rules. Avoid unnecessary copies of personal information, secrets or confidential material, while preserving the records needed for authorized review.
If the control fails
Escalate unmet critical performance or reliability criteria to release and business owners. Restrict scope, capacity or deployment until credible evidence supports operation.
Example in practice
Hypothetical example: A model answers quickly in single-user tests but loses response quality under production-like concurrency. Benchmarking identifies the limit and reduces approved capacity before release.
Related controls
- CF01-010: AI Change Quality Testing & Approval
- CF09-002: AI Model Performance Monitoring
- CF08-013: AI Output Reliability & Hallucination Testing
Framework mappings
- CSA AICM
- NIST AI RMF (MEASURE 2.2)
- EU AI Act (Art. 9(7))
- ISO 42001 (Cl. 8.2)
Review and maintenance
Review at the interval approved by the control owner and after relevant changes, incidents or failed assessments. Update affected procedures and configuration, address evidence gaps and verify corrective action before closing findings.
Version 1.0