AI Control Library / CF-08: Testing & Robustness
Designation: Supplemental | Control types represented: Operational and Technical
Control requirement
Define and use AI application security metrics that reflect exposure, control effectiveness and response, with clear populations, denominators, ownership and decision thresholds. Preserve limits and avoid reporting activity counts as assurance.
Why this matters
A rising number of scans or falling number of alerts can reflect changed coverage rather than improved security. Metrics need context and a link to a decision.
How the control types work together
Operational controls define scope, authority and the actions required when conditions are not met. Technical controls produce evidence or enforce the relevant boundaries. Both must apply to the same deployment and operating conditions.
When it applies
Security reporting for AI application development, release and operation, including shared platforms and critical suppliers.
Define the assessed boundary, dependencies and evidence limitations. Tailor depth and review frequency to potential impact, and document why a particular safeguard or exclusion is appropriate to the actual deployment.
Ownership
The application security owner maintains metrics with engineering, response and assurance input.
Implementation guidance
- Define the security question and decision for each metric, identifying source, scope, denominator and update cadence.
- Measure meaningful exposure such as unresolved exploitable findings, coverage gaps and intervention effectiveness as appropriate.
- Validate source quality and preserve scope changes, uncertainty and detection limitations.
- Assign review and action ownership, testing whether deteriorating measures lead to investigation or corrective action.
How to verify it
Use representative records and authorized controlled tests. Identify the relevant version, operating conditions and expected result before testing, and retain enough evidence to support the conclusion.
| Test | Expected result |
|---|---|
| Reproduce a metric | Source, population and calculation support the result. |
| Change scanning scope | Reporting identifies the discontinuity rather than claiming an unqualified improvement. |
| Trigger a material threshold | The intended owner records a response. |
Record a finding when required results cannot be demonstrated or practice differs materially from the approved conditions. Missing evidence limits the conclusion; it should not be treated as a successful test.
Evidence to retain
- Metric specification
- Source-quality and calculation checks
- Contextualized reports
- Review and corrective-action records
Protect evidence according to its sensitivity, access needs and retention rules. Avoid unnecessary copies of personal information, secrets or confidential material, while preserving the records needed for authorized review.
If the control fails
Escalate misleading or unavailable material metrics to the security owner. Qualify reporting and restore evidence before relying on it for release or risk decisions.
Example in practice
Hypothetical example: A dashboard reports fewer vulnerabilities after an unscanned service is added. Coverage reporting exposes the missing population, preventing a false claim of improved security.
Related controls
- CF08-011: AI Dependency & Library Vulnerability Scanning
- CF09-010: AI Monitoring Metrics & Reporting
- CF04-016: AI Governance Evidence Management
Framework mappings
- CSA AICM (AIS-03)
- NIST AI RMF (MEASURE 1.1)
- ISO 42001 (Cl. 9.1)
Review and maintenance
Review at the interval approved by the control owner and after relevant changes, incidents or failed assessments. Update affected procedures and configuration, address evidence gaps and verify corrective action before closing findings.
Version 1.0