AI Control Library / CF-13: Model & Adversarial Security
Designation: Core | Control types represented: Operational and Technical
Control requirement
Assess and improve resistance to adversarial inputs designed to cause incorrect or unsafe inference. Define threat assumptions, attack budgets and acceptable performance degradation for the actual task.
Why this matters
Small or strategically selected changes can alter a model’s decision while preserving the apparent meaning of an input. Ordinary accuracy testing does not establish resistance to adaptive attacks.
How the control types work together
Operational controls define scope, authority and the actions required when conditions are not met. Technical controls produce evidence or enforce the relevant boundaries. Both must apply to the same deployment and operating conditions.
When it applies
Apply to exposed classifiers, detectors and other models where manipulated inputs could cause harm. Match testing to realistic attacker knowledge and capabilities.
Define the assessed boundary, dependencies and evidence limitations. Tailor depth and review frequency to potential impact, and document why a particular safeguard or exclusion is appropriate to the actual deployment.
Ownership
The model owner defines task acceptance criteria; security testers design authorized attacks; risk owners approve residual exposure.
Implementation guidance
- Define attacker objectives, access and valid input constraints.
- Evaluate representative clean and adversarial cases, including adaptive tests.
- Apply suitable preprocessing, training or detection defenses.
- Retest combined defenses and operational consequences before release.
How to verify it
Use representative records and authorized controlled tests. Identify the relevant version, operating conditions and expected result before testing, and retain enough evidence to support the conclusion.
| Test | Expected result |
|---|---|
| Adversarial benchmark | Performance meets documented acceptance criteria. |
| Adaptive attack against defense | Residual weaknesses are recorded and assessed. |
| Clean inputs after mitigation | Utility and false-positive limits remain acceptable. |
Record a finding when required results cannot be demonstrated or practice differs materially from the approved conditions. Missing evidence limits the conclusion; it should not be treated as a successful test.
Evidence to retain
- Threat model
- Versioned attack cases
- Evaluation results
- Residual-risk decisions
Protect evidence according to its sensitivity, access needs and retention rules. Avoid unnecessary copies of personal information, secrets or confidential material, while preserving the records needed for authorized review.
If the control fails
Restrict the affected use, route uncertain cases to safer handling and retest remediation against the demonstrated attack.
Example in practice
Hypothetical example: A document classifier is tested with meaning-preserving formatting changes. Failures affecting routing lead to an additional verification step for consequential classifications.
Related controls
- CF13-007: Runtime Adversarial Input Detection
- CF08-001: AI Testing & Robustness Policy
- CF09-001: AI Monitoring & Drift Detection Policy
Framework mappings
- NIST AI RMF (MEASURE 2.5)
- MITRE ATLAS
Review and maintenance
Review at the interval approved by the control owner and after relevant changes, incidents or failed assessments. Update affected procedures and configuration, address evidence gaps and verify corrective action before closing findings.
Back to Model & Adversarial Security
Version 1.0