AI Control Library / CF-13: Model & Adversarial Security
Designation: Core | Control types represented: Operational and Technical
Control requirement
Detect and safely handle suspicious inference inputs using threat-informed runtime controls. Define response paths that consider detector uncertainty, legitimate unusual inputs and the consequences of false alarms.
Why this matters
Attack patterns evolve and can resemble normal edge cases. Detection supports layered defenses; it cannot guarantee that every adversarial input will be recognized.
How the control types work together
Operational controls define scope, authority and the actions required when conditions are not met. Technical controls produce evidence or enforce the relevant boundaries. Both must apply to the same deployment and operating conditions.
When it applies
Apply to exposed inference services and high-consequence decisions where adversarial manipulation is plausible. Include multimodal and encoded inputs where supported.
Define the assessed boundary, dependencies and evidence limitations. Tailor depth and review frequency to potential impact, and document why a particular safeguard or exclusion is appropriate to the actual deployment.
Ownership
The security monitoring owner maintains detections; model teams assess behavioral signals; operations manages handling and escalation.
Implementation guidance
- Define relevant attack signals and detection thresholds.
- Validate incoming format, size and task constraints before inference.
- Route suspicious cases to rate limits, safe fallback or human review.
- Measure bypass and false-positive behavior and update detections after incidents.
How to verify it
Use representative records and authorized controlled tests. Identify the relevant version, operating conditions and expected result before testing, and retain enough evidence to support the conclusion.
| Test | Expected result |
|---|---|
| Known attack cases | Configured response occurs and is traceable. |
| Legitimate unusual inputs | False alarms remain within approved tolerance. |
| Detector unavailable | Documented fallback protects consequential paths. |
Record a finding when required results cannot be demonstrated or practice differs materially from the approved conditions. Missing evidence limits the conclusion; it should not be treated as a successful test.
Evidence to retain
- Detection rules
- Evaluation datasets
- Alert outcomes
- Fallback tests
Protect evidence according to its sensitivity, access needs and retention rules. Avoid unnecessary copies of personal information, secrets or confidential material, while preserving the records needed for authorized review.
If the control fails
Activate safer handling for the affected path, investigate bypasses and avoid broad blocking without evaluating user impact.
Example in practice
Hypothetical example: A claims service flags manipulated image inputs for specialist review while allowing ordinary low-light images that testing showed were frequently misclassified by the detector.
Related controls
- CF13-002: Evasion Attack Resistance
- CF12-010: Prompt Injection Defense
- CF09-001: AI Monitoring & Drift Detection Policy
Framework mappings
- NIST AI RMF (MEASURE 2.5)
- MITRE ATLAS
Review and maintenance
Review at the interval approved by the control owner and after relevant changes, incidents or failed assessments. Update affected procedures and configuration, address evidence gaps and verify corrective action before closing findings.
Back to Model & Adversarial Security
Version 1.0