AI Control Library / CF-13: Model & Adversarial Security
Designation: Core | Control types represented: Operational and Technical
Control requirement
Assess and reduce the risk that model access reveals sensitive training information through inversion or membership inference. Evaluate the actual interface, data sensitivity and attacker capabilities.
Why this matters
A model may reveal information without reproducing an entire training record. Membership signals can themselves be sensitive, while repeated outputs or confidence values can support reconstruction.
How the control types work together
Operational controls define scope, authority and the actions required when conditions are not met. Technical controls produce evidence or enforce the relevant boundaries. Both must apply to the same deployment and operating conditions.
When it applies
Apply where training or fine-tuning uses confidential or personal data. EU AI Act Article 10(5) is a conditional special-category-data provision for specified bias work, rather than a general requirement for every model.
Define the assessed boundary, dependencies and evidence limitations. Tailor depth and review frequency to potential impact, and document why a particular safeguard or exclusion is appropriate to the actual deployment.
Ownership
The privacy and model owners define exposure tolerance; security performs authorized attack testing; API owners reduce unnecessary disclosure.
Implementation guidance
- Identify sensitive training populations and exposed output signals.
- Test representative membership and reconstruction attacks against the deployed interface.
- Minimize outputs, restrict queries and consider privacy-preserving training where suitable.
- Evaluate privacy and utility together and document remaining exposure.
How to verify it
Use representative records and authorized controlled tests. Identify the relevant version, operating conditions and expected result before testing, and retain enough evidence to support the conclusion.
| Test | Expected result |
|---|---|
| Membership attack evaluation | Risk meets approved criteria against a suitable baseline. |
| Repeated-query test | Interface limits unnecessary exploitable signals. |
| Mitigation utility test | Relevant task performance remains acceptable. |
Record a finding when required results cannot be demonstrated or practice differs materially from the approved conditions. Missing evidence limits the conclusion; it should not be treated as a successful test.
Evidence to retain
- Privacy threat model
- Attack methodology
- Exposure results
- Mitigation decisions
Protect evidence according to its sensitivity, access needs and retention rules. Avoid unnecessary copies of personal information, secrets or confidential material, while preserving the records needed for authorized review.
If the control fails
Restrict the interface or affected model, investigate potential disclosure and engage privacy incident handling when warranted.
Example in practice
Hypothetical example: A health model’s confidence responses enable stronger membership inference than its class labels. The owner removes unnecessary confidence detail and retests the revised service.
Related controls
- CF12-013: AI Sensitive Information Disclosure Prevention
- CF13-009: Model Inference API Abuse Detection
- CF05-001: Data Security & Privacy Policy for AI
Framework mappings
- NIST AI RMF (MEASURE 2.5)
- EU AI Act (Art. 10(5))
- ISO 42001 (Cl. 8.2)
- MITRE ATLAS
Review and maintenance
Review at the interval approved by the control owner and after relevant changes, incidents or failed assessments. Update affected procedures and configuration, address evidence gaps and verify corrective action before closing findings.
Back to Model & Adversarial Security
Version 1.0