AI Control Library / CF-12: Security & Abuse Resistance
Designation: Core | Control types represented: Operational, Technical and Contextual
Control requirement
Reduce prompt-injection risk across direct inputs, retrieved documents, tool results and other untrusted content. Enforce data and action boundaries independently of model instructions and detect attempted boundary violations.
Why this matters
An attacker can place instructions inside material the model is expected to read. Instruction formatting and filtering may help, but cannot establish reliable authorization for consequential actions.
How the control types work together
Operational controls define review and intervention duties. Technical controls enforce limits or supply relevant evidence. Contextual controls shape the information, interface and conditions in which people interpret results and decide whether to act.
When it applies
Apply to LLM systems that process external or user-controlled content, especially those with retrieval, tools, memory or multimodal inputs.
Define the assessed boundary, dependencies and evidence limitations. Tailor depth and review frequency to potential impact, and document why a particular safeguard or exclusion is appropriate to the actual deployment.
Ownership
The application owner defines trust boundaries; security tests attacks; tool and data owners enforce underlying permissions.
Implementation guidance
- Identify untrusted content and the actions it could influence.
- Keep instructions and external data clearly distinguished and provenance visible.
- Restrict tool permissions, destinations and persistent writes outside the model.
- Test direct and indirect attacks against complete workflows.
How to verify it
Use representative records and authorized controlled tests. Identify the relevant version, operating conditions and expected result before testing, and retain enough evidence to support the conclusion.
| Test | Expected result |
|---|---|
| Malicious retrieved instruction | Unauthorized action remains blocked. |
| Encoded or multimodal injection | Test records actual exposure and residual risk. |
| Injection requests secret export | Data and destination controls prevent transfer. |
Record a finding when required results cannot be demonstrated or practice differs materially from the approved conditions. Missing evidence limits the conclusion; it should not be treated as a successful test.
Evidence to retain
- Trust-boundary map
- Attack cases
- End-to-end results
- Remediation records
Protect evidence according to its sensitivity, access needs and retention rules. Avoid unnecessary copies of personal information, secrets or confidential material, while preserving the records needed for authorized review.
If the control fails
Suspend vulnerable capabilities, contain exposed data paths and investigate successful actions rather than counting refusals alone.
Example in practice
Hypothetical example: A web page tells a research agent to email private documents. The agent’s mail capability is unavailable and the document service denies access beyond the approved collection.
Related controls
- CF12-004: AI Knowledge Base & RAG Access Control
- CF12-006: AI Agent Access Restriction
- CF14-012: Agentic Memory & Context Security
Framework mappings
- NIST AI RMF (MEASURE 2.5)
- MITRE ATLAS
- OWASP LLM Top 10
Review and maintenance
Review at the interval approved by the control owner and after relevant changes, incidents or failed assessments. Update affected procedures and configuration, address evidence gaps and verify corrective action before closing findings.
Back to Security & Abuse Resistance
Version 1.0