AI Control Library / CF-12: Security & Abuse Resistance
Designation: Core | Control types represented: Operational and Technical
Control requirement
Treat model output as untrusted input to downstream software. Validate it for the specific consumer and require independent authorization before using it as code, queries, commands, markup or tool arguments.
Why this matters
A plausible answer may contain executable or malformed content. General text filtering does not replace parameterized queries, contextual escaping or strict tool schemas.
How the control types work together
Operational controls define scope, authority and the actions required when conditions are not met. Technical controls produce evidence or enforce the relevant boundaries. Both must apply to the same deployment and operating conditions.
When it applies
Apply wherever generated content enters applications, databases, browsers, command interpreters or business-action interfaces.
Define the assessed boundary, dependencies and evidence limitations. Tailor depth and review frequency to potential impact, and document why a particular safeguard or exclusion is appropriate to the actual deployment.
Ownership
The consuming application owner implements validation; security reviews execution paths; tool owners enforce permissible actions.
Implementation guidance
- Map output consumers and identify execution-sensitive contexts.
- Use strict schemas and parameterized interfaces instead of string-built commands.
- Apply consumer-specific encoding and reject invalid or excessive arguments.
- Test malicious output and ensure rejected content cannot reach alternate execution paths.
How to verify it
Use representative records and authorized controlled tests. Identify the relevant version, operating conditions and expected result before testing, and retain enough evidence to support the conclusion.
| Test | Expected result |
|---|---|
| Generated SQL injection string | Parameterized interface treats it as data. |
| Unsafe rendered markup | Consumer encoding prevents execution. |
| Unauthorized tool arguments | Validation and authorization deny the action. |
Record a finding when required results cannot be demonstrated or practice differs materially from the approved conditions. Missing evidence limits the conclusion; it should not be treated as a successful test.
Evidence to retain
- Consumer inventory
- Validation rules
- Security test cases
- Rejection logs
Protect evidence according to its sensitivity, access needs and retention rules. Avoid unnecessary copies of personal information, secrets or confidential material, while preserving the records needed for authorized review.
If the control fails
Disable unsafe execution paths, inspect completed actions and repair the consuming interface before restoring automation.
Example in practice
Hypothetical example: An agent proposes a customer-search query containing executable syntax. The application passes the search value as a database parameter rather than concatenating generated SQL.
Related controls
- CF12-005: AI Output Modification Authorization
- CF12-007: AI API Security
- CF14-001: Agentic System Security Boundaries
Framework mappings
- NIST AI RMF (MEASURE 2.5)
- MITRE ATLAS
- OWASP LLM Top 10
Review and maintenance
Review at the interval approved by the control owner and after relevant changes, incidents or failed assessments. Update affected procedures and configuration, address evidence gaps and verify corrective action before closing findings.
Back to Security & Abuse Resistance
Version 1.0