AI Control Library / CF-09: Monitoring & Drift Detection
Designation: Core | Control types represented: Operational and Technical
Control requirement
Detect deterioration in the relationship between AI inputs and relevant outcomes, distinguishing concept change, data quality, workflow change and other causes. Use appropriate outcome evidence and explicit uncertainty when labels are delayed or unavailable.
Why this matters
Input distributions can appear stable while the meaning of a useful predictor changes. Conversely, a changed error rate does not by itself identify concept drift as the cause.
How the control types work together
Operational controls define scope, authority and the actions required when conditions are not met. Technical controls produce evidence or enforce the relevant boundaries. Both must apply to the same deployment and operating conditions.
When it applies
Deployed AI whose learned relationships or task usefulness can change over time, including changed business conditions and user practices.
Define the assessed boundary, dependencies and evidence limitations. Tailor depth and review frequency to potential impact, and document why a particular safeguard or exclusion is appropriate to the actual deployment.
Ownership
The model and business owners define outcome relevance; evaluation and monitoring teams investigate observed decay.
Implementation guidance
- Define task outcomes and reference performance, documenting how and when trustworthy labels or qualified judgments become available.
- Monitor relevant errors and conditional patterns over time, accounting for sampling, delay and changes in the decision process.
- Use proxy indicators only with recorded limits; investigate competing explanations such as upstream defects or altered human handling.
- Re-evaluate the model and workflow before retraining or replacement, preserving holdout checks and release authorization.
How to verify it
Use representative records and authorized controlled tests. Identify the relevant version, operating conditions and expected result before testing, and retain enough evidence to support the conclusion.
| Test | Expected result |
|---|---|
| Introduce a controlled changed-outcome relationship | Evaluation identifies relevant deterioration under the available evidence. |
| Review a proxy-only alert | The report does not claim confirmed concept drift without supporting outcomes. |
| Trace a retraining decision | Diagnosis and controlled evaluation precede deployment. |
Record a finding when required results cannot be demonstrated or practice differs materially from the approved conditions. Missing evidence limits the conclusion; it should not be treated as a successful test.
Evidence to retain
- Outcome and label specification
- Reference and decay analysis
- Diagnostic investigation
- Re-evaluation and change decisions
Protect evidence according to its sensitivity, access needs and retention rules. Avoid unnecessary copies of personal information, secrets or confidential material, while preserving the records needed for authorized review.
If the control fails
Escalate material decay or unresolved consequential uncertainty to model and business owners. Limit reliance or use a supported fallback while the cause and treatment are assessed.
Example in practice
Hypothetical example: A risk model sees similar inputs but outcomes change after a business rule change. Investigation separates process effects from model decay before selecting remediation.
Related controls
- CF09-002: AI Model Performance Monitoring
- CF09-003: Input Data Drift Detection
- CF01-010: AI Change Quality Testing & Approval
Framework mappings
- NIST AI RMF (MANAGE 4.1)
- EU AI Act (Art. 72)
- ISO 42001 (Cl. 9.1)
Review and maintenance
Review at the interval approved by the control owner and after relevant changes, incidents or failed assessments. Update affected procedures and configuration, address evidence gaps and verify corrective action before closing findings.
Back to Monitoring & Drift Detection
Version 1.0