Silent model degradation is the biggest operational risk in production ML. A fraud model trained on pre-COVID spending patterns degrades severely during economic shocks. A churn model trained on 2023 customer behaviour becomes less predictive as the product evolves. Without monitoring, UK businesses discover model failure via business metrics decline — often months after the degradation began. FCA SS1/23 explicitly requires ongoing model performance monitoring with defined thresholds and escalation procedures.
Three types of drift to monitor: data drift (input feature distribution shifts — detect with PSI for continuous, chi-squared for categorical, Earth Mover's Distance), concept drift (relationship between features and target changes — harder to detect without fresh labels, use proxy metrics), prediction drift (output distribution shifts — simplest to monitor, useful leading indicator). Monitoring frequency: daily for fraud/credit, weekly for churn/demand forecasting.
Monitoring tooling for UK production: Evidently AI (open-source, generates drift reports and data quality checks, integrates with Grafana), Arize AI (enterprise, with explainability), WhyLabs (privacy-preserving monitoring), MLflow experiment tracking (custom metric logging). Alert routing: PagerDuty for P1 drift (model offline), Slack for P2 (degraded performance), weekly Grafana dashboard for P3 (monitoring trend).
Retraining triggers and automation: drift alert → automatic data validation → Champion-Challenger test on holdout data → human approval gate → rolling deployment. For FCA-regulated models, retraining requires model risk committee sign-off — automate data pipeline but require human approval for deployment. Codeworth MLOps retainer (from £800/month) includes monitoring setup, drift alerting, and quarterly retraining governance.
Related Codeworth Services
Need Help with Implementation?
The Codeworth team is ready to implement these ideas in your project. Free consultation.
Discuss the ProjectCodeworth team specialist.