A Unified Observability Framework for Cloud-Native Machine Learning Systems: Architectural Primitives, Lineage Correlation, and Cross-Stage Reliability Objectives

Abstract

The observability gap in modern cloud-native organizations reflects a fundamental disconnect between data engineering teams, which focus on pipeline throughput and data freshness, and machine learning teams, which monitor inference latency and prediction drift. This separation is largely caused by telemetry systems that lack a unified identity model capable of linking datasets, features, model versions, and production environments across the machine learning lifecycle. To address this challenge, this paper proposes a unified observability framework that integrates telemetry primitives, lineage-aware correlation, and end-to-end reliability objectives connecting data quality and freshness with model performance outcomes. The framework introduces a minimal set of universal telemetry tags, including environment, workload identifier, dataset/feature/model version, and execution run identifier, enabling consistent cross-lifecycle correlation and incident analysis. A comparative evaluation is conducted against existing observability solutions, including MLflow integrated with OpenTelemetry, Monte Carlo data observability, and WhyLogs. The results indicate that the proposed framework offers superior capabilities for cross-stage incident attribution by linking failures occurring across data pipelines, feature engineering processes, model training, and inference services. Concept validation is performed using the Alibaba Cluster Trace Dataset and Evidently AI Drift Detection Dataset, demonstrating the practicality and applicability of the proposed telemetry primitives in real-world scenarios. The study further shows that lineage-aware correlation can reveal operational dependencies and failure propagation patterns that remain undetected by conventional component-level monitoring approaches. In addition, the framework defines a tool-agnostic event format and supports machine learning–based incident classification for common cross-stage failure modes, including training-serving skew and prediction degradation caused by data freshness issues. An incremental adoption strategy is proposed to facilitate implementation, beginning with high-impact production models and expanding according to demonstrated operational value.