When Pipelines Break Silently: The Case for Self-Adaptive Data Cleaning

Abstract

Schema changes have transitioned from occasional maintenance events to a routine operational condition in modern data engineering environments. Yet, the theoretical foundations for handling it at the operator level remain underdeveloped. Upstream teams modify field names, remove columns, alter data types, and restructure nested objects as a natural consequence of product iteration and infrastructure migration. At the same time, downstream cleaning logic continues executing against structural assumptions that no longer hold. The result is pipelines that survive schema change while silently degrading the quality of their output in ways that standard monitoring frameworks do not immediately surface. Recent advances in schema evolution cataloging, data quality optimization, provenance tracking, and CI/CD governance have each strengthened individual components of the pipeline reliability problem. Still, they have treated these concerns as neighboring domains rather than as parts of a unified theory. The concept of operator robustness to schema evolution addresses this fragmentation directly by defining robustness not as functional continuity alone, but as the joint preservation of function and data quality after structural change. Mapping the cleaning operator's sensitivity to elementary schema modification operations yields a decision framework that transforms reactive maintenance into an analyzable adaptation problem. Embedding that framework within a self-adaptive architectural loop—monitoring, analysis, planning, and execution over shared knowledge—establishes the conceptual infrastructure for pipelines that respond to structural uncertainty autonomously rather than depending on manual repair. The forward agenda includes tri-objective optimization across quality, latency, and compute cost, provenance-guided operator repair, contract-aware adaptation policy, and empirical benchmarking against realistic long-running pipelines with controlled schema injection.