An MLOps pipeline is a set of linked pipelines that takes a model from raw data to a monitored prediction service, and then back into training. Three pipelines do the core work. Data management covers gathering, cleaning and preprocessing data, and feature engineering. Model creation covers training, hyperparameter tuning and evaluation. Deployment covers packaging, integration and serving through inference APIs. Around them sit feature and metadata stores, a model registry, CI/CD automation, runtime monitoring and retraining triggers. In our view the hard part of MLOps is the joins between the components: the points where data changes shape, a model gets promoted, or a signal one team emits never reaches the team that needs it.
Which components does an MLOps pipeline need, from data ingestion to monitoring?
Google’s MLOps architecture guidance describes a level 1 pipeline that automates continuous training. To get there it adds automated data validation, model validation, pipeline triggers and metadata management to a basic train-and-deploy flow. A 2024 systematic mapping study of 43 papers on MLOps architecture found 35 distinct components and grouped them into six categories: data curation, storage and versioning, ML training, CI/CD, inference, and infrastructure and supporting services. The table below is the working set we would expect in a production pipeline that retrains itself. Each row says what the component does and why production needs it.
Component map of a production MLOps pipeline
| Component | What it does | Why it matters in production |
|---|---|---|
| Data ingestion | Merges data from several sources into one repository the trainer can read | Enterprise data usually sits in several storage systems with different characteristics, so it needs a conversion step before training. |
| Data validation | Checks each batch against a schema and decides whether to retrain or stop the pipeline | Catches bad data before a model is trained on it. |
| Feature store | Gives the same feature definitions and values to experiments, the continuous-training pipeline and the prediction service | Avoids rebuilding and duplicating features, and keeps training and serving consistent. |
| Training and experiment tracking | Trains candidate models and records what was tried | Engineers hand tracking to tools but still choose the next experiment themselves. |
| Offline model validation | Scores the new model on a test set and compares it with the current production model | A model is promoted only if it beats the current one. |
| Metadata store | Records each run’s date, steps and arguments, the evaluation results and pointers to earlier model versions | Supports debugging, reproduction and rollback. |
| Model registry | Versions trained models, links them to their data and hyperparameters, and gates promotion | Only validated models reach production, and rollback stays possible. |
| CI/CD | Builds, tests and packages pipeline code and model components, then deploys them to staging or production | Pipeline changes ship as tested artefacts, not by hand. |
| Staged and online validation | Runs canary or A/B deployments, often across clusters of increasing size | Catches problems before every user sees them. |
| Runtime monitoring | Watches model performance, data and infrastructure in real time | Its output is the signal that starts retraining or a new experiment. |
| Retraining trigger | Starts retraining on events, intervals or a performance threshold seen by the monitor | Links observation to action. |
Practitioners describe a similar structure from the inside. An interview study of 18 ML engineers, published in 2024, found four stages: data preparation, experimentation, evaluation and deployment, and monitoring and response. The engineers ran these as a continuous loop, not a line. The same study treats evaluation and deployment as one stage. Deployments run for long periods and evaluation happens several times during each one, so the two are tightly intertwined.
What an MLOps pipeline architecture diagram shows, and what it blurs
The baseline architecture in the 2024 mapping study shows the data flow first. Raw data goes through a preprocessor into a dataset repository. The experiment pipeline trains on that data and writes models to a model repository. After deployment, a runtime monitor reports performance. More complete diagrams add an arrow from the monitor back to the training pipeline: a periodic or event-based retraining request that retrains and redeploys automatically. Some vendor reference architectures organise everything into phases. Microsoft’s MLOps v2 pattern, for example, has four: data estate, administration and setup, an inner loop for model development, and an outer loop for staging, deployment and monitoring.
Two diagrams labelled “MLOps pipeline architecture” can describe very different things. The 2024 mapping study found that most of the figures it reviewed combined architecture, process and stakeholder roles in one picture. That makes a diagram compact but less clear. Detail varied from figures with six high-level components to combined figures with more than 40 entities. Names were inconsistent too. Some papers used “Feature Store” for what was really a dataset store, or “Artifact Store” for a model store. Almost half the architectures contained domain-specific components that do not carry over to other settings.
For a broader view of the stages MLOps adds on top of a DevOps flow, see the end-to-end machine learning pipeline diagram. For how these components map onto shared accelerator infrastructure, see MLOps architecture for GPU clusters.
Five checks to run on any MLOps architecture diagram
- Separate the views. Mark which boxes are components, which are process steps and which are people. A diagram that mixes all three hides who owns each component.
- Find the trigger arrow. Check whether anything links monitoring back to training. If nothing does, retraining depends on someone starting it by hand.
- Look for a comparison gate. Without a component that compares the new model with the deployed one, a newly trained model is always deployed, even when it performs worse.
- Trace the feature path. Check whether training and serving read features from the same place. Breck et al. (MLSys 2019) found separate code paths for offline and online feature generation to be the most common cause of training-serving skew.
- Check what each named box actually does. Because components are named so inconsistently, confirm what a “Feature Store” or “Artifact Store” does before assuming it does what the name suggests.
Where do MLOps pipelines break once the model is in production?
Production ML pipelines rarely fail loudly. A 2025 study of monitoring practice puts it plainly: ML systems fail silently, not with crashes or error logs but by making wrong decisions. The research reports several distinct failure points. They are different problems, so each gets a short treatment of its own.
Data that is wrong but well-formed
In the 2025 monitoring study, data integrity issues were the most common failure mode, raised in six of seven practitioner sessions. Polyzotis and colleagues (SIGMOD Record, 2018) explain why these issues do so much damage. Training tolerates errors such as a country code changing from “US” to “us”, or an age feature switching units. Instead of crashing, the model simply loses accuracy. In their example, a data-source bug pins a feature to -1. The model trained on that data is worse, and its predictions are logged and fed back as the next round’s training data, so the error spreads downstream.
Training-serving skew is the related failure where production feature data deviates from the data used in training. Breck et al. found that its most common cause is separate code paths for generating training data and serving data. Those paths exist because offline training and online serving have very different latency and throughput needs. In one case reported in that paper, Google Play found features that were always present in training but always missing from serving logs. Removing the skew raised the app install rate on the main landing page by 2% in an online A/B experiment.
Quick fixes that turn into pipeline jungles
Sculley et al. (NeurIPS 2015) named “pipeline jungles”: data preparation that grows into a tangle of scrapes, joins and sampling steps, which are costly to manage, test and recover. The 2024 interview study shows how jungles form in practice. Engineers under pressure to restore performance add non-ML rules and filters. Most of the pipelines the interviewees described had become jungles of model versions, rules and filters. Sculley et al. also flag fixed decision thresholds in dynamic systems as a maintenance cost. A threshold set by hand may no longer be valid once the model retrains on new data.
Alerts nobody trusts and signals that never connect
The alerting trade-off is between recall and precision. Polyzotis and colleagues describe the cycle: after an incident, a team adds alerts for every conceivable data issue, the alerts fire daily, and people start ignoring all of them. Engineers in the 2024 interview study named alert fatigue as a pain point and listed “muting alerts” among the anti-patterns they had seen. Coverage gaps cause the opposite problem. When a downstream signal fires and no upstream signal does, teams cannot tell a critical failure from a harmless change. When an upstream signal fires with no downstream signal, they cannot tell whether it matters. Instrumentation is weakest across components owned by different teams, so root-cause analysis remains mostly manual.
Late labels and mismatched environments
Ground-truth labels for live predictions often arrive after a delay that varies unpredictably. That makes it hard to know real-time performance or when to retrain. Development and production environments can also differ in ways that create bugs. The 2024 study gives examples: GPUs for training and CPUs for inference, or Python in development and C++ in serving.
The research also contains a finding that cuts the other way. Papasian and Underwood (USENIX OpML 2020) examined outages in one of Google’s largest and oldest continuous ML pipelines. Most of the outages were not ML-centric. They were more related to the pipeline’s distributed character. Our reading is that a hardening plan aimed only at drift will miss the failures that come from ordinary distributed systems.
How drift monitoring and retraining triggers fit into an MLOps pipeline
Monitoring’s output is a decision: run the pipeline or start a new experiment. Data validation makes the first decision, and the decision depends on the kind of change. Google’s architecture guidance separates two kinds. A data schema skew means the input does not comply with the expected schema: the pipeline should stop so the data science team can investigate. A data values skew means the statistical properties of the data have changed: this should trigger retraining. An MLOps survey gives the reasoning. A schema change means the new features differ partly or completely from what the model was trained on, so retraining on that data cannot produce a valid model.
Further on, monitoring compares production inputs and outputs with a baseline. Data drift compares input distributions with the training data or recent production data. Prediction drift does the same for model outputs. In Model Monitoring, alerts fire when a feature’s statistical distance from its baseline exceeds a threshold set for that feature.
Retraining trigger types and when each fits
| Trigger | When it fires | What to watch |
|---|---|---|
| On demand | A person starts the pipeline. | Useful as a fallback, but it depends on someone noticing the problem. |
| Scheduled | The pipeline runs at a fixed interval. | Engineers often pick the interval arbitrarily. Monthly runs can leave the model stale. Hourly runs can retrain on data that has not yet been validated and cleaned. |
| New data available | A new batch arrives and starts a run. | In CD4ML-style setups, changing a dataset in place does not start the pipeline unless the data is versioned. |
| Performance degradation | Measured model quality drops noticeably. | Needs labels, which often arrive late. |
| Concept shift | The distributions of prediction features change significantly, even when online performance is hard to measure directly. | Not every shift is a problem. Some are harmless, and telling which is which needs domain context. |
The last row is where retraining automation meets a structural problem. Practitioners find distribution shifts easily but struggle to tell technical failures from harmless shifts. Practitioners rely on analysts and domain experts to explain the events behind them, such as campaigns, seasonality or supply disruptions. Without that context, automated retraining risks overfitting to temporary patterns. In our view, a drift alert should feed a check before retraining: is the shift persistent, and did an upstream fault cause it? It should not retrain on the alert alone.
What happens after retraining matters as much. Microsoft’s reference pattern sends a retrained model back to staging and testing before it can reach production. Promotion from staging can then go through human-approved gates. The 2024 interviewees retrained automatically but still reviewed validation metrics and supervised each promotion by hand. Many also fed live failure modes back into their validation datasets, so the same failure would be caught next time. CD4ML recommends threshold tests that stop a new model from performing worse than a known baseline.
Production AI reliability is an engineering discipline distinct from model accuracy. Its deliverables are artefacts that catch regressions before customers do: task-specific evaluation harnesses, regression suites, drift detection and release-readiness gates. The choice between scheduled batch retraining, online learning and event-triggered pipelines has its own trade-offs, covered in our comparison of batch retraining, online learning and triggered pipelines.
Why MLOps CI/CD differs from traditional CI/CD
ML CI/CD validates more than code, and it deploys a system that keeps producing new models. Google’s guidance sets out three differences. CI also tests data, data schemas and models. CD deploys a training pipeline that in turn deploys a prediction service. Continuous training (CT) is a property only ML systems have. If your question is how much of this is plain DevOps under a new name, the MLOps operating model covers that argument.
Practitioners cite non-determinism as a major testing hurdle. NVIDIA’s TAO Toolkit documentation states that several GPU kernels are non-deterministic by default, including cuDNN convolution algorithms and fused attention backward passes. They use atomic accumulation or auto-tuned algorithm selection, so gradients can differ between runs even with a fixed seed. Deterministic code paths are available, but they are slower and use more memory. Even with them, reproducibility holds only per hardware setup. Changing the GPU model, the CUDA or cuDNN version, or the number of GPUs can still change the results.
Practitioners report a related pressure in CI. In that practitioner survey, 75% expected ML projects to have longer build durations than non-ML projects. They cited model training, compute demands, data handling and dependency management as the causes. Non-determinism was among the most frequently cited testing obstacles. Respondents also noted that CI environments start fresh, so ML dependencies and CUDA libraries are reinstalled on every run. Adoption reflects this. Earlier analysis of 4,031 GitHub ML projects, cited in the survey, found that only 37% used CI services.
The CD4ML answer is to make the release process reproducible even when model outputs are not. CD4ML states the goal directly: model outputs may be hard to reproduce, but releasing them should be reliable and reproducible. For a test suite, we would assert on metric bands against a baseline, not on bitwise-identical weights. We would pin GPU architecture and library versions for any test that does check bitwise reproducibility. And we would run cheap unit and data checks before the expensive model tests, which the 2025 survey also recommends.
Sizing the pipeline to the model you actually run
Teams that install enterprise-grade MLOps infrastructure before their first production model can spend months maintaining a platform that delivers no value yet. Teams with no infrastructure discipline end up with fragility that is expensive to fix once the model becomes business-critical. Teams new to MLOps often build stacks with more tools than they need, and the cost of running those tools becomes the failure mode. How much MLOps infrastructure to build at each stage is a question of its own.
Google’s maturity model gives a sequence. Level 0 has no automation. Level 1 automates the ML pipeline. The highest level automates CI/CD as well. In our view, you are ready for level 1 when your answer to “which signal is allowed to retrain this model without a human?” is specific: a value skew on named features, against a threshold someone chose deliberately, with a comparison gate before promotion. If the answer is “the schedule”, the pipeline is automated but not yet reliable.
This is the stage where TechnoLynx’s MLOps engineering fits: a pipeline is in production, but the artefacts that catch its regressions (regression suites, drift detection wired to deliberate triggers, release gates) are missing.
