# ML Prediction Pipeline Orchestration with ML Control Center

DevFeed: [ML Prediction Pipeline Orchestration with ML Control Center](<https://devfeed.tech/articles/ml-prediction-pipeline-orchestration-with-ml-control-center-22613.md>)

Original publisher: [Read original article](<https://medium.com/glassdoor-engineering/ml-prediction-pipeline-orchestration-with-ml-control-center-812f46631484?source=rss----288d984af747---4>)

Author: Vance Thornton

Published: 2022-04-01T23:32:32Z

Content type: article

Language: en

Sources: [Glassdoor Engineering](<https://devfeed.tech/sources/glassdoor-engineering.md>)

Topics: [MLOps](<https://devfeed.tech/topics/mlops.md>), [Machine learning](<https://devfeed.tech/topics/machine-learning.md>), [YAML](<https://devfeed.tech/topics/yaml.md>), [data-processing](<https://devfeed.tech/topics/data-processing.md>), [GitOps](<https://devfeed.tech/topics/gitops.md>), [model-deployment](<https://devfeed.tech/topics/model-deployment.md>), [Open Source](<https://devfeed.tech/topics/open-source.md>), [Library](<https://devfeed.tech/topics/library.md>), [real-time](<https://devfeed.tech/topics/real-time.md>)

Tags: [data-pipeline](<https://devfeed.tech/tags/data-pipeline.md>), [data-processing](<https://devfeed.tech/tags/data-processing.md>), [gitops](<https://devfeed.tech/tags/gitops.md>), [library](<https://devfeed.tech/tags/library.md>), [low-latency](<https://devfeed.tech/tags/low-latency.md>), [machine-learning](<https://devfeed.tech/tags/machine-learning.md>), [ml](<https://devfeed.tech/tags/ml.md>), [mlops](<https://devfeed.tech/tags/mlops.md>), [mlops-platform](<https://devfeed.tech/tags/mlops-platform.md>), [models](<https://devfeed.tech/tags/models.md>), [open-source](<https://devfeed.tech/tags/open-source.md>), [orchestration](<https://devfeed.tech/tags/orchestration.md>), [pipeline](<https://devfeed.tech/tags/pipeline.md>), [real-time](<https://devfeed.tech/tags/real-time.md>), [yaml](<https://devfeed.tech/tags/yaml.md>)

## AI overview

This article describes Glassdoor's ML Control Center, an MLOps project for connecting machine learning components and orchestrating prediction pipelines. Pipelines are defined with YAML configuration, support reusable and custom components, and can run for online, near-real-time, offline batch, or automated testing use cases.

## Source excerpt

A YAML based approach for building data processing pipelinesIntroduction In a previous blog post we described Glassdoor's a la carte approach to MLOps. We chose this approach because the MLOps landscape is rapidly evolving and we want to have the flexibility to use the best available options. ML Control Center (MLCC) is a project that we are developing at Glassdoor which will act as the glue to connect and unify these disparate components. Our plan is to move to a GitOps oriented approach where YAML configuration files in our Git repositories control the deployment and execution of all the tasks and services needed for our ML projects. One part of this is support for ML prediction pipelines which we recently released as an open source project at: https://gitlab.com/glassdoor-open-source/ml-control-center. A ML prediction pipeline typically involves gathering feature data from various sources, providing the data to feature extractors and/or ML models for prediction, and then writing the output to a data store. In MLCC these pipelines are defined using YAML files which specify the configuration of the operations to perform and the flow of data input/output of those operations. One of our primary design goals is to make it easy to implement the most common use cases quickly with simple YAML configuration and minimal coding. We want to allow ML scientists and engineers to focus on defining what they want the pipeline to do with many of the engineering implementation details such as parallelization, metrics, and retry logic taken care of automatically. MLCC has a library of reusable components which provide the functionality that is typically needed for our use cases and it is easy to add custom components when needed. This approach promotes flexibility and reuse by encouraging a modular implementation. Another trend that we have seen at Glassdoor is that online and near real time ML processing is becoming more common. The design of MLCC prediction pipelines makes them w