# Discovering the systematic errors made by machine learning models

DevFeed: [Discovering the systematic errors made by machine learning models](<https://devfeed.tech/articles/discovering-the-systematic-errors-made-by-machine-learning-models-7581.md>)

Original publisher: [Read original article](<https://ai.stanford.edu/blog/domino/>)

Author: Sabri Eyuboglu, Maya Varma, Khaled Saab, Jared Dunnmon, James Zou and Chris Ré

Published: 2022-04-07T07:00:00Z

Content type: article

Language: en

Sources: [The Stanford AI Lab Blog](<https://devfeed.tech/sources/the-stanford-ai-lab-blog.md>)

Topics: [Machine Learning & Artificial Intelligence](<https://devfeed.tech/topics/machine-learning-artificial-intelligence.md>), [Embeddings](<https://devfeed.tech/topics/embeddings.md>), [data](<https://devfeed.tech/topics/data.md>), [model-deployment](<https://devfeed.tech/topics/model-deployment.md>), [debug](<https://devfeed.tech/topics/debug.md>), [Optimization](<https://devfeed.tech/topics/optimization.md>)

Tags: [data](<https://devfeed.tech/tags/data.md>), [datasets](<https://devfeed.tech/tags/datasets.md>), [debug](<https://devfeed.tech/tags/debug.md>), [embeddings](<https://devfeed.tech/tags/embeddings.md>), [evaluation](<https://devfeed.tech/tags/evaluation.md>), [machine-learning](<https://devfeed.tech/tags/machine-learning.md>), [model](<https://devfeed.tech/tags/model.md>), [model-deployment](<https://devfeed.tech/tags/model-deployment.md>), [optimization](<https://devfeed.tech/tags/optimization.md>), [validation](<https://devfeed.tech/tags/validation.md>)

## AI overview

This article introduces Domino, a method for discovering systematic errors that machine learning models make on coherent, underperforming slices of validation data. It explains how slice awareness supports model evaluation, deployment decisions, debugging, robustness improvements, and safety and fairness analysis, including through cross-modal embeddings.

## Source excerpt

Discovering systematic errors with cross-modal embeddings In this blog post, we introduce Domino, a new approach for discovering systematic errors made by machine learning models. We also discuss a framework for quantitatively evaluating methods like Domino. Links: 📄 Paper (ICLR 2022) 🌍 Longer Walkthrough 💻 GitHub 📘 Docs 📒 Google Colab Machine learning models that achieve high overall accuracy often make systematic errors on coherent slices of validation data. What is a slice? A slice is a set of data samples that share a common characteristic. As an example, in large image datasets, photos of vintage cars comprise a slice (i.e. all images in the slice share a common subject). The term slice has a number of synonyms that you might be more familiar with (e.g. subgroup, subpopulation, stratum). These terms are largely interchangeable, but we'll stick with "slice" throughout this post. We say that a model underperforms on a slice if performance on the data samples in the slice is significantly worse than its overall performance. The search for underperforming slices is a critical, but often overlooked, part of model evaluation. When practitioners are aware of the slices on which their models underperform, they can make more informed decisions around model deployment. This is particularly important in safety-critical settings like medicine: a diagnostic model that underperforms on younger patients should likely not be deployed at a pediatric hospital. Slice awareness can also help practitioners debug and improve models: after an underperforming slice is identified, we can improve model robustness by either updating the training dataset or using robust optimization techniques (e.g. Sohoni et al., 2020; Sagawa et al., 2020). Deploying models that underperform on critical data slices may have significant safety or fairness consequences. For example, models trained to detect collapsed lungs in chest X-rays have been shown to make predictions based on the presence of chest d