# Implementing Data Validation with Great Expectations in Hybrid Environments

DevFeed: [Implementing Data Validation with Great Expectations in Hybrid Environments](<https://devfeed.tech/articles/implementing-data-validation-with-great-expectations-in-hybrid-environments-28036.md>)

Original publisher: [Read original article](<https://tech.trivago.com/post/2023-04-25-implementing-data-validation-with-great-expectations-in-hybrid-environments/>)

Author: Kamila Widyanto Full time DevOps; Site Reliability Engineer; Part Time Rubberduck Linkedin Profile

Published: 2023-04-25T00:00:00Z

Content type: tutorial

Language: en

Sources: [Trivago](<https://devfeed.tech/sources/trivago.md>)

Topics: [data-processing](<https://devfeed.tech/topics/data-processing.md>), [Hadoop](<https://devfeed.tech/topics/hadoop.md>), [hdfs](<https://devfeed.tech/topics/hdfs.md>), [integrity](<https://devfeed.tech/topics/integrity.md>), [Apache Spark](<https://devfeed.tech/topics/spark.md>), [configuration](<https://devfeed.tech/topics/configuration.md>), [Library](<https://devfeed.tech/topics/library.md>), [Python](<https://devfeed.tech/topics/python.md>), [JSON](<https://devfeed.tech/topics/json.md>), [Open Source](<https://devfeed.tech/topics/open-source.md>), [YAML](<https://devfeed.tech/topics/yaml.md>), [version-control](<https://devfeed.tech/topics/version-control.md>)

Tags: [configuration](<https://devfeed.tech/tags/configuration.md>), [data](<https://devfeed.tech/tags/data.md>), [data-pipeline](<https://devfeed.tech/tags/data-pipeline.md>), [data-processing](<https://devfeed.tech/tags/data-processing.md>), [data-validation](<https://devfeed.tech/tags/data-validation.md>), [devops](<https://devfeed.tech/tags/devops.md>), [distributed](<https://devfeed.tech/tags/distributed.md>), [hadoop](<https://devfeed.tech/tags/hadoop.md>), [hdfs](<https://devfeed.tech/tags/hdfs.md>), [integrity](<https://devfeed.tech/tags/integrity.md>), [json](<https://devfeed.tech/tags/json.md>), [library](<https://devfeed.tech/tags/library.md>), [open-source](<https://devfeed.tech/tags/open-source.md>), [python](<https://devfeed.tech/tags/python.md>), [version-control](<https://devfeed.tech/tags/version-control.md>), [workflow](<https://devfeed.tech/tags/workflow.md>), [yaml](<https://devfeed.tech/tags/yaml.md>)

## AI overview

This article describes implementing Great Expectations for data validation in a hybrid Hadoop environment. It explains the framework's core concepts and how the authors ran it as a PySpark job in an automated data pipeline, including configuring the Data Context for HDFS constraints.

## Source excerpt

Data validation is an essential step in any data processing pipeline, as it ensures the integrity and accuracy of the data to be used across all subsequent processing steps.