# Using Play validation with Scala for strongly typed Hadoop and Spark data pipelines

DevFeed: [Using Play validation with Scala for strongly typed Hadoop and Spark data pipelines](<https://devfeed.tech/articles/strange-bedfellows-how-a-web-tier-validation-framework-enables-strongly-typed-big-data-pipelines-20850.md>)

Original publisher: [Read original article](<http://themodernlife.net/scala/validation/play/spark/2014/08/07/serlialization-validation-in-play-and-spark/>)

Published: 2014-08-07T11:50:13Z

Content type: article

Language: en

Sources: [Ian Hummel](<https://devfeed.tech/sources/ian-hummel.md>)

Topics: [Scala](<https://devfeed.tech/topics/scala.md>), [data-processing](<https://devfeed.tech/topics/data-processing.md>), [Apache Spark](<https://devfeed.tech/topics/spark.md>), [Hadoop](<https://devfeed.tech/topics/hadoop.md>), [Back end](<https://devfeed.tech/topics/backend.md>), [Exception](<https://devfeed.tech/topics/exception.md>)

Tags: [backend](<https://devfeed.tech/tags/backend.md>), [big-data](<https://devfeed.tech/tags/big-data.md>), [data-processing](<https://devfeed.tech/tags/data-processing.md>), [framework](<https://devfeed.tech/tags/framework.md>), [hadoop](<https://devfeed.tech/tags/hadoop.md>), [play](<https://devfeed.tech/tags/play.md>), [scala](<https://devfeed.tech/tags/scala.md>), [spark](<https://devfeed.tech/tags/spark.md>), [validation](<https://devfeed.tech/tags/validation.md>)

## AI overview

The article examines whether Play's validation API, commonly used for HTML form and JSON submissions, can also support backend data pipelines powered by Hadoop or Spark. It describes applying validation during translation of encoded records into strongly typed Scala data.

## Source excerpt

The other day I was talking with a colleague about data validation and the Play web framework came up. Play has a nice API for validating HTML form and JSON submissions. This works great when you're processing small amounts of data from the web-tier of your application. But could that same tech benefit a Big Data team working on a backend powered by Hadoop or Spark? We decided to find out and the results were encouraging. The secret sauce? Play's combinator-based approach to data validation. Whether your data is big or small, garbage in is garbage out MediaMath processes TBs of online user behavior and advertising data every day. It's inevitable that with hundreds of machines spread across multiple datacenters, legacy systems and partner provided APIs we receive bad data or invalid records from time to time. Systems built around file formats like CSV or TSV are especially susceptible to encoding errors that can cause headaches for downstream processing systems. So what are your options? Quite often the first step for most data processing pipelines (be they single node scripts or massive Hadoop jobs) is translating some kind of encoded wire format T into a record of type D for partitioning, joining, filtering or aggregating. In mathematical terms, you need a function translate: (input: T) => D where input could be a parsed JSON object, a snippet of XML, an array of bytes or an array of stings in the case of tab or comma delimited files. But what if the translation fails? Think about the scenario of processing a CSV file line by line. Each line has columns of different types (strings, integers, floating point numbers). What if someone puts "#@$?" where you were expecting a number? Or leaves a required field blank? In other words, our function is only defined for some values of T (it's a partial function). At MediaMath we use Scala, so the natural choice would be to model this by throwing an exception or returning an Option[D]. That said, a richer validation API would