# Real-time Big Data at Target

DevFeed: [Real-time Big Data at Target](<https://devfeed.tech/articles/real-time-big-data-at-target-20394.md>)

Original publisher: [Read original article](<https://target.github.io/analytics/big-data-storm>)

Author: Target Brands, Inc

Published: 2015-11-11T06:00:00Z

Content type: article

Language: en

Sources: [Target](<https://devfeed.tech/sources/target.md>)

Topics: [big-data](<https://devfeed.tech/topics/big-data.md>), [Hadoop](<https://devfeed.tech/topics/hadoop.md>), [stream-processing](<https://devfeed.tech/topics/stream-processing.md>), [streaming-data-processing](<https://devfeed.tech/topics/streaming-data-processing.md>), [Kafka](<https://devfeed.tech/topics/kafka.md>), [Latency](<https://devfeed.tech/topics/latency.md>), [Low Latency](<https://devfeed.tech/topics/low-latency.md>), [Open Source](<https://devfeed.tech/topics/open-source.md>)

Tags: [analytics](<https://devfeed.tech/tags/analytics.md>), [apache-kafka](<https://devfeed.tech/tags/apache-kafka.md>), [batch](<https://devfeed.tech/tags/batch.md>), [big-data](<https://devfeed.tech/tags/big-data.md>), [data](<https://devfeed.tech/tags/data.md>), [hadoop](<https://devfeed.tech/tags/hadoop.md>), [kafka](<https://devfeed.tech/tags/kafka.md>), [latency](<https://devfeed.tech/tags/latency.md>), [low-latency](<https://devfeed.tech/tags/low-latency.md>), [open-source](<https://devfeed.tech/tags/open-source.md>), [real-time](<https://devfeed.tech/tags/real-time.md>), [storm](<https://devfeed.tech/tags/storm.md>), [stream-processing](<https://devfeed.tech/tags/stream-processing.md>), [streaming](<https://devfeed.tech/tags/streaming.md>)

## AI overview

Target's Big Data platform team describes its effort to move data into Hadoop in real time using flexible open source components. The article outlines requirements for resilience, message-format support, usability, and low latency, then evaluates Apache Storm and identifies recovery and batching-related latency issues in testing.

## Source excerpt

An enterprise as large as Target generates a lot of data and on my Big Data platform team we want to make it as easy as possible for our users to get it into Hadoop in real-time. I want to discuss how we are starting to approach this problem, what we've done so far, and what is still to come. Our requirements We wanted to build a system with flexible open source components. Our experience with proprietary products on Hadoop is that they tend to be inflexible and only work with a narrow set of use cases. That can also be true of open source products, but we have found them to be easier to adapt to our needs. Indeed, that ended up being the case as we went through this journey and made several contributions to Apache projects. More specifically, we wanted to create a system that: was highly resilient to the failure of any individual component supported a variety of message formats delivered data to Hadoop with very low latency made the data immedaiately usable streamed data from Apache Kafka sources A Streaming Framework There are many excellent comparisons of streaming frameworks available, and I won't attempt to recreate them here. The first criteria we considered was what tooling was needed to monitor and administer the streaming framework, with a strong preference to use our primary Hadoop administration tool, Apache Ambari. Apache Storm fit that bill and was also a proven solution for stream processing. If Storm could meet our other requirements, it would be our first choice. To test its resiliency we ran a simple scenario: start a data stream into Hadoop, disable HDFS, and then reenable it. Streaming would obviously fail while HDFS was disabled, but we needed the system to recover gracefully when HDFS came back online. Unfortunately our first test of this scenario left our Storm topology in an unrecoverable state, which required a manual restart. That's not something we could live with. We also needed very fine control over the latency of arriving data. In gener