# Optimizing Text Processing for 1.75 Billion Lines of Hadoop Output

DevFeed: [Optimizing Text Processing for 1.75 Billion Lines of Hadoop Output](<https://devfeed.tech/articles/adventures-in-optimizing-text-processing-19708.md>)

Original publisher: [Read original article](<https://word.bitly.com/post/74069870671>)

Author: Wordbitly

Published: 2014-01-21T16:34:03Z

Content type: article

Language: en

Sources: [Bitly](<https://devfeed.tech/sources/bitly.md>)

Topics: [Hadoop](<https://devfeed.tech/topics/hadoop.md>), [data](<https://devfeed.tech/topics/data.md>), [Python](<https://devfeed.tech/topics/python.md>)

Tags: [bash](<https://devfeed.tech/tags/bash.md>), [data](<https://devfeed.tech/tags/data.md>), [hadoop](<https://devfeed.tech/tags/hadoop.md>), [performance](<https://devfeed.tech/tags/performance.md>), [python](<https://devfeed.tech/tags/python.md>)

## AI overview

The article describes attempts to post-process 1.75 billion lines of Hadoop output from an EMR job. It examines Hadoop output plugins, secondary Hadoop jobs, and shell-based text-processing approaches for separating data by account and metric type, including their operational and performance problems.

## Source excerpt

Lessons learned while post-processing 1.75 billion lines of Hadoop output. The Problem Recently, I encountered a problem. I had a nightly Hadoop job running on EMR that churned over the past 30 days' worth of Bitly redirect data in order to run reach analysis pertaining to about 1000 of our paid accounts. This job resulted in 175 gzipped "part" files, each containing at least 10 million lines of data. I needed to collate that data after the Hadoop job ran. > ls part-*.gz part-0000.gz ... part-0174.gz > zcat part-0000.gz | wc -l 10000000 The Hadoop output data inside the part files consisted of things like this: "3,g,05-02,12SIMV6" 329 "175,geo,05,US,GA,Atlanta" 9987 "10,phrase,05,egg foo young" 1093 "11,n_clicks,05" 393999 Those were comma-delimited keys with the following structure and a count: "[ACCOUNT_ID],[METRIC_TYPE],[DATE],[VALUE]" COUNT The challenge was this: How do I efficiently separate out this data by ACCOUNT_ID and METRIC_TYPE? That is, I wanted one file per ACCOUNT_ID-METRIC_TYPE combination. First, Look on the Shelf Like many people churning through volumes of data, we make use of the the mr_job python package for our Hadoop processing. At first I thought this was a no-brainer: "I'll use the oddjob plugin. Yay, a solution already exists!" The plugin's description was tailor-made for me: "oddjob.MultipleJSONOutputFormat - Writes to the directories specified by the first element in the key" - https://github.com/jblomo/oddjob Wrong. oddjob plugin wouldn't run at all on our Hadoop cluster. oddjob plugin wouldn't run consistently on EMR This approach resulted in 890 x 175 x 5 = ~800K part files. To scp 800K files from EMR is a nightmare of a long time. Secondary Hadoop Jobs After days of struggling with oddjob, I cut bait on it and looked at running a set of secondary Hadoop jobs, using the output from the first Hadoop job as input to the second ones. Something like this: for account_id in $account_ids do run_emr_job_to_extract $account_id done Even if ea