# Speeding things up with Redshift

DevFeed: [Speeding things up with Redshift](<https://devfeed.tech/articles/speeding-things-up-with-redshift-19701.md>)

Original publisher: [Read original article](<https://word.bitly.com/post/48854093418>)

Author: Wordbitly

Published: 2013-04-25T14:30:19Z

Content type: opinion

Language: en

Sources: [Bitly](<https://devfeed.tech/sources/bitly.md>)

Topics: [Amazon Redshift](<https://devfeed.tech/topics/amazon-redshift.md>), [Data analysis](<https://devfeed.tech/topics/data-analysis.md>), [SQL](<https://devfeed.tech/topics/sql.md>), [Hadoop](<https://devfeed.tech/topics/hadoop.md>), [Python](<https://devfeed.tech/topics/python.md>), [amazon](<https://devfeed.tech/topics/amazon.md>)

Tags: [amazon](<https://devfeed.tech/tags/amazon.md>), [analysis](<https://devfeed.tech/tags/analysis.md>), [data-analysis](<https://devfeed.tech/tags/data-analysis.md>), [gotchas](<https://devfeed.tech/tags/gotchas.md>), [hadoop](<https://devfeed.tech/tags/hadoop.md>), [python](<https://devfeed.tech/tags/python.md>), [redshift](<https://devfeed.tech/tags/redshift.md>), [speed](<https://devfeed.tech/tags/speed.md>), [sql](<https://devfeed.tech/tags/sql.md>), [workflow](<https://devfeed.tech/tags/workflow.md>)

## AI overview

The article describes bitly's experience using Amazon Redshift to speed up ad hoc analysis of large volumes of click data. It contrasts Redshift SQL queries with a slower Hadoop and Python MapReduce workflow, reporting answers within minutes or seconds and an overall positive experience, while noting some gotchas.

## Source excerpt

Recently we've started to experiment with using Redshift, Amazon's new data warehousing service. More specifically, we're using it to speed up and expand our ad hoc data analysis. The Challenge bitly sees billions of clicks and shortens each month. Often we have various questions about the data generated from this activity. Sometimes these questions are driven by business needs (how much traffic do we see from a potential enterprise customer), sometimes they are more technically driven (how much traffic will a new sub-system need to deal with), and sometimes we like to just have fun (what are the top trashy celeb stories this week). Unfortunately, when working with that volume of data it can be pretty difficult to do much of anything quickly. Pre-Redshift, all of these questions were answered by writing map-reduce jobs to be run on our Hadoop cluster or on Amazon's EMR. Whenever we wanted to answer a question with our data, the process would look something like this: Write map-reduce job in Python Run it on some local test data Fix bugs. Run it on the Hadoop cluster Wait 20-30 minutes for results Get an error back from Hadoop Dig through the logs to find the error. GOTO 3 This is clearly not ideal when all you want to do is get a simple count. For a lot of the work we do Hadoop + Python make for an awesome combination, but for these ad hoc aggregation queries they're very blunt instruments. In both cases, they are general purpose tools that are super flexible, but slow and difficult to use for this specific use case. Redshift, on the other hand, is specifically built and optimized for doing aggregation queries over large sets of data. When we want to answer a question with Redshift, we just write a SQL query and get an answer within a few minutes--if not seconds. Overall, our experience with Redshift has been a positive one but we have run into some gotchas that we'll get into below. The Good News User Experience From a user perspective, we're really happy with Redsh