# How to Process Large Data Sets with Ruby

DevFeed: [How to Process Large Data Sets with Ruby](<https://devfeed.tech/articles/how-to-process-large-data-sets-with-ruby-21036.md>)

Original publisher: [Read original article](<https://jakeyesbeck.com/2015/08/09/how-to-process-large-data-sets-with-ruby/>)

Published: 2015-08-09T12:00:00Z

Content type: tutorial

Language: en

Sources: [Jake Yesbeck](<https://devfeed.tech/sources/jake-yesbeck.md>)

Topics: [Ruby](<https://devfeed.tech/topics/ruby.md>), [Rails](<https://devfeed.tech/topics/rails.md>), [migration](<https://devfeed.tech/topics/migration.md>), [Redis](<https://devfeed.tech/topics/redis.md>), [systems](<https://devfeed.tech/topics/systems.md>), [jobs](<https://devfeed.tech/topics/jobs.md>)

Tags: [activerecord](<https://devfeed.tech/tags/activerecord.md>), [data-migrations](<https://devfeed.tech/tags/data-migrations.md>), [how-to](<https://devfeed.tech/tags/how-to.md>), [jobs](<https://devfeed.tech/tags/jobs.md>), [rails](<https://devfeed.tech/tags/rails.md>), [redis](<https://devfeed.tech/tags/redis.md>), [ruby](<https://devfeed.tech/tags/ruby.md>), [ruby-on-rails](<https://devfeed.tech/tags/ruby-on-rails.md>), [scale](<https://devfeed.tech/tags/scale.md>), [time](<https://devfeed.tech/tags/time.md>)

## AI overview

This tutorial explains how to process a large Ruby on Rails data migration more efficiently. It first shows why a serial script for updating two million users is too slow, then introduces Resque and Redis to distribute the work across background jobs and multiple workers.

## Source excerpt

The need for data migrations in mature systems is real. At times, requests for these migrations can appear at random. One minute, a system is behaving as specified, happily fulfilling requests, and then bam! All the user objects suddenly need an extremely crucial attribute. Well that seems relatively simple, right? All that is needed is a simple ruby script to iterate over all users and update every user with this essential piece of data. To demonstrate such a problem, we can assume the following: A Ruby on Rails application exists with a User class, each with a phone_number attribute. class User < ActiveRecord::Base validates_presence_of :phone_number end The application is relatively popular, which results in 2,000,000 users: User.count #=> 2000000 Finally, the requested task: Add a "+1" to the beginning of all user phone numbers (we are assuming that all phone numbers belong to users in the USA or Canada). Disclaimer: Ruby is probably not the best tool for this kind of data migration but for argument's sake we can assume it is the only one available. Serial scripts at scale are slow Without giving it too much thought, an approach to solve this problem might look something like: User.find_each do |user| user.phone_number = "+1#{ user.phone_number }" user.save! end This approach will work. The find_each method will make sure the memory footprint of the script stays low (it will not load every user into memory at once) and the phone numbers will be updated. However, this will be painfully slow. Even if the system is able to update 20 users per second, it will take approximately 27 hours to complete. 2,000,000 users / 20 users per second = 100,000 seconds to process all users 100,000 seconds / 60 seconds ~= 1,666 minutes 1,666 minutes / 60 minutes ~= 27 hours Resque to the Rescue Resque is a very useful Ruby library for creating background jobs. Redis is utilized as the storage for these jobs and individual Resque workers pick one job off each queue at a time. With a