# Post-mortem: The big outage on January 25, 2014

DevFeed: [Post-mortem: The big outage on January 25, 2014](<https://devfeed.tech/articles/post-mortem-the-big-outage-on-january-25-2014-19989.md>)

Original publisher: [Read original article](<http://engineering.hackerearth.com/2014/01/27/big-outage-25-january/>)

Published: 2014-01-27T00:00:00Z

Content type: article

Language: en

Sources: [HackerEarth](<https://devfeed.tech/sources/hackerearth.md>)

Topics: [Post Mortem](<https://devfeed.tech/topics/post-mortem.md>), [servers](<https://devfeed.tech/topics/servers.md>), [Website](<https://devfeed.tech/topics/website.md>), [Django](<https://devfeed.tech/topics/django.md>), [Python](<https://devfeed.tech/topics/python.md>)

Tags: [coding](<https://devfeed.tech/tags/coding.md>), [django](<https://devfeed.tech/tags/django.md>), [engineering](<https://devfeed.tech/tags/engineering.md>), [outage](<https://devfeed.tech/tags/outage.md>), [post-mortem](<https://devfeed.tech/tags/post-mortem.md>), [python](<https://devfeed.tech/tags/python.md>), [servers](<https://devfeed.tech/tags/servers.md>)

## AI overview

This post-mortem explains why HackerEarth's January Jackpot 2014 challenge was cancelled during a January 25, 2014 outage. Requests accumulated in queues and timed out even though server CPU usage stayed below 30%, so adding servers did not help. The article attributes the problem to collecting large amounts of user-activity data for a new search engine while using an old, non-thread-safe Haystack wrapper that threw exceptions nondeterministically.

## Source excerpt

25th January was a rather unfortunate day for us. The monthly challenge - January Jackpot 2014 which was scheduled at 9:30 PM that day was cancelled due to turn of events going wrong at the worst possible time. We regret once again for the incovenience caused to you, and this is a postemortem of what really happened behind the scenes. It was Saturday, and the day was sunny here. Everything was running smoothly as usual. The whole HackerEarth team was out roaming in Bangalore, went to a lake, did boat ride, went for a lunch then and came back to office in evening. We all sang with the guitar and played counter strike. At the same time, there was a college contest - Epiphany January Challenge going on. Everything was smooth, the servers were sending replies happily and we didn't have to worry about anything. This is the email I received in the evening from Ravi from NIT Surat who handles the coding contests there. from: Ravi Ojha to: Vivek Prakash date: Sat, Jan 25, 2014 at 7:37 PM First, Thank You for HackerEarth!! Its such a lucid platform for organizing online contests. 
 ####How it went wrong Little did we know that there was a catastrophe waiting for us in the contest that was going to run at 9:30 PM. First let me assure you that the last thing we need to worry about is the number of requests that hit our servers. Read [Scaling Python/Django application with Apache and mod_wsgi](http://engineering.hackerearth.com/2013/11/21/scaling-python-django-application-apache-mod_wsgi/) to understand why. And in this case, there were not many as the website had stopped respoding anyway. Here is the request graph of last one week: The graph shows two significant spikes and two small spikes. The first one at 15,000 was on 24th January. The second one at 20,000+ was in the day on 25th January when Epiphany January contest was going on. The spike which just crosses 5000 in the later part of 25th was for the January Jackpot challenge. It's clear that something else was wrong.