# Scaling database with Django and HAProxy

DevFeed: [Scaling database with Django and HAProxy](<https://devfeed.tech/articles/scaling-database-with-django-and-haproxy-19985.md>)

Original publisher: [Read original article](<http://engineering.hackerearth.com/2013/10/07/scaling-database-with-django-and-haproxy/>)

Published: 2013-10-07T00:00:00Z

Content type: article

Language: en

Sources: [HackerEarth](<https://devfeed.tech/sources/hackerearth.md>)

Topics: [Databases](<https://devfeed.tech/topics/databases.md>), [MySQL](<https://devfeed.tech/topics/mysql.md>), [Scalability](<https://devfeed.tech/topics/scalability.md>), [Django](<https://devfeed.tech/topics/django.md>), [Architecture & Design](<https://devfeed.tech/topics/architecture-design.md>), [Object-relational mapping](<https://devfeed.tech/topics/orm.md>), [Replication](<https://devfeed.tech/topics/replication.md>), [Availability](<https://devfeed.tech/topics/availability.md>)

Tags: [architecture](<https://devfeed.tech/tags/architecture.md>), [availability](<https://devfeed.tech/tags/availability.md>), [database](<https://devfeed.tech/tags/database.md>), [distributed](<https://devfeed.tech/tags/distributed.md>), [django](<https://devfeed.tech/tags/django.md>), [frontend](<https://devfeed.tech/tags/frontend.md>), [haproxy](<https://devfeed.tech/tags/haproxy.md>), [load-balancer](<https://devfeed.tech/tags/load-balancer.md>), [mysql](<https://devfeed.tech/tags/mysql.md>), [replication](<https://devfeed.tech/tags/replication.md>), [scalability](<https://devfeed.tech/tags/scalability.md>), [uptime](<https://devfeed.tech/tags/uptime.md>)

## AI overview

This article describes HackerEarth's effort to scale a Django application whose MySQL database became a major bottleneck as data volume and request rates grew. The team rearchitected the system by sharding the database, adding database routers and Django ORM wrappers, placing HAProxy in front of the MySQL databases, and optimizing the codebase. It also discusses read replicas, routing writes to the master, and avoiding stale data for reads that require current results.

## Source excerpt

###MySQL - Primary data store At HackerEarth, we use MySQL database as the primary data store. We have experimented with a few NoSQL databases on the way, but the results have been largely unsatisfactory. The distributed databases like MongoDB or CouchDB aren't very scalable or stable. Right now, our status monitoring services use RethinkDB for storing the data in JSON format and that's all for the NoSQL database usage right now. With the growing data and number of requests/sec, it turns out that the database becomes the major bottlneck to scale the application dynamically. At this point if you are thinking that there are mythical (cloud) providers who can handle the growing need of your application, you can't be more wrong. To make the problem even harder, you can't spin a new database whenever you want to just like your frontend servers. To achieve a horizontal scalability at all levels, it requires massive rearchitecture of the system while being completely transparent to the end user. This is what a part of our team has focussed on in last few months, resulting in very high uptime and availability. The master (and only) MySQL database had started being under heavy load recently. We thought we will delay any scalability at this level till the single database could handle the load, and we will work on other high priority things. But that was not supposed to go as planned and we experienced a few downtimes. After that we did a rearchitecture of our application, sharded the database, wrote database routers and wrappers on top of django ORM, put HAProxy load balancer infront of the MySQL databases, and refactored our codebase to optimize it significantly. The image below shows a part of the architecture we have at HackerEarth. Many other components have been omitted for simplicity. ###Database slaves and router The idea was to create read replicas and route the write queries to master database and read queries to slave (read replica) databases. But that was not so si