# Logging millions of requests everyday and what it takes

DevFeed: [Logging millions of requests everyday and what it takes](<https://devfeed.tech/articles/logging-millions-of-requests-everyday-and-what-it-takes-19996.md>)

Original publisher: [Read original article](<http://engineering.hackerearth.com/2015/02/26/logging-millions-requests-what-it-takes/>)

Published: 2015-02-26T00:00:00Z

Content type: article

Language: en

Sources: [HackerEarth](<https://devfeed.tech/sources/hackerearth.md>)

Topics: [Logging](<https://devfeed.tech/topics/logging.md>), [Django](<https://devfeed.tech/topics/django.md>), [Scalability](<https://devfeed.tech/topics/scalability.md>), [Architecture & Design](<https://devfeed.tech/topics/architecture-design.md>), [log management](<https://devfeed.tech/topics/log-management.md>), [Kafka](<https://devfeed.tech/topics/kafka.md>), [Back end](<https://devfeed.tech/topics/backend.md>), [MongoDB](<https://devfeed.tech/topics/mongodb.md>)

Tags: [architecture](<https://devfeed.tech/tags/architecture.md>), [django](<https://devfeed.tech/tags/django.md>), [kafka](<https://devfeed.tech/tags/kafka.md>), [logging](<https://devfeed.tech/tags/logging.md>), [logs](<https://devfeed.tech/tags/logs.md>), [metrics](<https://devfeed.tech/tags/metrics.md>), [mongodb](<https://devfeed.tech/tags/mongodb.md>), [scalability](<https://devfeed.tech/tags/scalability.md>), [scale](<https://devfeed.tech/tags/scale.md>), [stateless](<https://devfeed.tech/tags/stateless.md>), [thread](<https://devfeed.tech/tags/thread.md>)

## AI overview

This article describes HackerEarth's architecture for collecting and processing millions of web request logs. It explains how Django middleware asynchronously forwards request data to a stateless Transporter Cluster, which routes messages to systems such as Kafka, MongoDB, and RabbitMQ while keeping web-server overhead low and supporting horizontal scaling.

## Source excerpt

HackerEarth's web servers handle millions of requests every day. These request logs can be analyzed to mine some highly useful insights as well as metrics critical for the business, for example, no. of views per day, no. of views per sub product, most popular user navigation flow etc. Initial Thoughts HackerEarth uses Django as its primary web development framework and a host of other components which have been customized for performance and scalability. During normal operations, our servers handle 80-90 requests/sec on an average and this surges to 200-250 requests/sec when multiple contests overlap in a time delta. We needed a system which could easily scale to a peak traffic 500 requests/sec. Also, this system should add minimum processing overhead to the webservers and the data collected should be stored for crunching and offline processing. Architecture The diagram above shows a high level architecture of our request log collection system. The solid connection lines represent the data flow between different components and the dotted lines represent the communications. The whole architecture is message based and stateless and so individual components can easily be removed/replaced without any downtime. Below is a more detailed explanation about each component in the order of data flow. Web Servers On the web servers, we employ a Django Middleware that asynchronously retrieves required data for a given request and then forwards it to the Transporter Cluster servers. This is done using a thread and the middleware adds an overhead of 2 milli seconds to the Django request/response cycle. class RequestLoggerMiddleware(object): """ Logs data from requests """ def process_request(self, request): if settings.LOCAL or settings.DEBUG: return None if request.is_ajax(): is_ajax = True request.META['IS_AJAX'] = is_ajax before = datetime.datetime.now() DISALLOWED_USER_AGENTS = ["ELB-HealthChecker/1.0"] http_user_agent = request.environ.get('HTTP_USER_AGENT','') if http_user_a