# Logging millions of requests reliably with our new data ingestion pipeline

DevFeed: [Logging millions of requests reliably with our new data ingestion pipeline](<https://devfeed.tech/articles/logging-millions-of-requests-reliably-with-our-new-data-ingestion-pipeline-20018.md>)

Original publisher: [Read original article](<http://engineering.hackerearth.com/2022/07/01/logging-millions-of-requests-reliably-with-our-new-data-ingestion-pipeline/>)

Published: 2022-07-01T00:00:00Z

Content type: article

Language: en

Sources: [HackerEarth](<https://devfeed.tech/sources/hackerearth.md>)

Topics: [Logging](<https://devfeed.tech/topics/logging.md>), [Amazon Redshift](<https://devfeed.tech/topics/amazon-redshift.md>), [log management](<https://devfeed.tech/topics/log-management.md>), [real-time](<https://devfeed.tech/topics/real-time.md>), [Amazon Web Services](<https://devfeed.tech/topics/aws.md>), [cloud-infrastructure](<https://devfeed.tech/topics/cloud-infrastructure.md>), [web applications](<https://devfeed.tech/topics/web-applications.md>), [Security](<https://devfeed.tech/topics/security.md>), [Architecture & Design](<https://devfeed.tech/topics/architecture-design.md>)

Tags: [analytics](<https://devfeed.tech/tags/analytics.md>), [aws](<https://devfeed.tech/tags/aws.md>), [infrastructure](<https://devfeed.tech/tags/infrastructure.md>), [logging](<https://devfeed.tech/tags/logging.md>), [pipeline](<https://devfeed.tech/tags/pipeline.md>), [real-time](<https://devfeed.tech/tags/real-time.md>), [redshift](<https://devfeed.tech/tags/redshift.md>), [security](<https://devfeed.tech/tags/security.md>), [server](<https://devfeed.tech/tags/server.md>)

## AI overview

HackerEarth describes redesigning its request-logging architecture to reliably ingest millions of HTTP/HTTPS requests in near real time. The new pipeline uses Kinesis Firehose data streams to deliver compressed and encrypted data from web servers to Amazon Redshift, reducing operational overhead and infrastructure cost while providing retries and delivery within 10-15 minutes.

## Source excerpt

Introduction HackerEarth handles millions of requests every day. To understand the user access patterns, get the usage of any particular feature or a page, figure out the daily active users or users who have been active for the past 6 months, etc in near real time, it is important to stream that data from across different services and ingest it to the analytics pipeline reliably. Problem Our old request logging architecture was complex and has many moving components. There were a lot of operational overheads involved in maintaining and scaling each of those components independently to ensure that all the self-hosted components were up and running all the time. Architecture Solution Last year, we revamped the way we log our web requests. It was done mainly to increase the reliability in logging the HTTP/HTTPS request data from across web services and also to reduce the operational overheads and the infrastructure cost associated with it. The new flow is making use of Kinesis Firehose data streams to deliver the request data from our web servers to Redshift (the database that we use to log and query request data) reliably with much lower cost. Amazon Kinesis Firehose is a fully managed service that automatically scales to match the throughput of our incoming request log data and requires no ongoing administration. It also allows us to compress and encrypt the data before loading it, minimizing the amount of storage used at the destination with increased security and we only need to pay for the amount of data we transmit through the service. The new flow is a fully managed solution with almost no operational overhead. We tried to keep the flow simple and straightforward with less number of moving components. The request data is now guaranteed to appear in the Redshift table within 10-15 minutes from the point we received a request at our web server's end. There are retries configured between consecutive infra components in the new flow to make sure there are no message