# HackerEarth Blog about engineering and technology at HackerEarth. This is one page of public article previews, not the complete archive. Follow Next page to continue. Summaries are not the original full articles. ## Computing accurate skill percentile with DDSketch DevFeed: [Computing accurate skill percentile with DDSketch]() Original publisher: [Read original article]() Published: 2023-09-17T00:00:00Z Content type: article Language: en Sources: [HackerEarth]() Topics: [Algorithms, Complexity](), [benchmarking](), [data](), [Data structures](), [real-time]() Tags: [algorithms](), [analysis](), [benchmark](), [benchmarking](), [benchmarks](), [complexity](), [data](), [flow](), [pipeline](), [real-time](), [space]() ### AI overview This article explains how HackerEarth built a global benchmarking platform for candidate skill percentiles. It replaced a cron-based, deterministic approach with data ingestion and read flows using probabilistic algorithms, especially DDSketch, to estimate quantiles from compressed data. The article compares DDSketch with T-Digest using accuracy, time, and space complexity analyses. ### Source excerpt Introduction HackerEarth has lots of candidates getting evaluated on a daily basis. We have a feature that benchmarks candidates across the platform. Benchmarking is the process of creating the profile of the ideal candidate for a position, and then measuring all candidates against that profile. To benchmark candidate skills against our millions of candidates, we decided to move away from our regular cron solution to build a more reliable and accurate data pipeline. To support this, we created a new data ingestion flow and data read flow. We moved away from our deterministic algorithms to probabilistic algorithms with DDSketch. Problem Our old benchmarking solution was trying to compute the global benchmarking of a candidate on the fly by calculating the solve percentage of the individual skills and returning the geometric mean of all the skill benchmarks. We handle huge volumes of data every day. Analyzing this data itself--for example, calculating a quantile was optimal in terms of resources. Solution We came up with a solution that computes an approximate quantile from a compressed representation of that data. We first need to appropriately summarize that data without incurring an excessive loss of fidelity. We do this by creating a sketch. Sketch algorithms generate sketches: smaller, more manageable data structures, from which we can calculate some properties of the original data. We considered various algorithms to accurately compute percentiles on noisy, large-scale, real-time data that we were receiving from candidates' skill scores. We considered using Tdigest and DDSketch. For our use case, DDSketch served the purpose.We did a POC and compared the accuracy of both the algorithms as shown below to come up with a finalized algorithm. POC results and observations We compared the actual percentile ranges in comparison to the two probabilistic approaches we mentioned(DDSketch and T-DIgest), and these were the results. Note: we have run these tests on the random ## Logging millions of requests reliably with our new data ingestion pipeline DevFeed: [Logging millions of requests reliably with our new data ingestion pipeline]() Original publisher: [Read original article]() Published: 2022-07-01T00:00:00Z Content type: article Language: en Sources: [HackerEarth]() Topics: [Logging](), [Amazon Redshift](), [log management](), [real-time](), [Amazon Web Services](), [cloud-infrastructure](), [web applications](), [Security](), [Architecture & Design]() Tags: [analytics](), [aws](), [infrastructure](), [logging](), [pipeline](), [real-time](), [redshift](), [security](), [server]() ### AI overview HackerEarth describes redesigning its request-logging architecture to reliably ingest millions of HTTP/HTTPS requests in near real time. The new pipeline uses Kinesis Firehose data streams to deliver compressed and encrypted data from web servers to Amazon Redshift, reducing operational overhead and infrastructure cost while providing retries and delivery within 10-15 minutes. ### Source excerpt Introduction HackerEarth handles millions of requests every day. To understand the user access patterns, get the usage of any particular feature or a page, figure out the daily active users or users who have been active for the past 6 months, etc in near real time, it is important to stream that data from across different services and ingest it to the analytics pipeline reliably. Problem Our old request logging architecture was complex and has many moving components. There were a lot of operational overheads involved in maintaining and scaling each of those components independently to ensure that all the self-hosted components were up and running all the time. Architecture Solution Last year, we revamped the way we log our web requests. It was done mainly to increase the reliability in logging the HTTP/HTTPS request data from across web services and also to reduce the operational overheads and the infrastructure cost associated with it. The new flow is making use of Kinesis Firehose data streams to deliver the request data from our web servers to Redshift (the database that we use to log and query request data) reliably with much lower cost. Amazon Kinesis Firehose is a fully managed service that automatically scales to match the throughput of our incoming request log data and requires no ongoing administration. It also allows us to compress and encrypt the data before loading it, minimizing the amount of storage used at the destination with increased security and we only need to pay for the amount of data we transmit through the service. The new flow is a fully managed solution with almost no operational overhead. We tried to keep the flow simple and straightforward with less number of moving components. The request data is now guaranteed to appear in the Redshift table within 10-15 minutes from the point we received a request at our web server's end. There are retries configured between consecutive infra components in the new flow to make sure there are no message ## Building a reliable asynchronous job pipeline DevFeed: [Building a reliable asynchronous job pipeline]() Original publisher: [Read original article]() Published: 2022-06-17T00:00:00Z Content type: article Language: en Sources: [HackerEarth]() Topics: [Amazon Simple Queue Service (SQS)](), [Library](), [Scalability](), [web applications](), [Architecture & Design](), [Amazon Web Services](), [Amazon S3](), [Serverless](), [Provisioning](), [Security](), [Kafka](), [Self-hosted]() Tags: [applications](), [architecture](), [aws](), [infrastructure](), [jobs](), [kafka](), [libraries](), [open-source](), [pipeline](), [provisioning](), [s3](), [scalability](), [security](), [serverless]() ### AI overview The article presents he-messenger, a custom serverless library for reliable asynchronous background-job processing. Built on AWS SQS, SNS, and S3, it replaces a self-hosted RabbitMQ and Kafka solution to reduce operational overhead, improve security, and support scalable service-to-service communication. ### Source excerpt Asynchronous background jobs can dramatically improve the performance and scalability of web applications by offloading resource-intensive and time consuming processing from the request-response cycle of an application. Last year, in an effort to make our asynchronous flows more reliable, secure, and scalable, we decided to move away from our self-hosted solution that was based on RabbitMQ and Kafka, to a fully-managed one. This was done mainly to reduce the operational overheads in managing and scaling the underlying infrastructure and also to improve our overall security posture. To support this flow, we created a new library called he-messenger that implements an end-to-end queuing solution for allowing different services or different components of the same service to communicate with each other asynchronously. This library is built on top of the SQS, SNS, and S3 - managed services provided by AWS. It is a fully serverless solution that ingests events from different services, buffers them, and then delivers those events to the subscribed services in a reliable way. It simplifies the otherwise laborious process of provisioning and scaling self-hosted infrastructure. Since its introduction, this library has become one of the critical pieces of our architecture powering a lot of different use-cases with a very high number of transactions There are many open-source libraries available in the market that use AWS managed services to support asynchronous background jobs. However, none of them offered an end-to-end solution and the kind of guarantees we needed. Therefore, we decided to implement our own custom library to support this flow. The purpose of this blog is to give you an overview of the internal and the code level details of the he-messenger library along with a sample reference architecture, supported flows, features,and benefits of this solution. Terminology Channel: An abstract communication layer responsible for passing messages between producers and consu ## How to set a React Component or dom element as a background image DevFeed: [How to set a React Component or dom element as a background image]() Original publisher: [Read original article]() Published: 2021-08-12T00:00:00Z Content type: tutorial Language: en Sources: [HackerEarth]() Topics: [React](), [Document Object Model (DOM)](), [Code](), [Front end]() Tags: [code](), [container](), [frontend](), [how-to](), [react](), [react-component](), [z-index]() ### AI overview A step-by-step tutorial for displaying a React Component as a background-like layer behind a textarea. It uses a relatively positioned wrapper, an absolutely positioned component, a transparent textarea background, a negative z-index, and disabled pointer events on the component. ### Source excerpt During my internship at HackerEarth, I faced an interesting problem. This blog is about that and how I solved it. Problem: To set a background image to the textarea element. My initial impression on seeing the design was that it would be easy. I thought it's a image but after exploring the code I came to know that it's not an image that we have to show as background instead, it's a React Component. So now what? To solve this problem we need to think from scratch. Solution: First, we will think about how to do it and then will implement it step by step. 1. Think From Scratch In this example, as you can see the content in the body element overlap the background-image. To solve this problem we just need to overlap the React Component with the textarea. 2. Implementation Create a textarea and React Component which we are going to use. The text area is to the left and to the right is the React Component which we are going to set as a background image in the textarea element. Steps: 1.Create a div which wrap textarea and React Component. Set the div position: relative and React Component position: absolute, top: 0 & left: 0.
// position: relative