# Computing accurate skill percentile with DDSketch

DevFeed: [Computing accurate skill percentile with DDSketch](<https://devfeed.tech/articles/computing-accurate-skill-percentile-with-ddsketch-20019.md>)

Original publisher: [Read original article](<http://engineering.hackerearth.com/2023/09/17/building-a-relaible-global-benchmarking-platform/>)

Published: 2023-09-17T00:00:00Z

Content type: article

Language: en

Sources: [HackerEarth](<https://devfeed.tech/sources/hackerearth.md>)

Topics: [Algorithms, Complexity](<https://devfeed.tech/topics/algorithms-complexity.md>), [benchmarking](<https://devfeed.tech/topics/benchmarking.md>), [data](<https://devfeed.tech/topics/data.md>), [Data structures](<https://devfeed.tech/topics/data-structures.md>), [real-time](<https://devfeed.tech/topics/real-time.md>)

Tags: [algorithms](<https://devfeed.tech/tags/algorithms.md>), [analysis](<https://devfeed.tech/tags/analysis.md>), [benchmark](<https://devfeed.tech/tags/benchmark.md>), [benchmarking](<https://devfeed.tech/tags/benchmarking.md>), [benchmarks](<https://devfeed.tech/tags/benchmarks.md>), [complexity](<https://devfeed.tech/tags/complexity.md>), [data](<https://devfeed.tech/tags/data.md>), [flow](<https://devfeed.tech/tags/flow.md>), [pipeline](<https://devfeed.tech/tags/pipeline.md>), [real-time](<https://devfeed.tech/tags/real-time.md>), [space](<https://devfeed.tech/tags/space.md>)

## AI overview

This article explains how HackerEarth built a global benchmarking platform for candidate skill percentiles. It replaced a cron-based, deterministic approach with data ingestion and read flows using probabilistic algorithms, especially DDSketch, to estimate quantiles from compressed data. The article compares DDSketch with T-Digest using accuracy, time, and space complexity analyses.

## Source excerpt

Introduction HackerEarth has lots of candidates getting evaluated on a daily basis. We have a feature that benchmarks candidates across the platform. Benchmarking is the process of creating the profile of the ideal candidate for a position, and then measuring all candidates against that profile. To benchmark candidate skills against our millions of candidates, we decided to move away from our regular cron solution to build a more reliable and accurate data pipeline. To support this, we created a new data ingestion flow and data read flow. We moved away from our deterministic algorithms to probabilistic algorithms with DDSketch. Problem Our old benchmarking solution was trying to compute the global benchmarking of a candidate on the fly by calculating the solve percentage of the individual skills and returning the geometric mean of all the skill benchmarks. We handle huge volumes of data every day. Analyzing this data itself--for example, calculating a quantile was optimal in terms of resources. Solution We came up with a solution that computes an approximate quantile from a compressed representation of that data. We first need to appropriately summarize that data without incurring an excessive loss of fidelity. We do this by creating a sketch. Sketch algorithms generate sketches: smaller, more manageable data structures, from which we can calculate some properties of the original data. We considered various algorithms to accurately compute percentiles on noisy, large-scale, real-time data that we were receiving from candidates' skill scores. We considered using Tdigest and DDSketch. For our use case, DDSketch served the purpose.We did a POC and compared the accuracy of both the algorithms as shown below to come up with a finalized algorithm. POC results and observations We compared the actual percentile ranges in comparison to the two probabilistic approaches we mentioned(DDSketch and T-DIgest), and these were the results. Note: we have run these tests on the random