# t-digest

Published articles for t-digest.

This is one page of public article previews, not the complete archive. Follow Next page to continue. Summaries are not the original full articles.

## Computing accurate skill percentile with DDSketch

DevFeed: [Computing accurate skill percentile with DDSketch](<https://devfeed.tech/articles/computing-accurate-skill-percentile-with-ddsketch-20019.md>)

Original publisher: [Read original article](<http://engineering.hackerearth.com/2023/09/17/building-a-relaible-global-benchmarking-platform/>)

Published: 2023-09-17T00:00:00Z

Content type: article

Language: en

Sources: [HackerEarth](<https://devfeed.tech/sources/hackerearth.md>)

Topics: [Algorithms](<https://devfeed.tech/topics/algorithms.md>), [Analytics Pipeline](<https://devfeed.tech/topics/analytics-pipeline.md>), [computational complexity](<https://devfeed.tech/topics/computational-complexity.md>), [Benchmark](<https://devfeed.tech/topics/benchmark.md>), [streaming data analytics](<https://devfeed.tech/topics/streaming-data-analytics.md>), [Data serialization](<https://devfeed.tech/topics/data-serialization.md>)

Tags: [algorithm](<https://devfeed.tech/tags/algorithm.md>), [algorithms](<https://devfeed.tech/tags/algorithms.md>), [asynchronous](<https://devfeed.tech/tags/asynchronous.md>), [background](<https://devfeed.tech/tags/background.md>), [batch](<https://devfeed.tech/tags/batch.md>), [benchmark](<https://devfeed.tech/tags/benchmark.md>), [benchmarking](<https://devfeed.tech/tags/benchmarking.md>), [data-ingestion](<https://devfeed.tech/tags/data-ingestion.md>), [data-pipeline](<https://devfeed.tech/tags/data-pipeline.md>), [data-structures](<https://devfeed.tech/tags/data-structures.md>), [ddsketch](<https://devfeed.tech/tags/ddsketch.md>), [probabilistic](<https://devfeed.tech/tags/probabilistic.md>), [quantile](<https://devfeed.tech/tags/quantile.md>), [real-time](<https://devfeed.tech/tags/real-time.md>), [sketch](<https://devfeed.tech/tags/sketch.md>), [skills](<https://devfeed.tech/tags/skills.md>), [t-digest](<https://devfeed.tech/tags/t-digest.md>), [time](<https://devfeed.tech/tags/time.md>)

### AI overview

HackerEarth moved candidate skill percentile benchmarking to an asynchronous pipeline using DDSketch to summarize large score sets and update thresholds near real time. Its POC found T-Digest more accurate (deviation under 0.2% versus DDSketch under about 0.6%), but much slower and larger: at 100,000 samples, sketch creation took 298.89 ms for DDSketch versus 7.243 seconds for T-Digest, and serialized sizes were 6,224 versus 22,049 bytes. Given the stated tolerance for deviation near 1% and the importance of speed and memory, HackerEarth chose DDSketch. The pipeline batches skill scores, builds and merges sketches, calculates thresholds, and stores results. The reported benchmarks used random POC samples; the article says the system supports asynchronous benchmarking for 1 million or more candidates.

### Source excerpt

Introduction HackerEarth has lots of candidates getting evaluated on a daily basis. We have a feature that benchmarks candidates across the platform. Benchmarking is the process of creating the profile of the ideal candidate for a position, and then measuring all candidates against that profile. To benchmark candidate skills against our millions of candidates, we decided to move away from our regular cron solution to build a more reliable and accurate data pipeline. To support this, we created a new data ingestion flow and data read flow. We moved away from our deterministic algorithms to probabilistic algorithms with DDSketch. Problem Our old benchmarking solution was trying to compute the global benchmarking of a candidate on the fly by calculating the solve percentage of the individual skills and returning the geometric mean of all the skill benchmarks. We handle huge volumes of data every day. Analyzing this data itself--for example, calculating a quantile was optimal in terms of resources. Solution We came up with a solution that computes an approximate quantile from a compressed representation of that data. We first need to appropriately summarize that data without incurring an excessive loss of fidelity. We do this by creating a sketch. Sketch algorithms generate sketches: smaller, more manageable data structures, from which we can calculate some properties of the original data. We considered various algorithms to accurately compute percentiles on noisy, large-scale, real-time data that we were receiving from candidates' skill scores. We considered using Tdigest and DDSketch. For our use case, DDSketch served the purpose.We did a POC and compared the accuracy of both the algorithms as shown below to come up with a finalized algorithm. POC results and observations We compared the actual percentile ranges in comparison to the two probabilistic approaches we mentioned(DDSketch and T-DIgest), and these were the results. Note: we have run these tests on the random