# cardinality Published articles for cardinality. This is one page of public article previews, not the complete archive. Follow Next page to continue. Summaries are not the original full articles. ## Why your index is not being used, and why the planner is usually right DevFeed: [Why your index is not being used, and why the planner is usually right]() Original publisher: [Read original article]() Author: hello@ankit-rana.com Published: 2026-08-16T00:00:00Z Content type: tutorial Language: en Sources: [Ankit Rana | Mechanical Sympathy]() Topics: [Query (disambiguation)](), [MySQL](), [bug]() Tags: [bug](), [cardinality](), [database-performance](), [explain](), [function](), [indexes](), [indexing](), [mysql](), [postgresql](), [predicate](), [query](), [query-planner]() ### AI overview This tutorial explains why database indexes may not be used even when they exist. It focuses on inaccurate cardinality estimates caused by stale statistics, predicates that prevent index matching, implicit casts, and cases where sequential scans are the cheaper choice. ### Source excerpt An unused index is almost never a planner bug. It is usually a predicate the planner cannot match to the index, such as a function or an implicit cast applied to the column, or a cardinality estimate that is wrong because statistics are stale. When the estimate is right and the planner still refuses, it is often correct: past a few percent of the table, random access through an index costs more than reading the table sequentially. The diagnostic that matters is the gap between estimated and actual rows in EXPLAIN ANALYZE. ## Metric cardinality limits in OpenTelemetry: a practical guide DevFeed: [Metric cardinality limits in OpenTelemetry: a practical guide]() Original publisher: [Read original article]() Author: OpenTelemetry Authors; Docs CC BY Published: 2026-08-06T07:43:47Z Content type: article Language: en Sources: [Blog on OpenTelemetry]() Topics: [OpenTelemetry](), [SDK](), [telemetry](), [dashboards]() Tags: [cardinality](), [dashboards](), [filter](), [guide](), [memory](), [metric](), [metrics](), [opentelemetry](), [practical](), [production]() ### AI overview This practical guide explains how OpenTelemetry metric cardinality limits protect process memory from unbounded growth. When a metric stream overflows, total values remain correct, but attribute-based filtering and grouping can undercount, affecting dashboards, SLOs, and alerts. ### Source excerpt OpenTelemetry metrics are designed to be safe to use in production. One part of that safety is the cardinality limit in the metrics SDK. The limit protects your process from unbounded memory growth when a metric receives too many unique attribute combinations. That protection is useful, but it has a consequence many users do not expect: when a metric stream overflows, the total value remains correct, while queries that filter or group by attributes can undercount. This can affect dashboards, service-level objectives (SLOs), and alerts that looked correct before overflow started. ## Leveraging PyFixest for High-Cardinality Marketplace Modeling at Instacart DevFeed: [Leveraging PyFixest for High-Cardinality Marketplace Modeling at Instacart]() Original publisher: [Read original article]() Author: Benjamin Knight Published: 2026-06-29T16:06:24Z Content type: article Language: en Sources: [Instacart]() Topics: [benchmarking](), [math](), [Algorithms, Complexity](), [Software](), [Programming](), [Routing (disambiguation)]() Tags: [benchmarking](), [bias](), [cardinality](), [data-science](), [delivery](), [efficiency](), [estimator](), [fixed-effects-model](), [instacart](), [linear-regression](), [measurement](), [memory](), [precision](), [pyfixest](), [regression](), [routing](), [speed](), [statistics](), [techniques]() ### AI overview This Instacart article explains why ordinary least squares regression becomes computationally impractical for marketplace experiments with high-cardinality categories. It presents the mathematical basis for using Fixest and Pyfixest, discusses switchback experiment designs for addressing treatment spillover, and describes benchmarks comparing processing speed, memory efficiency, and estimator precision. ### Source excerpt Benjamin S. Knight Scaling Marketplace experiments requires specialized statistical techniques. We examine why standard ordinary least squares regression (OLS) becomes computationally intractable when controlling for high-cardinality categories. We then dive into the underlying math and demonstrate how modern packages -- specifically Fixest and Pyfixest -- bypass these limitations. We conclude by benchmarking these methods to show their real-world impact on processing speed, memory efficiency, and estimator precision. At Instacart we strive to give our customers access to all the fresh foods and ingredients that they would normally get from a trip to the grocery store, but without the hassle of driving, finding parking, waiting in line, etc. Instacart's Marketplace team is responsible for surfacing customers' orders to shoppers, aligning Instacart's delivery windows with shoppers' projected availabilities as efficiently as possible. This entails a careful balancing act. If we offer delivery windows that are sooner / more popular, then we risk overextending shoppers' ability to fulfill those orders on time. If we are too conservative in our delivery option offerings, then we risk losing potential orders. Accurately measuring the impact of changes in our batching and routing algorithms requires thoughtful experiment design and software. Better predictions of future demand / time-to-fulfill allow Instacart to offer more convenient delivery windows.Experimentation on Marketplace One of our primary concerns in Marketplace is treatment spillage. For example, if we adjust our batching algorithm and increase the rate at which multiple orders are combined into batches in Brooklyn and Queens, then we face a real risk of also influencing the rate of batch creation / completion in Staten Island, the Bronx, and Manhattan. In this case the treatment impacts the control group -- a classic source of measurement bias as a consequence of violating the Stable Unit Treatment Value Assumpt ## Deep Dive: Telemetry cardinality in the Apollo GraphOS Router DevFeed: [Deep Dive: Telemetry cardinality in the Apollo GraphOS Router]() Original publisher: [Read original article]() Author: Nick Marsh Published: 2026-02-10T09:00:00Z Content type: tutorial Language: en Sources: [Apollo Blog]() Topics: [GraphOS](), [OpenTelemetry](), [telemetry](), [Instrumentation](), [SDK](), [Rust]() Tags: [announcement](), [cardinality](), [config](), [graphos](), [instrumentation](), [metric](), [metrics](), [observability](), [opentelemetry](), [rust](), [scale](), [sdk](), [telemetry](), [traces]() ### AI overview This technical article explains how the Apollo GraphOS Router uses OpenTelemetry for metrics and traces, including instrumentation, metric providers, readers, and exporters. It also describes the OpenTelemetry SDK's 2,000-cardinality limit and how reaching that limit affects metric attributes. ### Source excerpt Learn how telemetry works in the Apollo GraphOS Router, from OpenTelemetry instrumentation to metric exporters. Understand cardinality limits, diagnose overflow issues, and tune your router config for production-scale observability. ## ClickStack: A High-Performance OSS Observability Stack on ClickHouse DevFeed: [ClickStack: A High-Performance OSS Observability Stack on ClickHouse]() Original publisher: [Read original article]() Author: Mike Shi Published: 2025-05-29T00:00:00Z Content type: news Language: en Sources: [ClickHouse Blog]() Topics: [clickhouse](), [observability](), [Open Source](), [OpenTelemetry](), [telemetry](), [session replay](), [Traces]() Tags: [announce](), [cardinality](), [clickhouse](), [compression](), [logs](), [metrics](), [observability](), [opentelemetry](), [oss](), [performance](), [saas](), [session-replay](), [traces]() ### AI overview ClickHouse announces ClickStack, an open-source observability stack powered by ClickHouse and HyperDX. It provides logs, metrics, traces, and session replay through an integrated experience with OpenTelemetry collection, a wide-events UI, natural-language querying, alerting, and high-performance storage and analytics. ### Source excerpt We're delighted to announce ClickStack: the open-source observability stack built on ClickHouse - logs, metrics, traces, and session replay in one blazing-fast, developer-friendly platform. ## Evaluating Equality Predicates with RangeBitmap DevFeed: [Evaluating Equality Predicates with RangeBitmap]() Original publisher: [Read original article]() Author: Richard Startin's Blog Published: 2022-12-18T00:00:00Z Content type: article Language: en Sources: [Richard Startin's Blog]() Topics: [Data structures](), [Library](), [benchmarking](), [data observability]() Tags: [benchmarking](), [cardinality](), [comparison](), [data-engineering](), [data-structure](), [java](), [library](), [memory](), [pinot](), [range](), [roaring](), [speed]() ### AI overview This article evaluates equality and inequality queries using RangeBitmap in the RoaringBitmap library. It explains how the enhancement can support equality filtering as a compact inverted-index alternative, including as a fallback for Apache Pinot range indexes, and reports faster selection than a stream-based scan in the described example while using less space than some inverted indexes. ### Source excerpt I have just implemented support for (in)equality queries against a RangeBitmap, a succinct data structure in the RoaringBitmap library which supports range queries. RangeBitmap was designed to support range queries in Apache Pinot (more details here) but this enhancement would allow a range index to be used as a fallback for (in)equality queries in case nothing better is available. Supporting (in)equality queries allows a RangeBitmap to be used as a kind of compact inverted index, trading space for time, capable of supporting high cardinality gracefully. Since RangeBitmap supports memory mapping from files, I think that it could be used for data engineering beyond Apache Pinot. ## Modelling UI State on Android DevFeed: [Modelling UI State on Android]() Original publisher: [Read original article]() Author: Stojan Anastasov Published: 2021-01-29T00:00:00Z Content type: tutorial Language: en Sources: [Stojan Anastasov's blog]() Topics: [Android](), [android-development](), [Kotlin](), [Data structures](), [Development]() Tags: [android](), [android-development](), [cardinality](), [data-class](), [data-structure](), [fp](), [functional](), [kotlin](), [livedata](), [rxjava](), [sealed-class](), [stateflow](), [types](), [ui](), [viewmodel]() ### AI overview This article explains how to model Android UI state in Kotlin using data classes and sealed classes. It connects types with sets and cardinality, then describes product types and sum types as tools for representing valid application states. ### Source excerpt The recommended approach from Google for Android development is holding the UI state in a ViewModel and having the View observe it. To achieve that one can use LiveData, StateFlow, RxJava or a similar tool. But how to model the UI state? Use a data class or a sealed class? Use one observable property or many? I will describe the tradeoffs between the approaches and present a tool to help you decide which one to use. This article is heavily inspired by Types as Sets from the Elm guide, a large part is a translation from Elm to Kotlin. Photo by Marc-Olivier Jodoin on Unsplash Types as sets By Making Data Structure we can make sure the possible values in code exactly match the valid values in real life. Doing that helps to avoid a whole class of bugs related to invalid data. To achieve that, first we need to understand the relationship between Types and Sets. We can think of Types as sets of values, they contain unique elements and there is no ordering between them. For example: Nothing - the empty set, it contains no elements Unit - the singleton set, it contains one element - Unit Boolean - contains the elements true and false Int - contains the elements: ... -2, -1, 0, 1, 2 ... Float - contains the elements: 0.1, 0.01, 1.0 .... String - contains the elements: "", "a", "b", "Kotlin", "Android", "Hello world!"... So when you write: val x: Boolean it means x belongs to the set of Boolean values and can be either true or false. Cardinality In Mathematics, Cardinality is the measure of "number of elements" of a Set. For example the set of Boolean contains the elements [true, false] so it has a cardinality = 2. Let's take a look at the cardinality of the sets mentioned above: Nothing - 0 Unit - 1 Boolean - 2 Short - 65535 Int - ∞ Float - ∞ String- ∞ Note: The cardinality of Int and Float is not exactly infinity, it's 2^32 however that is a huge number. When building apps, we use built-in types and create custom types using constructs like data classes and sealed classes. Product ## Better URL Search with Elasticsearch DevFeed: [Better URL Search with Elasticsearch]() Original publisher: [Read original article]() Author: Jorge Luis Betancourt Follow Published: 2020-02-11T00:00:00Z Content type: tutorial Language: en Sources: [Trivago]() Topics: [elasticsearch](), [Logging](), [log management](), [tokenization](), [Query (disambiguation)]() Tags: [ascii](), [cardinality](), [elasticsearch](), [kibana](), [logging](), [logs](), [logstash](), [monitoring](), [pipeline](), [protocol](), [tokenization]() ### AI overview This article explains how trivago searches URL components stored in Elasticsearch within an ELK-based logging pipeline. It examines how the Standard Analyzer tokenizes URL-like query strings and discusses preprocessing and flattened-field approaches for matching query-parameter key/value pairs, including their tradeoffs. ### Source excerpt At trivago, we generate a huge amount of logs and we have our own custom setup for shipping logs using mostly ## Methods of Proof -- Diagonalization DevFeed: [Methods of Proof -- Diagonalization]() Original publisher: [Read original article]() Published: 2015-06-08T09:00:00Z Content type: tutorial Language: en Sources: [Jeremy Kun]() Topics: [Mathematics](), [Math and Logic](), [Collections]() Tags: [bijections](), [cardinality](), [diagonalization](), [halting-problem](), [mathematics](), [methods-of-proof](), [turing-machines](), [uncountability]() ### AI overview This tutorial introduces diagonalization as an advanced method of mathematical proof. It explains the table-and-diagonal construction and presents the theorem that no bijection exists between the natural numbers and the real numbers. ### Source excerpt A while back we featured a post about why learning mathematics can be hard for programmers, and I claimed a major issue was not understanding the basic methods of proof (the lingua franca between intuition and rigorous mathematics). I boiled these down to the "basic four," direct implication, contrapositive, contradiction, and induction. But in mathematics there is an ever growing supply of proof methods. There are books written about the "probabilistic method," and I recently went to a lecture where the "linear algebra method" was displayed. ## Abbreviated keys for numeric to accelerate numeric sorts DevFeed: [Abbreviated keys for numeric to accelerate numeric sorts]() Original publisher: [Read original article]() Author: Peter Geoghegan (noreply@blogger.com) Published: 2015-04-04T16:19:00Z Content type: article Language: en Sources: [Peter Geoghegan's blog]() Topics: [PostgreSQL](), [Sorting](), [Development]() Tags: [abbreviation](), [benchmarks](), [c](), [cardinality](), [commit](), [count](), [fast](), [improvements](), [indexes](), [internals](), [maintenance](), [postgresql](), [sorting](), [table]() ### AI overview The article discusses PostgreSQL numeric abbreviated keys, a committed patch intended to accelerate numeric sorting. It reports 6x-7x improvements for representative in-memory queries and a 12x improvement in a PostgreSQL 9.5 text-column CREATE INDEX benchmark compared with PostgreSQL 9.4. ### Source excerpt Andrew Gierth's numeric abbreviated keys patch was committed recently. This commit added abbreviation/sortsupport for the numeric type (the PostgreSQL type which allows practically arbitrary precision, typically recommended for representing monetary values). The encoding scheme that Andrew came up with is rather clever - it has an excellent tendency to concentrate entropy from the original values into the generated abbreviated keys in real world cases. As far as accelerating sorts goes, numeric abbreviation is at least as effective as the original text abbreviation scheme. I easily saw improvements of 6x-7x with representative queries that did not spill to disk (i.e. that used quicksort). In essence, the patch makes sorting numeric values almost as cheap as sorting simple integers, since that is often all that is actually required during sorting proper (the abbreviated keys compare as integers, except that the comparison is inverted to comport with how abbreviation builds abbreviated values from numerics as tuples are copied into local memory ahead of sorting - see the patch for exact details). Separately, over lunch at pgConf.US in New York, Corey Huinker complained about a slow, routine data warehousing CREATE INDEX operation that took far too long. The indexes in question were built on a single text column. I suggested that Corey check out how PostgreSQL 9.5 performs, where this operation is accelerated by text abbreviation, often very effectively. Corey chose an organic set of data that could be taken as a reasonable proxy for how PostgreSQL behaves when he performs these routine index builds. In all cases maintenance_work_mem was set to 64MB, meaning that an external tapesort is always required - those details were consistent. This was a table with 18 million rows. Apparently, on PostgreSQL 9.4, without abbreviation, the CREATE INDEX took 10 minutes and 19 seconds in total. On PostgreSQL 9.5, with identical settings, it took only 51.3 seconds - a 12x improvemen ## ThingMonk 2014: Toward a more intelligent IoT DevFeed: [ThingMonk 2014: Toward a more intelligent IoT]() Original publisher: [Read original article]() Published: 2014-12-05T00:00:00Z Content type: opinion Language: en Sources: [Tim Kellogg]() Topics: [Internet of things](), [data](), [MQTT](), [CoAP]() Tags: [bloom-filter](), [cardinality](), [data](), [data-structures](), [iot](), [london](), [mqtt]() ### AI overview A recap of ThingMonk 2014 argues that the Internet of Things needs shared ways to organize, interpret, and connect sensor data, beyond efficient communication protocols. The article highlights metadata, open data, and data structures such as Bloom filters and HyperLogLog. ### Source excerpt This week I was fortunate enough to attend ThingMonk in London. RedMonk were excellent hosts and managed to put together a tremendous lineup of speakers and talks that I hadn't anticipated. There were only 150 attendees, but each one of them brought something unique. Here I attempt to summarize some of the day, I know I've missed several truly great talks, but I just wanted to keep it short. ## Redis new data structure: the HyperLogLog DevFeed: [Redis new data structure: the HyperLogLog]() Original publisher: [Read original article]() Published: 2014-04-01T08:16:35Z Content type: release Language: en Sources: [Antirez]() Topics: [Redis](), [Data structures](), [Algorithms]() Tags: [algorithm](), [algorithms](), [cardinality](), [data-structure](), [memory](), [redis]() ### AI overview The article introduces HyperLogLog as a new Redis data structure for approximating the number of unique elements in a set. It explains that the algorithm uses a small, constant amount of memory and reports that the Redis implementation uses 12 kB per key with a standard error of 0.81%. ### Source excerpt Generally speaking, I love randomized algorithms, but there is one I love particularly since even after you understand how it works, it still remains magical from a programmer point of view. It accomplishes something that is almost illogical given how little it asks for in terms of time or space. This algorithm is called HyperLogLog, and today it is introduced as a new data structure for Redis. Counting unique things === Usually counting unique things, for example the number of unique IPs that connected today to your web site, or the number of unique searches that your users performed, requires to remember all the unique elements encountered so far, in order to match the next element with the set of already seen elements, and increment a counter only if the new element was never seen before. This requires an amount of memory proportional to the cardinality (number of items) in the set we are counting, which is, often absolutely prohibitive. There is a class of algorithms that use randomization in order to provide an approximation of the number of unique elements in a set using just a constant, and small, amount of memory. The best of such algorithms currently known is called HyperLogLog, and is due to Philippe Flajolet. HyperLogLog is remarkable as it provides a very good approximation of the cardinality of a set even using a very small amount of memory. In the Redis implementation it only uses 12kbytes per key to count with a standard error of 0.81%, and there is no limit to the number of items you can count, unless you approach 2^64 items (which seems quite unlikely). The algorithm is documented in the original paper [1], and its practical implementation and variants were covered in depth by a 2013 paper from Google [2]. [1] http://algo.inria.fr/flajolet/Publications/FlFuGaMe07.pdf [2] http://static.googleusercontent.com/media/research.google.com/en//pubs/archive/40671.pdf How it works? === There are plenty of wonderful resources to learn more about HyperLogLog, s ## HyperLogLog Unions DevFeed: [HyperLogLog Unions]() Original publisher: [Read original article]() Author: Dimitri Fontaine PostgreSQL Major Contributor; Author Published: 2013-02-26T11:44:00Z Content type: article Language: en Sources: [Dimitri Fontaine]() Topics: [PostgreSQL](), [Algorithm](), [Data structures](), [SQL](), [data]() Tags: [algorithm](), [article](), [cardinality](), [data-structure](), [postgresql](), [sql]() ### AI overview This article explains how PostgreSQL's HyperLogLog extension maintains a fixed-size data structure for estimating cardinality and supports union operations for counting unique values across sets, including use through SQL aggregates and window functions. ### Source excerpt In the article from yesterday we talked about PostgreSQL HyperLogLog with some details. The real magic of that extension has been skimmed over though, and needs another very small article all by itself, in case you missed it. *Which Set Operation do you want for counting unique values?* The first query here has the default level of magic in it, really. What happens is that each time we do an update of the HyperLogLog hash value, we update some data which are allowing us to compute its cardinality. ## PostgreSQL HyperLogLog DevFeed: [PostgreSQL HyperLogLog]() Original publisher: [Read original article]() Author: Dimitri Fontaine PostgreSQL Major Contributor; Author Published: 2013-02-25T09:23:00Z Content type: tutorial Language: en Sources: [Dimitri Fontaine]() Topics: [PostgreSQL](), [Algorithm](), [Statistics](), [Extension]() Tags: [algorithm](), [bug](), [cardinality](), [concurrency](), [extension](), [postgresql](), [statistics]() ### AI overview This tutorial explains how to use the postgresql-hll extension with PostgreSQL for HyperLogLog cardinality estimation. It covers installation, estimating unique values such as IP addresses, handling concurrency with queued asynchronous updates, and protecting against empty-set and NULL behavior. The article also discusses a possible aggregate bug. ### Source excerpt If you've been following along at home the newer statistics developments, you might have heard about this new State of The Art Cardinality Estimation Algorithm called HyperLogLog. This technique is now available for PostgreSQL in the extension postgresql-hll available at https://github.com/aggregateknowledge/postgresql-hll and soon to be in debian. ## Set Theory--A Primer DevFeed: [Set Theory--A Primer]() Original publisher: [Read original article]() Published: 2011-07-09T18:14:59Z Content type: tutorial Language: en Sources: [Jeremy Kun]() Topics: [Mathematics](), [Math and Logic]() Tags: [axiom-of-choice](), [bijections](), [cardinality](), [countability](), [example](), [functions](), [mathematics](), [numbers](), [power-set](), [primer](), [set](), [set-theory](), [symbols](), [theory](), [variable]() ### AI overview This primer introduces set theory by defining sets, elements, membership, cardinality, notation, and several ways to construct sets. It uses numerical examples and introduces natural numbers, integers, and rational numbers while noting that unrestricted operations can lead to paradoxes. ### Source excerpt It's often that a student's first exposure to rigorous mathematics is through set theory, as originally studied by Georg Cantor. This means we will not treat set theory axiomatically (as in ZF set theory), but rather we will take the definition of a set for granted, and allow any operation to be performed on a set. This will be clear when we present examples, and it will be clear why this is a bad idea when we present paradoxes.