# Forget Table: Tracking Changing Categorical Distributions in Data Streams

DevFeed: [Forget Table: Tracking Changing Categorical Distributions in Data Streams](<https://devfeed.tech/articles/forget-table-19699.md>)

Original publisher: [Read original article](<https://word.bitly.com/post/41284219720>)

Author: Wordbitly

Published: 2013-01-23T16:46:00Z

Content type: article

Language: en

Sources: [Bitly](<https://devfeed.tech/sources/bitly.md>)

Topics: [Statistics](<https://devfeed.tech/topics/statistics.md>), [data](<https://devfeed.tech/topics/data.md>), [Streams](<https://devfeed.tech/topics/streams.md>), [datasets](<https://devfeed.tech/topics/datasets.md>)

Tags: [analytics](<https://devfeed.tech/tags/analytics.md>), [data](<https://devfeed.tech/tags/data.md>), [dataset](<https://devfeed.tech/tags/dataset.md>), [statistics](<https://devfeed.tech/tags/statistics.md>), [streams](<https://devfeed.tech/tags/streams.md>)

## AI overview

Forget Table addresses the challenge of tracking categorical distributions in continuously changing data streams. It uses recent data to better represent shifting patterns and support anomaly detection, such as identifying unexpectedly high traffic from a location.

## Source excerpt

Forget Table is a solution to the problem of storing the recent dynamics of categorical distributions that change over time (ie: non-stationary distributions). What does this mean? Why is this useful? What makes this the most important thing since sliced bread?! Read on! Storing distributions is crucial for doing any sort of statistics on a dataset. Normally, this problem is as easy as keeping counters of how many times we've seen certain quantities, but when dealing with data streams that constantly deliver new data, these simple counters no longer do the job. They fail because data we saw weeks ago has the same weight as data we have just seen, even though the fundamental distribution the data is describing may be changing. In addition it provides an engineering challenge since the counters would strictly grow and very soon use all the resources available to it. This is why we created Forget Table. Background A categorical distribution describes the probability of seeing an event occur out of a set of possible events. So an example of this at bitly is that every click coming through our system comes from one of a fixed number of country codes (there are about 260). We would like to maintain a categorical distribution, that assigns a probability to each country, that describes how likely any given click comes from each country. With bitly's data, this is going to give a high weight to the US and lower weights to Japan and Brazil, for example. It's very useful for bitly to store distributions like this, as it gives us a good idea as to what's considered 'normal'. Most of the time we produce analytics that show, for example, how many clicks came from a given country, or referrer. While this gives our clients a sense of where there traffic is coming from, it doesn't directly express how surprising their traffic is. This can be remedied by maintaining a distribution over countries for all of bitly, so that we can identify anomalous traffic, ie: when a particular link i