# How FINN Redesigned Its Big Data Statistics System

DevFeed: [How FINN Redesigned Its Big Data Statistics System](<https://devfeed.tech/articles/finding-gold-in-big-data-31993.md>)

Original publisher: [Read original article](<https://tech.finn.no2015/04/27/finding-gold-in-big-data/>)

Author: mick

Published: 2015-04-27T09:00:00Z

Content type: tutorial

Language: en

Sources: [Finn.no](<https://devfeed.tech/sources/finn-no.md>)

Topics: [big-data](<https://devfeed.tech/topics/big-data.md>), [Architecture & Design](<https://devfeed.tech/topics/architecture-design.md>), [Scalability](<https://devfeed.tech/topics/scalability.md>), [Apache Cassandra](<https://devfeed.tech/topics/cassandra.md>), [Hadoop](<https://devfeed.tech/topics/hadoop.md>), [web applications](<https://devfeed.tech/topics/web-applications.md>), [Back end](<https://devfeed.tech/topics/backend.md>)

Tags: [architecture](<https://devfeed.tech/tags/architecture.md>), [backend](<https://devfeed.tech/tags/backend.md>), [big-data](<https://devfeed.tech/tags/big-data.md>), [cassandra](<https://devfeed.tech/tags/cassandra.md>), [hadoop](<https://devfeed.tech/tags/hadoop.md>), [scalability](<https://devfeed.tech/tags/scalability.md>), [web-applications](<https://devfeed.tech/tags/web-applications.md>)

## AI overview

This technical article explains how FINN redesigned its statistics system for asynchronous processing, fault tolerance, linear scalability, and durability. The design separates event collection from statistics display, using Scribe, Cassandra, Thrift, and Hadoop MapReduce.

## Source excerpt

In the previous article we introduced our own introduction into the world of Big Data, and explored what it meant for FINN. Here we'll go into the technical depth about the implementation of our Big Data needs. rehashing the previous article FINN is a busy site, the busiest in Norway, and we display over 80 million ad pages each day. Back when it was around 50 million views per day, the old system responsible for collecting statistics was performing up to a thousand database writes per second during peak traffic. Like a lot of web applications we had a modern scalable presentation and logic tier based upon ten tomcat servers but just one not-so-scalable monster database sitting in the data tier. The procedure responsible for writing to the statistics table in the relational database was our biggest thorn. It had indeed gotten so bad that: during peak traffic; operations had to at the first sign of trouble turn off this database procedure - that is when users were getting the most traffic to their ads we had to stop collecting their statistics. At this time we were also in the process of modularising the FINN web application. The time was right to turn our statistics system into something modern and modular. We wanted an asynchronous, fault-tolerance, linearly scaling, and durable solution. the design The new design uses the Command Query Separation pattern by using two separate modules: one for the collecting of events and one for displaying statistics. The event collecting system achieves asynchronousity, scalability, and durability by using Scribe. The backend persistence and statistics module achieves all goals by using Cassandra and Thrift. As an extension of the push-on-change model: the event collection stores denormalised data and it is later aggregated and normalised to the views the statistics module requires; we use MapReduce jobs within a Hadoop cluster. the statistics module At FINN all our modular architecture is built either upon REST or interfaces def