# Building a Data Pipeline to Track Strava's Bad Events

DevFeed: [Building a Data Pipeline to Track Strava's Bad Events](<https://devfeed.tech/articles/an-eventful-summer-at-strava-26571.md>)

Original publisher: [Read original article](<https://medium.com/strava-engineering/an-eventful-summer-at-strava-5692882e5f4f?source=rss----89d4108ce2a3---4>)

Author: Bisman Sodhi

Published: 2024-01-08T20:19:46Z

Content type: opinion

Language: en

Sources: [Strava Engineering](<https://devfeed.tech/sources/strava-engineering.md>)

Topics: [data](<https://devfeed.tech/topics/data.md>), [airflow](<https://devfeed.tech/topics/airflow.md>), [Amazon S3](<https://devfeed.tech/topics/amazon-s3.md>), [Python](<https://devfeed.tech/topics/python.md>), [Amazon Web Services](<https://devfeed.tech/topics/aws.md>), [JSON](<https://devfeed.tech/topics/json.md>), [dashboards](<https://devfeed.tech/topics/dashboards.md>), [Computer science](<https://devfeed.tech/topics/computer-science.md>)

Tags: [airflow](<https://devfeed.tech/tags/airflow.md>), [apache-airflow](<https://devfeed.tech/tags/apache-airflow.md>), [aws](<https://devfeed.tech/tags/aws.md>), [data](<https://devfeed.tech/tags/data.md>), [data-engineering](<https://devfeed.tech/tags/data-engineering.md>), [data-pipeline](<https://devfeed.tech/tags/data-pipeline.md>), [data-platforms](<https://devfeed.tech/tags/data-platforms.md>), [integrity](<https://devfeed.tech/tags/integrity.md>), [json](<https://devfeed.tech/tags/json.md>), [python](<https://devfeed.tech/tags/python.md>), [s3](<https://devfeed.tech/tags/s3.md>), [software-engineering](<https://devfeed.tech/tags/software-engineering.md>), [strava](<https://devfeed.tech/tags/strava.md>), [tableau](<https://devfeed.tech/tags/tableau.md>), [warehouse](<https://devfeed.tech/tags/warehouse.md>)

## AI overview

A software engineering intern describes building a daily Apache Airflow pipeline that extracts schema-invalid user behavior events from S3, decompresses them into JSON, and loads them into Snowflake. Staging tables protect production data from partial loads, while materialized SQL views and a Tableau dashboard improve querying and monitoring.

## Source excerpt

Hi my name is Bisman and I studied Computer Science at University of California, Santa Barbara. During summer of 2022, I had the most amazing experience working as a Software Engineer Intern on Strava's Data Platform Team. In the first fews weeks, I learned the tools my team uses and then spent the rest of the time working on my project. TRACKING BAD EVENTS For my major summer project, I created a data pipeline that pulls user behavior data out of external storage and persists it in our data warehouse. Strava uses a service called Snowplow to collect this user behavior data, like loading a club page or uploading a profile photo. Sometimes, this data fails to match the schema that we've set, and a piece of data that fails this schema validation is called a bad event. Previously, these bad events were temporarily stored in an Elastic Search. Persisting this data in Snowflake, our data warehouse, makes it accessible to a wider audience. It also makes it easier to incorporate the bad events data with other services used at Strava. To start my project, I created a directed acyclic graph in Apache Airflow, a scheduling framework, using python that extracts bad events data from the S3, AWS's storage service, buckets on a daily cadence. This data was stored as gzip files on S3 which I decompressed and stored the data as JSON blobs. As I was working with billions of rows of data, it was important to maintain data integrity and take measures in case data failed to load from S3. Therefore, I loaded data into a staging table in Snowflake. The staging table ensured that if loading from S3 failed, the production table would remain untouched. This data was then loaded into the production table free of any partial data. After all the data was loaded into the production table, I created six view tables because there were six different types of bad events stored in the production table. I collaborated with our stakeholders -- data analysts -- throughout this process to craft tables bas