# Engineering a Scalable Topic Pipeline: A BERTopic and GenAI Case Study

DevFeed: [Engineering a Scalable Topic Pipeline: A BERTopic and GenAI Case Study](<https://devfeed.tech/articles/engineering-a-scalable-topic-pipeline-a-bertopic-and-genai-case-study-35066.md>)

Original publisher: [Read original article](<https://medium.com/gumgum-tech/engineering-a-scalable-topic-pipeline-a-bertopic-and-genai-case-study-2d9b1161dfc5?source=rss----d4c1dee0f87b---4>)

Author: Saniya Naphade

Published: 2025-08-28T06:25:05Z

Content type: tutorial

Language: en

Sources: [GumGum](<https://devfeed.tech/sources/gumgum.md>)

Topics: [genai](<https://devfeed.tech/topics/genai.md>), [Artificial Intelligence](<https://devfeed.tech/topics/ai.md>), [Algorithm](<https://devfeed.tech/topics/algorithm.md>), [data](<https://devfeed.tech/topics/data.md>), [GPU](<https://devfeed.tech/topics/gpu.md>), [datasets](<https://devfeed.tech/topics/datasets.md>)

Tags: [ai](<https://devfeed.tech/tags/ai.md>), [algorithm](<https://devfeed.tech/tags/algorithm.md>), [case-study](<https://devfeed.tech/tags/case-study.md>), [clustering](<https://devfeed.tech/tags/clustering.md>), [data](<https://devfeed.tech/tags/data.md>), [datasets](<https://devfeed.tech/tags/datasets.md>), [engineering](<https://devfeed.tech/tags/engineering.md>), [genai](<https://devfeed.tech/tags/genai.md>), [gpu](<https://devfeed.tech/tags/gpu.md>), [llm-applications](<https://devfeed.tech/tags/llm-applications.md>), [topic-modeling](<https://devfeed.tech/tags/topic-modeling.md>), [workflow](<https://devfeed.tech/tags/workflow.md>)

## AI overview

GumGum describes a scalable hybrid topic-modeling pipeline that combines deterministic BERTopic clustering with generative AI post-processing. The workflow processes impression data weekly, uses cuML for GPU acceleration, and addresses BERTopic's high outlier rate to recover more topic-related insights.

## Source excerpt

Introduction: In a world where trends change in an instant, relying on static taxonomies to target ad campaigns is no longer effective. At GumGum, we face a constant torrent of digital content, ingesting millions of web pages every day. This massive flow of information must be turned into actionable insights, but because trends are dynamic, traditional methods like hard-coded taxonomies and slow hand-labeling are simply not viable at this scale. While large language models (LLMs) can offer some insight, using them at this volume is often cost-prohibitive and can lead to inconsistent results. To solve this, we've developed a lightweight and repeatable hybrid pipeline. This approach combines a deterministic clustering method -- specifically BERTopic¹ -- with a strategic post-processing layer using generative AI. The result is a more efficient and powerful topic modeling workflow that gives us the stability needed for large datasets and provides valuable, human-centric insights. Let's now walk through the key stages of this pipeline, beginning with data preparation. Data preparation The pipeline's first step is a critical data preparation phase built to handle immense scale. We process a comprehensive collection of impression data on a weekly basis to ensure we can capture not only major trends like the Summer Box Office but also the smaller, more nuanced shifts within that topic. For instance, in a given week, the pipeline might detect a specific surge of interest around a new movie's opening weekend, which would be lost in a less frequent processing schedule. This raw data is then thoroughly cleaned and prepared for the next stage, laying the groundwork for accurate and timely topic modeling. Dynamic Topic Modeling With this clean, prepared data, the real work of topic modeling begins. The pipeline then leverages BERTopic¹, a topic modeling algorithm, in conjunction with cuML³ for GPU acceleration, that is both powerful and efficient, and serves as our deterministic co