# Udemy Engineering Learn about cool projects, product initiatives, and company culture from the data science and engineering teams at Udemy. Find these projects fun? We're hiring! https://about.udemy.com/careers/ - Medium This is one page of public article previews, not the complete archive. Follow Next page to continue. Summaries are not the original full articles. ## From Drift to Direction: The Architecture Behind Digression Control (Role Play) DevFeed: [From Drift to Direction: The Architecture Behind Digression Control (Role Play)]() Original publisher: [Read original article]() Author: Raka Dalal Published: 2026-04-09T12:25:37Z Content type: article Language: en Sources: [Udemy Engineering]() Topics: [Large Language Model](), [Artificial Intelligence](), [systems](), [Learning]() Tags: [ai](), [architecture](), [blocking](), [conversational-ai](), [data-science](), [large-language-models](), [learning](), [llm](), [llm-evaluation](), [prompt-engineering](), [systems]() ### AI overview Udemy's Role Play experience uses open-ended LLM-mediated conversations for practice with defined learning goals. The article examines digression as a systems challenge and introduces a Response Steering Layer intended to preserve conversational naturalness while improving goal adherence. ### Source excerpt Role Play @UdemyIntroduction AI-powered role play systems aim to recreate one of the most effective learning modalities: practicing real-world conversations in a safe, repeatable environment. In Udemy's Role Play experience, learners engage in open-ended, free-text dialogues with an AI character that embodies a specific persona -- such as an interviewer, a stakeholder, or a customer -- while working toward clearly defined learning goals. This contrasts with many traditional digital learning tools, which rely on static content, multiple-choice interactions, or scripted simulations that limit conversational variability. This design prioritizes realism and skill transfer, allowing learners to respond naturally rather than selecting from predefined options. Open-ended conversations mediated by LLM-based systems, however, introduce a core systems challenge: digression. In real conversations, people may drift off-topic, but human tutors actively manage and redirect such drift. In a learning-oriented role play mediated by an AI system, persistent digressions undermine the experience in more subtle but consequential ways. Learners may shift into meta-conversations, attempt to override the role being played, or explore tangents unrelated to the scenario's objectives. When this happens, learning goals become harder to assess, feedback loses grounding in the conversation, and the interaction no longer reflects the real-world situation the learner is meant to practice. A naive solution is to tightly constrain the conversation -- blocking inputs, rejecting turns, or forcing the dialogue back onto a scripted path. While this approach improves goal adherence, it comes at the cost of immersion and learner agency. Overly rigid controls make the AI feel less like a conversational partner and more like an instructional interface. On the other hand, allowing unrestricted conversational freedom preserves naturalness but leads to goal drift, inconsistent learning signals, and reduced reliab ## From Zero to Hero: Localization-Led Generative AI at Udemy DevFeed: [From Zero to Hero: Localization-Led Generative AI at Udemy]() Original publisher: [Read original article]() Author: Nishanth Gandhidoss Published: 2025-09-22T17:47:18Z Content type: article Language: en Sources: [Udemy Engineering]() Topics: [Localization (l10n)](), [Generative AI](), [Large Language Model](), [datasets]() Tags: [ai](), [features](), [generative-ai](), [generative-ai-tools](), [implementation](), [llm](), [llm-applications](), [machine-learning](), [machine-translation](), [retrieval-augmented-gen](), [techniques](), [translation]() ### AI overview Udemy describes how it localized generative AI features, including Skills Mapping and its in-course AI Assistant, to serve learners in multiple languages. The article compares translation management systems, multilingual LLMs, and hybrid strategies, emphasizing trade-offs among implementation speed, language coverage, fluency, cultural nuance, and domain precision. ### Source excerpt By Nishanth Gandhidoss, Senior Staff Data Scientist @ Udemy Why Multilingual Generative AI Matters Generative AI is rapidly changing how people learn and work, but those gains are only realized when the technology speaks the user's language. Over the past year at Udemy, we prioritized breaking the English-only barrier so our Gen AI features, most critically Skills Mapping and the in-course AI Assistant, could deliver value for learners across the globe. Our approach balanced speed, coverage, and quality, allowing us to move quickly without sacrificing fidelity in markets where nuance matters. Below we describe the localization strategy we adopted, why we made those trade-offs, and the operational practices that made the rollout repeatable, scalable, and quick. Localization Strategies When deciding how to localize a Gen AI product, it is helpful to think of options on a spectrum from low to high complexity. The simplest approach provides broad language coverage fast, while the most involved approach optimizes for fluency, cultural nuance, and domain precision. At Udemy, we researched and organized our work around three primary strategies: a translation management system (TMS) for rapid implementation with broader coverage, a multilingual LLM system (MLS) for high-fidelity experiences, and a hybrid approach to balance reach and quality. As each strategy is introduced below, we briefly explain the industry-standard localization techniques they draw on, with examples, so the differences are clear. Translation Management System (TMS) At the simplest end of the spectrum, we have the TMS approach. With a TMS, the user interacts in their own language, but the AI model itself operates in English input/output. The system first detects the user's input language, say, in Spanish, and sends the input through a machine translation (MT) service that converts it into English. The LLM then generates an English response, and that response is translated back into the user's input lang ## From siloed DataOps, MLOps, and LLMOps to a unified data-intelligence platform DevFeed: [From siloed DataOps, MLOps, and LLMOps to a unified data-intelligence platform]() Original publisher: [Read original article]() Author: Rajit Saha Published: 2025-08-04T18:03:19Z Content type: opinion Language: en Sources: [Udemy Engineering]() Topics: [DataOps](), [MLOps](), [Generative AI](), [Apache Spark](), [Amazon Bedrock](), [Amazon S3](), [Amazon Redshift](), [Amazon SageMaker](), [apache-flink]() Tags: [ai](), [aiops](), [amazon-s3](), [analytics](), [apache-flink](), [apache-spark](), [bedrock](), [dataops](), [large-language-models](), [llmops](), [mlops]() ### AI overview The article describes how DataOps, MLOps, and AI/LLM Ops commonly rely on separate systems and teams for data processing, model deployment, inference, evaluation, orchestration, governance, and monitoring. It then introduces Databricks' Data Intelligence Platform as a unified environment intended to bring these domains together. ### Source excerpt Introduction In modern data-driven businesses, the pace of innovation in analytics and artificial intelligence has outstripped the capacity of many teams. Three distinct disciplines emerged to handle this expansion: Data platform (DataOps) teams built data lakes on cloud storage such as Amazon S3, processed them with Apache Spark and Hive on EMR, ingested streaming data with Spark Structured Streaming or Apache Flink, and loaded tabular copies into MPP warehouses like Redshift for interactive SQL and BI. Cataloguing and governance were offloaded to external tools such as DataHub, and fine-grained access controls required third-party services like Privacera. This architecture worked, but it required separate workflows for batch and streaming, extra systems for lineage and governance, and a mosaic of operational teams. MLOps teams provided an additional layer. Data scientists used notebook environments (for example, Amazon SageMaker) to preprocess data, train, and evaluate models. Deploying models meant writing integration code to move features into a serving layer, to register models in disparate registries and to build custom APIs for inference. Feature stores and model registries were bought from additional vendors. Updates and monitoring were often manual processes. AI/LLM Ops teams are a new addition because generative AI requires specialized components: LLM gateways (e.g., Amazon Bedrock) to proxy access to foundation models; evaluation tooling to compare large language models; orchestration frameworks for agents; vector databases for retrieval augmented generation; and of course another layer of security, access management and cost control. These tools seldom integrate seamlessly with existing data and ML pipelines. This fragmented state makes it difficult to react quickly when product requirements change. Each new capability requires another system, another integration, and another team. Meanwhile, budgets tighten and go-to-market timelines shrink. The questio ## Evolution of the Udemy AI Assistant Intent Understanding System DevFeed: [Evolution of the Udemy AI Assistant Intent Understanding System]() Original publisher: [Read original article]() Author: Jack Kwok Published: 2025-05-28T16:35:59Z Content type: article Language: en Sources: [Udemy Engineering]() Topics: [Artificial Intelligence](), [Language models](), [context](), [API](), [Development]() Tags: [ai](), [api](), [artificial-intelligence](), [chat](), [chatbots](), [context](), [cost](), [development](), [function](), [generation](), [language](), [language-models](), [llm](), [llms](), [models](), [quality](), [responses](), [search](), [speed](), [udemy]() ### AI overview This article describes the evolution of Udemy's AI Assistant intent-understanding system. It explains how the system predicts a learner's intent and invokes chains of actions to retrieve relevant context from data sources and APIs, including lecture summaries and course search, while balancing response quality, speed, and cost. ### Source excerpt Introduction Imagine having a virtual learning companion by your side every time you tackle a new course or challenging concept, ready to clarify doubts, summarize lessons, or quiz your understanding on demand. This isn't just an idea for the future. Thanks to the Udemy AI Assistant, it's a reality for every Udemy learner today. Powered by AI, the Assistant is actively supporting learners all over the world, guiding them step by step through their skills development journeys and making learning smoother, faster, and more personalized than ever. Behind the scenes, delivering such smart, relevant help relies on one crucial ability: understanding exactly what the learner wants. From course-related clarifications to lecture summaries and beyond, the AI Assistant must accurately grasp the intent behind every question to deliver the perfect response. In this article, we'll take you behind the curtain to explore how we've evolved the Udemy AI Assistant's intent understanding system. You'll discover key lessons from our journey: the challenges we faced as we expanded features, breakthroughs that boosted accuracy, and how we balanced response quality, speed, and cost. Background With Large Language Models (LLMs), setting the relevant context in the prompt is key to eliciting high-quality, high-relevance chat responses. To that end, we enable the AI Assistant to connect to various data sources and services to obtain the most relevant context. Behind the scenes, the AI Assistant predicts the learner's intent and then invokes the corresponding chain of actions to fetch the most helpful context from data sources and APIs. For example, if the learner intends to ask for a summary of the current video lecture, a chain of actions would include invoking an API call to fetch the pre-generated lecture summary and send it to the LLM for final response generation. As another example, if the learner intends to search for a topic within the course, a function call will execute a search t ## Migrating Udemy's Homepage to Micro Frontends DevFeed: [Migrating Udemy's Homepage to Micro Frontends]() Original publisher: [Read original article]() Author: Eugenia Leong Kato Published: 2024-05-01T20:41:02Z Content type: article Language: en Sources: [Udemy Engineering]() Topics: [migration](), [Front end](), [Development](), [Web](), [React](), [modules](), [User Stories](), [Requirements]() Tags: [architecture](), [developer-velocity](), [development](), [frontend](), [micro-frontends](), [migration](), [module](), [react](), [requirements](), [user-stories]() ### AI overview Udemy describes migrating its logged-out homepage from a monolithic architecture to micro frontends. The article covers the migration's motivations, preparation through user stories and requirements, modular team organization, React component reuse, and shared npm packages, with reported improvements in development speed and web performance motivating the architectural change. ### Source excerpt By Eugenia Kato, Anthony Nichols, Sol Zhu, Eyupcan Bodur, Stephen Duke, David Gao, Michael Palisano, Izabela Bulska, Elaine Zhou, Matt Schaefer, Marty Note, Matthew Bise, Chris Nienhuis, Jilles Soeters, Trey Briggs, Seth Hodgson Introduction This blog delves into the migration of Udemy's logged-out homepage from a monolith architecture to a micro frontends architecture, detailing the motivations behind the migration, challenges faced, and the impact on developer velocity and web performance metrics as a result of the migration. Udemy's logged-out homepage is the second most frequently visited destination, following the course landing pages. Our current focus revolves around professional skills development to cater the needs of professionals seeking upskilling and reskilling opportunities. However, the design and content of the logged-out homepage did not align with this focus. As a result, a new homepage was redesigned that aimed to showcase Udemy's range of offerings and emphasize its commitment to professional skills development. Beyond a mere facelift, the redesign presented an opportunity for us to transition the logged-out homepage from a monolith architecture to micro frontend architecture. With the enhanced web performance metrics, especially for international users, and increased development speed demonstrated in the previous blog post, we are motivated to adopt the new frontend architecture. Preparing for the migration We laid the groundwork for the new logged-out homepage by creating a new micro-frontend application, keeping it isolated from other existing frontend applications. This allows for a more streamlined and efficient development process, as React components and pages that are not relevant can be excluded. As a preparatory step for the migration, we broke down each of the homepage modules into user stories. Then, we collaborated with our design and product partners to define clear and concise requirements for each module. This structured approach ## A Journey from Monolith to Micro frontends at Udemy -- Part 3 of 3 DevFeed: [A Journey from Monolith to Micro frontends at Udemy -- Part 3 of 3]() Original publisher: [Read original article]() Author: Hamza ERBAY Published: 2024-01-16T18:23:06Z Content type: article Language: en Sources: [Udemy Engineering]() Topics: [Front end](), [migration](), [Design system](), [GitHub Actions](), [Monitoring](), [npm packages](), [Hackathon]() Tags: [architecture](), [design-system](), [development](), [development-process](), [frontend](), [github-actions](), [micro-frontends](), [migration](), [monitoring](), [npm-packages](), [overcoming-obstacles](), [performance-metrics]() ### AI overview Part 3 of Udemy's series describes implementing a migration from a monolithic frontend to a more modular micro-frontend architecture. It covers migration guidance, shared design-system npm packages, team education, monitoring with Datadog and Sentry, simplified local development, and GitHub Actions-based releases. ### Source excerpt Transforming Frontend Architecture: A Journey from Monolith to Micro Frontends at Udemy -- Part 3 of 3 By: Chris Nienhuis, Hamza Erbay, Matthew Bise, Nathan Chapman, Seth Hodgson, Trey Briggs, Charles Pham Photo by Venti Views on UnsplashIntroduction In the previous section, we walked through the Udemy Hackathon event, and the decisions and preparations made that laid the foundation for our move toward a more modular frontend architecture. In this section, we will focus on the implementation of our vision post-Hackathon, how we are planning to execute the migration, and the impact it has had so far. If you missed Part 1 and Part 2 of this series, you can find them here: Transforming frontend Architecture: A Journey from Monolith to Micro frontends at Udemy -- Part 1... Transforming frontend Architecture: A Journey from Monolith to Micro frontends at Udemy -- Part 2... Post-Hackathon: Implementing the Vision Following the green light from the Shark Tank jury, we've taken decisive steps to translate our hackathon prototype into a scalable and robust frontend architecture. A dedicated frontend team has been formed at Udemy with the express purpose of driving this evolution. "Where the journey began... A throwback to the spark that ignited our incredible voyage. Thanks to Gregg (our former CEO) for recognizing the hard work! #Milestone #Hackathon #OriginStory" Our commitment to this new direction is reflected in a series of strategic actions: Migration Guidance: We've laid out clear guidelines for transitioning (transitioning implies migration) from the monolithic application, ensuring a smooth and structured shift to the new architecture. Collaboration with Design Systems: By partnering with the design system team, we've begun decoupling foundational components and design tokens, packaging them into shared npm packages. This initiative is critical for consistency and efficiency across all applications. Education and Alignment: Recognizing the importance of knowledge sharing, we ## A Journey from Monolith to Micro frontends at Udemy -- Part 2 of 3 DevFeed: [A Journey from Monolith to Micro frontends at Udemy -- Part 2 of 3]() Original publisher: [Read original article]() Author: Hamza ERBAY Published: 2024-01-08T18:30:04Z Content type: article Language: en Sources: [Udemy Engineering]() Topics: [Front end](), [Next.js](), [React](), [Hackathon](), [Web Development](), [GraphQL]() Tags: [architecture](), [collaboration](), [decoupling](), [frontend](), [graphql](), [hackathon](), [hackathons](), [infrastructure](), [micro-frontends](), [next-js](), [performance](), [react](), [static-site]() ### AI overview This second article in a three-part series describes Udemy's hackathon exploration of decoupling its monolithic frontend into micro frontends. It explains the limitations of tightly coupled React components, the choice of Next.js, and criteria used to select the Featured topics page as an initial candidate. ### Source excerpt Transforming Frontend Architecture: A Journey from Monolith to Micro Frontends at Udemy -- Part 2 of 3 By: Chris Nienhuis, Hamza Erbay, Matthew Bise, Nathan Chapman, Seth Hodgson, Trey Briggs, Charles Pham Photo by Hamza Erbay on UnsplashIntroduction In the previous section, we discussed the challenges in our current monolith architecture and the reasons why we are keen on transitioning to a micro frontend architecture. Now, we are going to delve into the hackathon event at Udemy where we explored the idea of decoupling our frontend structure, the preparations we made, and the options we considered for this transformation. If you missed Part 1 of this series, it is located here: Transforming frontend Architecture: A Journey from Monolith to Micro frontends at Udemy -- Part 1... Udemy Hackathon During the Hackathon event at Udemy, which we organize every six months, I had the opportunity to meet some amazing people in 2021. We collaborated on ideas for decoupling the frontend structure, which was a great experience. Our team comprised Chris Nienhuis, Hamza Erbay, Matthew Bise, Nathan Chapman, and Trey Briggs. We would like to thank Seth Hodgson for bringing us together. Preparation Our React components are currently tightly linked to the monolith runtime environment, making it impossible to use them across our various sites or in different rendering environments -- client-side, server-side, or offline via static site generation tools (B2C/B2B, support center, blogs, microsites, etc.). Udemy Monolith's high-level architecture Decoupling the frontend would improve our daily workflow and make it easier to make significant style changes across the apps. After researching modern frontend technologies, we chose Next.js for its strong community support and performance. Then, we worked on the high-level structure for the hackathon. Analyze the inventory The hackathon posed challenges related to our infrastructure and GraphQL integration that needed to be addressed. Our goal was t ## A Journey from Monolith to Micro frontends at Udemy -- Part 1 of 3 DevFeed: [A Journey from Monolith to Micro frontends at Udemy -- Part 1 of 3]() Original publisher: [Read original article]() Author: Hamza ERBAY Published: 2024-01-02T18:47:48Z Content type: article Language: en Sources: [Udemy Engineering]() Topics: [Front end](), [Architecture & Design](), [Scalability](), [Development]() Tags: [architecture](), [development](), [frontend](), [innovation](), [modular](), [monolithic-architecture](), [performance](), [scalability](), [user-experience]() ### AI overview Udemy describes its migration from a monolithic frontend architecture toward micro frontends. The article explains that scaling created bottlenecks in development speed, release processes, CDN use, and user experience, motivating a more modular and scalable architecture. ### Source excerpt Transforming Frontend Architecture: A Journey from Monolith to Micro Frontends at Udemy -- Part 1 of 3 By: Chris Nienhuis, Hamza Erbay, Matthew Bise, Nathan Chapman, Seth Hodgson, Trey Briggs, Charles Pham Photo by Annie Spratt on UnsplashIntroduction At Udemy, we have over 67M learners and over 75K instructors, and we're working on a fast and accessible experience for them. Our underlying application architecture is a monolith -- a single, unified codebase that houses our marketplace, consumer subscription services, and business applications. The engine that powers our global platform is constantly being improved by our dedicated teams working to enhance the learning and teaching experience. In the spirit of innovation, we pitched an idea at the 2021 hackathon to decouple frontend engineering from the monolithic structure. The concept resonated with the hackathon committee (Shark Tank Council), earning a decisive nod. Thus began our odyssey toward a more modular, resilient, and scalable frontend architecture. Motivation to migrate monolith to micro frontend At the heart of our decision to migrate from a monolithic architecture to micro frontends was the need to address critical pain points impeding our ability to scale and innovate effectively. Development Speed and Scalability Our monolithic application served us well in the early days, but as we scaled, it became a double-edged sword. Development speed slowed to a crawl due to two main bottlenecks: When you start a new project, the build process takes approximately 10-15 mins. Cold Start (blue and orange), Warm Start (only orange boxes)Complex Release Process Deploying our code was a difficult and time-consuming process. Any changes to the code had to go through a slow and tedious cycle of review, approval, and queuing, which often took several days, especially during release freezes. This slow process was not sustainable for a growing global team generating many experiments. The release process can take days.In 20 ## Data Quality at Udemy -- Part 1 DevFeed: [Data Quality at Udemy -- Part 1]() Original publisher: [Read original article]() Author: Murat Migdisoglu Published: 2023-09-06T22:01:16Z Content type: article Language: en Sources: [Udemy Engineering]() Topics: [Data Quality](), [data-architecture](), [data lake](), [Apache Spark](), [airflow](), [Streaming]() Tags: [airflow](), [architecture](), [data-architecture](), [data-catalog](), [data-governance](), [data-lake](), [data-lineage](), [data-quality](), [data-quality-management](), [pipelines](), [principal-engineer](), [quality](), [spark](), [streaming]() ### AI overview This article describes Udemy's efforts to improve data quality by establishing an end-to-end data lineage solution. It explains how distributed data ownership and self-service analytics make lineage important for impact analysis, change management, and identifying unused columns or orphan tables. ### Source excerpt Data Quality at Udemy -- Part 1Data Lineage Demystified- Why it Matters and How to Leverage its Magic for Informed Business Success! In late 2020, upon joining Udemy as a principal engineer for the data platform team, my focus shifted toward enhancing data quality within the organization. My journey began with conducting a comprehensive poll across the data organization, aimed at identifying the key pain points of data users. The results of the poll were eye-opening, revealing that 78% of users considered the absence of data provenance/lineage as a data quality issue. Furthermore, it was obvious that for a vast majority of the users, the inability to track data lineage was an important problem in impact analysis and detecting unused columns or orphan tables in the system. Inspired by these insights, I took the initiative to propose and launch two transformative projects. The first one, which is the subject of this article, is an ambitious initiative to establish a comprehensive end-to-end data lineage solution that will revolutionize our data ecosystem. The second project centers around data monitoring, which will be explored in another post. Udemy's sophisticated data architecture revolves around a data lake fed by diverse pipelines: system logs, streaming data from services, CDC listeners for replicated service databases, and more. The backbone of data transformations lies in Hive and Spark, while Airflow takes charge of orchestrating thousands of these pipelines. Unraveling Data Flow Complexity: Data Lineage in Growing Data Driven Organizations In the early stages of an organization's data-driven journey, data lineage may not be deemed crucial. With just a few pipelines managed by a centralized team, the dependency tree of the workflow orchestration typically suffices to comprehend the relationships between data entities. However, as the business scales up, relying on a single centralized team for all data flows becomes a bottleneck. Consequently, data organizatio ## Accelerating Online Experiments that Target Quantile Treatment Effects DevFeed: [Accelerating Online Experiments that Target Quantile Treatment Effects]() Original publisher: [Read original article]() Author: Zach Flynn Published: 2023-08-31T14:27:25Z Content type: tutorial Language: en Sources: [Udemy Engineering]() Topics: [experiments]() Tags: [analysis](), [automated](), [data-science](), [experiment](), [experimentation](), [experiments](), [metric](), [platform](), [precision](), [statistics](), [tests]() ### AI overview An Udemy engineering article explains how variance-reduction methods such as CUPED accelerate A/B experiments targeting average treatment effects, while introducing the challenge of applying variance reduction to quantile treatment effects such as the median or 90th percentile. It describes the motivation for keeping the analysis practical enough for automated experiment analysis, but the supplied text ends before the proposed quantile approach is presented. ### Source excerpt At Udemy, we run hundreds of A/B-style experiments per year on our experimentation platform. One of the biggest challenges we've faced is the need to run experiments faster to enable our product and engineering teams to rapidly ship more new product innovations to our customers. One way we've accelerated experimentation at Udemy is by building variance reduction methods -- such as CUPED -- into our experimentation platform. Doing so has had a powerful impact, helping us reduce the average time to run some product A/B tests by 30% or more. However, variance reduction methods like CUPED are designed to work for experiments that target mean or average treatment effects. They are not designed to work with quantile treatment effects, such as the median or 90th percentile impact of an experiment. In this short blog, we show how we've also reduced the variance of quantile treatment effects at Udemy, while still keeping the analysis practical and simple enough to build into our automated experiment analysis system. Variance Reduction for Average Treatment Effects Control function methods, like CUPED, are a type of variance reduction method used to increase the precision of average treatment effect estimates. They control for differences between the experimental units (e.g., site visitors, accounts) that could not have been the result of the change the experiment introduces. Removing this irrelevant variation in the outcome metric reduces noise in estimating the treatment effect. Usually, these methods control for features that existed prior to the experiment being launched, such as total minutes a user spent learning in the 30 days prior to entering the experiment. Clearly, this feature is unrelated to the experiment because it predates the unit's exposure to the experiment. But it likely explains a good portion of the variation in post-treatment minutes spent learning because, even in the absence of the experiment, people who learned more last month are likely to learn more