# data-pipeline Published articles for data-pipeline. This is one page of public article previews, not the complete archive. Follow Next page to continue. Summaries are not the original full articles. ## Building an Operational Ontology: An E-Commerce Walkthrough DevFeed: [Building an Operational Ontology: An E-Commerce Walkthrough]() Original publisher: [Read original article]() Author: Togo YAMANAKA Published: 2026-08-26T14:29:30Z Content type: tutorial Language: en Sources: [Data Engineering Weekly]() Topics: [semantic-layer](), [SQL](), [TypeScript](), [Open Source](), [Databases]() Tags: [customer](), [data-pipeline](), [implementation](), [integration](), [open-source](), [semantic-layer](), [sql](), [typescript]() ### AI overview A walkthrough builds an operational ontology over integrated e-commerce order data from two systems with different schemas and status encodings. It models customers, orders, products, and relationships, then introduces named actions, business rules, and write-back to systems of record. ### Source excerpt The write side of the ontology conversation: named actions, business rules, and write-back to the systems of record -- a pattern already running at enterprise scale. ## Building Menu Vision: Real-Time Dish Recognition DevFeed: [Building Menu Vision: Real-Time Dish Recognition]() Original publisher: [Read original article]() Author: Arpitha Dudi, Growth Tech Lead Published: 2026-08-20T00:00:00Z Content type: article Language: en Sources: [Yelp]() Topics: [Computer vision](), [Artificial Intelligence](), [Android](), [cameraX](), [ML Kit](), [Hackathon](), [Development](), [iOS]() Tags: [ai](), [android](), [camerax](), [data-pipeline](), [development](), [hackathon](), [ios](), [ml-kit](), [pipeline](), [prototype](), [real-time](), [recognition](), [vision]() ### AI overview Yelp describes Menu Vision, a feature that uses a phone camera, AI, augmented reality, and text recognition to identify dishes on restaurant menus and surface related user photos and reviews. The article covers its hackathon-origin Android prototype and the production system's on-device machine learning, real-time computer vision, fuzzy matching, and distributed data pipeline. ### Source excerpt Menus aren't just lists, they're a window into a restaurant's unique offerings, specialties, and personality, shaping where and what we choose to eat. But here's the challenge: reading "Kung Pao Chicken - Stir-fried chicken with peanuts in spicy sauce" doesn't tell you what the portions look like, whether other diners loved it, or if it matches your expectations. At Yelp, we knew we had the solution sitting in our user-generated content: hundreds of millions of photos, reviews, and prices for dishes. The problem? Users had to manually search for each dish, an experience that doesn't work well when you're at... ## Chaining Activities -- from text to vectors DevFeed: [Chaining Activities -- from text to vectors]() Original publisher: [Read original article]() Author: Houman Kargaran Published: 2026-08-12T00:00:00Z Content type: tutorial Language: en Sources: [Temporal Blog]() Topics: [data](), [Embeddings](), [on-prem](), [pii](), [Redis](), [retry](), [Sequences]() Tags: [article](), [community](), [data-pipeline](), [embeddings](), [on-prem](), [pii](), [redis](), [retries](), [storage]() ### AI overview This guest post explains a Temporal workflow that chains discrete Activities to process complaints while maintaining a PII boundary. It uses an on-premises all-MiniLM-L6-v2 model for embeddings, caches vectors in Redis, stores them externally, and relies on deterministic workflow sequencing and retry policies. ### Source excerpt Learn how to build a durable, PII-conscious data pipeline with Temporal using on-prem embeddings, idempotent Activities, external vector storage, and retries. ## The foundation: Why Temporal for a data pipeline? DevFeed: [The foundation: Why Temporal for a data pipeline?]() Original publisher: [Read original article]() Author: Houman Kargaran Published: 2026-07-23T00:00:00Z Content type: tutorial Language: en Sources: [Temporal Blog]() Topics: [data](), [Orchestration](), [pii](), [Requirements](), [Resilience](), [consistency](), [Availability](), [observability](), [AI search](), [Embeddings]() Tags: [auditability](), [community](), [compliance](), [data](), [data-pipeline](), [embeddings](), [execution](), [foundation](), [requirements](), [resilience](), [search](), [temporal]() ### AI overview This tutorial explains how to justify Temporal as the orchestration layer for a PII-compliant complaint-ingestion pipeline. It covers requirements for resilience, consistency, availability, observability, redaction, internal classification, and semantic search over complaint data. ### Source excerpt How to justify Temporal for a regulated data pipeline: the requirements, the naive approach, and why Durable Execution wins on resilience and auditability. ## Data Engineering Weekly #278 DevFeed: [Data Engineering Weekly #278]() Original publisher: [Read original article]() Author: Ananth Packkildurai Published: 2026-07-13T02:44:42Z Content type: article Language: en Sources: [Data Engineering Weekly]() Topics: [data-engineering](), [AI, ML & Data Engineering](), [Apache Spark](), [Kafka](), [Data Quality]() Tags: [agents](), [ai](), [data-engineering](), [data-pipeline](), [data-quality](), [engineering](), [kafka](), [releases](), [spark]() ### AI overview Data Engineering Weekly #278 is a curated newsletter covering AI-assisted engineering workflows, new leetdata.ai features, agent-oriented data systems, multilingual AI concerns, visualization with Flint, and data platform fundamentals. ### Source excerpt The Weekly Data Engineering Newsletter ## A 15-Second Health Check for Your Heroku Connect Data Pipeline DevFeed: [A 15-Second Health Check for Your Heroku Connect Data Pipeline]() Original publisher: [Read original article]() Author: Nick Prey Published: 2026-06-24T15:35:14Z Content type: tutorial Language: en Sources: [Heroku]() Topics: [Heroku](), [Command-line interface](), [Data Management](), [SQL](), [Databases]() Tags: [cli](), [data-pipeline](), [engineering](), [heroku](), [heroku-connect](), [migrations](), [salesforce](), [schema](), [synchronization](), [troubleshooting]() ### AI overview This tutorial explains how the Heroku Connect CLI plugin's diagnostic command checks connection health, schema alignment, and field configurations in a Salesforce-to-Heroku Postgres data pipeline. It highlights how the command can expose configuration mismatches and provide actionable warnings before schema updates or migrations. ### Source excerpt Heroku Connect is a fully managed, bidirectional data sync service between Salesforce and Heroku Postgres that lets developers read and write Salesforce data using standard SQL. The sync is straightforward to set up and operates smoothly in the background, but when a field does not update as expected or data seems to lag, you do not need to guess what went wrong. The post A 15-Second Health Check for Your Heroku Connect Data Pipeline appeared first on Heroku. ## The 5 Silent Failures in Data Pipelines DevFeed: [The 5 Silent Failures in Data Pipelines]() Original publisher: [Read original article]() Author: SeattleDataGuy Published: 2026-04-24T19:06:02Z Content type: article Language: en Sources: [SeattleDataGuy's Newsletter]() Topics: [Data pipelines](), [data](), [CSV]() Tags: [csv](), [dashboard](), [data-pipeline](), [data-pipelines](), [pipelines](), [reports](), [schema]() ### AI overview The article explains how data pipelines can fail silently without triggering errors or obvious warnings, causing stale or incorrect data to reach dashboards and reports. It introduces schema drift as one failure mode, including unexpected changes to CSV or XML files loaded from SFTP. ### Source excerpt How Your Pipelines Lie to You Without Throwing a Single Error ## Mapping Brazilian Cell Towers DevFeed: [Mapping Brazilian Cell Towers]() Original publisher: [Read original article]() Author: Carlos Alexandro Becker Published: 2026-03-30T18:18:07Z Content type: opinion Language: en Sources: [Carlos Becker]() Topics: [data](), [Python](), [Cloudflare](), [Open Source](), [Front end]() Tags: [brazil](), [cities](), [cloudflare](), [data](), [data-pipeline](), [frontend](), [mapping](), [open-source](), [pages](), [python](), [tower]() ### AI overview The author describes building temsinal.org, an open-source map of Brazilian cell towers. It combines tower data from ANATEL with population and municipality data from IBGE to help people compare infrastructure, coverage claims, and local commitments. ### Source excerpt I was curious about how many cell towers were around me - so I built the tool I wanted. ## Data Is the Bottleneck in Many AI Projects DevFeed: [Data Is the Bottleneck in Many AI Projects]() Original publisher: [Read original article]() Author: Brian Jenney Published: 2026-02-07T16:25:26Z Content type: opinion Language: en Sources: [Brian Jenney]() Topics: [data](), [Web Scraping](), [Structured-data](), [Artificial Intelligence]() Tags: [ai](), [data](), [data-pipeline](), [grounding](), [web-scraping]() ### AI overview The article argues that data collection and preparation, rather than wiring together AI components, are often the main bottlenecks in AI projects. It discusses messy real-world data, data pipelines, vector databases, and web scraping as ways to ground useful AI outputs. ### Source excerpt This is the first Saturday I've had off in a few weeks. ## Redpanda Serverless GA: AWS PrivateLink & metrics DevFeed: [Redpanda Serverless GA: AWS PrivateLink & metrics]() Original publisher: [Read original article]() Author: Towfiqa Yasmeen Published: 2026-02-03T00:00:00Z Content type: article Language: en Sources: [Redpanda]() Topics: [Serverless](), [Streaming](), [Kafka](), [Amazon Web Services](), [Security](), [observability](), [Self-hosted](), [Supabase]() Tags: [announcement](), [aws](), [data-pipeline](), [event-streaming](), [kafka](), [metrics](), [observability](), [security](), [self-hosted](), [serverless](), [streaming]() ### AI overview Redpanda Serverless on AWS is generally available with AWS PrivateLink, production-ready observability, elastic scaling, and a hybrid networking model that supports secure private traffic alongside a public endpoint on the same cluster. The managed, consumption-based service targets Apache Kafka-compatible streaming workloads and removes much of the operational burden of running streaming infrastructure. ### Source excerpt Zero-ops simplicity meets enterprise-grade security to unlock production-ready data streaming for builders. Here's what's new in Redpanda Serverless. ## What It Actually Takes to Build a Data Pipeline System DevFeed: [What It Actually Takes to Build a Data Pipeline System]() Original publisher: [Read original article]() Author: SeattleDataGuy Published: 2026-01-14T17:59:10Z Content type: article Language: en Sources: [SeattleDataGuy's Newsletter]() Topics: [data]() Tags: [building](), [components](), [data](), [data-pipeline](), [pipeline](), [system]() ### AI overview The article breaks down the components, tradeoffs, and practical realities of building your own data pipeline system. ### Source excerpt A breakdown of the components, tradeoffs, and realities of building your own data pipeline system ## Common Data Pipeline Patterns You'll See in the Real World DevFeed: [Common Data Pipeline Patterns You'll See in the Real World]() Original publisher: [Read original article]() Author: SeattleDataGuy Published: 2026-01-05T19:58:06Z Content type: article Language: en Sources: [SeattleDataGuy's Newsletter]() Topics: [Data pipelines]() Tags: [data](), [data-pipeline](), [patterns](), [practical](), [real-world]() ### AI overview The article provides a practical overview of the different ways data pipelines appear within real companies. ### Source excerpt A practical look at the many ways data pipelines show up inside real companies ## How we Use Dagster Automations in our Data Pipeline DevFeed: [How we Use Dagster Automations in our Data Pipeline]() Original publisher: [Read original article]() Author: Delphine Rabiller Published: 2025-12-10T11:32:42Z Content type: article Language: en Sources: [FreeAgent]() Topics: [Automation](), [data](), [migration]() Tags: [analytics-engineering](), [automation](), [dagster](), [data-ml](), [data-pipeline](), [data-pipelines](), [declarative](), [migration](), [pipeline](), [scheduled](), [sensors](), [tooling](), [triggers]() ### AI overview This engineering post explains how FreeAgent is migrating data pipelines to Dagster and re-architecting its automation logic. It describes three approaches to automating asset materialization: schedules, declarative automation, and asset sensors, including when schedules are appropriate and how declarative automation uses asset dependencies and materialization status. ### Source excerpt Introduction The heart of a reliable data platform are robust and automated data pipelines. As our team migrates our data pipelines to Dagster, re-architecting our automation logic is a crucial task. Dagster offers condition-based approaches to creating or updating a data asset (table or file), moving us toward a modern, asset-centric view of data. This [...] ## Streaming optimized data to S3 for analytics with Parquet DevFeed: [Streaming optimized data to S3 for analytics with Parquet]() Original publisher: [Read original article]() Author: Chandler Mayo Published: 2025-08-13T00:00:00Z Content type: tutorial Language: en Sources: [Redpanda]() Topics: [parquet](), [Redpanda-Connect](), [Amazon S3](), [data analytics](), [Streaming](), [Apache Spark](), [pandas](), [Serverless]() Tags: [analytics](), [analytics-systems-data-feed](), [apache-spark-and-athena-data-queries](), [athena](), [automating-data-pipelines-in-s3](), [building-dashboards-with-s3-data](), [compressing-data-with-parquet](), [data](), [data-pipeline](), [data-pipeline-for-analytics](), [event-driven-pipelines-with-s3](), [how-to](), [learn](), [pandas](), [parquet](), [parquet-files-in-amazon-s3](), [pipeline](), [real-time-data-streaming](), [redpanda-connect](), [redpanda-data-streaming](), [s3](), [serverless](), [setting-up-redpanda-observability](), [spark](), [streaming](), [streaming-data-to-s3-with-parquet](), [tutorial]() ### AI overview This tutorial explains how to use Redpanda Connect to continuously batch streaming data and write compressed Apache Parquet files to Amazon S3 for analytical workloads. It covers the benefits of Parquet and querying the resulting files with tools such as Pandas, Apache Spark, and Athena. ### Source excerpt Learn how to build a powerful data pipeline that feeds analytics systems from Redpanda using clean, compressed Parquet files in Amazon S3. ## Efficient MultiModal Data Pipeline DevFeed: [Efficient MultiModal Data Pipeline]() Original publisher: [Read original article]() Author: Aritra Roy Gosthipaty; Luis; Andres Marafioti; Sergio Paniego; Pedro Cuenca Published: 2025-07-08T00:00:00Z Content type: article Language: en Sources: [Hugging Face - Blog]() Topics: [multimodal](), [data](), [datasets](), [GPU](), [Mathematics](), [Open Source]() Tags: [artificial-intelligence](), [batching](), [community](), [data](), [data-pipeline](), [gpu](), [hardware](), [multimodal](), [nanovlm](), [open](), [open-source](), [pipeline](), [research](), [vlm]() ### AI overview This article explains how to build an efficient multimodal data pipeline for nanoVLM training. It examines waste caused by idle GPUs and excessive padding, then introduces progressively improved data preparation and batching strategies, including a knapsack-based approach to fit more useful data into each batch. ### Source excerpt We're on a journey to advance and democratize artificial intelligence through open source and open science. ## Rain: A key-value store for Strava's scale DevFeed: [Rain: A key-value store for Strava's scale]() Original publisher: [Read original article]() Author: Derick Yang Published: 2025-01-24T18:02:11Z Content type: article Language: en Sources: [Strava Engineering]() Topics: [data](), [datasets](), [Amazon S3](), [Apache Cassandra](), [Deployment](), [Databases]() Tags: [caching](), [cassandra](), [data](), [data-pipeline](), [databases](), [deployment](), [key-value-store](), [maps](), [memory](), [performance](), [production](), [s3](), [scale](), [spark]() ### AI overview Strava's Geo and Metro teams work with large, immutable datasets derived from map, GPS, and third-party data. The article describes the challenges of serving these datasets and the limitations of using PalDB and Cassandra, including slow deployments, duplicated memory usage, cost constraints, and schema inflexibility. ### Source excerpt Much of our heatmaps are built on batch data outputs stored in Rain At Strava, we love maps -- some of our most loved features are nestled on map surfaces. My team, the Geo team, is focused on building and improving these products. On the Geo and Metro teams, we tend to work with large datasets: aggregations of open source map data via OpenStreetMaps, GPS data points from uploaded activities, third-party datasets for properties like elevation, and beyond. This aggregated dataset eventually turns into Geo features we know and love, like the global heatmap, Strava Metro, the routing product, route suggestions, elevation profiles, and points of interest. We perform these data aggregations in a rather hefty data pipeline, run on a regular cadence to ensure we're serving up-to-date geo data. One of the Geo team's key challenges is efficiently serving large, immutable (write-once, read-many) datasets produced by our pipeline. This is particularly hard for compute-intensive use cases like routing, where: Write-Optimized vs. Read-Optimized Conflict: Traditional read-optimized data stores struggle with large batch writes without impacting read performance or introducing significant operational complexity. Cost Constraints: Storing rarely accessed datasets in production databases can be prohibitively expensive -- especially for projects like Strava Metro, which are accessed sporadically. Schema Complexity: Defining schemas externally from the service that uses them can be costly and inflexible for developers. Previously Our previous solution to large writes used a combination of datastores: PalDB and Cassandra. PalDB is a binary data format ideal for small datasets. The README states that it is optimized for "side data", relatively small datasets you read "in passing" on your service. PalDB is unideal, however, for larger datasets. In our case, since each service deployment required downloading the key-value file from S3, our deployments were taking upwards of twenty minutes. S ## From batch processing to streaming: AMP's data journey from open-source to ClickHouse Cloud DevFeed: [From batch processing to streaming: AMP's data journey from open-source to ClickHouse Cloud]() Original publisher: [Read original article]() Author: Chris Lawrence Published: 2025-01-20T12:45:59Z Content type: article Language: en Sources: [ClickHouse Blog]() Topics: [clickhouse](), [Data Infrastructure](), [real-time](), [Streaming](), [Shopify](), [Cloud](), [Open Source]() Tags: [batch](), [clickhouse](), [data-infrastructure](), [data-pipeline](), [ecommerce](), [open-source](), [performance](), [postgres](), [real-time](), [shopify](), [streaming]() ### AI overview AMP migrated its analytics data infrastructure from a PostgreSQL-based batch workflow to ClickHouse Cloud. The change enabled real-time streaming and improved the speed and reliability of analytics for Shopify merchants as data volume and platform complexity increased. ### Source excerpt Learn how AMP revamped its data infrastructure with ClickHouse Cloud, enabling real-time streaming for faster, more reliable analytics for Shopify merchants. ## Building a Data Pipeline to Track Strava's Bad Events DevFeed: [Building a Data Pipeline to Track Strava's Bad Events]() Original publisher: [Read original article]() Author: Bisman Sodhi Published: 2024-01-08T20:19:46Z Content type: opinion Language: en Sources: [Strava Engineering]() Topics: [data](), [airflow](), [Amazon S3](), [Python](), [Amazon Web Services](), [JSON](), [dashboards](), [Computer science]() Tags: [airflow](), [apache-airflow](), [aws](), [data](), [data-engineering](), [data-pipeline](), [data-platforms](), [integrity](), [json](), [python](), [s3](), [software-engineering](), [strava](), [tableau](), [warehouse]() ### AI overview A software engineering intern describes building a daily Apache Airflow pipeline that extracts schema-invalid user behavior events from S3, decompresses them into JSON, and loads them into Snowflake. Staging tables protect production data from partial loads, while materialized SQL views and a Tableau dashboard improve querying and monitoring. ### Source excerpt Hi my name is Bisman and I studied Computer Science at University of California, Santa Barbara. During summer of 2022, I had the most amazing experience working as a Software Engineer Intern on Strava's Data Platform Team. In the first fews weeks, I learned the tools my team uses and then spent the rest of the time working on my project. TRACKING BAD EVENTS For my major summer project, I created a data pipeline that pulls user behavior data out of external storage and persists it in our data warehouse. Strava uses a service called Snowplow to collect this user behavior data, like loading a club page or uploading a profile photo. Sometimes, this data fails to match the schema that we've set, and a piece of data that fails this schema validation is called a bad event. Previously, these bad events were temporarily stored in an Elastic Search. Persisting this data in Snowflake, our data warehouse, makes it accessible to a wider audience. It also makes it easier to incorporate the bad events data with other services used at Strava. To start my project, I created a directed acyclic graph in Apache Airflow, a scheduling framework, using python that extracts bad events data from the S3, AWS's storage service, buckets on a daily cadence. This data was stored as gzip files on S3 which I decompressed and stored the data as JSON blobs. As I was working with billions of rows of data, it was important to maintain data integrity and take measures in case data failed to load from S3. Therefore, I loaded data into a staging table in Snowflake. The staging table ensured that if loading from S3 failed, the production table would remain untouched. This data was then loaded into the production table free of any partial data. After all the data was loaded into the production table, I created six view tables because there were six different types of bad events stored in the production table. I collaborated with our stakeholders -- data analysts -- throughout this process to craft tables bas ## How a Geo Data Pipeline Bug Disrupted Skyscanner Flight Search DevFeed: [How a Geo Data Pipeline Bug Disrupted Skyscanner Flight Search]() Original publisher: [Read original article]() Author: Skyscanner Engineering Published: 2023-11-21T11:36:51Z Content type: article Language: en Sources: [Stories by Skyscanner Engineering on Medium]() Topics: [Graphs](), [data](), [dataset](), [systems](), [Kubernetes clusters]() Tags: [bug](), [data-pipeline](), [dataset](), [graph](), [issue](), [kubernetes-clusters](), [recursion](), [systems]() ### AI overview A bug in Skyscanner's geo data materialisation process made some locations their own parents, creating loops in the location hierarchy and disrupting flight search. The article explains the geo data model and the incident's technical context. ### Source excerpt How recursion brought down flight search at Skyscanner On 14th September, 2023 8:05 AM UTC (all timestamps in UTC) there was a critical bug in the output of our geo data pipeline which resulted in a number of geo locations being set as parents of themselves. This caused disruption to our service, and for this we're sorry. It gave us the opportunity to evaluate what went wrong, what we learned, and ways we could prevent a situation like this from happening again. Let's take a deep dive and explain things further. What is Geo Data? Geo data is a key dataset at Skyscanner which is used to provide systems, industry partners and travellers with a complete and accurate representation of the world. In simpler terms, any time you see an Airport, City, Region or Country used in Skyscanner, it's originating from this dataset. The most visible example across our offering can be seen when searching for flights where you will specify an airport, city or country for your journey. We use the geo data to populate origins/destinations and look for flights What was the issue? Skyscanner has been on a journey to upgrade our geo data set. At this time there are two versions, two geo models, running in parallel. Flights generally need to know about airports, cities, countries, contrary to other parts of the business where we need to model more complex relations such as districts, countries, islands, etc. Those relationships form a complex graph where locations are related to each others as parents and children. For this reason we kept our original "heritage" data set, merged it with our canonical dataset, our source of truth. We then basically generate (or reconstruct) our heritage dataset from the canonical data every day at 8am UTC. This generation is referred to as materialisation. The Materialisation Process On 14th September, a bug in the materialisation process updated some locations to be the parent of themselves. For example Scotland is now parent of Scotland. We've created a lo ## Implementing Data Validation with Great Expectations in Hybrid Environments DevFeed: [Implementing Data Validation with Great Expectations in Hybrid Environments]() Original publisher: [Read original article]() Author: Kamila Widyanto Full time DevOps; Site Reliability Engineer; Part Time Rubberduck Linkedin Profile Published: 2023-04-25T00:00:00Z Content type: tutorial Language: en Sources: [Trivago]() Topics: [data-processing](), [Hadoop](), [hdfs](), [integrity](), [Apache Spark](), [configuration](), [Library](), [Python](), [JSON](), [Open Source](), [YAML](), [version-control]() Tags: [configuration](), [data](), [data-pipeline](), [data-processing](), [data-validation](), [devops](), [distributed](), [hadoop](), [hdfs](), [integrity](), [json](), [library](), [open-source](), [python](), [version-control](), [workflow](), [yaml]() ### AI overview This article describes implementing Great Expectations for data validation in a hybrid Hadoop environment. It explains the framework's core concepts and how the authors ran it as a PySpark job in an automated data pipeline, including configuring the Data Context for HDFS constraints. ### Source excerpt Data validation is an essential step in any data processing pipeline, as it ensures the integrity and accuracy of the data to be used across all subsequent processing steps. ## ML Prediction Pipeline Orchestration with ML Control Center DevFeed: [ML Prediction Pipeline Orchestration with ML Control Center]() Original publisher: [Read original article]() Author: Vance Thornton Published: 2022-04-01T23:32:32Z Content type: article Language: en Sources: [Glassdoor Engineering]() Topics: [MLOps](), [Machine learning](), [YAML](), [data-processing](), [GitOps](), [model-deployment](), [Open Source](), [Library](), [real-time]() Tags: [data-pipeline](), [data-processing](), [gitops](), [library](), [low-latency](), [machine-learning](), [ml](), [mlops](), [mlops-platform](), [models](), [open-source](), [orchestration](), [pipeline](), [real-time](), [yaml]() ### AI overview This article describes Glassdoor's ML Control Center, an MLOps project for connecting machine learning components and orchestrating prediction pipelines. Pipelines are defined with YAML configuration, support reusable and custom components, and can run for online, near-real-time, offline batch, or automated testing use cases. ### Source excerpt A YAML based approach for building data processing pipelinesIntroduction In a previous blog post we described Glassdoor's a la carte approach to MLOps. We chose this approach because the MLOps landscape is rapidly evolving and we want to have the flexibility to use the best available options. ML Control Center (MLCC) is a project that we are developing at Glassdoor which will act as the glue to connect and unify these disparate components. Our plan is to move to a GitOps oriented approach where YAML configuration files in our Git repositories control the deployment and execution of all the tasks and services needed for our ML projects. One part of this is support for ML prediction pipelines which we recently released as an open source project at: https://gitlab.com/glassdoor-open-source/ml-control-center. A ML prediction pipeline typically involves gathering feature data from various sources, providing the data to feature extractors and/or ML models for prediction, and then writing the output to a data store. In MLCC these pipelines are defined using YAML files which specify the configuration of the operations to perform and the flow of data input/output of those operations. One of our primary design goals is to make it easy to implement the most common use cases quickly with simple YAML configuration and minimal coding. We want to allow ML scientists and engineers to focus on defining what they want the pipeline to do with many of the engineering implementation details such as parallelization, metrics, and retry logic taken care of automatically. MLCC has a library of reusable components which provide the functionality that is typically needed for our use cases and it is easy to add custom components when needed. This approach promotes flexibility and reuse by encouraging a modular implementation. Another trend that we have seen at Glassdoor is that online and near real time ML processing is becoming more common. The design of MLCC prediction pipelines makes them w ## Integrating Confluent Schema Registry with Apache Spark applications DevFeed: [Integrating Confluent Schema Registry with Apache Spark applications]() Original publisher: [Read original article]() Author: Dominik Liebler Published: 2022-01-24T08:04:19Z Content type: tutorial Language: en Sources: [YAZIO Engineering - Medium]() Topics: [Kafka](), [Apache Spark](), [Streaming](), [data lake](), [parquet](), [Kotlin](), [Kubernetes](), [ceph](), [JSON Schema]() Tags: [apache-spark](), [backpressure](), [ceph](), [confluent](), [data-engineering](), [data-lake](), [data-pipeline](), [json](), [kafka](), [kotlin](), [kubernetes](), [parquet](), [payload](), [pipelines](), [schema](), [schemaregistry](), [serialization](), [spark](), [streaming]() ### AI overview This engineering article explains YAZIO's data pipeline from mobile and web applications through Kafka and Spark Structured Streaming into a Ceph-based data lake. It discusses why schemas matter and describes replacing JSON with Apache Avro and Confluent Schema Registry to reduce message size while keeping schema information externally stored and cached. ### Source excerpt At YAZIO, we believe in making decisions backed by data to help people live healthier lives through better nutrition. For each new and existing feature we want to evaluate how well it performs and how our users interact with it. In order to do so, we need a lot of data and we need to handle backpressure in our systems. To cope with that we use a Kafka cluster managed by Strimzi operators running in Kubernetes. The data itself is being ingested from our mobile and web apps via HTTP or TCP endpoints serialized into JSON and stored in Kafka by a small application written in Kotlin/JVM. Overview of our data pipeline architecture At the other end of the pipeline, different Spark Structured Streaming applications (also written in Kotlin) dump this information into our data lake residing in a Ceph bucket. They read data from Kafka, deserialize it, transform some of the fields and write Parquet files into the data lake using a new schema. Why schemas? Schemas play an important role in data pipelines because they give meaning and context to data. In a world without schemas we would still do random interpretations about the context and meaning of data every now and then when using it. As you might have guessed already this would lead to a lot of bugs and misunderstandings. Photo by EJ Strat https://unsplash.com/photos/VjWi56AWQ9k Similar to a legal contract that binds you to certain limits, a schema binds the data to certain limits and meaning which narrow down the need of interpretation. Choice of serialization formats At the time of writing, Confluent Schema Registry supports these three serialization formats: Apache Avro Protocol Buffers (protobuf) JSON Schema From those choices, only two really provide more than just validation of the data that is ingested and transmitted through our data pipelines. Avro and Protobuf also allow us to shrink the sizes of our topics because only the payload is contained in a message, while the repeating schema will not be stored. In the cas ## How Working as a Product Owner Helped Me Be a Better Engineer (and vice versa) DevFeed: [How Working as a Product Owner Helped Me Be a Better Engineer (and vice versa)]() Original publisher: [Read original article]() Author: Pejman Poh Published: 2020-08-19T00:00:00Z Content type: article Language: en Sources: [Trivago]() Topics: [Job](), [Learning](), [Graphs](), [data](), [Code]() Tags: [code](), [collaboration](), [data-pipeline](), [engineering-culture](), [frontend](), [graph](), [job](), [learning](), [product-owner]() ### AI overview An engineer at trivago describes taking on a Product Owner role while remaining the sole engineer on a project. The experience involved stakeholder collaboration, KPI planning, design, and building a data pipeline that led to a price graph tested with more than 2 million people. The author explains how understanding user problems and broader company context improved their engineering practice. ### Source excerpt Over the past few months, I was given the opportunity to try out the life of a Product Owner (PO), alongside retaining my responsibilities as an engineer. The life of a PO has always intrigued m... ## AWS SQS Pro Tip - Lower Your Costs By Batching Up Data DevFeed: [AWS SQS Pro Tip - Lower Your Costs By Batching Up Data]() Original publisher: [Read original article]() Author: Fuzzygroup Published: 2020-07-20T00:00:00Z Content type: tutorial Language: en Sources: [Scott Johnson]() Topics: [Amazon Simple Queue Service (SQS)](), [JSON](), [data](), [Amazon Web Services]() Tags: [aws](), [batching](), [cost](), [data](), [data-pipeline](), [messages](), [sqs]() ### AI overview The article describes batching small JSON blobs into larger messages before sending them to Amazon SQS. It explains measuring blob sizes, building arrays below the 256,000-byte packet limit, serializing them as JSON, and unpacking them when received. The approach adds queue-processing overhead but reduced costs for a large-scale data pipeline. ### Source excerpt This one is a short one but a good one. I've been using SQS (disclaimer - I love SQS) lately for a large scale data pipeline. We are processing literally billions of small JSON blobs and in our cost modeling, we initially came up with an estimate that made SQS something that we couldn't use because of the number of JSON blobs. And then one of our people, thank you Don Neufeld, came up with the idea of batching up the JSON blobs and having SQS give us a batch that we could process. And while this did complicate our queue processing a bit, it made SQS cost effective. Here were the key things: When we initially loaded the queue, we measured each blob's length and then built an array of blobs until we were just below the maximum packet size, 256_000 bytes. We then serialized the array into JSON and gave that JSON to SQS. Our routine which read things back off the queue looked something like this: messages = sqs_client.receive_message(QueueUrl=input_queue.url, MaxNumberOfMessages=MAX_QUEUE_MESSAGES) if 'Messages' in messages: # when the queue is exhausted, the response dict contains no 'Messages' key for message in messages['Messages']: # 'Messages' is a list message_bodies = json.loads(message['Body']) for json_str in message_bodies: Yes this was a bit more overhead but it has worked tremendously well and let us take full advantage of SQS without running up a huge bill. Recommended. [Next page]()