# data observability A data-engineering practice for monitoring and improving the health of data in applications and data pipelines. This is one page of public article previews, not the complete archive. Follow Next page to continue. Summaries are not the original full articles. ## Data Engineering Weekly #287 DevFeed: [Data Engineering Weekly #287]() Original publisher: [Read original article]() Author: Ananth Packkildurai Published: 2026-09-14T02:52:23Z Content type: article Language: en Sources: [Data Engineering Weekly]() Topics: [data-engineering](), [Kafka](), [data observability](), [Embeddings](), [Multi-tenancy](), [Event-Streaming](), [OpenAI](), [Library](), [Code]() Tags: [architecture](), [architectures](), [data-engineering](), [data-quality](), [embedding](), [engineering](), [kafka](), [multi-tenancy](), [object-storage](), [observability](), [openai](), [streaming](), [vector]() ### AI overview Data Engineering Weekly #287 covers building data platforms from scratch, including composable architectures, data quality, and observability. It also previews talks on governed machine-executable ontologies for marketing activation and fair, order-preserving Kafka consumption for many tenants. The issue links to OpenAI's storage platform scaling for ChatGPT and Pinterest's embedding retrieval platform. ### Source excerpt The Weekly Data Engineering Newsletter ## Data pipeline monitoring 101: Tracking health and performance across the data stack DevFeed: [Data pipeline monitoring 101: Tracking health and performance across the data stack]() Original publisher: [Read original article]() Author: Aaron Kaplan; Ryan Warrier Published: 2026-08-14T00:00:00Z Content type: article Language: en Sources: [Datadog | The Monitor blog]() Topics: [Monitoring](), [data observability](), [observability](), [data](), [data streams monitoring](), [Kafka](), [Apache Spark](), [event driven](), [Streaming]() Tags: [ai-ml](), [analytics](), [data](), [data-analytics](), [data-engineering](), [data-observability](), [data-streams-monitoring](), [event-driven](), [kafka](), [learn](), [monitoring](), [observability](), [performance](), [spark](), [streaming]() ### AI overview This article introduces end-to-end monitoring for modern data pipelines. It explains how to track pipeline health, performance, data quality, and availability across varied architectures and technology layers, with examples including Kafka, Flink, Apache Spark, data lakes, warehouses, and lakehouses. ### Source excerpt Learn about monitoring the end-to-end health and performance of modern data pipelines. ## Data Engineering Weekly #282 DevFeed: [Data Engineering Weekly #282]() Original publisher: [Read original article]() Author: Ananth Packkildurai Published: 2026-08-10T01:21:26Z Content type: article Language: en Sources: [Data Engineering Weekly]() Topics: [data-engineering](), [data-platforms](), [data observability](), [Graphs](), [semantic-layer](), [Large Language Model](), [Apache Iceberg](), [gRPC](), [Concurrency](), [Caching](), [Optimization](), [Streaming]() Tags: [caching](), [chaos](), [concurrency](), [data](), [data-engineering](), [data-platforms](), [data-quality](), [knowledge-graphs](), [llms](), [newsletter](), [observability](), [pipelines](), [semantic-layer](), [streaming]() ### AI overview Data Engineering Weekly #282 is a newsletter covering data platform fundamentals, semantic layers, ontology-backed knowledge graphs, converged databases, AI modernization, and Netflix's real-time distributed graph query architecture. It highlights composable architectures, data quality and observability, evolving schemas supported by LLM-assisted extraction, Iceberg full-text search, and optimization techniques including concurrency control, streaming filters, and caching. ### Source excerpt The Weekly Data Engineering Newsletter ## The continuous validation framework for data pipelines. DevFeed: [The continuous validation framework for data pipelines.]() Original publisher: [Read original article]() Author: Niruta Talwekar Published: 2026-07-23T05:40:01Z Content type: article Language: en Sources: [Platform Engineering Blog]() Topics: [data](), [DataOps](), [Data Quality](), [data observability](), [CI/CD](), [data-processing](), [DevOps](), [Platform Engineering]() Tags: [ci-cd](), [data](), [data-quality](), [devops](), [platform-engineering](), [provenance](), [software-testing](), [streaming]() ### AI overview The article introduces the Continuous Validation Framework (CVF), an end-to-end methodology for validating data pipelines through architectural isolation, configuration-driven data quality management, and continuous automation based on lineage-driven impact analysis. It reports production results including a 50% reduction in incidents and an 80% improvement in detecting data quality issues. ### Source excerpt A framework for automated, end-to-end data pipeline validation using isolation, declarative quality checks, and lineage-driven impact analysis. ## Agentic Data Engineering Is Here -- But Can It Close the Loop? DevFeed: [Agentic Data Engineering Is Here -- But Can It Close the Loop?]() Original publisher: [Read original article]() Author: Daniel Beach Published: 2026-07-22T14:05:27Z Content type: article Language: en Sources: [Data Engineering Central]() Topics: [data-engineering](), [Orchestration](), [data observability](), [Artificial Intelligence](), [AI-assisted coding](), [DuckDB](), [Apache Iceberg]() Tags: [agentic](), [ai-agents](), [data-engineering](), [duckdb](), [orchestration](), [pipelines]() ### AI overview A podcast conversation with Hugo Lu about agentic data engineering and the infrastructure needed for data platforms to execute work, observe outcomes, validate changes, and improve pipelines safely. It examines why production data systems remain difficult for AI, including schema changes, realistic testing, business semantics, and secure execution. ### Source excerpt a conversation with Hugo Lu ## Data Engineering Weekly #276 DevFeed: [Data Engineering Weekly #276]() Original publisher: [Read original article]() Author: Ananth Packkildurai Published: 2026-06-29T03:52:17Z Content type: article Language: en Sources: [Data Engineering Weekly]() Topics: [data-engineering](), [schema-evolution](), [data-architecture](), [Amazon OpenSearch Service](), [Resilience](), [data observability]() Tags: [architecture](), [compatibility](), [data-engineering](), [engineering](), [observability](), [opensearch](), [resilience](), [schema-evolution]() ### AI overview Data Engineering Weekly #276 is a newsletter roundup covering data platform fundamentals, storage and workload architecture, schema evolution in Pinterest's ingestion framework, zone-failure-resilient OpenSearch at Uber, AI modernization, and stateful reasoning systems for notebooks. ### Source excerpt The Weekly Data Engineering Newsletter ## Get reliable answers to business questions with Bits Data Analysis DevFeed: [Get reliable answers to business questions with Bits Data Analysis]() Original publisher: [Read original article]() Author: Jonathan Morin; Jonathan Parisot; Harel Shein Published: 2026-06-09T00:00:00Z Content type: article Language: en Sources: [Datadog | The Monitor blog]() Topics: [Data analysis](), [data observability](), [semantic-layer](), [SIEM, Security, Observability](), [Model Context Protocol](), [ai-coding]() Tags: [ai](), [ai-coding](), [analytics](), [bits-ai](), [data-analytics](), [data-engineering](), [data-observability](), [digital-experience-monitoring](), [mcp](), [observability](), [root-cause-analysis](), [semantic-layer]() ### AI overview Bits Data Analysis, now in Preview, helps teams answer business questions with governed context from their data stack and Datadog telemetry. It uses metric definitions, lineage, freshness, quality signals, application telemetry, and source code to select appropriate data and provide confidence indicators with links to the definitions and tables used. ### Source excerpt Learn how Bits Data Analysis answers business questions using governed data context from Datadog. ## Malicious Release of elementary-data PyPI Package Steals Cloud Credentials from Data Engineers DevFeed: [Malicious Release of elementary-data PyPI Package Steals Cloud Credentials from Data Engineers]() Original publisher: [Read original article]() Author: Liran Tal Published: 2026-04-27T23:00:00Z Content type: news Language: en Sources: [Blog RSS Feed | Snyk]() Topics: [data observability](), [GitHub Actions](), [Security](), [supply-chain-security](), [vulnerability](), [backdoor](), [Python](), [data-engineering](), [Cloud](), [ssh]() Tags: [awareness](), [aws](), [backdoor](), [blog](), [data-engineering](), [developer](), [devops](), [devrel](), [docker](), [github-actions](), [kubernetes](), [python](), [security](), [snyk-platform](), [ssh](), [supply-chain](), [supply-chain-security](), [vulnerability]() ### AI overview Attackers compromised the elementary-data Python package's GitHub Actions publication pipeline through script injection and released malicious content that stole cloud credentials and SSH secrets from data engineering environments. ### Source excerpt Attackers exploited a GitHub Actions script injection vulnerability to publish a malicious version of the elementary-data Python CLI (v0.23.3), embedding a credential-stealing backdoor that targeted dbt profiles, cloud provider keys, and SSH secrets from data engineering environments. ## Chainguard customers safe from elementary-data compromise DevFeed: [Chainguard customers safe from elementary-data compromise]() Original publisher: [Read original article]() Published: 2026-04-25T00:00:00Z Content type: article Language: en Sources: [Chainguard: Unchained]() Topics: [chainguard](), [Malware](), [data observability](), [Data Quality](), [chainguard containers](), [chainguard libraries]() Tags: [chainguard](), [chainguard-containers](), [chainguard-customers](), [chainguard-libraries](), [data-observability](), [data-quality](), [elementary-data-compromise](), [malware](), [pypi](), [pypi-malware]() ### AI overview Chainguard reports that customers using its Python Libraries and Container images were unaffected by the compromised elementary-data 0.23.3 package on PyPI. Chainguard detected malicious patterns before building the package, while the compromised release was quarantined and related GitHub and Docker artifacts were removed. ### Source excerpt Malicious elementary-data version hit PyPI. Chainguard customers stayed protected by detecting malware pre-build and serving only verified safe versions. ## Data-to-Production: Bridging the Gap Between Iceberg and Live Microservices DevFeed: [Data-to-Production: Bridging the Gap Between Iceberg and Live Microservices]() Original publisher: [Read original article]() Author: Wix Engineering Published: 2026-02-17T11:04:08Z Content type: article Language: en Sources: [Wix Engineering]() Topics: [Apache Iceberg](), [data observability](), [Microservice](), [clickhouse](), [Amazon S3](), [Architecture & Design](), [Apache Spark](), [airflow](), [Back end](), [API]() Tags: [airflow](), [amazon-s3](), [apache-iceberg](), [api](), [architecture](), [clickhouse](), [data](), [data-engineering](), [json](), [microservices](), [spark]() ### AI overview Wix describes Data-to-Production, a platform that activates data from Amazon S3 and Apache Iceberg for backend microservices. The system ingests Iceberg data into ClickHouse and serves it through a type-safe JSON API, using metadata governance and an Airflow and Python ingestion engine. ### Source excerpt At Wix, our Data Warehouse (DWH) is a massive repository of insights. Built on Amazon S3 using Apache Iceberg table formats, and populated by Trino and Spark jobs, it houses petabytes of data--from user segmentation and logs to AI chat analytics. However, storage is only half the battle. The real challenge--and the "holy grail" for many data engineering teams--is Activation : taking that petabyte-scale data and exposing it to backend microservices with millisecond latency, high availability, and... ## Monitoring Temporal Cloud with ClickStack DevFeed: [Monitoring Temporal Cloud with ClickStack]() Original publisher: [Read original article]() Author: The ClickStack Team Published: 2026-01-28T14:25:05Z Content type: article Language: en Sources: [ClickHouse Blog]() Topics: [clickhouse](), [observability](), [Monitoring](), [data observability](), [OpenTelemetry](), [Scalability](), [cloud-infrastructure]() Tags: [clickhouse](), [code](), [metrics](), [monitoring](), [observability](), [open-source](), [opentelemetry](), [scalability](), [telemetry](), [traces]() ### AI overview ClickStack integrates with Temporal Cloud's OpenMetrics endpoint to provide scalable observability for long-running workflows and high-cardinality telemetry. Built on ClickHouse, it supports logs, metrics, and traces for monitoring, troubleshooting, and performance analysis. ### Source excerpt Learn how ClickStack integrates with Temporal Cloud to deliver fast, scalable observability for high-cardinality metrics, helping you spot bottlenecks, debug failures, and understand workflow health in minutes. ## Streaming IoT and event data into Snowflake and ClickHouse DevFeed: [Streaming IoT and event data into Snowflake and ClickHouse]() Original publisher: [Read original article]() Author: Mdu Sibisi Published: 2025-12-09T00:00:00Z Content type: tutorial Language: en Sources: [Redpanda]() Topics: [Internet of things](), [Streaming](), [Redpanda-Connect](), [clickhouse](), [data observability](), [Compression](), [data-governance](), [DataOps](), [Database]() Tags: [clickhouse](), [cloud-storage](), [compliance](), [data](), [data-governance](), [event](), [iot](), [pipelines](), [redpanda-connect](), [streaming]() ### AI overview A guide to streaming IoT and event data through Redpanda and Redpanda Connect into Snowflake and ClickHouse. It compares ClickHouse for real-time analysis with Snowflake for scalable cloud storage, historical reporting, and querying, while discussing governance, compression, data freshness, and robust pipeline design. ### Source excerpt Learn how to stream IoT and event data into Snowflake and ClickHouse using Redpanda ## No, AI is not replacing Data Engineers. DevFeed: [No, AI is not replacing Data Engineers.]() Original publisher: [Read original article]() Published: 2025-05-29T00:00:00Z Content type: opinion Language: en Sources: [Mutt0-ds Notes]() Topics: [AI, ML & Data Engineering](), [data observability](), [Claude](), [ChatGPT](), [Python]() Tags: [ai](), [ai-tools](), [chatgpt](), [claude](), [data-engineering](), [python]() ### AI overview An opinion article argues that AI is not replacing data engineers, focusing on the continued importance of greenfield prototyping in data engineering. It describes messy, real-world work involving APIs, spreadsheets, data cleaning, and Python scripts, contrasting it with AI demonstrations and synthetic coding benchmarks. ### Source excerpt Claude 4 landed, and as usual, it kicked off the usual hype cycle. "Best coding model ever" "It writes stories!" "AI is replacing... ## April 2025 Newsletter DevFeed: [April 2025 Newsletter]() Original publisher: [Read original article]() Author: Mark Needham Published: 2025-04-16T08:29:42Z Content type: article Language: en Sources: [ClickHouse Blog]() Topics: [clickhouse](), [real-time](), [Data Infrastructure](), [data observability](), [Artificial Intelligence](), [airflow]() Tags: [2025](), [ai](), [airflow](), [aws](), [clickhouse](), [data](), [data-infrastructure](), [data-observability](), [newsletter](), [openai](), [real-time]() ### AI overview The April 2025 ClickHouse newsletter covers CloudQuery's experience with ClickHouse, the new query condition cache in version 25.3, ClickHouse's Rust development, the acquisition of HyperDX, community news, upcoming events, training, and ClickHouse's role in data observability and AI applications. ### Source excerpt Welcome to the April ClickHouse newsletter, which will round up what's happened in real-time data warehouses over the last month. ## Overclocking dbt: Discord's Custom Solution in Processing Petabytes of Data DevFeed: [Overclocking dbt: Discord's Custom Solution in Processing Petabytes of Data]() Original publisher: [Read original article]() Author: Chris Dong Published: 2025-04-09T00:00:00Z Content type: article Language: en Sources: [Discord Blog]() Topics: [data](), [developer-productivity](), [data observability](), [DataOps](), [Testing](), [SQL](), [Open Source](), [Software Engineering]() Tags: [analytics](), [blog-post](), [code](), [data](), [developer](), [developer-productivity](), [open-source](), [performance](), [productivity](), [quality](), [software-engineering](), [sql](), [testing](), [tool]() ### AI overview Discord describes scaling dbt to process petabytes of warehouse data while supporting more than 100 developers and over 2,500 models. It explains how custom extensions addressed long recompilation times, inefficient incremental processing, conflicting test tables, breaking changes, and complex calculations, improving performance, developer productivity, and data quality. ### Source excerpt Explore how Discord supercharged dbt with a tailored solution designed for performance, developer productivity, and data quality. ## Open Preference Dataset for Text-to-Image Generation by the 🤗 Community DevFeed: [Open Preference Dataset for Text-to-Image Generation by the 🤗 Community]() Original publisher: [Read original article]() Author: David Berenstein; ben burtenshaw; Daniel Vila; Daniel van Strien; Sayak Paul; Ame Vi; Linoy Tsaban Published: 2024-12-09T00:00:00Z Content type: article Language: en Sources: [Hugging Face - Blog]() Topics: [Open Source Models & Datasets](), [datasets](), [Synthetic Data Generation](), [flux](), [stable-diffusion](), [argilla](), [distilabel](), [data observability](), [GitHub](), [hugging face]() Tags: [argilla](), [artificial-intelligence](), [data](), [data-is-better-together](), [data-quality](), [datasets](), [distilabel](), [flux](), [generation](), [github](), [hugging-face](), [image](), [open](), [open-source](), [stable-diffusion](), [synthetic-data](), [synthetic-data-generation](), [text-to-image]() ### AI overview The article describes an open community effort to create an image-preference dataset for text-to-image generation. It covers prompt preparation with distilabel, synthetic data generation, image generation with Flux and Stable Diffusion, and filtering with text- and image-based classifiers plus manual review. The resulting dataset and related code are available through the Hugging Face Hub and GitHub. ### Source excerpt We're on a journey to advance and democratize artificial intelligence through open source and open science. ## Data quality testing DevFeed: [Data quality testing]() Original publisher: [Read original article]() Author: Lambert Le Manh Published: 2024-09-04T16:30:00Z Content type: article Language: en Sources: [The incident.io Blog]() Topics: [Data Quality](), [data observability](), [DataOps](), [ci](), [Transactions]() Tags: [ci](), [data-observability](), [data-quality](), [flaky](), [incident](), [incident-channel](), [incident-management](), [incident-response](), [observability](), [outage](), [post-mortem](), [slack-incident](), [testing](), [transactions](), [workflow]() ### AI overview The article explains how incident.io uses dbt's native testing features for data quality testing and data observability. It describes integrating validation into production and CI workflows, including custom tests and relationship tests, and addresses flaky failures caused by ingestion and transformation pipelines running at different times. ### Source excerpt Our data observability workflow uses data quality testing to ensure data meets accuracy, consistency, and reliability standards, enabling confident, data-driven decisions. See how we built it, the common challenges we encountered, and the solutions. ## "The dashboard looks broken!": How should data teams respond to incidents? DevFeed: ["The dashboard looks broken!": How should data teams respond to incidents?]() Original publisher: [Read original article]() Author: Jack Colsey Published: 2023-11-21T14:54:00Z Content type: opinion Language: en Sources: [The incident.io Blog]() Topics: [incident management](), [data observability](), [Software Engineering](), [observability](), [dashboards](), [Slack]() Tags: [dashboards](), [data](), [incident](), [incident-channel](), [incident-management](), [incident-response](), [observability](), [outage](), [post-mortem](), [slack](), [slack-incident](), [software-engineering](), [testing]() ### AI overview This article argues that incident management platforms are a good fit for data teams because data pipeline failures are inevitable and require coordinated fixes. It describes how automated testing and observability can identify failing tests, route issues to owners, detect anomalies, and notify tools such as Slack, while incident management supports the subsequent communication and collaboration needed to resolve data incidents. ### Source excerpt For data teams, the thought of using an incident management tool feels like an odd fit. But if you look deeper, it just makes sense. ## Can Debezium Lose Events? DevFeed: [Can Debezium Lose Events?]() Original publisher: [Read original article]() Published: 2023-11-14T14:00:00Z Content type: article Language: en Sources: [Gunnar Morling]() Topics: [data observability](), [Replication](), [Database](), [configuration](), [MySQL](), [Amazon RDS]() Tags: [amazon-rds](), [configuration](), [data-engineering](), [database](), [disk-space](), [mysql](), [replication](), [sql-server]() ### AI overview The article explains that Debezium should generally not miss database change events because its delivery semantics are at-least-once. Event loss can occur when operational issues cause required transaction-log data to be discarded before a connector captures it, such as when retention limits are exceeded during extended downtime. It discusses retention behavior for MySQL, Amazon RDS, SQL Server, and PostgreSQL replication slots. ### Source excerpt This question came up on the Data Engineering sub-reddit the other day: Can Debezium lose any events? I.e. can there be a situation where a record in a database get inserted, updated, or deleted, but Debezium fails to capture that event from the transaction log and propagate it to downstream consumers? ## Evaluating Equality Predicates with RangeBitmap DevFeed: [Evaluating Equality Predicates with RangeBitmap]() Original publisher: [Read original article]() Author: Richard Startin's Blog Published: 2022-12-18T00:00:00Z Content type: article Language: en Sources: [Richard Startin's Blog]() Topics: [Data structures](), [Library](), [benchmarking](), [data observability]() Tags: [benchmarking](), [cardinality](), [comparison](), [data-engineering](), [data-structure](), [java](), [library](), [memory](), [pinot](), [range](), [roaring](), [speed]() ### AI overview This article evaluates equality and inequality queries using RangeBitmap in the RoaringBitmap library. It explains how the enhancement can support equality filtering as a compact inverted-index alternative, including as a fallback for Apache Pinot range indexes, and reports faster selection than a stream-based scan in the described example while using less space than some inverted indexes. ### Source excerpt I have just implemented support for (in)equality queries against a RangeBitmap, a succinct data structure in the RoaringBitmap library which supports range queries. RangeBitmap was designed to support range queries in Apache Pinot (more details here) but this enhancement would allow a range index to be used as a fallback for (in)equality queries in case nothing better is available. Supporting (in)equality queries allows a RangeBitmap to be used as a kind of compact inverted index, trading space for time, capable of supporting high cardinality gracefully. Since RangeBitmap supports memory mapping from files, I think that it could be used for data engineering beyond Apache Pinot. ## How a strategically designed data platform elevates product outcomes DevFeed: [How a strategically designed data platform elevates product outcomes]() Original publisher: [Read original article]() Author: Doximity Published: 2022-12-09T16:00:00Z Content type: article Language: en Sources: [Doximity]() Topics: [data-engineering](), [Domain-driven design (DDD)](), [Data Infrastructure](), [data observability]() Tags: [building](), [data-engineering](), [data-infrastructure](), [domain-driven-design](), [engineering]() ### AI overview Doximity describes organizing its data function around cross-functional product teams that combine data analysts, data engineers, and product managers. The article contrasts this model with functionally separated data teams and argues that domain-focused ownership better supports data-driven products. ### Source excerpt If this article piques your interest and you would like an opportunity to do your best data work in a unique and truly data-driven environment, you're in luck: ⚡ we're hiring! ⚡ Building a company where data is fundamental has resulted in many unique challenges--one of which is how to organize our data teams. But before diving into Doximity's data team structure, let's set the stage. Doximity is commonly mentioned as the "LinkedIn for doctors" and although partially accurate it only captures a portion of our story. Today, Doximity assists medical professionals with career navigation, secure peer-to-peer communication, telehealth (Dialer), medical news and research discovery, Locum Tenens opportunities (Curative), and, most recently, clinical scheduling (Amion). I like to say that we are more of a Swiss army knife for medical professionals. Already unique in terms of our offerings and network, which includes over 2 million U.S. healthcare professionals, including over 80% of U.S. physicians, we are also unique in that we are a profoundly data-driven company. Not only do we leverage data for business intelligence and decision support, but data directly drives the majority of our products. Through talking with industry peers, it is my experience that even in today's increasingly data-driven world, many organizations still leverage an older strategy for organizing their data teams. In short, most companies have a data analyst team and a data engineering team. The division is purely functional, and if there's a need for data infrastructure, you will often find it managed by members of the general operations team. Being big proponents of domain-driven design we do not believe it is possible to achieve the best performance for modern high-performance data organizations using this model. Instead, at Doximity, we have multiple cross-functional product teams where each team is responsible for a specific product (or part of a product). These product teams are generally composed ## Algo Hour - Large Scale Data & ML Monitoring with whylogs | Alessya Visnjic DevFeed: [Algo Hour - Large Scale Data & ML Monitoring with whylogs | Alessya Visnjic]() Original publisher: [Read original article]() Published: 2022-09-29T09:00:00Z Content type: article Language: en Sources: [Stitch Fix]() Topics: [Data Quality](), [data observability](), [Monitoring](), [Machine learning](), [ai observability]() Tags: [data](), [data-ml](), [data-pipelines](), [data-quality](), [monitoring](), [observability](), [open-source]() ### AI overview This talk explains how the open-source whylogs library supports end-to-end data quality and monitoring across machine learning pipelines. It covers whylogs' lightweight statistical data collection, language- and platform-agnostic approach, architecture, and application to existing data and ML pipelines. ### Source excerpt Title: Large Scale Data & ML Monitoring with whylogs Talk Abstract: In the era of microservices, decentralized ML architectures and complex data pipelines, data quality has become a bigger challenge than ever. When data is involved in complex business processes and decisions, bad data can, and will, affect the bottom line. As a result, ensuring data quality across the entire ML pipeline is both costly, and cumbersome while data monitoring is often fragmented and performed ad hoc. An open source library called whylogs is built to address these challenges. It is a lightweight data profiling library that enables end-to-end data monitoring across the entire software stack. The library implements a language and platform agnostic approach to data quality and data monitoring. It's been deployed at massive-scale data environments, on structured and unstructured data modalities, and across a range of points in the ML lifecycle. In this talk, we will provide an overview of the whylogs architecture, including its lightweight statistical data collection approach and we will show how users can apply this library to existing data and ML pipelines. Date and Time: The talk will be held on Tuesday, October 11th at 1:00PM PDT. Recording Info: This talk was recorded live and is viewable below: Speaker Info: Alessya Visnjic is the CEO of WhyLabs, the AI Observability company building tools that power robust and responsible AI deployment. Prior to WhyLabs, Alessya was a CTO-in-residence at the Allen Institute for AI, where she evaluated commercial potential for the latest AI research. Earlier, Alessya spent 9 years at Amazon leading ML initiatives, including forecasting and data science platforms. Alessya is also the founder of Rsqrd AI, a global community of 1,000+ AI practitioners who are committed to making enterprise AI technology responsible. ## Announcing Presto Conference Tokyo 2020 DevFeed: [Announcing Presto Conference Tokyo 2020]() Original publisher: [Read original article]() Author: Yuya Ebihara, LINE Published: 2020-10-21T00:00:00Z Content type: news Language: en Sources: [Trino Blog]() Topics: [data observability](), [Amazon Web Services]() Tags: [amazon-web-services](), [conference](), [data-engineering](), [developer](), [event](), [japan]() ### AI overview Presto Conference Tokyo 2020 will be held online on November 20, featuring six sessions from Treasure Data, Amazon Web Services Japan, Repro, and LINE, plus open sessions with Presto community members. The event is aimed at people using Presto for data engineering and those interested in adopting it. ### Source excerpt Last year, Presto Conference Tokyo 2019 was held in Japan with Martin Traverso, Dain Sundstrom and David Phillips, the founders of the Presto Software Foundation. This year, the event changes to be an online only event. Presto Conference Tokyo 2020 is happening on the 20th of November. You can find out details and register right now! The event includes six sessions from Treasure Data, Amazon Web Services Japan, Repro and LINE, as well as open sessions with Martin and Brian Olsen, a Developer Advocate at Starburst Data. This is a valuable opportunity to hear from engineers who are actually using Presto. It has something for those who are using Presto for data engineering and those who don't use Presto yet but are interested in it. ## Why Data Pipelines Should Avoid Monolithic Queue Processors DevFeed: [Why Data Pipelines Should Avoid Monolithic Queue Processors]() Original publisher: [Read original article]() Author: Fuzzygroup Published: 2020-07-20T00:00:00Z Content type: article Language: en Sources: [Scott Johnson]() Topics: [data observability](), [Amazon Simple Queue Service (SQS)](), [debugging](), [Amazon Web Services](), [GPU](), [cpu](), [Tensorflow]() Tags: [aws](), [cpu](), [data-pipeline](), [data-pipelines](), [data-processing](), [debugging](), [gpu](), [process](), [sqs](), [tensorflow]() ### AI overview The article describes lessons from building a high-performance, near-real-time data pipeline on AWS with SQS and machine learning components. It argues that a monolithic queue processor can increase cloud costs when GPU-backed processing is applied to work that does not require a GPU, make debugging harder, and limit independent scaling of different processing routines. ### Source excerpt I am in the process of wrapping up a year long engagement where I: Built a high performance data pipeline Capable of processing all of Twitter in real time / near real time Applied multiple tools to the data at different stages of the pipeline Applied one or more Machine Learning models at different stages of the pipeline Operated on AWS using SQS as the queueing structure This blog post talks about one of the key things I learned in terms of the data pipeline, specifically: **Do Not Build Data Pipelines Around a Monolithic Queue Processor ** Note: By queue processor I mean the bit of software which pulls the data of the queue, operates on it and then puts it back. This lesson may be obvious to some but we took a meandering approach to this problem where we started with the idea of distributed pipeline components, moved to a monolithic approach and then ended up back at a distributed approach. As with a lot of research endeavors, the obvious conclusion wasn't quite so obvious in the throes of the research. Lesson 01: Monolithic Queue Processing Raises Your Costs At the heart of our queue processing were a number of Machine Learning components (python / tensorflow) that really needed a GPU for efficient data processing. The problem here is that when you have a monolithic queue processor, all your processing happens on a box with the GPU whether or not all that processing needs the GPU. When you are using cloud computing, you pay for the GPU whether not not it is being used for a given operation. And since GPU boxes generally cost at least 5x to 6x more than CPU only boxes, well, our monolithic queue processor proved to be an economic disaster. Lesson 02: Monolithic Queue Processing Is Harder to Debug After realizing Lesson 01, I took our monolithic queue processor apart and broke it down into 8 (ultimately 9) individual queue processors. One thing that I quickly found is that debugging the 8 individual queue processors was dramatically easier than debugging the singl