# schema-evolution The practice of safely changing schemas over time while maintaining compatibility with existing data producers and consumers. This is one page of public article previews, not the complete archive. Follow Next page to continue. Summaries are not the original full articles. ## Introducing WalShadow: Sub-second Postgres replication to ClickHouse from physical WAL DevFeed: [Introducing WalShadow: Sub-second Postgres replication to ClickHouse from physical WAL]() Original publisher: [Read original article]() Author: Sai Srirampur Published: 2026-09-10T15:53:42Z Content type: article Language: en Sources: [ClickHouse Blog]() Topics: [clickhouse](), [Database](), [schema-evolution](), [Latency]() Tags: [analytics](), [architecture](), [benchmarks](), [clickhouse](), [github](), [latency](), [open-source](), [postgres](), [replication](), [schema-evolution](), [transactions]() ### AI overview WalShadow is an open-source engine that replicates PostgreSQL data to ClickHouse from the physical WAL stream. The article describes benchmark results of about 200 ms visibility latency and 289K rows per second, plus support for initial loads, continuous replication, schema evolution, recovery, and source switchovers. ### Source excerpt WalShadow replicates Postgres data directly from physical WAL into ClickHouse, delivering around 200 ms latency and 289,000 rows per second in benchmarks. ## MCP went stateless: Is your AWS MCP server deployment well-architected? DevFeed: [MCP went stateless: Is your AWS MCP server deployment well-architected?]() Original publisher: [Read original article]() Author: Anand Komandooru Published: 2026-09-01T13:09:19Z Content type: article Language: en Sources: [AWS Architecture Blog]() Topics: [Model Context Protocol](), [Deployment](), [schema-evolution](), [Amazon Bedrock]() Tags: [amazon-bedrock](), [announcements](), [aws](), [best-practices](), [deployment](), [mcp](), [mcp-server](), [observability]() ### AI overview The article explains MCP's move to a stateless core and how it simplifies AWS MCP server deployments by removing the need for sticky sessions and shared session stores. It maps the MCP 2026-07-28 specification to AWS Well-Architected Agentic AI Lens practices and recommends migration. ### Source excerpt On July 28, 2026, MCP made its protocol core stateless, removing the initialize handshake and session header. This post maps the MCP 2026-07-28 specification to the AWS Well-Architected Agentic AI Lens, pillar by pillar, and shows why the stateless design lets you delete the sticky sessions and session stores your MCP servers needed on AWS. ## Schema Evolution: Changing the Contract Without Breaking What Runs DevFeed: [Schema Evolution: Changing the Contract Without Breaking What Runs]() Original publisher: [Read original article]() Author: ByteByteGo Published: 2026-08-20T15:32:18Z Content type: article Language: en Sources: [ByteByteGo]() Topics: [schema-evolution]() Tags: [article](), [evolution](), [schema](), [schema-evolution]() ### AI overview The article examines schema evolution and strategies for managing changes to data schemas. ### Source excerpt In this article, we will look at schema evolution and strategies for the same. ## Tableflow: Turn Kafka Topics into Iceberg Tables DevFeed: [Tableflow: Turn Kafka Topics into Iceberg Tables]() Original publisher: [Read original article]() Author: Mohtasham Sayeed Mohiuddin Published: 2026-07-10T15:36:14Z Content type: tutorial Language: en Sources: [Confluent: Data in motion]() Topics: [Apache Iceberg](), [Kafka](), [schema-evolution](), [Confluent Cloud](), [parquet](), [Amazon S3](), [Streaming](), [AWS Glue](), [Amazon Redshift](), [BigQuery]() Tags: [amazon-redshift](), [amazon-s3](), [apache-iceberg](), [aws-glue](), [bigquery](), [confluent-cloud](), [kafka](), [parquet](), [schema-evolution](), [streaming](), [technologies](), [tutorial]() ### AI overview This tutorial explains how Confluent Cloud Tableflow continuously materializes Apache Kafka topics as Apache Iceberg or Delta Lake tables. It covers automatic schema handling, type conversion, schema evolution, Parquet conversion, catalog publishing, and table maintenance for querying streaming data with analytics engines and warehouses. ### Source excerpt Learn how Confluent Tableflow turns Kafka topics into Iceberg tables for zero-ETL analytics with automatic schema evolution and open catalog access. ## Data Engineering Weekly #276 DevFeed: [Data Engineering Weekly #276]() Original publisher: [Read original article]() Author: Ananth Packkildurai Published: 2026-06-29T03:52:17Z Content type: article Language: en Sources: [Data Engineering Weekly]() Topics: [data-engineering](), [schema-evolution](), [data-architecture](), [Amazon OpenSearch Service](), [Resilience](), [data observability]() Tags: [architecture](), [compatibility](), [data-engineering](), [engineering](), [observability](), [opensearch](), [resilience](), [schema-evolution]() ### AI overview Data Engineering Weekly #276 is a newsletter roundup covering data platform fundamentals, storage and workload architecture, schema evolution in Pinterest's ingestion framework, zone-failure-resilient OpenSearch at Uber, AI modernization, and stateful reasoning systems for notebooks. ### Source excerpt The Weekly Data Engineering Newsletter ## Automated Schema Evolution in Pinterest's Next-Generation DB Ingestion Framework DevFeed: [Automated Schema Evolution in Pinterest's Next-Generation DB Ingestion Framework]() Original publisher: [Read original article]() Author: Pinterest Engineering Published: 2026-06-24T18:01:01Z Content type: article Language: en Sources: [Pinterest Engineering Blog - Medium]() Topics: [schema-evolution](), [Apache Iceberg](), [Automation](), [Databases](), [Apache Spark](), [Kafka](), [Pull Request](), [Code generation](), [datasets](), [monitor](), [Provisioning]() Tags: [automation](), [code-generation](), [datasets](), [engineering](), [icebergs](), [kafka](), [monitor](), [pinterest](), [process](), [schema-evolution](), [software](), [spark](), [storage]() ### AI overview Pinterest describes an automated schema-evolution framework for its CDC-based database ingestion platform, which uses Kafka, Flink, Spark, and Iceberg. The framework propagates supported schema changes across the stack, uses pull-request-based versioning and auditing, targets SLA-based consistency between online and offline schemas, and provides recovery paths for unsupported or ambiguous changes. ### Source excerpt Yisheng Zhou | Software Engineer II Liang Mou | Sr Staff Software Engineer Gabriel Raphael Garcia Montoya | Staff Software Engineer Istvan Podor | Staff Software Engineer Introduction In the first post of this series, we introduced Pinterest's next-generation CDC-based ingestion platform built on Kafka, Flink, Spark, and Iceberg. In production, upstream schemas are constantly evolving, and in a distributed CDC pipeline, schema is not just metadata -- it is a cross-system contract spanning ingestion, transformation, storage, and historical backfill. A schema change that is not handled carefully can break Flink jobs, block Spark upserts, or create inconsistencies between online and offline representations. This post walks through how we make schema evolution safe in practice: the onboarding model it builds on, the changes we support and the tradeoffs we accept, how updates propagate across the stack, and how we monitor and recover when things go wrong. We end with where we're heading next -- zero-gap schema evolution. Background & Motivation Schema evolution was particularly challenging in our environment because our ingestion pipeline is composed of multiple tightly related stages, each of which depends on schema in a different way. A single table onboarding flow spans CDC source configuration, Kafka provisioning, Flink and Spark code generation, Iceberg CDC and base table creation, and historical bootstrap -- all driven by the same schema. Once a source schema changes, that change must be reflected across code, metadata, and storage in a controlled and auditable way. Without a unified automation framework, manual updates across these layers increase the risk of drift, failed deployments, and inconsistent datasets. We therefore needed a way to make schema evolution safe, repeatable, and scalable. Our Solution: SLA-Based Automated Schema Evolution To address this problem, we built an automated schema evolution framework on top of our CDC ingestion platform, providing: au ## How Spyne simplified their CDC pipeline with ClickPipes and ClickHouse Cloud DevFeed: [How Spyne simplified their CDC pipeline with ClickPipes and ClickHouse Cloud]() Original publisher: [Read original article]() Author: ClickHouse Published: 2026-06-22T11:37:29Z Content type: article Language: en Sources: [ClickHouse Blog]() Topics: [clickhouse](), [schema-evolution](), [Kafka](), [MongoDB](), [MySQL](), [Streaming](), [developer velocity](), [Serverless](), [Cloud](), [real-time](), [DevOps]() Tags: [clickhouse](), [cloud-infrastructure](), [data](), [developer-velocity](), [devops](), [kafka](), [mysql](), [open-source](), [real-time](), [schema-evolution](), [serverless](), [streaming]() ### AI overview Spyne migrated its change data capture pipeline from a self-managed Debezium and Kafka stack to serverless ClickPipes and ClickHouse Cloud. The change reduced table onboarding time from more than 1.5 hours to minutes and automated schema evolution, improving data consistency and developer velocity. ### Source excerpt Spyne cut table onboarding from 1.5+ hours to minutes and eliminated schema drift by migrating their CDC pipeline from a self-managed Debezium/Kafka stack to ClickPipes and ClickHouse Cloud. ## The open ecosystem around ClickHouse DevFeed: [The open ecosystem around ClickHouse]() Original publisher: [Read original article]() Author: Al Brown Published: 2026-06-17T15:41:30Z Content type: article Language: en Sources: [ClickHouse Blog]() Topics: [clickhouse](), [Model Context Protocol](), [Open Source](), [OpenTelemetry](), [Apache Iceberg](), [Database](), [MCP Server](), [observability](), [Prometheus](), [schema-evolution](), [Artificial Intelligence]() Tags: [ai](), [clickhouse](), [database](), [mcp-server](), [model-context-protocol](), [observability](), [open](), [open-source](), [opentelemetry](), [prometheus](), [schema-evolution]() ### AI overview ClickHouse's open ecosystem connects databases with observability, AI tooling, and lakehouse technologies through open standards. The article highlights OpenTelemetry and Prometheus support, an open-source MCP server compatible with multiple agent frameworks, and support for Iceberg and Delta Lake formats. ### Source excerpt Around any good database sits the clients you import, the dashboards your team shares, the pipelines that feed it, the projects that build on top of it, and now the agents that query it. ## Announcing Apache Iceberg 1.11.0 DevFeed: [Announcing Apache Iceberg 1.11.0]() Original publisher: [Read original article]() Author: Google Open Source (noreply@blogger.com) Published: 2026-05-27T18:30:00Z Content type: release Language: en Sources: [Google Open Source Blog]() Topics: [Apache Iceberg](), [Apache Spark](), [apache-flink](), [schema-evolution](), [Streaming](), [Encryption](), [Cryptography]() Tags: [apache-flink](), [apache-iceberg](), [apache-spark](), [encryption](), [kms](), [lakehouse](), [releases](), [schema-evolution](), [streaming]() ### AI overview Apache Iceberg 1.11.0 adds support for Apache Spark 4.1 and Apache Flink 2.1, makes both default build targets, and introduces changes including server-side REST catalog scan planning, partition statistics APIs, built-in envelope encryption with Google KMS support, and Google Storage Analytics integration. ### Source excerpt by Alex Stephen & Talat Uyarer, Lakehouse Apache Iceberg project has just launched version 1.11.0! A lot has happened since the last version. Iceberg 1.11.0 adds support for Apache Spark 4.1 and Apache Flink 2.1, the latest releases of the two engines and makes both the default build targets The rest are more structural. The REST catalog learns to plan scans server-side, shifting metadata work off the query engine. A new partition statistics scan API gives optimizers a clean, supported way to read a table's shape. Built-in table encryption arrives with envelope encryption and Google KMS support. And Google Storage Analytics library integration makes your Iceberg workloads faster than before. Let's take a look at some of the biggest changes. Spark & Flink Updates As Spark and Flink are moving forward, the 1.11.0 release is pushing forward for new version support in both. Spark 4.1 & DSv2 Migration: Spark 4.1 unlocks is MERGE INTO with automatic schema evolution: Spark's newer MERGE syntax accepts a WITH SCHEMA EVOLUTION clause, so a MERGE whose source carries columns the target table lacks can add those columns to the table within the same statement, with no separate ALTER TABLE round trip. Beyond the version bump, the 1.11 Spark connector also modernizes against Spark's newer DataSource V2 APIs and adds an asynchronous micro-batch planner that speeds up Structured Streaming. Flink Ecosystem Updates: Initial work for Flink 2.1 support has landed in the core repository, continuing Iceberg's promise of providing first-class, low-latency streaming sink capabilities. The centerpiece of the Flink work is the DynamicIcebergSink, an experimental sink that breaks the old one-sink-per-table model: a single sink routes each record to a table chosen at runtime, creating tables on demand and evolving their schemas and partition specs on the fly as the input changes including dropping columns once you opt in with dropUnusedColumns. In addition to DynamicIcebergSInk work Flink sta ## Integrating the Rust Delta Kernel into ClickHouse DevFeed: [Integrating the Rust Delta Kernel into ClickHouse]() Original publisher: [Read original article]() Author: Melvyn Peignon; Kseniia Sumarokova; Raúl Marín Published: 2026-05-22T09:32:20Z Content type: article Language: en Sources: [ClickHouse Blog]() Topics: [clickhouse](), [Rust](), [interoperability](), [schema-evolution](), [Data Management]() Tags: [clickhouse](), [data-management](), [guides](), [interoperability](), [performance](), [rust](), [schema-evolution](), [sql]() ### AI overview This article explains how ClickHouse integrated the Rust Delta Kernel to replace its native Delta Lake implementation. The integration provides a maintained interface for working with the table format, reduces integration and maintenance complexity, and enables features including writes, schema evolution, time travel, and partition pruning. ### Source excerpt How we integrated the Rust Delta Kernel to replace our native Delta Lake implementation, reducing maintenance overhead while unlocking writes, schema evolution, time travel, and partition pruning. ## May 2026 Updates #2 for XCP-ng 8.3 LTS DevFeed: [May 2026 Updates #2 for XCP-ng 8.3 LTS]() Original publisher: [Read original article]() Author: Samuel Verschelde Published: 2026-05-07T17:32:25Z Content type: release Language: en Sources: [XCP-ng Blog]() Topics: [schema-evolution](), [API](), [Feathers]() Tags: [2026](), [api](), [compatibility](), [issue](), [lts](), [maintenance](), [maintenance-updates](), [storage](), [update](), [updates]() ### AI overview This release provides maintenance updates for XCP-ng 8.3 LTS, fixing a backward-compatibility issue that affected rolling pool updates using XOSTOR. ### Source excerpt Today's update addresses an issue with the rolling pool update process on pools using XOSTOR. ## Introducing Iceberg output for Redpanda Connect DevFeed: [Introducing Iceberg output for Redpanda Connect]() Original publisher: [Read original article]() Author: Prakhar Garg Published: 2026-03-05T00:00:00Z Content type: release Language: en Sources: [Redpanda]() Topics: [Apache Iceberg](), [Redpanda-Connect](), [Streaming](), [Routing (disambiguation)](), [schema-evolution](), [YAML](), [data]() Tags: [apache-iceberg](), [connectors](), [integration](), [pii](), [redpanda-connect](), [routing](), [schema-evolution](), [streaming](), [yaml]() ### AI overview Redpanda introduces an Iceberg output for Redpanda Connect that writes streaming data directly to Apache Iceberg tables through declarative YAML pipelines. The component supports transformations, enrichment, PII removal, and content-based routing across Redpanda Connect inputs and processors, enabling data from sources such as HTTP webhooks, Postgres CDC streams, and GCP Pub/Sub to reach lakehouse tables with less infrastructure and configuration overhead. ### Source excerpt Land any data into Apache Iceberg with Redpanda Connect. Maximize infra density, automate schema evolution, and scale 1-to-many routing without the toil. ## Making large Postgres migrations practical: 1TB in 2 hours with PeerDB DevFeed: [Making large Postgres migrations practical: 1TB in 2 hours with PeerDB]() Original publisher: [Read original article]() Author: Amogh Bharadwaj Published: 2026-02-19T15:26:55Z Content type: comparison Language: en Sources: [ClickHouse Blog]() Topics: [Databases](), [migration](), [PostgreSQL](), [benchmarking](), [Replication](), [schema-evolution](), [Open Source](), [AWS Database Migration Service]() Tags: [benchmarking](), [databases](), [migration](), [open-source](), [postgres](), [postgresql](), [replication](), [schema-evolution]() ### AI overview This article explains how PeerDB supports large-scale Postgres-to-Postgres migrations with minimal downtime. It compares a 1TB migration using pg_dump/pg_restore, native logical replication, and PeerDB, focusing on initial-load performance, continuous change data capture, operational trade-offs, and support for complex PostgreSQL workloads. ### Source excerpt PeerDB's Postgres to Postgres migration features enable teams to migrate their existing Postgres databases quickly and with minimal downtime. ## Zero-Copy Graph Analytics: Getting Started With LakeHouse Graph DevFeed: [Zero-Copy Graph Analytics: Getting Started With LakeHouse Graph]() Original publisher: [Read original article]() Author: Maruthi Lokanathan Published: 2026-02-03T15:39:34Z Content type: tutorial Language: en Sources: [ClickHouse Blog]() Topics: [graph-database](), [clickhouse](), [Graphs](), [Databases](), [schema-evolution](), [Latency](), [SQL](), [data]() Tags: [clickhouse](), [database](), [graph-database](), [latency](), [real-time](), [schema-evolution](), [sql](), [systems]() ### AI overview This tutorial explains why copying analytical data into a separate graph database creates ETL, synchronization, schema-evolution, storage, and latency problems. It introduces zero-copy graph analytics, using PuppyGraph with ClickHouse to query relationships directly over existing tables through Cypher or Gremlin without moving the data. ### Source excerpt Why copying data for graph analytics doesn't scale, and what you can do instead ## Real-time analytics at scale: Redpanda and Snowflake Streaming DevFeed: [Real-time analytics at scale: Redpanda and Snowflake Streaming]() Original publisher: [Read original article]() Author: Ben Barkhouse Published: 2025-10-02T00:00:00Z Content type: article Language: en Sources: [Redpanda]() Topics: [Streaming](), [benchmarking](), [real-time](), [Latency](), [Low Latency](), [Amazon EC2](), [Amazon Web Services](), [data](), [schema-evolution](), [Terraform](), [Ansible](), [datasets]() Tags: [analytics](), [ansible](), [aws](), [benchmark](), [benchmarks](), [deployment](), [latency](), [low-latency](), [pipelines](), [real-time](), [schema-evolution](), [streaming](), [terraform]() ### AI overview This article benchmarks a Redpanda and Snowflake streaming pipeline for near-real-time analytics on equity market data. It describes the AWS-based architecture, automated deployment with Terraform and Ansible, Redpanda Connect connectors, schema evolution, parallel processing, batching, exactly-once delivery, and a high-volume workload containing 3.8 billion AVRO-encoded messages. ### Source excerpt Turn streaming data into insights faster. Learn how Redpanda Connect delivers high-volume, low-latency pipelines directly into Snowflake tables. ## IoT for fun and Prophet: Scaling IoT and predicting the future DevFeed: [IoT for fun and Prophet: Scaling IoT and predicting the future]() Original publisher: [Read original article]() Author: Bryan Wood Published: 2025-07-22T00:00:00Z Content type: tutorial Language: en Sources: [Redpanda]() Topics: [Apache Iceberg](), [Internet of things](), [Redpanda-Connect](), [Streaming](), [Time Series](), [ESP32](), [Amazon Web Services](), [Amazon S3](), [schema-evolution](), [event driven](), [Hardware](), [Kafka]() Tags: [apache-iceberg](), [apache-iceberg-for-scalable-storage](), [aws](), [core](), [devices](), [esp32-mqtt-iot-example](), [event-driven](), [hardware](), [integrating-iceberg-and-redpanda](), [iot](), [iot-data-pipeline-architecture](), [iot-predictive-analytics](), [iot-sensor-data-forecasting](), [predictive-analysis-with-iot](), [prophet-forecasting-for-iot](), [real-time-iot-data-streaming](), [redpanda-connect](), [redpanda-connect-for-iot](), [s3](), [scaling-iot-with-prophet](), [schema-evolution](), [schema-management-in-iot](), [sensor](), [streaming](), [thought-leadership]() ### AI overview This tutorial presents a scalable IoT data pipeline for real-time streaming and forecasting. It combines Redpanda and Redpanda Connect for ingestion and messaging, Apache Iceberg for scalable time-series data storage, AWS IoT for device-to-cloud communication, and ESP32 hardware for affordable connected-device experiments and deployments. ### Source excerpt Check out this real-world example of scaling IoT for predictive analysis with Redpanda, Iceberg, and Prophet--without high costs or complexity. ## Building on open table formats DevFeed: [Building on open table formats]() Original publisher: [Read original article]() Author: Prashant Sridharan Published: 2025-05-29T07:00:00Z Content type: article Language: en Sources: [Supabase Blog]() Topics: [Apache Iceberg](), [data](), [datasets](), [schema-evolution](), [Amazon S3](), [cloud-infrastructure](), [Databases](), [parquet](), [systems](), [Filesystems](), [Streaming](), [Supabase]() Tags: [amazon-s3](), [analytics](), [apache-iceberg](), [compatibility](), [data](), [data-platforms](), [data-processing](), [databricks](), [datasets](), [object-storage](), [open](), [parquet](), [schema-evolution](), [spark](), [storage](), [streaming](), [systems](), [transactions]() ### AI overview The article explains how open table formats organize and manage large datasets on distributed storage. It compares Apache Iceberg, Delta Lake, and Apache Hudi, then focuses on Apache Iceberg's support for analytics, ACID transactions, schema evolution, partition evolution, hidden partitioning, time travel, and safe concurrent access across processing engines. Iceberg operates over Parquet and other data files in object storage such as Supabase Storage, Amazon S3, Google Cloud Storage, and Azure Blob Storage. ### Source excerpt Open table formats like Apache Iceberg, Delta Lake, and Apache Hudi are transforming how developers manage large-scale data on object storage systems. ## Scaling Nextdoor's Datastores: Part 3 DevFeed: [Scaling Nextdoor's Datastores: Part 3]() Original publisher: [Read original article]() Author: Ronak Shah Published: 2025-03-19T15:08:43Z Content type: article Language: en Sources: [Nextdoor]() Topics: [Caching](), [Databases](), [schema-evolution](), [Object-relational mapping](), [Django](), [Python](), [Redis](), [valkey]() Tags: [blog](), [cache](), [cache-invalidation](), [caching](), [compatibility](), [database](), [django](), [lua](), [python](), [redis](), [schema](), [serialization-format](), [thundering-herd](), [valkey]() ### AI overview Part 3 of Nextdoor's datastore-scaling series explains how applications serialize database objects for Redis or Valkey look-aside caches. It describes compatibility problems caused by runtime, package, and schema changes, including cache misses and thundering-herd effects during migrations. ### Source excerpt In this part of the Scaling Nextdoor's Datastores blog series, we'll explore how the Core-Services team at Nextdoor serializes database data for caching while ensuring forward and backward compatibility between the cache and application code. In part 1 of this series we discussed how ORMs, object-relational mapping frameworks, help abstract away database specific schemas and queries from application code. Developers simply utilize objects in their application's language to access database data. Here's a simple example of using Python's Django ORM to define a model: from django.db import models class Users(models.Model): first_name = models.CharField(max_length=30) last_name = models.CharField(max_length=30) The associated SQL create table would look like: CREATE TABLE users ( "id" bigint NOT NULL PRIMARY KEY GENERATED BY DEFAULT AS IDENTITY, "first_name" varchar(30) NOT NULL, "last_name" varchar(30) NOT NULL ); Developers would then access database data like this: user_id = 123 user = User.objects.get(id=user_id) print(user.first_name)Object Byte Serialization for Caching An issue arises when adding a look-aside cache such as Redis/Valkey to an application: How do you store what you got from the database in the cache? A common solution to caching complex objects, such as those from ORMs, is object byte serialization. This process converts language objects into bytes before storing them in the cache. When reading from the cache the process is done in reverse where the byte data is turned into language objects. For instance in Python this is often done with the pickle package. The interaction between the application, database, and the cache looks like this: Look-Aside Cacheimport pickle # Try getting from cache ('None' if not in cache) user_bytes = cache.get("user_123") if user_bytes is not None: # Read bytes using pickle user = pickle.loads(user_bytes) else: # Fetch from database user = User.objects.get(id=123) # Convert to bytes user_bytes = pickle.dumps(user) # Sto ## An API to Track Database Schema Changes DevFeed: [An API to Track Database Schema Changes]() Original publisher: [Read original article]() Author: Luis Tavares Published: 2025-01-09T17:08:23Z Content type: article Language: en Sources: [Blog -- Neon Docs]() Topics: [API](), [schema-evolution](), [Database Migration](), [Databases](), [Repl.it](), [CI/CD](), [GitHub Actions](), [Artificial Intelligence](), [Large Language Model]() Tags: [agentic](), [api](), [ci-cd](), [database](), [github](), [github-action](), [llms](), [migration](), [postgres](), [product](), [schema-evolution]() ### AI overview The article introduces the compare_schema API endpoint for Neon's Schema Diff feature. It explains how the API compares schemas between Neon databases and supports CI/CD automation, agentic systems, database migrations, and schema evolution, including use cases involving Replit Agent and Postgres. ### Source excerpt We keep expanding our Schema Diff feature, this time adding an API endpoint: compare_schema. You can use it in all your Neon projects, including the Free Plan. Schema Diff allows you to easily compare schemas between Neon databases. It was first made available via the Neon Consol... ## Modernizing Your Android App's Data Storage: SharedPreferences to DataStore DevFeed: [Modernizing Your Android App's Data Storage: SharedPreferences to DataStore]() Original publisher: [Read original article]() Author: Michael Evans Published: 2024-10-10T15:53:57Z Content type: tutorial Language: en Sources: [Gadget Habit]() Topics: [Android](), [migration](), [schema-evolution](), [reactive](), [Coroutines](), [Google]() Tags: [android](), [best-practices](), [coroutines](), [flow-api](), [migration](), [pitfalls](), [protocol](), [reactive](), [schema-evolution](), [testing]() ### AI overview This tutorial explains how to migrate Android app data from SharedPreferences to Proto DataStore. It covers dependencies, schema definition, serialization, repository design, migration testing, schema evolution, error handling, performance, and testing practices. ### Source excerpt SharedPreferences has long been a staple for storing small pieces of data and user preferences in Android apps. However, it has notable limitations, such as a lack of type safety, no support for safe schema evolution, and potential performance issues on the main thread. Google introduced Proto DataStore as a modern and robust alternative, offering: Strong typing with Protocol Buffers Safe schema evolution Built-in migration support Flow API for reactive programming Coroutines support for main-thread safety In this post, we'll walk through the process of migrating your existing SharedPreferences data to Proto DataStore without data loss, including best practices and common pitfalls to avoid. Step 1: Add Dependencies First, add the necessary dependencies to your app's build.gradle file: 1 2 3 4 5 6 7 8 9 dependencies { def datastore_version = "1.0.0" // Proto DataStore implementation "androidx.datastore:datastore:$datastore_version" // Protocol Buffers implementation "com.google.protobuf:protobuf-javalite:3.18.0" } Step 2: Define Your Proto DataStore Schema Create a new .proto file in app/src/main/proto/my_data.proto to define your data schema: 1 2 3 4 5 6 7 8 9 10 11 12 13 14 syntax = "proto3"; option java_package = "com.example.app"; option java_multiple_files = true; message UserPreferences { // Define your fields with unique numbers string user_name = 1; bool notifications_enabled = 2; string theme = 3; // Optional: Add a version field for future schema evolution int32 schema_version = 999; } Note the schema_version field - this helps manage schema evolution as your app grows. Step 3: Create a Proto DataStore Serializer The serializer handles reading and writing your protocol buffer messages: 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 object UserPreferencesSerializer : Serializer { override val defaultValue: UserPreferences = UserPreferences.getDefaultInstance() override suspend fun readFrom(input: InputStream): UserPreferences { try { return UserPrefere ## Spotlight on SIG API Machinery DevFeed: [Spotlight on SIG API Machinery]() Original publisher: [Read original article]() Author: The Kubernetes Authors Published: 2024-08-07T00:00:00Z Content type: article Language: en Sources: [Kubernetes Contributors Blog]() Topics: [Kubernetes](), [API](), [schema-evolution](), [Back end](), [Front end]() Tags: [api](), [backend](), [frontend](), [kubernetes](), [schema](), [schema-evolution](), [spotlight]() ### AI overview An interview with Federico Bongiovanni and David Eads, chairs of Kubernetes SIG API Machinery, explores the group's scope and responsibilities. They discuss the kube-apiserver, backend storage contracts, API schema evolution, serialization, client and controller patterns, and the importance of avoiding regressions or incompatible changes in the Kubernetes control plane. ### Source excerpt We recently talked with Federico Bongiovanni (Google) and David Eads (Red Hat), Chairs of SIG API Machinery, to know a bit more about this Kubernetes Special Interest Group. Introductions Frederico (FSM): Hello, and thank your for your time. To start with, could you tell us about yourselves and how you got involved in Kubernetes? David: I started working on OpenShift (the Red Hat distribution of Kubernetes) in the fall of 2014 and got involved pretty quickly in API Machinery. My first PRs were fixing kube-apiserver error messages and from there I branched out to kubectl (kubeconfigs are my fault!), auth (RBAC and *Review APIs are ports from OpenShift), apps (workqueues and sharedinformers for example). Don't tell the others, but API Machinery is still my favorite :) Federico: I was not as early in Kubernetes as David, but now it's been more than six years. At my previous company we were starting to use Kubernetes for our own products, and when I came across the opportunity to work directly with Kubernetes I left everything and boarded the ship (no pun intended). I joined Google and Kubernetes in early 2018, and have been involved since. SIG Machinery's scope FSM: It only takes a quick look at the SIG API Machinery charter to see that it has quite a significant scope, nothing less than the Kubernetes control plane. Could you describe this scope in your own words? David: We own the kube-apiserver and how to efficiently use it. On the backend, that includes its contract with backend storage and how it allows API schema evolution over time. On the frontend, that includes schema best practices, serialization, client patterns, and controller patterns on top of all of it. Federico: Kubernetes has a lot of different components, but the control plane has a really critical mission: it's your communication layer with the cluster and also owns all the extensibility mechanisms that make Kubernetes so powerful. We can't make mistakes like a regression, or an incompatible change, ## A report from the Trino Conference Tokyo 2023 DevFeed: [A report from the Trino Conference Tokyo 2023]() Original publisher: [Read original article]() Author: Yuya Ebihara Published: 2023-10-11T00:00:00Z Content type: article Language: en Sources: [Trino Blog]() Topics: [Apache Iceberg](), [schema-evolution](), [Apache Spark](), [Data Infrastructure](), [migration](), [Serverless](), [Kubernetes](), [Amazon Web Services](), [Amazon S3]() Tags: [analytics](), [article](), [aws](), [conference](), [data-infrastructure](), [data-lake](), [japan](), [k8s](), [migration](), [performance](), [report](), [s3](), [schema-evolution](), [serverless](), [spark]() ### AI overview A report on the Trino Conference Tokyo 2023, covering sessions about Trino, Starburst products, Iceberg-based log infrastructure, Spark and Trino data infrastructure on Kubernetes, migration from Impala, and serverless Athena analytics. ### Source excerpt The Trino community in Japan held an online event on October 5th, 2023. This article is a summary of the conference aiming to share the presentations and provide an overview. ## Trino for lakehouses, data oceans, and beyond DevFeed: [Trino for lakehouses, data oceans, and beyond]() Original publisher: [Read original article]() Author: Martin Traverso, Cole Bowden Published: 2023-06-22T00:00:00Z Content type: article Language: en Sources: [Trino Blog]() Topics: [data](), [schema-evolution](), [migration](), [tracing](), [OpenTelemetry](), [SQL](), [JSON](), [Redis](), [Python](), [Java](), [GitHub]() Tags: [data](), [github](), [json](), [migration](), [opentelemetry](), [python](), [redis](), [schema-evolution](), [sql](), [tracing]() ### AI overview This recap covers Trino's recent project activity and roadmap presented at Trino Fest 2023. It highlights releases, commits, new maintainers, table functions, configuration and performance improvements, schema evolution, lakehouse migration, OpenTelemetry tracing, Python client improvements, and dbt Cloud support. Planned work includes SQL 2023 enhancements, JSON functions, numeric literals, new or improved connectors, Java 21, and Project Hummingbird. ### Source excerpt Trino Fest 2023 got off to a bang, as Trino co-creator and maintainer Martin Traverso gave an update on all the amazing things that have happened to Trino since Trino Summit last year. He also provided some insight into what's coming down the pipeline for Trino, with a brief look at the project's roadmap. You can watch the recording of the talk if you want to see for yourself, or you can read on for the highlights. ## Future Proofing SQL with Carefully Placed Errors DevFeed: [Future Proofing SQL with Carefully Placed Errors]() Original publisher: [Read original article]() Author: Haki Benita Published: 2022-10-05T21:00:00Z Content type: tutorial Language: en Sources: [Haki Benita]() Topics: [SQL](), [schema-evolution]() Tags: [articles](), [best-practices](), [compatibility](), [errors](), [postgresql](), [sql]() ### AI overview This article explains how to future-proof SQL by deliberately placing errors in queries so that unsupported changes, such as a newly added payment method, are detected instead of silently producing incorrect business results. ### Source excerpt There are many best practices for maintaining backward and forward compatibility in application code, but it's not very commonly mentioned in relation to SQL. SQL is used to produce critical business information for applications and decision-making, so there's no reason it shouldn't benefit from similar practices. In this article, I present a simple way to future-proof SQL. [Next page]()