# parquet

Apache Parquet is an open-source, column-oriented data file format designed for efficient data storage and retrieval.

This is one page of public article previews, not the complete archive. Follow Next page to continue. Summaries are not the original full articles.

## Aderant modernizes analytics data pipeline with Apache Iceberg and AWS Lambda

DevFeed: [Aderant modernizes analytics data pipeline with Apache Iceberg and AWS Lambda](<https://devfeed.tech/articles/aderant-modernizes-analytics-data-pipeline-with-apache-iceberg-and-aws-lambda-141916.md>)

Original publisher: [Read original article](<https://aws.amazon.com/blogs/architecture/aderant-modernizes-analytics-data-pipeline-with-apache-iceberg-and-aws-lambda/>)

Author: Sham Birbahadur

Published: 2026-11-19T00:36:23Z

Content type: article

Language: en

Sources: [AWS Architecture Blog](<https://devfeed.tech/sources/aws-architecture-blog.md>)

Topics: [Analytics Pipeline](<https://devfeed.tech/topics/analytics-pipeline.md>), [Apache Iceberg](<https://devfeed.tech/topics/apache-iceberg.md>), [AWS Glue](<https://devfeed.tech/topics/aws-glue.md>), [event-driven data processing](<https://devfeed.tech/topics/event-driven-data-processing.md>), [streaming data analytics](<https://devfeed.tech/topics/streaming-data-analytics.md>), [Event-Streaming](<https://devfeed.tech/topics/event-streaming.md>), [Apache Spark](<https://devfeed.tech/topics/spark.md>), [AWS Database Migration Service](<https://devfeed.tech/topics/aws-database-migration-service.md>), [schema-evolution](<https://devfeed.tech/topics/schema-evolution.md>), [parquet](<https://devfeed.tech/topics/parquet.md>), [Amazon EventBridge](<https://devfeed.tech/topics/amazon-eventbridge.md>), [CI/CD](<https://devfeed.tech/topics/cicd.md>)

Tags: [acid](<https://devfeed.tech/tags/acid.md>), [advanced-300](<https://devfeed.tech/tags/advanced-300.md>), [aggregation](<https://devfeed.tech/tags/aggregation.md>), [ai](<https://devfeed.tech/tags/ai.md>), [ai-powered](<https://devfeed.tech/tags/ai-powered.md>), [analytics](<https://devfeed.tech/tags/analytics.md>), [apache-iceberg](<https://devfeed.tech/tags/apache-iceberg.md>), [architecture](<https://devfeed.tech/tags/architecture.md>), [aws](<https://devfeed.tech/tags/aws.md>), [aws-glue](<https://devfeed.tech/tags/aws-glue.md>), [aws-lambda](<https://devfeed.tech/tags/aws-lambda.md>), [batch](<https://devfeed.tech/tags/batch.md>), [change-data-capture](<https://devfeed.tech/tags/change-data-capture.md>), [cost](<https://devfeed.tech/tags/cost.md>), [customer-solutions](<https://devfeed.tech/tags/customer-solutions.md>), [data-pipeline](<https://devfeed.tech/tags/data-pipeline.md>), [event-driven](<https://devfeed.tech/tags/event-driven.md>), [faster](<https://devfeed.tech/tags/faster.md>), [iceberg](<https://devfeed.tech/tags/iceberg.md>), [legal](<https://devfeed.tech/tags/legal.md>), [pipeline](<https://devfeed.tech/tags/pipeline.md>), [queries](<https://devfeed.tech/tags/queries.md>)

### AI overview

Aderant replaced Stridyn's 15-minute batch pipeline with an event-driven medallion architecture that processes CDC changes from 700 client environments in single-digit seconds to bronze-layer Apache Iceberg tables. AWS DMS delivers Parquet files to S3; S3 notifications and SQS invoke Lambda for ingestion, while scheduled AWS Glue jobs transform data through silver and gold layers. Incremental Glue jobs handle 77 standard gold tables; 34 complex tables use Iceberg materialized views and snapshot comparisons to send changed rows to native Amazon Redshift tables. Aderant reports 10-26x faster queries on native Redshift tables, under-100-millisecond dashboard queries, and 77 percent lower pipeline compute costs versus streaming alternatives. The account says this approach supports schema changes and concurrent tenant workloads; it also describes a possible future Kinesis path for high-volume tenants.

### Source excerpt

Learn how Aderant modernized the data pipeline behind its Stridyn legal analytics platform with Apache Iceberg, AWS Lambda, and AWS Glue, moving from 15-minute batch cycles to an event-driven medallion architecture that processes change data capture events in single-digit seconds for 700 tenants, with 10-26x faster queries and 77 percent lower cost.

## Orchestrate DuckDB analytics pipelines with Amazon MWAA

DevFeed: [Orchestrate DuckDB analytics pipelines with Amazon MWAA](<https://devfeed.tech/articles/orchestrate-duckdb-analytics-pipelines-with-amazon-mwaa-141881.md>)

Original publisher: [Read original article](<https://aws.amazon.com/blogs/big-data/orchestrate-duckdb-analytics-pipelines-with-amazon-mwaa/>)

Author: Vinod Jayendra

Published: 2026-11-18T20:57:09Z

Content type: tutorial

Language: en

Sources: [AWS Big Data Blog](<https://devfeed.tech/sources/aws-big-data-blog.md>)

Topics: [Data pipelines](<https://devfeed.tech/topics/data-pipelines.md>), [Analytics Pipeline](<https://devfeed.tech/topics/analytics-pipeline.md>), [DuckDB](<https://devfeed.tech/topics/duckdb.md>), [Amazon S3 Tables](<https://devfeed.tech/topics/amazon-s3-tables.md>), [Amazon S3](<https://devfeed.tech/topics/amazon-s3.md>), [Parquet files in Amazon S3](<https://devfeed.tech/topics/parquet-files-in-amazon-s3.md>), [parquet](<https://devfeed.tech/topics/parquet.md>)

Tags: [advanced-300](<https://devfeed.tech/tags/advanced-300.md>), [airflow](<https://devfeed.tech/tags/airflow.md>), [amazon](<https://devfeed.tech/tags/amazon.md>), [amazon-managed-workflows-for-apache-airflow-amazon-mwaa](<https://devfeed.tech/tags/amazon-managed-workflows-for-apache-airflow-amazon-mwaa.md>), [amazon-s3](<https://devfeed.tech/tags/amazon-s3.md>), [amazon-s3-tables](<https://devfeed.tech/tags/amazon-s3-tables.md>), [analytics](<https://devfeed.tech/tags/analytics.md>), [apache](<https://devfeed.tech/tags/apache.md>), [apache-iceberg](<https://devfeed.tech/tags/apache-iceberg.md>), [apache-iceberg-tables](<https://devfeed.tech/tags/apache-iceberg-tables.md>), [data](<https://devfeed.tech/tags/data.md>), [duckdb](<https://devfeed.tech/tags/duckdb.md>), [effective](<https://devfeed.tech/tags/effective.md>), [embedded](<https://devfeed.tech/tags/embedded.md>), [how-to](<https://devfeed.tech/tags/how-to.md>), [iceberg](<https://devfeed.tech/tags/iceberg.md>), [learn-how-to](<https://devfeed.tech/tags/learn-how-to.md>), [model](<https://devfeed.tech/tags/model.md>), [parquet](<https://devfeed.tech/tags/parquet.md>), [pipelines](<https://devfeed.tech/tags/pipelines.md>), [retry](<https://devfeed.tech/tags/retry.md>), [s3](<https://devfeed.tech/tags/s3.md>), [technical-how-to](<https://devfeed.tech/tags/technical-how-to.md>)

### AI overview

Amazon MWAA can orchestrate bounded SQL analytics by running DuckDB inside Airflow worker tasks, without a separate query cluster. The walkthrough reads S3 Parquet files, deduplicates events, uses ASOF LEFT JOINs to apply rates and ownership effective at each event time, and merges results into three Iceberg tables in Amazon S3 Tables. A second task validates counts, uniqueness, and allocation states. On supported MWAA Provisioned environments running Airflow 3.3.1, DuckDB and its providers plus the duckdb_default AWS-aware connection are preinstalled; the execution role supplies temporary credentials. The example's unchanged-input reruns retained 300,000 unique events and duplicate-free summaries, but the excerpt says this does not establish recovery from partial failures or concurrent external writes. Each task shares one worker's CPU, memory, and storage, so limits and concurrency need tuning with representative data. Use Glue or EMR for steps needing horizontal scaling or more capacity than one worker provides. Three table commits are not one transaction, and Iceberg update/delete support can vary by version.

### Source excerpt

Learn how to orchestrate embedded DuckDB analytics pipelines with Amazon MWAA. This post shows how to run DuckDB inside Airflow tasks to read Parquet data from Amazon S3, apply time-effective model rates, and publish Apache Iceberg tables to Amazon S3 Tables, with retry-safe publishing and worker-sizing considerations.

## A modern dilemma: Consume data from a stream or a lake

DevFeed: [A modern dilemma: Consume data from a stream or a lake](<https://devfeed.tech/articles/a-modern-dilemma-consume-data-from-a-stream-or-a-lake-141161.md>)

Original publisher: [Read original article](<https://medium.com/capital-one-tech/a-modern-dilemma-consume-data-from-a-stream-or-a-lake-296bbb354094?source=rss----3db3a67cb648---4>)

Author: Capital One Tech

Published: 2026-11-18T17:35:14Z

Content type: article

Language: en

Sources: [Capital One Tech](<https://devfeed.tech/sources/capital-one-tech.md>)

Topics: [software-architecture](<https://devfeed.tech/topics/software-architecture.md>), [stream-processing](<https://devfeed.tech/topics/stream-processing.md>), [Streaming](<https://devfeed.tech/topics/streaming.md>), [etl](<https://devfeed.tech/topics/etl.md>), [deltalake](<https://devfeed.tech/topics/deltalake.md>), [Apache Kafka](<https://devfeed.tech/topics/apache-kafka.md>), [Apache Flink](<https://devfeed.tech/topics/apache-flink.md>), [parquet](<https://devfeed.tech/topics/parquet.md>), [Data Quality](<https://devfeed.tech/topics/data-quality.md>)

Tags: [ai](<https://devfeed.tech/tags/ai.md>), [apis](<https://devfeed.tech/tags/apis.md>), [architecture](<https://devfeed.tech/tags/architecture.md>), [batch](<https://devfeed.tech/tags/batch.md>), [big-data](<https://devfeed.tech/tags/big-data.md>), [business-processes](<https://devfeed.tech/tags/business-processes.md>), [cloud](<https://devfeed.tech/tags/cloud.md>), [cloud-computing](<https://devfeed.tech/tags/cloud-computing.md>), [data](<https://devfeed.tech/tags/data.md>), [data-engineering](<https://devfeed.tech/tags/data-engineering.md>), [data-lakes](<https://devfeed.tech/tags/data-lakes.md>), [delta-lake](<https://devfeed.tech/tags/delta-lake.md>), [enrichment](<https://devfeed.tech/tags/enrichment.md>), [etl](<https://devfeed.tech/tags/etl.md>), [event](<https://devfeed.tech/tags/event.md>), [flink](<https://devfeed.tech/tags/flink.md>), [kafka](<https://devfeed.tech/tags/kafka.md>), [latency](<https://devfeed.tech/tags/latency.md>), [legacy](<https://devfeed.tech/tags/legacy.md>), [log](<https://devfeed.tech/tags/log.md>), [mapping](<https://devfeed.tech/tags/mapping.md>), [model-context-protocol](<https://devfeed.tech/tags/model-context-protocol.md>), [software-architecture](<https://devfeed.tech/tags/software-architecture.md>), [software-engineering](<https://devfeed.tech/tags/software-engineering.md>), [stream](<https://devfeed.tech/tags/stream.md>)

### AI overview

Choosing a data consumption pattern early can avoid costly architectural changes later. APIs support synchronous, record-by-record requests and paginated multirecord results, making them suitable for low-latency needs; Model Context Protocol can provide similar access for agents and is often an API wrapper. Event streams, commonly built with Kafka, Flink or Spark Streaming, suit sub-second latency and enable per-event calculations, transformations and enrichment. The excerpt also identifies Delta Change Data Feed (CDF) as a record-by-record pattern and traditional batch ETL as a multirecord pattern, but ends before explaining CDF's mechanics or comparing these approaches further.

### Source excerpt

Data is a critical component of software architecture, providing the foundation for business processes, including those powered by artificial intelligence (AI). Quality, volume, latency, consistency, ordering and duplicates are essential considerations in software design. Modern cloud architectures consume data in various ways, such as through application programming interfaces (APIs), streams and data lakes. You can consume data at the individual event or record level as it arrives or in larger batches. Legacy architectures often process data in scheduled batches, forcing surrounding architecture and business processes to depend on that approach. Getting the data consumption approach right up front is vital, because rearchitecting later is costly. General patterns to consume data There are several general data consumption patterns. These include APIs (record-by-record or multirecord results), event stream consumption (record-by-record), Delta Change Data Feed (CDF) consumption (record-by-record) or traditional batch ETL-extract, transform, load (multirecord results). APIs are ideal for low-latency, record-by-record requirements. They also support multirecord results, typically through pagination. APIs usually follow a synchronous approach, where the client makes a call and waits for a response before moving to the next step. Model Context Protocol is an industry specification for agents to consume data and can support record-by-record or batch results via pagination, similar to APIs. They are often a wrapper on top of APIs. The event stream pattern works well for sub-second latency requirements and typically runs on a tech stack like Kafka, Flink or Spark Streaming. These technologies excel at record-level calculations (like on-the-fly calculation), transformations (like column mapping) or enrichment (like adding data to an existing event). Delta Lake is often used in data lakes and serves as a transaction log to standard Apache Parquet data files (source). CDF is

## Getting started with Apache Polaris Catalog in Fivetran's Managed Data Lake Service | Blog | Fivetran

DevFeed: [Getting started with Apache Polaris Catalog in Fivetran's Managed Data Lake Service | Blog | Fivetran](<https://devfeed.tech/articles/getting-started-with-apache-polaris-catalog-in-fivetran-s-managed-data-lake-service-blog-fivetran-140235.md>)

Original publisher: [Read original article](<https://wf.fivetran.com/blog/getting-started-with-apache-polaris-catalog-in-fivetrans-managed-data-lake-service>)

Author: Andrew Madson

Published: 2026-11-18T04:00:44Z

Content type: tutorial

Language: en

Sources: [Fivetran Blog](<https://devfeed.tech/sources/fivetran-blog.md>)

Topics: [data-engineering](<https://devfeed.tech/topics/data-engineering.md>), [data lake](<https://devfeed.tech/topics/data-lake.md>), [Apache Spark](<https://devfeed.tech/topics/spark.md>), [parquet](<https://devfeed.tech/topics/parquet.md>), [schema-evolution](<https://devfeed.tech/topics/schema-evolution.md>), [database-consistency](<https://devfeed.tech/topics/database-consistency.md>)

Tags: [acid](<https://devfeed.tech/tags/acid.md>), [amazon-s3](<https://devfeed.tech/tags/amazon-s3.md>), [apache](<https://devfeed.tech/tags/apache.md>), [apache-iceberg](<https://devfeed.tech/tags/apache-iceberg.md>), [apache-polaris](<https://devfeed.tech/tags/apache-polaris.md>), [best-practices](<https://devfeed.tech/tags/best-practices.md>), [catalog](<https://devfeed.tech/tags/catalog.md>), [configure](<https://devfeed.tech/tags/configure.md>), [data-engineer](<https://devfeed.tech/tags/data-engineer.md>), [data-lake](<https://devfeed.tech/tags/data-lake.md>), [engines](<https://devfeed.tech/tags/engines.md>), [fivetran](<https://devfeed.tech/tags/fivetran.md>), [getting-started](<https://devfeed.tech/tags/getting-started.md>), [how-to](<https://devfeed.tech/tags/how-to.md>), [managed](<https://devfeed.tech/tags/managed.md>), [matters](<https://devfeed.tech/tags/matters.md>), [popular](<https://devfeed.tech/tags/popular.md>), [practices](<https://devfeed.tech/tags/practices.md>), [query](<https://devfeed.tech/tags/query.md>), [service](<https://devfeed.tech/tags/service.md>)

### AI overview

Fivetran's Managed Data Lake Service uses Apache Polaris as its default Iceberg REST catalog, maintaining table metadata for Parquet data in supported cloud destinations. Query tables through the catalog so engines read committed snapshots, apply deletes, follow schema changes, and avoid hard-coded file paths; Fivetran alone can modify catalog metadata. The article gives OAuth2 setup examples for Spark, DuckDB, Snowflake, and Trino/Starburst. Requirements and capabilities differ: Spark needs Iceberg 1.9.0+ with Spark 3.4 or 3.5; DuckDB's REST support is experimental, supports SELECT and time travel, and is currently limited to S3-compatible storage; Snowflake external-table reads are generally available while writes are in preview, with configurable refresh; Trino/Starburst use REST catalog properties and vended credentials. Protect metadata files and configure refresh and snapshot retention for freshness and time-travel needs.

### Source excerpt

Why the catalog matters, how to configure it with popular query engines, and the essential best practices every data engineer should follow.

## How Mycom optimized its SaaS data fabric with Amazon S3 Access Points and Amazon FSx for NetApp ONTAP

DevFeed: [How Mycom optimized its SaaS data fabric with Amazon S3 Access Points and Amazon FSx for NetApp ONTAP](<https://devfeed.tech/articles/how-mycom-optimized-its-saas-data-fabric-with-amazon-s3-access-points-and-amazon-fsx-for-netapp-ontap-140642.md>)

Original publisher: [Read original article](<https://aws.amazon.com/blogs/storage/how-mycom-optimized-its-saas-data-fabric-with-amazon-s3-access-points-and-amazon-fsx-for-netapp-ontap/>)

Author: Zhiying Lee

Published: 2026-11-17T23:05:19Z

Content type: article

Language: en

Sources: [AWS Storage Blog](<https://devfeed.tech/sources/aws-storage-blog.md>)

Topics: [Amazon S3](<https://devfeed.tech/topics/amazon-s3.md>), [data-engineering](<https://devfeed.tech/topics/data-engineering.md>), [data lake](<https://devfeed.tech/topics/data-lake.md>), [parquet](<https://devfeed.tech/topics/parquet.md>), [Authorization](<https://devfeed.tech/topics/authorization.md>)

Tags: [access-points](<https://devfeed.tech/tags/access-points.md>), [ai-agents](<https://devfeed.tech/tags/ai-agents.md>), [ai-and-machine-learning](<https://devfeed.tech/tags/ai-and-machine-learning.md>), [ai-applications](<https://devfeed.tech/tags/ai-applications.md>), [amazon-athena](<https://devfeed.tech/tags/amazon-athena.md>), [amazon-eks](<https://devfeed.tech/tags/amazon-eks.md>), [amazon-elastic-kubernetes-service-amazon-eks](<https://devfeed.tech/tags/amazon-elastic-kubernetes-service-amazon-eks.md>), [amazon-fsx-for-netapp-ontap](<https://devfeed.tech/tags/amazon-fsx-for-netapp-ontap.md>), [amazon-s3](<https://devfeed.tech/tags/amazon-s3.md>), [amazon-simple-storage-service-amazon-s3](<https://devfeed.tech/tags/amazon-simple-storage-service-amazon-s3.md>), [amazon-simple-storage-service-s3](<https://devfeed.tech/tags/amazon-simple-storage-service-s3.md>), [analytics](<https://devfeed.tech/tags/analytics.md>), [analytics-pipeline](<https://devfeed.tech/tags/analytics-pipeline.md>), [api](<https://devfeed.tech/tags/api.md>), [aws-glue](<https://devfeed.tech/tags/aws-glue.md>), [carbon](<https://devfeed.tech/tags/carbon.md>), [cost](<https://devfeed.tech/tags/cost.md>), [customer-solutions](<https://devfeed.tech/tags/customer-solutions.md>), [data-fabric](<https://devfeed.tech/tags/data-fabric.md>), [intermediate-200](<https://devfeed.tech/tags/intermediate-200.md>), [nfs](<https://devfeed.tech/tags/nfs.md>), [saas](<https://devfeed.tech/tags/saas.md>), [telco](<https://devfeed.tech/tags/telco.md>)

### AI overview

Mycom replaced a duplicate S3 analytics copy with Amazon S3 Access Points attached to its Amazon FSx for NetApp ONTAP volume. Operational services continue to use NFS files, while AWS Glue, Athena, and other consumers access the same data through the S3 API, avoiding copy synchronization and application changes. Mycom reported a 20% reduction in analytics storage footprint, with associated cost and carbon reductions, and tighter consistency between operational and analytics views. The design uses existing ONTAP file permissions alongside scoped IAM access point policies; access points are VPC-restricted. Deployment requires two provisioning passes, and existing bucket data is migrated by streaming it to an NFS-mounted volume, followed by a brief producer write pause for the final sync. A configuration switch lets tenants use either the access point or native bucket. A single-AZ volume is an availability, cost, and sustainability tradeoff evaluated per customer and dataset.

### Source excerpt

Mycom collapsed two copies of its telco assurance data (one file store, one object store) into a single copy on Amazon FSx for NetApp ONTAP, served through both NFS and the Amazon S3 API using S3 Access Points. Learn how this removed duplicate storage, reduced cost and carbon, and kept analytics and operational apps consistent.

## Migrating from Apache Hudi-based data lake to Apache Iceberg-based data lake

DevFeed: [Migrating from Apache Hudi-based data lake to Apache Iceberg-based data lake](<https://devfeed.tech/articles/migrating-from-apache-hudi-based-data-lake-to-apache-iceberg-based-data-lake-130716.md>)

Original publisher: [Read original article](<https://aws.amazon.com/blogs/big-data/migrating-from-apache-hudi-based-data-lake-to-apache-iceberg-based-data-lake/>)

Author: Nitin Kumar

Published: 2026-11-12T15:50:48Z

Content type: tutorial

Language: en

Sources: [AWS Big Data Blog](<https://devfeed.tech/sources/aws-big-data-blog.md>)

Topics: [Apache Iceberg](<https://devfeed.tech/topics/apache-iceberg.md>), [AWS Glue](<https://devfeed.tech/topics/aws-glue.md>), [data lake](<https://devfeed.tech/topics/data-lake.md>), [Amazon S3 Tables](<https://devfeed.tech/topics/amazon-s3-tables.md>), [parquet](<https://devfeed.tech/topics/parquet.md>), [Apache Spark](<https://devfeed.tech/topics/spark.md>), [Parquet files in Amazon S3](<https://devfeed.tech/topics/parquet-files-in-amazon-s3.md>), [deltalake](<https://devfeed.tech/topics/deltalake.md>)

Tags: [advanced-300](<https://devfeed.tech/tags/advanced-300.md>), [amazon](<https://devfeed.tech/tags/amazon.md>), [amazon-athena](<https://devfeed.tech/tags/amazon-athena.md>), [amazon-s3](<https://devfeed.tech/tags/amazon-s3.md>), [amazon-s3-tables](<https://devfeed.tech/tags/amazon-s3-tables.md>), [apache](<https://devfeed.tech/tags/apache.md>), [apache-hudi](<https://devfeed.tech/tags/apache-hudi.md>), [apache-iceberg](<https://devfeed.tech/tags/apache-iceberg.md>), [athena](<https://devfeed.tech/tags/athena.md>), [aws](<https://devfeed.tech/tags/aws.md>), [aws-glue](<https://devfeed.tech/tags/aws-glue.md>), [based](<https://devfeed.tech/tags/based.md>), [data](<https://devfeed.tech/tags/data.md>), [data-lake](<https://devfeed.tech/tags/data-lake.md>), [data-lakes](<https://devfeed.tech/tags/data-lakes.md>), [files](<https://devfeed.tech/tags/files.md>), [full](<https://devfeed.tech/tags/full.md>), [how-to](<https://devfeed.tech/tags/how-to.md>), [hudi](<https://devfeed.tech/tags/hudi.md>), [iceberg](<https://devfeed.tech/tags/iceberg.md>), [insert](<https://devfeed.tech/tags/insert.md>), [technical-how-to](<https://devfeed.tech/tags/technical-how-to.md>)

### AI overview

The excerpt presents three Hudi-to-Iceberg migration paths on AWS. In-place migration reuses Parquet files, saving rewrite time and storage but retaining the source schema, partitioning, and file layout. Iceberg's `add_files` suits a one-time conversion and requires Hudi cleanup first; for MoR, compact logs and clean old versions to avoid missing updates or duplicate rows. Apache XTable translates Hudi metadata and can sync ongoing writes, but MoR logs must be compacted before each conversion. Both approaches share data files, so avoid destructive Iceberg maintenance until cutover; `add_files` and `snapshot` also lack column statistics in Iceberg 1.9.0-1.10.0, with the issue fixed in 1.10.1. Full migration with CTAS or CREATE plus INSERT rewrites data, allowing schema and partition changes, including hidden partitioning, at higher compute, time, and storage cost. For large tables, partitioned INSERT batches support parallelism and resumability. The excerpt demonstrates AWS Glue notebooks, general S3 buckets, and Amazon S3 Tables, and recommends validating against Hudi before cutover.

### Source excerpt

Learn how to migrate transactional data lakes from Apache Hudi to Apache Iceberg on AWS. This post walks through in-place migration with the Iceberg add_files procedure and Apache XTable, plus full table migration with CTAS and INSERT, using AWS Glue notebooks and Amazon S3 Tables, then validates the results in Amazon Athena.

## Hybrid Product Search Without the Copy: Lexical, Semantic and Filtered Retrieval Served Directly from a Data Lake

DevFeed: [Hybrid Product Search Without the Copy: Lexical, Semantic and Filtered Retrieval Served Directly from a Data Lake](<https://devfeed.tech/articles/hybrid-product-search-without-the-copy-lexical-semantic-and-filtered-retrieval-served-directly-from-a-data-lake-125310.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.11031>)

Author: Raghvendra Yadav, Xiang Fu, Chinmay Soman

Published: 2026-11-10T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Databases](<https://devfeed.tech/sources/arxiv-databases.md>), [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [hybrid-search](<https://devfeed.tech/topics/hybrid-search.md>), [data lake](<https://devfeed.tech/topics/data-lake.md>), [parquet](<https://devfeed.tech/topics/parquet.md>), [search-engines](<https://devfeed.tech/topics/search-engines.md>), [streaming data analytics](<https://devfeed.tech/topics/streaming-data-analytics.md>)

Tags: [apache](<https://devfeed.tech/tags/apache.md>), [apache-iceberg](<https://devfeed.tech/tags/apache-iceberg.md>), [benchmark](<https://devfeed.tech/tags/benchmark.md>), [bm25](<https://devfeed.tech/tags/bm25.md>), [broker](<https://devfeed.tech/tags/broker.md>), [catalog](<https://devfeed.tech/tags/catalog.md>), [commit](<https://devfeed.tech/tags/commit.md>), [cs-db](<https://devfeed.tech/tags/cs-db.md>), [cs-ir](<https://devfeed.tech/tags/cs-ir.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [data-lake](<https://devfeed.tech/tags/data-lake.md>), [database](<https://devfeed.tech/tags/database.md>), [e-commerce](<https://devfeed.tech/tags/e-commerce.md>), [engine](<https://devfeed.tech/tags/engine.md>), [files](<https://devfeed.tech/tags/files.md>), [fusion](<https://devfeed.tech/tags/fusion.md>), [hybrid](<https://devfeed.tech/tags/hybrid.md>), [hybrid-search](<https://devfeed.tech/tags/hybrid-search.md>), [iceberg](<https://devfeed.tech/tags/iceberg.md>), [product-search](<https://devfeed.tech/tags/product-search.md>), [retrieval](<https://devfeed.tech/tags/retrieval.md>), [search](<https://devfeed.tech/tags/search.md>), [semantic](<https://devfeed.tech/tags/semantic.md>)

### AI overview

The paper describes serving hybrid product search directly from Apache Iceberg tables, using Apache Pinot with Lucene BM25, corpus-wide term statistics, vector retrieval, and SQL reciprocal rank fusion. A stream-fed real-time table can be merged with the lake table. On Amazon ESCI, the approach reports nDCG@10 of .361, matching exact retrieval; registered files become queryable in 149 seconds, with commits searchable in 3.8 seconds. On a 277-million-product, 0.8-TB brand-sorted lake, one 16-vCPU node reports 6.5-9 ms median latency for brand-scoped vector, BM25, and client-fused hybrid searches, and under 180 ms p99 at 1,100 vector or 635 BM25 searches per second. It adds search indexes without a catalog copy, sync pipeline, or second schema. These are reported results for the described datasets and setup.

### Source excerpt

arXiv:2611.11031v1 Announce Type: new Abstract: E-commerce catalogs increasingly live in lakehouse tables: Apache Iceberg metadata over Parquet files in object storage. Product search, however, still runs on copies of that catalog, in a lexical engine and a vector database, each with its own ingestion pipeline, schema, staleness window and bill. We show that search applications can be hosted directly on the data lake, with the lakehouse table as the single system of record. We extend Apache Pinot, a real-time OLAP engine that already serves Iceberg tables in place, with BM25 relevance scoring over its Lucene text index, corpus-wide term statistics so that scores are comparable across files, and reciprocal rank fusion in plain SQL; a real-time table fed by a stream can sit behind the same table, merged by the broker. On the Amazon ESCI benchmark (1.2M US products, 22,458 test queries), hybrid search served from the lake matches exact retrieval (nDCG@10 .361). The lake table is queryable 149 s after its files are registered, and a lake commit is searchable in 3.8 s, close to indexing directly into a search engine. On a brand-sorted lake of 277M products (0.8 TB), one 16-vCPU node answers brand-scoped vector, BM25 and client-fused hybrid searches in 6.5-9 ms (p50), sustains 1,100 vector and 635 BM25 searches per second at under 180 ms p99, and makes a commit searchable in 4.3 s. Search adds only its indexes: no copy of the catalog, no sync pipeline and no second schema, which cuts storage, rebuild time and the operational work of keeping copies consistent.

## Comparing Write, Read, and Query Performance of General-Purpose and Columnar Data Serialization Formats at Increasing Scale: An Empirical Benchmark Study with Statistical Validation on Synthetic and Real-World Data

DevFeed: [Comparing Write, Read, and Query Performance of General-Purpose and Columnar Data Serialization Formats at Increasing Scale: An Empirical Benchmark Study with Statistical Validation on Synthetic and Real-World Data](<https://devfeed.tech/articles/comparing-write-read-and-query-performance-of-general-purpose-and-columnar-data-serialization-formats-at-increasing-scale-an-empirical-benchmark-study-with-statistical-va-125664.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.10398>)

Author: Asra Fathima

Published: 2026-11-10T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Databases](<https://devfeed.tech/sources/arxiv-databases.md>), [arXiv: Software Engineering](<https://devfeed.tech/sources/arxiv-software-engineering.md>)

Topics: [Data serialization](<https://devfeed.tech/topics/data-serialization.md>), [Benchmark](<https://devfeed.tech/topics/benchmark.md>), [parquet](<https://devfeed.tech/topics/parquet.md>), [Serialization protocols](<https://devfeed.tech/topics/serialization-protocols.md>), [IO](<https://devfeed.tech/topics/io.md>)

Tags: [benchmark](<https://devfeed.tech/tags/benchmark.md>), [columnar](<https://devfeed.tech/tags/columnar.md>), [cs-db](<https://devfeed.tech/tags/cs-db.md>), [cs-pf](<https://devfeed.tech/tags/cs-pf.md>), [cs-se](<https://devfeed.tech/tags/cs-se.md>), [data-serialization](<https://devfeed.tech/tags/data-serialization.md>), [performance](<https://devfeed.tech/tags/performance.md>), [query](<https://devfeed.tech/tags/query.md>), [scale](<https://devfeed.tech/tags/scale.md>)

### AI overview

Across four dataset scales and two different hosts, the benchmark found that selective queries requiring full deserialization were one to three orders of magnitude more expensive than approaches that avoid it. Indexed SQLite was significantly fastest for selective access at every scale; Parquet's pyarrow pushdown was significantly cheaper than naive materialization. Pickle was significantly faster to write than Parquet on both hosts, a result supported by 20 trials rather than the earlier five-trial pilot. A 1.47-million-row public fraud dataset reproduced the query-performance gap and indexed SQLite advantage. The study compared write time, storage size, read time, and query time, using statistical tests to assess differences.

### Source excerpt

arXiv:2611.10398v1 Announce Type: new Abstract: This study presents an empirical benchmark comparing nine format and access-method combinations across six data serialization technologies: CSV (including a column-projected variant), gzip CSV, JSON Lines, Python Pickle, SQLite (unindexed and indexed), and Apache Parquet (naive and pyarrow pushdown). A synthetic dataset was generated at four scales (50,000 to 3,000,000 rows), with write time, storage size, read time, and query time measured across twenty trials on two independent hosts, a native 24 GB Apple Silicon machine and a memory-constrained, single-core 4 GB Linux virtual machine. Mean, standard deviation, and 95 percent confidence intervals are reported for each host. The most robust finding, stable across every scale and both hosts, is that flat formats requiring full deserialization before filtering are one to three orders of magnitude more expensive for selective queries than any approach that avoids it. Welch t-tests, corroborated by Mann-Whitney U tests, show SQLite (indexed) is significantly fastest for selective access at every scale on both hosts, with the pushdown path of Parquet significantly cheaper than naive materialization. With the increased trial count, the write-time variance of Pickle narrowed enough to show it is significantly faster than Parquet at write time on both hosts, not the statistical tie an earlier five-trial pilot suggested. These findings were confirmed on a real, public 1.47-million-row e-commerce fraud dataset, which reproduced the same order-of-magnitude gap between flat and non-flat formats and the same SQLite (indexed) advantage. These results offer statistically grounded guidance for format selection in data engineering pipelines.

## Ducks in a Column

DevFeed: [Ducks in a Column](<https://devfeed.tech/articles/ducks-in-a-column-127523.md>)

Original publisher: [Read original article](<https://duckdb.org/2026/11/10/ducks-in-a-column.html>)

Author: The DuckDB team

Published: 2026-11-10T00:00:00Z

Content type: article

Language: en

Sources: [DuckDB](<https://devfeed.tech/sources/duckdb.md>)

Topics: [DuckDB](<https://devfeed.tech/topics/duckdb.md>), [parquet](<https://devfeed.tech/topics/parquet.md>), [database-optimization](<https://devfeed.tech/topics/database-optimization.md>), [table-partitioning](<https://devfeed.tech/topics/table-partitioning.md>), [CSV](<https://devfeed.tech/topics/csv.md>), [data lake](<https://devfeed.tech/topics/data-lake.md>)

Tags: [aggregation](<https://devfeed.tech/tags/aggregation.md>), [columnar](<https://devfeed.tech/tags/columnar.md>), [duckdb](<https://devfeed.tech/tags/duckdb.md>), [parquet](<https://devfeed.tech/tags/parquet.md>), [querying](<https://devfeed.tech/tags/querying.md>), [sql](<https://devfeed.tech/tags/sql.md>), [using-duckdb](<https://devfeed.tech/tags/using-duckdb.md>), [window-functions](<https://devfeed.tech/tags/window-functions.md>), [writing](<https://devfeed.tech/tags/writing.md>)

### AI overview

DuckDB's columnar storage lets analytical queries read only named columns, and Parquet adds row-group statistics that can skip data excluded by filters. The article demonstrates querying CSV, Parquet, remote files and DataFrames in place; using EXCLUDE, REPLACE and COLUMNS() for column-level SQL; and simplifying analysis with GROUP BY ALL, FILTER, arg_max, PIVOT/UNPIVOT, window functions, QUALIFY and ASOF JOIN. For faster later reads, it recommends partitioning on commonly filtered columns, sorting before writing Parquet, choosing narrow types, and checking EXPLAIN ANALYZE to see which files and columns were scanned. It also notes that approximate aggregates can use less memory than exact counterparts, while giving a small-table example where an approximate distinct count differs from the exact count.

### Source excerpt

DuckDB is fast among other reasons because it stores data by column, so a query reads only the columns it names. This post covers DuckDB features that take advantage of that design choice: querying files in place, column-level SQL, aggregation shortcuts, window functions, ASOF joins, and writing well-organized Parquet.

## Parquet Bloom Filters in Apache Iceberg: Configuration, Selectivity, and Row-Group Skipping

DevFeed: [Parquet Bloom Filters in Apache Iceberg: Configuration, Selectivity, and Row-Group Skipping](<https://devfeed.tech/articles/parquet-bloom-filters-in-apache-iceberg-configuration-selectivity-and-row-group-skipping-123565.md>)

Original publisher: [Read original article](<https://www.dremio.com/blog/parquet-bloom-filters-in-apache-iceberg-configuration-selectivity-and-row-group-skipping/>)

Author: Alex Merced

Published: 2026-11-09T13:42:04Z

Content type: article

Language: en

Sources: [Dremio](<https://devfeed.tech/sources/dremio.md>)

Topics: [Apache Iceberg](<https://devfeed.tech/topics/apache-iceberg.md>), [parquet](<https://devfeed.tech/topics/parquet.md>), [configuration](<https://devfeed.tech/topics/configuration.md>)

Tags: [alerts](<https://devfeed.tech/tags/alerts.md>), [apache](<https://devfeed.tech/tags/apache.md>), [apache-iceberg](<https://devfeed.tech/tags/apache-iceberg.md>), [benchmark](<https://devfeed.tech/tags/benchmark.md>), [benchmarking](<https://devfeed.tech/tags/benchmarking.md>), [bloom](<https://devfeed.tech/tags/bloom.md>), [bloom-filter](<https://devfeed.tech/tags/bloom-filter.md>), [bloom-filters](<https://devfeed.tech/tags/bloom-filters.md>), [cardinality](<https://devfeed.tech/tags/cardinality.md>), [clustering](<https://devfeed.tech/tags/clustering.md>), [column](<https://devfeed.tech/tags/column.md>), [compaction](<https://devfeed.tech/tags/compaction.md>), [compatibility](<https://devfeed.tech/tags/compatibility.md>), [configuration](<https://devfeed.tech/tags/configuration.md>), [configure](<https://devfeed.tech/tags/configure.md>), [data-lakehouse](<https://devfeed.tech/tags/data-lakehouse.md>), [dremio-blog-open-data-insights](<https://devfeed.tech/tags/dremio-blog-open-data-insights.md>), [file](<https://devfeed.tech/tags/file.md>), [iceberg](<https://devfeed.tech/tags/iceberg.md>), [memory](<https://devfeed.tech/tags/memory.md>), [parquet](<https://devfeed.tech/tags/parquet.md>), [size](<https://devfeed.tech/tags/size.md>)

### AI overview

Parquet Bloom filters let Iceberg readers skip a row group only when its filter proves a searched value is absent; a possible match still requires reading the group. They are most useful for frequent, selective equality predicates, while range predicates generally gain little. Iceberg can configure filters at table and column level, but exact properties depend on the Iceberg version and writer, and reader or planner support varies by engine. Filters are created at write time, so existing files need rewriting to gain them, and rewrites can leave mixed coverage. To evaluate them, test a representative sample against a no-filter baseline: measure row groups skipped, bytes scanned, query performance, write CPU and memory, and file-size growth. Broader use can add write cost, storage overhead, and reader or coordinator memory pressure without meaningful savings.

### Source excerpt

Configure Parquet Bloom filters for Iceberg columns, choose selective predicates, measure row-group skipping, and avoid memory and file-size waste. The post Parquet Bloom Filters in Apache Iceberg: Configuration, Selectivity, and Row-Group Skipping appeared first on Dremio.

## Oracle to S3 as Parquet: Export Tables and Replicate Changes with CDC

DevFeed: [Oracle to S3 as Parquet: Export Tables and Replicate Changes with CDC](<https://devfeed.tech/articles/oracle-to-s3-as-parquet-export-tables-and-replicate-changes-with-cdc-121706.md>)

Original publisher: [Read original article](<https://estuary.dev/blog/oracle-to-s3>)

Author: Seán Whelan

Published: 2026-11-06T12:27:12Z

Content type: article

Language: en

Sources: [Estuary](<https://devfeed.tech/sources/estuary.md>)

Topics: [streaming data to S3 with Parquet](<https://devfeed.tech/topics/streaming-data-to-s3-with-parquet.md>), [parquet](<https://devfeed.tech/topics/parquet.md>), [Parquet files in Amazon S3](<https://devfeed.tech/topics/parquet-files-in-amazon-s3.md>), [Oracle Database](<https://devfeed.tech/topics/oracle-database.md>), [AWS Glue](<https://devfeed.tech/topics/aws-glue.md>), [Debezium Connector](<https://devfeed.tech/topics/debezium-connector.md>)

Tags: [aws](<https://devfeed.tech/tags/aws.md>), [aws-glue](<https://devfeed.tech/tags/aws-glue.md>), [cdc](<https://devfeed.tech/tags/cdc.md>), [changes](<https://devfeed.tech/tags/changes.md>), [compare](<https://devfeed.tech/tags/compare.md>), [dms](<https://devfeed.tech/tags/dms.md>), [export](<https://devfeed.tech/tags/export.md>), [goldengate](<https://devfeed.tech/tags/goldengate.md>), [managed](<https://devfeed.tech/tags/managed.md>), [oracle](<https://devfeed.tech/tags/oracle.md>), [parquet](<https://devfeed.tech/tags/parquet.md>), [pipeline](<https://devfeed.tech/tags/pipeline.md>), [replicate](<https://devfeed.tech/tags/replicate.md>), [s3](<https://devfeed.tech/tags/s3.md>), [tables](<https://devfeed.tech/tags/tables.md>)

### AI overview

Oracle-to-S3 Parquet choices depend on whether you need change capture: DMS and log-based CDC tools read redo and capture deletes, while exports and scheduled query jobs see surviving rows only. All approaches must handle Oracle type mapping and immutable Parquet files, where updates and deletes become new rows; current-state queries therefore deduplicate by key and filter the latest delete. The excerpt details DMS setup, including archived redo and supplemental logging, Parquet and timestamp settings, bounded LOB sizing, and file batching. It also describes Estuary's LogMiner capture and S3 materialization: backfill runs alongside redo mining, changes append on a configurable upload interval, and deletes and key changes are represented as rows. Estuary's stated tradeoffs include skipped LOB and several other column types, string or numeric mappings that may need configuration, no hard-delete or truncate handling, and retained historical files. Alternatives include Autonomous Database exports for simple full-query exports, and Spark or Glue for custom transforms; those query-based approaches do not capture deletes.

### Source excerpt

Move Oracle to S3 as Parquet with AWS DMS, AWS Glue, GoldenGate or a managed CDC pipeline. Compare how each handles NUMBER, DATE, LOBs and deletes.

## S3 to Kafka: How to Turn New S3 Files into Kafka Messages

DevFeed: [S3 to Kafka: How to Turn New S3 Files into Kafka Messages](<https://devfeed.tech/articles/s3-to-kafka-how-to-turn-new-s3-files-into-kafka-messages-121707.md>)

Original publisher: [Read original article](<https://estuary.dev/blog/s3-to-kafka>)

Author: Seán Whelan

Published: 2026-11-06T12:27:02Z

Content type: article

Language: en

Sources: [Estuary](<https://devfeed.tech/sources/estuary.md>)

Topics: [Apache Kafka](<https://devfeed.tech/topics/apache-kafka.md>), [Confluent Cloud](<https://devfeed.tech/topics/confluent-cloud.md>), [Amazon MSK](<https://devfeed.tech/topics/amazon-msk.md>), [parquet](<https://devfeed.tech/topics/parquet.md>)

Tags: [aws](<https://devfeed.tech/tags/aws.md>), [aws-lambda](<https://devfeed.tech/tags/aws-lambda.md>), [compare](<https://devfeed.tech/tags/compare.md>), [files](<https://devfeed.tech/tags/files.md>), [how-to](<https://devfeed.tech/tags/how-to.md>), [kafka](<https://devfeed.tech/tags/kafka.md>), [managed](<https://devfeed.tech/tags/managed.md>), [messages](<https://devfeed.tech/tags/messages.md>), [ordering](<https://devfeed.tech/tags/ordering.md>), [pipeline](<https://devfeed.tech/tags/pipeline.md>), [s3](<https://devfeed.tech/tags/s3.md>), [setup](<https://devfeed.tech/tags/setup.md>), [source](<https://devfeed.tech/tags/source.md>), [spark](<https://devfeed.tech/tags/spark.md>)

### AI overview

S3 files can become Kafka messages through Kafka Connect, S3 event notifications with Lambda, Spark or Flink, or a managed pipeline. The main differences are how each discovers and tracks files, supported parsing and transformation, and setup; most approaches deliver at least once, so consumers may need to handle duplicates. Listing tools can miss late-arriving keys when they track only the last key, while re-reading modified files can produce duplicates. The guide details S3 and Kafka permissions, format and message-size constraints, and operational tradeoffs. Estuary's described route lists a bucket prefix, parses new files into a durable collection, then materializes records to Kafka; it runs every five minutes by default, delivers at least once, and has specific Parquet size, field truncation, schema-subject, authentication, and deletion limitations. Event-triggered Lambda is positioned for a modest number of uploads needing quick reaction, and Spark or Flink for files requiring transformation.

### Source excerpt

Move S3 data into Kafka with an S3 source connector, AWS Lambda, Spark or a managed pipeline. Compare file ordering, formats, duplicates and setup.

## Salesforce to S3: Export Objects as Parquet and Capture Changes

DevFeed: [Salesforce to S3: Export Objects as Parquet and Capture Changes](<https://devfeed.tech/articles/salesforce-to-s3-export-objects-as-parquet-and-capture-changes-121709.md>)

Original publisher: [Read original article](<https://estuary.dev/blog/salesforce-to-s3>)

Author: Seán Whelan

Published: 2026-11-06T12:26:46Z

Content type: tutorial

Language: en

Sources: [Estuary](<https://devfeed.tech/sources/estuary.md>)

Topics: [Parquet files in Amazon S3](<https://devfeed.tech/topics/parquet-files-in-amazon-s3.md>), [parquet](<https://devfeed.tech/topics/parquet.md>), [Amazon S3](<https://devfeed.tech/topics/amazon-s3.md>), [AWS Glue](<https://devfeed.tech/topics/aws-glue.md>)

Tags: [17](<https://devfeed.tech/tags/17.md>), [2](<https://devfeed.tech/tags/2.md>), [3](<https://devfeed.tech/tags/3.md>), [about](<https://devfeed.tech/tags/about.md>), [amazon](<https://devfeed.tech/tags/amazon.md>), [aws](<https://devfeed.tech/tags/aws.md>), [aws-glue](<https://devfeed.tech/tags/aws-glue.md>), [capture](<https://devfeed.tech/tags/capture.md>), [changes](<https://devfeed.tech/tags/changes.md>), [compare](<https://devfeed.tech/tags/compare.md>), [export](<https://devfeed.tech/tags/export.md>), [fields](<https://devfeed.tech/tags/fields.md>), [managed](<https://devfeed.tech/tags/managed.md>), [parquet](<https://devfeed.tech/tags/parquet.md>), [pipeline](<https://devfeed.tech/tags/pipeline.md>), [python](<https://devfeed.tech/tags/python.md>), [s3](<https://devfeed.tech/tags/s3.md>), [salesforce](<https://devfeed.tech/tags/salesforce.md>), [types](<https://devfeed.tech/tags/types.md>), [with](<https://devfeed.tech/tags/with.md>)

### AI overview

Salesforce-to-S3 Parquet exports rely on an initial object load followed by polling `SystemModstamp` or consuming change events; updates append new rows, and deletes appear only when the method includes deleted records. The choice depends on scale and destination: AppFlow suits a few objects in AWS but has a 15 GB per-run cap and needs `preserveSourceDataTyping` enabled to retain types; Glue zero-ETL writes Iceberg tables that can be compacted but omits some objects; Python allows custom logic; managed pipelines suit many objects or destinations. Plan for shared Salesforce API limits, per-object S3 prefixes, and incremental recovery. Formula values can change without `SystemModstamp`, so they need periodic rereads; append-only files then require a query that combines the latest full row with refreshed formula values. Purged records cannot be recovered, and reloading can duplicate rows. Validate types, edits, deletes, and new fields after loading.

### Source excerpt

Move Salesforce to S3 as Parquet with Amazon AppFlow, AWS Glue, Python or a managed pipeline. Compare how each handles types, formula fields and deletes.

## Polars vs pandas: What's the Difference?

DevFeed: [Polars vs pandas: What's the Difference?](<https://devfeed.tech/articles/polars-vs-pandas-what-s-the-difference-121070.md>)

Original publisher: [Read original article](<https://realpython.com/polars-vs-pandas/>)

Author: Ian Eyre

Published: 2026-11-05T14:00:00Z

Content type: tutorial

Language: en

Sources: [Real Python](<https://devfeed.tech/sources/real-python.md>)

Topics: [pandas](<https://devfeed.tech/topics/pandas.md>), [Data analysis](<https://devfeed.tech/topics/data-analysis.md>), [Python](<https://devfeed.tech/topics/python.md>), [data transformation](<https://devfeed.tech/topics/data-transformation.md>), [Data visualization](<https://devfeed.tech/topics/data-visualization.md>), [data type](<https://devfeed.tech/topics/data-type.md>), [parquet](<https://devfeed.tech/topics/parquet.md>), [pip](<https://devfeed.tech/topics/pip.md>)

Tags: [analysis](<https://devfeed.tech/tags/analysis.md>), [data](<https://devfeed.tech/tags/data.md>), [data-analysis](<https://devfeed.tech/tags/data-analysis.md>), [difference](<https://devfeed.tech/tags/difference.md>), [faster](<https://devfeed.tech/tags/faster.md>), [pandas](<https://devfeed.tech/tags/pandas.md>), [polars](<https://devfeed.tech/tags/polars.md>), [python](<https://devfeed.tech/tags/python.md>), [vs](<https://devfeed.tech/tags/vs.md>)

### AI overview

Polars and pandas can produce the same analysis, but their APIs and execution options differ. Polars uses expressions and contexts to build chained queries without mutating the original DataFrame; pandas supports both index-based operations and method chains. In the tutorial's grouped-sales benchmark, Polars and its LazyFrame ran faster than pandas, while pandas created its DataFrame faster. These timings vary by workload and setup; LazyFrames cannot handle every operation lazily, and streaming is limited to supported operations. Polars streaming processes data in chunks and may help when data exceeds memory, though combining batches can add time; pandas has no built-in streaming. The tutorial also shows `.to_pandas()` and `from_pandas()` conversions, Narwhals for a shared API that preserves the input backend, and built-in plotting through Altair in Polars and Matplotlib in pandas. Both target single-node use; the text points to PySpark for distributed processing.

### Source excerpt

Discover the key differences in Polars vs pandas to help you choose the right Python library for faster, more efficient data analysis.

## Introducing Apache Spark® 4.3 - Now available in Databricks Runtime 19

DevFeed: [Introducing Apache Spark® 4.3 - Now available in Databricks Runtime 19](<https://devfeed.tech/articles/introducing-apache-spark-4-3-now-available-in-databricks-runtime-19-117923.md>)

Original publisher: [Read original article](<https://www.databricks.com/blog/introducing-apache-spark-43-now-available-databricks-runtime-19>)

Author: Jungtaek Lim; Wenchen Fan; Andreas Neumann; Serge Rielau; Hyukjin Kwon; Jerry Peng; DB Tsai; Daniel Tenedorio; Xiao Li; Reynold Xin

Published: 2026-11-05T00:00:00Z

Content type: release

Language: en

Sources: [Databricks](<https://devfeed.tech/sources/databricks.md>)

Topics: [Apache Spark](<https://devfeed.tech/topics/spark.md>), [streaming data analytics](<https://devfeed.tech/topics/streaming-data-analytics.md>), [Stateful](<https://devfeed.tech/topics/stateful.md>), [Event-Streaming](<https://devfeed.tech/topics/event-streaming.md>), [scheduling](<https://devfeed.tech/topics/scheduling.md>), [apache-arrow](<https://devfeed.tech/topics/apache-arrow.md>), [parquet](<https://devfeed.tech/topics/parquet.md>), [schema-evolution](<https://devfeed.tech/topics/schema-evolution.md>), [databricks](<https://devfeed.tech/topics/databricks.md>), [Streaming](<https://devfeed.tech/topics/streaming.md>), [Apache Kafka](<https://devfeed.tech/topics/apache-kafka.md>)

Tags: [aggregation](<https://devfeed.tech/tags/aggregation.md>), [apache](<https://devfeed.tech/tags/apache.md>), [apache-spark](<https://devfeed.tech/tags/apache-spark.md>), [engineering](<https://devfeed.tech/tags/engineering.md>), [open-source](<https://devfeed.tech/tags/open-source.md>), [real-time](<https://devfeed.tech/tags/real-time.md>), [spark](<https://devfeed.tech/tags/spark.md>), [stateful](<https://devfeed.tech/tags/stateful.md>)

### AI overview

Apache Spark 4.3 extends Structured Streaming's Real-Time Mode with stateful aggregation, deduplication and custom processing, reporting P99 end-to-end latency below 150 ms across three tested benchmarks; those measurements are not a guarantee for all workloads. It enables continuous data flow through concurrent stage scheduling and pipelined shuffles, while retaining Spark's state store and checkpointing. The release also adds Auto CDC SCD Type 2 history through Python, SQL and Spark Connect, requiring a catalog connector that supports Data Source V2 row-level operations (built-in Parquet does not). Other additions include opt-in shuffle-free execution for eligible small queries, Spark Connect session and operation visibility, JSON and VARIANT processing, and ASOF JOIN and nanosecond timestamp previews that may change. Spark 4.3 starts a quarterly feature-release cadence.

### Source excerpt

IntroductionApache Spark 4.3 brings stateful processing to Real-Time Mode, achieving...

## Amazon S3 Tables now offer managed migration of Apache Hive tables to Apache Iceberg

DevFeed: [Amazon S3 Tables now offer managed migration of Apache Hive tables to Apache Iceberg](<https://devfeed.tech/articles/amazon-s3-tables-now-offer-managed-migration-of-apache-hive-tables-to-apache-iceberg-118925.md>)

Original publisher: [Read original article](<https://aws.amazon.com/blogs/aws/amazon-s3-tables-now-offer-managed-migration-of-apache-hive-tables-to-apache-iceberg/>)

Author: Channy Yun (윤석찬)

Published: 2026-11-04T23:34:52Z

Content type: release

Language: en

Sources: [AWS News Blog](<https://devfeed.tech/sources/aws-news-blog.md>)

Topics: [Amazon S3 Tables](<https://devfeed.tech/topics/amazon-s3-tables.md>), [Apache Iceberg](<https://devfeed.tech/topics/apache-iceberg.md>), [Amazon S3](<https://devfeed.tech/topics/amazon-s3.md>), [AWS Glue](<https://devfeed.tech/topics/aws-glue.md>), [parquet](<https://devfeed.tech/topics/parquet.md>), [Amazon SageMaker](<https://devfeed.tech/topics/amazon-sagemaker.md>)

Tags: [amazon-s3-tables](<https://devfeed.tech/tags/amazon-s3-tables.md>), [analytics](<https://devfeed.tech/tags/analytics.md>), [apache-hive](<https://devfeed.tech/tags/apache-hive.md>), [apache-iceberg](<https://devfeed.tech/tags/apache-iceberg.md>), [aws-glue](<https://devfeed.tech/tags/aws-glue.md>), [launch](<https://devfeed.tech/tags/launch.md>), [migration](<https://devfeed.tech/tags/migration.md>), [news](<https://devfeed.tech/tags/news.md>), [storage](<https://devfeed.tech/tags/storage.md>)

### AI overview

Amazon S3 Tables can migrate AWS Glue-registered Hive tables stored as Parquet to managed Apache Iceberg tables in the same AWS account and Region. S3 Tables checks compatibility and source objects before copying, then handles type conversions, recompression, validation, and compaction. The destination remains read-only during migration; you can sync changes repeatedly without stopping source writes, then complete cutover, which takes the source table offline for a few minutes. The service charges for data copied, with no Hive-to-Iceberg conversion fee, and leaves S3 Intelligent-Tiering objects in their current tier. The feature is available in the listed US, Asia Pacific, and European Regions.

### Source excerpt

You can now migrate existing Apache Hive datasets in Amazon S3 to fully managed Apache Iceberg tables in Amazon S3 Tables, with no pipelines to build or compute to provision, while production workloads keep running.

## MongoDB to S3: Parquet Files, Change Streams & Nested Documents Guide

DevFeed: [MongoDB to S3: Parquet Files, Change Streams & Nested Documents Guide](<https://devfeed.tech/articles/mongodb-to-s3-parquet-files-change-streams-nested-documents-guide-118922.md>)

Original publisher: [Read original article](<https://estuary.dev/blog/mongodb-to-s3>)

Author: Seán Whelan

Published: 2026-11-04T22:41:24Z

Content type: article

Language: en

Sources: [Estuary](<https://devfeed.tech/sources/estuary.md>)

Topics: [streaming data to S3 with Parquet](<https://devfeed.tech/topics/streaming-data-to-s3-with-parquet.md>), [MongoDB](<https://devfeed.tech/topics/mongodb.md>), [Parquet files in Amazon S3](<https://devfeed.tech/topics/parquet-files-in-amazon-s3.md>), [Amazon S3](<https://devfeed.tech/topics/amazon-s3.md>), [parquet](<https://devfeed.tech/topics/parquet.md>), [AWS Database Migration Service](<https://devfeed.tech/topics/aws-database-migration-service.md>), [AWS Glue](<https://devfeed.tech/topics/aws-glue.md>), [compressing data with Parquet](<https://devfeed.tech/topics/compressing-data-with-parquet.md>), [DLT](<https://devfeed.tech/topics/data-load-tool.md>), [Amazon DocumentDB](<https://devfeed.tech/topics/amazon-documentdb.md>)

Tags: [change](<https://devfeed.tech/tags/change.md>), [mongodb](<https://devfeed.tech/tags/mongodb.md>), [nested](<https://devfeed.tech/tags/nested.md>), [parquet](<https://devfeed.tech/tags/parquet.md>), [parquet-files](<https://devfeed.tech/tags/parquet-files.md>), [s3](<https://devfeed.tech/tags/s3.md>)

### AI overview

MongoDB change streams can feed append-only Parquet files in S3, but updates and deletes become new rows; readers must retain the latest row per `_id` and filter delete markers to represent current state. Change streams require a replica set or sharded cluster and an oplog window long enough to cover outages. A time-window copy or batch job misses deletes. Atlas Data Federation, AWS DMS, custom Spark/Glue/dlt jobs, and managed CDC services differ in setup and how they map nested BSON data; structs, flattened columns, and JSON text are the described options. Schema changes affect later files, and Decimal128 or digit-only strings can lose precision or leading zeros unless written as suitable decimal or string types. Estuary's described route captures changes durably and appends Parquet on a configurable interval, with nested-field depth settings; its Decimal128 and integer inference behavior may require `castToString`. The excerpt also gives S3 permissions, lifecycle, validation, and recovery requirements, including re-reading collections after an oplog history gap.

### Source excerpt

Move MongoDB to S3 as Parquet with Atlas Data Federation, AWS DMS, Spark or a managed pipeline. Compare how each handles nested documents and deletes.

## Migrate Google BigQuery to an open lakehouse on AWS with Apache Iceberg

DevFeed: [Migrate Google BigQuery to an open lakehouse on AWS with Apache Iceberg](<https://devfeed.tech/articles/migrate-google-bigquery-to-an-open-lakehouse-on-aws-with-apache-iceberg-114202.md>)

Original publisher: [Read original article](<https://aws.amazon.com/blogs/big-data/migrate-google-bigquery-to-an-open-lakehouse-on-aws-with-apache-iceberg/>)

Author: Saman Irfan

Published: 2026-11-02T17:21:17Z

Content type: article

Language: en

Sources: [AWS Big Data Blog](<https://devfeed.tech/sources/aws-big-data-blog.md>)

Topics: [Apache Iceberg](<https://devfeed.tech/topics/apache-iceberg.md>), [BigQuery](<https://devfeed.tech/topics/bigquery.md>), [Amazon Web Services](<https://devfeed.tech/topics/aws.md>), [Amazon S3 Tables](<https://devfeed.tech/topics/amazon-s3-tables.md>), [AWS Glue](<https://devfeed.tech/topics/aws-glue.md>), [AWS Lake Formation](<https://devfeed.tech/topics/aws-lake-formation.md>), [Amazon Bedrock Knowledge Bases](<https://devfeed.tech/topics/amazon-bedrock-knowledge-bases.md>), [AWS Step Functions](<https://devfeed.tech/topics/aws-step-functions.md>), [Amazon SageMaker](<https://devfeed.tech/topics/amazon-sagemaker.md>), [parquet](<https://devfeed.tech/topics/parquet.md>), [Amazon MSK](<https://devfeed.tech/topics/amazon-msk.md>), [Apache Kafka](<https://devfeed.tech/topics/apache-kafka.md>)

Tags: [advanced-300](<https://devfeed.tech/tags/advanced-300.md>), [amazon](<https://devfeed.tech/tags/amazon.md>), [amazon-athena](<https://devfeed.tech/tags/amazon-athena.md>), [amazon-redshift](<https://devfeed.tech/tags/amazon-redshift.md>), [amazon-s3](<https://devfeed.tech/tags/amazon-s3.md>), [apache](<https://devfeed.tech/tags/apache.md>), [apache-iceberg](<https://devfeed.tech/tags/apache-iceberg.md>), [architecture](<https://devfeed.tech/tags/architecture.md>), [aws](<https://devfeed.tech/tags/aws.md>), [bigquery](<https://devfeed.tech/tags/bigquery.md>), [data](<https://devfeed.tech/tags/data.md>), [design](<https://devfeed.tech/tags/design.md>), [future](<https://devfeed.tech/tags/future.md>), [google](<https://devfeed.tech/tags/google.md>), [google-bigquery](<https://devfeed.tech/tags/google-bigquery.md>), [iceberg](<https://devfeed.tech/tags/iceberg.md>), [lakehouse](<https://devfeed.tech/tags/lakehouse.md>), [migrate](<https://devfeed.tech/tags/migrate.md>), [migration](<https://devfeed.tech/tags/migration.md>), [open](<https://devfeed.tech/tags/open.md>), [optimize](<https://devfeed.tech/tags/optimize.md>), [over](<https://devfeed.tech/tags/over.md>), [technical-how-to](<https://devfeed.tech/tags/technical-how-to.md>), [with](<https://devfeed.tech/tags/with.md>)

### AI overview

The article presents a phased path from BigQuery to an AWS lakehouse using Apache Iceberg tables on Amazon S3, with governance through AWS Glue Data Catalog and Lake Formation and engines selected by workload. It recommends assessing tables, query complexity, consumers, volumes, and access controls; designing storage, partitioning, ingestion, and governance; then converting a representative slice and validating results, performance, and cost. For bulk transfer, export BigQuery tables to Parquet in Google Cloud Storage, move them to S3 with AWS DataSync, create or register Iceberg tables, and synchronize incremental changes during transition. Cutover includes parallel validation and deciding whether each consumer will reconnect, be rebuilt, or relay data. The article distinguishes managed S3 Tables, which handle maintenance automatically, from self-managed Iceberg on general-purpose S3, which requires tuning. It also cautions that BigQuery scan-based and Redshift compute-based pricing are not directly comparable, recommending workload-based benchmarks and monthly TCO estimates. Post-migration steps include query and cost optimization.

### Source excerpt

Learn how to migrate Google BigQuery workloads to an open lakehouse on AWS built on Apache Iceberg and Amazon S3. This post explains why an open lakehouse future-proofs your data, presents the target architecture on AWS, and walks through a step-by-step migration: assess, design, convert, migrate, cut over, and optimize.

## Apache Iceberg Clustering and Sort Order Evolution: When to Rewrite Data

DevFeed: [Apache Iceberg Clustering and Sort Order Evolution: When to Rewrite Data](<https://devfeed.tech/articles/apache-iceberg-clustering-and-sort-order-evolution-when-to-rewrite-data-102840.md>)

Original publisher: [Read original article](<https://www.dremio.com/blog/apache-iceberg-clustering-and-sort-order-evolution-when-to-rewrite-data/>)

Author: Alex Merced

Published: 2026-10-27T18:56:57Z

Content type: article

Language: en

Sources: [Dremio](<https://devfeed.tech/sources/dremio.md>)

Topics: [Apache Iceberg](<https://devfeed.tech/topics/apache-iceberg.md>), [Sorting](<https://devfeed.tech/topics/sorting.md>), [Apache Spark](<https://devfeed.tech/topics/spark.md>), [parquet](<https://devfeed.tech/topics/parquet.md>), [Grafana](<https://devfeed.tech/topics/grafana.md>), [Amazon S3 Tables](<https://devfeed.tech/topics/amazon-s3-tables.md>)

Tags: [apache-iceberg](<https://devfeed.tech/tags/apache-iceberg.md>), [data-lakehouse](<https://devfeed.tech/tags/data-lakehouse.md>), [dremio-blog-open-data-insights](<https://devfeed.tech/tags/dremio-blog-open-data-insights.md>), [evolution](<https://devfeed.tech/tags/evolution.md>), [rewrite](<https://devfeed.tech/tags/rewrite.md>), [sort](<https://devfeed.tech/tags/sort.md>)

### AI overview

Changing an Apache Iceberg sort order updates table metadata for future writes; it does not reorder existing files or refresh their file statistics. Rewriting old files is worthwhile only when measured pruning and query savings justify the I/O, commit load, and operational risk. The text recommends identifying frequent, high-value predicates, recording baseline files read, bytes scanned, and query latency, then testing the new sort order and a bounded rewrite on representative files. Compare pruning and end-to-end performance after each batch, while monitoring commit latency, retries, storage activity, and coordinator load. If gains persist, expand in controlled waves; stop if payback is weak or operational costs exceed the measured benefit. Exact DDL and rewrite procedures depend on the Iceberg, engine, and catalog versions, so verify supported syntax and APIs in the deployed environment. The suggested batch sizes and cluster observations are examples, not universal guarantees.

### Source excerpt

Evolve Iceberg sort orders safely, measure pruning quality, choose sort or Z-order rewrites, and decide when existing data justifies a controlled rewrite. The post Apache Iceberg Clustering and Sort Order Evolution: When to Rewrite Data appeared first on Dremio.

## Making the Telemetry Pipeline Measurable: An Edge IoT Testbed and Dataset for End-to-End Latency, Provenance, and Energy-Aware Benchmarking

DevFeed: [Making the Telemetry Pipeline Measurable: An Edge IoT Testbed and Dataset for End-to-End Latency, Provenance, and Energy-Aware Benchmarking](<https://devfeed.tech/articles/making-the-telemetry-pipeline-measurable-an-edge-iot-testbed-and-dataset-for-end-to-end-latency-provenance-and-energy-aware-benchmarking-101638.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2610.35591>)

Author: Dimitris Gkoulis

Published: 2026-10-27T04:00:00Z

Content type: article

Language: en

Sources: [arXiv: Distributed Computing](<https://devfeed.tech/sources/arxiv-distributed-computing.md>), [arXiv: Networking](<https://devfeed.tech/sources/arxiv-networking.md>), [arXiv: Software Engineering](<https://devfeed.tech/sources/arxiv-software-engineering.md>)

Topics: [streaming data analytics](<https://devfeed.tech/topics/streaming-data-analytics.md>), [parquet](<https://devfeed.tech/topics/parquet.md>), [data transformation](<https://devfeed.tech/topics/data-transformation.md>), [schema design](<https://devfeed.tech/topics/schema-design.md>)

Tags: [benchmarking](<https://devfeed.tech/tags/benchmarking.md>), [cs-dc](<https://devfeed.tech/tags/cs-dc.md>), [cs-ni](<https://devfeed.tech/tags/cs-ni.md>), [cs-se](<https://devfeed.tech/tags/cs-se.md>), [dataset](<https://devfeed.tech/tags/dataset.md>), [edge](<https://devfeed.tech/tags/edge.md>), [energy](<https://devfeed.tech/tags/energy.md>), [iot](<https://devfeed.tech/tags/iot.md>), [latency](<https://devfeed.tech/tags/latency.md>), [pipeline](<https://devfeed.tech/tags/pipeline.md>), [provenance](<https://devfeed.tech/tags/provenance.md>), [telemetry](<https://devfeed.tech/tags/telemetry.md>), [testbed](<https://devfeed.tech/tags/testbed.md>)

### AI overview

The work presents an edge IoT testbed and dataset designed to make telemetry timing, provenance, and energy measurable. Its role-based Source-Broker-Aggregator architecture uses an Avro TelemetryMessage envelope with correlation IDs and shared stage timestamps; an Eclipse Mosquitto plugin adds high-precision broker ingress and egress times. Data is provided as compressed, replayable Avro event streams and partitioned long-format Parquet tables produced by a deterministic transformation pipeline. The demonstration reports latency summaries, measurement-context distributions, and energy aggregates for one deployment. The authors characterize this as an integrated artifact built from established components: it does not introduce a new attribution method or establish a controlled performance benchmark, but supports exploratory analysis and future benchmarking.

### Source excerpt

arXiv:2610.35591v1 Announce Type: new Abstract: Edge-centric IoT systems are increasingly evaluated through sensor values or platform features, yet the telemetry pipeline itself, the sequence of sensing, serialization, message mediation, ingestion, and persistence, is rarely released as a measurable object. This limits reproducible benchmarking of timeliness, provenance, and operational cost, especially in early-stage IoT design where real deployments and high-quality open data are often unavailable. I present a measurement-first edge IoT testbed and dataset that elevate timeliness, provenance, and energy to first-class telemetry dimensions. The testbed operationalizes a role-based event-driven architecture with explicit Source-Broker-Aggregator responsibilities and a schema-governed TelemetryMessage envelope (Avro) that embeds correlation identifiers and a uniform, cross-role vocabulary of stage-level timestamps. To trace mediation effects, I extend an MQTT broker (Eclipse Mosquitto) with a plugin that annotates messages with high-precision ingress/egress timestamps. I release the dataset in a dual representation: (i) replayable raw events as compressed Avro streams and (ii) analysis-ready long-format Parquet tables partitioned for efficient query and benchmarking, generated by a deterministic raw-to-tabular transformation pipeline. The contribution is an integrated engineering artifact and dataset built from established components. The demonstration describes one deployment through latency summaries, measurement-context distributions, and energy aggregates; it does not establish a new attribution method or a controlled performance benchmark. By co-releasing heterogeneous telemetry with pipeline timing and energy signals in both replayable and queryable forms, the artifact provides a basis for reproducible exploratory analysis and subsequent benchmarking studies.

## A curated multi-endpoint drug-protein bioactivity dataset with construct-aware protein targets

DevFeed: [A curated multi-endpoint drug-protein bioactivity dataset with construct-aware protein targets](<https://devfeed.tech/articles/a-curated-multi-endpoint-drug-protein-bioactivity-dataset-with-construct-aware-protein-targets-99620.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2610.32490>)

Author: Lina Abou-Abbas, Nour Mhanna, Hasan Bazzi, Sleiman El Hajj, Abdulkader Helwan, Rabah Omrani, Khadidja Henni

Published: 2026-10-26T04:00:00Z

Content type: article

Language: en

Sources: [arXiv: Machine Learning](<https://devfeed.tech/sources/arxiv-machine-learning.md>)

Topics: [dataset](<https://devfeed.tech/topics/dataset.md>), [parquet](<https://devfeed.tech/topics/parquet.md>)

Tags: [classification](<https://devfeed.tech/tags/classification.md>), [code](<https://devfeed.tech/tags/code.md>), [cs-lg](<https://devfeed.tech/tags/cs-lg.md>), [data](<https://devfeed.tech/tags/data.md>), [dataset](<https://devfeed.tech/tags/dataset.md>), [evaluation](<https://devfeed.tech/tags/evaluation.md>), [machine-learning](<https://devfeed.tech/tags/machine-learning.md>), [provenance](<https://devfeed.tech/tags/provenance.md>), [regression](<https://devfeed.tech/tags/regression.md>)

### AI overview

Curate-DTI preserves experimental protein constructs that accession-only target processing could merge. Derived from the 1 July 2026 BindingDB export, it processes IC50, Ki, Kd and EC50 independently and contains 2,919,717 measurements, 9,668 sequence-defined protein entities and 2,275,683 endpoint-specific interactions. For 4,204 endpoint-specific ligand-accession groups, classification labels differed across sequence-defined entities. Classification can use exact and inequality-qualified observations; regression targets use exact values only. Connected Parquet tables retain measurements, standardized ligands, protein sequences, aggregated interactions and available assay and source provenance. The dataset supports endpoint-specific classification and regression with selectable molecular and protein representations and evaluation splits; preprocessing code and validation materials accompany the release.

### Source excerpt

arXiv:2610.32490v1 Announce Type: new Abstract: Public drug-protein bioactivity resources commonly identify targets by UniProt accession, although one accession may represent distinct experimental sequences. We present Curate-DTI, a construct-aware dataset derived from the 1 July 2026 BindingDB export, with IC50, Ki, Kd and EC50 processed independently. The release contains 2,919,717 measurements, 1,291,872 standardized ligands, 9,668 sequence-defined protein entities and 2,275,683 endpoint-specific interactions spanning 8,336 reviewed UniProt accessions. Protein entities combine the accession with the reported experimental sequence, preserving constructs that accession-only processing would merge. Across the four endpoints, 4,204 endpoint-specific ligand-accession groups showed different classification labels across sequence-defined protein entities. Exact and inequality-qualified observations provide relation-aware classification evidence, whereas regression targets use exact values only. Connected Parquet tables preserve individual measurements, standardized structures, protein sequences, aggregated interactions and available assay and source provenance. The resource supports endpoint-specific classification and regression while allowing users to select molecular representations, protein representations and evaluation splits. Preprocessing code, a column dictionary, exclusion reports and validation summaries accompany the release.

## Article: AI Changes the Data Interface, Not the Query Engine

DevFeed: [Article: AI Changes the Data Interface, Not the Query Engine](<https://devfeed.tech/articles/article-ai-changes-the-data-interface-not-the-query-engine-95267.md>)

Original publisher: [Read original article](<https://www.infoq.com/articles/ai-changes-data-interface/>)

Author: Peihao Yuan

Published: 2026-10-22T09:00:00Z

Content type: article

Language: en

Sources: [InfoQ](<https://devfeed.tech/sources/infoq.md>)

Topics: [DuckDB](<https://devfeed.tech/topics/duckdb.md>), [reproducibility](<https://devfeed.tech/topics/reproducibility.md>), [Disaster Recovery](<https://devfeed.tech/topics/disaster-recovery.md>), [parquet](<https://devfeed.tech/topics/parquet.md>), [llm-reasoning](<https://devfeed.tech/topics/llm-reasoning.md>), [Data pipelines](<https://devfeed.tech/topics/data-pipelines.md>), [Query (disambiguation)](<https://devfeed.tech/topics/query.md>), [Apache Spark](<https://devfeed.tech/topics/spark.md>)

Tags: [agent](<https://devfeed.tech/tags/agent.md>), [agentic-ai-architecture](<https://devfeed.tech/tags/agentic-ai-architecture.md>), [ai](<https://devfeed.tech/tags/ai.md>), [ai-agent](<https://devfeed.tech/tags/ai-agent.md>), [ai-architecture](<https://devfeed.tech/tags/ai-architecture.md>), [ai-changes-data-interface](<https://devfeed.tech/tags/ai-changes-data-interface.md>), [ai-ml-data-engineering](<https://devfeed.tech/tags/ai-ml-data-engineering.md>), [analysis](<https://devfeed.tech/tags/analysis.md>), [analytics](<https://devfeed.tech/tags/analytics.md>), [architecture](<https://devfeed.tech/tags/architecture.md>), [architecture-design](<https://devfeed.tech/tags/architecture-design.md>), [article](<https://devfeed.tech/tags/article.md>), [changes](<https://devfeed.tech/tags/changes.md>), [data](<https://devfeed.tech/tags/data.md>), [data-lake](<https://devfeed.tech/tags/data-lake.md>), [database](<https://devfeed.tech/tags/database.md>), [disaster](<https://devfeed.tech/tags/disaster.md>), [disaster-recovery](<https://devfeed.tech/tags/disaster-recovery.md>), [duckdb](<https://devfeed.tech/tags/duckdb.md>), [embedded](<https://devfeed.tech/tags/embedded.md>), [engine](<https://devfeed.tech/tags/engine.md>), [interface](<https://devfeed.tech/tags/interface.md>), [lakehouse](<https://devfeed.tech/tags/lakehouse.md>), [managed](<https://devfeed.tech/tags/managed.md>), [ml-data-engineering](<https://devfeed.tech/tags/ml-data-engineering.md>), [parquet](<https://devfeed.tech/tags/parquet.md>), [presents](<https://devfeed.tech/tags/presents.md>), [query](<https://devfeed.tech/tags/query.md>), [query-engine](<https://devfeed.tech/tags/query-engine.md>)

### AI overview

In the disaster-recovery workflow, DuckDB read Parquet and deterministically calculated traffic, availability, and latency facts; an AI agent generated constrained SQL or Python, interpreted those facts in service context, and explained the results. The workflow retained inputs, configuration, code, execution metadata, facts, and model outputs so analyses could be replayed and checked. The author argues an embedded engine can suffice for moderate, focused workloads with controlled writes. A shared catalog, open table format, or distributed engine becomes useful when independent consumers, large shared joins, concurrent writes, schema evolution, or organization-wide governance create concrete needs; these capabilities add integration and operational costs.

### Source excerpt

This article presents a disaster-recovery analytics workflow in which an AI agent generates constrained SQL or Python, DuckDB computes reproducible facts from Parquet, and the agent interprets the results. It explains the determinism split, replayable analysis, and four signals for deciding when an embedded engine is sufficient or a managed lakehouse is justified. By Peihao Yuan

## Apache Iceberg Architecture Explained

DevFeed: [Apache Iceberg Architecture Explained](<https://devfeed.tech/articles/apache-iceberg-architecture-explained-88575.md>)

Original publisher: [Read original article](<https://www.starburst.io/blog/apache-iceberg-architecture-explained>)

Published: 2026-10-16T08:00:04Z

Content type: article

Language: en

Sources: [Starburst Data](<https://devfeed.tech/sources/starburst-data.md>)

Topics: [Apache Iceberg](<https://devfeed.tech/topics/apache-iceberg.md>), [parquet](<https://devfeed.tech/topics/parquet.md>), [data lake](<https://devfeed.tech/topics/data-lake.md>), [table-partitioning](<https://devfeed.tech/topics/table-partitioning.md>), [Amazon S3](<https://devfeed.tech/topics/amazon-s3.md>)

Tags: [2](<https://devfeed.tech/tags/2.md>), [3](<https://devfeed.tech/tags/3.md>), [analytics](<https://devfeed.tech/tags/analytics.md>), [apache](<https://devfeed.tech/tags/apache.md>), [apache-iceberg](<https://devfeed.tech/tags/apache-iceberg.md>), [apache-iceberg-architecture](<https://devfeed.tech/tags/apache-iceberg-architecture.md>), [architecture](<https://devfeed.tech/tags/architecture.md>), [format](<https://devfeed.tech/tags/format.md>), [high-performance](<https://devfeed.tech/tags/high-performance.md>), [iceberg](<https://devfeed.tech/tags/iceberg.md>), [object](<https://devfeed.tech/tags/object.md>), [object-storage](<https://devfeed.tech/tags/object-storage.md>), [open](<https://devfeed.tech/tags/open.md>), [performance](<https://devfeed.tech/tags/performance.md>), [storage](<https://devfeed.tech/tags/storage.md>), [table](<https://devfeed.tech/tags/table.md>), [table-format](<https://devfeed.tech/tags/table-format.md>)

### AI overview

Apache Iceberg tracks tables through a catalog pointer to immutable metadata and data files, rather than inferring table contents from directory layouts. Reads use snapshot and manifest statistics to prune files; writes create new files and metadata, then use an optimistic, atomic catalog pointer swap. This supports concurrent engine access, snapshots and time travel, while row level deletes and updates use delete files. The document distinguishes v2 position and equality deletes from v3 deletion vectors, which replace position deletes, and notes v3's VARIANT type and other additions. Engine and catalog compatibility must be checked before upgrading. Ongoing maintenance--compaction, snapshot expiration, orphan file removal and manifest optimization--helps control small file fragmentation, metadata growth and storage costs.

### Source excerpt

Apache Iceberg is an open table format designed for high-performance analytics running on object storage. ...

## How to Architect an On-Premises Data Lakehouse

DevFeed: [How to Architect an On-Premises Data Lakehouse](<https://devfeed.tech/articles/how-to-architect-an-on-premises-data-lakehouse-87813.md>)

Original publisher: [Read original article](<https://www.starburst.io/blog/how-to-architect-an-on-premises-data-lakehouse>)

Published: 2026-10-15T08:00:34Z

Content type: article

Language: en

Sources: [Starburst Data](<https://devfeed.tech/sources/starburst-data.md>)

Topics: [data lake](<https://devfeed.tech/topics/data-lake.md>), [database-optimization](<https://devfeed.tech/topics/database-optimization.md>), [acid](<https://devfeed.tech/topics/acid.md>), [Transactions](<https://devfeed.tech/topics/transactions.md>), [Cache](<https://devfeed.tech/topics/cache.md>), [control-plane](<https://devfeed.tech/topics/control-plane.md>), [parquet](<https://devfeed.tech/topics/parquet.md>), [AI Infrastructure](<https://devfeed.tech/topics/ai-infrastructure.md>), [AI-Ready Data](<https://devfeed.tech/topics/ai-ready-data.md>)

Tags: [access-control](<https://devfeed.tech/tags/access-control.md>), [ai](<https://devfeed.tech/tags/ai.md>), [ai-workloads](<https://devfeed.tech/tags/ai-workloads.md>), [air-gapped](<https://devfeed.tech/tags/air-gapped.md>), [amazon-s3](<https://devfeed.tech/tags/amazon-s3.md>), [analytics](<https://devfeed.tech/tags/analytics.md>), [apache-iceberg](<https://devfeed.tech/tags/apache-iceberg.md>), [apache-iceberg-tables](<https://devfeed.tech/tags/apache-iceberg-tables.md>), [architect](<https://devfeed.tech/tags/architect.md>), [audit](<https://devfeed.tech/tags/audit.md>), [availability](<https://devfeed.tech/tags/availability.md>), [bandwidth](<https://devfeed.tech/tags/bandwidth.md>), [batch](<https://devfeed.tech/tags/batch.md>), [benchmark](<https://devfeed.tech/tags/benchmark.md>), [bi](<https://devfeed.tech/tags/bi.md>), [big-bang](<https://devfeed.tech/tags/big-bang.md>), [build](<https://devfeed.tech/tags/build.md>), [built-in](<https://devfeed.tech/tags/built-in.md>), [data-lakehouse](<https://devfeed.tech/tags/data-lakehouse.md>), [lakehouse](<https://devfeed.tech/tags/lakehouse.md>)

### AI overview

An on-premises lakehouse can serve sovereignty, regulatory, latency, or egress-cost needs using S3-compatible object storage, Apache Iceberg, and Trino, while separating storage from compute. Plan capacity, durability, network bandwidth, and the Iceberg catalog early; compatibility varies by provider: open-source Trino is tested with Amazon S3 and MinIO, while Starburst Enterprise documents its supported stores. The article recommends starting federation-first, then materializing frequently used paths into Iceberg, and building access controls, audit, and governed data products in from day one. Migrate from Hadoop in stages: replace Hive or Impala with Trino while HDFS remains, write new data to Iceberg, selectively convert Hive tables, then retire legacy components. Starburst Enterprise multi-cluster availability features require Elite and were in public preview in 480-e; running queries may fail during coordinator failover, so the article advises testing recovery plans that do not depend on those features.

### Source excerpt

An on-premises data lakehouse brings lakehouse capabilities into data centers you own or lease. It ...

[Next page](<https://devfeed.tech/topics/parquet.md?cursor=WyIyMDI2LTEwLTE1VDA4OjAwOjM0KzAwOjAwIiwgIjY4NWRkMGU5LTY5YjMtNGUxNS1iMTAxLWI5OTQ4MTYxMzY1NCJd>)