# table-partitioning

Table partitioning is a database technique that splits one logically large table into smaller physical partitions, improving manageability and potentially query performance.

This is one page of public article previews, not the complete archive. Follow Next page to continue. Summaries are not the original full articles.

## Ducks in a Column

DevFeed: [Ducks in a Column](<https://devfeed.tech/articles/ducks-in-a-column-127523.md>)

Original publisher: [Read original article](<https://duckdb.org/2026/11/10/ducks-in-a-column.html>)

Author: The DuckDB team

Published: 2026-11-10T00:00:00Z

Content type: article

Language: en

Sources: [DuckDB](<https://devfeed.tech/sources/duckdb.md>)

Topics: [DuckDB](<https://devfeed.tech/topics/duckdb.md>), [parquet](<https://devfeed.tech/topics/parquet.md>), [database-optimization](<https://devfeed.tech/topics/database-optimization.md>), [table-partitioning](<https://devfeed.tech/topics/table-partitioning.md>), [CSV](<https://devfeed.tech/topics/csv.md>), [data lake](<https://devfeed.tech/topics/data-lake.md>)

Tags: [aggregation](<https://devfeed.tech/tags/aggregation.md>), [columnar](<https://devfeed.tech/tags/columnar.md>), [duckdb](<https://devfeed.tech/tags/duckdb.md>), [parquet](<https://devfeed.tech/tags/parquet.md>), [querying](<https://devfeed.tech/tags/querying.md>), [sql](<https://devfeed.tech/tags/sql.md>), [using-duckdb](<https://devfeed.tech/tags/using-duckdb.md>), [window-functions](<https://devfeed.tech/tags/window-functions.md>), [writing](<https://devfeed.tech/tags/writing.md>)

### AI overview

DuckDB's columnar storage lets analytical queries read only named columns, and Parquet adds row-group statistics that can skip data excluded by filters. The article demonstrates querying CSV, Parquet, remote files and DataFrames in place; using EXCLUDE, REPLACE and COLUMNS() for column-level SQL; and simplifying analysis with GROUP BY ALL, FILTER, arg_max, PIVOT/UNPIVOT, window functions, QUALIFY and ASOF JOIN. For faster later reads, it recommends partitioning on commonly filtered columns, sorting before writing Parquet, choosing narrow types, and checking EXPLAIN ANALYZE to see which files and columns were scanned. It also notes that approximate aggregates can use less memory than exact counterparts, while giving a small-table example where an approximate distinct count differs from the exact count.

### Source excerpt

DuckDB is fast among other reasons because it stores data by column, so a query reads only the columns it names. This post covers DuckDB features that take advantage of that design choice: querying files in place, column-level SQL, aggregation shortcuts, window functions, ASOF joins, and writing well-organized Parquet.

## Apache Iceberg Partition Statistics: compute\_partition\_stats, Pruning, and Query Planning

DevFeed: [Apache Iceberg Partition Statistics: compute\_partition\_stats, Pruning, and Query Planning](<https://devfeed.tech/articles/apache-iceberg-partition-statistics-compute-partition-stats-pruning-and-query-planning-123560.md>)

Original publisher: [Read original article](<https://www.dremio.com/blog/apache-iceberg-partition-statistics-compute_partition_stats-pruning-and-query-planning/>)

Author: Alex Merced

Published: 2026-11-09T13:42:02Z

Content type: article

Language: en

Sources: [Dremio](<https://devfeed.tech/sources/dremio.md>)

Topics: [Apache Iceberg](<https://devfeed.tech/topics/apache-iceberg.md>), [table-partitioning](<https://devfeed.tech/topics/table-partitioning.md>), [Query (disambiguation)](<https://devfeed.tech/topics/query.md>), [observability](<https://devfeed.tech/topics/observability.md>)

Tags: [apache-iceberg](<https://devfeed.tech/tags/apache-iceberg.md>), [compute](<https://devfeed.tech/tags/compute.md>), [data-lakehouse](<https://devfeed.tech/tags/data-lakehouse.md>), [dremio-blog-open-data-insights](<https://devfeed.tech/tags/dremio-blog-open-data-insights.md>), [metadata](<https://devfeed.tech/tags/metadata.md>), [partition](<https://devfeed.tech/tags/partition.md>), [planning](<https://devfeed.tech/tags/planning.md>), [pruning](<https://devfeed.tech/tags/pruning.md>)

### AI overview

Iceberg partition statistics summarize file metrics by partition for a specific snapshot; compute_partition_stats writes them to a metadata file referenced by that snapshot, without rewriting data files or creating separate Puffin column statistics. Incremental runs can reduce work when few manifests change, though manifest churn may force rescans and missing file-level metrics limit the summaries. Planner support varies by engine and version, so verify snapshot freshness and compare EXPLAIN estimates, partitions and files scanned, bytes read, and query time before and after refresh. These statistics help most when partition keys match common filters; they do not replace file-level pruning for non-partition columns or fix a mismatched partition layout. The example Spark procedure signature and parallelism settings may vary by release.

### Source excerpt

Compute Iceberg partition statistics incrementally, understand their metadata model, validate pruning, and separate them from column NDV statistics. The post Apache Iceberg Partition Statistics: compute_partition_stats, Pruning, and Query Planning appeared first on Dremio.

## Database Sharding Clearly Explained

DevFeed: [Database Sharding Clearly Explained](<https://devfeed.tech/articles/database-sharding-clearly-explained-121476.md>)

Original publisher: [Read original article](<https://blog.levelupcoding.com/p/database-sharding-clearly-explained>)

Author: Nikki Siapno

Published: 2026-11-06T15:03:19Z

Content type: article

Language: en

Sources: [Level Up Coding System Design Newsletter](<https://devfeed.tech/sources/level-up-coding-system-design-newsletter.md>)

Topics: [sharding](<https://devfeed.tech/topics/sharding.md>), [database-scalability](<https://devfeed.tech/topics/database-scalability.md>), [table-partitioning](<https://devfeed.tech/topics/table-partitioning.md>), [read-replica](<https://devfeed.tech/topics/read-replica.md>), [database-optimization](<https://devfeed.tech/topics/database-optimization.md>), [connection pool](<https://devfeed.tech/topics/connection-pool.md>), [Memcached](<https://devfeed.tech/topics/memcached.md>)

Tags: [agents](<https://devfeed.tech/tags/agents.md>), [ai](<https://devfeed.tech/tags/ai.md>), [ai-engineering](<https://devfeed.tech/tags/ai-engineering.md>), [analytics](<https://devfeed.tech/tags/analytics.md>), [application](<https://devfeed.tech/tags/application.md>), [architecture](<https://devfeed.tech/tags/architecture.md>), [automate](<https://devfeed.tech/tags/automate.md>), [automation](<https://devfeed.tech/tags/automation.md>), [best-practices](<https://devfeed.tech/tags/best-practices.md>), [cache](<https://devfeed.tech/tags/cache.md>), [caching](<https://devfeed.tech/tags/caching.md>), [connection-pooling](<https://devfeed.tech/tags/connection-pooling.md>), [consistent-hashing](<https://devfeed.tech/tags/consistent-hashing.md>), [constraint](<https://devfeed.tech/tags/constraint.md>), [continuous](<https://devfeed.tech/tags/continuous.md>), [cpu](<https://devfeed.tech/tags/cpu.md>), [data](<https://devfeed.tech/tags/data.md>), [data-movement](<https://devfeed.tech/tags/data-movement.md>), [database](<https://devfeed.tech/tags/database.md>), [sharding](<https://devfeed.tech/tags/sharding.md>)

### AI overview

Sharding splits data across multiple database servers to distribute storage and workload when one server remains a constraint after simpler measures such as query optimization, caching, vertical scaling, read replicas, and partitioning. Horizontal sharding divides rows while keeping the same schema; vertical sharding separates tables or workloads. The key design challenge is choosing a shard key that spreads both data and traffic evenly. Queries and transactions spanning shards require coordination and add latency and complexity, while changing shard counts can require difficult data movement. The text recommends keeping common operations within one shard, planning for rebalancing, monitoring shards individually, automating maintenance, and testing realistic workloads. Sharding can extend scalability, but replaces a single-server bottleneck with distributed-system and operational challenges.

### Source excerpt

The hardest part of sharding isn't splitting the data.

## Kafka to ClickHouse: Streaming Ingestion, Duplicates & Tombstones Guide

DevFeed: [Kafka to ClickHouse: Streaming Ingestion, Duplicates & Tombstones Guide](<https://devfeed.tech/articles/kafka-to-clickhouse-streaming-ingestion-duplicates-tombstones-guide-121703.md>)

Original publisher: [Read original article](<https://estuary.dev/blog/kafka-to-clickhouse>)

Author: Seán Whelan

Published: 2026-11-06T12:26:41Z

Content type: article

Language: en

Sources: [Estuary](<https://devfeed.tech/sources/estuary.md>)

Topics: [Apache Kafka](<https://devfeed.tech/topics/apache-kafka.md>), [Event-Streaming](<https://devfeed.tech/topics/event-streaming.md>), [self-hosted databases vs managed services](<https://devfeed.tech/topics/self-hosted-databases-vs-managed-services.md>), [table-partitioning](<https://devfeed.tech/topics/table-partitioning.md>), [schema-evolution](<https://devfeed.tech/topics/schema-evolution.md>), [Amazon MSK](<https://devfeed.tech/topics/amazon-msk.md>), [Confluent Cloud](<https://devfeed.tech/topics/confluent-cloud.md>), [Apache Iceberg](<https://devfeed.tech/topics/apache-iceberg.md>)

Tags: [clickhouse](<https://devfeed.tech/tags/clickhouse.md>), [compare](<https://devfeed.tech/tags/compare.md>), [connect](<https://devfeed.tech/tags/connect.md>), [delivery](<https://devfeed.tech/tags/delivery.md>), [engine](<https://devfeed.tech/tags/engine.md>), [guarantees](<https://devfeed.tech/tags/guarantees.md>), [guide](<https://devfeed.tech/tags/guide.md>), [kafka](<https://devfeed.tech/tags/kafka.md>), [stream](<https://devfeed.tech/tags/stream.md>), [streaming](<https://devfeed.tech/tags/streaming.md>), [table](<https://devfeed.tech/tags/table.md>)

### AI overview

Kafka ingestion into ClickHouse is generally at least once, so replayed messages can create duplicates; reliable designs need batching and tables suited to event or keyed data. The article compares the Kafka table engine, ClickPipes, the Kafka Connect sink, and managed pipelines: the engine runs consumers on ClickHouse servers, ClickPipes is recommended for ClickHouse Cloud, and the sink fits teams already running Kafka Connect. For event streams, store partition and offset in a ReplacingMergeTree sorting key; keyed topics with tombstones need ReplacingMergeTree keyed by message key and queries with FINAL. Tombstone support differs: ClickPipes can delete rows in exactly-once mode or mark them deleted, while the Connect sink lists deletes as unsupported and the table engine's documentation does not describe null-value messages. Retention limits recovery, and compaction can remove tombstones after the configured retention period. Large, infrequent inserts reduce part pressure; monitor lag and active parts. The article also describes Estuary capture and materialization, which keeps Kafka progress in its own checkpoints and writes to ClickHouse on a configurable schedule.

### Source excerpt

Stream Kafka to ClickHouse with the Kafka table engine, ClickPipes or Kafka Connect. Compare 4 methods, delivery guarantees and duplicate handling.

## Normalizing a Wide Well and Pipeline Telemetry Table Without Re-Ingesting History

DevFeed: [Normalizing a Wide Well and Pipeline Telemetry Table Without Re-Ingesting History](<https://devfeed.tech/articles/normalizing-a-wide-well-and-pipeline-telemetry-table-without-re-ingesting-history-116242.md>)

Original publisher: [Read original article](<https://hackernoon.com/normalizing-a-wide-well-and-pipeline-telemetry-table-without-re-ingesting-history?source=rss>)

Author: Tiger Data (creators of TimescaleDB)

Published: 2026-11-04T08:13:35Z

Content type: tutorial

Language: en

Sources: [HackerNoon](<https://devfeed.tech/sources/hackernoon.md>)

Topics: [table-partitioning](<https://devfeed.tech/topics/table-partitioning.md>), [Data Quality](<https://devfeed.tech/topics/data-quality.md>)

Tags: [dashboards](<https://devfeed.tech/tags/dashboards.md>), [data-migration](<https://devfeed.tech/tags/data-migration.md>), [database-migration](<https://devfeed.tech/tags/database-migration.md>), [good-company](<https://devfeed.tech/tags/good-company.md>), [history](<https://devfeed.tech/tags/history.md>), [industrial-database](<https://devfeed.tech/tags/industrial-database.md>), [migrate](<https://devfeed.tech/tags/migrate.md>), [telemetry](<https://devfeed.tech/tags/telemetry.md>), [telemetry-database-design](<https://devfeed.tech/tags/telemetry-database-design.md>), [time-series-database](<https://devfeed.tech/tags/time-series-database.md>), [timescaledb](<https://devfeed.tech/tags/timescaledb.md>), [timescaledb-hypertable](<https://devfeed.tech/tags/timescaledb-hypertable.md>), [timescaledb-migration](<https://devfeed.tech/tags/timescaledb-migration.md>), [wide](<https://devfeed.tech/tags/wide.md>)

### AI overview

The guide describes an in-place migration from a wide TimescaleDB telemetry hypertable to a narrow `(recorded_at, tag_id, value)` hypertable plus a tag metadata table, keeping legacy dashboards working through a compatibility view. It checks that each tag's descriptors are stable, backfills metadata, moves readings in resumable time slices, then builds indexes and performs a brief lock-limited rename-and-view cutover. It highlights operational details: pause columnstore conversion during the backfill, freeze new tag registration or refresh metadata before slices, repoint dependent aggregates, policies, triggers and grants, and verify counts and sampled checksums before dropping the old table. The source reports under seven minutes for 100 million generated readings on its hardware, without competing live ingest. It excludes the time-dependent migration needed when tag descriptors change; it also notes extra schema and verification steps for non-float tags.

### Source excerpt

Learn how to migrate a wide TimescaleDB sensor table to a narrow hypertable in place, preserving historical data and keeping dashboards running.

## Improving Partition Pruning for Cross-Type Joins in Tanzu Greenplum

DevFeed: [Improving Partition Pruning for Cross-Type Joins in Tanzu Greenplum](<https://devfeed.tech/articles/improving-partition-pruning-for-cross-type-joins-in-tanzu-greenplum-112925.md>)

Original publisher: [Read original article](<https://blogs.vmware.com/tanzu/improving-partition-pruning-for-cross-type-joins-in-tanzu-greenplum/>)

Author: VMware Community

Published: 2026-11-02T09:38:00Z

Content type: article

Language: en

Sources: [VMware](<https://devfeed.tech/sources/vmware-blogs.md>)

Topics: [database-optimization](<https://devfeed.tech/topics/database-optimization.md>), [table-partitioning](<https://devfeed.tech/topics/table-partitioning.md>), [data type](<https://devfeed.tech/topics/data-type.md>), [Optimization](<https://devfeed.tech/topics/optimization.md>), [typesafety](<https://devfeed.tech/topics/typesafety.md>)

Tags: [constraint](<https://devfeed.tech/tags/constraint.md>), [cross](<https://devfeed.tech/tags/cross.md>), [framework](<https://devfeed.tech/tags/framework.md>), [greenplum](<https://devfeed.tech/tags/greenplum.md>), [inference](<https://devfeed.tech/tags/inference.md>), [modern-apps](<https://devfeed.tech/tags/modern-apps.md>), [partition](<https://devfeed.tech/tags/partition.md>), [vmware-tanzu-greenplum](<https://devfeed.tech/tags/vmware-tanzu-greenplum.md>)

### AI overview

GPORCA can now infer constraints through casts and other deterministic single-column expressions, allowing partition pruning for cross-type joins. In the example, a nanosecond timestamp filter is propagated through an implicit cast to a microsecond partition key, so the query can skip 364 of 366 daily partitions instead of scanning all of them. The change adds expression mapping, recursive tracing to base columns, and datetime type alignment; strict type compatibility checks are intended to prevent incorrect substitutions. Expression constraint handling is gated by the `optimizer_expression_constraints` GUC for incremental adoption, while the type correctness fixes are always active. The post says the optimization avoids query rewrites and manual redundant filters.

### Source excerpt

This technical post on VMware Tanzu Greenplum walks through the changes to GPORCA's constraint derivation framework that make partition pruning and predicate inference work for cross-type joins. The post Improving Partition Pruning for Cross-Type Joins in Tanzu Greenplum appeared first on Tanzu. The post Improving Partition Pruning for Cross-Type Joins in Tanzu Greenplum appeared first on VMware Blogs.

## Gems of DuckDB 2.0.0

DevFeed: [Gems of DuckDB 2.0.0](<https://devfeed.tech/articles/gems-of-duckdb-2-0-0-101870.md>)

Original publisher: [Read original article](<https://duckdb.org/2026/10/26/gems-of-duckdb-200.html>)

Author: Geertjan Wielenga

Published: 2026-10-26T00:00:00Z

Content type: article

Language: en

Sources: [DuckDB](<https://devfeed.tech/sources/duckdb.md>)

Topics: [DuckDB](<https://devfeed.tech/topics/duckdb.md>), [table-partitioning](<https://devfeed.tech/topics/table-partitioning.md>), [data type](<https://devfeed.tech/topics/data-type.md>), [aio](<https://devfeed.tech/topics/aio.md>)

Tags: [1-4](<https://devfeed.tech/tags/1-4.md>), [1-5](<https://devfeed.tech/tags/1-5.md>), [2-0-0](<https://devfeed.tech/tags/2-0-0.md>), [2018](<https://devfeed.tech/tags/2018.md>), [2019](<https://devfeed.tech/tags/2019.md>), [2020](<https://devfeed.tech/tags/2020.md>), [2021](<https://devfeed.tech/tags/2021.md>), [2023](<https://devfeed.tech/tags/2023.md>), [announcement](<https://devfeed.tech/tags/announcement.md>), [asynchronous](<https://devfeed.tech/tags/asynchronous.md>), [blog](<https://devfeed.tech/tags/blog.md>), [blog-post](<https://devfeed.tech/tags/blog-post.md>), [duckdb](<https://devfeed.tech/tags/duckdb.md>), [features](<https://devfeed.tech/tags/features.md>), [i-o](<https://devfeed.tech/tags/i-o.md>), [improvements](<https://devfeed.tech/tags/improvements.md>), [post](<https://devfeed.tech/tags/post.md>), [protocol](<https://devfeed.tech/tags/protocol.md>), [release](<https://devfeed.tech/tags/release.md>), [support](<https://devfeed.tech/tags/support.md>)

### AI overview

DuckDB 2.0.0 adds practical improvements across its Python client, CLI, SQL, and data export. Python gains an appender that buffers rows and flushes them when its context block exits; for data already in a DataFrame or Arrow table, `INSERT INTO ... SELECT` remains faster. The CLI adds formatted schema output, function help via `.manual`, calculator expressions, and queryable command history. SQL changes include `SECURE VIEW` to hide underlying rows from statistics and query plans, `SHOW` for settings, sequence reset with `setval`, and prepared-statement fallback to session variables. Export features include custom partition paths and ordering, plus GeoJSON and GeoJSONL output. Extensions can define window functions, and extension pages now list version and platform compatibility. The text also notes that `SHOW` for describing tables is deprecated in favor of `DESCRIBE`, and that not every extension is available on every version and platform.

### Source excerpt

We released DuckDB 2.0.0 last week with an improved Quack protocol and `VARIANT` support, asynchronous I/O and many other features. In this blog post, we highlight a few smaller features and improvements that did not make it into the release announcement.

## Shard design doesn't start from your tables

DevFeed: [Shard design doesn't start from your tables](<https://devfeed.tech/articles/shard-design-doesn-t-start-from-your-tables-98539.md>)

Original publisher: [Read original article](<https://planetscale.com/blog/shard-groups-the-unit-of-co-location>)

Author: Rohit Nayak

Published: 2026-10-23T00:00:00Z

Content type: article

Language: en

Sources: [PlanetScale](<https://devfeed.tech/sources/blog-planetscale.md>)

Topics: [table-partitioning](<https://devfeed.tech/topics/table-partitioning.md>), [schema design](<https://devfeed.tech/topics/schema-design.md>), [Transactions](<https://devfeed.tech/topics/transactions.md>), [rdbms](<https://devfeed.tech/topics/rdbms.md>), [JOIN](<https://devfeed.tech/topics/join.md>)

Tags: [64-bit](<https://devfeed.tech/tags/64-bit.md>), [aggregate](<https://devfeed.tech/tags/aggregate.md>), [algorithm](<https://devfeed.tech/tags/algorithm.md>), [atomic](<https://devfeed.tech/tags/atomic.md>), [best](<https://devfeed.tech/tags/best.md>), [can](<https://devfeed.tech/tags/can.md>), [catalog](<https://devfeed.tech/tags/catalog.md>), [change](<https://devfeed.tech/tags/change.md>), [clusters](<https://devfeed.tech/tags/clusters.md>), [column](<https://devfeed.tech/tags/column.md>), [commit](<https://devfeed.tech/tags/commit.md>), [concepts](<https://devfeed.tech/tags/concepts.md>), [copy](<https://devfeed.tech/tags/copy.md>), [cross](<https://devfeed.tech/tags/cross.md>), [customer](<https://devfeed.tech/tags/customer.md>), [design](<https://devfeed.tech/tags/design.md>), [engineering](<https://devfeed.tech/tags/engineering.md>), [key](<https://devfeed.tech/tags/key.md>), [neki](<https://devfeed.tech/tags/neki.md>), [postgres](<https://devfeed.tech/tags/postgres.md>), [queries](<https://devfeed.tech/tags/queries.md>), [tables](<https://devfeed.tech/tags/tables.md>), [transactions](<https://devfeed.tech/tags/transactions.md>)

### AI overview

Design shard groups around keys shared by important queries and transactions: co-locate related tables so joins and transactions stay on one shard. In Neki, matching shard keys and hashing place rows with the same key together; reference tables are copied to each listed shard, keeping joins local. Cross-shard transactions lack atomic commit and a shared snapshot. The article describes `xxhash` as its recommended index, `range` for direct integer ranges, and `modulo` as predictable but costly to change at scale. A global secondary index can support lookups by a non-shard key through two single-shard reads. MoveTables imports data and can create initial reference-table copies; Reshard changes shard keys or shard counts as query needs evolve.

### Source excerpt

Shard group design starts from your queries, not your tables. Find the key that your most important queries and transactions share, and put every table that carries that key in one group.

## Old tech is sometimes better - a case study based on blogpost from Now-Next...

DevFeed: [Old tech is sometimes better - a case study based on blogpost from Now-Next...](<https://devfeed.tech/articles/old-tech-is-sometimes-better-a-case-study-based-on-blogpost-from-now-next-88510.md>)

Original publisher: [Read original article](<https://www.depesz.com/2026/10/16/old-tech-is-sometimes-better-a-case-study-based-on-blogpost-from-now-next/>)

Author: depesz

Published: 2026-10-16T10:09:03Z

Content type: article

Language: en

Sources: [select \* from depesz;](<https://devfeed.tech/sources/select-from-depesz.md>)

Topics: [table-partitioning](<https://devfeed.tech/topics/table-partitioning.md>)

Tags: [audit](<https://devfeed.tech/tags/audit.md>), [benchmark](<https://devfeed.tech/tags/benchmark.md>), [changes](<https://devfeed.tech/tags/changes.md>), [hstore](<https://devfeed.tech/tags/hstore.md>), [jsonb](<https://devfeed.tech/tags/jsonb.md>), [logging](<https://devfeed.tech/tags/logging.md>), [now-next](<https://devfeed.tech/tags/now-next.md>), [postgresql](<https://devfeed.tech/tags/postgresql.md>), [reaction](<https://devfeed.tech/tags/reaction.md>), [trigger](<https://devfeed.tech/tags/trigger.md>), [uncategorized](<https://devfeed.tech/tags/uncategorized.md>)

### AI overview

The author benchmarks PostgreSQL trigger-based change logging using JSONB and hstore, comparing full-row logging with logging only changed columns. In the reported tests, hstore is faster and uses somewhat less disk space, with diff logging faster than full-row logging. The author cautions that hstore is not necessarily better for every use case.

### Source excerpt

Some time ago I read this blogpost. They wrote about logging changes to table using a trigger, but one thing that caught my attention was: Storing only the changed columns made the log five times smaller and the trigger slower. Figured I can try it myself... Unfortunately they didn't provide schema of table, nor test ... Continue reading "Old tech is sometimes better - a case study based on blogpost from Now-Next..."

## Apache Iceberg Architecture Explained

DevFeed: [Apache Iceberg Architecture Explained](<https://devfeed.tech/articles/apache-iceberg-architecture-explained-88575.md>)

Original publisher: [Read original article](<https://www.starburst.io/blog/apache-iceberg-architecture-explained>)

Published: 2026-10-16T08:00:04Z

Content type: article

Language: en

Sources: [Starburst Data](<https://devfeed.tech/sources/starburst-data.md>)

Topics: [Apache Iceberg](<https://devfeed.tech/topics/apache-iceberg.md>), [parquet](<https://devfeed.tech/topics/parquet.md>), [data lake](<https://devfeed.tech/topics/data-lake.md>), [table-partitioning](<https://devfeed.tech/topics/table-partitioning.md>), [Amazon S3](<https://devfeed.tech/topics/amazon-s3.md>)

Tags: [2](<https://devfeed.tech/tags/2.md>), [3](<https://devfeed.tech/tags/3.md>), [analytics](<https://devfeed.tech/tags/analytics.md>), [apache](<https://devfeed.tech/tags/apache.md>), [apache-iceberg](<https://devfeed.tech/tags/apache-iceberg.md>), [apache-iceberg-architecture](<https://devfeed.tech/tags/apache-iceberg-architecture.md>), [architecture](<https://devfeed.tech/tags/architecture.md>), [format](<https://devfeed.tech/tags/format.md>), [high-performance](<https://devfeed.tech/tags/high-performance.md>), [iceberg](<https://devfeed.tech/tags/iceberg.md>), [object](<https://devfeed.tech/tags/object.md>), [object-storage](<https://devfeed.tech/tags/object-storage.md>), [open](<https://devfeed.tech/tags/open.md>), [performance](<https://devfeed.tech/tags/performance.md>), [storage](<https://devfeed.tech/tags/storage.md>), [table](<https://devfeed.tech/tags/table.md>), [table-format](<https://devfeed.tech/tags/table-format.md>)

### AI overview

Apache Iceberg tracks tables through a catalog pointer to immutable metadata and data files, rather than inferring table contents from directory layouts. Reads use snapshot and manifest statistics to prune files; writes create new files and metadata, then use an optimistic, atomic catalog pointer swap. This supports concurrent engine access, snapshots and time travel, while row level deletes and updates use delete files. The document distinguishes v2 position and equality deletes from v3 deletion vectors, which replace position deletes, and notes v3's VARIANT type and other additions. Engine and catalog compatibility must be checked before upgrading. Ongoing maintenance--compaction, snapshot expiration, orphan file removal and manifest optimization--helps control small file fragmentation, metadata growth and storage costs.

### Source excerpt

Apache Iceberg is an open table format designed for high-performance analytics running on object storage. ...

## How Ridgeline unified Iceberg and ClickHouse into one analytics layer for investment management

DevFeed: [How Ridgeline unified Iceberg and ClickHouse into one analytics layer for investment management](<https://devfeed.tech/articles/how-ridgeline-unified-iceberg-and-clickhouse-into-one-analytics-layer-for-investment-management-87284.md>)

Original publisher: [Read original article](<https://clickhouse.com/blog/ridgeline-iceberg-clickhouse-cloud>)

Author: ClickHouse

Published: 2026-10-14T17:36:29Z

Content type: article

Language: en

Sources: [ClickHouse Blog](<https://devfeed.tech/sources/clickhouse-blog.md>)

Topics: [clickhouse](<https://devfeed.tech/topics/clickhouse.md>), [Apache Kafka](<https://devfeed.tech/topics/apache-kafka.md>), [table-partitioning](<https://devfeed.tech/topics/table-partitioning.md>), [Apache Spark](<https://devfeed.tech/topics/spark.md>), [Amazon Redshift](<https://devfeed.tech/topics/amazon-redshift.md>)

Tags: [architecture](<https://devfeed.tech/tags/architecture.md>), [aws](<https://devfeed.tech/tags/aws.md>), [clickhouse](<https://devfeed.tech/tags/clickhouse.md>), [cloud](<https://devfeed.tech/tags/cloud.md>), [cloud-native](<https://devfeed.tech/tags/cloud-native.md>), [cost](<https://devfeed.tech/tags/cost.md>), [data](<https://devfeed.tech/tags/data.md>), [deployments](<https://devfeed.tech/tags/deployments.md>), [iceberg](<https://devfeed.tech/tags/iceberg.md>), [it](<https://devfeed.tech/tags/it.md>), [performance](<https://devfeed.tech/tags/performance.md>), [querying](<https://devfeed.tech/tags/querying.md>)

### AI overview

Ridgeline moved its investment performance analytics from Postgres to ClickHouse Cloud, using Iceberg for older history and ClickHouse for recent data. The architecture handles larger customer workloads, cut reporting batches from hours to minutes, and helped the company more than double the customers live on its platform.

### Source excerpt

Ridgeline rebuilt its performance analytics on ClickHouse Cloud, querying decades of Iceberg history alongside recent data so it can serve investment managers 10x larger than before and more than double the customers live on its platform.

## What Is Inside an Iceberg Table?

DevFeed: [What Is Inside an Iceberg Table?](<https://devfeed.tech/articles/what-is-inside-an-iceberg-table-87646.md>)

Original publisher: [Read original article](<https://duckdb.org/2026/10/13/inside-iceberg-table.html>)

Author: The DuckDB team

Published: 2026-10-13T00:00:00Z

Content type: article

Language: en

Sources: [DuckDB](<https://devfeed.tech/sources/duckdb.md>)

Topics: [Apache Iceberg](<https://devfeed.tech/topics/apache-iceberg.md>), [DuckDB](<https://devfeed.tech/topics/duckdb.md>), [table-partitioning](<https://devfeed.tech/topics/table-partitioning.md>), [parquet](<https://devfeed.tech/topics/parquet.md>)

Tags: [2](<https://devfeed.tech/tags/2.md>), [2026](<https://devfeed.tech/tags/2026.md>), [can](<https://devfeed.tech/tags/can.md>), [data](<https://devfeed.tech/tags/data.md>), [deep-dive](<https://devfeed.tech/tags/deep-dive.md>), [delete](<https://devfeed.tech/tags/delete.md>), [duckdb](<https://devfeed.tech/tags/duckdb.md>), [file](<https://devfeed.tech/tags/file.md>), [files](<https://devfeed.tech/tags/files.md>), [iceberg](<https://devfeed.tech/tags/iceberg.md>), [inside](<https://devfeed.tech/tags/inside.md>), [json](<https://devfeed.tech/tags/json.md>), [list](<https://devfeed.tech/tags/list.md>), [manifest](<https://devfeed.tech/tags/manifest.md>), [metadata](<https://devfeed.tech/tags/metadata.md>), [parquet](<https://devfeed.tech/tags/parquet.md>), [parquet-files](<https://devfeed.tech/tags/parquet-files.md>), [sql](<https://devfeed.tech/tags/sql.md>), [table](<https://devfeed.tech/tags/table.md>), [what-is](<https://devfeed.tech/tags/what-is.md>), [with](<https://devfeed.tech/tags/with.md>)

### AI overview

An Iceberg table is an immutable file tree whose current version is selected by a catalog pointer to a metadata JSON file. Each snapshot points to an Avro manifest list and manifests, which describe Parquet data and delete files. With DuckDB's Iceberg and Avro extensions, SQL can inspect snapshots, file counts, partition values, and column bounds to investigate unexpected row counts or file reads. The example shows deletes recorded as position-delete Parquet files while leaving data files unchanged: this makes writes cheaper, but readers must combine delete and data files, and the cost grows with delete files. Small manifests can similarly reduce write effort while slowing reads. Snapshot summary counters are optional and may be absent for tables written by other engines. The article also notes that Iceberg format version 3 stores deletion vectors in Puffin files.

### Source excerpt

An Iceberg table is a tree of JSON, Avro and Parquet files. DuckDB can read every one of them, so you can walk the tree yourself with SQL: from the metadata file, through the manifest list and manifests, down to the data files and delete files.

## Apache Iceberg Table Statistics: NDV, Puffin and Query Plans

DevFeed: [Apache Iceberg Table Statistics: NDV, Puffin and Query Plans](<https://devfeed.tech/articles/apache-iceberg-table-statistics-ndv-puffin-and-query-plans-85690.md>)

Original publisher: [Read original article](<https://www.dremio.com/blog/apache-iceberg-table-statistics/>)

Author: Alex Merced

Published: 2026-10-12T19:28:22Z

Content type: tutorial

Language: en

Sources: [Dremio](<https://devfeed.tech/sources/dremio.md>)

Topics: [Apache Iceberg](<https://devfeed.tech/topics/apache-iceberg.md>), [table-partitioning](<https://devfeed.tech/topics/table-partitioning.md>), [Statistics](<https://devfeed.tech/topics/statistics.md>)

Tags: [analytics](<https://devfeed.tech/tags/analytics.md>), [apache-iceberg](<https://devfeed.tech/tags/apache-iceberg.md>), [count](<https://devfeed.tech/tags/count.md>), [data-lakehouse](<https://devfeed.tech/tags/data-lakehouse.md>), [datasketches](<https://devfeed.tech/tags/datasketches.md>), [distribution](<https://devfeed.tech/tags/distribution.md>), [dremio-blog-open-data-insights](<https://devfeed.tech/tags/dremio-blog-open-data-insights.md>), [query](<https://devfeed.tech/tags/query.md>), [statistics](<https://devfeed.tech/tags/statistics.md>)

### AI overview

The article explains how Apache Iceberg's automatically recorded per-file bounds help prune scans, while explicitly computed table-level NDV statistics in Puffin files help query optimizers estimate join sizes and choose join order. It outlines Spark procedures for computing table and partition statistics, recommends refreshing them as data distributions change, and advises checking whether the query engine uses them by comparing plans before and after computation.

### Source excerpt

Iceberg keeps per-file bounds automatically but table-level NDV statistics only when you compute them. Why that decides join order, and how to keep them fresh. The post Apache Iceberg Table Statistics: NDV, Puffin and Query Plans appeared first on Dremio.

## Article: When Should You Use Liquid Clustering? Benchmarking Lakehouse Layout Strategies

DevFeed: [Article: When Should You Use Liquid Clustering? Benchmarking Lakehouse Layout Strategies](<https://devfeed.tech/articles/article-when-should-you-use-liquid-clustering-benchmarking-lakehouse-layout-strategies-85212.md>)

Original publisher: [Read original article](<https://www.infoq.com/articles/liquid-clustering-lakehouse-layout/>)

Author: Kushal Kumar Vishwakarma

Published: 2026-10-12T11:00:00Z

Content type: article

Language: en

Sources: [InfoQ](<https://devfeed.tech/sources/infoq.md>)

Topics: [deltalake](<https://devfeed.tech/topics/deltalake.md>), [table-partitioning](<https://devfeed.tech/topics/table-partitioning.md>), [clustering](<https://devfeed.tech/topics/clustering.md>)

Tags: [ai](<https://devfeed.tech/tags/ai.md>), [ai-ml-data-engineering](<https://devfeed.tech/tags/ai-ml-data-engineering.md>), [application-performance-management](<https://devfeed.tech/tags/application-performance-management.md>), [architecture-design](<https://devfeed.tech/tags/architecture-design.md>), [article](<https://devfeed.tech/tags/article.md>), [benchmark](<https://devfeed.tech/tags/benchmark.md>), [benchmarking](<https://devfeed.tech/tags/benchmarking.md>), [clustering-caching](<https://devfeed.tech/tags/clustering-caching.md>), [data-lake](<https://devfeed.tech/tags/data-lake.md>), [data-storage](<https://devfeed.tech/tags/data-storage.md>), [delta-lake](<https://devfeed.tech/tags/delta-lake.md>), [engineering](<https://devfeed.tech/tags/engineering.md>), [infrastructure](<https://devfeed.tech/tags/infrastructure.md>), [lakehouse](<https://devfeed.tech/tags/lakehouse.md>), [liquid-clustering-lakehouse-layout](<https://devfeed.tech/tags/liquid-clustering-lakehouse-layout.md>), [ml-data-engineering](<https://devfeed.tech/tags/ml-data-engineering.md>), [partitioning](<https://devfeed.tech/tags/partitioning.md>), [performance](<https://devfeed.tech/tags/performance.md>), [performance-evaluation](<https://devfeed.tech/tags/performance-evaluation.md>), [query](<https://devfeed.tech/tags/query.md>)

### AI overview

The article benchmarks Liquid Clustering, ZOrdering, and partitioning on Delta Lake across four analytical workloads. Liquid Clustering has the lowest latency for point lookups, time-range aggregations, and large aggregations, while ZOrder performs best for selective multicolumn queries whose filters match its clustering keys. The author recommends validating layout choices against representative query patterns and notes that the benchmark uses synthetic data, so results may vary in production.

### Source excerpt

Physical data layout is one of the most consequential decisions in lakehouse engineering. The author compares Liquid Clustering, ZOrdering, and partitioning across four workload types on Delta Lake, uncovering two undocumented failure modes that likely affect production systems today and offering a decision framework for choosing the right layout strategy. By Kushal Kumar Vishwakarma

## How I Fixed a Data Ingestion Bottleneck With Date Partitioning

DevFeed: [How I Fixed a Data Ingestion Bottleneck With Date Partitioning](<https://devfeed.tech/articles/how-i-fixed-a-data-ingestion-bottleneck-with-date-partitioning-85090.md>)

Original publisher: [Read original article](<https://hackernoon.com/how-i-fixed-a-data-ingestion-bottleneck-with-date-partitioning?source=rss>)

Author: Rahul N

Published: 2026-10-11T19:36:29Z

Content type: tutorial

Language: en

Sources: [HackerNoon](<https://devfeed.tech/sources/hackernoon.md>)

Topics: [data-engineering](<https://devfeed.tech/topics/data-engineering.md>), [Data Quality](<https://devfeed.tech/topics/data-quality.md>), [table-partitioning](<https://devfeed.tech/topics/table-partitioning.md>)

Tags: [compatibility](<https://devfeed.tech/tags/compatibility.md>), [data](<https://devfeed.tech/tags/data.md>), [data-engineering](<https://devfeed.tech/tags/data-engineering.md>), [data-ingestion](<https://devfeed.tech/tags/data-ingestion.md>), [data-ingestion-pipelines](<https://devfeed.tech/tags/data-ingestion-pipelines.md>), [data-partitioning](<https://devfeed.tech/tags/data-partitioning.md>), [data-pipeline-reliability](<https://devfeed.tech/tags/data-pipeline-reliability.md>), [delivery](<https://devfeed.tech/tags/delivery.md>), [distributed-systems](<https://devfeed.tech/tags/distributed-systems.md>), [exactly-once](<https://devfeed.tech/tags/exactly-once.md>), [failed](<https://devfeed.tech/tags/failed.md>), [idempotent-data-processing](<https://devfeed.tech/tags/idempotent-data-processing.md>), [incremental-data-ingestion](<https://devfeed.tech/tags/incremental-data-ingestion.md>), [object-storage](<https://devfeed.tech/tags/object-storage.md>), [partitioning](<https://devfeed.tech/tags/partitioning.md>)

### AI overview

A data engineer worked with a vendor to add date-partitioned exports, letting the consumer target date-specific prefixes instead of enumerating the full export history. The article explains the benefits for discovery, retries, and backfills, along with late-arrival policies, replay safety, and migration planning. It describes an engineering approach rather than a benchmark or a guarantee of runtime improvements.

### Source excerpt

I fixed a recurring data ingestion bottleneck by working with a vendor to introduce date-partitioned exports and reduce unnecessary file discovery.

## How Meshtastic Metrics Exporter Turned Eight Prometheus Queries Into One with Tiger Data

DevFeed: [How Meshtastic Metrics Exporter Turned Eight Prometheus Queries Into One with Tiger Data](<https://devfeed.tech/articles/how-meshtastic-metrics-exporter-turned-eight-prometheus-queries-into-one-with-tiger-data-66753.md>)

Original publisher: [Read original article](<https://hackernoon.com/how-meshtastic-metrics-exporter-turned-eight-prometheus-queries-into-one-with-tiger-data?source=rss>)

Author: Tiger Data (creators of TimescaleDB)

Published: 2026-10-07T08:30:07Z

Content type: article

Language: en

Sources: [HackerNoon](<https://devfeed.tech/sources/hackernoon.md>)

Topics: [Grafana](<https://devfeed.tech/topics/grafana.md>), [table-partitioning](<https://devfeed.tech/topics/table-partitioning.md>), [telemetry](<https://devfeed.tech/topics/telemetry.md>), [pgwatch](<https://devfeed.tech/topics/pgwatch.md>)

Tags: [data](<https://devfeed.tech/tags/data.md>), [good-company](<https://devfeed.tech/tags/good-company.md>), [grafana](<https://devfeed.tech/tags/grafana.md>), [high-cardinality-metrics](<https://devfeed.tech/tags/high-cardinality-metrics.md>), [meshtastic-grafana-dashboards](<https://devfeed.tech/tags/meshtastic-grafana-dashboards.md>), [meshtastic-metrics-exporter](<https://devfeed.tech/tags/meshtastic-metrics-exporter.md>), [meshtastic-monitoring-database](<https://devfeed.tech/tags/meshtastic-monitoring-database.md>), [metrics](<https://devfeed.tech/tags/metrics.md>), [postgres](<https://devfeed.tech/tags/postgres.md>), [postgres-time-series](<https://devfeed.tech/tags/postgres-time-series.md>), [prometheus](<https://devfeed.tech/tags/prometheus.md>), [prometheus-cardinality-limits](<https://devfeed.tech/tags/prometheus-cardinality-limits.md>), [timescaledb](<https://devfeed.tech/tags/timescaledb.md>), [timescaledb-iot-telemetry](<https://devfeed.tech/tags/timescaledb-iot-telemetry.md>)

### AI overview

Meshtastic Metrics Exporter moved from Prometheus to TimescaleDB on Postgres after high-cardinality metrics from a growing mesh exceeded Prometheus's scrape budget. The change combined node metadata and time-series telemetry in one database, enabling Grafana dashboards to correlate node details with historical data through a single query. The project runs as three Docker containers, while retention and compression are handled through database policies.

### Source excerpt

See how Meshtastic Metrics Exporter moved from Prometheus to TimescaleDB, combining node metadata and time-series telemetry in one Postgres database.

## Query across accounts and table formats with multi-catalog in Amazon EMR

DevFeed: [Query across accounts and table formats with multi-catalog in Amazon EMR](<https://devfeed.tech/articles/query-across-accounts-and-table-formats-with-multi-catalog-in-amazon-emr-65456.md>)

Original publisher: [Read original article](<https://aws.amazon.com/blogs/big-data/query-across-accounts-and-table-formats-with-multi-catalog-in-amazon-emr/>)

Author: Suthan Phillips

Published: 2026-10-06T16:05:27Z

Content type: tutorial

Language: en

Sources: [AWS Big Data Blog](<https://devfeed.tech/sources/aws-big-data-blog.md>)

Topics: [table-partitioning](<https://devfeed.tech/topics/table-partitioning.md>), [Amazon Web Services](<https://devfeed.tech/topics/aws.md>), [Amazon S3 Tables](<https://devfeed.tech/topics/amazon-s3-tables.md>), [data lake](<https://devfeed.tech/topics/data-lake.md>)

Tags: [access-control](<https://devfeed.tech/tags/access-control.md>), [advanced-300](<https://devfeed.tech/tags/advanced-300.md>), [amazon-emr](<https://devfeed.tech/tags/amazon-emr.md>), [amazon-emr-on-eks](<https://devfeed.tech/tags/amazon-emr-on-eks.md>), [amazon-redshift](<https://devfeed.tech/tags/amazon-redshift.md>), [amazon-s3](<https://devfeed.tech/tags/amazon-s3.md>), [announcements](<https://devfeed.tech/tags/announcements.md>), [apache-hudi](<https://devfeed.tech/tags/apache-hudi.md>), [apache-iceberg](<https://devfeed.tech/tags/apache-iceberg.md>), [api](<https://devfeed.tech/tags/api.md>), [applications](<https://devfeed.tech/tags/applications.md>), [aws-emr](<https://devfeed.tech/tags/aws-emr.md>), [aws-glue](<https://devfeed.tech/tags/aws-glue.md>), [aws-key-management-service](<https://devfeed.tech/tags/aws-key-management-service.md>), [aws-lake-formation](<https://devfeed.tech/tags/aws-lake-formation.md>), [data-catalog](<https://devfeed.tech/tags/data-catalog.md>), [data-mesh](<https://devfeed.tech/tags/data-mesh.md>), [database](<https://devfeed.tech/tags/database.md>), [databases](<https://devfeed.tech/tags/databases.md>), [delta-lake](<https://devfeed.tech/tags/delta-lake.md>)

### AI overview

Amazon EMR 8.1.0 adds multi-catalog support that routes Iceberg, Delta Lake, Hudi, and Hive tables through one catalog and enables queries joining data across AWS accounts without copying it. The tutorial demonstrates configuring the RedirectingSessionCatalog and optionally using Glue catalog resolution for discovering remote catalogs at query time.

### Source excerpt

Amazon EMR 8.1.0 adds multi-catalog support through the RedirectingSessionCatalog, so you can query Iceberg, Delta Lake, Hudi, and Hive tables through a single catalog and join data across AWS accounts without copying it. This post shows how to put these capabilities into practice on Amazon EMR Serverless.

## Building a Unified Observability Platform on GreptimeDB: A Reference Architecture

DevFeed: [Building a Unified Observability Platform on GreptimeDB: A Reference Architecture](<https://devfeed.tech/articles/building-a-unified-observability-platform-on-greptimedb-a-reference-architecture-65202.md>)

Original publisher: [Read original article](<https://blog.devops.dev/building-a-unified-observability-platform-on-greptimedb-a-reference-architecture-35c754365411?source=rss----33f8b2d9a328---4>)

Author: Greptime

Published: 2026-10-05T20:11:01Z

Content type: article

Language: en

Sources: [DevOps.dev](<https://devfeed.tech/sources/devops-dev.md>)

Topics: [observability pipelines](<https://devfeed.tech/topics/observability-pipelines.md>), [OpenTelemetry](<https://devfeed.tech/topics/opentelemetry.md>), [Grafana](<https://devfeed.tech/topics/grafana.md>), [Prometheus](<https://devfeed.tech/topics/prometheus.md>), [table-partitioning](<https://devfeed.tech/topics/table-partitioning.md>)

Tags: [agent](<https://devfeed.tech/tags/agent.md>), [alerts](<https://devfeed.tech/tags/alerts.md>), [analysis](<https://devfeed.tech/tags/analysis.md>), [architecture](<https://devfeed.tech/tags/architecture.md>), [article](<https://devfeed.tech/tags/article.md>), [astronomy](<https://devfeed.tech/tags/astronomy.md>), [cluster](<https://devfeed.tech/tags/cluster.md>), [collectors](<https://devfeed.tech/tags/collectors.md>), [components](<https://devfeed.tech/tags/components.md>), [configuration](<https://devfeed.tech/tags/configuration.md>), [demo](<https://devfeed.tech/tags/demo.md>), [deploy](<https://devfeed.tech/tags/deploy.md>), [devops](<https://devfeed.tech/tags/devops.md>), [enterprise](<https://devfeed.tech/tags/enterprise.md>), [incident](<https://devfeed.tech/tags/incident.md>), [investigation](<https://devfeed.tech/tags/investigation.md>), [jaeger](<https://devfeed.tech/tags/jaeger.md>), [large](<https://devfeed.tech/tags/large.md>), [observability](<https://devfeed.tech/tags/observability.md>), [opentelemetry](<https://devfeed.tech/tags/opentelemetry.md>), [opentelemetry-demo](<https://devfeed.tech/tags/opentelemetry-demo.md>), [platform](<https://devfeed.tech/tags/platform.md>), [reference-architecture](<https://devfeed.tech/tags/reference-architecture.md>), [tutorial](<https://devfeed.tech/tags/tutorial.md>)

### AI overview

The architecture routes metrics, logs, and traces into one GreptimeDB while leaving collectors in place: dashboards and alerts use PromQL, and investigation and long-range analysis use SQL. The excerpt describes protocol endpoints, per-signal table and indexing choices, and three deployment tiers. Flow can maintain small aggregate tables for dashboards and alerts; frontend groups isolate query handling, while storage isolation through Read Replicas or Datanode groups is Enterprise-only. Object storage supports longer retention; Local WAL suits standalone and small clusters, while Remote WAL uses Kafka for faster recovery. Migration is described as dual-write, switch reads, then retire the old system. Compatibility has limits: Loki writes are supported but LogQL queries move to SQL; open-source Elasticsearch support covers bulk writes, not Query DSL. The article also reports better agent-diagnosis results on GreptimeDB, while cautioning that its comparison changed the full interface bundle and does not isolate the storage engine's contribution.

### Source excerpt

This article assumes you have already decided to put metrics, logs, and traces into one database. Why unify, and why now, is covered in our earlier post on the history of observability unification. This one covers what comes after that decision: which components to deploy, how data flows, which decisions are still yours to make, and what the topology looks like at three scales: standalone, small cluster, and large cluster. The architecture fits in one sentence: leave the collectors alone and point them at one GreptimeDB; dashboards and alerts go through PromQL, incident investigation and long-range analysis through SQL. The validation environment is the official OpenTelemetry Astronomy Shop demo: 14 services, SDKs in several languages, and a default backend of Jaeger, OpenSearch, and Prometheus. We replaced those three with a single GreptimeDB v1.2.0 standalone instance, plus a node_exporter and a Prometheus that only scrapes. The configuration, table schemas, query results, and incident records in this article come from that environment; the complete setup is in the greptimedb-observability-playground repository. Enterprise-only features are marked where they come up. Three workloads the platform has to carry Dashboards and alerts. Grafana panels refresh every few seconds and alert rules evaluate every minute, over the last few minutes to few hours of data. Many queries, each small, each latency-sensitive. Incident investigation. One investigation moves between metrics, logs, and traces: from an alert to the slow service, from the service to the slow call chain, from the call chain to the error logs and node resources of that time window. The queries are exploratory; nobody knows in advance how many there will be, and humans are no longer the only ones issuing them. In our Agent RCA Bench, one agent investigation averaged thirty to forty tool calls, with some approaching fifty. Long-range analysis. Capacity planning, weekly reports, cost attribution: scans over wee

## How MySQL Implicit Type Conversion Turned an Indexed Query Into a 7M-Row Scan

DevFeed: [How MySQL Implicit Type Conversion Turned an Indexed Query Into a 7M-Row Scan](<https://devfeed.tech/articles/how-mysql-implicit-type-conversion-turned-an-indexed-query-into-a-7m-row-scan-64959.md>)

Original publisher: [Read original article](<https://hackernoon.com/how-mysql-implicit-type-conversion-turned-an-indexed-query-into-a-7m-row-scan?source=rss>)

Author: Oluwaseun

Published: 2026-10-05T15:49:22Z

Content type: tutorial

Language: en

Sources: [HackerNoon](<https://devfeed.tech/sources/hackernoon.md>)

Topics: [MySQL](<https://devfeed.tech/topics/mysql.md>), [database-optimization](<https://devfeed.tech/topics/database-optimization.md>), [table-partitioning](<https://devfeed.tech/topics/table-partitioning.md>), [Tech Debt](<https://devfeed.tech/topics/tech-debt.md>)

Tags: [batching](<https://devfeed.tech/tags/batching.md>), [concurrency](<https://devfeed.tech/tags/concurrency.md>), [database-administration](<https://devfeed.tech/tags/database-administration.md>), [database-indexes](<https://devfeed.tech/tags/database-indexes.md>), [implicit-type-conversion](<https://devfeed.tech/tags/implicit-type-conversion.md>), [mvcc](<https://devfeed.tech/tags/mvcc.md>), [mysql](<https://devfeed.tech/tags/mysql.md>), [mysql-data-types](<https://devfeed.tech/tags/mysql-data-types.md>), [optimize-mysql-queries](<https://devfeed.tech/tags/optimize-mysql-queries.md>), [query](<https://devfeed.tech/tags/query.md>), [sql-performance-tuning](<https://devfeed.tech/tags/sql-performance-tuning.md>), [sql-query-optimization](<https://devfeed.tech/tags/sql-query-optimization.md>)

### AI overview

A production UPDATE intended to change a few customer records risked scanning millions of rows because a numeric literal was compared with a VARCHAR column, leading MySQL to apply implicit type conversion and bypass the index. The article describes correcting the literal, accounting for a date filter absent from the composite index, and safely extracting target IDs before updating them in batches with production safeguards.

### Source excerpt

A missing quote made MySQL scan 7.2 million rows. Here's how EXPLAIN exposed the problem and what it taught us about safe production updates.

## Normalizing a Wide Well and Pipeline Telemetry Table Without Re-Ingesting History

DevFeed: [Normalizing a Wide Well and Pipeline Telemetry Table Without Re-Ingesting History](<https://devfeed.tech/articles/normalizing-a-wide-well-and-pipeline-telemetry-table-without-re-ingesting-history-62632.md>)

Original publisher: [Read original article](<https://www.tigerdata.com/blog/normalizing-wide-well-pipeline-telemetry-table-without-re-ingesting-history>)

Author: Damaso Sanoja

Published: 2026-09-30T13:58:45Z

Content type: tutorial

Language: en

Sources: [Timescale](<https://devfeed.tech/sources/timescale.md>)

Topics: [table-partitioning](<https://devfeed.tech/topics/table-partitioning.md>)

Tags: [iot](<https://devfeed.tech/tags/iot.md>), [pipeline](<https://devfeed.tech/tags/pipeline.md>), [table](<https://devfeed.tech/tags/table.md>), [telemetry](<https://devfeed.tech/tags/telemetry.md>), [wide](<https://devfeed.tech/tags/wide.md>)

### AI overview

The article describes reshaping a wide oilfield SCADA telemetry table into a narrow hypertable with tag metadata, while keeping the change in place and dashboards running without re-ingesting historical data.

### Source excerpt

Reshape a wide oilfield SCADA table into a narrow hypertable plus tag metadata, in place, with dashboards running and history left where it is.

## Dasha 1.8: index recommendations, I/O analysis, schema checks and log insights

DevFeed: [Dasha 1.8: index recommendations, I/O analysis, schema checks and log insights](<https://devfeed.tech/articles/dasha-1-8-index-recommendations-i-o-analysis-schema-checks-and-log-insights-61633.md>)

Original publisher: [Read original article](<https://www.postgresql.org/about/news/dasha-18-index-recommendations-io-analysis-schema-checks-and-log-insights-3387/>)

Published: 2026-09-29T00:00:00Z

Content type: release

Language: en

Sources: [PostgreSQL news](<https://devfeed.tech/sources/postgresql-news.md>)

Topics: [table-partitioning](<https://devfeed.tech/topics/table-partitioning.md>), [read-replica](<https://devfeed.tech/topics/read-replica.md>), [Transactions](<https://devfeed.tech/topics/transactions.md>), [schema design](<https://devfeed.tech/topics/schema-design.md>), [transaction-log](<https://devfeed.tech/topics/transaction-log.md>), [rate-limiting](<https://devfeed.tech/topics/rate-limiting.md>)

Tags: [context](<https://devfeed.tech/tags/context.md>), [cve](<https://devfeed.tech/tags/cve.md>), [database](<https://devfeed.tech/tags/database.md>), [indexes](<https://devfeed.tech/tags/indexes.md>), [performance](<https://devfeed.tech/tags/performance.md>), [postgresql](<https://devfeed.tech/tags/postgresql.md>)

### AI overview

Dasha 1.8 adds PostgreSQL index recommendations, I/O analysis, schema checks and log insights. It analyzes statistics across cluster hosts and replicas, suggests B-tree indexes with supporting query and execution details, and adds schema findings with remediation SQL. Log search supports additional providers, and query plan comparisons can identify statements whose plans worsened across time windows.

### Source excerpt

Dasha is an open source performance dashboard for PostgreSQL fleets. It connects to your clusters with a read-only role, shows what the databases are doing right now, and explains what to do about it. Nothing is installed on the database hosts: no agent, and no extension beyond the ones you already have. One instance serves many clusters and all of their replicas. Most PostgreSQL statistics are per-instance and are not replicated, so Dasha reads every host of a cluster and reasons over the combined picture. An index that looks unused on the primary may be serving the entire read workload on a standby. New since the last announcement Index recommendations. Dasha reads pg_stat_statements on every host of a cluster and proposes the B-tree indexes that are missing. Each candidate comes with its column order, a ready CREATE INDEX CONCURRENTLY, the statements it covers and their share of execution time. Write counters and partitioned tables (ATTACH PARTITION) are taken into account. A candidate can be backed by auto_explain plans from the log: sequential scans of its table, what they cost and how many rows their filters discarded. A misestimated row count raises a warning to run ANALYZE first. I/O page built on pg_stat_io (PostgreSQL 16+). Server I/O is broken down by backend type, object and context (normal, vacuum, bulkread, bulkwrite), with WAL on PostgreSQL 18. There is a live mode and snapshot history, and cards for shared_buffers efficiency, vacuum cost and bulk operations. Schema checks. 17 checks for structural defects that stay invisible until something breaks, including sequences close to exhaustion, tables without a primary or unique key, forgotten UNLOGGED relations, a public schema anyone can create objects in (CVE-2018-1058), UUIDs stored as varchar, nullable foreign keys, NOT VALID constraints that were never validated and reserved keywords used as names. Each finding has a severity and remediation SQL. Log insights. Log search now works with OpenSearch, El

## Partition Finalization in Pinterest's Next-Generation DB Ingestion Framework

DevFeed: [Partition Finalization in Pinterest's Next-Generation DB Ingestion Framework](<https://devfeed.tech/articles/partition-finalization-in-pinterest-s-next-generation-db-ingestion-framework-60046.md>)

Original publisher: [Read original article](<https://medium.com/pinterest-engineering/partition-finalization-in-pinterests-next-generation-db-ingestion-framework-4c7da6e4cc8f?source=rss----4c5a5f6279b6---4>)

Author: Pinterest Engineering

Published: 2026-09-25T15:01:03Z

Content type: article

Language: en

Sources: [Pinterest](<https://devfeed.tech/sources/pinterest-engineering-blog-medium.md>)

Topics: [data-engineering](<https://devfeed.tech/topics/data-engineering.md>), [Data pipelines](<https://devfeed.tech/topics/data-pipelines.md>), [stream-processing](<https://devfeed.tech/topics/stream-processing.md>), [Apache Flink](<https://devfeed.tech/topics/apache-flink.md>), [Apache Iceberg](<https://devfeed.tech/topics/apache-iceberg.md>), [table-partitioning](<https://devfeed.tech/topics/table-partitioning.md>)

Tags: [agent](<https://devfeed.tech/tags/agent.md>), [automated](<https://devfeed.tech/tags/automated.md>), [batch](<https://devfeed.tech/tags/batch.md>), [challenge](<https://devfeed.tech/tags/challenge.md>), [complete](<https://devfeed.tech/tags/complete.md>), [completeness](<https://devfeed.tech/tags/completeness.md>), [consumer](<https://devfeed.tech/tags/consumer.md>), [customers](<https://devfeed.tech/tags/customers.md>), [data](<https://devfeed.tech/tags/data.md>), [data-completeness](<https://devfeed.tech/tags/data-completeness.md>), [database](<https://devfeed.tech/tags/database.md>), [engineering](<https://devfeed.tech/tags/engineering.md>), [flink](<https://devfeed.tech/tags/flink.md>), [format](<https://devfeed.tech/tags/format.md>), [framework](<https://devfeed.tech/tags/framework.md>), [generate](<https://devfeed.tech/tags/generate.md>), [generation](<https://devfeed.tech/tags/generation.md>), [hive](<https://devfeed.tech/tags/hive.md>), [icebergs](<https://devfeed.tech/tags/icebergs.md>), [next](<https://devfeed.tech/tags/next.md>), [partition](<https://devfeed.tech/tags/partition.md>), [pinterest](<https://devfeed.tech/tags/pinterest.md>), [spark](<https://devfeed.tech/tags/spark.md>)

### AI overview

Pinterest's streaming ingestion framework restores a reliable "safe to read" signal by deriving partition finalization from event time. The Flink-to-Iceberg sink records per-checkpoint event-time statistics in snapshot metadata, including compact percentile sketches, then computes a watermark from recent commits. Its default uses the minimum event time across a configurable lookback and rounds down to the hour; the watermark never regresses, and late data can trigger an alert. The CDC table stores the watermark as an Iceberg property, which Spark copies to the base table during upserts so downstream jobs can gate processing through metadata sensors. A conservative default favors completeness; tables can use a percentile policy for fresher data while accepting a rare late record. Generic accumulator and commit-processor extensions enable the mechanism in other Flink-to-Iceberg pipelines without extra operators or an external service.

### Source excerpt

Qianrui Zhang | Sr Software Engineer, Logging Platform Kanchi Masalia | Software Engineer II, Stream Processing Platform Liang Mou | Sr Staff Software Engineer, Logging Platform Yi Pan | Principal Engineer, Agent Platform Introduction This is the third post in our series on Pinterest's next-generation database ingestion framework. Part 1 introduced the DB ingestion framework built on Kafka, Flink, Spark, and Iceberg, and Part 2 covered automated schema evolution. This post tackles another challenge in migrating downstream customers to the new ingestion framework: knowing when data is complete enough to read. We'll walk through that data completeness challenge and how we solve it: what "partition finalization" means, why it matters to downstream customers, how we built a unified mechanism to generate partition finalization markers, and how those markers are consumed. We'll also look at how this mechanism generalizes beyond DB ingestion. Background & Motivation Before explaining partition finalization, it helps to define what a partition is and why a downstream consumer cares when one partition is "done." What is a Partition A time partition is a subset of a table whose rows are grouped by a time column, e.g. all rows whose timestamp falls within a given hour. Grouping data this way lets a consumer read just the slice it needs, such as the 1 AM hour, instead of scanning the whole table. How that grouping is physically represented depends on the table format: Hive uses explicit, directory-based partitioning: each partition maps to a physical directory in S3. Iceberg uses hidden partitioning: the partitioning is defined in the metadata layer rather than by directory layout, so consumers don't need to know the physical file layout to query a partition. What Finalization Means Now consider a typical consumer of these tables: a downstream batch job that processes a single hour's partition and runs only once for that hour. The consumer faces the question: when should my job

## It's Time to Break Up with Archive Tables

DevFeed: [It's Time to Break Up with Archive Tables](<https://devfeed.tech/articles/it-s-time-to-break-up-with-archive-tables-50495.md>)

Original publisher: [Read original article](<https://sqlperformance.com/2026/07/partitioning/its-time-to-break-up-with-archive-tables>)

Author: Aaron Bertrand

Published: 2026-07-27T09:00:52Z

Content type: article

Language: en

Sources: [SQLPerformance.com](<https://devfeed.tech/sources/sqlperformance-com.md>)

Topics: [table-partitioning](<https://devfeed.tech/topics/table-partitioning.md>), [sql-server](<https://devfeed.tech/topics/sql-server.md>), [Database](<https://devfeed.tech/topics/database.md>), [transaction-log](<https://devfeed.tech/topics/transaction-log.md>), [backups](<https://devfeed.tech/topics/backups.md>)

Tags: [backups](<https://devfeed.tech/tags/backups.md>), [database-design](<https://devfeed.tech/tags/database-design.md>), [partitioning](<https://devfeed.tech/tags/partitioning.md>), [sql](<https://devfeed.tech/tags/sql.md>), [sql-performance](<https://devfeed.tech/tags/sql-performance.md>), [sql-server](<https://devfeed.tech/tags/sql-server.md>), [table-partitioning](<https://devfeed.tech/tags/table-partitioning.md>)

### AI overview

Aaron Bertrand explains why moving aging SQL Server data row by row into archive tables can create unnecessary writes and operational overhead. He presents table partitioning as an alternative for managing data retention and workload impact.

### Source excerpt

Aaron Bertrand talks about archive tables in SQL Server and how to improve the process using table partitioning. The post It's Time to Break Up with Archive Tables appeared first on SQLPerformance.com.

## A Deep Dive into Table partitioning Part 5: Partitioning an environment not individual tables

DevFeed: [A Deep Dive into Table partitioning Part 5: Partitioning an environment not individual tables](<https://devfeed.tech/articles/a-deep-dive-into-table-partitioning-part-5-partitioning-an-environment-not-individual-tables-26252.md>)

Original publisher: [Read original article](<https://medium.com/adyen/a-deep-dive-into-table-partitioning-part-5-partitioning-an-environment-not-individual-tables-a890ca302d86?source=rss----64941d9fbc09---4>)

Author: Adyen

Published: 2026-07-13T09:18:45Z

Content type: article

Language: en

Sources: [Adyen Tech](<https://devfeed.tech/sources/adyen-tech.md>)

Topics: [Database](<https://devfeed.tech/topics/database.md>), [table-partitioning](<https://devfeed.tech/topics/table-partitioning.md>), [rdbms](<https://devfeed.tech/topics/rdbms.md>)

Tags: [adyen](<https://devfeed.tech/tags/adyen.md>), [database](<https://devfeed.tech/tags/database.md>), [partitioning](<https://devfeed.tech/tags/partitioning.md>), [performance](<https://devfeed.tech/tags/performance.md>), [postgresql](<https://devfeed.tech/tags/postgresql.md>), [table-partitioning](<https://devfeed.tech/tags/table-partitioning.md>)

### AI overview

The excerpt's central lesson is to plan partitioning across a relational database environment, not table by table: many tables are tightly coupled to information in other tables. The author says this broader perspective should have come before the series' earlier focus on why, how, and when to partition and how to maintain partitioned tables. The series also describes building a custom framework because existing solutions could not minimize table locks in Adyen's continuously pressured databases, and cautions that PostgreSQL default partitions can cause problems despite occasional useful exceptions. The excerpt ends before giving the promised cross-table planning guidance.

### Source excerpt

By Derk van Veen, Database Engineer, Adyen If we had known better, this would have been the first blog post about partitioning to publish. The reality is, the first article was published two years and two weeks ago, as I write these opening sentences. So this article will not only give insight into the most important decisions you will have to make when considering your partitioning planning, but as a bonus, it will give some insights into how much it takes to write an article like this. One of my values is sharing knowledge and giving back to the community. I share what I learn and take people along on my journey and investigations into the problems we've faced. For this reason, the previous articles have documented these learnings, and this one is about a lesson we learned a little too late. The Journey so far Up to this point: We've shared the what, why and when to partition in the first blog post. In the second article, we covered how to maintain partitioned tables, as they provide some additional challenges when it comes to adding/removing partitions and managing indexes and foreign keys. Only in the third article did we disclose why we decided to work on our own partitioning framework: the existing partitioning solutions didn't provide us with the tools to minimize locks on tables. Our databases are always up and pressured, and the locks around partitioning were simply impossible to deal with. The key lesson from this article was that the default partition is the root of (almost) all evil regarding table partitioning in PostgreSQL. In the last article, we shared a cool solution to a problem we faced, where the default partition actually became the hero of the story. Yes, default partitions are usually the root of partitioning evil, but they also provide a welcome exception to the rule. Just be very careful with its usage. The Journey Continues... What ingredients can be missing after reading these four articles? You already know all about partitioning tables by

[Next page](<https://devfeed.tech/topics/table-partitioning.md?cursor=WyIyMDI2LTA3LTEzVDA5OjE4OjQ1KzAwOjAwIiwgImFhODdhZmM3LTZlM2MtNDZiNC05ZTUzLTBkNjdiNzBiNGVlZiJd>)