# compressing data with Parquet

Compressing data with Parquet uses Parquet's compression support to reduce file size and improve data read/write performance.

This is one page of public article previews, not the complete archive. Follow Next page to continue. Summaries are not the original full articles.

## Announcing DuckDB 2.0.1 LTS

DevFeed: [Announcing DuckDB 2.0.1 LTS](<https://devfeed.tech/articles/announcing-duckdb-2-0-1-lts-146570.md>)

Original publisher: [Read original article](<https://duckdb.org/2026/11/20/announcing-duckdb-201.html>)

Author: The DuckDB team

Published: 2026-11-20T00:00:00Z

Content type: release

Language: en

Sources: [DuckDB](<https://devfeed.tech/sources/duckdb.md>)

Topics: [DuckDB](<https://devfeed.tech/topics/duckdb.md>), [patches](<https://devfeed.tech/topics/patches.md>), [least privilege](<https://devfeed.tech/topics/least-privilege.md>), [compressing data with Parquet](<https://devfeed.tech/topics/compressing-data-with-parquet.md>), [Linux](<https://devfeed.tech/topics/linux.md>)

Tags: [bugfixes](<https://devfeed.tech/tags/bugfixes.md>), [custom](<https://devfeed.tech/tags/custom.md>), [duckdb](<https://devfeed.tech/tags/duckdb.md>), [duckdb-2-0](<https://devfeed.tech/tags/duckdb-2-0.md>), [extension](<https://devfeed.tech/tags/extension.md>), [improvements](<https://devfeed.tech/tags/improvements.md>), [lts](<https://devfeed.tech/tags/lts.md>), [patch](<https://devfeed.tech/tags/patch.md>), [performance](<https://devfeed.tech/tags/performance.md>), [release](<https://devfeed.tech/tags/release.md>), [repositories](<https://devfeed.tech/tags/repositories.md>), [security](<https://devfeed.tech/tags/security.md>), [security-patches](<https://devfeed.tech/tags/security-patches.md>)

### AI overview

DuckDB 2.0.1 is the first patch release in the 2.0 LTS line, with bug and security fixes, performance improvements, and a new way to register custom extension repositories by hostname. Host-based registration verifies the host's repository pointer over HTTPS, then pins signing keys from the destination repository; it avoids a central registry and lets an organization use its domain while serving extensions elsewhere. The default repository policy now permits host-based repositories, while prefix-based repositories require a broader setting; `forbidden` disables custom repositories. Review host and bucket `SECRET` scopes, which now require exact host or bucket names (and explicit ports), and `MERGE` or `UPDATE ... FROM`, which again update once per source match and may produce non-deterministic final values. Other fixes address 1.x database compatibility, external-access controls, memory handling on affected Linux kernels, and numerous query, storage, and extension issues. The announcement recommends upgrading from 1.4 LTS, whose community support ended November 17, 2026; 2.0 LTS is supported until at least October 2027.

### Source excerpt

Today we are releasing DuckDB 2.0.1, the first patch release of our 2.0 LTS line. This release ships bugfixes, performance improvements, and security patches. In addition, a capability is added to define custom extension repositories with only their hostname.

## Can AI Write Database Internals?

DevFeed: [Can AI Write Database Internals?](<https://devfeed.tech/articles/can-ai-write-database-internals-142367.md>)

Original publisher: [Read original article](<https://www.scylladb.com/2026/11/18/can-ai-write-database-internals/>)

Author: Cynthia Dunlop

Published: 2026-11-18T13:29:09Z

Content type: opinion

Language: en

Sources: [ScyllaDB](<https://devfeed.tech/sources/scylladb.md>)

Topics: [scylladb](<https://devfeed.tech/topics/scylladb.md>), [AI tools for developers](<https://devfeed.tech/topics/ai-tools-for-developers.md>), [Claude](<https://devfeed.tech/topics/claude.md>), [compressing data with Parquet](<https://devfeed.tech/topics/compressing-data-with-parquet.md>), [Apache Cassandra](<https://devfeed.tech/topics/cassandra.md>)

Tags: [ai](<https://devfeed.tech/tags/ai.md>), [cache](<https://devfeed.tech/tags/cache.md>), [claude](<https://devfeed.tech/tags/claude.md>), [database](<https://devfeed.tech/tags/database.md>), [engineering](<https://devfeed.tech/tags/engineering.md>), [internals](<https://devfeed.tech/tags/internals.md>), [invariant](<https://devfeed.tech/tags/invariant.md>), [leadership-team](<https://devfeed.tech/tags/leadership-team.md>), [mvcc](<https://devfeed.tech/tags/mvcc.md>), [p99-conf](<https://devfeed.tech/tags/p99-conf.md>), [policy](<https://devfeed.tech/tags/policy.md>), [write](<https://devfeed.tech/tags/write.md>)

### AI overview

The author reports that AI helped implement W-TinyLFU for ScyllaDB after years of dormancy, but the work required weeks of testing and debugging before showing gains. The policy combines a small admission window, a probation area and a main cache, using approximate frequency counts to resist scan-driven cache pollution. Tests showed higher hit rates and fewer evictions than LRU; P99 latency was nearly unchanged when disk was not a bottleneck, but fell from 27 ms to 6.7 ms with artificial disk delays. The attempt also exposed a correctness risk: W-TinyLFU's promotions and demotions could violate an MVCC invariant requiring older partition versions to be evicted first. Reviewers caught the issue; the author says the pull request is awaiting a separate change that bypasses MVCC for partitions without clustering keys. The account presents AI as a major coding accelerator, while emphasizing that it missed this invariant and still needed experienced human review. The excerpt also describes Parquet work, reporting substantial compression savings alongside higher point-read costs for multi-column searches.

### Source excerpt

A new cache policy in days, weeks stuck with no measurable gain, and an MVCC invariant Claude overlooked

## MongoDB to S3: Parquet Files, Change Streams & Nested Documents Guide

DevFeed: [MongoDB to S3: Parquet Files, Change Streams & Nested Documents Guide](<https://devfeed.tech/articles/mongodb-to-s3-parquet-files-change-streams-nested-documents-guide-118922.md>)

Original publisher: [Read original article](<https://estuary.dev/blog/mongodb-to-s3>)

Author: Seán Whelan

Published: 2026-11-04T22:41:24Z

Content type: article

Language: en

Sources: [Estuary](<https://devfeed.tech/sources/estuary.md>)

Topics: [streaming data to S3 with Parquet](<https://devfeed.tech/topics/streaming-data-to-s3-with-parquet.md>), [MongoDB](<https://devfeed.tech/topics/mongodb.md>), [Parquet files in Amazon S3](<https://devfeed.tech/topics/parquet-files-in-amazon-s3.md>), [Amazon S3](<https://devfeed.tech/topics/amazon-s3.md>), [parquet](<https://devfeed.tech/topics/parquet.md>), [AWS Database Migration Service](<https://devfeed.tech/topics/aws-database-migration-service.md>), [AWS Glue](<https://devfeed.tech/topics/aws-glue.md>), [compressing data with Parquet](<https://devfeed.tech/topics/compressing-data-with-parquet.md>), [DLT](<https://devfeed.tech/topics/data-load-tool.md>), [Amazon DocumentDB](<https://devfeed.tech/topics/amazon-documentdb.md>)

Tags: [change](<https://devfeed.tech/tags/change.md>), [mongodb](<https://devfeed.tech/tags/mongodb.md>), [nested](<https://devfeed.tech/tags/nested.md>), [parquet](<https://devfeed.tech/tags/parquet.md>), [parquet-files](<https://devfeed.tech/tags/parquet-files.md>), [s3](<https://devfeed.tech/tags/s3.md>)

### AI overview

MongoDB change streams can feed append-only Parquet files in S3, but updates and deletes become new rows; readers must retain the latest row per `_id` and filter delete markers to represent current state. Change streams require a replica set or sharded cluster and an oplog window long enough to cover outages. A time-window copy or batch job misses deletes. Atlas Data Federation, AWS DMS, custom Spark/Glue/dlt jobs, and managed CDC services differ in setup and how they map nested BSON data; structs, flattened columns, and JSON text are the described options. Schema changes affect later files, and Decimal128 or digit-only strings can lose precision or leading zeros unless written as suitable decimal or string types. Estuary's described route captures changes durably and appends Parquet on a configurable interval, with nested-field depth settings; its Decimal128 and integer inference behavior may require `castToString`. The excerpt also gives S3 permissions, lifecycle, validation, and recovery requirements, including re-reading collections after an oplog history gap.

### Source excerpt

Move MongoDB to S3 as Parquet with Atlas Data Federation, AWS DMS, Spark or a managed pipeline. Compare how each handles nested documents and deletes.

## Row-Group Control, Faster S3 Reads, Footer Caching: Hardwood 1.1.0.Beta2 Is Out

DevFeed: [Row-Group Control, Faster S3 Reads, Footer Caching: Hardwood 1.1.0.Beta2 Is Out](<https://devfeed.tech/articles/row-group-control-faster-s3-reads-footer-caching-hardwood-1-1-0-beta2-is-out-85597.md>)

Original publisher: [Read original article](<https://www.morling.dev/blog/row-group-control-faster-s3-reads-footer-caching-hardwood-1-1-0-beta2/>)

Published: 2026-10-12T19:00:00Z

Content type: release

Language: en

Sources: [Gunnar Morling](<https://devfeed.tech/sources/gunnar-morling.md>)

Topics: [parquet](<https://devfeed.tech/topics/parquet.md>), [Cache](<https://devfeed.tech/topics/cache.md>), [compressing data with Parquet](<https://devfeed.tech/topics/compressing-data-with-parquet.md>)

Tags: [apache-parquet](<https://devfeed.tech/tags/apache-parquet.md>), [binaries](<https://devfeed.tech/tags/binaries.md>), [bloom-filter](<https://devfeed.tech/tags/bloom-filter.md>), [cli](<https://devfeed.tech/tags/cli.md>), [improvements](<https://devfeed.tech/tags/improvements.md>), [maven](<https://devfeed.tech/tags/maven.md>), [maven-central](<https://devfeed.tech/tags/maven-central.md>), [performance](<https://devfeed.tech/tags/performance.md>), [release](<https://devfeed.tech/tags/release.md>), [releases](<https://devfeed.tech/tags/releases.md>), [s3](<https://devfeed.tech/tags/s3.md>)

### AI overview

Hardwood 1.1.0.Beta2 adds writer controls for Parquet row groups, record rejection handling, Bloom filters and indexes, plus faster reads from S3 and other workloads. It introduces an experimental API for caching file footers and dictionary-ID access, alongside query execution, decoder, CLI, and compatibility improvements. The release reports that 21 of 25 benchmark scenarios were faster than Beta1, with the remaining four within noise; one Bloom filter lookup on S3 fell from 6.8 seconds to 163 milliseconds.

### Source excerpt

Table of Contents Writer Improvements Performance Improvements Footer Caching and Reuse Id-Based Access to Dictionary-Encoded Columns Also in This Release I'm happy to announce the release of Hardwood 1.1.0.Beta2! The latest version of the Hardwood open-source library for reading and writing Apache Parquet files brings new features for the writer, significantly improves performance for many different workloads, provides a new API for caching and reusing Parquet file footers, id-based dictionary access, and much more. You can retrieve Hardwood from Maven Central; for the platform-specific CLI binaries check out our GitHub Releases page.

## Converting Snowflake Native Tables to Iceberg -- Part 1

DevFeed: [Converting Snowflake Native Tables to Iceberg -- Part 1](<https://devfeed.tech/articles/converting-snowflake-native-tables-to-iceberg-part-1-78486.md>)

Original publisher: [Read original article](<https://medium.com/snowflake/converting-snowflake-native-tables-to-iceberg-part-1-603fdb0fdcd1?source=rss----34b6daafc07---4>)

Author: Leo Pasta

Published: 2026-10-02T14:01:08Z

Content type: article

Language: en

Sources: [Snowflake Builders](<https://devfeed.tech/sources/snowflake-builders.md>)

Topics: [Apache Iceberg](<https://devfeed.tech/topics/apache-iceberg.md>), [compressing data with Parquet](<https://devfeed.tech/topics/compressing-data-with-parquet.md>), [Snowflake streaming data architecture](<https://devfeed.tech/topics/snowflake-streaming-data-architecture.md>)

Tags: [amazon-athena](<https://devfeed.tech/tags/amazon-athena.md>), [apache-iceberg](<https://devfeed.tech/tags/apache-iceberg.md>), [apache-spark](<https://devfeed.tech/tags/apache-spark.md>), [article](<https://devfeed.tech/tags/article.md>), [conversion](<https://devfeed.tech/tags/conversion.md>), [data](<https://devfeed.tech/tags/data.md>), [data-engineering](<https://devfeed.tech/tags/data-engineering.md>), [data-migration](<https://devfeed.tech/tags/data-migration.md>), [databricks](<https://devfeed.tech/tags/databricks.md>), [duckdb](<https://devfeed.tech/tags/duckdb.md>), [ecosystem](<https://devfeed.tech/tags/ecosystem.md>), [etl](<https://devfeed.tech/tags/etl.md>), [feature](<https://devfeed.tech/tags/feature.md>), [format](<https://devfeed.tech/tags/format.md>), [governance](<https://devfeed.tech/tags/governance.md>), [iceberg](<https://devfeed.tech/tags/iceberg.md>), [latency](<https://devfeed.tech/tags/latency.md>), [native](<https://devfeed.tech/tags/native.md>), [performance](<https://devfeed.tech/tags/performance.md>), [platform](<https://devfeed.tech/tags/platform.md>), [snowflake](<https://devfeed.tech/tags/snowflake.md>), [tables](<https://devfeed.tech/tags/tables.md>)

### AI overview

Iceberg conversion is most useful when specific datasets need access from multiple engines; converting every table can add complexity and cost without equivalent benefit. Start with a targeted, phased migration and inventory each table's consumers, engine versions, data types, write patterns, streams, and swap-based pipelines before choosing settings. Consumer capabilities constrain Iceberg version, types, partition pruning, codecs, and delete behavior. Resolve unsupported or lossy types such as TIMESTAMP_TZ, VECTOR, and some V2 conversions, and account for limitations involving defaults and collation for external engines. Choose CTAS for tables around 1-2 TB or with types that make Parquet staging risky; for larger compatible tables, Parquet export with COPY INTO ... LOAD_MODE=ADD_FILES_COPY may be faster and use fewer credits. Select partitioning or clustering based on consumer pruning and update workload, and choose storage and serialization settings according to cloud, access, key-management, and operational needs. Validate a representative sample before conversion; execution, validation, and cutover are reserved for Part 2.

### Source excerpt

Part 1 of 2 -- this article covers why you might convert to Iceberg, the decisions you need to make and how to approach them. Part 2 covers the conversion execution, validation and cutover. Why convert to Iceberg? Snowflake native tables are tightly integrated into the Snowflake platform: They offer the most complete support in areas such as performance optimisation, disaster recovery, replication and the full range of governance features. For workloads that live entirely within Snowflake, native tables are usually the right choice. But enterprise data environments are growing more and more sophisticated, so adopting an open table format like Iceberg can solve real challenges, like having multiple engines -- Apache Spark, Trino, Amazon Athena, Databricks, DuckDB and many others -- able to read and, in some configurations, write the same Iceberg tables without relying on Snowflake to process the query or on ETL jobs duplicating the data with the complexities and latency challenges it brings. Snowflake responded to this trend by offering what is widely regarded as the most complete support for the Apache Iceberg ecosystem, both as a manager and consumer of Iceberg catalogs. Conversion is not free of trade-offs though. The Iceberg ecosystem is also more complex to manage, with more options and configurations to consider and often more vendors and technologies involved. It is also less mature and well known than "traditional" Snowflake tables, which usually means a longer learning and adoption curve for engineers. The Apache Iceberg project is managed by a committee which, while fundamental for the extensive vendor support it has attracted, also means the adoption of new features and capabilities might happen later than with native tables. Over time, as the Iceberg format evolves and Snowflake features are released, the feature gap between native tables and Iceberg tables continues to narrow, but it is not zero yet. A few of those limitations will be discussed in the rest

## They All Write, Store, and Read Data: How to Pick The Right Database

DevFeed: [They All Write, Store, and Read Data: How to Pick The Right Database](<https://devfeed.tech/articles/they-all-write-store-and-read-data-how-to-pick-the-right-database-78938.md>)

Original publisher: [Read original article](<https://motherduck.com/blog/how-to-pick-the-right-database>)

Author: Dumky de Wilde

Published: 2026-09-15T00:00:00Z

Content type: article

Language: en

Sources: [MotherDuck](<https://devfeed.tech/sources/motherduck.md>)

Topics: [Database](<https://devfeed.tech/topics/database.md>), [DuckDB](<https://devfeed.tech/topics/duckdb.md>), [database-optimization](<https://devfeed.tech/topics/database-optimization.md>), [sql-database](<https://devfeed.tech/topics/sql-database.md>), [graph-database](<https://devfeed.tech/topics/graph-database.md>), [Transactions](<https://devfeed.tech/topics/transactions.md>), [compressing data with Parquet](<https://devfeed.tech/topics/compressing-data-with-parquet.md>), [database-consistency](<https://devfeed.tech/topics/database-consistency.md>), [LanceDB vector database](<https://devfeed.tech/topics/lancedb-vector-database.md>)

Tags: [aggregation](<https://devfeed.tech/tags/aggregation.md>), [analysis](<https://devfeed.tech/tags/analysis.md>), [analytics](<https://devfeed.tech/tags/analytics.md>), [atomic](<https://devfeed.tech/tags/atomic.md>), [audio](<https://devfeed.tech/tags/audio.md>), [cache](<https://devfeed.tech/tags/cache.md>), [data](<https://devfeed.tech/tags/data.md>), [database](<https://devfeed.tech/tags/database.md>), [duckdb](<https://devfeed.tech/tags/duckdb.md>), [postgres](<https://devfeed.tech/tags/postgres.md>), [questions](<https://devfeed.tech/tags/questions.md>), [sqlite](<https://devfeed.tech/tags/sqlite.md>), [store](<https://devfeed.tech/tags/store.md>), [write](<https://devfeed.tech/tags/write.md>)

### AI overview

Choose a database around the queries and workload you expect: SQLite suits small, mostly single-writer projects; Postgres is a strong general-purpose choice for shared transactional data; DuckDB and other analytical databases suit large aggregations. The article explains that DuckDB reduces work through columnar reads, row groups, predicate pushdown and pruning, compression, and query-plan optimization. Its embedded, in-process design can put compute alongside the application and avoid a shared database server. Other layouts fit specific needs: key/value stores provide fast known-key lookups but cannot answer unplanned queries; document databases allow flexible nested records but can make cross-record analysis costly; graph databases model relationships; vector indexes speed similarity search at the cost of approximate results; time-series databases optimize time-windowed data; and geospatial indexes support location queries. The guidance is workload-dependent: Postgres is presented as a practical default, with specialized systems useful when its performance or cost no longer fits.

### Source excerpt

SQLite, Postgres, DuckDB and 20 others: they all write, store and read data. The questions you ask every day tell you which one you need.

## Parquet File Write Support, Bloom Filters, Improved Performance: Hardwood 1.1.0.Beta1 Is Out

DevFeed: [Parquet File Write Support, Bloom Filters, Improved Performance: Hardwood 1.1.0.Beta1 Is Out](<https://devfeed.tech/articles/parquet-file-write-support-bloom-filters-improved-performance-hardwood-1-1-0-beta1-is-out-18856.md>)

Original publisher: [Read original article](<https://www.morling.dev/blog/parquet-file-write-support-bloom-filters-improved-performance-hardwood-1-1-0-beta1/>)

Published: 2026-08-31T19:36:00Z

Content type: release

Language: en

Sources: [Gunnar Morling](<https://devfeed.tech/sources/gunnar-morling.md>)

Topics: [parquet](<https://devfeed.tech/topics/parquet.md>), [compressing data with Parquet](<https://devfeed.tech/topics/compressing-data-with-parquet.md>)

Tags: [apache](<https://devfeed.tech/tags/apache.md>), [apache-parquet](<https://devfeed.tech/tags/apache-parquet.md>), [api](<https://devfeed.tech/tags/api.md>), [apis](<https://devfeed.tech/tags/apis.md>), [arrays](<https://devfeed.tech/tags/arrays.md>), [bloom-filters](<https://devfeed.tech/tags/bloom-filters.md>), [columnar](<https://devfeed.tech/tags/columnar.md>), [compaction](<https://devfeed.tech/tags/compaction.md>), [library](<https://devfeed.tech/tags/library.md>), [parquet](<https://devfeed.tech/tags/parquet.md>), [performance](<https://devfeed.tech/tags/performance.md>), [query](<https://devfeed.tech/tags/query.md>), [write](<https://devfeed.tech/tags/write.md>)

### AI overview

Hardwood 1.1 Beta1 adds an initial preview of Parquet writing, expanding the project beyond its reader. Its record-based API accepts one record at a time and is suited to complex schemas with structs, lists, or maps; the engine buffers records and creates row groups and pages with defaults. The batch-oriented API accepts column values in arrays and is optimized for performance-intensive work such as compaction. Writer settings can control row-group and page sizes, encodings, compression codecs, and statistics globally or per column. The excerpt also announces Bloom filters, dictionary-based row-group pruning, performance improvements, and a faster CLI, but does not provide details on those features.

### Source excerpt

Table of Contents Write Support Query Evaluation: Bloom Filters and Dictionary-Based Row-Group Pruning Performance Improvements Hardwood CLI Closing Thoughts "When is write support gonna land in Hardwood?" That's probably the most common question I got over the last few months. As of today, I am very happy to share that the answer has changed from "It's coming soon" to "A first cut is there, give it a try" -- the first Beta of Hardwood 1.1 is out! This is a major milestone for the project, marking the first step in evolving Hardwood from being solely a Parquet parser to a complete library for this widely used columnar file format. But there's more. This release also comes with significant enhancements to the query layer (Bloom filters, dictionary-based row-group pruning), many performance improvements such as a fast path for effectively fixed-length list columns, an even snappier CLI, and much more. Let's dig into some of the new features and changes!

## Compression Dictionary Transport

DevFeed: [Compression Dictionary Transport](<https://devfeed.tech/articles/compression-dictionary-transport-89124.md>)

Original publisher: [Read original article](<https://specification.website/spec/performance/compression-dictionary-transport/>)

Author: Joost de Valk

Published: 2026-06-10T00:00:00Z

Content type: tutorial

Language: en

Sources: [The Website Specification](<https://devfeed.tech/sources/the-website-specification.md>)

Topics: [Compression](<https://devfeed.tech/topics/compression.md>), [compressing data with Parquet](<https://devfeed.tech/topics/compressing-data-with-parquet.md>), [Latency](<https://devfeed.tech/topics/latency.md>)

Tags: [assets](<https://devfeed.tech/tags/assets.md>), [browser](<https://devfeed.tech/tags/browser.md>), [cache](<https://devfeed.tech/tags/cache.md>), [compression](<https://devfeed.tech/tags/compression.md>), [curl](<https://devfeed.tech/tags/curl.md>), [how-to](<https://devfeed.tech/tags/how-to.md>), [https](<https://devfeed.tech/tags/https.md>), [network](<https://devfeed.tech/tags/network.md>), [optional](<https://devfeed.tech/tags/optional.md>), [performance](<https://devfeed.tech/tags/performance.md>)

### AI overview

Compression Dictionary Transport (RFC 9842) lets servers use a previously served resource or a dedicated dictionary to compress updates to assets, reducing transfer size. Browsers that do not support it continue to receive ordinary gzip or Brotli responses. The tutorial explains dictionary selection and scope, cache variation, CDN configuration, common mistakes, and verification.

### Source excerpt

Use a previously served response, or a dedicated dictionary, as a Brotli/Zstandard dictionary so updated assets compress to a fraction of their size. Pure progressive enhancement over ordinary compression.

## Building a WHOOP sleep data pipeline with dlt and Cursor

DevFeed: [Building a WHOOP sleep data pipeline with dlt and Cursor](<https://devfeed.tech/articles/from-no-pipelines-to-building-one-in-just-1-hour-with-cursor-80432.md>)

Original publisher: [Read original article](<https://dlthub.com/blog/whoop-sleep-data-pipeline>)

Author: Roshni Melwani

Published: 2025-06-29T00:00:00Z

Content type: tutorial

Language: en

Sources: [DLT Hub](<https://devfeed.tech/sources/dlt-hub.md>)

Topics: [data-engineering](<https://devfeed.tech/topics/data-engineering.md>), [cursor](<https://devfeed.tech/topics/cursor.md>), [DLT](<https://devfeed.tech/topics/data-load-tool.md>), [compressing data with Parquet](<https://devfeed.tech/topics/compressing-data-with-parquet.md>), [Data serialization](<https://devfeed.tech/topics/data-serialization.md>)

Tags: [ai-coding](<https://devfeed.tech/tags/ai-coding.md>), [community](<https://devfeed.tech/tags/community.md>), [cursor](<https://devfeed.tech/tags/cursor.md>), [dev](<https://devfeed.tech/tags/dev.md>), [parquet](<https://devfeed.tech/tags/parquet.md>), [pipelines](<https://devfeed.tech/tags/pipelines.md>), [tutorials](<https://devfeed.tech/tags/tutorials.md>)

### AI overview

A beginner describes learning pipeline fundamentals through a dlt course, then using Cursor to build a WHOOP sleep data pipeline that saves sleep records locally in Parquet. Cursor sped up setup and coding, but required documentation context and manual corrections for authentication, pagination, and destination choices.

### Source excerpt

Dev takes Alena's dlt course, then uses AI to build a WHOOP sleep data pipeline to Parquet, proving beginners can master pipelines fast, hands-on.

## What is Airbyte? Uses, Features, and Top Alternatives

DevFeed: [What is Airbyte? Uses, Features, and Top Alternatives](<https://devfeed.tech/articles/what-is-airbyte-uses-features-and-top-alternatives-82441.md>)

Original publisher: [Read original article](<https://estuary.dev/blog/what-is-airbyte>)

Author: Dani Pálma

Published: 2025-04-08T16:19:46Z

Content type: comparison

Language: en

Sources: [Estuary](<https://devfeed.tech/sources/estuary.md>)

Topics: [etl](<https://devfeed.tech/topics/etl.md>), [compressing data with Parquet](<https://devfeed.tech/topics/compressing-data-with-parquet.md>), [build vs buy database solutions](<https://devfeed.tech/topics/build-vs-buy-database-solutions.md>), [CI/CD](<https://devfeed.tech/topics/cicd.md>), [Service fabric](<https://devfeed.tech/topics/service-fabric.md>)

Tags: [airbyte](<https://devfeed.tech/tags/airbyte.md>), [alternatives](<https://devfeed.tech/tags/alternatives.md>), [cloud](<https://devfeed.tech/tags/cloud.md>), [comparison](<https://devfeed.tech/tags/comparison.md>), [cost](<https://devfeed.tech/tags/cost.md>), [data](<https://devfeed.tech/tags/data.md>), [data-integration](<https://devfeed.tech/tags/data-integration.md>), [data-pipelines](<https://devfeed.tech/tags/data-pipelines.md>), [data-replication](<https://devfeed.tech/tags/data-replication.md>), [database](<https://devfeed.tech/tags/database.md>)

### AI overview

This guide reviews Airbyte as an open-source ELT platform, covering its connectors, data replication and pipeline architecture. It describes limitations in latency, reliability, and scaling, then compares Airbyte with alternatives including Estuary, Fivetran, Stitch, and Meltano.

### Source excerpt

Confused about Airbyte? This guide breaks it down. Learn what Airbyte does, how it works, and if it fits your data needs. Explore top Airbyte alternatives like Estuary!

## Memory Efficient Data Streaming To Parquet Files

DevFeed: [Memory Efficient Data Streaming To Parquet Files](<https://devfeed.tech/articles/memory-efficient-data-streaming-to-parquet-files-82214.md>)

Original publisher: [Read original article](<https://estuary.dev/blog/memory-efficient-streaming-parquet>)

Author: Dani Pálma; William Baker

Published: 2024-07-29T18:37:30Z

Content type: tutorial

Language: en

Sources: [Estuary](<https://devfeed.tech/sources/estuary.md>)

Topics: [streaming data to S3 with Parquet](<https://devfeed.tech/topics/streaming-data-to-s3-with-parquet.md>), [parquet](<https://devfeed.tech/topics/parquet.md>), [Streaming](<https://devfeed.tech/topics/streaming.md>), [compressing data with Parquet](<https://devfeed.tech/topics/compressing-data-with-parquet.md>)

Tags: [compression](<https://devfeed.tech/tags/compression.md>), [data](<https://devfeed.tech/tags/data.md>), [data-pipelines](<https://devfeed.tech/tags/data-pipelines.md>), [data-streaming](<https://devfeed.tech/tags/data-streaming.md>), [memory](<https://devfeed.tech/tags/memory.md>), [parquet](<https://devfeed.tech/tags/parquet.md>), [parquet-files](<https://devfeed.tech/tags/parquet-files.md>)

### AI overview

The article explains a two-pass method for streaming data into Parquet files with limited memory. It writes incoming rows to small Parquet row groups in a disk-based scratch file, then reads the data column by column to assemble larger output row groups. The method reduces memory use in constrained connectors, with added encoding and decoding overhead and metadata-size limitations for datasets with very many columns.

### Source excerpt

How to efficiently stream data into Parquet files.

## How dlt uses Apache Arrow

DevFeed: [How dlt uses Apache Arrow](<https://devfeed.tech/articles/how-dlt-uses-apache-arrow-80333.md>)

Original publisher: [Read original article](<https://dlthub.com/blog/how-dlt-uses-apache-arrow>)

Author: Jorrit Sandbrink

Published: 2024-07-11T00:00:00Z

Content type: article

Language: en

Sources: [DLT Hub](<https://devfeed.tech/sources/dlt-hub.md>)

Topics: [DLT](<https://devfeed.tech/topics/data-load-tool.md>), [apache-arrow](<https://devfeed.tech/topics/apache-arrow.md>), [parquet](<https://devfeed.tech/topics/parquet.md>), [compressing data with Parquet](<https://devfeed.tech/topics/compressing-data-with-parquet.md>), [Data pipelines](<https://devfeed.tech/topics/data-pipelines.md>), [Python](<https://devfeed.tech/topics/python.md>), [High-Performance Computing](<https://devfeed.tech/topics/high-performance-computing.md>)

Tags: [apache-arrow](<https://devfeed.tech/tags/apache-arrow.md>), [batch](<https://devfeed.tech/tags/batch.md>), [c-plus-plus](<https://devfeed.tech/tags/c-plus-plus.md>), [community](<https://devfeed.tech/tags/community.md>), [data-processing](<https://devfeed.tech/tags/data-processing.md>), [memory](<https://devfeed.tech/tags/memory.md>), [optimization](<https://devfeed.tech/tags/optimization.md>), [performance](<https://devfeed.tech/tags/performance.md>), [pipelines](<https://devfeed.tech/tags/pipelines.md>), [python](<https://devfeed.tech/tags/python.md>), [tabular-data](<https://devfeed.tech/tags/tabular-data.md>)

### AI overview

The article explains how the open source Python library dlt uses Apache Arrow to make data pipelines faster. It compares dlt's traditional route, which processes native Python objects row by row, with its Arrow route, which uses columnar data, Parquet files, and C++ batch transformations. The article also notes that Arrow's advantages depend on destination support for Parquet loading and suggests Arrow IPC serialization as an untested possible optimization.

### Source excerpt

`dlt` uses Apache Arrow to make pipelines faster. The Arrow format is a better way to represent tabular data in memory than native Python objects (list of dictionaries). It enables offloading computation to Arrow's fast C++ library and prevents processing rows one by one. Speed matters. Pipelines should move data quickly and efficiently. The bigger the data, the more that holds. Growing data volumes force performance optimization upon data processing tools.

## Using ConnectorX, Arrow, and dlt to speed up PostgreSQL data processing

DevFeed: [Using ConnectorX, Arrow, and dlt to speed up PostgreSQL data processing](<https://devfeed.tech/articles/get-30x-speedups-when-reading-databases-with-connectorx-arrow-dlt-80287.md>)

Original publisher: [Read original article](<https://dlthub.com/blog/dlt-arrow-loading>)

Author: Marcin Rudolf

Published: 2023-10-23T00:00:00Z

Content type: tutorial

Language: en

Sources: [DLT Hub](<https://devfeed.tech/sources/dlt-hub.md>)

Topics: [data-engineering](<https://devfeed.tech/topics/data-engineering.md>), [DuckDB](<https://devfeed.tech/topics/duckdb.md>), [etl](<https://devfeed.tech/topics/etl.md>), [compressing data with Parquet](<https://devfeed.tech/topics/compressing-data-with-parquet.md>)

Tags: [arrow](<https://devfeed.tech/tags/arrow.md>), [data-engineering](<https://devfeed.tech/tags/data-engineering.md>), [databases](<https://devfeed.tech/tags/databases.md>), [duckdb](<https://devfeed.tech/tags/duckdb.md>), [engineering](<https://devfeed.tech/tags/engineering.md>), [parquet](<https://devfeed.tech/tags/parquet.md>), [postgres](<https://devfeed.tech/tags/postgres.md>), [rust](<https://devfeed.tech/tags/rust.md>)

### AI overview

A tutorial compares processing 10 million PostgreSQL rows with ConnectorX and Arrow versus SQLAlchemy in dlt. In the reported local database test, extraction and normalization took about 16 seconds with Arrow, compared with about 8 minutes with SQLAlchemy. The comparison excludes the loading step and notes that remote database network latency would reduce the speedup.

### Source excerpt

If rust + arrow + duckb is a new data engineering stack, now you can get a feel of it with `dlt`. We recently added native arrow tables (and panda frames) loading. What it means? You can pass an Arrow table to `dlt` **pipeline.run** or **pipeline.extract** methods, have it normalized, saved to parquet and loaded to your destination. Here we achieved ~30x speedups when loading data from (local) postgres database using ConnectorX + Arrow compared to SqlAlchemy + json.

## A simple way to get more value from metrics

DevFeed: [A simple way to get more value from metrics](<https://devfeed.tech/articles/a-simple-way-to-get-more-value-from-metrics-77721.md>)

Original publisher: [Read original article](<https://danluu.com/metrics-analytics/>)

Published: 2020-05-30T07:06:34Z

Content type: article

Language: en

Sources: [Danluu](<https://devfeed.tech/sources/danluu.md>)

Topics: [observability](<https://devfeed.tech/topics/observability.md>), [compressing data with Parquet](<https://devfeed.tech/topics/compressing-data-with-parquet.md>), [Prometheus](<https://devfeed.tech/topics/prometheus.md>)

Tags: [cost](<https://devfeed.tech/tags/cost.md>), [infra](<https://devfeed.tech/tags/infra.md>), [metrics](<https://devfeed.tech/tags/metrics.md>), [optimization](<https://devfeed.tech/tags/optimization.md>), [performance](<https://devfeed.tech/tags/performance.md>)

### AI overview

The article describes LongTermMetrics, a batch system for querying historical metrics across services and hosts. It explains how the system stores selected performance and capacity data in Parquet on HDFS to enable SQL analysis, and discusses its applications, design choices, operational tradeoffs, and implementation mistakes.

### Source excerpt

We spent one day1 building a system that immediately found a mid 7 figure optimization (which ended up shipping). In the first year, we shipped mid 8 figures per year worth of cost savings as a result. The key feature this system introduces is the ability to query metrics data across all hosts and all services and over any period of time (since inception), so we've called it LongTermMetrics (LTM) internally since I like boring, descriptive, names. This got started when I was looking for a starter project that would both help me understand the Twitter infra stack and also have some easily quantifiable value. Andy Wilcox suggested looking at JVM survivor space utilization for some large services. If you're not familiar with what survivor space is, you can think of it as a configurable, fixed-size buffer, in the JVM (at least if you use the GC algorithm that's default at Twitter). At the time, if you looked at a random large services, you'd usually find that either: The buffer was too small, resulting in poor performance, sometimes catastrophically poor when under high load. The buffer was too large, resulting in wasted memory, i.e., wasted money. But instead of looking at random services, there's no fundamental reason that we shouldn't be able to query all services and get a list of which services have room for improvement in their configuration, sorted by performance degradation or cost savings. And if we write that query for JVM survivor space, this also goes for other configuration parameters (e.g., other JVM parameters, CPU quota, memory quota, etc.). Writing a query that worked for all the services turned out to be a little more difficult than I was hoping due to a combination of data consistency and performance issues. Data consistency issues included things like: Any given metric can have ~100 names, e.g., I found 94 different names for JVM survivor space I suspect there are more, these were just the ones I could find via a simple search The same metric name mi