# apache-arrow Published articles for apache-arrow. This is one page of public article previews, not the complete archive. Follow Next page to continue. Summaries are not the original full articles. ## Shehab Amin on Spark Compatibility, Rust, and LakeSail DevFeed: [Shehab Amin on Spark Compatibility, Rust, and LakeSail]() Original publisher: [Read original article]() Author: Daniel Beach Published: 2026-09-02T12:22:23Z Content type: article Language: en Sources: [Data Engineering Central]() Topics: [Apache Spark](), [Rust](), [data-engineering](), [apache-arrow](), [agentic-coding]() Tags: [apache-arrow](), [apache-spark](), [data-engineering](), [rust]() ### AI overview A podcast conversation with LakeSail co-founder and CEO Shehab Amin about Spark compatibility, Rust, Apache Arrow, DataFusion, and data infrastructure. It also discusses data-stack choices, streaming and batch processing, agentic coding, and using existing data pipelines as a basis for AI pipelines. ### Source excerpt Data Engineering Central Podcast. ## The Creator of Pandas on AI, Apache Arrow, and the Future of Software Engineering DevFeed: [The Creator of Pandas on AI, Apache Arrow, and the Future of Software Engineering]() Original publisher: [Read original article]() Author: Daniel Beach Published: 2026-07-08T12:16:09Z Content type: article Language: en Sources: [Data Engineering Central]() Topics: [pandas](), [data-engineering](), [Artificial Intelligence](), [DuckDB](), [data-platforms](), [software-development](), [future of software](), [Apache Spark](), [Hadoop](), [Open Source](), [parquet]() Tags: [ai](), [apache-arrow](), [arrow](), [data](), [data-engineering](), [data-platforms](), [duckdb](), [hadoop](), [large-language-models](), [open-source](), [pandas](), [parquet](), [software-development]() ### AI overview An interview with Wes McKinney covers the origins of pandas and Apache Arrow, the evolution of modern data engineering from Hadoop to lakehouse architectures, and the roles of tools such as Parquet, DuckDB, DataFusion, and Spark. McKinney also discusses how AI affects software development, arguing that it can improve experienced engineers' productivity but does not replace software engineering, architecture, or judgment. ### Source excerpt interview with Wes McKinney ## OTel-Arrow Phase 2: From Efficient Transport to Efficient Telemetry Pipelines DevFeed: [OTel-Arrow Phase 2: From Efficient Transport to Efficient Telemetry Pipelines]() Original publisher: [Read original article]() Author: OpenTelemetry Authors; Docs CC BY Published: 2026-06-13T00:47:46Z Content type: article Language: en Sources: [Blog on OpenTelemetry]() Topics: [OpenTelemetry](), [telemetry](), [Structured-data](), [systems](), [Rust](), [Instrumentation]() Tags: [apache-arrow](), [memory](), [opentelemetry](), [overhead](), [pipelines](), [processors](), [rust](), [telemetry]() ### AI overview This article presents Phase 2 of OTel-Arrow, which uses Apache Arrow not only for transport through OTAP but also as the internal representation of telemetry pipelines. It describes the Rust-based OTel-Arrow Dataflow Engine and reports substantially lower transport and processing costs when telemetry remains on the OTAP path end to end. ### Source excerpt Phase 1 of OTel-Arrow established OTAP, the OpenTelemetry Arrow Protocol, as an efficient transport protocol for OpenTelemetry. Apache Arrow is a language-independent, columnar in-memory format designed to move and process structured data efficiently across systems. We demonstrated that telemetry could be transported with significantly lower network overhead while preserving compatibility with the OpenTelemetry data model. Phase 2 asked a different question: what happens if Arrow is used not only on the wire, but also as the representation the pipeline works with internally? ## Introducing Apache Arrow Support in mssql-python DevFeed: [Introducing Apache Arrow Support in mssql-python]() Original publisher: [Read original article]() Author: Saumya Garg Published: 2026-05-04T04:33:00Z Content type: release Language: en Sources: [Microsoft Python Engineering]() Topics: [sql-server](), [data-processing](), [interoperability](), [data](), [pandas]() Tags: [apache-arrow](), [arrow](), [azure](), [azure-sql](), [client-driver](), [data-processing](), [interoperability](), [pandas](), [python](), [python-driver-for-azure-sql](), [python-driver-for-sql-server](), [sql-server](), [sql-server-2025](), [zero-copy]() ### AI overview Microsoft introduces Apache Arrow support in mssql-python, enabling SQL Server data to be fetched directly into Arrow structures for Polars, Pandas, DuckDB, and other Arrow-native libraries. The approach is intended to reduce Python object creation and memory overhead during data processing. ### Source excerpt Reviewed by Sumit Sarabhai Fetching a million rows from SQL Server into a Polars DataFrame used to mean a million Python objects, a million GC allocations, and then throwing it all away to build a DataFrame. Not anymore. mssql-python now supports fetching SQL Server data directly as Apache Arrow structures - a faster and more [...] The post Introducing Apache Arrow Support in mssql-python appeared first on Microsoft for Python Developers Blog. ## ClickHouse Release 25.8 DevFeed: [ClickHouse Release 25.8]() Original publisher: [Read original article]() Author: ClickHouse Published: 2025-09-05T00:00:00Z Content type: release Language: en Sources: [ClickHouse Blog]() Topics: [clickhouse](), [parquet](), [Flight](), [Apache Iceberg]() Tags: [apache-arrow](), [bug](), [bug-fixes](), [clickhouse](), [data](), [data-lake](), [features](), [files](), [format](), [integration](), [new-features](), [parallelism](), [parquet](), [partitioning](), [performance](), [release](), [storage](), [streaming]() ### AI overview ClickHouse 25.8 introduces 45 new features, 47 performance optimizations, and 119 bug fixes. Highlights include a faster experimental native Parquet reader, Data Lake improvements, Hive-style partitioned writes, initial PromQL support, and Arrow Flight integration. ### Source excerpt ClickHouse 25.8 is available. In this post, you will learn about new features, including a new Parquet reader and Arrow Flight integration. ## A behind-the-scenes look at building interactive analysis capabilities in Benchling DevFeed: [A behind-the-scenes look at building interactive analysis capabilities in Benchling]() Original publisher: [Read original article]() Author: Wonja Fairbrother Published: 2024-06-11T13:01:25Z Content type: article Language: en Sources: [Benchling]() Topics: [data-processing](), [web applications](), [Amazon EKS](), [Amazon S3](), [API](), [Back end](), [Front end](), [JSON]() Tags: [apache-arrow](), [apache-parquet](), [api](), [architecture](), [backend](), [data](), [data-analysis](), [data-transformation](), [data-visualization](), [duckdb](), [frontend](), [json](), [real-time](), [s3](), [storage](), [systems]() ### AI overview This article explains the architecture and design decisions behind Benchling's Interactive Analysis capabilities. The system lets scientists select data from multiple sources, transform and analyze it in real time, and visualize results within Benchling. Its architecture uses the Benchling web application, a stateless service running on EKS, temporary S3 storage, and a JSON-based transformation API. ### Source excerpt Authors: Wonja Fairbrother and Eli Levine Science is iterative. To design the next experiment, scientists need to analyze the results of previous ones. Interactive Analysis in Benchling allows scientists to perform real-time data transformation, visualization, and analysis without having to transfer it into other systems. In this post we will describe the architecture behind interactive analysis capabilities in Benchling and give a peek into the decision journey we took along the way¹. Interactive Analysis allows scientists to: 1. Select data from many sources: Benchling entity and results data Instrument data Notebook tables Data upload via both API and UI 2. Transform, visualize, and analyze data in real time, without leaving Benchling: Data transformations: filtering, aggregations, window functions, etc. Visualizations: line chart, bar chart, scatter plot, etc. Scientific analysis methods: IC50 and various curve fitting functions Overall architecture The architecture backing Interactive Analysis consists of: The Benchling web application An auto-scaling stateless internal service running on EKS that performs the transformations Temporary S3 storage locations for input and output data, shared between the web app and the service The frontend of the application is responsible for taking in input datasets and transformation configurations from users. The backend of the web application collects all the input data from the appropriate sources, serializes and uploads the data to S3, and sends a synchronous transformation request to the service. The service's API consists of one main endpoint that takes in a JSON payload of transformation parameters. The service can accept a single transformation, or a list of many transformations to perform. In this endpoint, the service downloads and deserializes the input data, performs the transformation with an analysis engine, and serializes and uploads the resulting data to S3. Each request spins up its own self-contained in-memor