# Data Engineering Central Long Live the Data Engineer. No holds barred. This is one page of public article previews, not the complete archive. Follow Next page to continue. Summaries are not the original full articles. ## The Future of Data Engineering in the Age of AI | Erfan Hesami DevFeed: [The Future of Data Engineering in the Age of AI | Erfan Hesami]() Original publisher: [Read original article]() Author: Daniel Beach Published: 2026-09-16T13:21:19Z Content type: article Language: en Sources: [Data Engineering Central]() Topics: [data-engineering](), [AI Engineering](), [agent observability](), [coding](), [Security]() Tags: [agents](), [ai](), [ai-agents](), [ai-engineering](), [coding](), [data-engineering](), [fundamentals](), [governance](), [security]() ### AI overview An interview with Erfan Hesami examines how AI and agents may change data engineering, including the evolving role of data engineers, the overlap with AI engineering, the continuing importance of fundamentals, and the need to manage governance, security, costs, technical debt, and human judgment. ### Source excerpt AI Agents, Coding & Fundamentals ## Back to the Basics. Threads. DevFeed: [Back to the Basics. Threads.]() Original publisher: [Read original article]() Author: Daniel Beach Published: 2026-09-14T12:19:11Z Content type: tutorial Language: en Sources: [Data Engineering Central]() Topics: [Concurrency](), [Rust](), [Concurrent Programming](), [Programming](), [Python]() Tags: [concurrency](), [programming](), [python](), [rust](), [threads]() ### AI overview An introductory, applied discussion of threads and concurrency, focusing on Rust and briefly recalling Python ProcessPools for parallel data and file processing. ### Source excerpt staying grounded in an agentic world ## Big Tech Laid Him Off. He Realized He Didn't Need a Job. -- Asian Dad Energy DevFeed: [Big Tech Laid Him Off. He Realized He Didn't Need a Job. -- Asian Dad Energy]() Original publisher: [Read original article]() Author: Daniel Beach Published: 2026-09-09T14:07:55Z Content type: opinion Language: en Sources: [Data Engineering Central]() Topics: [Job](), [future of software](), [Software Engineering](), [Artificial Intelligence](), [cobol](), [mainframes](), [ibm]() Tags: [ai](), [career](), [cobol](), [future-of-software](), [ibm](), [job](), [layoff](), [mainframes](), [software-engineering](), [tech]() ### AI overview A Data Engineering Central Podcast episode features AsianDadEnergy discussing his career from BASIC, IBM 386 systems, and COBOL mainframes to consulting and a chief architect role in Big Tech, followed by a layoff. The conversation covers financial independence, AI's effects on software engineering, communication skills, identity beyond work, and content creation. ### Source excerpt Data Engineering Central Podcast ## Shehab Amin on Spark Compatibility, Rust, and LakeSail DevFeed: [Shehab Amin on Spark Compatibility, Rust, and LakeSail]() Original publisher: [Read original article]() Author: Daniel Beach Published: 2026-09-02T12:22:23Z Content type: article Language: en Sources: [Data Engineering Central]() Topics: [Apache Spark](), [Rust](), [data-engineering](), [apache-arrow](), [agentic-coding]() Tags: [apache-arrow](), [apache-spark](), [data-engineering](), [rust]() ### AI overview A podcast conversation with LakeSail co-founder and CEO Shehab Amin about Spark compatibility, Rust, Apache Arrow, DataFusion, and data infrastructure. It also discusses data-stack choices, streaming and batch processing, agentic coding, and using existing data pipelines as a basis for AI pipelines. ### Source excerpt Data Engineering Central Podcast. ## Omnigent: A Meta-Harness for Coding Agents DevFeed: [Omnigent: A Meta-Harness for Coding Agents]() Original publisher: [Read original article]() Author: Daniel Beach Published: 2026-08-31T12:04:20Z Content type: opinion Language: en Sources: [Data Engineering Central]() Topics: [Artificial Intelligence](), [AI-assisted coding](), [Claude Code](), [codex](), [Model Context Protocol]() Tags: [agents](), [bash](), [claude-code](), [codex](), [coding-agents](), [cursor](), [mcp]() ### AI overview The author introduces Omnigent as a meta-harness for agents while discussing the overload of coding-agent tools, configurations, plugins, MCP servers, and Bash-command controls. ### Source excerpt I got you boo ## Lakesail: A Rust-based alternative to Spark without the JVM DevFeed: [Lakesail: A Rust-based alternative to Spark without the JVM]() Original publisher: [Read original article]() Author: Daniel Beach Published: 2026-08-18T21:11:44Z Content type: opinion Language: en Sources: [Data Engineering Central]() Topics: [Apache Spark](), [Rust]() Tags: [apache-spark](), [rust](), [spark]() ### AI overview This commentary article introduces Lakesail as a Rust-based implementation or alternative to Spark that avoids the JVM overhead. It places Lakesail within the broader data-engineering ecosystem, where established tools such as Spark and Kafka continue to coexist with newer projects including Polars, DuckDB, and Daft. ### Source excerpt ya know ... black magic stuff ## Quasi-Agentic Pipelines with Databricks and Apache Airflow DevFeed: [Quasi-Agentic Pipelines with Databricks and Apache Airflow]() Original publisher: [Read original article]() Author: Daniel Beach Published: 2026-08-10T21:23:57Z Content type: tutorial Language: en Sources: [Data Engineering Central]() Topics: [databricks](), [airflow](), [Data pipelines](), [Large Language Model](), [data-engineering]() Tags: [airflow](), [apache-airflow](), [data-engineering](), [databricks](), [llms](), [pipelines]() ### AI overview A practical developer discussion of incorporating LLMs and agents into existing data workflows using Databricks and Apache Airflow. It also examines determinism in data pipelines and the gap between business requirements and engineering implementation. ### Source excerpt the strange space in between ## Keeping Technical Skills in the Age of the LLM DevFeed: [Keeping Technical Skills in the Age of the LLM]() Original publisher: [Read original article]() Author: Daniel Beach Published: 2026-08-03T12:03:12Z Content type: opinion Language: en Sources: [Data Engineering Central]() Topics: [Large Language Model](), [coding](), [Claude]() Tags: [claude](), [llm](), [model](), [technical]() ### AI overview The article reflects on the challenge of maintaining technical skills as large language models become part of software development. The author describes personal uncertainty about when to use Claude versus writing code independently, placing that dilemma in the context of a lifelong interest in mechanical and digital tinkering. ### Source excerpt the end of the era and new one beginning ## Data Modeling's Relevance in Modern Data Engineering DevFeed: [Data Modeling's Relevance in Modern Data Engineering]() Original publisher: [Read original article]() Author: Daniel Beach Published: 2026-07-27T12:20:21Z Content type: opinion Language: en Sources: [Data Engineering Central]() Topics: [data-modeling](), [data](), [Artificial Intelligence]() Tags: [ai](), [article](), [data-modeling](), [opinions]() ### AI overview This opinion article examines whether data modeling remains relevant in modern data work. It presents differing views, notes the longstanding role of relational modeling, and questions why the topic has recently been treated as potentially obsolete. ### Source excerpt was it never alive? ## Why I Miss Writing Rust Code by Hand DevFeed: [Why I Miss Writing Rust Code by Hand]() Original publisher: [Read original article]() Author: Daniel Beach Published: 2026-07-23T12:47:19Z Content type: opinion Language: en Sources: [Data Engineering Central]() Topics: [Rust](), [Code](), [systems]() Tags: [code](), [rust]() ### AI overview A personal commentary about missing the challenge of writing Rust code by hand. The author argues that exercising the mind through detailed software work supports effective systems design and software problem-solving, while also mentioning broader responsibilities such as architecture, business requirements, and project planning. ### Source excerpt back to the old days ## Agentic Data Engineering Is Here -- But Can It Close the Loop? DevFeed: [Agentic Data Engineering Is Here -- But Can It Close the Loop?]() Original publisher: [Read original article]() Author: Daniel Beach Published: 2026-07-22T14:05:27Z Content type: article Language: en Sources: [Data Engineering Central]() Topics: [data-engineering](), [Orchestration](), [data observability](), [Artificial Intelligence](), [AI-assisted coding](), [DuckDB](), [Apache Iceberg]() Tags: [agentic](), [ai-agents](), [data-engineering](), [duckdb](), [orchestration](), [pipelines]() ### AI overview A podcast conversation with Hugo Lu about agentic data engineering and the infrastructure needed for data platforms to execute work, observe outcomes, validate changes, and improve pipelines safely. It examines why production data systems remain difficult for AI, including schema changes, realistic testing, business semantics, and secure execution. ### Source excerpt a conversation with Hugo Lu ## ADBC: Arrow's Cross-Platform Database Connectivity API DevFeed: [ADBC: Arrow's Cross-Platform Database Connectivity API]() Original publisher: [Read original article]() Author: Daniel Beach Published: 2026-07-20T13:57:32Z Content type: article Language: en Sources: [Data Engineering Central]() Topics: [API](), [Databases](), [data](), [client]() Tags: [apis](), [arrow](), [client](), [data](), [database](), [rust]() ### AI overview This article explains ADBC, Arrow's database connectivity API. It describes ADBC implementations across several languages and databases, and compares its columnar, minimal-overhead approach with JDBC and ODBC for working with Arrow data. ### Source excerpt Guest Post by Anonymous Rust Dev ## Michael Drogalis on Building ShadowTraffic as a Solo B2B Software Company DevFeed: [Michael Drogalis on Building ShadowTraffic as a Solo B2B Software Company]() Original publisher: [Read original article]() Author: Daniel Beach Published: 2026-07-15T13:24:12Z Content type: article Language: en Sources: [Data Engineering Central]() Topics: [Software Engineering](), [Kafka](), [stream-processing]() Tags: [kafka](), [software-engineering](), [stream-processing]() ### AI overview A conversation with Michael Drogalis about building ShadowTraffic as a solo B2B software company. It discusses realistic production-traffic generation for testing, demos, and development, along with customer discovery, marketing, AI use, and engineering careers. ### Source excerpt A Conversation with Michael Drogalis ## Evaluating Cloudflare as a Data Platform DevFeed: [Evaluating Cloudflare as a Data Platform]() Original publisher: [Read original article]() Author: Daniel Beach Published: 2026-07-13T12:39:09Z Content type: article Language: en Sources: [Data Engineering Central]() Topics: [Cloudflare](), [data](), [Apache Iceberg]() Tags: [apache-iceberg](), [cloudflare](), [data](), [data-platform]() ### AI overview The article examines Cloudflare's data products and whether they can support a complete data stack, including how to build with them and what they cost. The supplied excerpt establishes the article's scope but does not provide its final findings. ### Source excerpt new kid on the block ## The Creator of Pandas on AI, Apache Arrow, and the Future of Software Engineering DevFeed: [The Creator of Pandas on AI, Apache Arrow, and the Future of Software Engineering]() Original publisher: [Read original article]() Author: Daniel Beach Published: 2026-07-08T12:16:09Z Content type: article Language: en Sources: [Data Engineering Central]() Topics: [pandas](), [data-engineering](), [Artificial Intelligence](), [DuckDB](), [data-platforms](), [software-development](), [future of software](), [Apache Spark](), [Hadoop](), [Open Source](), [parquet]() Tags: [ai](), [apache-arrow](), [arrow](), [data](), [data-engineering](), [data-platforms](), [duckdb](), [hadoop](), [large-language-models](), [open-source](), [pandas](), [parquet](), [software-development]() ### AI overview An interview with Wes McKinney covers the origins of pandas and Apache Arrow, the evolution of modern data engineering from Hadoop to lakehouse architectures, and the roles of tools such as Parquet, DuckDB, DataFusion, and Spark. McKinney also discusses how AI affects software development, arguing that it can improve experienced engineers' productivity but does not replace software engineering, architecture, or judgment. ### Source excerpt interview with Wes McKinney ## AWS Lambda MicroVMs Are Positioned Between Lambda and EC2 DevFeed: [AWS Lambda MicroVMs Are Positioned Between Lambda and EC2]() Original publisher: [Read original article]() Author: Daniel Beach Published: 2026-07-06T12:24:57Z Content type: opinion Language: en Sources: [Data Engineering Central]() Topics: [AWS Lambda](), [Amazon EC2](), [Amazon Web Services]() Tags: [aws](), [aws-lambda](), [ec2](), [lambda]() ### AI overview The article offers commentary on AWS Lambda MicroVMs, describing them as a compute option between AWS Lambda and EC2 instances. It considers whether they address broader demand or a narrower niche, but the supplied text does not provide a definitive conclusion. ### Source excerpt example with Polars and CSVs ## From COBOL to Copilot: 30 Years of Data, BI, and AI with David Langer DevFeed: [From COBOL to Copilot: 30 Years of Data, BI, and AI with David Langer]() Original publisher: [Read original article]() Author: Daniel Beach Published: 2026-07-01T13:43:11Z Content type: article Language: en Sources: [Data Engineering Central]() Topics: [Artificial Intelligence](), [cobol](), [Data Science](), [mainframes](), [Software Engineering](), [AI Strategy](), [self-service](), [dashboards](), [jupyter notebooks]() Tags: [adoption](), [ai](), [business-intelligence](), [cobol](), [data-science](), [enterprise](), [jupyter-notebooks](), [mainframes](), [programming](), [self-service](), [semantic]() ### AI overview A podcast conversation with Dave Langer about nearly three decades spanning COBOL, enterprise architecture, business intelligence, analytics, data science, machine learning, and AI. It discusses persistent data-industry problems, self-service analytics, dimensional modeling, AI adoption, semantic layers, governance, and career advice for data professionals. ### Source excerpt What happens when someone who started programming on a Commodore 64 watches AI reshape the entire data industry?