# Data Engineering Central

Long Live the Data Engineer. No holds barred.

This is one page of public article previews, not the complete archive. Follow Next page to continue. Summaries are not the original full articles.

## What the !@#$% is Ontology?

DevFeed: [What the !@#$% is Ontology?](<https://devfeed.tech/articles/what-the-is-ontology-136528.md>)

Original publisher: [Read original article](<https://dataengineeringcentral.substack.com/p/what-the-is-ontology>)

Author: Daniel Beach

Published: 2026-11-16T14:43:28Z

Content type: article

Language: en

Sources: [Data Engineering Central](<https://devfeed.tech/sources/data-engineering-central.md>)

Topics: [semantic-layer](<https://devfeed.tech/topics/semantic-layer.md>), [AI Context Management](<https://devfeed.tech/topics/ai-context-management.md>), [business logic](<https://devfeed.tech/topics/business-logic.md>), [Data Management](<https://devfeed.tech/topics/data-management.md>)

Tags: [2](<https://devfeed.tech/tags/2.md>), [2006](<https://devfeed.tech/tags/2006.md>), [2025](<https://devfeed.tech/tags/2025.md>), [3](<https://devfeed.tech/tags/3.md>), [about](<https://devfeed.tech/tags/about.md>), [absolute](<https://devfeed.tech/tags/absolute.md>), [acceptance](<https://devfeed.tech/tags/acceptance.md>), [agent](<https://devfeed.tech/tags/agent.md>), [agent-context](<https://devfeed.tech/tags/agent-context.md>), [agentcore](<https://devfeed.tech/tags/agentcore.md>), [agentic](<https://devfeed.tech/tags/agentic.md>), [agentic-tooling](<https://devfeed.tech/tags/agentic-tooling.md>), [agentic-world](<https://devfeed.tech/tags/agentic-world.md>), [agents](<https://devfeed.tech/tags/agents.md>), [ai](<https://devfeed.tech/tags/ai.md>), [ai-search](<https://devfeed.tech/tags/ai-search.md>), [data](<https://devfeed.tech/tags/data.md>), [data-mesh](<https://devfeed.tech/tags/data-mesh.md>), [mesh](<https://devfeed.tech/tags/mesh.md>), [ontology](<https://devfeed.tech/tags/ontology.md>)

### AI overview

The article frames ontology as a machine-readable model of domain concepts, relationships, and rules that gives AI agents business context. It argues that this extends a longstanding data problem: encoding institutional knowledge so technology can use it consistently. Its examples show different approaches. NVIDIA Auto-Ontology scans SQL data sources, proposes business terms, metrics, and joins for human review, then answers questions with SQL and supporting steps; the author sees it as infrastructure-heavy and notes its apparent lack of document ingestion. AWS's Context Ontology Accelerator is tightly integrated with AWS services and costly to run, but can build knowledge graphs from PDFs and Word files. Databricks Genie Ontology combines human-authored context with knowledge extracted from data, then ranks and delivers relevant information to agents. The article says open-source options are scarce and treats the distinction between ontology and semantic layers as unsettled; it calls for evaluations showing whether the added infrastructure improves results enough to justify its cost.

### Source excerpt

the new Data Mesh?

## AI Is Changing Software Engineering Forever | Gregor Ojstersek

DevFeed: [AI Is Changing Software Engineering Forever | Gregor Ojstersek](<https://devfeed.tech/articles/ai-is-changing-software-engineering-forever-gregor-ojstersek-129778.md>)

Original publisher: [Read original article](<https://dataengineeringcentral.substack.com/p/ai-is-changing-software-engineering>)

Author: Daniel Beach

Published: 2026-11-11T13:25:32Z

Content type: article

Language: en

Sources: [Data Engineering Central](<https://devfeed.tech/sources/data-engineering-central.md>)

Topics: [Software Engineering](<https://devfeed.tech/topics/software-engineering.md>), [engineering-leadership](<https://devfeed.tech/topics/engineering-leadership.md>), [engineering-culture](<https://devfeed.tech/topics/engineering-culture.md>)

Tags: [advice](<https://devfeed.tech/tags/advice.md>), [age-of-ai](<https://devfeed.tech/tags/age-of-ai.md>), [ai](<https://devfeed.tech/tags/ai.md>), [ai-generated-code](<https://devfeed.tech/tags/ai-generated-code.md>), [book](<https://devfeed.tech/tags/book.md>), [career](<https://devfeed.tech/tags/career.md>), [code](<https://devfeed.tech/tags/code.md>), [computer-science](<https://devfeed.tech/tags/computer-science.md>), [cto](<https://devfeed.tech/tags/cto.md>), [data-engineering](<https://devfeed.tech/tags/data-engineering.md>), [developer](<https://devfeed.tech/tags/developer.md>), [engineer](<https://devfeed.tech/tags/engineer.md>), [engineering](<https://devfeed.tech/tags/engineering.md>), [engineering-jobs](<https://devfeed.tech/tags/engineering-jobs.md>), [engineering-leadership](<https://devfeed.tech/tags/engineering-leadership.md>), [engineering-management](<https://devfeed.tech/tags/engineering-management.md>), [engineers](<https://devfeed.tech/tags/engineers.md>), [fundamentals](<https://devfeed.tech/tags/fundamentals.md>), [software](<https://devfeed.tech/tags/software.md>), [software-engineering](<https://devfeed.tech/tags/software-engineering.md>)

### AI overview

The episode's central takeaway is that writing code may matter less as AI changes engineering work, while engineering fundamentals, problem-solving, and leadership remain important. Gregor Ojstersek discusses how AI may affect junior through principal roles, engineering management, and the path to CTO; the risks of AI-generated code, weak engineering practices, and losing fundamental skills; and why choosing the right work matters. The excerpt also describes his journey from self-taught programmer to CTO, his book The Multiplier Mindset, and the growth of his Engineering Leadership Newsletter to over 200,000 subscribers. It lists topics and chapter timestamps but does not provide the detailed discussion or specific advice.

### Source excerpt

From Engineer to CTO

## Life After Big Tech

DevFeed: [Life After Big Tech](<https://devfeed.tech/articles/life-after-big-tech-117851.md>)

Original publisher: [Read original article](<https://dataengineeringcentral.substack.com/p/life-after-big-tech>)

Author: Daniel Beach

Published: 2026-11-04T14:34:31Z

Content type: article

Language: en

Sources: [Data Engineering Central](<https://devfeed.tech/sources/data-engineering-central.md>)

Topics: [Finance](<https://devfeed.tech/topics/finance.md>)

Tags: [2026](<https://devfeed.tech/tags/2026.md>), [ai](<https://devfeed.tech/tags/ai.md>), [apple](<https://devfeed.tech/tags/apple.md>), [big-tech](<https://devfeed.tech/tags/big-tech.md>), [career](<https://devfeed.tech/tags/career.md>), [financial](<https://devfeed.tech/tags/financial.md>), [fire](<https://devfeed.tech/tags/fire.md>), [inflation](<https://devfeed.tech/tags/inflation.md>), [job](<https://devfeed.tech/tags/job.md>), [layoffs](<https://devfeed.tech/tags/layoffs.md>), [leaving](<https://devfeed.tech/tags/leaving.md>), [life](<https://devfeed.tech/tags/life.md>), [lifestyle](<https://devfeed.tech/tags/lifestyle.md>), [mindset](<https://devfeed.tech/tags/mindset.md>), [money](<https://devfeed.tech/tags/money.md>), [mountains](<https://devfeed.tech/tags/mountains.md>), [nevada](<https://devfeed.tech/tags/nevada.md>), [next](<https://devfeed.tech/tags/next.md>), [outdoors](<https://devfeed.tech/tags/outdoors.md>), [tech](<https://devfeed.tech/tags/tech.md>)

### AI overview

After nearly 13 years at Apple, Kevin Martin left corporate work and completed hundreds of miles of backpacking, including the Benton MacKaye and John Muir trails. He reflects that leaving the corporate world did not automatically shed the intense planning and expectations that had made him successful there, raising the question of how to enjoy financial independence rather than chase another milestone. The conversation also touches on Big Tech's changing landscape, AI-driven layoffs, lifestyle inflation, FIRE, money psychology, and how difficult outdoor experiences can shift perspective. The supplied text is an episode description, so it does not provide detailed arguments or practical financial guidance.

### Source excerpt

Bears, Mountains, and Finding Purpose

## Databricks. Data Quality. MPC Server. SDK.

DevFeed: [Databricks. Data Quality. MPC Server. SDK.](<https://devfeed.tech/articles/databricks-data-quality-mpc-server-sdk-115747.md>)

Original publisher: [Read original article](<https://dataengineeringcentral.substack.com/p/databricks-data-quality-mpc-server>)

Author: Daniel Beach

Published: 2026-11-03T13:19:51Z

Content type: opinion

Language: en

Sources: [Data Engineering Central](<https://devfeed.tech/sources/data-engineering-central.md>)

Topics: [AI Chat](<https://devfeed.tech/topics/ai-chat.md>), [databricks](<https://devfeed.tech/topics/databricks.md>), [Data Quality](<https://devfeed.tech/topics/data-quality.md>), [SDK](<https://devfeed.tech/topics/sdk.md>)

Tags: [agent](<https://devfeed.tech/tags/agent.md>), [agentic](<https://devfeed.tech/tags/agentic.md>), [agents](<https://devfeed.tech/tags/agents.md>), [ai](<https://devfeed.tech/tags/ai.md>), [ai-hype](<https://devfeed.tech/tags/ai-hype.md>), [ai-slop](<https://devfeed.tech/tags/ai-slop.md>), [analysis](<https://devfeed.tech/tags/analysis.md>), [code](<https://devfeed.tech/tags/code.md>), [data](<https://devfeed.tech/tags/data.md>), [data-analysis](<https://devfeed.tech/tags/data-analysis.md>), [data-context](<https://devfeed.tech/tags/data-context.md>), [data-platform](<https://devfeed.tech/tags/data-platform.md>), [data-quality](<https://devfeed.tech/tags/data-quality.md>), [databricks](<https://devfeed.tech/tags/databricks.md>), [drive](<https://devfeed.tech/tags/drive.md>), [engineer](<https://devfeed.tech/tags/engineer.md>), [english](<https://devfeed.tech/tags/english.md>), [google-drive](<https://devfeed.tech/tags/google-drive.md>), [mpc](<https://devfeed.tech/tags/mpc.md>), [sdk](<https://devfeed.tech/tags/sdk.md>)

### AI overview

Andreas Kretz's video demonstrates building a Data Quality Assistant connected to Databricks and Google Drive, using MPC and SDKs from Zapier. The author describes shifting from manually researching data and building charts with SQL and Spark to using English instructions and domain knowledge to direct agents, while taking a researcher role. The post presents this as a practical example of how agentic tools might reduce technical workload and help explore data, while emphasizing that users still need to check the tools' work for accuracy. It treats the specific tools as less important than learning how to apply LLMs, agents, RAG, and MPC servers to data problems. This overview is based on the supplied excerpt.

### Source excerpt

we live in a new world

## Polars 2.0 vs DuckDB 2.0

DevFeed: [Polars 2.0 vs DuckDB 2.0](<https://devfeed.tech/articles/polars-2-0-vs-duckdb-2-0-112907.md>)

Original publisher: [Read original article](<https://dataengineeringcentral.substack.com/p/polars-20-vs-duckdb-20>)

Author: Daniel Beach

Published: 2026-11-02T13:11:02Z

Content type: comparison

Language: en

Sources: [Data Engineering Central](<https://devfeed.tech/sources/data-engineering-central.md>)

Topics: [DuckDB](<https://devfeed.tech/topics/duckdb.md>), [Apache Iceberg](<https://devfeed.tech/topics/apache-iceberg.md>), [apache-arrow](<https://devfeed.tech/topics/apache-arrow.md>)

Tags: [aggregation](<https://devfeed.tech/tags/aggregation.md>), [benchmark](<https://devfeed.tech/tags/benchmark.md>), [benchmarking](<https://devfeed.tech/tags/benchmarking.md>), [benchmarks](<https://devfeed.tech/tags/benchmarks.md>), [duckdb](<https://devfeed.tech/tags/duckdb.md>), [duckdb-2-0](<https://devfeed.tech/tags/duckdb-2-0.md>), [polars](<https://devfeed.tech/tags/polars.md>), [vs](<https://devfeed.tech/tags/vs.md>)

### AI overview

The author's 2 GB Linode test processed about 23 GB of CSV data on S3 with aggregation and window operations. DuckDB 2.0 completed in 837 seconds. Polars 2.0 failed even when the machine was increased to 8 GB; the author attributes this to then-missing spill-to-disk support for group-bys, which the cited release notes said was planned. After Polars 2.1 added group-by and window spill support, it completed the same test on 2 GB in 438 seconds, roughly half DuckDB's reported time. The comparison reflects this specific setup and query. The text also describes Polars 2.0's streaming default, SQL support, and schema checks, and DuckDB 2.0's Quack networking protocol, asynchronous Parquet/CSV I/O, VARIANT type, and triggers.

### Source excerpt

who's been up to what?

## DuckDB + ArrowFlight = GizmoData

DevFeed: [DuckDB + ArrowFlight = GizmoData](<https://devfeed.tech/articles/duckdb-arrowflight-gizmodata-92869.md>)

Original publisher: [Read original article](<https://dataengineeringcentral.substack.com/p/duckdb-arrowflight-gizmodata>)

Author: Daniel Beach

Published: 2026-10-21T13:36:34Z

Content type: tutorial

Language: en

Sources: [Data Engineering Central](<https://devfeed.tech/sources/data-engineering-central.md>)

Topics: [DuckDB](<https://devfeed.tech/topics/duckdb.md>), [Apache Iceberg](<https://devfeed.tech/topics/apache-iceberg.md>), [elt](<https://devfeed.tech/topics/elt.md>), [scheduling](<https://devfeed.tech/topics/scheduling.md>)

Tags: [ai](<https://devfeed.tech/tags/ai.md>), [airflow](<https://devfeed.tech/tags/airflow.md>), [apache-airflow](<https://devfeed.tech/tags/apache-airflow.md>), [apache-arrow](<https://devfeed.tech/tags/apache-arrow.md>), [apache-iceberg](<https://devfeed.tech/tags/apache-iceberg.md>), [apache-spark](<https://devfeed.tech/tags/apache-spark.md>), [arrow](<https://devfeed.tech/tags/arrow.md>), [authentication](<https://devfeed.tech/tags/authentication.md>), [aws](<https://devfeed.tech/tags/aws.md>), [backblaze](<https://devfeed.tech/tags/backblaze.md>), [benchmark](<https://devfeed.tech/tags/benchmark.md>), [benchmarks](<https://devfeed.tech/tags/benchmarks.md>), [bi-tools](<https://devfeed.tech/tags/bi-tools.md>), [bike](<https://devfeed.tech/tags/bike.md>), [binaries](<https://devfeed.tech/tags/binaries.md>), [catalog](<https://devfeed.tech/tags/catalog.md>), [cloud](<https://devfeed.tech/tags/cloud.md>), [cloud-storage](<https://devfeed.tech/tags/cloud-storage.md>), [cluster](<https://devfeed.tech/tags/cluster.md>), [duckdb](<https://devfeed.tech/tags/duckdb.md>)

### AI overview

GizmoSQL wraps DuckDB in a client-server service using Apache Arrow Flight SQL to handle remote query execution and columnar results. The article describes running the open-source server with Docker, querying it from Python via ADBC, and integrating it with dbt and Airflow. In a described test, eight writer processes sent batches through separate ADBC sessions while readers polled; the author reports no failures. For a larger cloud workload, a two-thread, 7.4 GiB cluster processed a three-year Iceberg dataset from S3 in 17 minutes 54 seconds, using 12 monthly chunks with 31-day margins for window calculations. The author says local DuckDB checks also worked with Iceberg and Delta Lake, while identifying first-class Unity and Polaris catalog support and pipeline features as areas they would like to see added.

### Source excerpt

let the games begin!

## Building a Rust tool with Delta Kernel to import and export Delta Lake tables

DevFeed: [Building a Rust tool with Delta Kernel to import and export Delta Lake tables](<https://devfeed.tech/articles/delta-kernel-with-rust-doing-stuff-90337.md>)

Original publisher: [Read original article](<https://dataengineeringcentral.substack.com/p/delta-kernel-with-rust-doing-stuff>)

Author: Daniel Beach

Published: 2026-10-19T13:17:32Z

Content type: tutorial

Language: en

Sources: [Data Engineering Central](<https://devfeed.tech/sources/data-engineering-central.md>)

Topics: [deltalake](<https://devfeed.tech/topics/deltalake.md>), [Rust](<https://devfeed.tech/topics/rust.md>)

Tags: [build-tools](<https://devfeed.tech/tags/build-tools.md>), [cli](<https://devfeed.tech/tags/cli.md>), [clickhouse](<https://devfeed.tech/tags/clickhouse.md>), [kernel](<https://devfeed.tech/tags/kernel.md>), [rust](<https://devfeed.tech/tags/rust.md>)

### AI overview

The article introduces Delta Kernel as a reusable library that helps query engines read and write Delta Lake tables without implementing the protocol themselves. It walks through building a Rust command-line tool that exports an S3-hosted Delta table to Parquet and imports CSV data into a new Delta table, while noting that the Rust implementation is experimental and that production use may need additional support for multiple writers and Unity Catalog.

### Source excerpt

look at me go ma!

## The Modern CDP Manifesto

DevFeed: [The Modern CDP Manifesto](<https://devfeed.tech/articles/the-modern-cdp-manifesto-87193.md>)

Original publisher: [Read original article](<https://dataengineeringcentral.substack.com/p/the-modern-cdp-manifesto>)

Author: Daniel Beach

Published: 2026-10-14T14:31:26Z

Content type: opinion

Language: en

Sources: [Data Engineering Central](<https://devfeed.tech/sources/data-engineering-central.md>)

Topics: [data-engineering](<https://devfeed.tech/topics/data-engineering.md>), [AI for customer service automation](<https://devfeed.tech/topics/ai-for-customer-service-automation.md>), [Data pipelines](<https://devfeed.tech/topics/data-pipelines.md>), [internal developer platform automation](<https://devfeed.tech/topics/internal-developer-platform-automation.md>), [AI Chat](<https://devfeed.tech/topics/ai-chat.md>), [AI Infrastructure](<https://devfeed.tech/topics/ai-infrastructure.md>)

Tags: [agent](<https://devfeed.tech/tags/agent.md>), [ai](<https://devfeed.tech/tags/ai.md>), [cdp](<https://devfeed.tech/tags/cdp.md>), [data](<https://devfeed.tech/tags/data.md>), [data-platform](<https://devfeed.tech/tags/data-platform.md>), [platform](<https://devfeed.tech/tags/platform.md>)

### AI overview

The article defines a customer data platform (CDP) as infrastructure for collecting, identifying, modeling, governing, and activating customer data across systems. It argues that modern CDPs need reliable data foundations, clear governance and access boundaries, machine-readable interfaces, and support for agents that analyze data and take action. It also describes how CDPs can work alongside warehouses and lakehouses to reduce the complexity of fragmented customer data and manual workflows.

### Source excerpt

Building a Modern (agentic) Customer Data Platform From the Ground Up

## DuckDB vs Polars vs Spark on Databricks Serverless with Joins and Window Functions.

DevFeed: [DuckDB vs Polars vs Spark on Databricks Serverless with Joins and Window Functions.](<https://devfeed.tech/articles/duckdb-vs-polars-vs-spark-on-databricks-serverless-with-joins-and-window-functions-65104.md>)

Original publisher: [Read original article](<https://dataengineeringcentral.substack.com/p/duckdb-vs-polars-vs-spark-on-databricks>)

Author: Daniel Beach

Published: 2026-10-05T11:56:19Z

Content type: comparison

Language: en

Sources: [Data Engineering Central](<https://devfeed.tech/sources/data-engineering-central.md>)

Topics: [DuckDB](<https://devfeed.tech/topics/duckdb.md>), [databricks](<https://devfeed.tech/topics/databricks.md>), [Apache Spark](<https://devfeed.tech/topics/spark.md>), [Databricks Unity Catalog integration](<https://devfeed.tech/topics/databricks-unity-catalog-integration.md>)

Tags: [databricks](<https://devfeed.tech/tags/databricks.md>), [duckdb](<https://devfeed.tech/tags/duckdb.md>), [serverless](<https://devfeed.tech/tags/serverless.md>), [spark](<https://devfeed.tech/tags/spark.md>), [window-functions](<https://devfeed.tech/tags/window-functions.md>)

### AI overview

The author compares DuckDB, Polars, and Spark on Databricks Serverless using a large dataset and join/window-function workload. In this informal benchmark, DuckDB ran only after bypassing Unity Catalog and accessing remote Delta tables directly, Polars failed during execution with memory-related errors, and Spark on Serverless was reported as fastest. The author cautions that the test was not scientific.

### Source excerpt

hehehehehe

## AI Is a Mirror: MLOps, LLMOps & the Future of AI Engineering | Maria Vechtomova

DevFeed: [AI Is a Mirror: MLOps, LLMOps & the Future of AI Engineering | Maria Vechtomova](<https://devfeed.tech/articles/ai-is-a-mirror-mlops-llmops-the-future-of-ai-engineering-maria-vechtomova-62786.md>)

Original publisher: [Read original article](<https://dataengineeringcentral.substack.com/p/ai-is-a-mirror-mlops-llmops-and-the>)

Author: Daniel Beach

Published: 2026-09-30T12:58:38Z

Content type: article

Language: en

Sources: [Data Engineering Central](<https://devfeed.tech/sources/data-engineering-central.md>)

Topics: [MLOps](<https://devfeed.tech/topics/mlops.md>), [AI Engineering](<https://devfeed.tech/topics/ai-engineering.md>), [Prompt Engineering](<https://devfeed.tech/topics/prompt-engineering.md>), [Software Engineering](<https://devfeed.tech/topics/software-engineering.md>), [data observability](<https://devfeed.tech/topics/data-observability.md>), [rachels-ramblings](<https://devfeed.tech/topics/rachels-ramblings.md>)

Tags: [ai](<https://devfeed.tech/tags/ai.md>), [ai-assisted](<https://devfeed.tech/tags/ai-assisted.md>), [ai-engineering](<https://devfeed.tech/tags/ai-engineering.md>), [ai-generated-code](<https://devfeed.tech/tags/ai-generated-code.md>), [code-testing](<https://devfeed.tech/tags/code-testing.md>), [governance](<https://devfeed.tech/tags/governance.md>), [llmops](<https://devfeed.tech/tags/llmops.md>), [observability](<https://devfeed.tech/tags/observability.md>)

### AI overview

This podcast conversation examines how production machine learning has evolved into modern MLOps and LLMOps, including the challenges of evaluating, monitoring, tracing, and governing LLM applications and agents. Maria Vechtomova discusses AI coding tools, testing generated code, and the continuing importance of engineering judgment and practical skills.

### Source excerpt

Data Engineering Central Podcast

## From Podcast Video to Searchable Insight with Databricks FILE data type

DevFeed: [From Podcast Video to Searchable Insight with Databricks FILE data type](<https://devfeed.tech/articles/from-podcast-video-to-searchable-insight-with-databricks-file-data-type-61297.md>)

Original publisher: [Read original article](<https://dataengineeringcentral.substack.com/p/from-podcast-video-to-searchable>)

Author: Daniel Beach

Published: 2026-09-28T15:51:45Z

Content type: article

Language: en

Sources: [Data Engineering Central](<https://devfeed.tech/sources/data-engineering-central.md>)

Topics: [databricks](<https://devfeed.tech/topics/databricks.md>), [Databricks Unity Catalog integration](<https://devfeed.tech/topics/databricks-unity-catalog-integration.md>), [data](<https://devfeed.tech/topics/data.md>)

Tags: [data-processing](<https://devfeed.tech/tags/data-processing.md>), [databricks](<https://devfeed.tech/tags/databricks.md>), [lakehouse](<https://devfeed.tech/tags/lakehouse.md>)

### AI overview

The article discusses Databricks' FILE data type for bringing unstructured assets such as video and audio into the Lakehouse. It highlights unified data management and processing, Unity Catalog governance, and reduced architectural complexity.

### Source excerpt

the Lakehouse expands to multi-modal

## Databricks LLM (APIs)/Modeling Serving. Thoughts from the Frontline.

DevFeed: [Databricks LLM (APIs)/Modeling Serving. Thoughts from the Frontline.](<https://devfeed.tech/articles/databricks-llm-apis-modeling-serving-thoughts-from-the-frontline-55690.md>)

Original publisher: [Read original article](<https://dataengineeringcentral.substack.com/p/databricks-llm-apismodeling-serving>)

Author: Daniel Beach

Published: 2026-09-20T23:48:19Z

Content type: article

Language: en

Sources: [Data Engineering Central](<https://devfeed.tech/sources/data-engineering-central.md>)

Topics: [databricks](<https://devfeed.tech/topics/databricks.md>), [Large Language Model](<https://devfeed.tech/topics/llm.md>), [model-serving](<https://devfeed.tech/topics/model-serving.md>), [Artificial Intelligence](<https://devfeed.tech/topics/ai.md>), [data-engineering](<https://devfeed.tech/topics/data-engineering.md>), [Machine learning](<https://devfeed.tech/topics/machine-learning.md>)

Tags: [agentic](<https://devfeed.tech/tags/agentic.md>), [ai](<https://devfeed.tech/tags/ai.md>), [apis](<https://devfeed.tech/tags/apis.md>), [databricks](<https://devfeed.tech/tags/databricks.md>), [llm](<https://devfeed.tech/tags/llm.md>), [model-serving](<https://devfeed.tech/tags/model-serving.md>)

### AI overview

The author shares a conceptual perspective on serving LLM traffic through Databricks and choosing options for LLM or agentic applications. The discussion emphasizes that decisions depend on content, business context, cost, and engineering requirements.

### Source excerpt

a brave new world

## The Future of Data Engineering in the Age of AI | Erfan Hesami

DevFeed: [The Future of Data Engineering in the Age of AI | Erfan Hesami](<https://devfeed.tech/articles/the-future-of-data-engineering-in-the-age-of-ai-erfan-hesami-38718.md>)

Original publisher: [Read original article](<https://dataengineeringcentral.substack.com/p/the-future-of-data-engineering-in>)

Author: Daniel Beach

Published: 2026-09-16T13:21:19Z

Content type: article

Language: en

Sources: [Data Engineering Central](<https://devfeed.tech/sources/data-engineering-central.md>)

Topics: [data-engineering](<https://devfeed.tech/topics/data-engineering.md>), [AI Engineering](<https://devfeed.tech/topics/ai-engineering.md>), [agent observability](<https://devfeed.tech/topics/agent-observability.md>), [Programming](<https://devfeed.tech/topics/programming.md>), [Security](<https://devfeed.tech/topics/security.md>)

Tags: [agents](<https://devfeed.tech/tags/agents.md>), [ai](<https://devfeed.tech/tags/ai.md>), [ai-agents](<https://devfeed.tech/tags/ai-agents.md>), [ai-engineering](<https://devfeed.tech/tags/ai-engineering.md>), [coding](<https://devfeed.tech/tags/coding.md>), [data-engineering](<https://devfeed.tech/tags/data-engineering.md>), [fundamentals](<https://devfeed.tech/tags/fundamentals.md>), [governance](<https://devfeed.tech/tags/governance.md>), [security](<https://devfeed.tech/tags/security.md>)

### AI overview

An interview with Erfan Hesami examines how AI and agents may change data engineering, including the evolving role of data engineers, the overlap with AI engineering, the continuing importance of fundamentals, and the need to manage governance, security, costs, technical debt, and human judgment.

### Source excerpt

AI Agents, Coding & Fundamentals

## Back to the Basics. Threads.

DevFeed: [Back to the Basics. Threads.](<https://devfeed.tech/articles/back-to-the-basics-threads-38706.md>)

Original publisher: [Read original article](<https://dataengineeringcentral.substack.com/p/back-to-the-basics-threads>)

Author: Daniel Beach

Published: 2026-09-14T12:19:11Z

Content type: tutorial

Language: en

Sources: [Data Engineering Central](<https://devfeed.tech/sources/data-engineering-central.md>)

Topics: [Concurrency](<https://devfeed.tech/topics/concurrency.md>), [Rust](<https://devfeed.tech/topics/rust.md>), [Programming](<https://devfeed.tech/topics/programming.md>), [Python](<https://devfeed.tech/topics/python.md>)

Tags: [concurrency](<https://devfeed.tech/tags/concurrency.md>), [programming](<https://devfeed.tech/tags/programming.md>), [python](<https://devfeed.tech/tags/python.md>), [rust](<https://devfeed.tech/tags/rust.md>), [threads](<https://devfeed.tech/tags/threads.md>)

### AI overview

An introductory, applied discussion of threads and concurrency, focusing on Rust and briefly recalling Python ProcessPools for parallel data and file processing.

### Source excerpt

staying grounded in an agentic world

## Big Tech Laid Him Off. He Realized He Didn't Need a Job. -- Asian Dad Energy

DevFeed: [Big Tech Laid Him Off. He Realized He Didn't Need a Job. -- Asian Dad Energy](<https://devfeed.tech/articles/big-tech-laid-him-off-he-realized-he-didn-t-need-a-job-asian-dad-energy-38707.md>)

Original publisher: [Read original article](<https://dataengineeringcentral.substack.com/p/big-tech-laid-him-off-he-realized>)

Author: Daniel Beach

Published: 2026-09-09T14:07:55Z

Content type: opinion

Language: en

Sources: [Data Engineering Central](<https://devfeed.tech/sources/data-engineering-central.md>)

Topics: [Job](<https://devfeed.tech/topics/job.md>), [future of software](<https://devfeed.tech/topics/future-of-software.md>), [Software Engineering](<https://devfeed.tech/topics/software-engineering.md>), [Artificial Intelligence](<https://devfeed.tech/topics/ai.md>), [cobol](<https://devfeed.tech/topics/cobol.md>), [Mainframe](<https://devfeed.tech/topics/mainframe.md>), [ibm](<https://devfeed.tech/topics/ibm.md>)

Tags: [ai](<https://devfeed.tech/tags/ai.md>), [career](<https://devfeed.tech/tags/career.md>), [cobol](<https://devfeed.tech/tags/cobol.md>), [future-of-software](<https://devfeed.tech/tags/future-of-software.md>), [ibm](<https://devfeed.tech/tags/ibm.md>), [job](<https://devfeed.tech/tags/job.md>), [layoff](<https://devfeed.tech/tags/layoff.md>), [mainframes](<https://devfeed.tech/tags/mainframes.md>), [software-engineering](<https://devfeed.tech/tags/software-engineering.md>), [tech](<https://devfeed.tech/tags/tech.md>)

### AI overview

A Data Engineering Central Podcast episode features AsianDadEnergy discussing his career from BASIC, IBM 386 systems, and COBOL mainframes to consulting and a chief architect role in Big Tech, followed by a layoff. The conversation covers financial independence, AI's effects on software engineering, communication skills, identity beyond work, and content creation.

### Source excerpt

Data Engineering Central Podcast

## Shehab Amin on Spark Compatibility, Rust, and LakeSail

DevFeed: [Shehab Amin on Spark Compatibility, Rust, and LakeSail](<https://devfeed.tech/articles/spark-isn-t-going-anywhere-so-they-rebuilt-it-in-rust-shehab-amin-ceo-of-lakesail-38716.md>)

Original publisher: [Read original article](<https://dataengineeringcentral.substack.com/p/spark-isnt-going-anywhere-so-they>)

Author: Daniel Beach

Published: 2026-09-02T12:22:23Z

Content type: article

Language: en

Sources: [Data Engineering Central](<https://devfeed.tech/sources/data-engineering-central.md>)

Topics: [Apache Spark](<https://devfeed.tech/topics/spark.md>), [Rust](<https://devfeed.tech/topics/rust.md>), [data-engineering](<https://devfeed.tech/topics/data-engineering.md>), [apache-arrow](<https://devfeed.tech/topics/apache-arrow.md>), [agentic-coding](<https://devfeed.tech/topics/agentic-coding.md>)

Tags: [apache-arrow](<https://devfeed.tech/tags/apache-arrow.md>), [apache-spark](<https://devfeed.tech/tags/apache-spark.md>), [data-engineering](<https://devfeed.tech/tags/data-engineering.md>), [rust](<https://devfeed.tech/tags/rust.md>)

### AI overview

A podcast conversation with LakeSail co-founder and CEO Shehab Amin about Spark compatibility, Rust, Apache Arrow, DataFusion, and data infrastructure. It also discusses data-stack choices, streaming and batch processing, agentic coding, and using existing data pipelines as a basis for AI pipelines.

### Source excerpt

Data Engineering Central Podcast.

## Omnigent: A Meta-Harness for Coding Agents

DevFeed: [Omnigent: A Meta-Harness for Coding Agents](<https://devfeed.tech/articles/omnigent-too-afraid-to-ask-what-a-meta-harness-for-agents-is-38712.md>)

Original publisher: [Read original article](<https://dataengineeringcentral.substack.com/p/omnigent-too-afraid-to-ask-what-a>)

Author: Daniel Beach

Published: 2026-08-31T12:04:20Z

Content type: opinion

Language: en

Sources: [Data Engineering Central](<https://devfeed.tech/sources/data-engineering-central.md>)

Topics: [Artificial Intelligence](<https://devfeed.tech/topics/ai.md>), [AI-assisted coding](<https://devfeed.tech/topics/ai-assisted-coding.md>), [Claude Code](<https://devfeed.tech/topics/claude-code.md>), [codex](<https://devfeed.tech/topics/codex.md>), [Model Context Protocol](<https://devfeed.tech/topics/model-context-protocol.md>)

Tags: [agents](<https://devfeed.tech/tags/agents.md>), [bash](<https://devfeed.tech/tags/bash.md>), [claude-code](<https://devfeed.tech/tags/claude-code.md>), [codex](<https://devfeed.tech/tags/codex.md>), [coding-agents](<https://devfeed.tech/tags/coding-agents.md>), [cursor](<https://devfeed.tech/tags/cursor.md>), [mcp](<https://devfeed.tech/tags/mcp.md>)

### AI overview

The author introduces Omnigent as a meta-harness for agents while discussing the overload of coding-agent tools, configurations, plugins, MCP servers, and Bash-command controls.

### Source excerpt

I got you boo

## Lakesail: A Rust-based alternative to Spark without the JVM

DevFeed: [Lakesail: A Rust-based alternative to Spark without the JVM](<https://devfeed.tech/articles/lakesail-replacing-spark-with-rust-but-with-spark-38711.md>)

Original publisher: [Read original article](<https://dataengineeringcentral.substack.com/p/lakesail-replacing-spark-with-rust>)

Author: Daniel Beach

Published: 2026-08-18T21:11:44Z

Content type: opinion

Language: en

Sources: [Data Engineering Central](<https://devfeed.tech/sources/data-engineering-central.md>)

Topics: [Apache Spark](<https://devfeed.tech/topics/spark.md>), [Rust](<https://devfeed.tech/topics/rust.md>)

Tags: [apache-spark](<https://devfeed.tech/tags/apache-spark.md>), [rust](<https://devfeed.tech/tags/rust.md>), [spark](<https://devfeed.tech/tags/spark.md>)

### AI overview

This commentary article introduces Lakesail as a Rust-based implementation or alternative to Spark that avoids the JVM overhead. It places Lakesail within the broader data-engineering ecosystem, where established tools such as Spark and Kafka continue to coexist with newer projects including Polars, DuckDB, and Daft.

### Source excerpt

ya know ... black magic stuff

## Quasi-Agentic Pipelines with Databricks and Apache Airflow

DevFeed: [Quasi-Agentic Pipelines with Databricks and Apache Airflow](<https://devfeed.tech/articles/quasi-agentic-pipelines-with-databricks-and-apache-airflow-38713.md>)

Original publisher: [Read original article](<https://dataengineeringcentral.substack.com/p/quasi-agentic-pipelines-with-databricks>)

Author: Daniel Beach

Published: 2026-08-10T21:23:57Z

Content type: tutorial

Language: en

Sources: [Data Engineering Central](<https://devfeed.tech/sources/data-engineering-central.md>)

Topics: [databricks](<https://devfeed.tech/topics/databricks.md>), [airflow](<https://devfeed.tech/topics/airflow.md>), [Data pipelines](<https://devfeed.tech/topics/data-pipelines.md>), [Large Language Model](<https://devfeed.tech/topics/llm.md>), [data-engineering](<https://devfeed.tech/topics/data-engineering.md>)

Tags: [airflow](<https://devfeed.tech/tags/airflow.md>), [apache-airflow](<https://devfeed.tech/tags/apache-airflow.md>), [data-engineering](<https://devfeed.tech/tags/data-engineering.md>), [databricks](<https://devfeed.tech/tags/databricks.md>), [llms](<https://devfeed.tech/tags/llms.md>), [pipelines](<https://devfeed.tech/tags/pipelines.md>)

### AI overview

A practical developer discussion of incorporating LLMs and agents into existing data workflows using Databricks and Apache Airflow. It also examines determinism in data pipelines and the gap between business requirements and engineering implementation.

### Source excerpt

the strange space in between

## Keeping Technical Skills in the Age of the LLM

DevFeed: [Keeping Technical Skills in the Age of the LLM](<https://devfeed.tech/articles/keeping-technical-skills-in-the-age-of-the-llm-38710.md>)

Original publisher: [Read original article](<https://dataengineeringcentral.substack.com/p/keeping-technical-skills-in-the-age>)

Author: Daniel Beach

Published: 2026-08-03T12:03:12Z

Content type: opinion

Language: en

Sources: [Data Engineering Central](<https://devfeed.tech/sources/data-engineering-central.md>)

Topics: [Large Language Model](<https://devfeed.tech/topics/llm.md>), [Programming](<https://devfeed.tech/topics/programming.md>), [Claude](<https://devfeed.tech/topics/claude.md>)

Tags: [claude](<https://devfeed.tech/tags/claude.md>), [llm](<https://devfeed.tech/tags/llm.md>), [model](<https://devfeed.tech/tags/model.md>), [technical](<https://devfeed.tech/tags/technical.md>)

### AI overview

The article reflects on the challenge of maintaining technical skills as large language models become part of software development. The author describes personal uncertainty about when to use Claude versus writing code independently, placing that dilemma in the context of a lifelong interest in mechanical and digital tinkering.

### Source excerpt

the end of the era and new one beginning

## Data Modeling's Relevance in Modern Data Engineering

DevFeed: [Data Modeling's Relevance in Modern Data Engineering](<https://devfeed.tech/articles/so-is-data-modeling-dead-38714.md>)

Original publisher: [Read original article](<https://dataengineeringcentral.substack.com/p/so-is-data-modeling-dead>)

Author: Daniel Beach

Published: 2026-07-27T12:20:21Z

Content type: opinion

Language: en

Sources: [Data Engineering Central](<https://devfeed.tech/sources/data-engineering-central.md>)

Topics: [data-modeling](<https://devfeed.tech/topics/data-modeling.md>), [data](<https://devfeed.tech/topics/data.md>), [Artificial Intelligence](<https://devfeed.tech/topics/ai.md>)

Tags: [ai](<https://devfeed.tech/tags/ai.md>), [article](<https://devfeed.tech/tags/article.md>), [data-modeling](<https://devfeed.tech/tags/data-modeling.md>), [opinions](<https://devfeed.tech/tags/opinions.md>)

### AI overview

This opinion article examines whether data modeling remains relevant in modern data work. It presents differing views, notes the longstanding role of relational modeling, and questions why the topic has recently been treated as potentially obsolete.

### Source excerpt

was it never alive?

## Why I Miss Writing Rust Code by Hand

DevFeed: [Why I Miss Writing Rust Code by Hand](<https://devfeed.tech/articles/warning-this-rust-code-was-written-by-hand-38720.md>)

Original publisher: [Read original article](<https://dataengineeringcentral.substack.com/p/warning-this-rust-code-was-written>)

Author: Daniel Beach

Published: 2026-07-23T12:47:19Z

Content type: opinion

Language: en

Sources: [Data Engineering Central](<https://devfeed.tech/sources/data-engineering-central.md>)

Topics: [Rust](<https://devfeed.tech/topics/rust.md>), [Code](<https://devfeed.tech/topics/code.md>), [systems](<https://devfeed.tech/topics/systems.md>)

Tags: [code](<https://devfeed.tech/tags/code.md>), [rust](<https://devfeed.tech/tags/rust.md>)

### AI overview

A personal commentary about missing the challenge of writing Rust code by hand. The author argues that exercising the mind through detailed software work supports effective systems design and software problem-solving, while also mentioning broader responsibilities such as architecture, business requirements, and project planning.

### Source excerpt

back to the old days

## Agentic Data Engineering Is Here -- But Can It Close the Loop?

DevFeed: [Agentic Data Engineering Is Here -- But Can It Close the Loop?](<https://devfeed.tech/articles/agentic-data-engineering-is-here-but-can-it-close-the-loop-38703.md>)

Original publisher: [Read original article](<https://dataengineeringcentral.substack.com/p/agentic-data-engineering-is-here>)

Author: Daniel Beach

Published: 2026-07-22T14:05:27Z

Content type: article

Language: en

Sources: [Data Engineering Central](<https://devfeed.tech/sources/data-engineering-central.md>)

Topics: [data-engineering](<https://devfeed.tech/topics/data-engineering.md>), [Orchestration](<https://devfeed.tech/topics/orchestration.md>), [data observability](<https://devfeed.tech/topics/data-observability.md>), [Artificial Intelligence](<https://devfeed.tech/topics/ai.md>), [AI-assisted coding](<https://devfeed.tech/topics/ai-assisted-coding.md>), [DuckDB](<https://devfeed.tech/topics/duckdb.md>), [Apache Iceberg](<https://devfeed.tech/topics/apache-iceberg.md>)

Tags: [agentic](<https://devfeed.tech/tags/agentic.md>), [ai-agents](<https://devfeed.tech/tags/ai-agents.md>), [data-engineering](<https://devfeed.tech/tags/data-engineering.md>), [duckdb](<https://devfeed.tech/tags/duckdb.md>), [orchestration](<https://devfeed.tech/tags/orchestration.md>), [pipelines](<https://devfeed.tech/tags/pipelines.md>)

### AI overview

A podcast conversation with Hugo Lu about agentic data engineering and the infrastructure needed for data platforms to execute work, observe outcomes, validate changes, and improve pipelines safely. It examines why production data systems remain difficult for AI, including schema changes, realistic testing, business semantics, and secure execution.

### Source excerpt

a conversation with Hugo Lu

## ADBC: Arrow's Cross-Platform Database Connectivity API

DevFeed: [ADBC: Arrow's Cross-Platform Database Connectivity API](<https://devfeed.tech/articles/adbc-38702.md>)

Original publisher: [Read original article](<https://dataengineeringcentral.substack.com/p/adbc>)

Author: Daniel Beach

Published: 2026-07-20T13:57:32Z

Content type: article

Language: en

Sources: [Data Engineering Central](<https://devfeed.tech/sources/data-engineering-central.md>)

Topics: [API](<https://devfeed.tech/topics/api.md>), [Database](<https://devfeed.tech/topics/database.md>), [data](<https://devfeed.tech/topics/data.md>), [client](<https://devfeed.tech/topics/client.md>)

Tags: [apis](<https://devfeed.tech/tags/apis.md>), [arrow](<https://devfeed.tech/tags/arrow.md>), [client](<https://devfeed.tech/tags/client.md>), [data](<https://devfeed.tech/tags/data.md>), [database](<https://devfeed.tech/tags/database.md>), [rust](<https://devfeed.tech/tags/rust.md>)

### AI overview

This article explains ADBC, Arrow's database connectivity API. It describes ADBC implementations across several languages and databases, and compares its columnar, minimal-overhead approach with JDBC and ODBC for working with Arrow data.

### Source excerpt

Guest Post by Anonymous Rust Dev

[Next page](<https://devfeed.tech/sources/data-engineering-central.md?cursor=WyIyMDI2LTA3LTIwVDEzOjU3OjMyKzAwOjAwIiwgImI3ZTczM2FhLWE4NDktNDYzYy1hZDRjLTdmODMwNmVjNDZmMSJd>)