# DataExpert.io Newsletter

A newsletter dedicated to talking about data engineering, AI, and data science trends

This is one page of public article previews, not the complete archive. Follow Next page to continue. Summaries are not the original full articles.

## The State of the 2026 Data Engineering Job Market

DevFeed: [The State of the 2026 Data Engineering Job Market](<https://devfeed.tech/articles/the-state-of-the-2026-data-engineering-job-market-106847.md>)

Original publisher: [Read original article](<https://blog.dataexpert.io/p/the-state-of-the-2026-data-engineering>)

Author: Zach Wilson

Published: 2026-10-29T20:02:21Z

Content type: article

Language: en

Sources: [DataExpert.io Newsletter](<https://devfeed.tech/sources/dataexpert-io-newsletter.md>)

Topics: [data-engineering](<https://devfeed.tech/topics/data-engineering.md>), [databricks](<https://devfeed.tech/topics/databricks.md>), [Machine learning](<https://devfeed.tech/topics/machine-learning.md>), [Learning](<https://devfeed.tech/topics/learning.md>)

Tags: [2](<https://devfeed.tech/tags/2.md>), [2015](<https://devfeed.tech/tags/2015.md>), [2021](<https://devfeed.tech/tags/2021.md>), [2022](<https://devfeed.tech/tags/2022.md>), [2026](<https://devfeed.tech/tags/2026.md>), [22](<https://devfeed.tech/tags/22.md>), [3](<https://devfeed.tech/tags/3.md>), [about](<https://devfeed.tech/tags/about.md>), [account](<https://devfeed.tech/tags/account.md>), [ai](<https://devfeed.tech/tags/ai.md>), [ai-data](<https://devfeed.tech/tags/ai-data.md>), [data](<https://devfeed.tech/tags/data.md>), [data-engineering](<https://devfeed.tech/tags/data-engineering.md>), [databricks](<https://devfeed.tech/tags/databricks.md>), [engineering](<https://devfeed.tech/tags/engineering.md>), [how-to](<https://devfeed.tech/tags/how-to.md>), [job](<https://devfeed.tech/tags/job.md>), [job-market](<https://devfeed.tech/tags/job-market.md>), [project](<https://devfeed.tech/tags/project.md>), [with](<https://devfeed.tech/tags/with.md>)

### AI overview

The excerpt describes a difficult 2026 market, especially for junior candidates, alongside reported data engineering job growth. It cautions that broad Data & Analytics declines do not show data engineering's specific trajectory, since analysts make up most of that category; a separate report cited in the article shows continuous growth from August to October, with only 5.5% of openings for juniors. To stand out, it recommends a deployed project combining a data pipeline with an AI agent that takes actions, illustrated by a Databricks Free Edition stock trading app using Spark, Lakebase, Agent Bricks, and an Alpaca Markets account. For job applications, it advises testing channels beyond LinkedIn Easy Apply, seeking warm introductions, and tailoring resumes to job descriptions with AI. These are the article's recommendations and reported figures; the excerpt does not establish their outcomes.

### Source excerpt

And how to get hired with an AI Data Engineering project on Databricks

## Big Tech Speedrunning Senior to Staff Data Engineer

DevFeed: [Big Tech Speedrunning Senior to Staff Data Engineer](<https://devfeed.tech/articles/big-tech-speedrunning-senior-to-staff-data-engineer-61305.md>)

Original publisher: [Read original article](<https://blog.dataexpert.io/p/big-tech-speedrunning-senior-to-staff>)

Author: Zach Wilson

Published: 2026-09-28T21:29:35Z

Content type: article

Language: en

Sources: [DataExpert.io Newsletter](<https://devfeed.tech/sources/dataexpert-io-newsletter.md>)

Topics: [Data Management](<https://devfeed.tech/topics/data-management.md>)

Tags: [big-tech](<https://devfeed.tech/tags/big-tech.md>), [career](<https://devfeed.tech/tags/career.md>), [data](<https://devfeed.tech/tags/data.md>), [engineering](<https://devfeed.tech/tags/engineering.md>), [engineering-manager](<https://devfeed.tech/tags/engineering-manager.md>)

### AI overview

The author describes moving from senior data engineer at Netflix to staff data engineer at Airbnb in two years, attributing the accelerated promotion to hard work, relationship building, high leverage projects, bravery, and luck. The advice includes seeking work with product and revenue impact, developing system design and product sense, maintaining professional relationships, and changing roles or companies when a manager blocks growth.

### Source excerpt

Staff engineer is the beginning of the "second" ladder in big tech.

## The Complete Databricks Learning Roadmap for 2026

DevFeed: [The Complete Databricks Learning Roadmap for 2026](<https://devfeed.tech/articles/the-complete-databricks-learning-roadmap-for-2026-27257.md>)

Original publisher: [Read original article](<https://blog.dataexpert.io/p/the-2026-mastering-databricks-roadmap>)

Author: Zach Wilson

Published: 2026-08-14T20:11:45Z

Content type: tutorial

Language: en

Sources: [DataExpert.io Newsletter](<https://devfeed.tech/sources/dataexpert-io-newsletter.md>)

Topics: [databricks](<https://devfeed.tech/topics/databricks.md>), [Learning](<https://devfeed.tech/topics/learning.md>)

Tags: [2026](<https://devfeed.tech/tags/2026.md>), [databricks](<https://devfeed.tech/tags/databricks.md>), [learn](<https://devfeed.tech/tags/learn.md>), [learning](<https://devfeed.tech/tags/learning.md>), [master](<https://devfeed.tech/tags/master.md>), [skip](<https://devfeed.tech/tags/skip.md>)

### AI overview

A 2026 learning roadmap for Databricks covering what to learn, what to skip, and the recommended order for mastering the platform.

### Source excerpt

What to learn, what to skip, and the right order to master Databricks.

## How do One Big Table and AI fit together

DevFeed: [How do One Big Table and AI fit together](<https://devfeed.tech/articles/how-do-one-big-table-and-ai-fit-together-27248.md>)

Original publisher: [Read original article](<https://blog.dataexpert.io/p/how-to-data-model-for-your-ai-context>)

Author: Zach Wilson

Published: 2026-04-06T20:54:07Z

Content type: tutorial

Language: en

Sources: [DataExpert.io Newsletter](<https://devfeed.tech/sources/dataexpert-io-newsletter.md>)

Topics: [Artificial Intelligence](<https://devfeed.tech/topics/ai.md>)

Tags: [ai](<https://devfeed.tech/tags/ai.md>)

### AI overview

The article discusses how One Big Table relates to AI and states that dimensional modeling is dead.

### Source excerpt

Dimensional Modeling is dead

## Databricks is abstracting away physical data engineering controls

DevFeed: [Databricks is abstracting away physical data engineering controls](<https://devfeed.tech/articles/databricks-is-no-longer-about-tuning-knobs-27242.md>)

Original publisher: [Read original article](<https://blog.dataexpert.io/p/databricks-is-for-data-analysts-not>)

Author: Zach Wilson

Published: 2026-02-24T01:03:11Z

Content type: opinion

Language: en

Sources: [DataExpert.io Newsletter](<https://devfeed.tech/sources/dataexpert-io-newsletter.md>)

Topics: [databricks](<https://devfeed.tech/topics/databricks.md>), [data-engineering](<https://devfeed.tech/topics/data-engineering.md>), [Apache Iceberg](<https://devfeed.tech/topics/apache-iceberg.md>), [Open Source](<https://devfeed.tech/topics/open-source.md>)

Tags: [architecture](<https://devfeed.tech/tags/architecture.md>), [data-engineering](<https://devfeed.tech/tags/data-engineering.md>), [databricks](<https://devfeed.tech/tags/databricks.md>), [open-source](<https://devfeed.tech/tags/open-source.md>), [partition](<https://devfeed.tech/tags/partition.md>), [partitioning](<https://devfeed.tech/tags/partitioning.md>), [sorting](<https://devfeed.tech/tags/sorting.md>), [spark](<https://devfeed.tech/tags/spark.md>)

### AI overview

This opinion article argues that Databricks is shifting away from hands-on data engineering by abstracting physical data modeling through features such as liquid clustering and predictive optimization. It also criticizes Databricks' support for managed Apache Iceberg tables after acquiring Tabular.

### Source excerpt

Databricks abstracts away almost all of the data engineering skills. Liquid clustering is the first place where things will get messy!

## The 2026 AI Data Engineer Roadmap

DevFeed: [The 2026 AI Data Engineer Roadmap](<https://devfeed.tech/articles/the-2026-ai-data-engineer-roadmap-27256.md>)

Original publisher: [Read original article](<https://blog.dataexpert.io/p/the-2026-ai-data-engineer-roadmap>)

Author: Zach Wilson

Published: 2026-02-05T20:26:37Z

Content type: article

Language: en

Sources: [DataExpert.io Newsletter](<https://devfeed.tech/sources/dataexpert-io-newsletter.md>)

Topics: [Artificial Intelligence](<https://devfeed.tech/topics/ai.md>), [data-engineering](<https://devfeed.tech/topics/data-engineering.md>), [AI-assisted coding](<https://devfeed.tech/topics/ai-assisted-coding.md>), [Apache Spark](<https://devfeed.tech/topics/spark.md>), [airflow](<https://devfeed.tech/topics/airflow.md>), [Apache Flink](<https://devfeed.tech/topics/apache-flink.md>)

Tags: [ai](<https://devfeed.tech/tags/ai.md>), [ai-coding-agents](<https://devfeed.tech/tags/ai-coding-agents.md>), [best-practices](<https://devfeed.tech/tags/best-practices.md>), [data](<https://devfeed.tech/tags/data.md>), [data-engineering](<https://devfeed.tech/tags/data-engineering.md>)

### AI overview

This article presents a 2026 roadmap for data engineering as AI automates more pipeline, SQL, Spark, testing, migration, and operational work. It compares responsibilities that are becoming more automated with areas such as system design, deep technical debt, performance tradeoffs, and organizational constraints where human expertise remains important.

### Source excerpt

And how to avoid getting replaced

## Processing 1 TB with DuckDB in less than 30 seconds

DevFeed: [Processing 1 TB with DuckDB in less than 30 seconds](<https://devfeed.tech/articles/processing-1-tb-with-duckdb-in-less-than-30-seconds-27250.md>)

Original publisher: [Read original article](<https://blog.dataexpert.io/p/i-processed-1-tb-with-duckdb-in-30>)

Author: Matt Martin

Published: 2025-12-23T19:58:37Z

Content type: article

Language: en

Sources: [DataExpert.io Newsletter](<https://devfeed.tech/sources/dataexpert-io-newsletter.md>)

Topics: [DuckDB](<https://devfeed.tech/topics/duckdb.md>), [Benchmark](<https://devfeed.tech/topics/benchmark.md>), [data](<https://devfeed.tech/topics/data.md>), [parquet](<https://devfeed.tech/topics/parquet.md>), [Python](<https://devfeed.tech/topics/python.md>), [Process](<https://devfeed.tech/topics/process.md>)

Tags: [benchmark](<https://devfeed.tech/tags/benchmark.md>), [data](<https://devfeed.tech/tags/data.md>), [duckdb](<https://devfeed.tech/tags/duckdb.md>), [parquet](<https://devfeed.tech/tags/parquet.md>), [process](<https://devfeed.tech/tags/process.md>), [python](<https://devfeed.tech/tags/python.md>)

### AI overview

This article benchmarks DuckDB on progressively larger datasets, reporting that it read approximately 200 GB in under 10 seconds and 500 GB in about 40 seconds. It also describes generating a 1 TB dataset in roughly 70 minutes on an M2 Pro Mac with 16 GB of RAM using 10 parallel workers.

### Source excerpt

And so can you

## A Datestamp-Based Alternative to Slowly Changing Dimensions

DevFeed: [A Datestamp-Based Alternative to Slowly Changing Dimensions](<https://devfeed.tech/articles/scd-2-considered-harmful-part-2-27254.md>)

Original publisher: [Read original article](<https://blog.dataexpert.io/p/stop-using-slowly-changing-dimensions>)

Author: Sahar Massachi

Published: 2025-11-04T20:44:28Z

Content type: tutorial

Language: en

Sources: [DataExpert.io Newsletter](<https://devfeed.tech/sources/dataexpert-io-newsletter.md>)

Topics: [data](<https://devfeed.tech/topics/data.md>), [Structured-data](<https://devfeed.tech/topics/structured-data.md>), [dataset](<https://devfeed.tech/topics/dataset.md>)

Tags: [2025](<https://devfeed.tech/tags/2025.md>), [data](<https://devfeed.tech/tags/data.md>), [data-recovery](<https://devfeed.tech/tags/data-recovery.md>), [pipeline](<https://devfeed.tech/tags/pipeline.md>), [sql](<https://devfeed.tech/tags/sql.md>), [time](<https://devfeed.tech/tags/time.md>)

### AI overview

This second installment in a data warehousing series explains the difficulty of querying historical state across multiple tables with slowly changing dimensions. It proposes datestamps and idempotent pipelines as a simpler approach for historical queries, backfills, data recovery, and operational alerts.

### Source excerpt

Date stamp your data!

## A Date-Stamped Data Warehouse Setup

DevFeed: [A Date-Stamped Data Warehouse Setup](<https://devfeed.tech/articles/the-data-warehouse-setup-no-one-taught-you-27258.md>)

Original publisher: [Read original article](<https://blog.dataexpert.io/p/the-data-warehouse-setup-no-one-taught>)

Author: Sahar Massachi

Published: 2025-10-24T21:03:30Z

Content type: article

Language: en

Sources: [DataExpert.io Newsletter](<https://devfeed.tech/sources/dataexpert-io-newsletter.md>)

Topics: [data-engineering](<https://devfeed.tech/topics/data-engineering.md>), [data](<https://devfeed.tech/topics/data.md>), [Apache Iceberg](<https://devfeed.tech/topics/apache-iceberg.md>)

Tags: [ab-testing](<https://devfeed.tech/tags/ab-testing.md>), [data](<https://devfeed.tech/tags/data.md>), [data-engineering](<https://devfeed.tech/tags/data-engineering.md>), [storage](<https://devfeed.tech/tags/storage.md>), [warehouse](<https://devfeed.tech/tags/warehouse.md>)

### AI overview

The article presents date-stamping data as a simple organizing principle for building resilient data warehouses and pipelines. It explains how this approach can manage changes over time, support experimentation and metrics, and work with systems including Hive metastore, Iceberg, Delta, and Hudi.

### Source excerpt

Storage is cheap, your time is not!

## The 2025 AI + Data Engineering Roadmap

DevFeed: [The 2025 AI + Data Engineering Roadmap](<https://devfeed.tech/articles/the-2025-ai-data-engineering-roadmap-27255.md>)

Original publisher: [Read original article](<https://blog.dataexpert.io/p/the-2025-breaking-into-data-engineering-roadmap>)

Author: Zach Wilson

Published: 2025-10-17T22:35:45Z

Content type: tutorial

Language: en

Sources: [DataExpert.io Newsletter](<https://devfeed.tech/sources/dataexpert-io-newsletter.md>)

Topics: [data-engineering](<https://devfeed.tech/topics/data-engineering.md>), [Python](<https://devfeed.tech/topics/python.md>), [SQL](<https://devfeed.tech/topics/sql.md>), [Artificial Intelligence](<https://devfeed.tech/topics/ai.md>), [airflow](<https://devfeed.tech/topics/airflow.md>), [BigQuery](<https://devfeed.tech/topics/bigquery.md>), [Orchestration](<https://devfeed.tech/topics/orchestration.md>), [Apache Spark](<https://devfeed.tech/topics/spark.md>), [data-modeling](<https://devfeed.tech/topics/data-modeling.md>), [Data Quality](<https://devfeed.tech/topics/data-quality.md>), [Embeddings](<https://devfeed.tech/topics/embeddings.md>), [Retrieval Augmented Generation (RAG)](<https://devfeed.tech/topics/retrieval-augmented-generation-rag.md>)

Tags: [2025](<https://devfeed.tech/tags/2025.md>), [ai](<https://devfeed.tech/tags/ai.md>), [airflow](<https://devfeed.tech/tags/airflow.md>), [bigquery](<https://devfeed.tech/tags/bigquery.md>), [count](<https://devfeed.tech/tags/count.md>), [course](<https://devfeed.tech/tags/course.md>), [data-engineering](<https://devfeed.tech/tags/data-engineering.md>), [data-modeling](<https://devfeed.tech/tags/data-modeling.md>), [data-quality](<https://devfeed.tech/tags/data-quality.md>), [embeddings](<https://devfeed.tech/tags/embeddings.md>), [engineering](<https://devfeed.tech/tags/engineering.md>), [framer](<https://devfeed.tech/tags/framer.md>), [newsletter](<https://devfeed.tech/tags/newsletter.md>), [orchestration](<https://devfeed.tech/tags/orchestration.md>), [python](<https://devfeed.tech/tags/python.md>), [rag](<https://devfeed.tech/tags/rag.md>), [real-time](<https://devfeed.tech/tags/real-time.md>), [right-join](<https://devfeed.tech/tags/right-join.md>), [spark](<https://devfeed.tech/tags/spark.md>), [sql](<https://devfeed.tech/tags/sql.md>)

### AI overview

A 2025 roadmap for entering data engineering, covering foundational SQL and Python skills, distributed computing, orchestration, data modeling, data quality, AI and data integrations, portfolio projects, and personal branding.

### Source excerpt

Getting a data engineering job is complicated.

## Migrating 13,000 Iceberg Tables in 4 hours to Glue Catalog

DevFeed: [Migrating 13,000 Iceberg Tables in 4 hours to Glue Catalog](<https://devfeed.tech/articles/migrating-13-000-iceberg-tables-in-4-hours-to-glue-catalog-27247.md>)

Original publisher: [Read original article](<https://blog.dataexpert.io/p/how-i-migrated-13000-iceberg-tables>)

Author: Zach Wilson

Published: 2025-09-17T22:12:17Z

Content type: article

Language: en

Sources: [DataExpert.io Newsletter](<https://devfeed.tech/sources/dataexpert-io-newsletter.md>)

Topics: [Apache Iceberg](<https://devfeed.tech/topics/apache-iceberg.md>)

### AI overview

The article describes migrating 13,000 Apache Iceberg tables to the Glue Catalog in four hours. The supplied text also reports a message saying that Tabular would be sunsetted within 24 hours.

### Source excerpt

At midnight on September 16th, Jason Reid (data engineering advocate at Databricks) messages me on LinkedIn saying, "Tabular will be sunsetted in 24 hours.