# Dagster

Read the latest from the Dagster team: insights, tutorials, and updates on data engineering, orchestration, and building better pipelines.

This is one page of public article previews, not the complete archive. Follow Next page to continue. Summaries are not the original full articles.

## The Data Plane Is Yours: Dagster+ Hybrid on Kubernetes and Azure

DevFeed: [The Data Plane Is Yours: Dagster+ Hybrid on Kubernetes and Azure](<https://devfeed.tech/articles/the-data-plane-is-yours-dagster-hybrid-on-kubernetes-and-azure-83746.md>)

Original publisher: [Read original article](<https://dagster.io/blog/the-data-plane-is-yours-dagster-hybrid-on-kubernetes-and-azure>)

Author: Mieszko Półgrabsk

Published: 2026-09-08T00:00:00Z

Content type: article

Language: en

Sources: [Dagster](<https://devfeed.tech/sources/dagster.md>)

Topics: [Distributed Systems & Parallel Computing](<https://devfeed.tech/topics/distributed-systems-parallel-computing.md>), [Azure Kubernetes Service](<https://devfeed.tech/topics/azure-kubernetes-service.md>), [internal developer platform automation](<https://devfeed.tech/topics/internal-developer-platform-automation.md>)

Tags: [azure](<https://devfeed.tech/tags/azure.md>), [capacity](<https://devfeed.tech/tags/capacity.md>), [deployment](<https://devfeed.tech/tags/deployment.md>), [engineering](<https://devfeed.tech/tags/engineering.md>), [enterprise-it](<https://devfeed.tech/tags/enterprise-it.md>), [helm](<https://devfeed.tech/tags/helm.md>), [isolation](<https://devfeed.tech/tags/isolation.md>), [kubernetes](<https://devfeed.tech/tags/kubernetes.md>), [networking](<https://devfeed.tech/tags/networking.md>), [ownership](<https://devfeed.tech/tags/ownership.md>)

### AI overview

The article explains how a platform team operates Dagster+ Hybrid on Kubernetes and Azure, with Dagster Labs running the control plane and the team owning the data plane. It describes workload ownership, isolation, identity and networking, capacity planning, monitoring, upgrades, and the operational boundaries between product and platform teams. Its central recommendation is to connect Dagster runs, Kubernetes jobs and pods, and Azure capacity signals into a traceable support path, while keeping product-team configuration constrained through templates and pull request guardrails.

### Source excerpt

This article draws on a Dagster+ Hybrid deployment on Kubernetes and Azure at a very large enterprise. It covers the work outside the Helm chart: workload ownership, isolation, networking, identity, sizing, and capacity.

## Orchestration is More than Scheduling: Declarative Automation in Dagster

DevFeed: [Orchestration is More than Scheduling: Declarative Automation in Dagster](<https://devfeed.tech/articles/orchestration-is-more-than-scheduling-declarative-automation-in-dagster-83711.md>)

Original publisher: [Read original article](<https://dagster.io/blog/orchestration-is-more-than-scheduling-declarative-automation-in-dagster>)

Author: Catherine Noll

Published: 2026-08-06T00:00:00Z

Content type: tutorial

Language: en

Sources: [Dagster](<https://devfeed.tech/sources/dagster.md>)

Topics: [Orchestration](<https://devfeed.tech/topics/orchestration.md>), [Declarative programming](<https://devfeed.tech/topics/declarative-programming.md>), [scheduling](<https://devfeed.tech/topics/scheduling.md>), [Python](<https://devfeed.tech/topics/python.md>)

Tags: [automation](<https://devfeed.tech/tags/automation.md>), [dagster](<https://devfeed.tech/tags/dagster.md>), [data-pipelines](<https://devfeed.tech/tags/data-pipelines.md>), [declarative](<https://devfeed.tech/tags/declarative.md>), [product](<https://devfeed.tech/tags/product.md>)

### AI overview

Dagster's Declarative Automation lets engineers define when data assets should be considered up to date. Dagster evaluates these conditions against asset dependencies, partitions, and execution state to request work. The tutorial demonstrates built-in policies, composed conditions, job-level automation in preview, and isolated tests of automation behavior.

### Source excerpt

Define the outcome, not the orchestration. Declarative Automation lets you express your desired asset state while Dagster continuously handles the work needed to achieve it.

## Classifying a Million Snowflake Columns in 9 Days, Solo, with Dagster

DevFeed: [Classifying a Million Snowflake Columns in 9 Days, Solo, with Dagster](<https://devfeed.tech/articles/classifying-a-million-snowflake-columns-in-9-days-solo-with-dagster-83613.md>)

Original publisher: [Read original article](<https://dagster.io/blog/classifying-a-million-snowflake-columns-in-9-days-solo-with-dagster>)

Author: Mayank Sethi

Published: 2026-07-16T00:00:00Z

Content type: article

Language: en

Sources: [Dagster](<https://devfeed.tech/sources/dagster.md>)

Topics: [data-engineering](<https://devfeed.tech/topics/data-engineering.md>), [human review](<https://devfeed.tech/topics/human-review.md>), [Publish-subscribe pattern](<https://devfeed.tech/topics/pubsub.md>), [self-service](<https://devfeed.tech/topics/self-service.md>), [data type](<https://devfeed.tech/topics/data-type.md>), [schema-evolution](<https://devfeed.tech/topics/schema-evolution.md>)

Tags: [dagster](<https://devfeed.tech/tags/dagster.md>), [data-governance](<https://devfeed.tech/tags/data-governance.md>), [engineering](<https://devfeed.tech/tags/engineering.md>), [human-in-the-loop](<https://devfeed.tech/tags/human-in-the-loop.md>), [legacy](<https://devfeed.tech/tags/legacy.md>), [legacy-systems](<https://devfeed.tech/tags/legacy-systems.md>), [orchestration](<https://devfeed.tech/tags/orchestration.md>), [snowflake](<https://devfeed.tech/tags/snowflake.md>)

### AI overview

An engineer describes building a Dagster-orchestrated system to classify and tag more than a million columns across 62 Snowflake databases during a legacy-system migration. The design combines ordered rules, Snowflake classification tools, and Cortex for ambiguous cases, with a human override workflow and shared tagging path. Dagster sensors, run keys, and asset metadata support triggering, duplicate prevention, and run visibility.

### Source excerpt

We're consolidating a decade of legacy systems into Snowflake, which leaves us with over a million columns that all need to be classified and tagged for data governance. I built the system that does it: a tiered classification engine, a human-in-the-loop override workflow, and the orchestration that ties them together. This is the engineering story, the challenges that made it interesting, and where Dagster fits as the orchestration layer. Built solo, to production, in nine days.

## Prefect is Acquiring Dagster

DevFeed: [Prefect is Acquiring Dagster](<https://devfeed.tech/articles/prefect-is-acquiring-dagster-83718.md>)

Original publisher: [Read original article](<https://dagster.io/blog/prefect-is-acquiring-dagster>)

Author: Nick Schrock

Published: 2026-07-13T00:00:00Z

Content type: news

Language: en

Sources: [Dagster](<https://devfeed.tech/sources/dagster.md>)

Topics: [Data pipelines](<https://devfeed.tech/topics/data-pipelines.md>), [argilla](<https://devfeed.tech/topics/argilla.md>), [Calculate Pi](<https://devfeed.tech/topics/calculate-pi.md>)

Tags: [dagster](<https://devfeed.tech/tags/dagster.md>), [open-source](<https://devfeed.tech/tags/open-source.md>), [orchestration](<https://devfeed.tech/tags/orchestration.md>), [product](<https://devfeed.tech/tags/product.md>)

### AI overview

Dagster founder Nick Schrock announces that Prefect is acquiring Dagster Labs. The letter says the companies' workflow orchestration products serve distinct, complementary needs, and that Dagster's open-source project will continue active development, Dagster+ will retain its brand, and about 40 team members will join Prefect. Schrock also says he is leaving the project and company.

### Source excerpt

A letter from Nick Schrock, founder and creator of Dagster

## Community-built projects using Dagster for data engineering

DevFeed: [Community-built projects using Dagster for data engineering](<https://devfeed.tech/articles/community-showcase-part-2-83617.md>)

Original publisher: [Read original article](<https://dagster.io/blog/community-showcase-part-2>)

Author: Dennis Hume

Published: 2026-06-30T00:00:00Z

Content type: article

Language: en

Sources: [Dagster](<https://devfeed.tech/sources/dagster.md>)

Topics: [Tool](<https://devfeed.tech/topics/tool.md>), [hugging face](<https://devfeed.tech/topics/hugging-face.md>), [Cloud Architecture](<https://devfeed.tech/topics/cloud-architecture.md>)

Tags: [dagster](<https://devfeed.tech/tags/dagster.md>), [data-orchestration](<https://devfeed.tech/tags/data-orchestration.md>), [data-pipelines](<https://devfeed.tech/tags/data-pipelines.md>), [data-science](<https://devfeed.tech/tags/data-science.md>), [product](<https://devfeed.tech/tags/product.md>), [projects](<https://devfeed.tech/tags/projects.md>), [showcase](<https://devfeed.tech/tags/showcase.md>)

### AI overview

This showcase features two community projects built with Dagster. One uses Dagster, dbt, DuckDB, and related tools to process Danish parliamentary data, while another integrates Dagster with Hugging Face Datasets to support dataset workflows, including streaming and metadata handling.

### Source excerpt

Some of the most interesting Dagster projects come from the community. This post highlights creative community-built applications.

## Operationalizing Data Orchestration: Best Practices for DevOps, Infra, and Code Locations

DevFeed: [Operationalizing Data Orchestration: Best Practices for DevOps, Infra, and Code Locations](<https://devfeed.tech/articles/operationalizing-data-orchestration-best-practices-for-devops-infra-and-code-locations-83744.md>)

Original publisher: [Read original article](<https://dagster.io/blog/the-dagster-almanack-operationalizing-data-orchestration>)

Author: Simon Späti

Published: 2026-06-24T00:00:00Z

Content type: tutorial

Language: en

Sources: [Dagster](<https://devfeed.tech/sources/dagster.md>)

Topics: [DevOps](<https://devfeed.tech/topics/devops.md>), [CI/CD](<https://devfeed.tech/topics/cicd.md>), [DataOps](<https://devfeed.tech/topics/dataops.md>), [finops](<https://devfeed.tech/topics/finops.md>), [schema design](<https://devfeed.tech/topics/schema-design.md>)

Tags: [architecture](<https://devfeed.tech/tags/architecture.md>), [automate](<https://devfeed.tech/tags/automate.md>), [automation](<https://devfeed.tech/tags/automation.md>), [branch](<https://devfeed.tech/tags/branch.md>), [devops](<https://devfeed.tech/tags/devops.md>), [engineering](<https://devfeed.tech/tags/engineering.md>), [has](<https://devfeed.tech/tags/has.md>), [infra](<https://devfeed.tech/tags/infra.md>)

### AI overview

The article explains how to operationalize data orchestration with Dagster, focusing on GitOps and DevOps practices, deployment structure, code locations, and separation of pipeline logic from infrastructure. It covers deployment choices such as Kubernetes, Docker, and Dagster+, along with scaling, data quality checks, governance, and multi-team workflows.

### Source excerpt

A complete guide with all the insights, tips, and some predictions for the data platform engineer, just like an Almanack provides, with practical information for daily life.

## How to Make the Architectural Case for Dagster

DevFeed: [How to Make the Architectural Case for Dagster](<https://devfeed.tech/articles/how-to-make-the-architectural-case-for-dagster-83692.md>)

Original publisher: [Read original article](<https://dagster.io/blog/how-to-make-the-architectural-case-for-dagster>)

Author: James Dzidek

Published: 2026-06-09T00:00:00Z

Content type: article

Language: en

Sources: [Dagster](<https://devfeed.tech/sources/dagster.md>)

Topics: [DataOps](<https://devfeed.tech/topics/dataops.md>), [internal developer platform](<https://devfeed.tech/topics/internal-developer-platform.md>), [self-service](<https://devfeed.tech/topics/self-service.md>), [engineering-leadership](<https://devfeed.tech/topics/engineering-leadership.md>), [Plover](<https://devfeed.tech/topics/plover.md>)

Tags: [dagster](<https://devfeed.tech/tags/dagster.md>), [deployments](<https://devfeed.tech/tags/deployments.md>), [orchestration](<https://devfeed.tech/tags/orchestration.md>), [product](<https://devfeed.tech/tags/product.md>), [scale](<https://devfeed.tech/tags/scale.md>), [self-service](<https://devfeed.tech/tags/self-service.md>), [teams](<https://devfeed.tech/tags/teams.md>)

### AI overview

The article argues that mature, job-centric data orchestration systems can hide dependencies and make data freshness, quality, and safe reruns difficult to assess. It presents Dagster's asset-aware model and an Orchestration Maturity Model as a way to make lineage and dependencies explicit, automate based on data state, and let product teams contribute pipelines within a shared platform.

### Source excerpt

Mature orchestration environments often work operationally while still leaving critical data dependencies implicit. This post introduces the Orchestration Maturity Model, explains the architectural ceiling of job-centric systems, and shows how Dagster's asset-aware approach helps teams reason about freshness, lineage, quality, and self-service at enterprise scale.

## A community-built data pipeline project using Dagster

DevFeed: [A community-built data pipeline project using Dagster](<https://devfeed.tech/articles/community-showcase-part-1-83616.md>)

Original publisher: [Read original article](<https://dagster.io/blog/community-showcase-part-1>)

Author: Dennis Hume

Published: 2026-06-03T00:00:00Z

Content type: article

Language: en

Sources: [Dagster](<https://devfeed.tech/sources/dagster.md>)

Topics: [CI/CD](<https://devfeed.tech/topics/cicd.md>), [hugging face](<https://devfeed.tech/topics/hugging-face.md>), [Fedora Project Community](<https://devfeed.tech/topics/fedora-project-community.md>), [Query (disambiguation)](<https://devfeed.tech/topics/query.md>)

Tags: [activation](<https://devfeed.tech/tags/activation.md>), [agent](<https://devfeed.tech/tags/agent.md>), [apache-arrow](<https://devfeed.tech/tags/apache-arrow.md>), [api](<https://devfeed.tech/tags/api.md>), [applications](<https://devfeed.tech/tags/applications.md>), [arrow](<https://devfeed.tech/tags/arrow.md>), [best-practices](<https://devfeed.tech/tags/best-practices.md>), [code](<https://devfeed.tech/tags/code.md>), [coding](<https://devfeed.tech/tags/coding.md>), [coding-agents](<https://devfeed.tech/tags/coding-agents.md>), [product](<https://devfeed.tech/tags/product.md>), [showcase](<https://devfeed.tech/tags/showcase.md>)

### AI overview

A data engineer describes building a Dagster-based system for pipelines over public datasets and other domains. The project combines tools including Polars, DuckDB, Parquet, and Apache Arrow, using reusable asset factories for orchestration, checks, metadata, and agent-assisted development. The author recommends starting with a complete, simple pipeline and adding complexity gradually.

### Source excerpt

Some of the most interesting Dagster projects come from the community. This post highlights creative community-built applications.

## How Dagster Compass Powers Brooklyn Data's Self-Service Analytics

DevFeed: [How Dagster Compass Powers Brooklyn Data's Self-Service Analytics](<https://devfeed.tech/articles/how-dagster-compass-powers-brooklyn-data-s-self-service-analytics-83688.md>)

Original publisher: [Read original article](<https://dagster.io/blog/how-dagster-compass-powers-brooklyn-datas-self-service-analytics>)

Author: David A. Gelman

Published: 2026-06-01T00:00:00Z

Content type: article

Language: en

Sources: [Dagster](<https://devfeed.tech/sources/dagster.md>)

Topics: [product analytics](<https://devfeed.tech/topics/product-analytics.md>), [internal developer portal](<https://devfeed.tech/topics/internal-developer-portal.md>), [Amazon Redshift](<https://devfeed.tech/topics/amazon-redshift.md>), [semantic-layer](<https://devfeed.tech/topics/semantic-layer.md>), [BigQuery](<https://devfeed.tech/topics/bigquery.md>), [treeherder](<https://devfeed.tech/topics/treeherder.md>)

Tags: [analytics](<https://devfeed.tech/tags/analytics.md>), [capacity-planning](<https://devfeed.tech/tags/capacity-planning.md>), [data](<https://devfeed.tech/tags/data.md>), [dbt](<https://devfeed.tech/tags/dbt.md>), [governance](<https://devfeed.tech/tags/governance.md>), [product](<https://devfeed.tech/tags/product.md>), [self-service](<https://devfeed.tech/tags/self-service.md>), [self-service-analytics](<https://devfeed.tech/tags/self-service-analytics.md>), [slack](<https://devfeed.tech/tags/slack.md>), [snowflake](<https://devfeed.tech/tags/snowflake.md>)

### AI overview

The guest post describes Brooklyn Data's evaluation and use of Dagster Compass, a Slack based text-to-analytics tool connected to Snowflake. It discusses how table access controls and pull request reviews govern model context, how users contributed business context, and how validating answers against known dashboard outputs informed adoption. Brooklyn Data reports that some operational questions that took 5-10 minutes in its PSA took 1-2 minutes with Compass.

### Source excerpt

Text-to-analytics promises self-service access to data, but adoption depends on usability, governance, and trust. In this guest post, Brooklyn Data explains how it evaluated Compass, deployed it on top of Snowflake, and enabled teams to answer operational questions directly in Slack while maintaining centralized governance and business context.

## Snowflake Runs Your Data: Dagster Runs Everything Else

DevFeed: [Snowflake Runs Your Data: Dagster Runs Everything Else](<https://devfeed.tech/articles/snowflake-runs-your-data-dagster-runs-everything-else-83740.md>)

Original publisher: [Read original article](<https://dagster.io/blog/snowflake-runs-your-data-dagster-runs-everything-else>)

Author: Dennis Hume

Published: 2026-05-28T00:00:00Z

Content type: article

Language: en

Sources: [Dagster](<https://devfeed.tech/sources/dagster.md>)

Topics: [data-engineering](<https://devfeed.tech/topics/data-engineering.md>), [Snowflake streaming data architecture](<https://devfeed.tech/topics/snowflake-streaming-data-architecture.md>), [Data Management](<https://devfeed.tech/topics/data-management.md>), [Apache Kafka](<https://devfeed.tech/topics/apache-kafka.md>)

Tags: [costs](<https://devfeed.tech/tags/costs.md>), [dagster](<https://devfeed.tech/tags/dagster.md>), [data-freshness](<https://devfeed.tech/tags/data-freshness.md>), [data-platform](<https://devfeed.tech/tags/data-platform.md>), [data-products](<https://devfeed.tech/tags/data-products.md>), [dbt](<https://devfeed.tech/tags/dbt.md>), [orchestration](<https://devfeed.tech/tags/orchestration.md>), [product](<https://devfeed.tech/tags/product.md>), [snowflake](<https://devfeed.tech/tags/snowflake.md>), [sql](<https://devfeed.tech/tags/sql.md>)

### AI overview

The article explains how Dagster complements Snowflake as Snowflake takes on more transformation and data freshness work. It describes defining Dagster assets from SQL, representing Snowflake Dynamic Tables as virtual assets and triggering downstream work after refreshes, and attributing Snowflake query costs to pipeline assets.

### Source excerpt

Snowflake increasingly handles transformation and data freshness internally through features like Dynamic Tables and Cortex. Dagster complements Snowflake by providing orchestration, lineage, automation, and cost visibility across your broader data platform from SQL-defined assets to downstream automation and Snowflake query attribution.

## We Tried ty for Performance. It Found Real Bugs

DevFeed: [We Tried ty for Performance. It Found Real Bugs](<https://devfeed.tech/articles/we-tried-ty-for-performance-it-found-real-bugs-83756.md>)

Original publisher: [Read original article](<https://dagster.io/blog/we-tried-ty-for-performance-it-found-real-bugs>)

Author: Dennis Hume

Published: 2026-05-21T00:00:00Z

Content type: article

Language: en

Sources: [Dagster](<https://devfeed.tech/sources/dagster.md>)

Topics: [Mercury](<https://devfeed.tech/topics/mercury-lang.md>), [Static code analysis](<https://devfeed.tech/topics/static-code-analysis.md>), [CI/CD](<https://devfeed.tech/topics/cicd.md>)

Tags: [ci](<https://devfeed.tech/tags/ci.md>), [coding-agents](<https://devfeed.tech/tags/coding-agents.md>), [engineering](<https://devfeed.tech/tags/engineering.md>), [monorepo](<https://devfeed.tech/tags/monorepo.md>), [python](<https://devfeed.tech/tags/python.md>), [speed](<https://devfeed.tech/tags/speed.md>), [type](<https://devfeed.tech/tags/type.md>)

### AI overview

Dagster migrated its large Python monorepo from Pyright to Astral's type checker, ty. The article reports that CI type-checking time fell from about 15 minutes to 1-2 minutes, and ty identified potential runtime bugs Pyright had accepted, alongside false positives and other diagnostic noise. The team migrated incrementally across roughly 100 pull requests, using coding agents to help process packages.

### Source excerpt

We adopted Astral's new Python type checker, ty, to speed up type checking in the Dagster monorepo. The performance gains were dramatic, but the bigger surprise was that ty caught real runtime bugs Pyright missed. Here's what we learned migrating a large Python codebase incrementally to ty.

## The Dagster Almanack: From Complexity to Composability

DevFeed: [The Dagster Almanack: From Complexity to Composability](<https://devfeed.tech/articles/the-dagster-almanack-from-complexity-to-composability-83743.md>)

Original publisher: [Read original article](<https://dagster.io/blog/the-dagster-almanack-from-complexity-to-composability>)

Author: Simon Späti

Published: 2026-05-19T00:00:00Z

Content type: article

Language: en

Sources: [Dagster](<https://devfeed.tech/sources/dagster.md>)

Topics: [Data Management](<https://devfeed.tech/topics/data-management.md>), [observability](<https://devfeed.tech/topics/observability.md>), [Functional programming](<https://devfeed.tech/topics/functional-programming.md>), [AI Infrastructure](<https://devfeed.tech/topics/ai-infrastructure.md>)

Tags: [composability](<https://devfeed.tech/tags/composability.md>), [dagster](<https://devfeed.tech/tags/dagster.md>), [data-platform](<https://devfeed.tech/tags/data-platform.md>), [engineer](<https://devfeed.tech/tags/engineer.md>), [engineering](<https://devfeed.tech/tags/engineering.md>)

### AI overview

The article presents Dagster as a composable, open data platform for data engineering. It describes how data asset based orchestration, declarative workflows, integrations, and a centralized control plane can help teams manage complex data systems and build reliable pipelines.

### Source excerpt

A complete guide with all the insights, tips, and some predictions for the data platform engineer, just like an Almanack provides, with practical information for daily life.

## Announcing the Dagster+ Terraform Provider

DevFeed: [Announcing the Dagster+ Terraform Provider](<https://devfeed.tech/articles/announcing-the-dagster-terraform-provider-83601.md>)

Original publisher: [Read original article](<https://dagster.io/blog/announcing-the-dagster-terraform-provider>)

Author: Dennis Hume

Published: 2026-04-28T00:00:00Z

Content type: release

Language: en

Sources: [Dagster](<https://devfeed.tech/sources/dagster.md>)

Topics: [Terraform](<https://devfeed.tech/topics/terraform.md>), [CI/CD](<https://devfeed.tech/topics/cicd.md>)

Tags: [ci-cd](<https://devfeed.tech/tags/ci-cd.md>), [infrastructure](<https://devfeed.tech/tags/infrastructure.md>), [product](<https://devfeed.tech/tags/product.md>), [pull-requests](<https://devfeed.tech/tags/pull-requests.md>), [terraform](<https://devfeed.tech/tags/terraform.md>)

### AI overview

Dagster introduces an open-source Terraform provider for managing Dagster+ deployments, access controls, alerting, integrations, and other platform configuration as code. Platform teams can define environments declaratively, review changes through pull requests, and use existing infrastructure workflows to improve consistency and detect configuration drift.

### Source excerpt

The Dagster+ Terraform provider lets platform teams manage deployments, access controls, alerting, and more as code. Define entire environments declaratively, review changes through pull requests, and integrate Dagster+ into your existing infrastructure workflows.

## The Missing Half of the Enterprise Context Layer

DevFeed: [The Missing Half of the Enterprise Context Layer](<https://devfeed.tech/articles/the-missing-half-of-the-enterprise-context-layer-83747.md>)

Original publisher: [Read original article](<https://dagster.io/blog/the-missing-half-of-the-enterprise-context-layer>)

Author: Alex Noonan

Published: 2026-04-22T00:00:00Z

Content type: article

Language: en

Sources: [Dagster](<https://devfeed.tech/sources/dagster.md>)

Topics: [AI Context Management](<https://devfeed.tech/topics/ai-context-management.md>), [observability](<https://devfeed.tech/topics/observability.md>), [Retrieval Augmented Generation (RAG)](<https://devfeed.tech/topics/retrieval-augmented-generation-rag.md>), [mcp grafana](<https://devfeed.tech/topics/mcp-grafana.md>)

Tags: [ai](<https://devfeed.tech/tags/ai.md>), [context-graph](<https://devfeed.tech/tags/context-graph.md>), [data](<https://devfeed.tech/tags/data.md>), [data-stack](<https://devfeed.tech/tags/data-stack.md>), [embeddings](<https://devfeed.tech/tags/embeddings.md>), [pipeline](<https://devfeed.tech/tags/pipeline.md>), [product](<https://devfeed.tech/tags/product.md>)

### AI overview

Enterprise AI agents need operational context alongside business definitions to judge whether data is current and trustworthy. The article describes how Dagster's asset materializations, run status, dependencies, freshness signals, and quality results can flow into Atlan's Enterprise Context Layer, including for structured and unstructured data workflows.

### Source excerpt

AI agents that only understand business definitions without knowing whether the underlying pipeline actually succeeded are confidently wrong and operational context from the orchestrator is the missing piece.

## How to Orchestrate Across Multiple Databricks Workspaces Without Losing Your Mind

DevFeed: [How to Orchestrate Across Multiple Databricks Workspaces Without Losing Your Mind](<https://devfeed.tech/articles/how-to-orchestrate-across-multiple-databricks-workspaces-without-losing-your-mind-83693.md>)

Original publisher: [Read original article](<https://dagster.io/blog/how-to-orchestrate-across-multiple-databricks-workspaces-without-losing-your-mind>)

Author: Eric Thomas

Published: 2026-04-20T00:00:00Z

Content type: article

Language: en

Sources: [Dagster](<https://devfeed.tech/sources/dagster.md>)

Topics: [databricks](<https://devfeed.tech/topics/databricks.md>), [Distributed Systems & Parallel Computing](<https://devfeed.tech/topics/distributed-systems-parallel-computing.md>), [etl](<https://devfeed.tech/topics/etl.md>)

Tags: [dagster](<https://devfeed.tech/tags/dagster.md>), [data-products](<https://devfeed.tech/tags/data-products.md>), [databricks](<https://devfeed.tech/tags/databricks.md>), [engineering](<https://devfeed.tech/tags/engineering.md>), [multiple](<https://devfeed.tech/tags/multiple.md>), [orchestration](<https://devfeed.tech/tags/orchestration.md>)

### AI overview

The article explains how Dagster can coordinate data pipelines across multiple Databricks workspaces by representing jobs as assets in a shared graph. It uses a finance dataset and an ML training job to show how declared cross-workspace dependencies and freshness policies can prevent downstream jobs from running on stale inputs.

### Source excerpt

Once your pipelines span multiple Databricks workspaces, you're no longer orchestrating a single system you're coordinating a distributed one.

## Dagster 1.13 adds AI tooling support, partitioned asset checks, virtual assets, and integrations

DevFeed: [Dagster 1.13 adds AI tooling support, partitioned asset checks, virtual assets, and integrations](<https://devfeed.tech/articles/dagster-1-13-octopus-s-garden-83624.md>)

Original publisher: [Read original article](<https://dagster.io/blog/dagster-1-13-octopuss-garden>)

Author: Colton Padden

Published: 2026-04-09T00:00:00Z

Content type: release

Language: en

Sources: [Dagster](<https://devfeed.tech/sources/dagster.md>)

Topics: [internal developer platform](<https://devfeed.tech/topics/internal-developer-platform.md>), [CI/CD](<https://devfeed.tech/topics/cicd.md>)

Tags: [agentic-coding](<https://devfeed.tech/tags/agentic-coding.md>), [bigquery](<https://devfeed.tech/tags/bigquery.md>), [blob-storage](<https://devfeed.tech/tags/blob-storage.md>), [build](<https://devfeed.tech/tags/build.md>), [ci-cd](<https://devfeed.tech/tags/ci-cd.md>), [claude-code](<https://devfeed.tech/tags/claude-code.md>), [cli](<https://devfeed.tech/tags/cli.md>), [coding-agents](<https://devfeed.tech/tags/coding-agents.md>), [coding-tools](<https://devfeed.tech/tags/coding-tools.md>), [configuration](<https://devfeed.tech/tags/configuration.md>), [dagster](<https://devfeed.tech/tags/dagster.md>), [releases](<https://devfeed.tech/tags/releases.md>), [skills](<https://devfeed.tech/tags/skills.md>)

### AI overview

Dagster 1.13 adds reusable skills for AI coding tools, partition-aware asset checks, a more programmatic CLI, and preview support for virtual assets such as database views. The release also enables persisted state by default for state-backed components and expands integrations across data and cloud platforms.

### Source excerpt

Dagster skills, partitioned asset checks, state backed components, virtual assets, and stronger integrations.

## Monorepos, the hub-and-spoke model, and Copybara

DevFeed: [Monorepos, the hub-and-spoke model, and Copybara](<https://devfeed.tech/articles/monorepos-the-hub-and-spoke-model-and-copybara-83703.md>)

Original publisher: [Read original article](<https://dagster.io/blog/monorepos-the-hub-and-spoke-model-and-copybara>)

Author: Colton Padden

Published: 2026-04-03T00:00:00Z

Content type: article

Language: en

Sources: [Dagster](<https://devfeed.tech/sources/dagster.md>)

Topics: [CI/CD](<https://devfeed.tech/topics/cicd.md>), [Submodules](<https://devfeed.tech/topics/submodules.md>), [gitflow](<https://devfeed.tech/topics/gitflow.md>)

Tags: [devtools](<https://devfeed.tech/tags/devtools.md>), [engineering](<https://devfeed.tech/tags/engineering.md>), [github](<https://devfeed.tech/tags/github.md>), [github-actions](<https://devfeed.tech/tags/github-actions.md>), [repositories](<https://devfeed.tech/tags/repositories.md>)

### AI overview

Dagster describes consolidating development in an internal monorepo while maintaining public repositories as spokes, using Copybara for bidirectional synchronization. It explains revision labels and subtree checks for preventing sync loops, Buildkite pipelines, and a GitHub Actions merge queue gate to avoid landing changes before synchronization catches up. Public contributors can continue opening pull requests against the Dagster repository without managing the internal sync process.

### Source excerpt

How we configure Copybara for bi-directional syncing to enable a hub-and-spoke model for Git repositories

## Making Dagster Easier to Contribute to in an AI-Driven World

DevFeed: [Making Dagster Easier to Contribute to in an AI-Driven World](<https://devfeed.tech/articles/making-dagster-easier-to-contribute-to-in-an-ai-driven-world-83702.md>)

Original publisher: [Read original article](<https://dagster.io/blog/making-dagster-easier-to-contribute>)

Author: Dennis Hume

Published: 2026-04-01T00:00:00Z

Content type: article

Language: en

Sources: [Dagster](<https://devfeed.tech/sources/dagster.md>)

Topics: [Maintainers](<https://devfeed.tech/topics/maintainers.md>), [Code review](<https://devfeed.tech/topics/code-review.md>), [Open Source](<https://devfeed.tech/topics/open-source.md>), [Artificial Intelligence](<https://devfeed.tech/topics/ai.md>)

Tags: [ai-assisted-workflows](<https://devfeed.tech/tags/ai-assisted-workflows.md>), [ai-tools](<https://devfeed.tech/tags/ai-tools.md>), [contribute](<https://devfeed.tech/tags/contribute.md>), [development](<https://devfeed.tech/tags/development.md>), [open-source](<https://devfeed.tech/tags/open-source.md>), [product](<https://devfeed.tech/tags/product.md>), [review](<https://devfeed.tech/tags/review.md>), [tooling](<https://devfeed.tech/tags/tooling.md>)

### AI overview

Dagster describes efforts to make open-source contributions easier to review and maintain as AI tools make pull requests simpler to generate. It is adding repository-aware review tooling and clearer contributor guidance, and recommends focused, limited-scope changes to help reviewers evaluate and merge contributions.

### Source excerpt

AI has made contributing to open source easier but reviewing contributions is still hard. At Dagster, we're improving the contributor experience with smarter review tooling, clearer guidelines, and a focus on contributions that are easier to evaluate, merge, and maintain.

## DataOps with Dagster: A Practical Guide to Building a Reliable Data Platform

DevFeed: [DataOps with Dagster: A Practical Guide to Building a Reliable Data Platform](<https://devfeed.tech/articles/dataops-with-dagster-a-practical-guide-to-building-a-reliable-data-platform-83663.md>)

Original publisher: [Read original article](<https://dagster.io/blog/dataops-with-dagster-a-practical-guide-to-building-a-reliable-data-platform>)

Author: Alex Noonan

Published: 2026-03-17T00:00:00Z

Content type: tutorial

Language: en

Sources: [Dagster](<https://devfeed.tech/sources/dagster.md>)

Topics: [DataOps](<https://devfeed.tech/topics/dataops.md>)

Tags: [dagster](<https://devfeed.tech/tags/dagster.md>), [data](<https://devfeed.tech/tags/data.md>), [dataops](<https://devfeed.tech/tags/dataops.md>), [engineering](<https://devfeed.tech/tags/engineering.md>), [guide](<https://devfeed.tech/tags/guide.md>)

### AI overview

This practical guide explains how Dagster supports DataOps across development and production. It covers pipeline development and branch deployments, then describes operational practices such as retries, concurrency controls, run priorities and timeouts, backfills, data quality checks, asset visibility, alerts, and operational metrics.

### Source excerpt

DataOps is about building a system that provides visibility into what's happening and control over how it behaves

## Unlocking the Full Value of Your Databricks

DevFeed: [Unlocking the Full Value of Your Databricks](<https://devfeed.tech/articles/unlocking-the-full-value-of-your-databricks-83752.md>)

Original publisher: [Read original article](<https://dagster.io/blog/unlocking-the-full-value-of-your-databricks>)

Author: Dennis Hume

Published: 2026-03-12T00:00:00Z

Content type: article

Language: en

Sources: [Dagster](<https://devfeed.tech/sources/dagster.md>)

Topics: [databricks](<https://devfeed.tech/topics/databricks.md>), [Databricks Unity Catalog integration](<https://devfeed.tech/topics/databricks-unity-catalog-integration.md>), [data-engineering](<https://devfeed.tech/topics/data-engineering.md>)

Tags: [airbyte](<https://devfeed.tech/tags/airbyte.md>), [dagster](<https://devfeed.tech/tags/dagster.md>), [data-products](<https://devfeed.tech/tags/data-products.md>), [databricks](<https://devfeed.tech/tags/databricks.md>), [dbt](<https://devfeed.tech/tags/dbt.md>), [product](<https://devfeed.tech/tags/product.md>), [unity-catalog](<https://devfeed.tech/tags/unity-catalog.md>)

### AI overview

The article argues that standardizing on Databricks does not by itself provide an operating model across enterprise teams and downstream systems. It presents Dagster as a complementary coordination layer that connects assets and dependencies across tools, with Unity Catalog remaining the governance authority for Databricks assets. The proposed shared operational view is intended to help teams track lineage, freshness, ownership, and downstream impact.

### Source excerpt

Standardizing on Databricks is a smart strategic move, but consolidation alone does not create a working operating model across teams, tools, and downstream systems. By pairing Databricks and Unity Catalog with Dagster, enterprises can add the coordination layer needed for dependency visibility, end-to-end lineage, and faster, more confident delivery at scale.

## Announcing AI Driven Data Engineering

DevFeed: [Announcing AI Driven Data Engineering](<https://devfeed.tech/articles/announcing-ai-driven-data-engineering-83599.md>)

Original publisher: [Read original article](<https://dagster.io/blog/announcing-ai-driven-data-engineering>)

Author: Dennis Hume

Published: 2026-03-05T00:00:00Z

Content type: release

Language: en

Sources: [Dagster](<https://devfeed.tech/sources/dagster.md>)

Topics: [Loop Engineering](<https://devfeed.tech/topics/loop-engineering.md>), [streaming data to S3 with Parquet](<https://devfeed.tech/topics/streaming-data-to-s3-with-parquet.md>), [GitHub Copilot](<https://devfeed.tech/topics/github-copilot.md>), [data-engineering](<https://devfeed.tech/topics/data-engineering.md>)

Tags: [ai-assisted-development](<https://devfeed.tech/tags/ai-assisted-development.md>), [ai-coding-agents](<https://devfeed.tech/tags/ai-coding-agents.md>), [dagster](<https://devfeed.tech/tags/dagster.md>), [data-engineering](<https://devfeed.tech/tags/data-engineering.md>), [engineering](<https://devfeed.tech/tags/engineering.md>)

### AI overview

Dagster is introducing an eight-lesson Dagster University course on building an ELT pipeline with AI coding agents. It covers agent guidance, Dagster best practices, verification of generated output, and workflows for managing project structure, sessions, and context. The course is free and open to everyone.

### Source excerpt

AI coding agents are changing how data engineers work. This Dagster University course shows how to build a production-ready ELT pipeline from prompts while learning practical patterns for reliable AI-assisted development.

## When to Move from Dagster OSS to Dagster+

DevFeed: [When to Move from Dagster OSS to Dagster+](<https://devfeed.tech/articles/when-to-move-from-dagster-oss-to-dagster-83763.md>)

Original publisher: [Read original article](<https://dagster.io/blog/when-to-move-from-dagster-oss-to-dagster>)

Author: Dennis Hume

Published: 2026-02-26T00:00:00Z

Content type: article

Language: en

Sources: [Dagster](<https://devfeed.tech/sources/dagster.md>)

Topics: [lead time](<https://devfeed.tech/topics/lead-time.md>), [internal developer platform automation](<https://devfeed.tech/topics/internal-developer-platform-automation.md>), [CI/CD](<https://devfeed.tech/topics/cicd.md>), [Frontend & real-time](<https://devfeed.tech/topics/frontend-real-time.md>)

Tags: [access-control](<https://devfeed.tech/tags/access-control.md>), [access-management](<https://devfeed.tech/tags/access-management.md>), [alert](<https://devfeed.tech/tags/alert.md>), [alerting](<https://devfeed.tech/tags/alerting.md>), [all](<https://devfeed.tech/tags/all.md>), [allows](<https://devfeed.tech/tags/allows.md>), [also](<https://devfeed.tech/tags/also.md>), [analyst](<https://devfeed.tech/tags/analyst.md>), [analysts](<https://devfeed.tech/tags/analysts.md>), [apis](<https://devfeed.tech/tags/apis.md>), [automate](<https://devfeed.tech/tags/automate.md>), [compliance](<https://devfeed.tech/tags/compliance.md>), [control-plane](<https://devfeed.tech/tags/control-plane.md>), [dagster](<https://devfeed.tech/tags/dagster.md>), [data-products](<https://devfeed.tech/tags/data-products.md>), [engineering](<https://devfeed.tech/tags/engineering.md>), [operational](<https://devfeed.tech/tags/operational.md>), [oss](<https://devfeed.tech/tags/oss.md>), [platform](<https://devfeed.tech/tags/platform.md>), [product](<https://devfeed.tech/tags/product.md>), [products](<https://devfeed.tech/tags/products.md>)

### AI overview

The article explains when teams using Dagster OSS may benefit from moving to Dagster+, focusing on the operational work of running the platform as teams and data products grow. It describes managed infrastructure, role-based access control, environment isolation, branch deployments, observability and asset lineage, while emphasizing that teams can keep their existing Dagster asset model and Python code.

### Source excerpt

Dagster OSS is built for builders. But as teams grow, the operational burden of running the platform can quietly consume engineering time. This guide explains when it makes sense to move to Dagster+ and shift your focus back to building data products.

## Sample-Level Versioning for ML Pipelines with Dagster and Metaxy

DevFeed: [Sample-Level Versioning for ML Pipelines with Dagster and Metaxy](<https://devfeed.tech/articles/sample-level-versioning-for-ml-pipelines-with-dagster-and-metaxy-83609.md>)

Original publisher: [Read original article](<https://dagster.io/blog/building-real-time-interactive-avatars-with-metaxy>)

Author: Daniel Gafni

Published: 2026-02-13T00:00:00Z

Content type: tutorial

Language: en

Sources: [Dagster](<https://devfeed.tech/sources/dagster.md>)

Topics: [Data pipelines](<https://devfeed.tech/topics/data-pipelines.md>), [data observability](<https://devfeed.tech/topics/data-observability.md>), [CI/CD](<https://devfeed.tech/topics/cicd.md>)

Tags: [dagster](<https://devfeed.tech/tags/dagster.md>), [engineering](<https://devfeed.tech/tags/engineering.md>), [ml](<https://devfeed.tech/tags/ml.md>), [pipelines](<https://devfeed.tech/tags/pipelines.md>), [versioning](<https://devfeed.tech/tags/versioning.md>)

### AI overview

An MLOps engineer explains how Metaxy adds sample-level and field-level versioning to multimodal data pipelines, integrating with Dagster to identify which samples need processing after an upstream change. This selective recomputation can avoid rerunning downstream steps that do not depend on the changed data.

### Source excerpt

Learn how Metaxy can be used to build multimodal data pipelines with sample-level granularity on Dagster

## How Dagster Evaluates Its Agent Skills

DevFeed: [How Dagster Evaluates Its Agent Skills](<https://devfeed.tech/articles/evaluating-skills-83680.md>)

Original publisher: [Read original article](<https://dagster.io/blog/evaluating-agent-skills>)

Author: Colton Padden

Published: 2026-02-06T00:00:00Z

Content type: article

Language: en

Sources: [Dagster](<https://devfeed.tech/sources/dagster.md>)

Topics: [Agent Skills](<https://devfeed.tech/topics/agent-skills.md>), [Developer skills](<https://devfeed.tech/topics/developer-skills.md>), [ai observability](<https://devfeed.tech/topics/ai-observability.md>)

Tags: [agent](<https://devfeed.tech/tags/agent.md>), [ai](<https://devfeed.tech/tags/ai.md>), [dagster](<https://devfeed.tech/tags/dagster.md>), [engineering](<https://devfeed.tech/tags/engineering.md>), [evaluation](<https://devfeed.tech/tags/evaluation.md>), [framework](<https://devfeed.tech/tags/framework.md>), [skills](<https://devfeed.tech/tags/skills.md>)

### AI overview

Dagster describes a lightweight framework for evaluating whether Claude uses its skills correctly and produces expected outputs. The framework runs prompts against skills, records tool use, token usage, and outputs in versioned snapshots, and validates resulting Dagster definitions. The team reports that concise skills, decision trees, purpose-built command-line utilities, and feedback loops improved their ability to assess and refine agent workflows.

### Source excerpt

We built a light weight evaluation framework to quantiatively measure the effectiveness of the Dagster Skills, and these are our findings.

[Next page](<https://devfeed.tech/sources/dagster.md?cursor=WyIyMDI2LTAyLTA2VDAwOjAwOjAwKzAwOjAwIiwgIjMyMjlkZTA5LTYxNmItNDJjMS04NGI0LTQzNzFlNGY2NjEwMSJd>)