# Scheduler Published articles for Scheduler. This is one page of public article previews, not the complete archive. Follow Next page to continue. Summaries are not the original full articles. ## Kubernetes Architecture: Control Plane, Scheduler, and Kubelet DevFeed: [Kubernetes Architecture: Control Plane, Scheduler, and Kubelet]() Original publisher: [Read original article]() Author: Nimesha Jinarajadasa Published: 2026-09-17T06:25:02Z Content type: tutorial Language: en Sources: [Kubernetes - KodeKloud Blog | DevOps, Cloud, Kubernetes, AI Tutorials & More]() Topics: [Kubernetes](), [control-plane](), [api server](), [Containers](), [etcd](), [kubectl]() Tags: [api](), [api-server](), [containers](), [control-plane](), [controllers](), [etcd](), [kubectl](), [kubernetes](), [kubernetes-architecture](), [kubernetes-networking](), [node](), [scheduler]() ### AI overview A beginner-friendly tutorial explaining how Kubernetes maintains the desired state of containerized applications. It covers the control plane, API server, etcd, controllers, scheduler, and kubelet. ### Source excerpt Learn how Kubernetes works, from the control plane and scheduler to kubelet, pods, and Services, with simple answers to interview questions. ## Linux Randomly Picking Idle CPU Cores Shows Modest Win On High Core Count Systems DevFeed: [Linux Randomly Picking Idle CPU Cores Shows Modest Win On High Core Count Systems]() Original publisher: [Read original article]() Author: Michael Larabel Published: 2026-09-16T13:30:04Z Content type: news Language: en Sources: [Phoronix]() Topics: [Linux](), [Linux Kernel](), [cpu](), [benchmarking](), [systems]() Tags: [benchmarking](), [cpu](), [desktop-linux](), [linux](), [linux-benchmarking](), [linux-hardware-benchmarks](), [linux-hardware-reviews](), [linux-how-to](), [linux-kernel](), [linux-performance](), [linux-server-benchmarks](), [open-source-graphics](), [patches](), [phoronix](), [phoronix-test-suite](), [scheduler](), [ubuntu-benchmarks](), [ubuntu-hardware]() ### AI overview A Linux kernel patch series proposes randomizing the selection among equally shallow idle CPU cores to reduce scan-order bias and prevent concurrent tasks from converging on the same core. Testing on a 160-core Ampere Altra system showed throughput improvements of up to a few percent in Stress-NG benchmarking, while the benefits on smaller systems remain uncertain. ### Source excerpt Arm engineer Christian Loehle sent out a patch series today proposing that the Linux scheduler code randomize its picking of equally shallow idle CPU cores. This randomization is showing to deliver a win at least for high CPU core counts for better distributing workloads... ## Monitor TAS and gang scheduling for AI training in Kubernetes DevFeed: [Monitor TAS and gang scheduling for AI training in Kubernetes]() Original publisher: [Read original article]() Author: David Lentz; Kathy Lin Published: 2026-09-15T00:00:00Z Content type: article Language: en Sources: [Datadog | The Monitor blog]() Topics: [Kubernetes](), [kueue](), [distributed-training](), [GPU](), [Monitoring](), [datadog]() Tags: [ai](), [batch](), [containers](), [datadog](), [distributed-training](), [gpu](), [gpu-monitoring](), [kubernetes](), [kueue](), [monitoring](), [scheduler]() ### AI overview This article explains why Kubernetes scheduling is insufficient for distributed AI training workloads and how topology-aware scheduling and gang scheduling address hardware placement and simultaneous startup requirements. It discusses implementing these capabilities with Kueue and the Coscheduling plugin, and monitoring and troubleshooting them with Datadog GPU Monitoring. ### Source excerpt Learn how Datadog helps you correlate Kueue, Coscheduling, GPU, and training framework signals to validate gang scheduling and topology-aware scheduling. ## A Backward NTP Clock Step Delayed a Python Scheduler's Measurements by Nine Days DevFeed: [A Backward NTP Clock Step Delayed a Python Scheduler's Measurements by Nine Days]() Original publisher: [Read original article]() Author: Philipp Lehmann (philipp.lehmann@gruppe.ai) Published: 2026-09-14T07:00:00Z Content type: tutorial Language: en Sources: [Philipp Lehmann]() Topics: [Python](), [Linux](), [systemd](), [Single Board Computer]() Tags: [embedded](), [linux](), [ntp](), [python](), [reliability](), [scheduler](), [single-board-computer](), [systemd](), [time](), [wall-clock-time]() ### AI overview A Python scheduler on an embedded Linux device silently stopped running after NTP corrected the system clock backward by about nine days. Because it used wall-clock time to measure elapsed time, the scheduler waited roughly nine days before running again. The article explains why a monotonic clock is the appropriate fix and mentions testing with an injected clock jump. ### Source excerpt time.time() vs time.monotonic() in a Python scheduler: a backwards NTP step delayed runs by nine days. Monotonic durations and a jump-injecting test. ## DB-Scheduler: A Persistent, Cluster-Friendly Scheduler for Java DevFeed: [DB-Scheduler: A Persistent, Cluster-Friendly Scheduler for Java]() Original publisher: [Read original article]() Author: Graham Cox Published: 2026-08-22T23:11:36Z Content type: tutorial Language: en Sources: [Baeldung]() Topics: [Java](), [Quartz]() Tags: [how-to](), [java](), [persistence](), [persistence-popular-quartz](), [popular](), [postgresql](), [quartz](), [scheduler](), [tutorial]() ### AI overview This tutorial explains how to use the db-scheduler library for persistent, cluster-aware task scheduling in Java. It covers one-time and recurring tasks, project dependencies, the required database table, PostgreSQL schema setup, and Scheduler configuration. ### Source excerpt Learn how to use the db-scheduler library to store, manage and execute persistent scheduled tasks in Java. The post DB-Scheduler: A Persistent, Cluster-Friendly Scheduler for Java first appeared on Baeldung. ## How dbt works, and why orchestrators shouldn't split it into tasks DevFeed: [How dbt works, and why orchestrators shouldn't split it into tasks]() Original publisher: [Read original article]() Author: Ruben Fiszel Published: 2026-08-20T00:00:00Z Content type: article Language: en Sources: [Windmill Blog]() Topics: [Compiler](), [Job](), [airflow](), [data](), [Command-line interface]() Tags: [airflow](), [bigquery](), [cli](), [dagster](), [data](), [data-pipelines](), [dbt](), [dbt-data-pipelines-airflow-dagster-orchestration](), [job](), [orchestration](), [scheduler](), [series]() ### AI overview This primer explains dbt's architecture and examines five ways teams orchestrate it in production. It argues that running a project as one dbt command, while reading dbt's per-model execution state, is generally more efficient and less fragile than creating one orchestrator task per model. It discusses Dagster and astronomer-cosmos's convergence on this approach, including a reported cost comparison. ### Source excerpt How is dbt actually orchestrated, and why does running it as one job beat one task per model? A primer on dbt as a compiler with a scheduler attached, the five ways teams wrap it, and why both Dagster and astronomer-cosmos converged on a single dbt invocation with per-model state projected out of it. ## Kubernetes Control Plane Internals and Reconciliation DevFeed: [Kubernetes Control Plane Internals and Reconciliation]() Original publisher: [Read original article]() Author: Alexandre Zajac Published: 2026-08-03T15:31:23Z Content type: tutorial Language: en Sources: [Hungry Minds]() Topics: [Kubernetes](), [distributed-systems](), [Containers]() Tags: [container](), [internals](), [kubernetes](), [reconciliation](), [scheduler]() ### AI overview This article explains how Kubernetes manages distributed workloads through its control plane, worker nodes, and reconciliation loop. It covers the API server, etcd, scheduling, kubelet, desired-versus-actual state, and automated recovery. ### Source excerpt PLUS: Ray 2x throughput boost ⚡, take-home interview breakdown 🔬, great work principles 📚 ## Keeping the peering inventory honest DevFeed: [Keeping the peering inventory honest]() Original publisher: [Read original article]() Author: Bastiaan Brink Published: 2026-07-30T00:00:00Z Content type: tutorial Language: en Sources: [AS215248 - Notes]() Topics: [Database](), [API](), [Job](), [Python]() Tags: [api](), [authentication](), [automation](), [cloudflare](), [database](), [ops](), [peering](), [python](), [scheduler]() ### AI overview The article describes keeping Peering Manager's inventory synchronized with router state by mirroring portal changes and running a 15-minute reconciliation job. It also documents API integration pitfalls and fixes a failed daily PeeringDB sync by pinning the service image to a digest. ### Source excerpt A database that says you have a session is worthless if the router disagrees. A 15-minute reconcile keeps Peering Manager and the routers in step. ## How the Operating System Runs Many Tasks on One CPU DevFeed: [How the Operating System Runs Many Tasks on One CPU]() Original publisher: [Read original article]() Author: Franco Fernando Published: 2026-07-11T08:25:54Z Content type: article Language: en Sources: [The Polymathic Engineer]() Topics: [Operating system](), [cpu](), [context]() Tags: [context](), [cpu](), [multitasking](), [scheduler](), [switching]() ### AI overview This article explains how an operating system runs many tasks on one CPU through multitasking, context switching, and scheduling. ### Source excerpt Multitasking, context switching, and the scheduler that makes it all work. ## How the Kubernetes control plane works DevFeed: [How the Kubernetes control plane works]() Original publisher: [Read original article]() Author: LearnKube Published: 2026-04-17T00:00:00Z Content type: tutorial Language: en Sources: [Learnk8s]() Topics: [Kubernetes](), [API](), [Cloud Native Ecosystem]() Tags: [api](), [api-server](), [authentication](), [authorization](), [declarative](), [kubernetes](), [reconciliation](), [scheduler](), [schema]() ### AI overview This tutorial explains how the Kubernetes control plane accepts changes, stores desired state, and keeps a cluster aligned with that state. It introduces the API server, etcd, scheduler, and controller manager, then describes request validation and persistence in etcd. ### Source excerpt A visual map of the Kubernetes control plane with linked deep dives into the API server, scheduler, controller manager, and etcd. ## How does the Kubernetes scheduler work? DevFeed: [How does the Kubernetes scheduler work?]() Original publisher: [Read original article]() Author: Daniele Polencic Published: 2026-02-09T00:00:00Z Content type: tutorial Language: en Sources: [Learnk8s]() Topics: [Kubernetes]() Tags: [deployment](), [kubernetes](), [scheduler](), [work]() ### AI overview This tutorial explains how the Kubernetes scheduler assigns pods to worker nodes by filtering unsuitable nodes and scoring the remaining candidates. It also clarifies the roles of the controller manager, ReplicaSets, the API server, and the scheduler queue. ### Source excerpt The Kubernetes scheduler decides where your pods run. Learn how it filters nodes, scores candidates, and assigns workloads to the right place in your cluster. ## Redesigning my microkernel from the ground up DevFeed: [Redesigning my microkernel from the ground up]() Original publisher: [Read original article]() Author: January Published: 2026-01-12T00:00:00Z Content type: opinion Language: en Sources: [Drew DeVault]() Topics: [systems](), [Kernel](), [kernels](), [Development](), [Code](), [Unix]() Tags: [code](), [cpu](), [development](), [kernel](), [kernels](), [operating-systems](), [os](), [scheduler](), [systems](), [unix]() ### AI overview The author explains why the Helios microkernel project was abandoned after design and implementation problems, and describes starting Hermes from scratch while reusing selected code from Helios and Bunnix. Hermes is not yet feature-complete, but the author says it has a more robust design and supports symmetric multiprocessing. ### Source excerpt As you may recall, circa 2022-2023 I was working on a microkernel written in Hare named Helios. Helios was largely inspired by and modelled after the design of seL4 and was my first major foray into modern OS development that was serious enough to get to a somewhat useful state of functionality, with drives for some real hardware, filesystems, and an environment for running user programs of a reasonable level of sophistication. Helios development went strong for a while but eventually it slowed and eventually halted in a state of design hell. Since Helios was my first major OS project at this scale and with this much ambition, the design and implementation ended up with a lot of poor assumptions that made it a pretty weak foundation for building a complete OS upon. In late 2023 I more or less gave up on it and moved my OS development work out of the realm of writing code and back into the realm of thinking really hard about how to design operating systems. What followed was a couple of years of design thinking, developing small scale design experiments, and doing deeper research into prior art - reading papers and studying existing kernels. It was also during this period that I wrote Bunnix, a working Unix clone, motivated in part by a desire to gain some first-hand experience working in the design and implementation of Unix-style operating systems - a fertile environment for learning a lot of the nuts and bolts of OS implementations by working against a complete and proven design. In August I was finally prepared to have another go. I decided to start over from scratch, importing and adapting and rewriting code from Helios and Bunnix on an as-needed basis to speed things up, and writing from scratch anything where the lessons learned in hindsight outweighed the benefits of adapting existing code.1 The result is Hermes. Hermes has not yet reached feature parity with Helios, lacking some IPC features and an aarch64 port, but already it's significantly more robust and ## Temporal Ruby SDK Reaches General Availability with Rust Core and Deterministic Fiber Scheduler DevFeed: [Temporal Ruby SDK Reaches General Availability with Rust Core and Deterministic Fiber Scheduler]() Original publisher: [Read original article]() Author: Chad Retz Published: 2025-10-16T00:00:00Z Content type: article Language: en Sources: [Temporal Blog]() Topics: [Ruby](), [SDKs](), [Rust](), [Programming](), [cancellation](), [Code](), [Open Source]() Tags: [announcements](), [asynchronous](), [cancellation](), [fiber](), [guardrails](), [open-source](), [ruby](), [rust](), [scheduler](), [sdk](), [workflows]() ### AI overview Temporal Ruby is now generally available, providing Ruby developers with a native-feeling SDK for building durable software and Workflows. The article introduces its programming model and demonstrates Activities, deterministic fibers, durable timers, cancellation, and updates, while also discussing the Rust-powered implementation and execution guardrails. ### Source excerpt Temporal Ruby SDK is GA. Build durable Ruby Workflows with native APIs, a Rust-powered core, a deterministic fiber scheduler, and guardrails for safe execution. ## How we replaced the default K8s scheduler to optimize our Continuous Integration builds DevFeed: [How we replaced the default K8s scheduler to optimize our Continuous Integration builds]() Original publisher: [Read original article]() Author: Vadim Gusev Published: 2025-07-07T11:57:20Z Content type: article Language: en Sources: [Codefresh]() Topics: [Kubernetes](), [ci]() Tags: [clusters](), [codefresh](), [continuous-integration](), [devops](), [glossary](), [k8s](), [kubernetes](), [orchestration](), [pipeline](), [product-and-tools](), [scheduler]() ### AI overview Codefresh describes replacing the default Kubernetes scheduler for clusters running ephemeral Continuous Integration pipelines. The article explains how this workload differed from long-running applications and motivated a different approach to Kubernetes scheduling. ### Source excerpt The default Kubernetes scheduler works great when your cluster is destined for long running applications. At Codefresh we use our Kubernetes clusters for running Continuous Integration pipelines which means our workloads are ephemeral (they are discarded when a pipeline has finished). This allowed us to look at the Kubernetes scheduler from a different perspective and [...] The post How we replaced the default K8s scheduler to optimize our Continuous Integration builds appeared first on Codefresh. ## Scheduling In Go : Part II - Go Scheduler DevFeed: [Scheduling In Go : Part II - Go Scheduler]() Original publisher: [Read original article]() Published: 2025-04-14T00:00:00Z Content type: article Language: en Sources: [William Kennedy]() Topics: [Go Language](), [Concurrency](), [Optimization](), [systems]() Tags: [ardan-labs](), [blog](), [concurrency](), [concurrent](), [context](), [go](), [go-programming](), [golang](), [optimization](), [os](), [parallel](), [programming](), [scheduler](), [switching](), [systems](), [threads]() ### AI overview The second article in a three-part series explains the Go scheduler at a semantic level. It covers logical processors, global and local run queues, context switching, work stealing, and synchronous and asynchronous system calls. ### Source excerpt This blogpost is the second installment in a three-part series exploring the mechanics and semantics of the Go scheduler. Despite being published in 2018, the content remains relevant today, as the Go scheduler's design continues to influence the development of efficient and scalable concurrent systems. In this post, we will go into the inner workings of the Go scheduler, discussing its components, such as the Global Run Queue (GRQ) and Local Run Queue (LRQ), and its behavior, including context switching, work stealing, and the handling of synchronous and asynchronous system calls. By understanding these concepts, developers can make informed decisions about concurrency and optimization in their Go applications. ## Eco and Mini Plans Now Generally Available DevFeed: [Eco and Mini Plans Now Generally Available]() Original publisher: [Read original article]() Author: Chris Peterson Published: 2022-11-08T00:10:00Z Content type: release Language: en Sources: [Heroku]() Topics: [Heroku](), [upgrade](), [Redis](), [Databases](), [dashboards]() Tags: [applications](), [apps](), [data](), [databases](), [dynos](), [free](), [heroku](), [heroku-postgres](), [news](), [redis](), [scheduler](), [upgrade]() ### AI overview Heroku announces the general availability of Eco dynos and Mini plans for Heroku Postgres and Heroku Data for Redis. The plans are available for new and existing applications, with free access for eligible card-paying customers until November 30, 2022, before charges begin on December 1, 2022. ### Source excerpt In September, we announced our new low-cost Eco dynos plan and Mini plans for Heroku Postgres and Heroku Data for Redis®. The time has come! These plans are available today for new and existing applications. For customers paying by credit or debit card, the Eco dynos and Mini data plans are free until November 30th, [...] The post Eco and Mini Plans Now Generally Available appeared first on Heroku. ## How to test intermediate steps in suspending functions DevFeed: [How to test intermediate steps in suspending functions]() Original publisher: [Read original article]() Published: 2022-09-30T00:00:00Z Content type: tutorial Language: en Sources: [Arkadiusz Chmura]() Topics: [Testing](), [Coroutines](), [execution]() Tags: [code-testing](), [coroutines](), [dispatcher](), [scheduler](), [testing](), [virtual]() ### AI overview This tutorial explains how to test intermediate state changes inside suspending functions. It shows how coroutine test dispatchers and virtual-time controls can verify behavior before and after a suspended network call without waiting through real delays. ### Source excerpt Testing the final result of a suspending function is easy, but what about verifying what happens inside it during the execution? ## Lessons Learned From Running Apache Airflow at Scale DevFeed: [Lessons Learned From Running Apache Airflow at Scale]() Original publisher: [Read original article]() Author: Megan Parker Published: 2022-05-23T14:00:01Z Content type: article Language: en Sources: [Shopify Engineering](), [Shopify Engineering - Shopify Engineering]() Topics: [airflow](), [Orchestration](), [Shopify](), [Kubernetes](), [Google Cloud Platform (GCP)](), [MySQL](), [Apache Iceberg](), [Training AI Models](), [data-modeling]() Tags: [airflow](), [apache](), [cloud](), [cloud-storage](), [data-modeling](), [google-cloud](), [kubernetes](), [load](), [machine-learning](), [model-training](), [monitoring](), [mount](), [mysql](), [orchestration](), [scale](), [scheduler](), [shopify](), [solutions](), [workflows]() ### AI overview Shopify explains lessons from operating Apache Airflow at scale for more than two years. The deployment runs on Kubernetes with the Celery executor and MySQL, supporting data extraction, machine learning model training, Apache Iceberg table maintenance, and DBT-powered data modeling. The article describes scaling to more than 10,000 DAGs, over 400 concurrent tasks, and more than 150,000 daily runs, along with challenges involving file access, DAG capabilities, traffic variability, and resource contention. ### Source excerpt Shopify shares some of the lessons we learned and solutions we built in order to run Airflow at scale. ## How to schedule jobs with Quartz on Ktor DevFeed: [How to schedule jobs with Quartz on Ktor]() Original publisher: [Read original article]() Author: Marco Gomiero Published: 2022-04-22T00:00:00Z Content type: tutorial Language: en Sources: [Posts on Marco Gomiero]() Topics: [Ktor](), [Quartz](), [Back end](), [Database](), [Persistence](), [MySQL]() Tags: [backend](), [database](), [how-to](), [ktor](), [mysql](), [persistence](), [quartz](), [scheduler](), [scheduling]() ### AI overview This tutorial explains how to schedule recurring jobs in a Ktor backend project using Quartz. It covers Quartz's persistence of job state, database table setup, logging configuration, and scheduler creation, with MySQL used in the example. ### Source excerpt SERIES: Building a backend with Ktor Part 1: Structuring a Ktor project Part 2: How to persist Ktor logs Part 3: How to use an in-memory database for testing on Ktor Part 4: How to handle database migrations with Liquibase on Ktor Part 5: Generate API documentation from Swagger on Ktor Part 6: How to schedule jobs with Quartz on Ktor Part 7: Moving from mobile to backend development with Ktor Sometimes, on a backend project, there is the need to run one or more tasks periodically, like for system administration, maintenance, backup, syncing content in the background, etc. These types of tasks can be scheduled "manually" with a cron job or with a scheduling library like Quartz, that makes easy for example the persistence of the task's state even after a reboot of the server. ## Tutorial: Authentication and Authorization for Kubernetes DevFeed: [Tutorial: Authentication and Authorization for Kubernetes]() Original publisher: [Read original article]() Author: info@goteleport.com (Lukonde Mwila) Published: 2022-02-15T00:00:00Z Content type: tutorial Language: en Sources: [Teleport]() Topics: [Kubernetes](), [Tutorial](), [Authentication](), [Authorization]() Tags: [access-control](), [api](), [api-server](), [authentication](), [authn](), [authorization](), [authz](), [cli](), [crud](), [devops](), [kubernetes](), [permissions](), [scheduler](), [security](), [tutorial]() ### AI overview This tutorial explains the difference between authentication and authorization in Kubernetes. It describes how the Kubernetes API server validates the identity of users or other entities and checks whether they have permission to perform requested operations. ### Source excerpt The difference between who gets to access (AuthN) a Kubernetes cluster and what they can do with those permissions (AuthZ) ## How etcd works with and without Kubernetes DevFeed: [How etcd works with and without Kubernetes]() Original publisher: [Read original article]() Author: Emanuel Evans Published: 2021-07-21T00:00:00Z Content type: tutorial Language: en Sources: [Learnk8s]() Topics: [Kubernetes](), [Database](), [API](), [Processes]() Tags: [api](), [authentication](), [authorization](), [database](), [deep-dive](), [deployment](), [distributed](), [key-value-store](), [kubernetes](), [process](), [scheduler](), [server]() ### AI overview A deep dive into etcd as Kubernetes' database, explaining how it fits into the control plane and how to build, configure, and troubleshoot a 3-node etcd cluster. ### Source excerpt Deep dive into etcd distributed key-value store and how Kubernetes uses it as its database. Learn to build, configure, and troubleshoot a 3-node etcd cluster with practical examples. ## Unit Tests and Concurrency DevFeed: [Unit Tests and Concurrency]() Original publisher: [Read original article]() Author: Stojan Anastasov Published: 2021-01-06T00:00:00Z Content type: tutorial Language: en Sources: [Stojan Anastasov's blog]() Topics: [RxJava](), [Concurrency](), [Android](), [Testing]() Tags: [android](), [api](), [concurrency](), [junit](), [observeon](), [recyclerview](), [rxjava](), [scheduler](), [testing](), [tests](), [trampoline]() ### AI overview This tutorial explains how replacing RxJava schedulers with a single test scheduler can hide concurrency problems. It presents an Android example involving concurrent API calls and discusses refactoring the RxJava chain to update the UI as results become available. ### Source excerpt Once Retrofit added RxJava support, RxJava became my go-to concurrency framework for writing Android apps. One of the great things about RxJava is the excellent testing support. It includes TestObserver, TestScheduler, RxJavaPlugins so you can switch your schedulers in tests. A common approach in testing RxJava code is using a JUnit rule that replaces the Scheduler pools with Schedulers.trampoline() before tests are run and resets them to the original thread pools after the tests. This makes the whole Observable chain runs on a single thread, the same thread the test runs on, which means we can write assertions without worrying about concurrency. However the production code usually is not single threaded. IO operations are done on the IO thread pool, views are updated on the main thread and everything else happens on the computation pool. By using different schedulers in the tests and using a different strategy (single threaded) we make those unit tests useless in catching concurrency issues. A real world scenario I was working on a side project. The screen consists of a RecyclerView displaying a list of elements. To get the elements I need to perform two different API calls. The first API call returns a list with N elements, then for each item in the list I need to perform the second call. After combining the data I send it to the UI for displaying. Using RxJava this looks like: // Emits Loading then Content or Problem private fun requestData(): Observable = service.firstApiCall() .observeOn(Schedulers.computation()) .map { it.message } .flatMap(this::secondApiCall) .map { ViewState.Content(it) } .startWith(Single.just(ViewState.Loading)) .onErrorReturn { ViewState.Problem } .toObservable() // Concurrently executes secondApiCall for each element in list. // Transforms the result to ViewEntity, combines everything in a list private fun secondApiCall(list: List): Single> = Observable.fromIterable(list) .concatMapEager { ## Apache Airflow 2.0 is here! DevFeed: [Apache Airflow 2.0 is here!]() Original publisher: [Read original article]() Author: Apache Airflow Published: 2020-12-17T00:00:00Z Content type: release Language: en Sources: [Apache Airflow Blog]() Topics: [airflow](), [API](), [REST API](), [OpenAPI Specification](), [resiliency]() Tags: [airflow](), [announce](), [apache-airflow](), [api](), [api-documentation](), [ha](), [improvements](), [openapi](), [release](), [rest-api](), [scheduler]() ### AI overview Apache Airflow 2.0.0 has been released with the TaskFlow API for authoring DAGs, a fully supported REST API with an OpenAPI specification, scheduler performance improvements, support for multiple highly available schedulers, and Task Groups for organizing tasks. ### Source excerpt I am proud to announce that Apache Airflow 2.0.0 has been released. The full changelog is about 3,000 lines long (already excluding everything backported to 1.10), so for now I'll simply share some of the major features in 2.0.0 compared to 1.10.14: A new way of writing dags: the TaskFlow API (AIP-31) (Known in 2.0.0alphas as Functional DAGs.) DAGs are now much much nicer to author especially when using PythonOperator. Dependencies are handled more clearly and XCom is nicer to use Read more here: TaskFlow API Tutorial TaskFlow API Documentation A quick teaser of what DAGs can now look like: from airflow.decorators import dag, task from airflow.utils.dates import days_ago @dag(default_args={'owner': 'airflow'}, schedule_interval=None, start_date=days_ago(2)) def tutorial_taskflow_api_etl(): @task def extract(): return {"1001": 301.27, "1002": 433.21, "1003": 502.22} @task def transform(order_data_dict: dict) -> dict: total_order_value = 0 for value in order_data_dict.values(): total_order_value += value return {"total_order_value": total_order_value} @task() def load(total_order_value: float): print("Total order value is: %.2f" % total_order_value) order_data = extract() order_summary = transform(order_data) load(order_summary["total_order_value"]) tutorial_etl_dag = tutorial_taskflow_api_etl() Fully specified REST API (AIP-32) We now have a fully supported, no-longer-experimental API with a comprehensive OpenAPI specification Read more here: REST API Documentation. Massive Scheduler performance improvements As part of AIP-15 (Scheduler HA+performance) and other work Kamil did, we significantly improved the performance of the Airflow Scheduler. It now starts tasks much, MUCH quicker. Over at Astronomer.io we've benchmarked the scheduler--it's fast (we had to triple check the numbers as we don't quite believe them at first!) Scheduler is now HA compatible (AIP-15) It's now possible and supported to run more than a single scheduler instance. This is super useful for bot ## Apache Airflow For Newcomers DevFeed: [Apache Airflow For Newcomers]() Original publisher: [Read original article]() Author: Apache Airflow Published: 2020-08-17T00:00:00Z Content type: tutorial Language: en Sources: [Apache Airflow Blog]() Topics: [airflow](), [Development]() Tags: [airflow](), [apache-airflow](), [community](), [dependencies](), [python](), [scheduler](), [scheduling](), [tasks](), [workflows]() ### AI overview This tutorial introduces Apache Airflow for newcomers, explaining how to author workflows with Python scripts, represent them as DAGs, define task dependencies, and use the scheduler to execute tasks when dependencies are met. ### Source excerpt Apache Airflow is a platform to programmatically author, schedule, and monitor workflows. A workflow is a sequence of tasks that processes a set of data. You can think of workflow as the path that describes how tasks go from being undone to done. Scheduling, on the other hand, is the process of planning, controlling, and optimizing when a particular task should be done. Authoring Workflow in Apache Airflow. Airflow makes it easy to author workflows using python scripts. A Directed Acyclic Graph (DAG) represents a workflow in Airflow. It is a collection of tasks in a way that shows each task's relationships and dependencies. You can have as many DAGs as you want, and Airflow will execute them according to the task's relationships and dependencies. If task B depends on the successful execution of another task A, it means Airflow will run task A and only run task B after task A. This dependency is very easy to express in Airflow. For example, the above scenario is expressed as task_A >> task_B Also equivalent to task_A.set_downstream(task_B) That helps Airflow to know that it needs to execute task A before task B. Tasks can have far more complex relationships to each other than expressed above and Airflow figures out how and when to execute the tasks following their relationships and dependencies. Before we discuss the architecture of Airflow that makes scheduling, executing, and monitoring of workflow an easy thing, let us discuss the Breeze environment. Breeze Environment The breeze environment is the development environment for Airflow where you can run tests, build images, build documentations and so many other things. There are excellent documentation and video on Breeze environment. Please check them out. You enter the Breeze environment by running the ./breeze script. You can run all the commands mentioned here in the Breeze environment. Scheduler The scheduler is the component that monitors DAGs and triggers those tasks whose dependencies have been met. It watc [Next page]()