# distributed-computing

Published articles for distributed-computing.

This is one page of public article previews, not the complete archive. Follow Next page to continue. Summaries are not the original full articles.

## A Survey on Cloud-Edge-Terminal Collaborative Intelligence in AIoT Networks

DevFeed: [A Survey on Cloud-Edge-Terminal Collaborative Intelligence in AIoT Networks](<https://devfeed.tech/articles/a-survey-on-cloud-edge-terminal-collaborative-intelligence-in-aiot-networks-137008.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2508.18803>)

Author: Jiaqi Wu, Jing Liu, Yang Liu, Lixu Wang, Zehua Wang, Wei Chen, Zijian Tian, Richard Yu, Victor C. M. Leung

Published: 2026-11-17T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Networking](<https://devfeed.tech/sources/arxiv-networking.md>)

Topics: [Artificial Intelligence](<https://devfeed.tech/topics/ai.md>), [Internet of things](<https://devfeed.tech/topics/iot.md>), [Cloud Architecture](<https://devfeed.tech/topics/cloud-architecture.md>), [Cloud Networking](<https://devfeed.tech/topics/cloud-networking.md>), [virtualization](<https://devfeed.tech/topics/virtualization.md>), [AI Research](<https://devfeed.tech/topics/ai-research.md>)

Tags: [agents](<https://devfeed.tech/tags/agents.md>), [ai](<https://devfeed.tech/tags/ai.md>), [aiot](<https://devfeed.tech/tags/aiot.md>), [architectures](<https://devfeed.tech/tags/architectures.md>), [challenges](<https://devfeed.tech/tags/challenges.md>), [cloud](<https://devfeed.tech/tags/cloud.md>), [collaboration](<https://devfeed.tech/tags/collaboration.md>), [communication](<https://devfeed.tech/tags/communication.md>), [computing](<https://devfeed.tech/tags/computing.md>), [container-orchestration](<https://devfeed.tech/tags/container-orchestration.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-ni](<https://devfeed.tech/tags/cs-ni.md>), [deep-learning](<https://devfeed.tech/tags/deep-learning.md>), [digital-twin](<https://devfeed.tech/tags/digital-twin.md>), [distributed](<https://devfeed.tech/tags/distributed.md>), [distributed-computing](<https://devfeed.tech/tags/distributed-computing.md>), [edge](<https://devfeed.tech/tags/edge.md>), [edge-cloud](<https://devfeed.tech/tags/edge-cloud.md>), [edge-computing](<https://devfeed.tech/tags/edge-computing.md>), [federated-learning](<https://devfeed.tech/tags/federated-learning.md>), [networks](<https://devfeed.tech/tags/networks.md>), [survey](<https://devfeed.tech/tags/survey.md>)

### AI overview

The survey presents cloud-edge-terminal collaborative intelligence (CETCI) as a way to build deployable collaborative AIoT systems by coordinating computing and communication across cloud, edge, and terminal layers. It reviews enabling infrastructure such as network virtualization, container orchestration, and software-defined networking; collaboration approaches for task offloading, resource allocation, and optimization across heterogeneous systems; and learning frameworks including federated and distributed deep learning, edge-cloud model evolution, and reinforcement learning. It identifies scalability, heterogeneity, and interoperability as challenges, and points to 6G+, agents, quantum computing, and digital twins as future directions. This overview is based on the supplied abstract.

### Source excerpt

arXiv:2508.18803v2 Announce Type: replace Abstract: The proliferation of Internet of things (IoT) devices in smart cities, transportation, healthcare, and industrial applications, coupled with the explosive growth of AI-driven services, has increased demands for efficient distributed computing architectures and networks, driving cloud-edge-terminal collaborative intelligence (CETCI) as a fundamental paradigm within the artificial intelligence of things (AIoT) community. With advancements in deep learning, large language models (LLMs), and edge computing, CETCI has made significant progress with emerging AIoT applications, moving beyond isolated layer optimization to deployable collaborative intelligence systems for AIoT (CISAIOT), a practical research focus in AI, distributed computing, and communications. This survey describes foundational architectures, enabling technologies, and scenarios of CETCI paradigms, offering a tutorial-style review for CISAIOT beginners. We systematically analyze architectural components spanning cloud, edge, and terminal layers, examining core technologies including network virtualization, container orchestration, and software-defined networking, while presenting categorizations of collaboration paradigms that cover task offloading, resource allocation, and optimization across heterogeneous infrastructures. Furthermore, we explain intelligent collaboration learning frameworks by reviewing advances in federated learning, distributed deep learning, edge-cloud model evolution, and reinforcement learning-based methods. Finally, we discuss challenges (e.g., scalability, heterogeneity, interoperability) and future trends (e.g., 6G+, agents, quantum computing, digital twin), highlighting how integration of distributed computing and communication can address open issues and guide development of robust, efficient, and secure collaborative AIoT systems.

## HistogramTools for Efficient Data Analysis and Distribution Representation in Large Data Sets

DevFeed: [HistogramTools for Efficient Data Analysis and Distribution Representation in Large Data Sets](<https://devfeed.tech/articles/histogramtools-for-efficient-data-analysis-and-distribution-representation-in-large-data-sets-132665.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2504.00001>)

Author: Shubham Malhotra

Published: 2026-11-13T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Databases](<https://devfeed.tech/sources/arxiv-databases.md>)

Topics: [Data analysis](<https://devfeed.tech/topics/data-analysis.md>), [big-data](<https://devfeed.tech/topics/big-data.md>), [Data science and engineering](<https://devfeed.tech/topics/data-science-and-engineering.md>), [Data visualization](<https://devfeed.tech/topics/data-visualization.md>), [Apache Spark](<https://devfeed.tech/topics/spark.md>)

Tags: [analysis](<https://devfeed.tech/tags/analysis.md>), [applications](<https://devfeed.tech/tags/applications.md>), [approximation](<https://devfeed.tech/tags/approximation.md>), [cs-db](<https://devfeed.tech/tags/cs-db.md>), [cs-pf](<https://devfeed.tech/tags/cs-pf.md>), [data](<https://devfeed.tech/tags/data.md>), [data-analysis](<https://devfeed.tech/tags/data-analysis.md>), [data-processing](<https://devfeed.tech/tags/data-processing.md>), [distributed-computing](<https://devfeed.tech/tags/distributed-computing.md>), [distribution](<https://devfeed.tech/tags/distribution.md>), [error](<https://devfeed.tech/tags/error.md>), [manipulation](<https://devfeed.tech/tags/manipulation.md>), [mapreduce](<https://devfeed.tech/tags/mapreduce.md>), [protocol](<https://devfeed.tech/tags/protocol.md>), [r](<https://devfeed.tech/tags/r.md>), [storage](<https://devfeed.tech/tags/storage.md>), [visualization](<https://devfeed.tech/tags/visualization.md>)

### AI overview

HistogramTools extends R's built-in histogram features for large-scale and distributed data analysis. It supports serializing histograms with Protocol Buffers, merging and modifying them, and manipulating bins. The abstract also describes distance measures, quantile approximation, and methods to estimate and visualize information loss, including error in CDFs derived from histograms. The package is presented as suited to MapReduce settings where compact storage and data sharing matter; the supplied text does not provide performance results or implementation details.

### Source excerpt

arXiv:2504.00001v2 Announce Type: replace Abstract: Histograms provide a powerful means of summarizing large data sets by representing their distribution in a compact, binned form. The HistogramTools R package enhances R built-in histogram functionality, offering advanced methods for manipulating and analyzing histograms, especially in large-scale data environments. Key features include the ability to serialize histograms using Protocol Buffers for distributed computing tasks, tools for merging and modifying histograms, and techniques for measuring and visualizing information loss in histogram representations. The package is particularly suited for environments utilizing MapReduce, where efficient storage and data sharing are critical. This paper presents various methods of histogram bin manipulation, distance measures, quantile approximation, and error estimation in cumulative distribution functions (CDFs) derived from histograms. Visualization techniques and efficient storage representations are also discussed alongside applications for large data processing and distributed computing tasks.

## Resetting-based designated information distribution

DevFeed: [Resetting-based designated information distribution](<https://devfeed.tech/articles/resetting-based-designated-information-distribution-126746.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.12485>)

Author: Xiaojuan Ma, Yan Wang, Indranil Gupta

Published: 2026-11-10T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Distributed Computing](<https://devfeed.tech/sources/arxiv-distributed-computing.md>)

Topics: [Network](<https://devfeed.tech/topics/network.md>), [Electrical engineering and computer science (EECS)](<https://devfeed.tech/topics/electrical-engineering-and-computer-science-eecs.md>), [Quantum Computing](<https://devfeed.tech/topics/quantum-computing.md>)

Tags: [cond-mat-stat-mech](<https://devfeed.tech/tags/cond-mat-stat-mech.md>), [confidence-interval](<https://devfeed.tech/tags/confidence-interval.md>), [control](<https://devfeed.tech/tags/control.md>), [convergence](<https://devfeed.tech/tags/convergence.md>), [cs-dc](<https://devfeed.tech/tags/cs-dc.md>), [cs-si](<https://devfeed.tech/tags/cs-si.md>), [distributed-computing](<https://devfeed.tech/tags/distributed-computing.md>), [networks](<https://devfeed.tech/tags/networks.md>), [protocol](<https://devfeed.tech/tags/protocol.md>), [quantum-algorithms](<https://devfeed.tech/tags/quantum-algorithms.md>)

### AI overview

With resetting at rate β on a designated subset of r nodes in a random gossip network of n nodes, the expected total node value approaches (n/r) times the subset's initial total in the small-β, long-time limit. The work analyzes convergence speed and gives an approximate high-probability confidence interval for finite time and β. It also proposes switching resetting on and off to distribute initial values evenly across the designated nodes, up to a prescribed precision. The abstract presents local-operation quantum state preparation as a potential application; it does not provide further details here.

### Source excerpt

arXiv:2611.12485v1 Announce Type: new Abstract: For a network of $n$ nodes under random gossip dynamics, we show that by introducing the additional resetting mechanism (with rate $\beta$) to a designated subset of $r$ nodes, the expected sum of all node values is asymptotically equal to $\tfrac{n}{r}$ times the sum of the initial values of the $r$ designated nodes, in the small-$\beta$ and long-time limit. We also analyze the convergence speed to this asymptotic state and provide an approximate high-probability confidence interval for the expected sum at finite time $t$ and finite $\beta$. Operationally, we propose a protocol that distributes the initial node values over the designated subset evenly (up to a prescribed precision), by appropriately turning on and off the resetting dynamics. Our work not only offers insights into classical problems such as distributed computing and network control, but also serves as a potential foundation for designing quantum algorithms that prepare desired states in quantum networks using only local operations.

## PyTorch 2.15 tackles the 'one bad GPU' problem

DevFeed: [PyTorch 2.15 tackles the 'one bad GPU' problem](<https://devfeed.tech/articles/pytorch-2-15-tackles-the-one-bad-gpu-problem-122495.md>)

Original publisher: [Read original article](<https://www.devclass.com/devops/2026/11/07/pytorch-215-tackles-the-one-bad-gpu-problem/5308234>)

Author: Joab Jackson

Published: 2026-11-07T18:38:00Z

Content type: news

Language: en

Sources: [DevClass.com](<https://devfeed.tech/sources/devclass-com.md>)

Topics: [PyTorch](<https://devfeed.tech/topics/pytorch.md>), [NCCL](<https://devfeed.tech/topics/nccl.md>), [Multi-GPU](<https://devfeed.tech/topics/multi-gpu.md>), [Machine learning](<https://devfeed.tech/topics/machine-learning.md>)

Tags: [ai](<https://devfeed.tech/tags/ai.md>), [api](<https://devfeed.tech/tags/api.md>), [apple](<https://devfeed.tech/tags/apple.md>), [backend](<https://devfeed.tech/tags/backend.md>), [blackwell](<https://devfeed.tech/tags/blackwell.md>), [bottlenecks](<https://devfeed.tech/tags/bottlenecks.md>), [cluster](<https://devfeed.tech/tags/cluster.md>), [clusters](<https://devfeed.tech/tags/clusters.md>), [communication](<https://devfeed.tech/tags/communication.md>), [communications](<https://devfeed.tech/tags/communications.md>), [computing](<https://devfeed.tech/tags/computing.md>), [data-movement](<https://devfeed.tech/tags/data-movement.md>), [deadlocks](<https://devfeed.tech/tags/deadlocks.md>), [devops](<https://devfeed.tech/tags/devops.md>), [distributed](<https://devfeed.tech/tags/distributed.md>), [distributed-computing](<https://devfeed.tech/tags/distributed-computing.md>), [gpu](<https://devfeed.tech/tags/gpu.md>), [job](<https://devfeed.tech/tags/job.md>), [ml](<https://devfeed.tech/tags/ml.md>), [nccl](<https://devfeed.tech/tags/nccl.md>), [nvidia](<https://devfeed.tech/tags/nvidia.md>), [pytorch](<https://devfeed.tech/tags/pytorch.md>), [training](<https://devfeed.tech/tags/training.md>)

### AI overview

PyTorch 2.15 rewrites its NCCL backend to improve diagnosis and recovery in multi-GPU jobs. The former ProcessGroupNCCL could create many communicators per process group, contributing to intermittent deadlocks and unpredictable performance; the new backend limits each group to one communicator at initialization, so programs needing parallelism must dispatch more groups. It also supports easier reconfiguration when a GPU is slow or fails: a third-party orchestrator can exclude it and assign a replacement worker. A new flight recorder stores operation snapshots for inspection after failures. The legacy backend remains available as a rollback in 2.15 and is slated for removal in 2.16 if the replacement works as expected. Other reported additions include an asynchronous one-way Python transport API and parallel-processing improvements for Apple hardware and Nvidia Blackwell GPUs.

### Source excerpt

ML admins will be happy to learn that one bad GPU may no longer mean starting an entire training job ...

## Decentralized AI Inference Needs a Boring Job Queue

DevFeed: [Decentralized AI Inference Needs a Boring Job Queue](<https://devfeed.tech/articles/decentralized-ai-inference-needs-a-boring-job-queue-121014.md>)

Original publisher: [Read original article](<https://hackernoon.com/decentralized-ai-inference-needs-a-boring-job-queue?source=rss>)

Author: William Onyejiaka

Published: 2026-11-06T08:33:30Z

Content type: article

Language: en

Sources: [HackerNoon](<https://devfeed.tech/sources/hackernoon.md>)

Topics: [AI Infrastructure](<https://devfeed.tech/topics/ai-infrastructure.md>), [Job](<https://devfeed.tech/topics/job.md>), [AI Architecture](<https://devfeed.tech/topics/ai-architecture.md>), [DePIN](<https://devfeed.tech/topics/depin.md>), [Artificial Intelligence](<https://devfeed.tech/topics/ai.md>)

Tags: [ai](<https://devfeed.tech/tags/ai.md>), [ai-inference](<https://devfeed.tech/tags/ai-inference.md>), [ai-infrastructure](<https://devfeed.tech/tags/ai-infrastructure.md>), [ai-jobs](<https://devfeed.tech/tags/ai-jobs.md>), [backend](<https://devfeed.tech/tags/backend.md>), [bill](<https://devfeed.tech/tags/bill.md>), [bullmq](<https://devfeed.tech/tags/bullmq.md>), [decentralize-ai-hackathon](<https://devfeed.tech/tags/decentralize-ai-hackathon.md>), [decentralized](<https://devfeed.tech/tags/decentralized.md>), [decentralized-ai](<https://devfeed.tech/tags/decentralized-ai.md>), [distributed-computing](<https://devfeed.tech/tags/distributed-computing.md>), [duplicate-billing](<https://devfeed.tech/tags/duplicate-billing.md>), [engineer](<https://devfeed.tech/tags/engineer.md>), [fail](<https://devfeed.tech/tags/fail.md>), [fault-tolerant-systems](<https://devfeed.tech/tags/fault-tolerant-systems.md>), [gpu](<https://devfeed.tech/tags/gpu.md>), [gpu-inference](<https://devfeed.tech/tags/gpu-inference.md>), [idempotency](<https://devfeed.tech/tags/idempotency.md>), [inference](<https://devfeed.tech/tags/inference.md>), [inference-api](<https://devfeed.tech/tags/inference-api.md>), [job](<https://devfeed.tech/tags/job.md>), [jobs](<https://devfeed.tech/tags/jobs.md>), [queue](<https://devfeed.tech/tags/queue.md>), [queues](<https://devfeed.tech/tags/queues.md>), [remote](<https://devfeed.tech/tags/remote.md>), [retries](<https://devfeed.tech/tags/retries.md>), [software-engineering](<https://devfeed.tech/tags/software-engineering.md>)

### AI overview

The article proposes a reliable backend path for decentralized GPU inference, centered on job queues, persistent job records, retries, and idempotency. An Express.js and TypeScript API would return a job ID; BullMQ and Redis would coordinate work, while PostgreSQL preserves job and execution history. A provider adapter would normalize common operations, starting with one provider such as Nosana while retaining provider-specific differences. On timeouts, the system should check execution status and record uncertainty rather than blindly resubmit potentially billable work. Idempotency keys and billing safeguards can prevent duplicate logical jobs and records, but cannot guarantee a provider executes only once. Results need basic validation; hashes show file integrity, not answer correctness. This is an unbenchmarked proposal: the author plans to test duplicate submissions, worker crashes, recovery, and added latency. Multi-provider routing, escrow, permanent storage, and advanced verification are deferred.

### Source excerpt

A backend engineer's proposal for reliable decentralized GPU inference using job queues, retries, and idempotency, so remote AI jobs don't fail or bill twice.

## Scaling Model Evaluation at Pinterest with Ray (Part 4)

DevFeed: [Scaling Model Evaluation at Pinterest with Ray (Part 4)](<https://devfeed.tech/articles/scaling-model-evaluation-at-pinterest-with-ray-part-4-118874.md>)

Original publisher: [Read original article](<https://medium.com/pinterest-engineering/scaling-model-evaluation-at-pinterest-with-ray-part-4-4a96cd12cc11?source=rss----4c5a5f6279b6---4>)

Author: Pinterest Engineering

Published: 2026-11-04T16:01:04Z

Content type: article

Language: en

Sources: [Pinterest](<https://devfeed.tech/sources/pinterest-engineering-blog-medium.md>)

Topics: [AI Infrastructure](<https://devfeed.tech/topics/ai-infrastructure.md>), [Apache Spark](<https://devfeed.tech/topics/spark.md>), [Advanced AI workflows](<https://devfeed.tech/topics/advanced-ai-workflows.md>), [Inference](<https://devfeed.tech/topics/inference.md>), [AI Integration Strategies](<https://devfeed.tech/topics/ai-integration-strategies.md>)

Tags: [distributed-computing](<https://devfeed.tech/tags/distributed-computing.md>), [engineering](<https://devfeed.tech/tags/engineering.md>), [evaluation](<https://devfeed.tech/tags/evaluation.md>), [machine-learning](<https://devfeed.tech/tags/machine-learning.md>), [model-evaluation](<https://devfeed.tech/tags/model-evaluation.md>), [open-source](<https://devfeed.tech/tags/open-source.md>), [pinterest](<https://devfeed.tech/tags/pinterest.md>), [ray](<https://devfeed.tech/tags/ray.md>), [scaling](<https://devfeed.tech/tags/scaling.md>)

### AI overview

Pinterest describes applying Ray to model evaluation in feature importance workflows, combining Ray with Apache Spark to compute and aggregate metrics across large inference datasets. Accumulators introduced in its batch inference system support metrics such as ROC-AUC and cross entropy, calculated for each experiment and model head. The post says the platform overlaps metric computation with inference, amortizing the work and reducing added runtime cost; it reports up to a 32% reduction in end-to-end evaluation time while scaling linearly as experiments increase. The supplied text is an excerpt and does not include the implementation details behind these results.

### Source excerpt

Vishnu Arun | Tech Lead, Feature Importance Platform; Anderson Lam | Sr Software Engineer; Andrew Qiu | Machine Learning Engineer I; Helena Xu | Data Scientist I; Tianyou Wang | Data Scientist II Introduction In our multi-part blog series, we have discussed why we chose Ray as a last mile data processing framework, how we integrated Ray into our existing ML Training Infrastructure, and how we enabled efficient Batch Inference at scale. In this blog post, we will discuss how we were able to apply Ray to another critical part of our ML Infrastructure stack: Model Evaluation. In the Ray Batch Inference Blog Post, we ended on a section on accumulators (counters that would allow us to calculate metrics like ROC-AUC and cross entropy). This post is where those accumulators pay off: a model evaluation platform that combines the strengths of Ray and Apache Spark to compute metrics efficiently at scale. Model evaluation may appear to be a relatively straightforward step after inference or training. But as ML workflows scale, evaluation quickly becomes an infrastructure challenge. Systems that run many experiments in parallel, such as feature importance, must efficiently compute and aggregate metrics over large volumes of inference data, as shown in Diagram 1. Later in this post, we'll show how we amortize the metrics compute step, overlapping it with inference for minimal added runtime cost, cutting end-to-end evaluation time by up to 32% while scaling linearly as we run more experiments. Diagram 1: Depicts a feature importance system's architecture. We illustrate a multi-head model with h heads and metrics calculated per head. In this type of workflow, metrics are calculated for each experiment and model head. Note: We omit a batched view of this data found in large scale production ML systems for simplicity. Terminology Feature Importance (FI): A technique for quantifying how much each input feature contributes to a model's predictions. In this post, we will reference a pe

## Throughput-Optimal Control of Distributed Computing Networks for Service DAGs

DevFeed: [Throughput-Optimal Control of Distributed Computing Networks for Service DAGs](<https://devfeed.tech/articles/throughput-optimal-control-of-distributed-computing-networks-for-service-dags-116526.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2611.05007>)

Author: Peng Li, Jianan Zhang, Jaime Llorca, Xinzhe Fu, Xiang Cheng, Chen Chen

Published: 2026-11-04T05:00:00Z

Content type: article

Language: en

Sources: [arXiv: Networking](<https://devfeed.tech/sources/arxiv-networking.md>)

Topics: [Distributed Systems & Parallel Computing](<https://devfeed.tech/topics/distributed-systems-parallel-computing.md>), [scheduling](<https://devfeed.tech/topics/scheduling.md>), [queueing theory](<https://devfeed.tech/topics/queueing-theory.md>), [Combinatorial optimization](<https://devfeed.tech/topics/combinatorial-optimization.md>)

Tags: [ai](<https://devfeed.tech/tags/ai.md>), [cs-ni](<https://devfeed.tech/tags/cs-ni.md>), [dag](<https://devfeed.tech/tags/dag.md>), [distributed-computing](<https://devfeed.tech/tags/distributed-computing.md>), [networks](<https://devfeed.tech/tags/networks.md>), [optimal-control](<https://devfeed.tech/tags/optimal-control.md>), [throughput](<https://devfeed.tech/tags/throughput.md>), [wireless](<https://devfeed.tech/tags/wireless.md>), [workflow](<https://devfeed.tech/tags/workflow.md>)

### AI overview

The paper presents DCNC-DAG for executing service DAGs over wireless computing networks with stochastic arrivals, changing links, and interference. It maps functions to computation nodes and dependencies to routes, then uses queue-aware mappings and nearest-to-origin scheduling while synchronizing request inputs at merge points. The authors state that synchronization does not reduce the capacity region. With exact minimum-cost mappings, the policy is throughput-optimal and keeps physical queues rate-stable under bounded arrivals, even with finite mapping-update intervals and boundedly stale queue information. For larger DAGs, DCNC-DAG-H uses a queue-aware heuristic instead of exact optimization. Simulations report 2.7-5.5 times the throughput of queue-blind baselines for DCNC-DAG and up to 2.68 times that of corresponding baselines for DCNC-DAG-H.

### Source excerpt

arXiv:2611.05007v1 Announce Type: new Abstract: Executing service workflows with directed acyclic graph (DAG) dependencies over wireless distributed computing networks requires joint control of function placement, routing, and scheduling, together with synchronization of inputs from the same request at merge points. We study throughput-optimal control of these networks under stochastic request arrivals, time-varying wireless links, and interference constraints. We use a valid mapping abstraction that assigns each service function to a computation node and each DAG dependency to a routing path. Using this abstraction, we characterize the network capacity region to support service DAGs, and prove that synchronizing the packets of a request at merge points does not shrink the region. We develop Dynamic Computing Network Control for service DAGs (DCNC-DAG), a dynamic control policy that selects queue-aware valid mappings and schedules service DAG execution by nearest-to-origin (NTO) while enforcing input synchronization at merge points. With exact minimum-cost valid mappings, DCNC-DAG is throughput-optimal and ensures rate stability of the physical queues under bounded arrivals. These guarantees hold for any fixed finite interval between mapping updates, even when mapping decisions rely on boundedly stale queue length information. For larger service DAGs, we develop DCNC-DAG-H, which replaces exact mapping optimization with a queue-aware heuristic. Simulations show that DCNC-DAG supports $2.7\times$-$5.5\times$ the throughput of queue-blind baselines, while DCNC-DAG-H achieves up to $2.68\times$ those of the corresponding baselines on larger workflow and AI service DAGs.

## rush: Scalable Asynchronous Distributed Computing via Shared State in R

DevFeed: [rush: Scalable Asynchronous Distributed Computing via Shared State in R](<https://devfeed.tech/articles/rush-scalable-asynchronous-distributed-computing-via-shared-state-in-r-101581.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2606.21430>)

Author: Marc Becker, Bernd Bischl

Published: 2026-10-27T04:00:00Z

Content type: article

Language: en

Sources: [arXiv: Distributed Computing](<https://devfeed.tech/sources/arxiv-distributed-computing.md>)

Topics: [distributed algorithm](<https://devfeed.tech/topics/distributed-algorithm.md>), [Hyperparameter optimization](<https://devfeed.tech/topics/hyperparameter-optimization.md>), [shared mutable state](<https://devfeed.tech/topics/shared-mutable-state.md>)

Tags: [algorithms](<https://devfeed.tech/tags/algorithms.md>), [api](<https://devfeed.tech/tags/api.md>), [asynchronous](<https://devfeed.tech/tags/asynchronous.md>), [caching](<https://devfeed.tech/tags/caching.md>), [cs-dc](<https://devfeed.tech/tags/cs-dc.md>), [distributed-computing](<https://devfeed.tech/tags/distributed-computing.md>), [error-handling](<https://devfeed.tech/tags/error-handling.md>), [hyperparameter-optimization](<https://devfeed.tech/tags/hyperparameter-optimization.md>), [machine-learning](<https://devfeed.tech/tags/machine-learning.md>), [parallel-computing](<https://devfeed.tech/tags/parallel-computing.md>), [r](<https://devfeed.tech/tags/r.md>), [redis](<https://devfeed.tech/tags/redis.md>), [state](<https://devfeed.tech/tags/state.md>)

### AI overview

rush provides R with shared-state coordination for asynchronous, decentralized iterative algorithms. Workers use a Redis key-value store to read and write task data while running independent loops, avoiding a central controller-worker pattern. Its API manages task lifecycles and is described as offering sub-millisecond per-task overhead, automatic detection of lost workers, error handling, and caching. Optional integration with mlr3 supports asynchronous optimization through bbotk and mlr3tuning. The authors demonstrate asynchronous decentralized Bayesian optimization for LightGBM hyperparameter tuning across four datasets with 448 workers; the supplied abstract does not report benchmark outcomes.

### Source excerpt

arXiv:2606.21430v4 Announce Type: replace Abstract: Many algorithms in statistics and machine learning can be parallelized in an asynchronous manner where workers need to communicate through shared state rather than execute independent tasks dispatched by a central controller. Especially in modern hyperparameter optimization and parallel black-box optimization with expensive objectives, this decentralized approach has become widespread, and several Python frameworks adopt it (e.g., Optuna, DeepHyper, and Hyperopt). However, all popular R packages for parallel computing follow a centralized controller-worker architecture that does not support this pattern. We present rush, an R package that provides a shared-state coordination layer for asynchronously parallelized iterative algorithms. rush uses a Redis database as a shared key-value store: workers read and write task data through the database and independently execute their own loops. The package provides a high-level API for managing tasks and their lifecycle, featuring sub-millisecond per-task overhead, robust error handling with automatic detection of lost workers, and efficient caching. rush optionally integrates with the mlr3 ecosystem, powering asynchronous optimization in the bbotk and mlr3tuning packages. We demonstrate the practical utility of rush by implementing asynchronous decentralized Bayesian optimization (ADBO) and benchmarking it on hyperparameter optimization of LightGBM across four datasets using 448 workers.

## The End of Consensus Research?

DevFeed: [The End of Consensus Research?](<https://devfeed.tech/articles/the-end-of-consensus-research-95173.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2610.30114>)

Author: Cl\\'ement Burgelin, Rachid Guerraoui, Zicong Hong, Yinan Hu, Anne-Marie Kermarrec, Rafael Pires, Rasmus Moorits Veski, Martijn de Vos

Published: 2026-10-22T04:00:00Z

Content type: article

Language: en

Sources: [arXiv: Distributed Computing](<https://devfeed.tech/sources/arxiv-distributed-computing.md>)

Topics: [consensus algorithm](<https://devfeed.tech/topics/consensus-algorithm.md>), [Agent Harness](<https://devfeed.tech/topics/agent-harness.md>), [consistency](<https://devfeed.tech/topics/consistency.md>), [Multi Agent Systems](<https://devfeed.tech/topics/multi-agent-systems.md>), [vulnerability-research](<https://devfeed.tech/topics/vulnerability-research.md>)

Tags: [agent](<https://devfeed.tech/tags/agent.md>), [agent-harness](<https://devfeed.tech/tags/agent-harness.md>), [ai](<https://devfeed.tech/tags/ai.md>), [autonomous](<https://devfeed.tech/tags/autonomous.md>), [consensus](<https://devfeed.tech/tags/consensus.md>), [cs-ai](<https://devfeed.tech/tags/cs-ai.md>), [cs-dc](<https://devfeed.tech/tags/cs-dc.md>), [discovery](<https://devfeed.tech/tags/discovery.md>), [distributed-computing](<https://devfeed.tech/tags/distributed-computing.md>), [experimentation](<https://devfeed.tech/tags/experimentation.md>), [framework](<https://devfeed.tech/tags/framework.md>), [implementation](<https://devfeed.tech/tags/implementation.md>), [key-value-store](<https://devfeed.tech/tags/key-value-store.md>), [knowledge-base](<https://devfeed.tech/tags/knowledge-base.md>), [latency](<https://devfeed.tech/tags/latency.md>), [leader](<https://devfeed.tech/tags/leader.md>), [lean](<https://devfeed.tech/tags/lean.md>), [lease](<https://devfeed.tech/tags/lease.md>), [local](<https://devfeed.tech/tags/local.md>), [low-latency](<https://devfeed.tech/tags/low-latency.md>), [modular](<https://devfeed.tech/tags/modular.md>), [research](<https://devfeed.tech/tags/research.md>)

### AI overview

AI helped discover A-greement, a consensus protocol for geo-replicated state machines, but the authors argue that human experts remain essential to consensus research. Across six deployments with 5 or 9 replicas, it reduced mean latency over the fastest baseline by 12.2% on write-heavy workloads and 33.3% on read-heavy workloads, while matching the baseline on read-only workloads. Its paths combine lease-authorized local reads, direct client learning for leader-ordered writes, and fast fixed-quorum per-key writes, chosen according to network topology and workload. The AI-greement harness brings together a protocol knowledge base, modular implementation skeleton, and Lean-based verification framework; its formalization also exposed previously unreported vulnerabilities in existing protocols.

### Source excerpt

arXiv:2610.30114v1 Announce Type: new Abstract: Consensus is a fundamental problem in distributed computing and at the core of state machine replication (SMR). Despite decades of active human research, discovering practical low-latency solutions to the problem is still open, as it requires combining protocol knowledge, correctness reasoning, implementation, and experimental evaluation. We report on how AI helped us discover A-greement, a new consensus protocol for geo-replicated state machines that significantly outperforms state-of-the-art solutions. Averaged across six deployments with 5 or 9 replicas, A-greement reduces the mean latency of a geo-distributed key-value store over the fastest baseline by 12.2% and 33.3% on write-heavy and read-heavy workloads, respectively, and matches the fastest baseline on read-only workloads. A-greement combines lease-authorized local reads, direct client learning for leader-ordered writes, and a fast fixed-quorum per-key write path, selected according to network topology and workload. At the heart of our discovery lies AI-greement, an agent harness for the "autonomous" discovery, implementation, experimentation, and verification of candidate protocols. AI-greement integrates (1) a concept-oriented protocol knowledge base, (2) a modular implementation skeleton, and (3) a Lean-based verification framework with reusable safety definitions, reference specifications, and proofs. AI-greement is of independent interest: its Lean formalization uncovers several previously unreported vulnerabilities in existing consensus protocols, and it provides guidelines on how domain-specific harnesses can turn general-purpose coding agents into effective tools for (distributed) computing research. Yet we argue that human consensus experts remain essential before, during, and after the search: agents reshape the researcher's role rather than end consensus research.

## Runpod Global Networking Expands to 14 More Data Centers

DevFeed: [Runpod Global Networking Expands to 14 More Data Centers](<https://devfeed.tech/articles/runpod-global-networking-expands-to-14-more-data-centers-91126.md>)

Original publisher: [Read original article](<https://www.runpod.io/blog/runpod-global-networking-expansion>)

Author: Brendan McKeag

Published: 2026-10-20T18:19:28Z

Content type: release

Language: en

Sources: [Runpod Blog.](<https://devfeed.tech/sources/runpod-blog.md>)

Topics: [Cloud Networking](<https://devfeed.tech/topics/cloud-networking.md>), [data centers](<https://devfeed.tech/topics/data-centers.md>)

Tags: [accessibility](<https://devfeed.tech/tags/accessibility.md>), [ai](<https://devfeed.tech/tags/ai.md>), [ai-research](<https://devfeed.tech/tags/ai-research.md>), [ai-workloads](<https://devfeed.tech/tags/ai-workloads.md>), [asia](<https://devfeed.tech/tags/asia.md>), [data-center](<https://devfeed.tech/tags/data-center.md>), [data-centers](<https://devfeed.tech/tags/data-centers.md>), [data-residency](<https://devfeed.tech/tags/data-residency.md>), [distributed-computing](<https://devfeed.tech/tags/distributed-computing.md>), [edge](<https://devfeed.tech/tags/edge.md>), [eu](<https://devfeed.tech/tags/eu.md>), [europe](<https://devfeed.tech/tags/europe.md>), [feature](<https://devfeed.tech/tags/feature.md>), [federated](<https://devfeed.tech/tags/federated.md>), [federated-learning](<https://devfeed.tech/tags/federated-learning.md>), [global](<https://devfeed.tech/tags/global.md>), [gpu](<https://devfeed.tech/tags/gpu.md>), [latency](<https://devfeed.tech/tags/latency.md>), [networking](<https://devfeed.tech/tags/networking.md>), [north-america](<https://devfeed.tech/tags/north-america.md>)

### AI overview

Runpod says Global Networking has expanded to 14 additional data centers, extending private cross-data-center communication between pods. Pods communicate over the `.runpod.internal` network without exposing TCP or HTTP ports to the internet, enabling data sharing and client-server applications across regions. The article suggests possible uses including distributed training, federated learning, regional inference serving, and parallel simulations; these are presented as theoretical implementations. The excerpt does not list the new locations or include the promised instructions for enabling the feature.

### Source excerpt

Runpod's global networking feature is now available in 14 new data centers, improving latency and accessibility across North America, Europe, and Asia.

## OpenAI and Cursor agree on agent coordinators. They disagree on who runs them.

DevFeed: [OpenAI and Cursor agree on agent coordinators. They disagree on who runs them.](<https://devfeed.tech/articles/openai-and-cursor-agree-on-agent-coordinators-they-disagree-on-who-runs-them-59812.md>)

Original publisher: [Read original article](<https://thenewstack.io/openai-cursor-coordinator-agents/>)

Author: Robert Kimani

Published: 2026-09-25T12:00:00Z

Content type: news

Language: en

Sources: [The New Stack](<https://devfeed.tech/sources/the-new-stack.md>)

Topics: [OpenAI](<https://devfeed.tech/topics/openai.md>), [cursor](<https://devfeed.tech/topics/cursor.md>), [AI-assisted coding](<https://devfeed.tech/topics/ai-assisted-coding.md>), [codex](<https://devfeed.tech/topics/codex.md>), [Orchestration](<https://devfeed.tech/topics/orchestration.md>), [distributed-systems](<https://devfeed.tech/topics/distributed-systems.md>), [Artificial Intelligence](<https://devfeed.tech/topics/ai.md>), [Large Language Model](<https://devfeed.tech/topics/llm.md>)

Tags: [agent-workflows](<https://devfeed.tech/tags/agent-workflows.md>), [ai](<https://devfeed.tech/tags/ai.md>), [ai-agents](<https://devfeed.tech/tags/ai-agents.md>), [ai-engineering](<https://devfeed.tech/tags/ai-engineering.md>), [ai-infrastructure](<https://devfeed.tech/tags/ai-infrastructure.md>), [codex](<https://devfeed.tech/tags/codex.md>), [coding-agents](<https://devfeed.tech/tags/coding-agents.md>), [cursor](<https://devfeed.tech/tags/cursor.md>), [distributed-computing](<https://devfeed.tech/tags/distributed-computing.md>), [openai](<https://devfeed.tech/tags/openai.md>), [orchestration](<https://devfeed.tech/tags/orchestration.md>)

### AI overview

The article reports that OpenAI and Cursor are independently introducing coordinator architectures for managing multiple coding agents. It describes a coordinator-worker split in which an orchestrator manages context and routes work to specialized agents, reflecting a broader shift toward reliable multi-agent software development systems.

### Source excerpt

OpenAI opened its Agents API in public beta this month, exposing the harness that powers Codex with managed sessions, tool The post OpenAI and Cursor agree on agent coordinators. They disagree on who runs them. appeared first on The New Stack.

## MapReduce: The Abstraction Layer That Still Shapes How AI Workloads Scale

DevFeed: [MapReduce: The Abstraction Layer That Still Shapes How AI Workloads Scale](<https://devfeed.tech/articles/mapreduce-the-abstraction-layer-that-still-shapes-how-ai-workloads-scale-58365.md>)

Original publisher: [Read original article](<https://hackernoon.com/mapreduce-the-abstraction-layer-that-still-shapes-how-ai-workloads-scale?source=rss>)

Author: Darsh Shah

Published: 2026-09-22T21:05:14Z

Content type: article

Language: en

Sources: [HackerNoon](<https://devfeed.tech/sources/hackernoon.md>)

Topics: [distributed-systems](<https://devfeed.tech/topics/distributed-systems.md>), [data-processing](<https://devfeed.tech/topics/data-processing.md>), [AI Infrastructure](<https://devfeed.tech/topics/ai-infrastructure.md>), [abstraction](<https://devfeed.tech/topics/abstraction.md>), [Computing](<https://devfeed.tech/topics/computing.md>), [scheduling](<https://devfeed.tech/topics/scheduling.md>), [Large Language Model](<https://devfeed.tech/topics/llm.md>)

Tags: [ai-engineering](<https://devfeed.tech/tags/ai-engineering.md>), [ai-infrastructure](<https://devfeed.tech/tags/ai-infrastructure.md>), [architecture](<https://devfeed.tech/tags/architecture.md>), [data-processing](<https://devfeed.tech/tags/data-processing.md>), [distributed](<https://devfeed.tech/tags/distributed.md>), [distributed-computing](<https://devfeed.tech/tags/distributed-computing.md>), [distributed-systems](<https://devfeed.tech/tags/distributed-systems.md>), [fault-tolerance](<https://devfeed.tech/tags/fault-tolerance.md>), [infrastructure](<https://devfeed.tech/tags/infrastructure.md>), [llm](<https://devfeed.tech/tags/llm.md>), [load-balancing](<https://devfeed.tech/tags/load-balancing.md>), [mapreduce](<https://devfeed.tech/tags/mapreduce.md>), [scaling](<https://devfeed.tech/tags/scaling.md>), [scheduling](<https://devfeed.tech/tags/scheduling.md>), [software-architecture](<https://devfeed.tech/tags/software-architecture.md>), [software-engineering](<https://devfeed.tech/tags/software-engineering.md>), [system-design](<https://devfeed.tech/tags/system-design.md>)

### AI overview

The article reviews MapReduce as a programming model and runtime framework for large-scale distributed data processing. It explains how Map and Reduce functions separate developer logic from platform responsibilities such as partitioning, scheduling, fault tolerance, retries, data movement, and load balancing, then examines the model's relevance to modern AI and LLM infrastructure.

### Source excerpt

How MapReduce shaped modern AI infrastructure, from distributed computing and fault tolerance to data movement, scheduling, and scaling LLM workloads

## IonQ | The Top 5 Benefits of Modular Quantum Computing for Business

DevFeed: [IonQ | The Top 5 Benefits of Modular Quantum Computing for Business](<https://devfeed.tech/articles/ionq-the-top-5-benefits-of-modular-quantum-computing-for-business-83861.md>)

Original publisher: [Read original article](<https://ionq.com/blog/the-top-5-benefits-of-modular-quantum-computing-for-business>)

Author: IonQ Staff

Published: 2026-01-15T23:19:20Z

Content type: article

Language: en

Sources: [IonQ](<https://devfeed.tech/sources/ionq.md>)

Topics: [Quantum Computing](<https://devfeed.tech/topics/quantum-computing.md>), [Computing](<https://devfeed.tech/topics/computing.md>), [Post-quantum cryptography](<https://devfeed.tech/topics/post-quantum-cryptography.md>), [Distributed Systems & Parallel Computing](<https://devfeed.tech/topics/distributed-systems-parallel-computing.md>)

Tags: [capabilities](<https://devfeed.tech/tags/capabilities.md>), [challenges](<https://devfeed.tech/tags/challenges.md>), [change](<https://devfeed.tech/tags/change.md>), [cost](<https://devfeed.tech/tags/cost.md>), [cost-savings](<https://devfeed.tech/tags/cost-savings.md>), [customization](<https://devfeed.tech/tags/customization.md>), [cybersecurity](<https://devfeed.tech/tags/cybersecurity.md>), [distributed](<https://devfeed.tech/tags/distributed.md>), [distributed-computing](<https://devfeed.tech/tags/distributed-computing.md>), [engineering](<https://devfeed.tech/tags/engineering.md>), [flexibility](<https://devfeed.tech/tags/flexibility.md>), [hardware](<https://devfeed.tech/tags/hardware.md>), [hardware-design](<https://devfeed.tech/tags/hardware-design.md>), [ionq](<https://devfeed.tech/tags/ionq.md>), [quantum](<https://devfeed.tech/tags/quantum.md>), [quantum-computing](<https://devfeed.tech/tags/quantum-computing.md>)

### AI overview

The article describes five proposed business benefits of modular quantum computers: customization, resilience, upgrades, security, and lower costs. It argues that modular designs can support incremental changes, combine different quantum technologies, isolate compromised modules, and connect smaller quantum processing units to increase computing power.

### Source excerpt

Working to build the world's best quantum computers to solve the world's most complex problems

## Revisiting HPX

DevFeed: [Revisiting HPX](<https://devfeed.tech/articles/revisiting-hpx-25138.md>)

Original publisher: [Read original article](<https://blog.brakmic.com/revisiting-hpx/>)

Author: brakmic

Published: 2024-09-23T09:53:03Z

Content type: tutorial

Language: en

Sources: [Harris Brakmic - Coding](<https://devfeed.tech/sources/harris-brakmic-coding.md>)

Topics: [distributed-systems](<https://devfeed.tech/topics/distributed-systems.md>), [Programming](<https://devfeed.tech/topics/programming.md>), [Software Testing](<https://devfeed.tech/topics/software-testing.md>)

Tags: [article](<https://devfeed.tech/tags/article.md>), [asynchronous](<https://devfeed.tech/tags/asynchronous.md>), [c-plus-plus](<https://devfeed.tech/tags/c-plus-plus.md>), [coding](<https://devfeed.tech/tags/coding.md>), [cpp](<https://devfeed.tech/tags/cpp.md>), [distributed](<https://devfeed.tech/tags/distributed.md>), [distributed-computing](<https://devfeed.tech/tags/distributed-computing.md>), [examples](<https://devfeed.tech/tags/examples.md>), [framework](<https://devfeed.tech/tags/framework.md>), [hpx](<https://devfeed.tech/tags/hpx.md>), [installation](<https://devfeed.tech/tags/installation.md>), [macos](<https://devfeed.tech/tags/macos.md>), [parallel](<https://devfeed.tech/tags/parallel.md>), [parallelism](<https://devfeed.tech/tags/parallelism.md>), [programming](<https://devfeed.tech/tags/programming.md>), [testing](<https://devfeed.tech/tags/testing.md>)

### AI overview

This hands-on article revisits HPX, a runtime system for asynchronous, parallel, and distributed computing. It covers installation on macOS, configuration, compilation, and testing, and discusses HPX features including task-based parallelism and components.

### Source excerpt

This article explores the powerful parallelization and distributed computing capabilities of the HPX framework. Starting with installation on macOS, it walks through configuring, compiling, and testing HPX on a modest machine, highlighting key features like asynchronous programming, task-based parallelism, and HPX components. Whether new to HPX or looking for practical examples, this guide offers a hands-on approach to understanding and utilizing HPX in real-world scenarios.

## Amazon's Distributed Computing Manifesto

DevFeed: [Amazon's Distributed Computing Manifesto](<https://devfeed.tech/articles/amazon-s-distributed-computing-manifesto-12528.md>)

Original publisher: [Read original article](<http://brooker.co.za/blog/2022/11/22/manifesto.html>)

Author: Marc Brooker

Published: 2022-11-22T00:00:00Z

Content type: article

Language: en

Sources: [Marc Brooker's](<https://devfeed.tech/sources/marc-brooker-s-blog.md>)

Topics: [distributed-systems](<https://devfeed.tech/topics/distributed-systems.md>), [service](<https://devfeed.tech/topics/service.md>), [decoupling](<https://devfeed.tech/topics/decoupling.md>), [interface](<https://devfeed.tech/topics/interface.md>), [amazon](<https://devfeed.tech/topics/amazon.md>), [Back end](<https://devfeed.tech/topics/backend.md>), [P2P](<https://devfeed.tech/topics/p2p.md>)

Tags: [amazon](<https://devfeed.tech/tags/amazon.md>), [api](<https://devfeed.tech/tags/api.md>), [architecture](<https://devfeed.tech/tags/architecture.md>), [based](<https://devfeed.tech/tags/based.md>), [business-logic](<https://devfeed.tech/tags/business-logic.md>), [data](<https://devfeed.tech/tags/data.md>), [database](<https://devfeed.tech/tags/database.md>), [distributed-computing](<https://devfeed.tech/tags/distributed-computing.md>), [distributed-system](<https://devfeed.tech/tags/distributed-system.md>), [interfaces](<https://devfeed.tech/tags/interfaces.md>), [processing](<https://devfeed.tech/tags/processing.md>)

### AI overview

Amazon's 1998 manifesto argued that direct database access tied application code to the data model, making change and distributing work difficult as the business grew. It proposed a three-tier, service-based architecture: clients call well-defined interfaces that encapsulate business logic, keeping the underlying data model private. The article emphasizes that this required a developer mindset shift from designing around database schemas to designing client-service interfaces. The manifesto also identified centralized database workflows as a scaling bottleneck and proposed distributing workflow processing so data travels to the process. The excerpt presents these ideas as historically relevant while noting that later two-tier approaches are returning; it offers no detailed evaluation of those newer systems.

### Source excerpt

Amazon's Distributed Computing Manifesto Manifesto made manifest. In the Johannesburg of 1998, I was rocking a middle parting, my friend group was abuzz about the news that there was water (and therefore monsters) on Europa, and all the cool kids were getting satellite TV at home1. Over in Seattle, the folks at Amazon.com had started to notice that their architecture was in need of rethinking. $147 million in sales in 1997, and over $600 million in 1998, were proving to be challenging to deal with. In 1998, as Werner Vogels recently shared folks at Amazon wrote a distributed computing manifesto describing the problems they were seeing and the solutions they saw to those problems. The document itself, which you can (and should!) read in full over on Werner's blog is both something of a time capsule, and surprisingly relevant to many of the systems architecture debates going on today, and the challenges that nearly all growing architectures inevitably face. From the manifesto: The applications that run the business access the database directly and have knowledge of the data model embedded in them. This means that there is a very tight coupling between the applications and the data model, and data model changes have to be accompanied by application changes even if functionality remains the same. Despite being called a distributed computing manifesto, the Amazon of 1997 was already a distributed system by any reasonable measure. The problem was one of interfaces: the data store was serving as the interface between components and concerns, leading to tight coupling between storage and business logic. The architecture was difficult to scale, not in requests per second, but to adapt to new lines of business and the rate of overall change. This approach does not scale well and makes distributing and segregating processing based on where data is located difficult since the applications are sensitive to the interdependent relationships between data elements. The proposed solu

## Apache Flink Table Store 0.1.0 Release Announcement

DevFeed: [Apache Flink Table Store 0.1.0 Release Announcement](<https://devfeed.tech/articles/apache-flink-table-store-0-1-0-release-announcement-48441.md>)

Original publisher: [Read original article](<https://flink.apache.org/2022/05/11/apache-flink-table-store-0.1.0-release-announcement/>)

Published: 2022-05-11T08:00:00Z

Content type: release

Language: en

Sources: [Apache Flink](<https://devfeed.tech/sources/apache-flink-stateful-computations-over-data-streams-on-apache-flink.md>)

Topics: [Apache Flink](<https://devfeed.tech/topics/apache-flink.md>), [stream-processing](<https://devfeed.tech/topics/stream-processing.md>), [distributed-systems](<https://devfeed.tech/topics/distributed-systems.md>)

Tags: [apache](<https://devfeed.tech/tags/apache.md>), [apache-flink](<https://devfeed.tech/tags/apache-flink.md>), [batch](<https://devfeed.tech/tags/batch.md>), [beta](<https://devfeed.tech/tags/beta.md>), [distributed](<https://devfeed.tech/tags/distributed.md>), [distributed-computing](<https://devfeed.tech/tags/distributed-computing.md>), [flink](<https://devfeed.tech/tags/flink.md>), [latency](<https://devfeed.tech/tags/latency.md>), [release](<https://devfeed.tech/tags/release.md>), [stateful](<https://devfeed.tech/tags/stateful.md>), [stream-processing](<https://devfeed.tech/tags/stream-processing.md>)

### AI overview

Apache Flink announces the preview release of Flink Table Store 0.1.0, a beta project for unified storage across streaming and batch workloads. It aims to support large datasets, low-latency streaming queries, batch and OLAP queries, and incremental snapshots.

### Source excerpt

The Apache Flink community is pleased to announce the preview release of the Apache Flink Table Store (0.1.0). Please check out the full documentation for detailed information and user guides. Note: Flink Table Store is still in beta status and undergoing rapid development. We do not recommend that you use it directly in a production environment. What is Flink Table Store # In the past years, thanks to our numerous contributors and users, Apache Flink has established itself as one of the best distributed computing engines, especially for stateful stream processing at large scale.

## DTable - an early performance assessment of a new distributed table implementation

DevFeed: [DTable - an early performance assessment of a new distributed table implementation](<https://devfeed.tech/articles/dtable-an-early-performance-assessment-of-a-new-distributed-table-implementation-46333.md>)

Original publisher: [Read original article](<https://julialang.org/blog/2021/12/dtable-performance-assessment/index.html>)

Author: Krystian Guliński

Published: 2021-12-08T00:00:00Z

Content type: article

Language: en

Sources: [JuliaLang - The Julia programming language](<https://devfeed.tech/sources/julialang-the-julia-programming-language.md>)

Topics: [The Julia Language](<https://devfeed.tech/topics/julia.md>), [data-processing](<https://devfeed.tech/topics/data-processing.md>), [distributed-systems](<https://devfeed.tech/topics/distributed-systems.md>), [data](<https://devfeed.tech/topics/data.md>), [implementation](<https://devfeed.tech/topics/implementation.md>)

Tags: [data-processing](<https://devfeed.tech/tags/data-processing.md>), [distributed](<https://devfeed.tech/tags/distributed.md>), [distributed-computing](<https://devfeed.tech/tags/distributed-computing.md>), [partitioning](<https://devfeed.tech/tags/partitioning.md>), [partitions](<https://devfeed.tech/tags/partitions.md>), [performance](<https://devfeed.tech/tags/performance.md>)

### AI overview

This article presents an early performance assessment of DTable, a Julia table structure for processing tabular data larger than available RAM. DTable uses distributed partitioning and parallel operations, builds on Dagger for task scheduling and memory management, and emphasizes composability with existing Julia table interfaces and data tools.

### Source excerpt

DTable - an early performance assessment of a new distributed table implementation

## Helios: hyperscale indexing for the cloud & edge - part 1

DevFeed: [Helios: hyperscale indexing for the cloud & edge - part 1](<https://devfeed.tech/articles/helios-hyperscale-indexing-for-the-cloud-edge-part-1-28588.md>)

Original publisher: [Read original article](<https://blog.acolyer.org/2020/10/26/helios-part-1/>)

Author: adriancolyer

Published: 2020-10-26T20:22:00Z

Content type: article

Language: en

Sources: [Adrian Colyer](<https://devfeed.tech/sources/adrian-colyer.md>)

Topics: [data-processing](<https://devfeed.tech/topics/data-processing.md>), [big-data](<https://devfeed.tech/topics/big-data.md>), [Cloud](<https://devfeed.tech/topics/cloud.md>), [distributed-systems](<https://devfeed.tech/topics/distributed-systems.md>), [data](<https://devfeed.tech/topics/data.md>), [Edge](<https://devfeed.tech/topics/edge.md>), [Microsoft](<https://devfeed.tech/topics/microsoft.md>), [client](<https://devfeed.tech/topics/client.md>), [Server](<https://devfeed.tech/topics/server.md>)

Tags: [architecture](<https://devfeed.tech/tags/architecture.md>), [big-data](<https://devfeed.tech/tags/big-data.md>), [cloud](<https://devfeed.tech/tags/cloud.md>), [data](<https://devfeed.tech/tags/data.md>), [data-processing](<https://devfeed.tech/tags/data-processing.md>), [datastores](<https://devfeed.tech/tags/datastores.md>), [distributed](<https://devfeed.tech/tags/distributed.md>), [distributed-computing](<https://devfeed.tech/tags/distributed-computing.md>), [edge](<https://devfeed.tech/tags/edge.md>), [idc](<https://devfeed.tech/tags/idc.md>), [latency](<https://devfeed.tech/tags/latency.md>), [microsoft](<https://devfeed.tech/tags/microsoft.md>), [paper](<https://devfeed.tech/tags/paper.md>), [real-time](<https://devfeed.tech/tags/real-time.md>), [server](<https://devfeed.tech/tags/server.md>), [streams](<https://devfeed.tech/tags/streams.md>), [systems](<https://devfeed.tech/tags/systems.md>), [uncategorized](<https://devfeed.tech/tags/uncategorized.md>)

### AI overview

This first part of a discussion of Helios examines its role as a reference architecture for distributed big-data systems. It describes the pressures driving cloud platforms toward new approaches, including growing data volume, variable arrival rates, lower latency, reduced data-center costs, and stronger privacy considerations.

### Source excerpt

Helios: hyperscale indexing for the cloud & edge, Potharaju et al., PVLDB'20 On the surface this is a paper about fast data ingestion from high-volume streams, with indexing to support efficient querying. As a production system within Microsoft capturing around a quadrillion events and indexing 16 trillion search keys per day it would be interesting in its own right, ... Continue reading Helios: hyperscale indexing for the cloud & edge - part 1

## Applications of Blockchain to Programming Language Theory

DevFeed: [Applications of Blockchain to Programming Language Theory](<https://devfeed.tech/articles/applications-of-blockchain-to-programming-language-theory-29480.md>)

Original publisher: [Read original article](<http://lambda-the-ultimate.org/ltu-blockchain-plt-thinktank>)

Published: 2020-04-13T14:38:12Z

Content type: opinion

Language: en

Sources: [Lambda the Ultimate](<https://devfeed.tech/sources/lambda-the-ultimate.md>)

Topics: [Blockchain](<https://devfeed.tech/topics/blockchain.md>), [Programming](<https://devfeed.tech/topics/programming.md>), [Programming language](<https://devfeed.tech/topics/programming-language.md>), [smart contract](<https://devfeed.tech/topics/smart-contract.md>), [distributed-systems](<https://devfeed.tech/topics/distributed-systems.md>)

Tags: [blockchain](<https://devfeed.tech/tags/blockchain.md>), [bug](<https://devfeed.tech/tags/bug.md>), [distributed-computing](<https://devfeed.tech/tags/distributed-computing.md>), [ethereum](<https://devfeed.tech/tags/ethereum.md>), [fun](<https://devfeed.tech/tags/fun.md>), [implementation](<https://devfeed.tech/tags/implementation.md>), [programming](<https://devfeed.tech/tags/programming.md>), [programming-language](<https://devfeed.tech/tags/programming-language.md>), [semantics](<https://devfeed.tech/tags/semantics.md>), [smart-contract](<https://devfeed.tech/tags/smart-contract.md>), [transactions](<https://devfeed.tech/tags/transactions.md>), [verification](<https://devfeed.tech/tags/verification.md>)

### AI overview

A forum discussion explores how blockchain could contribute to programming language theory, practice, and developer productivity. It considers mutual and common knowledge, distributed computing fabrics, third-party transactions, and the verification and security of smart contracts, including an Ethereum smart-contract logic bug.

### Source excerpt

Let's talk about Blockchain. Goal is to use this forum topic to highlight its usefulness to programming language theory and practice. If you're familiar with existing research efforts, please share them here. In addition, feel free to generate ideas for how Blockchain could improve languages and developer productivity. As one tasty example: Blockchain helps to formalize thinking about mutual knowledge and common knowledge, and potentially think about sharing intergalactic computing power through vast distributed computing fabrics. If we can design contracts in such a way that maximizes the usage of mutual knowledge while minimizing common knowledge to situations where you have to "prove your collateral", third-party transactions could eliminate a lot of back office burden. But, there might be benefits in other areas of computer science from such research, as well. Some language researchers, like Mark S. Miller, have always dreamed of Agoric and the Decades-Long Quest for Secure Smart Contracts. Some may also be aware that verification of smart contracts is an important research area, because of the notorious theft of purse via logic bug in an Ethereum smart contract.

## Why do we need distributed systems?

DevFeed: [Why do we need distributed systems?](<https://devfeed.tech/articles/why-do-we-need-distributed-systems-12490.md>)

Original publisher: [Read original article](<http://brooker.co.za/blog/2020/01/02/why-distributed.html>)

Author: Marc Brooker

Published: 2020-01-02T00:00:00Z

Content type: article

Language: en

Sources: [Marc Brooker's](<https://devfeed.tech/sources/marc-brooker-s-blog.md>)

Topics: [distributed-systems](<https://devfeed.tech/topics/distributed-systems.md>), [systems](<https://devfeed.tech/topics/systems.md>), [Developer Tools](<https://devfeed.tech/topics/developer-tools.md>), [Load Balancing](<https://devfeed.tech/topics/load-balancing.md>)

Tags: [availability](<https://devfeed.tech/tags/availability.md>), [better](<https://devfeed.tech/tags/better.md>), [complex](<https://devfeed.tech/tags/complex.md>), [complexity](<https://devfeed.tech/tags/complexity.md>), [computing](<https://devfeed.tech/tags/computing.md>), [cost](<https://devfeed.tech/tags/cost.md>), [datacenter](<https://devfeed.tech/tags/datacenter.md>), [distributed](<https://devfeed.tech/tags/distributed.md>), [distributed-computing](<https://devfeed.tech/tags/distributed-computing.md>), [distributed-systems](<https://devfeed.tech/tags/distributed-systems.md>), [efficiency](<https://devfeed.tech/tags/efficiency.md>), [fail](<https://devfeed.tech/tags/fail.md>), [failure](<https://devfeed.tech/tags/failure.md>), [hardware](<https://devfeed.tech/tags/hardware.md>), [john](<https://devfeed.tech/tags/john.md>), [memory](<https://devfeed.tech/tags/memory.md>), [modern](<https://devfeed.tech/tags/modern.md>), [need](<https://devfeed.tech/tags/need.md>), [performance](<https://devfeed.tech/tags/performance.md>), [systems](<https://devfeed.tech/tags/systems.md>)

### AI overview

Distributed systems can improve availability, durability, scalability and efficiency by spreading state and workloads across machines. Replication and consensus avoid dependence on one server; distributed storage keeps multiple copies and can tolerate correlated failures; shared infrastructure and automatic scaling can match capacity to changing demand. Stateless systems are comparatively easy to scale, while stateful systems require decisions about data placement and routing, for which existing cloud services and software can help. These benefits come with real costs: distributed systems are harder to build, understand and operate, and single-machine designs can be faster and more efficient. The author argues that better tooling and services, including serverless, could make distributed capabilities simpler to use; the theory does not guarantee practical results.

### Source excerpt

Why do we need distributed systems? Building distributed systems is hard. It's expensive. It's complex. But we do it anyway. I grew up reading John Carmack's .plan file. His stories about the development of Doom, Quake and the rest were a formative experience for me, and a big reason I was interested in computers beyond just gaming1. I was a little bit disappointed to see this tweet: My formative memory of Python was when the Quake Live team used it for the back end work, and we wound up having serious performance problems with a few million users. My bias is that a lot (not all!) of complex "scalable" systems can be done with a simple, single C++ server. -- John Carmack (@ID_AA_Carmack) December 28, 2019 This isn't an isolated opinion, but I don't think it's a particularly good one. To be fair, there are a lot of good reasons not to build distributed systems. Complexity is one: distributed systems are legitimately harder to build, and significantly harder to understand and operate. Efficiency is another. As McSherry et al point out in Scalability! But at what COST?, single-system designs can have great performance and efficiency. Modern computers are huge and fast. I was not so much disappointed in John, as in our success at building distributed systems tools that make this untrue. Distributed computing could be much easier, and needs to be much easier. We need to get to a point, with services, tooling and technology, that monolithic systems aren't a good default. To understand why, let me answer the question in the post's title. Distributed systems offer better availability The availability of a monolithic system is limited to the availability of the piece of hardware it runs on. Modern hardware is pretty great, and combined with a good datacenter and good management practices servers can be expected to fail with an annual failure rate (AFR) in the single-digit percentages. That's OK, but not great in two ways. First, if you run a lot of systems fixing these server

## Teleport 2.5 release information

DevFeed: [Teleport 2.5 release information](<https://devfeed.tech/articles/teleport-2-5-release-information-29901.md>)

Original publisher: [Read original article](<https://goteleport.com/blog/teleport-2-5-released/>)

Author: ev@goteleport.com (Ev Kontsevoy)

Published: 2018-03-10T00:00:00Z

Content type: release

Language: en

Sources: [Teleport](<https://devfeed.tech/sources/teleport.md>)

Topics: [Amazon Web Services](<https://devfeed.tech/topics/aws.md>), [gateway](<https://devfeed.tech/topics/gateway.md>), [ssh](<https://devfeed.tech/topics/ssh.md>), [upgrade](<https://devfeed.tech/topics/upgrade.md>), [Authentication](<https://devfeed.tech/topics/authentication.md>), [certificates](<https://devfeed.tech/topics/certificates.md>), [distributed-systems](<https://devfeed.tech/topics/distributed-systems.md>), [Single sign-on (SSO)](<https://devfeed.tech/topics/sso.md>), [audit](<https://devfeed.tech/topics/audit.md>), [nginx](<https://devfeed.tech/topics/nginx.md>), [configuration](<https://devfeed.tech/topics/configuration.md>)

Tags: [audit](<https://devfeed.tech/tags/audit.md>), [authentication](<https://devfeed.tech/tags/authentication.md>), [aws](<https://devfeed.tech/tags/aws.md>), [certificates](<https://devfeed.tech/tags/certificates.md>), [configuration](<https://devfeed.tech/tags/configuration.md>), [distributed-computing](<https://devfeed.tech/tags/distributed-computing.md>), [load-balancer](<https://devfeed.tech/tags/load-balancer.md>), [release](<https://devfeed.tech/tags/release.md>), [ssh](<https://devfeed.tech/tags/ssh.md>), [sso](<https://devfeed.tech/tags/sso.md>), [teleport](<https://devfeed.tech/tags/teleport.md>), [tokens](<https://devfeed.tech/tags/tokens.md>), [upgrade](<https://devfeed.tech/tags/upgrade.md>)

### AI overview

Teleport 2.5 is released with improvements for deploying Teleport at scale on AWS, including highly available auth servers, proxy certificates behind load balancers, and node-token generation using AWS secret-management facilities. The release also documents a cluster upgrade procedure.

### Source excerpt

This release of Teleport brings easier AWS deployments at scale and includes several usability enhancements.

## One-box stream processing with CSP

DevFeed: [One-box stream processing with CSP](<https://devfeed.tech/articles/one-box-stream-processing-with-csp-32160.md>)

Original publisher: [Read original article](<https://adambard.com/blog/stream-processing-core-async/>)

Published: 2018-02-18T00:00:00Z

Content type: tutorial

Language: en

Sources: [Adam Bard](<https://devfeed.tech/sources/adam-bard.md>)

Topics: [stream-processing](<https://devfeed.tech/topics/stream-processing.md>), [Clojure](<https://devfeed.tech/topics/clojure.md>), [Concurrency](<https://devfeed.tech/topics/concurrency.md>), [distributed-systems](<https://devfeed.tech/topics/distributed-systems.md>)

Tags: [clojure](<https://devfeed.tech/tags/clojure.md>), [concurrent](<https://devfeed.tech/tags/concurrent.md>), [core-async](<https://devfeed.tech/tags/core-async.md>), [coroutine](<https://devfeed.tech/tags/coroutine.md>), [distributed-computing](<https://devfeed.tech/tags/distributed-computing.md>), [distributed-system](<https://devfeed.tech/tags/distributed-system.md>), [flink](<https://devfeed.tech/tags/flink.md>), [spark](<https://devfeed.tech/tags/spark.md>), [stream](<https://devfeed.tech/tags/stream.md>), [stream-processing](<https://devfeed.tech/tags/stream-processing.md>)

### AI overview

This article presents a small-scale stream-processing architecture in Clojure using core.async and component libraries. It describes modular components connected by asynchronous queues and explains how stream-processing design principles can support reusable, loosely coupled systems without requiring a distributed stream processor.

### Source excerpt

If you're like me (that is, employed by an ad tech company), stream processing is usually associated with frameworks like Storm, Flink, Spark Streaming, and other such solutions. However, a lot of real-life software can be described as stream processing - data comes in one end, is transformed or aggregated, and goes somewhere else. Many of these workloads don't justify the overhead of a stream processor, but that doesn't mean they can't benefit from some of the lessons of stream processing systems.

## The Fallacies of Distributed Computing (Applied to Front-End Performance)

DevFeed: [The Fallacies of Distributed Computing (Applied to Front-End Performance)](<https://devfeed.tech/articles/the-fallacies-of-distributed-computing-applied-to-front-end-performance-45628.md>)

Original publisher: [Read original article](<https://csswizardry.com/2017/11/the-fallacies-of-distributed-computing-applied-to-front-end-performance/>)

Author: Harry Roberts

Published: 2017-11-21T09:51:11Z

Content type: article

Language: en

Sources: [CSS Wizardry](<https://devfeed.tech/sources/css-wizardry.md>)

Topics: [distributed-systems](<https://devfeed.tech/topics/distributed-systems.md>), [Front end](<https://devfeed.tech/topics/frontend.md>), [Latency](<https://devfeed.tech/topics/latency.md>), [reliability-engineering](<https://devfeed.tech/topics/reliability-engineering.md>), [Resilience](<https://devfeed.tech/topics/resilience.md>)

Tags: [developers](<https://devfeed.tech/tags/developers.md>), [distributed](<https://devfeed.tech/tags/distributed.md>), [distributed-computing](<https://devfeed.tech/tags/distributed-computing.md>), [front-end](<https://devfeed.tech/tags/front-end.md>), [latency](<https://devfeed.tech/tags/latency.md>), [network](<https://devfeed.tech/tags/network.md>), [networking](<https://devfeed.tech/tags/networking.md>), [performance](<https://devfeed.tech/tags/performance.md>), [reliability](<https://devfeed.tech/tags/reliability.md>), [resilience](<https://devfeed.tech/tags/resilience.md>), [resource-loading](<https://devfeed.tech/tags/resource-loading.md>)

### AI overview

This article applies the classic Fallacies of Distributed Computing to front-end performance. It explains why developers should not assume reliable networks, zero latency, infinite bandwidth, or ideal user conditions, and argues for designing and testing for degraded conditions.

### Source excerpt

What can front-end and performance engineers learn from the Fallacies of Distributed Computing?

## How Target Performance Tunes Machine Learning Applications

DevFeed: [How Target Performance Tunes Machine Learning Applications](<https://devfeed.tech/articles/how-target-performance-tunes-machine-learning-applications-20400.md>)

Original publisher: [Read original article](<https://target.github.io/data%20science%20and%20engineering/numspark>)

Author: Target Brands, Inc

Published: 2016-09-29T05:00:00Z

Content type: article

Language: en

Sources: [Target](<https://devfeed.tech/sources/target.md>)

Topics: [Machine learning](<https://devfeed.tech/topics/machine-learning.md>), [big-data](<https://devfeed.tech/topics/big-data.md>), [Apache Spark](<https://devfeed.tech/topics/spark.md>), [dataset](<https://devfeed.tech/topics/dataset.md>), [Hadoop](<https://devfeed.tech/topics/hadoop.md>)

Tags: [algorithms](<https://devfeed.tech/tags/algorithms.md>), [apache-spark](<https://devfeed.tech/tags/apache-spark.md>), [applications](<https://devfeed.tech/tags/applications.md>), [data](<https://devfeed.tech/tags/data.md>), [data-science-and-engineering](<https://devfeed.tech/tags/data-science-and-engineering.md>), [distributed](<https://devfeed.tech/tags/distributed.md>), [distributed-computing](<https://devfeed.tech/tags/distributed-computing.md>), [engine](<https://devfeed.tech/tags/engine.md>), [engineering](<https://devfeed.tech/tags/engineering.md>), [file-system](<https://devfeed.tech/tags/file-system.md>), [hadoop](<https://devfeed.tech/tags/hadoop.md>), [large-datasets](<https://devfeed.tech/tags/large-datasets.md>), [leverage](<https://devfeed.tech/tags/leverage.md>), [machine-learning](<https://devfeed.tech/tags/machine-learning.md>), [matrix](<https://devfeed.tech/tags/matrix.md>), [open-source-software](<https://devfeed.tech/tags/open-source-software.md>), [performance](<https://devfeed.tech/tags/performance.md>), [scalability](<https://devfeed.tech/tags/scalability.md>), [science](<https://devfeed.tech/tags/science.md>)

### AI overview

Target describes evaluating large-scale matrix multiplication to improve the speed and relevance of machine-learning algorithms built on billions of guest interactions and item data. The excerpt says this work requires computation across multiple nodes, high-throughput access to distributed data, storage and compute that scale with added resources, and fault tolerance as failures become more common. It names Apache Hadoop for storing and interacting with data, Apache Spark as the compute engine and a distributed machine-learning tool, and Apache Mahout for building and applying distributed algorithms. The article begins its practical guidance with paper-and-pencil examples: try matrices of different shapes and properties, choose compatible dimensions, and work through operations such as multiplication, inversion, and transposition. The excerpt cuts off during this list.

### Source excerpt

At Target we aim to make shopping more fun and relevant for our guests through extensive use of data - and believe me, we have lots of data! Tens of millions of guests and hundreds of thousands of items lead to billions of transactions and interactions. We regularly employ a number of different machine learning techniques on such large datasets for dozens of algorithms. We are constantly looking for ways to improve speed and relevance of our algorithms and one such quest brought us to carefully evaluate matrix multiplications at scale - since that forms the bedrock for most algorithms. If we make matrix multiplication more efficient, we can speed up most of our algorithms! Before we dig in, let me describe some properties of the landscape we will be working in. First, what do I mean by large scale? A large scale application, at a minimum, will require its computation to be spread over multiple nodes of a distributed computing environment to finish in a reasonable amount of time. These calculations will use existing data that are stored on a distributed file system that provides high-throughput access from the computing environment. Scalability, in terms of storage and compute, should grow as we add to these resources. As the system grows larger and more complex, failures will become more commonplace. Thus, software should be fault-tolerant. Fortunately, there is a lot of existing open-source software that we can leverage to work in such an environment, particularly Apache Hadoop for storing and interacting with our data, Apache Spark as the compute engine, and both Apache Spark and Apache Mahout for applying and building distributed machine learning algorithms. There are many other tools that we can add to the mix as well, but for the purposes of this post we will limit our discussion to these three. With that out of the way, lets dig in! Don't Forget the Basics Begin with good old paper and pencil. Yeah, I know this is about large scale matrix operations that you c

[Next page](<https://devfeed.tech/tags/distributed-computing.md?cursor=WyIyMDE2LTA5LTI5VDA1OjAwOjAwKzAwOjAwIiwgImIwOWY0ODE1LWNjNDEtNDBhMi05Zjc0LTgxNjhlNDI1OGRkYSJd>)