# distributed system Published articles for distributed system. This is one page of public article previews, not the complete archive. Follow Next page to continue. Summaries are not the original full articles. ## How WebRTC Scales: Signaling, NAT Traversal, and the Mesh/SFU/MCU Tradeoff DevFeed: [How WebRTC Scales: Signaling, NAT Traversal, and the Mesh/SFU/MCU Tradeoff]() Original publisher: [Read original article]() Author: Karan Pratap Singh Published: 2026-09-15T15:57:39Z Content type: article Language: en Sources: [freeCodeCamp Programming Tutorials: Python, JavaScript, Git & More]() Topics: [WebRTC](), [real-time](), [browsers](), [API](), [JavaScript](), [WebSocket](), [Firewall](), [Network](), [servers]() Tags: [api](), [browsers](), [decoding](), [distributed-system](), [firewall](), [http](), [javascript](), [network](), [networking](), [real-time](), [server](), [webrtc]() ### AI overview This article explains how WebRTC enables browsers to exchange audio, video, and data directly. It covers the three WebRTC APIs, signaling through WebSockets or HTTP, NAT traversal using ICE, STUN, and TURN, and the mesh, SFU, and MCU approaches to scaling media delivery. ### Source excerpt Web Real-Time Communication (or WebRTC) is the open standard browsers use to send audio, video, and data straight to each other. There's no plugin or native app, nothing beyond an API that every brows ## Durable, flexible multi-agent systems DevFeed: [Durable, flexible multi-agent systems]() Original publisher: [Read original article]() Author: Melanie Warrick Published: 2026-08-06T00:00:00Z Content type: tutorial Language: en Sources: [Temporal Blog]() Topics: [Multi Agent Systems](), [Orchestration](), [systems](), [Langgraph](), [Persistence](), [Google]() Tags: [agents](), [distributed-system](), [google-adk](), [langgraph](), [multi-agent](), [orchestration](), [persistence](), [temporal-voices]() ### AI overview This article describes a multi-agent delivery fleet built with Google ADK and LangGraph, using Temporal for orchestration, persistence, retries, and resumability. It explains how to keep human oversight durable when agents or workers disconnect or when workflows wait for human responses. ### Source excerpt Running a multi-agent delivery fleet on Google ADK and LangGraph, and what it takes to make the human in the loop durable. ## Spotlight on SIG Storage DevFeed: [Spotlight on SIG Storage]() Original publisher: [Read original article]() Author: The Kubernetes Authors Published: 2026-06-15T00:00:00Z Content type: article Language: en Sources: [Kubernetes Contributors Blog]() Topics: [Kubernetes](), [systems](), [data]() Tags: [backups](), [containers](), [distributed-system](), [kubernetes](), [storage](), [volumes]() ### AI overview An interview with Xing Yang, co-chair of Kubernetes SIG Storage, covering the group's role in persistent data and volume management, Kubernetes CSI maintenance, recent storage features, and data protection work such as Volume Group Snapshot and Changed Block Tracking. ### Source excerpt In our ongoing SIG Spotlight series, we shine a light on the groups that keep the Kubernetes project moving forward. This time, we catch up with SIG Storage , the group responsible for persistent data, volume management, and the interfaces that connect Kubernetes workloads to the storage systems beneath them. We spoke with Xing Yang , Co-Chair of SIG Storage and Software Engineer at VMware by Broadcom, about the SIG's history, the features shipping in recent Kubernetes releases, and where storage in Kubernetes is headed as AI workloads become the norm. Introductions Could you introduce yourself and share your role(s) within SIG Storage? My name is Xing Yang , a software engineer at VMware by Broadcom. I'm a co-chair in SIG Storage, alongside another co-chair Saad Ali from Google. There are also two Tech Leads in SIG Storage: Michelle Au from Google and Jan Šafránek from Red Hat. What first drew you to storage in Kubernetes, and how did you start contributing? I have always been working in the storage domain, so SIG Storage was a natural place for me to get started when I began to learn Kubernetes. I started attending SIG Storage meetings , trying to figure out what I could do to help. This was before the first Container Storage Interface (CSI) release -- lots of things were still evolving. It was a very exciting time. What subprojects or areas do you actively maintain or review today? I'm a maintainer in Kubernetes CSI. There are multiple CSI sidecars -- such as csi-provisioner, csi-attacher, csi-resizer, and csi-snapshotter -- that we need to release following every Kubernetes release. I'm also a co-chair for a Data Protection Working Group co-sponsored by SIG Storage and SIG Apps . Several features have come out of that WG aimed at filling gaps in data protection support within Kubernetes. One is Volume Group Snapshot , which provides crash-consistent group snapshots for multiple volumes used by an application. Changed Block Tracking (CBT) is another critical feature ## The Distributed Systems Challenges of Deploying Agentic AI in Production DevFeed: [The Distributed Systems Challenges of Deploying Agentic AI in Production]() Original publisher: [Read original article]() Author: Quentin Packard Published: 2026-06-04T00:00:00Z Content type: opinion Language: en Sources: [Cockroach Labs]() Topics: [Artificial Intelligence](), [AI Agent](), [distributed-systems](), [Concurrency](), [consistency](), [incident](), [observability]() Tags: [agent](), [agentic-ai](), [ai](), [concurrency](), [consistency](), [distributed-system](), [incident](), [observability](), [systems]() ### AI overview This article argues that deploying agentic AI in production requires solving distributed-systems problems around memory state, concurrency, identity, blast radius, observability, and economics. It focuses on state management for agents serving concurrent users, including failures, consistency, durability, and recovery. ### Source excerpt Most enterprise AI teams have built an agent that was impressive; far fewer have shipped one without a production incident that made someone question the whole program. ## Dropbox's Former Most Senior Eng: Building Great Systems and Advice for the AI Era | James Cowling DevFeed: [Dropbox's Former Most Senior Eng: Building Great Systems and Advice for the AI Era | James Cowling]() Original publisher: [Read original article]() Author: Ryan Peterman Published: 2026-05-25T10:00:57Z Content type: opinion Language: en Sources: [The Developing Dev]() Topics: [systems](), [distributed-systems](), [Concurrency](), [Transactions](), [Computer science](), [Replication](), [Protocol (disambiguation)]() Tags: [concurrency](), [consensus](), [distributed](), [distributed-system](), [distributed-transaction](), [protocol](), [replication](), [systems](), [technical](), [transactions]() ### AI overview A podcast interview with James Cowling, CTO of Convex and former senior engineer at Dropbox, about designing abstractions and large-scale transactional systems. Cowling discusses Granola, distributed transaction coordination, concurrency, Byzantine fault tolerance, and career advice in the context of AI. ### Source excerpt Transcript & Audio ## Multi-Region Database Architecture: Three Patents on SQL Abstractions, Data Placement, and Locality-Aware Query Planning DevFeed: [Multi-Region Database Architecture: Three Patents on SQL Abstractions, Data Placement, and Locality-Aware Query Planning]() Original publisher: [Read original article]() Author: Rebecca Taft Published: 2026-04-09T00:00:00Z Content type: article Language: en Sources: [Cockroach Labs]() Topics: [Databases](), [CockroachDB](), [SQL](), [Cockroach Labs](), [Resilience](), [Latency](), [Availability]() Tags: [availability](), [cockroach-labs](), [cockroachdb](), [database](), [databases](), [distributed-system](), [latency](), [resilience](), [sql]() ### AI overview This article explains three Cockroach Labs patents covering SQL abstractions for multi-region database architecture, replica placement and data domiciling, and locality-aware query planning. It describes how these mechanisms address resilience, availability, compliance, data residency, and latency across regions. ### Source excerpt Multi-region deployments are often motivated by resilience: If a region goes down, your system should keep serving traffic. But resilience is only part of the story. ## AI Slop: A Slack API Rate Limiting Disaster DevFeed: [AI Slop: A Slack API Rate Limiting Disaster]() Original publisher: [Read original article]() Author: Daniel Doubrovkine (dblock@dblock.org) Published: 2026-03-12T00:00:00Z Content type: article Language: en Sources: [Daniel Doubrovkine]() Topics: [Slack](), [API](), [Code]() Tags: [ai](), [api](), [async](), [code](), [concurrent](), [distributed-system](), [rate-limiting](), [retry](), [script](), [slack]() ### AI overview The article examines an AI-generated cleanup job for closing old Slack group DM conversations. Because Slack's conversations.close endpoint has a global rate limit of one request per second, the implementation could exhaust the limit and disrupt other API calls. The author describes mitigating the issue with scheduled execution, an opt-in setting, a slow-drain script, and limiting the number of DMs closed concurrently. ### Source excerpt Yesterday I described AI-generated code as "plausible-looking, locally coherent, globally wrong." Here's a concrete example from my own codebase. I needed a cleanup job to close old Slack group DM conversations in my slack-sup2 app. The AI-generated solution looked perfectly reasonable: def close_old_sups! return 0 unless sup_close old_sups = sups.where(conditions) old_sups.each(&:close!) old_sups.count end def close! return unless conversation_id return if closed_at logger.info "Closing DM channel #{conversation_id}..." slack_client.conversations_close(channel: conversation_id) update_attributes!(closed_at: Time.now.utc) end This code looks pretty great and completely breaks the app. Slack's API has a global rate limit of 1 request per second for the conversations.close endpoint. So, when this job runs against a workspace with hundreds of old conversations, it immediately hits the rate limit. Worse, because rate limits are global across all endpoints, it takes down the entire application. Every other API call -- posting messages, fetching user info, everything -- starts failing. I asked AI to address this. It made the fix catastrophically worse. def close! # ... existing code ... begin slack_client.conversations_close(channel: conversation_id) rescue Slack::Web::Api::Errors::TooManyRequests => e sleep_time = e.retry_after || 60 sleep(sleep_time) retry end # ... rest of method ... end In socketry/async, sleep() blocks the entire fiber and prevents other concurrent operations from executing. You should use Async::Task.sleep() instead, but that still doesn't solve the fundamental architectural problem of making hundreds of sequential API calls. What made this particularly insidious is that the assistant wrote some pretty professional-looking code and seemingly handled obvious edge cases. Yet, it failed to consider the distributed system constraints or global invariants. The rate limiting problem wasn't visible in the local scope of the method - it was a system-wide conc ## Getting from tested to battle-tested DevFeed: [Getting from tested to battle-tested]() Original publisher: [Read original article]() Author: Doug Patti Published: 2025-12-03T00:00:00Z Content type: article Language: en Sources: [Jane Street]() Topics: [Testing](), [Software](), [Availability](), [incident](), [aria](), [systems](), [Network]() Tags: [automated](), [availability](), [development](), [distributed-system](), [high-availability](), [incident](), [latency](), [low-latency](), [software](), [systems](), [testing](), [tests]() ### AI overview The article explains how Jane Street works to make testing more effective for reliable, high-availability software. It focuses on Aria, an internal distributed low-latency message bus, and introduces Antithesis as an end-to-end automated testing platform used to uncover critical scenarios that conventional tests may miss. ### Source excerpt Testing is an essential part of building reliable software. It's a form of documentation, a reminder of mistakes of the past, and a boost of confidence when you want to refactor. But mostly, testing is a way of showing that your code is correct and resilient. Because it's so important, we've invested a lot of effort at Jane Street to develop techniques that make tests clearer, more effective, and more pleasant to write. ## Corrosion DevFeed: [Corrosion]() Original publisher: [Read original article]() Published: 2025-10-22T00:00:00Z Content type: article Language: en Sources: [The Fly Blog]() Topics: [distributed-systems](), [Concurrent Programming](), [Deadlock](), [Rust](), [Network](), [Orchestration](), [fly](), [fly.io]() Tags: [cdn](), [close-to-users](), [deadlock](), [deploy-app-servers](), [distributed-system](), [docker](), [docker-containers](), [elixir](), [fly](), [fly-io](), [heroku-alternative](), [heroku-competitor](), [hosting](), [i](), [networking](), [outage](), [pipelines](), [postgresql-clusters](), [routing](), [rust](), [servers](), [synchronization]() ### AI overview This article introduces Corrosion, Fly.io's open-source distributed state synchronization and service discovery system. It explains how Fly.io propagates workload and routing state across globally distributed servers and edge proxies, and recounts a severe outage caused by a Rust concurrency bug that triggered a contagious deadlock. The article also describes Fly.io's decentralized orchestration model, in which individual servers are authoritative for their workloads instead of relying on a centralized database. ### Source excerpt Fly.io transmogrifies Docker containers into Fly Machines: micro-VMs running on our own hardware all over the world. The hardest part of running this platform isn't managing the servers, and it isn't operating the network; it's gluing those two things together. Several times a second, as customer CI/CD pipelines tear up or bring down Fly Machines, our state synchronization system blasts updates across our internal mesh, so that edge proxies from Tokyo to Amsterdam can keep the accurate routing table that allows them to route requests for applications to the nearest customer instances. On September 1, 2024, at 3:30PM EST, a new Fly Machine came up with a new "virtual service" configuration option a developer had just shipped. Within a few seconds every proxy in our fleet had locked up hard. It was the worst outage we've experienced: a period during which no end-user requests could reach our customer apps at all. Distributed systems are blast amplifiers. By propagating data across a network, they also propagate bugs in the systems that depend on that data. In the case of Corrosion, our state distribution system, those bugs propagate quickly. The proxy code that handled that Corrosion update had succumbed to a notorious Rust concurrency footgun: an if let expression over an RWLock assumed (reasonably, but incorrectly) in its else branch that the lock had been released. Instant and virulently contagious deadlock. A lesson we've learned the hard way: never trust a distributed system without an interesting failure story. If a distributed system hasn't ruined a weekend or kept you up overnight, you don't understand it yet. Which is why that's how we're introducing Corrosion, an unconventional service discovery system we built for our platform and open sourced. Our Face-Seeking Rake State synchronization is the hardest problem in running a platform like ours. So why build a risky new distributed system for it? Because no matter what we try, that rake is waiting for our foot ## The Making of a Scalable URL Shortener DevFeed: [The Making of a Scalable URL Shortener]() Original publisher: [Read original article]() Author: Cyril Mottier Published: 2025-02-18T10:57:54Z Content type: article Language: en Sources: [Cyril Mottier Blog]() Topics: [Code](), [Amazon Web Services](), [DynamoDB](), [Go Language](), [Requirements](), [Database]() Tags: [article](), [aws](), [build](), [distributed-system](), [dynamodb](), [engineering](), [go](), [redirection](), [requirements](), [retry](), [scale]() ### AI overview The article describes the design of a custom URL shortener built with AWS Lambda, AWS DynamoDB, and Go. It outlines requirements such as fast redirection, short URLs, batch creation, and optional expiration, then examines slug-generation strategies including randomized slugs, incremental counters, and permutation of the slug space. ### Source excerpt This article was initially published as a thread of posts on X. For the sake of readability and long-term availability, it is now available here as a single-page article. Post 1 A few months ago, I built a custom URL shortener. Quick refresher: a URL shortener does two things: Creates short URLs that map to long URLs Redirects short URL requests to those long URLs. Designing this was a fantastic engineering challenge. Here's a look at my design process! ## Building Reliable Messaging Patterns in AWS with SQS and SNS DevFeed: [Building Reliable Messaging Patterns in AWS with SQS and SNS]() Original publisher: [Read original article]() Author: Guillermo Ojeda Published: 2024-12-20T15:20:38Z Content type: tutorial Language: en Sources: [Guille Ojeda]() Topics: [Amazon Simple Queue Service (SQS)](), [Amazon Web Services](), [Messaging](), [distributed-systems]() Tags: [architecture](), [aws](), [distributed-system](), [distributed-systems](), [error-handling](), [fifo](), [messaging](), [scale](), [sns](), [sqs](), [technical]() ### AI overview This tutorial explains how to build reliable distributed messaging systems with AWS SQS and SNS. It covers queue configuration, error handling, scaling patterns, message retention, long polling, visibility timeouts, and the differences between Standard and FIFO queues, using Node.js examples. ### Source excerpt Building distributed systems requires putting a lot of attention on communication between components. These components often need to exchange information asynchronously, and that's where message queues and pub/sub systems are the go-to solution. AWS ... ## Git Version Control Tutorial: A Beginner's Guide to Git Version Control DevFeed: [Git Version Control Tutorial: A Beginner's Guide to Git Version Control]() Original publisher: [Read original article]() Author: admin Published: 2024-05-03T07:28:35Z Content type: tutorial Language: en Sources: [Posts on Chris Banes]() Topics: [Git](), [version-control](), [Tutorial](), [coding](), [Development](), [Code]() Tags: [beginner](), [code](), [coding](), [collaboration](), [commands](), [commit](), [core](), [developers](), [distributed](), [distributed-system](), [efficiency](), [files](), [git](), [guide](), [machine](), [merge](), [projects](), [repository](), [tutorial](), [uncategorized](), [version-control]() ### AI overview A beginner-focused tutorial explaining Git as a distributed version control system for tracking changes, managing repositories, creating commits and branches, and collaborating with other developers. ### Source excerpt Have you ever worked on a project and accidentally overwritten your progress? Maybe you spent hours coding a new feature, then realized you needed to go back to a previous version. This is where git version control can help! Git is a lifesaver for developers, allowing you to track changes, collaborate effectively, and ensure you [...] ## Measuring uptime for Neon's multi-tenant architecture DevFeed: [Measuring uptime for Neon's multi-tenant architecture]() Original publisher: [Read original article]() Author: Stas Kelvich Published: 2023-12-08T21:49:03Z Content type: article Language: en Sources: [Blog -- Neon Docs]() Topics: [Multitenancy](), [incident](), [Monitoring](), [Incident response](), [systems](), [Cloud](), [Databases]() Tags: [architecture](), [cloud-native](), [community](), [databases](), [distributed](), [distributed-system](), [engineering](), [incident](), [incident-response](), [monitoring](), [monitoring-alerting](), [platform](), [postgres](), [storage](), [uptime]() ### AI overview Neon explains why it removed the default uptime percentage from its status page and is exploring project-level uptime ranges to better represent service health across its multi-tenant distributed system. ### Source excerpt In the past two months, we've had several incidents that affected different aspects of our service. We've been transparent about them, and we made sure that each major one included details on what happened, the size of the impact, and what we're doing to prevent it from happening... ## Run a Node Grants Round Grantee Announcement DevFeed: [Run a Node Grants Round Grantee Announcement]() Original publisher: [Read original article]() Author: Ethereum Foundation Ecosystem Support Program Published: 2023-12-06T00:00:00Z Content type: release Language: en Sources: [Ethereum Foundation Blog]() Topics: [Ethereum](), [Blockchain](), [Network](), [Remote Procedure Call (RPC)]() Tags: [africa](), [announcement](), [blockchain](), [build](), [developers](), [distributed-system](), [ecosystem-support-program](), [ethereum](), [internet](), [rpc](), [security](), [testing](), [university]() ### AI overview The Run A Node Grants Round selected 35 recipients from 23 countries from 243 applications. The grants support projects using Ethereum nodes for community building, research, data access, performance evaluation, education, and related activities beyond validator operation. ### Source excerpt The Run A Node Grants Round wanted to reach different users exploring blockchain, learning, community efforts, and research purposes. The aim of this round was to encourage the exploration of a diverse range of ideas for operating nodes beyond running a validator, such as facilitating data queries for research, examining... ## Gossip Protocol Explained DevFeed: [Gossip Protocol Explained]() Original publisher: [Read original article]() Author: NK Published: 2023-07-16T16:38:02Z Content type: article Language: en Sources: [High Scalability](), [High Scalability]() Topics: [Protocol (disambiguation)](), [distributed-systems](), [P2P](), [Availability](), [consistency](), [Scalability](), [Resilience](), [Network]() Tags: [availability](), [consistency](), [distributed](), [distributed-algorithm](), [distributed-system](), [distributed-systems](), [high-scalability](), [messages](), [network](), [protocol](), [resilience]() ### AI overview This article explains the gossip protocol as a peer-to-peer approach for managing state and communicating between nodes in distributed systems. It describes its relationship to high availability, eventual consistency, scalability, and resilience, and compares it with centralized state management, point-to-point broadcast, and eager reliable broadcast. ### Source excerpt You can subscribe to the system design newsletter to excel in system design interviews and software architecture. The original article was published on systemdesign.one website. What Is Gossip Protocol? The typical problems in a distributed system are the following [1], [11]: maintaining the system state (liveness of nodes) communication between nodes The potential solutions to these problems are as follows [1]: centralized state management service peer-to-peer state management service ## Gossip Protocol Explained DevFeed: [Gossip Protocol Explained]() Original publisher: [Read original article]() Author: NK Published: 2023-07-16T15:38:02Z Content type: tutorial Language: en Sources: [High Scalability]() Topics: [distributed-systems](), [Protocol (disambiguation)](), [Availability](), [consistency](), [Scalability](), [Resilience]() Tags: [architecture](), [availability](), [consistency](), [distributed-algorithm](), [distributed-system](), [distributed-systems](), [messages](), [protocol](), [resilience](), [scalability]() ### AI overview This article explains the gossip protocol as a peer-to-peer approach for maintaining state and communicating in distributed systems. It contrasts gossip-based state management with centralized services such as Apache Zookeeper and discusses availability, consistency, scalability, resilience, and message-broadcasting techniques. ### Source excerpt You can subscribe to the system design newsletter to excel in system design interviews and software architecture. The original article was published on systemdesign.one website. What Is Gossip Protocol? The typical problems in a distributed system are the following [1], [11]: maintaining the system state (liveness of nodes) communication ## Consistent hashing algorithm DevFeed: [Consistent hashing algorithm]() Original publisher: [Read original article]() Author: NK Published: 2023-02-22T16:39:15Z Content type: tutorial Language: en Sources: [High Scalability]() Topics: [consistent hashing](), [Algorithm](), [distributed-systems](), [Cache](), [Scalability](), [Server]() Tags: [algorithm](), [architecture](), [cache](), [consistent-hashing](), [distributed-system](), [hashing](), [high-availability](), [load-balancing](), [low-latency](), [scalability](), [sharding](), [system-design]() ### AI overview The article explains how consistent hashing places nodes and data keys on a virtual hash ring, then assigns data by traversing the ring clockwise to the next node. It discusses cache-server scaling, hotspot reduction, dynamic load, replication, availability, and latency in distributed systems. ### Source excerpt You can subscribe to the system design newsletter to excel in system design interviews and software architecture. You can view the original article Consistent hashing explained on systemdesign.one website. How does consistent hashing work? At a high level, consistent hashing performs the following operations: The output of the hash ## Consistent hashing algorithm DevFeed: [Consistent hashing algorithm]() Original publisher: [Read original article]() Author: NK Published: 2023-02-22T16:39:15Z Content type: tutorial Language: en Sources: [High Scalability](), [High Scalability]() Topics: [hashing](), [Algorithm](), [Cache](), [Scalability](), [distributed-systems](), [Software](), [Availability]() Tags: [algorithm](), [availability](), [cache](), [consistent-hashing](), [distributed-system](), [hashing](), [hotspot](), [load-balancing](), [low-latency](), [scalability](), [sharding](), [software-architecture]() ### AI overview A tutorial explaining consistent hashing: nodes and data keys are mapped onto a virtual hash ring, and clockwise traversal assigns data to nodes. It discusses terminology, cache-server scaling requirements, hotspot reduction, dynamic load, replication, availability, and latency. ### Source excerpt This is a guest article by NK. You can view the original article Consistent hashing explained on systemdesign.one website. How does consistent hashing work? At a high level, consistent hashing performs the following operations: The output of the hash function is placed on a virtual ring structure (known as the hash ring) The hashed IP addresses of the nodes are used to assign a position for the nodes on the hash ring The key of a data object is hashed using the same hash function to find the position of the key on the hash ring The hash ring is traversed in the clockwise direction starting from the position of the key until a node is found The data object is stored or retrieved from the node that was found Terminology ## How we used Go 1.18 when designing our Identifiers DevFeed: [How we used Go 1.18 when designing our Identifiers]() Original publisher: [Read original article]() Author: Dominic Black Published: 2022-03-25T00:00:00Z Content type: article Language: en Sources: [Encore Updates]() Topics: [Go Language](), [Architecture & Design](), [data](), [Traces](), [Amazon Web Services]() Tags: [api](), [aws](), [complexity](), [database](), [debug](), [distributed-system](), [go](), [identifier](), [security](), [traces]() ### AI overview This article explains how Encore considered identifier design in distributed systems while using Go 1.18. It discusses the limitations of auto-incrementing identifiers and outlines requirements including sortability, scalability, collision avoidance, zero configuration, and type safety. ### Source excerpt It turns out identifiers were generic the whole time ## Why the author stopped using the term "actors" when explaining the Actor Model DevFeed: [Why the author stopped using the term "actors" when explaining the Actor Model]() Original publisher: [Read original article]() Author: Sergey Bykov Published: 2021-05-05T07:00:00Z Content type: opinion Language: en Sources: [Temporal Blog]() Topics: [Actor](), [Concurrency](), [Scalability](), [.NET](), [Java]() Tags: [actor](), [c-sharp](), [concurrency](), [distributed-system](), [java](), [net](), [temporal-voices]() ### AI overview The author explains why he stopped using the term "actors" when discussing the Actor Model, focusing on conflation between local and distributed actors and their differing roles in concurrency. ### Source excerpt While it may have taken me quite some time to get around to answering it, 2020 wasn't an ordinary year by any measure. ## Dealing with failure DevFeed: [Dealing with failure]() Original publisher: [Read original article]() Author: Sergey Bykov Published: 2021-01-21T04:00:00Z Content type: opinion Language: en Sources: [Temporal Blog]() Topics: [Remote Procedure Call (RPC)](), [retry](), [HTTP](), [client](), [servers](), [systems]() Tags: [conference](), [distributed-system](), [error](), [how-to](), [http](), [opinion](), [retry](), [rpc]() ### AI overview The article examines how to handle failures in cloud and distributed systems, focusing on request-reply or RPC, persistent queues, and workflows. It explains the simplicity and limitations of RPC, including timeouts, ambiguous outcomes, retries, unavailable systems, and partial failures. ### Source excerpt I recently gave a talk at the CodeMesh conference, and I spent half of it reflecting on the seemingly boring topic of dealing with failures. ## Flow Control in Aeron DevFeed: [Flow Control in Aeron]() Original publisher: [Read original article]() Author: Michael Barker (noreply@blogger.com) Published: 2020-03-19T22:20:00Z Content type: tutorial Language: en Sources: [Bad Concurrency]() Topics: [networking](), [Messaging](), [Protocol (disambiguation)](), [Development]() Tags: [distributed-system](), [messaging](), [multicast](), [networking](), [protocol](), [udp]() ### AI overview This article explains flow control in Aeron, focusing on sliding-window behavior over UDP. It compares Aeron's receiver status messages with TCP acknowledgements and describes how receiver-window limits help prevent senders from overrunning receivers and causing message loss. ### Source excerpt One of my more recent projects has led me to become more involved in the Aeron project. If you are unaware of Aeron, then head over to the Github site and check it out. At its core is an reliable messaging system that works over UDP, Multicast UDP and IPC. It also contains an archiving feature for recording and replay and (still under active development) an implementation of the Raft protocol for clustering. Did I mention that it was fast too. I've spent the last few weeks buried in the various strategies the Aeron has for flow control. Specifically modifying the existing flow control strategies and adding more flexible configuration on a per channel basis. Before I jump into that it would be useful to cover a little background first. What is flow control? Within a distributed system the purpose of flow control is to limit the rate of a sender so that is does not overrun it's associated receiver. UDP does not come with any form of flow control, therefore it is easy to create a sender that will out pace the receiver, leading to message loss. There are a number of different forms of flow control, but I'm going to focus on the sliding window flow control protocol used by TCP and Aeron. The sliding window protocol requires that the sender maintain a buffer of data (referred to as a window). The size of this window will typically communicated from the receiver to the sender as part of the protocol. With a bi-directional protocol like TCP the size of the window is communicated in each TCP segment header. This is the amount of data that the sender can transmit to the receiver before having to wait until an acknowledgement is received. If the application thread on the receiver side is busy and does not read the data from the socket and the sender continues to transmit, the window size value will decrease until it reaches 0, at which time the sender must stop and wait for an acknowledgement with a non-zero window size before sending again. There is a lot more networking theo ## Eth2 Consensus Mechanisms: GHOST and LMD-GHOST Fork Choice DevFeed: [Eth2 Consensus Mechanisms: GHOST and LMD-GHOST Fork Choice]() Original publisher: [Read original article]() Author: Carl Beekhuizen Published: 2020-02-12T00:00:00Z Content type: tutorial Language: en Sources: [Ethereum Foundation Blog]() Topics: [Algorithm](), [Protocol (disambiguation)](), [Network](), [Latency]() Tags: [algorithm](), [consensus](), [distributed-system](), [latency](), [network](), [pos](), [protocol](), [research-development]() ### AI overview This article explains the consensus mechanisms behind eth2, focusing on GHOST and its proof-of-stake variant, Latest Message Driven GHOST (LMD-GHOST). It describes how LMD-GHOST uses each validator's latest vote to select the chain head and how GHOST can reduce attack effectiveness and chain reorganization depth under high network latency. ### Source excerpt Special thanks to Sacha Yves Saint-Leger & Danny Ryan for review. In this installment, we'll discuss the consensus mechanisms behind eth2. Eth2 has a novel approach to deciding which block is the head of the chain, along with which blocks are and are not a part of the chain.... ## Open-Sourcing Panoptes, Oath's distributed network telemetry collector DevFeed: [Open-Sourcing Panoptes, Oath's distributed network telemetry collector]() Original publisher: [Read original article]() Author: amberwilsonla-blog Published: 2018-10-04T20:59:00Z Content type: release Language: en Sources: [Yahoo]() Topics: [telemetry](), [Network](), [Kafka](), [Open Source](), [Development](), [Python](), [Scalability](), [Redis]() Tags: [distributed-system](), [kafka](), [network](), [open-source](), [python](), [redis](), [scalability](), [telemetry](), [yahoo-engineering]() ### AI overview The Oath network automation team announces the open-sourcing of Panoptes, a distributed system for collecting, enriching, and distributing network telemetry. The article describes its pluggable polling and discovery architecture, Kafka-based metrics bus, Python implementation, and Celery scheduling with Redis. ### Source excerpt yahoodevelopers: By Ian Flint, Network Automation Architect and Varun Varma, Senior Principal Engineer The Oath network automation team is proud to announce that we are open-sourcing Panoptes, a distributed system for collecting, enriching and distributing network telemetry. We developed Panoptes to address several issues inherent in legacy polling systems, including overpolling due to multiple point solutions for metrics, a lack of data normalization, consistent data enrichment and integration with infrastructure discovery systems. Panoptes is a pluggable, distributed, high-performance data collection system which supports multiple polling formats, including SNMP and vendor-specific APIs. It is also extensible to support emerging streaming telemetry standards including gNMI. Architecture The following block diagram shows the major components of Panoptes: Panoptes is written primarily in Python, and leverages multiple open-source technologies to provide the most value for the least development effort. At the center of Panoptes is a metrics bus implemented on Kafka. All data plane transactions flow across this bus; discovery publishes devices to the bus, polling publishes metrics to the bus, and numerous clients read the data off of the bus for additional processing and forwarding. This architecture enables easy data distribution and integration with other systems. For example, in preparing for open-source, we identified a need for a generally available time series datastore. We developed, tested and released a plugin to push metrics into InfluxDB in under a week. This flexibility allows Panoptes to evolve with industry standards. Check scheduling is accomplished using Celery, a horizontally scalable, open-source scheduler utilizing a Redis data store. Celery's scalable nature combined with Panoptes' distributed nature yields excellent scalability. Across Oath, Panoptes currently runs hundreds of thousands of checks per second, and the infrastructure has been tested [Next page]()