# Some risks of coordinating only sometimes

DevFeed: [Some risks of coordinating only sometimes](<https://devfeed.tech/articles/some-risks-of-coordinating-only-sometimes-12486.md>)

Original publisher: [Read original article](<http://brooker.co.za/blog/2019/05/01/emergent.html>)

Author: Marc Brooker

Published: 2019-05-01T00:00:00Z

Content type: article

Language: en

Sources: [Marc Brooker's Blog](<https://devfeed.tech/sources/marc-brooker-s-blog.md>), [Marc Brooker's Blog](<https://devfeed.tech/sources/marc-brooker-s-blog-2.md>)

Topics: [Cloud Architecture](<https://devfeed.tech/topics/cloud-architecture.md>), [systems](<https://devfeed.tech/topics/systems.md>), [Availability](<https://devfeed.tech/topics/availability.md>), [Latency](<https://devfeed.tech/topics/latency.md>), [API](<https://devfeed.tech/topics/api.md>), [Network](<https://devfeed.tech/topics/network.md>)

Tags: [api](<https://devfeed.tech/tags/api.md>), [architecture](<https://devfeed.tech/tags/architecture.md>), [availability](<https://devfeed.tech/tags/availability.md>), [bugs](<https://devfeed.tech/tags/bugs.md>), [cloud-architecture](<https://devfeed.tech/tags/cloud-architecture.md>), [code](<https://devfeed.tech/tags/code.md>), [latency](<https://devfeed.tech/tags/latency.md>), [network](<https://devfeed.tech/tags/network.md>), [operator](<https://devfeed.tech/tags/operator.md>), [outages](<https://devfeed.tech/tags/outages.md>), [recovery](<https://devfeed.tech/tags/recovery.md>)

## AI overview

The article examines risks in cloud systems that coordinate only intermittently. It explains how correlated failures can trigger sudden coordination and traffic bursts, overload controllers, increase recovery time, and cause large-scale outages.

## Source excerpt

Some risks of coordinating only sometimes Sometimes-coordinating systems have dangerous emergent behaviors A classic cloud architecture is built of small clusters of nodes (typically one to nine1), with coordination used inside each cluster to provide availability, durability and integrity in the face of node failures. Coordination between clusters is avoided, making it easier to scale the system while meeting tight availability and latency requirements. In reality, however, systems sometimes do need to coordinate between clusters, or clusters need to coordinate with a central controller. Some of these circumstances are operational, such as around adding or removing capacity. Others are triggered by the application, where the need to present a client API which appears consistent requires either the system itself, or a layer above it, to coordinate across otherwise-uncoordinated clusters. The costs and risks of re-introducing coordination to handle API requests or provide strong client guarantees are well explored in the literature. Unfortunately, other aspects of sometimes-coordinated systems do not get as much attention, and many designs are not robust in cases where coordination is required for large-scale operations. Results like CAP and CALM2 provide clear tools for thinking through when coordination must occur, but offer little help in understanding the dynamic behavior of the system when it does occur. One example of this problem is reacting to correlated failures. At scale, uncorrelated node failures happen all the time. Designing to handle them is straightforward, as the code and design is continuously validated in production. Large-scale correlated failures also happen, triggered by power and network failures, offered load, software bugs, operator mistakes, and all manner of unlikely events. If systems are designed to coordinate during failure handling, either as a mesh or by falling back to a controller, these correlated failures bring sudden bursts of coo