# acid Published articles for acid. This is one page of public article previews, not the complete archive. Follow Next page to continue. Summaries are not the original full articles. ## Why FinTech Is Moving to Distributed SQL Databases DevFeed: [Why FinTech Is Moving to Distributed SQL Databases]() Original publisher: [Read original article]() Author: David Weiss Published: 2025-12-12T00:00:00Z Content type: article Language: en Sources: [Cockroach Labs]() Topics: [Databases](), [SQL](), [systems](), [Transactions](), [consistency](), [Resilience](), [Availability]() Tags: [acid](), [availability](), [compliance](), [consistency](), [databases](), [distributed](), [fintech](), [global](), [legacy](), [nosql](), [payments](), [resilience](), [sql](), [trading](), [transactions]() ### AI overview This article explains why FinTech organizations are considering distributed SQL databases as they scale financial workloads across regions and clouds. It emphasizes correctness, consistent ACID transactions, resilience, availability, regulatory compliance, and the limitations of legacy relational and patchwork NoSQL systems. ### Source excerpt FinTech teams are operating under more pressure than ever. As digital financial services become the default for consumers and enterprises alike, the margin for error has all but disappeared. Users expect every interaction to be instantaneous, accurate, and always available, with very little patience for anything less. ## Data Modeling for Java Developers: Structuring With PostgreSQL and MongoDB DevFeed: [Data Modeling for Java Developers: Structuring With PostgreSQL and MongoDB]() Original publisher: [Read original article]() Author: Aasawari Sahasrabuddhe Published: 2025-04-27T12:00:00Z Content type: tutorial Language: en Sources: [Nic Raboy]() Topics: [data-modeling](), [Java](), [MongoDB](), [PostgreSQL](), [Databases](), [Application Development]() Tags: [acid](), [application-development](), [data-modeling](), [java](), [mongodb](), [postgres](), [postgresql](), [tutorial](), [vs]() ### AI overview This tutorial explains data modeling for Java developers by comparing PostgreSQL's relational approach with MongoDB's document-oriented model. It covers database relationships, including many-to-many relationships, and discusses trade-offs such as rigid schemas, ACID compliance, and scalability. ### Source excerpt Application and system designs have always been considered the most essential step in application development. All the later steps and technologies to be used depend on how the system has been designe... The post Data Modeling for Java Developers: Structuring With PostgreSQL and MongoDB appeared first on DEV. ## SQL vs NoSQL Explained DevFeed: [SQL vs NoSQL Explained]() Original publisher: [Read original article]() Author: Nelson Djalo Published: 2024-07-02T16:00:57Z Content type: comparison Language: en Sources: [Amigoscode Newsletter]() Topics: [NoSQL](), [SQL](), [Databases](), [clickhouse]() Tags: [acid](), [business-intelligence](), [consistency](), [data-analysis](), [database](), [database-scalability](), [durability](), [microsoft](), [mysql](), [nosql](), [olap](), [oracle](), [postgresql](), [rdbms](), [relational-database-management-systems-rdbms](), [relational-databases](), [schema](), [sql]() ### AI overview This comparison explains the differences between SQL and NoSQL databases, including their characteristics, scalability models, and common use cases. It covers SQL database features such as structured schemas and ACID compliance, along with relational and OLAP database categories. ### Source excerpt Choosing the Right Database for Your Needs ## DynamoDB Transactions: An E-Commerce with Amazon DynamoDB DevFeed: [DynamoDB Transactions: An E-Commerce with Amazon DynamoDB]() Original publisher: [Read original article]() Author: Guillermo Ojeda Published: 2023-11-09T18:42:01Z Content type: tutorial Language: en Sources: [Guille Ojeda]() Topics: [Amazon DynamoDB](), [Database](), [NoSQL](), [Amazon Web Services](), [Back end]() Tags: [acid](), [amazon-dynamodb](), [amazon-web-services](), [architecture](), [aws](), [backend](), [customers](), [database](), [databases](), [dynamodb](), [e-commerce](), [nosql](), [operations](), [products]() ### AI overview This tutorial explains how to use DynamoDB transactions in an e-commerce application to atomically verify product stock, decrement inventory, and create an order. It describes the race condition that can occur when separate read and write operations process simultaneous purchases. ### Source excerpt We're building an e-commerce app with DynamoDB for the database, pretty similar to the one we built for the DynamoDB Database Design article. No need to go read that issue (though I think it came up great), here's how our database works: Customers a... ## 25 key terms for speaking Distributed Systems and Temporal DevFeed: [25 key terms for speaking Distributed Systems and Temporal]() Original publisher: [Read original article]() Author: Emily Fortuna Published: 2023-06-29T16:00:00Z Content type: tutorial Language: en Sources: [Temporal Blog]() Topics: [distributed-systems](), [CAP theorem](), [Concurrency](), [consistency](), [reliability](), [Scalability](), [Database]() Tags: [acid](), [availability](), [cap-theorem](), [concurrency](), [consistency](), [distributed-systems](), [reliability](), [scalability](), [systems](), [temporal-concepts]() ### AI overview A quick-reference guide explains core distributed-systems terms, including concurrency, scalability, reliability, consistency models, the CAP theorem, availability, partition tolerance, and ACID. It is intended to help readers understand and discuss distributed systems and Temporal. ### Source excerpt Learn the minimum set of terms to understand and discuss distributed systems challenges and Temporal with all your friends, in plain English. ## Pinion -- The Load Framework Part-2 DevFeed: [Pinion -- The Load Framework Part-2]() Original publisher: [Read original article]() Author: Saurabh Jain Published: 2021-10-29T16:50:24Z Content type: article Language: en Sources: [Groupon Engineering -- Medium]() Topics: [data lake](), [Framework](), [Data Quality](), [Transactions](), [Streaming](), [Open Source]() Tags: [acid](), [audit](), [aws](), [cloud](), [data-validation](), [delta-lake](), [deltalake](), [hdfs](), [logging](), [optimization](), [reporting](), [s3](), [schema](), [science](), [spark](), [streaming](), [transactions]() ### AI overview This second post in the Pinion -- The Load Framework series explains how Pinion extends Apache Delta Lake APIs for slowly changing dimension operations. It covers data validation, compaction, auditing, streamlined logging, and chained APIs, and introduces Delta Lake capabilities such as ACID transactions, schema enforcement, batch and streaming interfaces, and time travel. ### Source excerpt Pinion -- The Load Framework Part-2 This post is the 2nd part of the "Pinion -- The Load Framework" series. In case you have not read the 1st post, you can read it here. In this post, we are going to cover the following topics. How does Pinion use Delta Lake for SCD operations? Small file problem with Delta Lake and its resolution. Before we dive into the topics of this post, let's look at the definition of DeltaLake to set the context right. Apache Delta Lake - Apache Delta Lake is an open-source framework that enables the addition of ACID transactions support to a new data lake or an existing data lake created on top of S3, GCS, and HDFS. In addition to this, it provides other features such as scalable metadata handling, unified interface for both batch and streaming application, schema enforcement, time travel, and a rich interface of APIs to enable complex use cases like change-data-capture (CDC) and slowly-changing-dimension (SCD) operations. To keep the post concise and to the point, I won't go into much detail here about Delta Lake, since there is already great documentation available about it, that you can read it here. How does Pinion use Delta Lake for SCD operations? - Apache Delta Lake provides a rich set of APIs to handle slowly-changing dimensions, however, those APIs were not enough alone to build the features that we want to have in The Pinion Framework. So, we decided to enrich the APIs provided by Delta Lake by adding the following features to it: Data Validation Compaction Audit Streamlined logging infrastructure to make the data engineer's life easier during debugging of a failed job Chained APIs Let's dive a little further into the features that we had listed above. Data Validation -- By default, schema enforcement is enabled in Pinion for all the APIs where we have a need of inserting rows from source data(LRFs) into the target table. In case of a schema mismatch, Pinion raises an error and stops processing of further stages. It ensures the data a ## PostgreSQL as a Microservice DevFeed: [PostgreSQL as a Microservice]() Original publisher: [Read original article]() Author: Dimitri Fontaine PostgreSQL Major Contributor; Author Published: 2021-06-08T11:40:00Z Content type: opinion Language: en Sources: [Dimitri Fontaine]() Topics: [PostgreSQL](), [Concurrency](), [Microservice](), [Architecture & Design](), [acid](), [rdbms]() Tags: [acid](), [architecture](), [concurrency](), [microservices](), [postgresql](), [rdbms]() ### AI overview The article examines PostgreSQL as a potential microservice and argues that its distinctive value is not merely storage. It emphasizes PostgreSQL's role in managing concurrent access to data while providing guarantees such as ACID properties, schemas, and constraints. ### Source excerpt The MACI French podcast honoured me with an invitation to a guest appearance on their weekly schedule. As you can imagine, we talked about many things related to PostgreSQL... and also reacted to some newsworthy articles carefully curated by the MACI team. One of the topics we discussed in the podcast started with looking at PostgreSQL through the angle of it being one of the microservices that your application would be composed of. ## Some opinionated thoughts on SQL databases DevFeed: [Some opinionated thoughts on SQL databases]() Original publisher: [Read original article]() Author: Nelson Elhage Published: 2021-03-30T17:32:31Z Content type: opinion Language: en Sources: [Nelson Elhage]() Topics: [Databases](), [SQL](), [MySQL](), [PostgreSQL](), [SQLite](), [backends](), [Web APIs](), [Latency]() Tags: [acid](), [backends](), [consistency](), [databases](), [durability](), [high-availability](), [low-latency](), [mysql](), [open-source](), [postgresql](), [sql](), [sqlite](), [web-apis]() ### AI overview An opinionated discussion of SQL databases, focused mainly on MySQL and PostgreSQL, with some coverage of SQLite. Drawing on experience using databases as backends for web applications and APIs, the article argues that open-source SQL databases provide excellent storage engines, throughput, durability, and transactional guarantees when properly tuned. ### Source excerpt People who work with me tend to realize that I have Opinions about databases, and SQL databases in particular. Last week, I wrote about a Postgres debugging story and tweeted about AWS' policy ban on internal use of SQL databases, and had occasion to discuss and debate some of those feelings on Twitter; this article is an attempt to write up more of them into a single place I can refer to. ## Dgraph Database Semantics DevFeed: [Dgraph Database Semantics]() Original publisher: [Read original article]() Published: 2020-06-26T00:00:00Z Content type: tutorial Language: en Sources: [William Kennedy]() Topics: [Database](), [graph-database](), [Scalability](), [Architecture & Design](), [Latency](), [Availability](), [Replication](), [Transactions](), [Resilience]() Tags: [acid](), [architecture](), [ardan-labs](), [availability](), [blog](), [database](), [distributed](), [engineering](), [go](), [go-programming](), [golang](), [graph-database](), [high-availability](), [latency](), [low-latency](), [programming](), [replication](), [resilience](), [scalability](), [servers](), [sharding](), [software](), [transactions]() ### AI overview This article explains the semantics behind Dgraph's description as a distributed graph database. It provides a high-level mental model for claims involving horizontal scalability, cluster-wide ACID transactions, low-latency arbitrary-depth joins, synchronous replication, high availability, and crash resilience, while leaving lower-level mechanics to the referenced paper. ### Source excerpt Introduction In this paper written by Manish Jain (the founder of Dgraph) he describes Dgraph as: a distributed graph database which provides horizontal scalability, distributed cluster-wide ACID transactions, low-latency arbitrary-depth joins, synchronous replication, high availability, and crash resilience. There are many claims being stated here which frankly I didn't understand the meaning of the first time I read this sentence. I decided I wanted to better understand these claims with the hope that it would help me understand Dgraph's architecture and engineering decisions better. ## PostgreSQL Concurrency: an Article Series DevFeed: [PostgreSQL Concurrency: an Article Series]() Original publisher: [Read original article]() Author: Dimitri Fontaine PostgreSQL Major Contributor; Author Published: 2018-08-14T08:49:02Z Content type: tutorial Language: en Sources: [Dimitri Fontaine]() Topics: [Concurrency](), [PostgreSQL](), [Databases](), [consistency](), [Open Source]() Tags: [acid](), [article](), [cache-invalidation](), [caching](), [concurrency](), [consistency](), [database](), [isolation-level](), [locking](), [postgresql](), [series](), [sql](), [transactions](), [triggers]() ### AI overview This article series explains how PostgreSQL handles concurrent access while maintaining data consistency. It covers data modification, isolation levels, locking, ACID transactions, concurrency-oriented data modeling, event processing, materialized views for caching, and triggers for transactionally correct cache maintenance. ### Source excerpt PostgreSQL is a relational database management system. It's even the world's most advanced open source one of them. As such, as its core, Postgres solves concurrent access to a set of data and maintains consistency while allowing concurrent operations. In the PostgreSQL Concurrency series of articles here we did see several aspects of how to handle concurrent use cases of your application design with PostgreSQL. The main thing to remember is that a Database Management System first task is to handle concurrency access to the data for you. ## PostgreSQL Concurrency: Isolation and Locking DevFeed: [PostgreSQL Concurrency: Isolation and Locking]() Original publisher: [Read original article]() Author: Dimitri Fontaine PostgreSQL Major Contributor; Author Published: 2018-07-03T11:30:13Z Content type: tutorial Language: en Sources: [Dimitri Fontaine]() Topics: [PostgreSQL](), [Concurrency](), [consistency](), [Transactions](), [acid]() Tags: [acid](), [concurrency](), [consistency](), [locking](), [pg-dump](), [postgresql](), [transactions]() ### AI overview This primer explains PostgreSQL isolation and locking as mechanisms for handling concurrent access while maintaining consistency. It discusses ACID-compliant transactions, isolation levels, locking control, and the snapshot behavior used by pg_dump for backups. ### Source excerpt PostgreSQL is a relational database management system. It's even the world's most advanced open source one of them. As such, as its core, Postgres solves concurrent access to a set of data and maintains consistency while allowing concurrent operations. This article is a primer on PostgreSQL Isolation and Locking properties and behaviors. You might be interested into the previous article in the series: PostgreSQL Concurrency: Data Modification Language. ## Mastering PostgreSQL in Application Development DevFeed: [Mastering PostgreSQL in Application Development]() Original publisher: [Read original article]() Published: 2017-05-21T23:16:47Z Content type: article Language: en Sources: [Dimitri Fontaine]() Topics: [PostgreSQL](), [Application Development](), [Development](), [Object-relational mapping](), [SQL](), [Databases]() Tags: [acid](), [application-development](), [database](), [development](), [orm](), [postgresql](), [properties](), [sql]() ### AI overview An overview of a book about using PostgreSQL in application development. It addresses how much work a database should handle, whether an ORM is always the best option, maintaining SQL, and the role of PostgreSQL as remote storage with ACID properties. ### Source excerpt I am writing a book to share my knowledge about how to best use PostgreSQL as an application developer. If you've evered ask yourself how much should I have my database do for me? this book is for you. If you're thinking that using an ORM might not always be the best option out there, if sometimes you wish SQL was easier to maintain than those concatenated strings with conditionals, then this book can't wait for you! ## You Do It Too: Forfeiting Network Partition Tolerance in Distributed Systems DevFeed: [You Do It Too: Forfeiting Network Partition Tolerance in Distributed Systems]() Original publisher: [Read original article]() Author: Nicolas Liochon (noreply@blogger.com) Published: 2015-07-06T17:25:00Z Content type: article Language: en Sources: [Nicolas Liochon]() Topics: [distributed-systems](), [systems](), [Availability]() Tags: [acid](), [availability](), [cap-theorem](), [consistency](), [database](), [distributed-system](), [distributed-systems](), [durability](), [latency](), [network](), [partition](), [protocol]() ### AI overview This article examines the CA category in the CAP theorem, arguing that systems can forfeit network partition tolerance and that doing so may sometimes be appropriate. It reviews the category's academic history, the overlap between CA and CP, and the need for recovery mechanisms when partitions occur. ### Source excerpt "64-node clusters ought to be enough for anybody." Bill Gates, quoted from memory. This post is part of the CAP theorem series. You may want to start by my post on ACID vs. CAP if you have a database background but have never really been exposed to the CAP theorem. The post discussing some traps in the 'Availability' and 'Consistency' definition of CAP could also be used as an introduction if you know CAP but haven't looked at its formal definition. The CA--consistent, available, but not network partition tolerant--category in CAP has a very specific history. Not only forfeiting "network partition tolerance" can be understood as impossible in theory and crazy in practice (P as an illusion of a choice), but there is also an overlap between the CA and CP categories. As a result, many consider that it's impossible to build a production CA system. But you can actually build a system without network partition tolerance, and sometimes you should. A brief history of the CA category Let's look at the academic history of the CA category in CAP: In 2000, Eric Brewer presents the CAP conjecture. In his presentation, CA exists, for example for systems using the two-phase commit protocol. He considers that "‹the whole space is useful." In 2002, Seth Gilbert and Nancy Lynch publish the CAP proof. CA exists: "Systems that run on intranets and LANs are an example of these types of algorithms." In 2010, Daniel Abadi raises the point that there is an overlap between CA and CP: "What does 'not tolerant' mean? In practice, it means that they lose availability if there is a partition. Hence CP and CA are essentially identical." Still in 2010, Michael Stonebraker publishes multiple documents around the limited importance of partitions, with the tagline "Myth #6: In CAP, choose AP over CA", considering that with the capacity of modern hardware, small distributed systems can solve most real-life issues, and that "it doesn't much matter what you do when confronted with network partitions." ## If CAP were real-time: adding timing requirements to the definition of availability DevFeed: [If CAP were real-time: adding timing requirements to the definition of availability]() Original publisher: [Read original article]() Author: Nicolas Liochon (noreply@blogger.com) Published: 2015-06-14T18:20:00Z Content type: article Language: en Sources: [Nicolas Liochon]() Topics: [Availability](), [Networks](), [Protocol (disambiguation)](), [Algorithms, Complexity]() Tags: [acid](), [asynchronous](), [availability](), [cap-theorem](), [consistency](), [database](), [distributed-system](), [durability](), [latency](), [network](), [partition](), [protocol](), [real-time](), [tcp]() ### AI overview This article explains why delays are often misunderstood when applying the CAP theorem. It distinguishes the asynchronous network model used in the CAP proof from TCP and argues that adding timing requirements to CAP can make the model applicable to delays as well as partitions. ### Source excerpt "If you are not too long, I will wait here for you all my socketReadTimeout." - Oscar Wilde, quoted from memory. This post is part of the CAP theorem series. You may want to start by my post on ACID vs. CAP if you have a database background but have never really been exposed to the CAP theorem. The post discussing some traps in the 'Availability' and 'Consistency' definition of CAP could also be used as an introduction if you know CAP but haven't looked at its formal definition. Delays are a hot source of confusion when using CAP. Confusion arises out of the understanding of what an asynchronous network--the network model used in the CAP proof--actually is. Confusion arises out of what TCP is exactly. Confusion arises out of the definition of availability in CAP--which does not have any timing requirement. In this post, I'm not only going to cover why speaking about delays when the network is asynchronous is a mistake or why TCP is more than an asynchronous network: I will also show that adding timing requirements to CAP makes it usable not only for partitions but also for delays. Why are there never any delays in academic networks? The CAP proof by Gilbert and Lynch uses an "asynchronous model, in which system components take steps at arbitrary speeds." The key point is that there can be no delay, because there is absolutely no timing guarantee, so a message is never late. That's by design, as stated by Lynch: "algorithms designed for the asynchronous model are general and portable, in that they are guaranteed to run correctly in networks with arbitrary timing behavior." On such a network model, you cannot build an application that needs a guaranteed response time. You cannot even provide an approximation of its expected response time. You can at best prove that the application will answer... someday. You can calculate the number of messages required by a given protocol, but you cannot, in any case, calculate a response time. Such a network is never sufficient to wri ## CAP: if all you have is a timeout, everything looks like a partition DevFeed: [CAP: if all you have is a timeout, everything looks like a partition]() Original publisher: [Read original article]() Author: Nicolas Liochon (noreply@blogger.com) Published: 2015-05-25T17:42:00Z Content type: article Language: en Sources: [Nicolas Liochon]() Topics: [Network](), [Hadoop](), [Processes](), [Database](), [Open Source]() Tags: [acid](), [apache](), [apache-zookeeper](), [availability](), [cap-theorem](), [consistency](), [database](), [distributed-system](), [durability](), [hadoop](), [latency](), [network](), [partition](), [process](), [synchronization](), [timeout]() ### AI overview This article distinguishes network partitions from node failures and process crashes in the CAP theorem. It examines how Apache HBase uses Apache ZooKeeper ephemeral nodes and session timeouts for failure detection, noting that timeout-based recovery affects mean time to repair across these failure scenarios. ### Source excerpt "I suppose it is tempting, if the only tool you have is a timeout, to treat everything as if it were a partition." - Abraham Maslow, quoted from memory This post is part of the CAP theorem series. You may want to start by my post on ACID vs. CAP if you have a database background but have never really been exposed to the CAP theorem. The post discussing some traps in the 'Availability' and 'Consistency' definition of CAP should also be used as an introduction if you know CAP but haven't looked at its formal definition. Partitions in the CAP theorem are network partitions, and nothing else. There're not node failure or process crash. But does this matter in real-life? Let's answer this question by looking at an optimisation implemented on Apache HBase and Apache ZooKeeper when handling process crash. This is applicable by anyone using ZooKeeper. Apache HBase, ZooKeeper and failure detection Using ZooKeeper website: "ZooKeeper is a centralized service for maintaining configuration information, naming, providing distributed synchronization, and providing group services. All of these kinds of services are used in some form or another by distributed applications." And HBase website: "Apache HBase™ is the Hadoop database, a distributed, scalable, big data store." HBase is an open source implementation of Google's Bigtable. Cluster size varies from 5 to x000 nodes. ZooKeeper is used by HBase for a few things, one of them being, like in Bigtable, detecting failures. ZooKeeper manages failures by giving the user the possibility to create "ephemeral nodes". "These znodes exist as long as the session that created the znode is active. When the session ends the znode is deleted." And then "Expirations happen when the cluster does not hear from the client within the specified session timeout period (i.e. no heartbeat). At session expiration the cluster will delete any/all ephemeral nodes owned by that session and immediately notify any/all connected clients of the change (anyone w ## Eventual Consistency and Durability DevFeed: [Eventual Consistency and Durability]() Original publisher: [Read original article]() Author: Nicolas Liochon (noreply@blogger.com) Published: 2015-05-12T15:51:00Z Content type: article Language: en Sources: [Nicolas Liochon]() Topics: [NoSQL](), [Databases](), [data]() Tags: [acid](), [availability](), [cap-theorem](), [consistency](), [data](), [database](), [databases](), [distributed-system](), [durability](), [latency](), [network](), [nosql](), [nosql-databases](), [partition](), [storage]() ### AI overview The article explains eventual consistency in NoSQL databases and distinguishes it from ACID consistency. It discusses durability, defines eventual consistency using cited formulations, and introduces a replicated key-value database model using N, W, and R parameters. ### Source excerpt "A son can bear with equanimity the loss of his father, but the loss of his data may drive him to despair." (Machiavelli, quoted from memory) While traditional databases are ACID, with the 'D' meaning Durability, the NoSQL databases are mainly described by their memory model, i.e. Strong Consistency vs. Eventual Consistency. We're going to see that durability is not something you can forget about so easily. Definition of Eventually Consistent Eventual Consistency was defined by Werner Vogels in [E2] "the storage system guarantees that if no new updates are made to the object, eventually all accesses will return the last updated value." It's one definition among many. Baillis and Ghodsi mention for example [E3] "Informally, it guarantees that, if no additional updates are made to a given data item, all reads to that item will eventually return the same value." This definition only says that the value will converge to something (say '42'). In 1995, Terry used [V5] "All servers move towards eventual consistency. That is, the Bayou system guarantees that all servers eventually receive all Writes.". As most implementations available today are inspired by the Dynamo paper and Vogels' works, it's safer to stick to Vogels' definition, who knew them all, and has chosen his words carefully. In the CAP series, I explained the importance of this definition. And finally, this consistency is not the same thing as the consistency in ACID. Using an eventually consistent database So if we have an eventually consistent key-value database, we can implement a client that would be doing something like this for example: ecStorage.put("anId", "someData"); while (ecStorage.get("anId") == null) sleep 1 second; We know that this program will finish: even if the first "gets" may not see our write, eventually they will see it. Eventually Consistent database - possible implementation The most usual implementation is the one described once again by Werner Vogels, once again in his [E2] paper: N ## The Consistency Series DevFeed: [The Consistency Series]() Original publisher: [Read original article]() Author: Nicolas Liochon (noreply@blogger.com) Published: 2015-05-12T15:43:00Z Content type: article Language: en Sources: [Nicolas Liochon]() Topics: [consistency](), [Databases](), [acid](), [Dynamo](), [CAP theorem]() Tags: [acid](), [availability](), [cap-theorem](), [consistency](), [database](), [databases](), [distributed-system](), [durability](), [dynamo](), [latency](), [network](), [partition](), [series]() ### AI overview The article introduces a new series on consistency models, focusing on practical details and their implementations. It distinguishes the topic from a simple strong-versus-eventual consistency trade-off and previews posts about durability, ACID, and the Dynamo model. ### Source excerpt "Those are my consistency models, and if you don't like them... well, I have others." - Groucho Marx, quoted from memory The CAP theorem series is coming to an end. I have a few posts planned, but they will be mainly about wrapping things-up-- I hope these posts will be great, though. Anyway, here starts a new series: the consistency series. This series is about getting into some practical details about the consistency models and their implementations. You think that the trade-off is only or mainly on strong consistency vs. eventual consistency? That everything is said with "W+R" vs. "N"? Then read this series. Settling on the color may not be enough when choosing a car. Posts, in recommended reading order The posts published so far are: Eventual Consistency and Durability: shows the link between durability and consistency, and why traditional databases have the 'D' in ACID. Tells you what 'W=1' implies in the Dynamo model. ## CAP, Availability, High-Availability and Big Data databases in a world of partitions DevFeed: [CAP, Availability, High-Availability and Big Data databases in a world of partitions]() Original publisher: [Read original article]() Author: Nicolas Liochon (noreply@blogger.com) Published: 2015-04-16T16:32:00Z Content type: article Language: en Sources: [Nicolas Liochon]() Topics: [distributed-systems](), [Databases](), [big-data](), [Availability]() Tags: [acid](), [availability](), [big-data](), [cap-theorem](), [consistency](), [database](), [databases](), [distributed-system](), [durability](), [high-availability](), [latency](), [network](), [partition]() ### AI overview This article uses the CAP theorem to categorize distributed systems and extends the discussion to big-data systems, where data does not fit on a single node. It distinguishes CAP consistency from ACID consistency, defines CAP availability, and proposes a comparable definition of high availability. ### Source excerpt "Those who would give up Consistency, to purchase a little temporary Availability, deserve neither Consistency nor Availability." (B. Franklin, quoted from memory) This post is part of the CAP theorem series. You may want to start by my post on ACID vs. CAP if you have a database background but have never really been exposed to the CAP theorem. The post discussing some traps in the 'Availability' and 'Consistency' definition of CAP should also be used as an introduction if you know CAP but haven't looked at its formal definition. This post, as was the previous one, is about using CAP to categorize distributed systems. It summarizes its results and extends them to "big data" distributed systems. Compared to other distributed systems, "big data" systems add another dimension: by definition, the data does not fit into a single node. As we will see, this has some implications. Science Fiction authors are often ahead of their time. Will this be the case here? Looking at the terminology - CAP theorem & Availability Here we will use the definitions from the proof of the CAP theorem by Gilbert and Lynch [C2]. Consistent is [C2]: "Atomic, linearizable, consistency [...]. There must exist a total order on all operations such that each operation looks as if it were completed at a single instant. This is equivalent to requiring requests of the distributed shared memory to act as if they were executing on a single node, responding to operations one at a time." We have seen that CAP consistency is different from ACID consistency. Partition is [C2]: "The network will be allowed to lose arbitrarily many messages sent from one node to another. When a network is partitioned, all messages sent from nodes in one component of the partition to nodes in another component are lost." We have seen that node failures and packet losses are not partitions. Available is [C2]: "For a distributed system to be continuously available, every request received by a non-failing node in the system must r ## The unclear CP vs. CA case in CAP DevFeed: [The unclear CP vs. CA case in CAP]() Original publisher: [Read original article]() Author: Nicolas Liochon (noreply@blogger.com) Published: 2015-04-05T16:53:00Z Content type: article Language: en Sources: [Nicolas Liochon]() Topics: [distributed-systems](), [Databases](), [SQL](), [Protocol (disambiguation)]() Tags: [acid](), [availability](), [cap-theorem](), [consistency](), [database](), [distributed-system](), [distributed-systems](), [durability](), [latency](), [network](), [partition](), [protocol](), [sql]() ### AI overview This post examines why the CAP theorem's AP, CP, and CA categories can be unclear and may say little about how a distributed system actually behaves. It uses several example architectures, including SQL databases, eventually consistent stores, multiple databases with two-phase commit, and distributed consensus servers, while distinguishing CAP terminology from ACID terminology. ### Source excerpt This post is part of the CAP theorem series. You may want to start by my post on ACID vs. CAP if you have a database background but have never really been exposed to the CAP theorem. The post discussing some traps in the 'Availability' and 'Consistency' definition of CAP should also be used as an introduction if you know CAP but haven't looked at its formal definition. CAP is a theorem, but it is also widely used to categorize distributed systems and to explain their trade-offs. It looks great: "Hey, it's based on a theorem! This is scientifically proven!" However, the meaning of these categories--AP, CP, CA--is not that clear, leading to an opposite result: the CAP categorization says very little on how the distributed system actually behaves. For example, look at the CA category--Consistent and Available but not Partition-tolerant. A widely accepted opinion is: "Partition cannot be avoided. Your distributed system can just choose to sacrifice either consistency or availability when a partition occurs. So a distributed system cannot be CA, by definition." Indeed, Eric Brewer himself, wrote in 2012 [C3]: "Does choosing consistency and availability (CA) as the '2 of 3' make sense? As some researchers correctly point out, exactly what it means to forfeit P is unclear." Should one use the CA category then? A slide saying that some systems are CA. Does its author really understand CAP? To answer this question, we're going to use CAP to categorize a few different distributed systems: A simple application using a single server with a traditional SQL database. The same application, but with an Eventually Consistent store instead of our traditional SQL one. An application using multiple databases and the two-phase commit protocol. A distributed consensus server, like Apache ZooKeeper or Google Chubby. Also, as we know, using CAP in a database context can be confusing. In this post, I always use CAP terminology and not ACID terminology. When I say "consistent" or "strongly c ## The confusing CAP and ACID wording DevFeed: [The confusing CAP and ACID wording]() Original publisher: [Read original article]() Author: Nicolas Liochon (noreply@blogger.com) Published: 2015-03-26T15:30:00Z Content type: article Language: en Sources: [Nicolas Liochon]() Topics: [distributed-systems](), [Databases](), [Transactions](), [integrity]() Tags: [acid](), [availability](), [cap-theorem](), [consistency](), [database](), [distributed-system](), [distributed-systems](), [durability](), [integrity](), [latency](), [network](), [partition](), [properties](), [systems](), [transactions]() ### AI overview This article explains why CAP and ACID use overlapping terms with different meanings in distributed systems and database systems. It compares concepts including consistency, atomicity, transactions, durability, availability, and partition tolerance, highlighting where the terminology overlaps and where the concepts differ. ### Source excerpt CAP and ACID share a common vocabulary: Atomic, Consistent, and so on. But there is a catch: the words are the same but they mean totally different things. CAP comes from the distributed systems theory, while ACID belongs to database systems one.The Distributed Databases use both CAP and ACID vocabulary, so this obviously creates a lot of confusion. When someone says: "one should not give up consistency" what's the consistency he is speaking about? Let's have a look at the definitions of Atomic-Consistent-Isolated-Durable and Consistent-Available-Partition-tolerant. ACID & CAP - A reminder ACID properties were identified in the 70's, then the term ACID was coined in 1983. We have: A for Atomicity C for Consistency I for for Isolation D for Durability Reminder: the ACID properties were defined in California during the 70's. The CAP theorem was conjectured by Eric Brewer in 2000, and proved in 2002 by Seth Gilbert and Nancy Lynch. We have: C for Consistency A for Availability P for Partition-tolerance. CAP and ACID wording - the confusion summarized First, let's see the global picture. Word Databases CAP Confusion Transaction A set of operations. The word and the concept are not used. No Durable "Once a transaction completes successfully, its changes to the state survive failures." [D2] The word and the concept are not used. No Consistent Integrity constraints on the data (data types, relations, ...) For CAP, Consistency is a shortcut for "Atomic Consistency". The atomic consistency is a consistency model. More on this later. Same word, different concepts Isolation "Even though transactions execute concurrently, it appears to each Transaction, T, that others executed either before either after T". [D2] The word is not used in CAP but the word Isolation as defined in ACID is a consistency model in CAP vocabulary. Different words but same concept Atomic All changes happen or none happen. For CAP, Atomic is a consistency model, the one used in the CAP proof. Same word, dif ## Don't use the CAP theorem for packet losses DevFeed: [Don't use the CAP theorem for packet losses]() Original publisher: [Read original article]() Author: Nicolas Liochon (noreply@blogger.com) Published: 2015-03-18T15:16:00Z Content type: article Language: en Sources: [Nicolas Liochon]() Topics: [CAP theorem](), [consistency](), [Networks](), [Protocol (disambiguation)]() Tags: [acid](), [availability](), [cap-theorem](), [consistency](), [database](), [distributed-system](), [durability](), [ip](), [latency](), [messages](), [network](), [networks](), [partition](), [protocol](), [tcp](), [theory](), [udp]() ### AI overview The article explains why ordinary packet loss on IP networks should not automatically be treated as a CAP theorem partition. It distinguishes congestion-related packet loss from the arbitrary message loss assumed by the CAP model and discusses how TCP responds to congestion. ### Source excerpt In the previous post, we looked at this common saying: "nodes fail, network packets get lost, partitions happen so you need to use CAP to understand your trade-offs." We saw that node failures were not partitions. What about packet losses? Most distributed applications use TCP or UDP on top of IP, and it is well known that IP is an asynchronous protocol and that it can lose packets. So should we use all the results from the theory of asynchronous networks? Must we use CAP to do some trade-offs if we are using a network that can drop packets? The answer is no. The root issue lies in the incompleteness of our description of IP. "IP is an asynchronous protocol and it can lose packets" is true, but incomplete, and this incompleteness is misleading. Let's discuss why. CAP - The usual reminder CAP says that a distributed system cannot be Consistent, Available and Partition tolerant. We use here the definitions from the proof [C2]. Consistent is: [C2] "Atomic, linearizable, consistency [...]. There must exist a total order on all operations such that each operation looks as if it were completed at a single instant. This is equivalent to requiring requests of the distributed shared memory to act as if they were executing on a single node, responding to operations one at a time." Available is: [C2] "For a distributed system to be continuously available, every request received by a non-failing node in the system must result in a response." Partition is: [C2] "The network will be allowed to lose arbitrarily many messages sent from one node to another. When a network is partitioned, all messages sent from nodes in one component of the partition to nodes in another component are lost." How can you lose packets on a network Packet loss happens when a network equipment receives more messages that it can send. Losing packets is common on an IP network. A TCP connection tries to use most of the bandwidth available. It sends nearly as many packets as it can, until it loses some. Losi ## CAP Theorem Scope: Why Node Failures Are Not Network Partitions DevFeed: [CAP Theorem Scope: Why Node Failures Are Not Network Partitions]() Original publisher: [Read original article]() Author: Nicolas Liochon (noreply@blogger.com) Published: 2015-03-13T10:22:00Z Content type: article Language: en Sources: [Nicolas Liochon]() Topics: [Process](), [data](), [Network](), [Availability]() Tags: [acid](), [availability](), [cap-theorem](), [consistency](), [database](), [distributed-system](), [durability](), [latency](), [network](), [partition](), [process](), [request]() ### AI overview This article examines the scope of the CAP theorem and argues that process crashes and node failures are not network partitions under CAP's definitions. It reviews the theorem's consistency, availability, and partition definitions, then evaluates the common intuition that a failed node is equivalent to a node separated by a partition. ### Source excerpt "Dead nodes don't bite." - Robert Louis Stevenson (quoted from memory) CAP is often described as a theorem you cannot avoid using. A common saying is "nodes fail, network packets get lost, partitions arise so you need to use CAP to choose the trade-offs." Actually, the CAP scope is not that wide. Let's look at why a process crash or a node failure are not partitions in CAP. CAP - the usual reminder CAP is a theorem, initially conjectured by Eric Brewer and proved in 2002 by Seth Gilbert and Nancy Lynch proved [C2]. Its catchy definition is: "Consistency, Availability, Partition tolerance: pick two." Let's remind the definitions from the proof [C2]. Consistent is: "Atomic, linearizable, consistency [...]. There must exist a total order on all operations such that each operation looks as if it were completed at a single instant. This is equivalent to requiring requests of the distributed shared memory to act as if they were executing on a single node, responding to operations one at a time." Available is: "For a distributed system to be continuously available, every request received by a non-failing node in the system must result in a response." Partition is: "The network will be allowed to lose arbitrarily many messages sent from one node to another. When a network is partitioned, all messages sent from nodes in one component of the partition to nodes in another component are lost." Node failure vs. partition - the wrong intuition A common and wrong intuition is: "from the point of view of the remaining nodes, it is impossible to distinguish a node that has failed from a node that is on another partition: in both cases, it does not respond nor sends any message. As a consequence, there is no difference between a network partition and a node failure." Let's try this intuition: first against the definition of CAP, then against its proof, and then against real-life data store implementations. Is this crocodile dead or is it just sleeping? There is no difference until yo ## Comparing Eventually Consistent and CP-as-in-CAP stores DevFeed: [Comparing Eventually Consistent and CP-as-in-CAP stores]() Original publisher: [Read original article]() Author: Nicolas Liochon (noreply@blogger.com) Published: 2015-03-11T15:13:00Z Content type: article Language: en Sources: [Nicolas Liochon]() Topics: [distributed-systems](), [data](), [Availability](), [Latency]() Tags: [acid](), [availability](), [cap-theorem](), [consistency](), [database](), [distributed-system](), [distributed-systems](), [durability](), [latency](), [network](), [partition](), [storage]() ### AI overview This article compares eventual consistency with CP data stores under the CAP theorem. It examines hypothetical key-value-store implementations to show that apparent read-speed advantages can violate the definition of eventual consistency, especially the requirement that all accesses eventually return the latest value. ### Source excerpt Eventual Consistency (EC) is a well known concept. The CAP theorem, which defines Consistency, Availability and Partition tolerance is also well known. It describes some distributed systems such as CP, which has two properties: Consistency and Partition tolerance. If we look at data stores just by how they stick to the definition of their consistency model and compare their speeds, which one would be the fastest? An EC or a CP store? Answering this question is a good excuse to go into some detail of their respective definitions and show some of their limits. Contrary to popular belief, data stores can be very fast Eventually Consistent Eventually Consistent was defined by Werner Vogels in [E2]: "the storage system guarantees that if no new updates are made to the object, eventually all accesses will return the last updated value." Let's try a few implementations of a hypothetical key value store. Implementation 1: Trying the simplest option void put(key, value){ // do nothing } value get(key){ // Hey, it's easy to implement! We're going // to pretend that we have not yet received // the write, it is enough! throw "no value for this key" } Is this an EC store? No, it is not, because it will never return the last updated value. To be an EC store you need to "eventually [...] return the last updated value." That's not what this implementation does, it will never return the last updated value. So let's try another one. Implementation 2: Trying to be fast for reads void put(key, value){ doRealPut(key, value) // does the 'real' job } value get(key){ if (random(2) == 1) // 50% of the time throw "no value for this key" else return doRealGet(key) // As, 50% of the time, I pretend // that I have not yet received // the first insertion, I // divide the latency by two! } Is this an EC store? Again, it is not because it does not respect the property "eventually all accesses will return the last updated value." The "all accesses" condition is not met. The definition does not prev ## The CAP theorem series DevFeed: [The CAP theorem series]() Original publisher: [Read original article]() Author: Nicolas Liochon (noreply@blogger.com) Published: 2015-03-10T15:10:00Z Content type: article Language: en Sources: [Nicolas Liochon]() Topics: [distributed-systems](), [Databases](), [Availability](), [big-data]() Tags: [acid](), [availability](), [big-data](), [cap-theorem](), [consistency](), [database](), [databases](), [distributed-system](), [distributed-systems](), [durability](), [latency](), [network](), [partition]() ### AI overview An introduction to a series examining the CAP theorem and databases. The series addresses commonly misunderstood distinctions involving consistency, availability, partitions, node failures, packet loss, ACID terminology, and distributed-system categories. ### Source excerpt Let me introduce a new series of posts on the CAP theorem. CAP is a well known theorem conjectured and proven by recognized researchers in distributed systems, namely Eric Brewer, Seth Gilbert and Nancy Lynch. It is also widely used to categorize distributed applications. With CAP, do computer scientists actually have a better classification tool than biologists? Why you want to read this series on CAP theorem and databases This series covers some points often forgotten or misunderstood and sometimes never mentioned, such as: '42' is not a valid answer for an eventually consistent database; Node failures and packet loss are not partitions; Consistent in ACID and Consistent in CAP are two different things; The CAP definition of availability is key but very different from what most people think; There are some systems that do not fit into the three CA/CP/CA categories; It's possible to use the CA category, really possible, There is no such thing as an eventually consistent and available big data database. The posts, in recommended reading order Comparing Eventually Consistent and CP-as-in-CAP stores: Introduces the definitions of "Consistent" and "Available" in the CAP theorem. By comparing them with the definition of "Eventual Consistency" we show the limits of these definitions. The confusing ACID and CAP wording: Explains the terminologies of ACID and CAP, and how they overlap. Covers as well the difference between the theoretical definition of the 'I' in ACID and its actual implementation with multiple "isolation degrees". It's a good starting point if you know the "Consistency" definition of ACID but not the "Consistency" definition of CAP. Don't use the CAP theorem for node failures: Looks at a common misunderstanding of the CAP theorem: node failures are not partitions. It's also a pretext to look at the actual proof of CAP, and to go into the definition of "Partition" in CAP. Don't use the CAP theorem for packet losses: Looks at another common misunderstanding [Next page]()