# Marc Brooker's

Published articles for Marc Brooker's.

This is one page of public article previews, not the complete archive. Follow Next page to continue. Summaries are not the original full articles.

## Traffic mix matters

DevFeed: [Traffic mix matters](<https://devfeed.tech/articles/traffic-mix-matters-140983.md>)

Original publisher: [Read original article](<http://brooker.co.za/blog/2026/11/18/workload-mix.html>)

Author: Marc Brooker

Published: 2026-11-18T00:00:00Z

Content type: article

Language: en

Sources: [Marc Brooker's](<https://devfeed.tech/sources/marc-brooker-s-blog.md>)

Topics: [Benchmark](<https://devfeed.tech/topics/benchmark.md>), [thread safe data sharing](<https://devfeed.tech/topics/thread-safe-data-sharing.md>), [CPU Cache](<https://devfeed.tech/topics/cpu-cache.md>)

Tags: [cache](<https://devfeed.tech/tags/cache.md>), [concurrency](<https://devfeed.tech/tags/concurrency.md>), [database](<https://devfeed.tech/tags/database.md>), [distribution](<https://devfeed.tech/tags/distribution.md>), [fast](<https://devfeed.tech/tags/fast.md>), [fifo](<https://devfeed.tech/tags/fifo.md>), [limit](<https://devfeed.tech/tags/limit.md>), [load](<https://devfeed.tech/tags/load.md>), [load-testing](<https://devfeed.tech/tags/load-testing.md>), [local](<https://devfeed.tech/tags/local.md>), [lock](<https://devfeed.tech/tags/lock.md>), [locks](<https://devfeed.tech/tags/locks.md>), [m365](<https://devfeed.tech/tags/m365.md>), [memory](<https://devfeed.tech/tags/memory.md>), [model](<https://devfeed.tech/tags/model.md>), [postgres](<https://devfeed.tech/tags/postgres.md>), [queue](<https://devfeed.tech/tags/queue.md>), [store](<https://devfeed.tech/tags/store.md>), [system](<https://devfeed.tech/tags/system.md>), [traffic](<https://devfeed.tech/tags/traffic.md>)

### AI overview

Workload mix can change throughput by orders of magnitude, so load tests and benchmarks need traffic that resembles production. In a 16-worker lock-table simulation, keys follow a Zipf distribution: greater skew concentrates contention on a few locks, lowering throughput and leaving other resources underused. In a second model, requests use an LRU cache backed by slower storage; greater skew improves cache hits and throughput across a broad range. The opposing results show that key skew can affect systems in different ways. The excerpt also notes a preference for optimistic concurrency control and a single cost/latency mode, and says Aurora DSQL sends reads to one fast storage layer without workers waiting on locks.

### Source excerpt

Traffic mix matters Wanna listen to my mixtape? The preliminary report on last week's M365 outage contains one statement that really caught my eye: Our normal load testing did not detect the latent issue because [of] the traffic mix and SKUs used in our labs. There's a lot going on in this report (as there is in any report for a complex system), but I wanted to zoom in on this one issue: traffic mix. It's important, because its the one thing that makes the difference between a great load test and a complete waste of time. The performance of computer systems is extremely sensitive to workload mix. Let's start with a simple simulator, looking at a system with a table of locks. Requests arrive with a key, hold the lock for their key for some period of time, and then free the lock and complete. This isn't an unreasonable mental model of a relational database, where each transaction takes a lock during its execution. arrivals backlog 16 worker threads lock table (one lock per key) done Poisson FIFO, never empty at saturation holder FIFO waiters hold ~Exp(mean 1 ms) key 1 key 2 key 3 ⋮ key N holding its key's lock blocked behind the holder (thread still occupied) Keys (N): Simulate Each request takes one of 16 worker threads, waits in a FIFO queue for the lock on its key, holds the lock for an exponentially distributed time (mean 1 ms), then releases it and frees its thread. Keys are drawn from a Zipf distribution over N keys; a coefficient of 0 is uniform. As we may expect, the more skewed the distribution, the less throughput you get out of the system. A small number of keys are highly contended, leading to under utilization of the rest of the resources of the machine, driven by the global concurrency limit. Now, let's consider a different mental model of a relational database (at least a single-system one like Postgres). A fast, local, in-memory cache containing some of the accessed data backed by a slower store containing the rest of the data. arrivals backlog 16 work

## What does it mean to understand what we built?

DevFeed: [What does it mean to understand what we built?](<https://devfeed.tech/articles/what-does-it-mean-to-understand-what-we-built-138175.md>)

Original publisher: [Read original article](<http://brooker.co.za/blog/2026/11/16/understanding.html>)

Author: Marc Brooker

Published: 2026-11-16T00:00:00Z

Content type: opinion

Language: en

Sources: [Marc Brooker's](<https://devfeed.tech/sources/marc-brooker-s-blog.md>)

Topics: [observability](<https://devfeed.tech/topics/observability.md>), [code analysis](<https://devfeed.tech/topics/code-analysis.md>)

Tags: [ai](<https://devfeed.tech/tags/ai.md>), [behavior](<https://devfeed.tech/tags/behavior.md>), [code](<https://devfeed.tech/tags/code.md>), [complex-systems](<https://devfeed.tech/tags/complex-systems.md>), [engineering](<https://devfeed.tech/tags/engineering.md>), [failures](<https://devfeed.tech/tags/failures.md>), [julia](<https://devfeed.tech/tags/julia.md>), [properties](<https://devfeed.tech/tags/properties.md>), [systems](<https://devfeed.tech/tags/systems.md>), [understand](<https://devfeed.tech/tags/understand.md>), [user](<https://devfeed.tech/tags/user.md>)

### AI overview

The author argues that software understanding should focus less on reading code and more on the behavior and properties of the built system. These include precise guarantees, such as a SELECT never exposing uncommitted transactions, user-facing qualities, and emergent behaviors such as metastable failures. Because such whole-system properties can be hard to infer from individual lines, the author calls for more powerful tools analogous to Bode plots or Nyquist stability criteria in other engineering fields, and sees AI as a way to help develop that system-level understanding. The text is an excerpt and does not specify particular tools or methods.

### Source excerpt

What does it mean to understand what we built? Let's try come to an understanding. Today, a couple of people sent me the No AI Slop Manifesto. I found it interesting, in part, because it touches on something that seems super important right now: understanding. Through this work, we have come to value: ... Understanding over speed I fully agree with that as a pithy statement, but I think we are working with different definitions of the word understanding. For example: We do our own thinking when working with a model. We explore resources, code, and prior art to truly understand the problem before the work starts. That's an example of a symptom of something that I think makes this document substantially less useful and forward-looking than it could have been. Specifically, it privileges code. That's natural for programmers, of course, because program text has been somewhat privileged in our field's thinking for something like 60 years. I believe that the goal of understanding code is, longer-term, holding us back from understanding what matters. I'm excited about the posibilties of what we can understand better when we are freed from understanding code, and can focus on other things. What are those other things? The properties of the as-built system. Precise properties like "a SELECT never shows results from uncommitted transactions". And fuzzy properties like "makes the user's life easier". Those are already non-local properties that come from the code as a whole, not from any individual line. In fact, trying to understand how they are synthesized out of individual lines is a mammoth task! One that we've spent six decades coming up with abstractions to handle. Then there are the really emergent properties. The dynamic behaviors that only come from the system all working together. Like metastable failures. As with the Lorenz Attractor or Julia Set it's generally intractable to extract an understanding of the behavior of these systems from the code. We need different, be

## Animations for System Builders: Metastability and Lost Writes

DevFeed: [Animations for System Builders: Metastability and Lost Writes](<https://devfeed.tech/articles/animations-for-system-builders-metastability-and-lost-writes-97874.md>)

Original publisher: [Read original article](<http://brooker.co.za/blog/2026/10/23/animations.html>)

Author: Marc Brooker

Published: 2026-10-23T00:00:00Z

Content type: article

Language: en

Sources: [Marc Brooker's](<https://devfeed.tech/sources/marc-brooker-s-blog.md>)

Topics: [Chaos Engineering](<https://devfeed.tech/topics/chaos-engineering.md>), [etcd](<https://devfeed.tech/topics/etcd.md>), [Raft](<https://devfeed.tech/topics/raft.md>), [observability](<https://devfeed.tech/topics/observability.md>), [Cache](<https://devfeed.tech/topics/cache.md>)

Tags: [animations](<https://devfeed.tech/tags/animations.md>), [cache](<https://devfeed.tech/tags/cache.md>), [etcd](<https://devfeed.tech/tags/etcd.md>), [metastability](<https://devfeed.tech/tags/metastability.md>), [postgres](<https://devfeed.tech/tags/postgres.md>), [raft](<https://devfeed.tech/tags/raft.md>), [software](<https://devfeed.tech/tags/software.md>), [storage](<https://devfeed.tech/tags/storage.md>), [systems](<https://devfeed.tech/tags/systems.md>)

### AI overview

The excerpt uses animations backed by test-environment traces to show metastable failures in real systems. In Postgres 16 running in a MicroVM, a load spike fills memory with queries (12 MB each); queries that had fit in the page cache then run more slowly from disk, despite the database handling 20 queries per second before the spike. In etcd, overloaded peers appear unavailable, prompting application-level resend storms. A separate etcd example describes a GC feedback loop: a read spike from 500 to 1,000 per second increases time spent in GC, raises average tenure, and leaves less CPU for reads; the system does not recover without intervention. The author says a similar allocator-driven loop without GC remains unverified. The excerpt also notes that a Raft cluster can lose writes if a member's storage layer falsely reports `fsync()` success. Animations were generated with Kiro and KiroCrew, mostly using Opus 5.5; the underlying results came from a real test environment.

### Source excerpt

Animations for System Builders: Metastability and Lost Writes Pretty pictures help us learn, maybe? One of the things I love using LLMs for is helping me understand systems better. In the Strands Decider post I wrote about how I often ask ask me questions to check I understand this. The other technique I'm enjoying right now, and am delighted to see folks around me using too, is building animations. Animations can help us build an intuition for how and why systems work, or how and why they fail. Here's one example: did you know that a Raft cluster can lose writes if even one of the cluster members has a storage layer that lies about fsync()? Your browser does not support the video tag. When I built Stability Sim, I wanted to help engineers understand metastable failures better. That worked pretty well on an abstract level (e.g. retries and queues make metastability really easy to trigger), but I still hear doubts that these behaviors happen in real systems. After all, those real systems are production-proven, and wouldn't we have shaken out the behaviors by now if they existed? Let's start with Postgres. Your browser does not support the video tag. In this example, captured from traces on real Postgres 16 running in a real MicroVM. Before a load spike, the database is easily able to handle its 20 queries-per-second load, thanks to the whole working set sitting in the page cache. But after the load spike, we've filled up a lot of the memory running queries (each takes 12MB), which now run slower because they're served off the disk rather than the page cache. I like this example, because it shows that even simple, non-distributed systems have these behaviors. What about etcd? Your browser does not support the video tag. Again, we have load spike triggering the metastable state. Writes, in this case. The overloaded peers look like they're unavailable, which triggers a storm of application-level resends. I like this example, because etcd is widely-used quality software.

## Strands Decider: Why Not an Encoder?

DevFeed: [Strands Decider: Why Not an Encoder?](<https://devfeed.tech/articles/strands-decider-why-not-an-encoder-64737.md>)

Original publisher: [Read original article](<http://brooker.co.za/blog/2026/10/04/encoders.html>)

Author: Marc Brooker

Published: 2026-10-04T00:00:00Z

Content type: comparison

Language: en

Sources: [Marc Brooker's](<https://devfeed.tech/sources/marc-brooker-s-blog.md>)

Topics: [ai-experiments](<https://devfeed.tech/topics/ai-experiments.md>)

Tags: [latency](<https://devfeed.tech/tags/latency.md>), [qwen3-5](<https://devfeed.tech/tags/qwen3-5.md>), [strands-decider-2b](<https://devfeed.tech/tags/strands-decider-2b.md>)

### AI overview

Experiments compare encoder, decoder, and encoder-decoder designs for Strands Decider. An optimized encoder offers lower median latency and similar benchmark performance, but scales worse with longer prompts. Differences in architectures, model sizes, and training confound the comparison, leaving no clear winner.

### Source excerpt

Strands Decider: Why Not an Encoder? Expertise level: exhausted. As I admitted in my last post, I am very much not an expert model developer. What follows is likely to be, at least partially, inaccurate. I have tried to be careful and quantitative, to partially balance a lack of deep expertise. Since we launched strands-decider-2B last week, a couple of people have asked me why it's a modified LLM and not an encoder (like BERT). Their instinct seems to be that bidirectional (i.e. each token's representation is built from tokens before and after it) is likely to out-perform causal (i.e. only before tokens are used). This is the same reason encoders are widely used as classifiers. This is also a fair question, because if you squint at it right, we're abusing a decoder as an encoder. To test this hypothesis, I compared five designs: v19 is the final design for Qwen3.5-based strands-decider (check out the previous post and strands-decider README for the details). b1 is an encoder-decoder design (like the original Transformer), based on T5Gemma 2 with 1B encoder params and 1B decoder params, minus the LM head and plus our pointer head. b2 is the same as b1, but 4B+4B. e1a is just the encoder half of T5Gemma (not T5Gemma 2), with 2.6B parameters. e1b is a latency-optimized version of e1a with a masked state cache, allowing for multiple questions to be evaluated in parallel without interference, and for the state cache to be reused1. The state attends to the whole state bidirectionally, and each question attends to state and itself. I also measured frozen ModernBERT and ettin-encoder-1b, but found that they under-performed T5Gemma parameter-for-parameter, and so didn't invest time in training them. First, let's look at accuracy and calibration. b2 does very well here, beating v19 on JevBench accuracy and Brier, and on JF100 Brier at the same accuracy. This is good, but recall that b2 has ~8B total parameters compared to v19's 2B, and so is not an apples-to-apples compariso

## Small Decisions: Engineering a Leading Model

DevFeed: [Small Decisions: Engineering a Leading Model](<https://devfeed.tech/articles/small-decisions-engineering-a-leading-model-61280.md>)

Original publisher: [Read original article](<http://brooker.co.za/blog/2026/09/28/engineering-system-one.html>)

Author: Marc Brooker

Published: 2026-09-28T00:00:00Z

Content type: article

Language: en

Sources: [Marc Brooker's](<https://devfeed.tech/sources/marc-brooker-s-blog.md>)

Topics: [Advanced AI workflows](<https://devfeed.tech/topics/advanced-ai-workflows.md>), [Fine-tuning](<https://devfeed.tech/topics/fine-tuning.md>), [Reinforcement learning](<https://devfeed.tech/topics/reinforcement-learning.md>), [agentic workflows](<https://devfeed.tech/topics/agentic-workflows.md>), [scaling laws](<https://devfeed.tech/topics/scaling-laws.md>)

Tags: [agents](<https://devfeed.tech/tags/agents.md>), [ai](<https://devfeed.tech/tags/ai.md>), [ai-agents](<https://devfeed.tech/tags/ai-agents.md>), [ai-infrastructure](<https://devfeed.tech/tags/ai-infrastructure.md>), [engineering](<https://devfeed.tech/tags/engineering.md>), [gpu](<https://devfeed.tech/tags/gpu.md>), [infrastructure](<https://devfeed.tech/tags/infrastructure.md>), [latency](<https://devfeed.tech/tags/latency.md>), [low-latency](<https://devfeed.tech/tags/low-latency.md>), [model](<https://devfeed.tech/tags/model.md>), [parallel](<https://devfeed.tech/tags/parallel.md>), [qwen3](<https://devfeed.tech/tags/qwen3.md>), [qwen3-5](<https://devfeed.tech/tags/qwen3-5.md>), [small](<https://devfeed.tech/tags/small.md>)

### AI overview

Hobson is a roughly 2B-parameter classifier built to deliver calibrated, low-latency decisions. It replaces Qwen3.5-2B's language-model head with a small pointer head that scores each answer option against the `<answer>` hidden state, using a rank-16 LoRA adapter on the torso. The author reports better calibration than reading logits from a similarly sized LLM, and top performance in its size range on the JevBench public set at the time described, while noting an unlisted decider-2b version that reportedly performs better. Training used one epoch of cross-entropy on 115,000 examples, shuffled options, and self-distillation to limit forgetting; held-out data set per-question-type temperatures for confidence calibration. The model scored 0.009 Brier and 100% accuracy on JevBench's easy set. It generalized usefully to unseen tasks but less well than on familiar task types. On a 3090, reported public-set latency was just over 100 ms at p50 and under 300 ms at p95; inference takes one pass for one question and two passes for multiple questions, with cached computation. The author identifies harder data synthesis and reinforcement learning as possible next steps.

### Source excerpt

Small Decisions: Engineering a Leading Model Or trying to, at least. My day job has been primarily in AI for three years now, but I'd be the first to admit that's been almost entirely in one corner of AI: infrastructure, safety, and tools for AI agents. That work has brought me in contact with a lot of the AI science (and I'd dabbled there over the previous decade), but I'm super far from the day-to-day of work like model building. I wanted to catch up a little (after all you have to know what you're talking about), and the last couple weeks provided a perfect opportunity. On the 15th of this month, the TypeSafe AI folks announced Jev, a kind of general purpose calibrated classifier. You can see this as something exciting or not, but it sure has captured the world's attention. And mine. I was particularly interested in the calibration, combined with low latency and the ability to answer questions in parallel, it's a great building block for the more workflowy end of the spectrum of agents. In an effort to understand these things well, it was time to build my own model: Hobson. A small one, because I wanted to use the GPU I have at home, and because I wanted to see if I could push the bounds on accuracy and calibration at very low latency. I decided to limit myself to about 2 billion parameters. How have I done so far? Fairly well, I think. You can read that as a trajectory of how versions of my model have performed on accuracy (on x) and calibration (on y) as I've made improvements. The pareto optimal is the bottom right. On the jevbench public set I'm at the top in my size range1. There's a new version of decider-2b which beats me, but hasn't been added to the leaderboard yet. I handily beat Qwen3.5-2B on both calibration and accuracy. I'm measuring calibration with the multi-class Brier score, roughly the mean-square prediction error in a range between 0 (perfect) and 2 (confidently wrong). Perhaps most usefully, Brier on the JevBench easy set is only 0.009 and ac

## Lorenz and Little: How Much Does Your Tail Cost?

DevFeed: [Lorenz and Little: How Much Does Your Tail Cost?](<https://devfeed.tech/articles/lorenz-and-little-how-much-does-your-tail-cost-12600.md>)

Original publisher: [Read original article](<http://brooker.co.za/blog/2026/07/29/lorenz-and-little.html>)

Author: Marc Brooker

Published: 2026-07-29T00:00:00Z

Content type: article

Language: en

Sources: [Marc Brooker's](<https://devfeed.tech/sources/marc-brooker-s-blog.md>)

Topics: [Concurrency](<https://devfeed.tech/topics/concurrency.md>), [cloud cost management](<https://devfeed.tech/topics/cloud-cost-management.md>)

Tags: [capacity](<https://devfeed.tech/tags/capacity.md>), [concurrency](<https://devfeed.tech/tags/concurrency.md>), [cost](<https://devfeed.tech/tags/cost.md>), [customer-experience](<https://devfeed.tech/tags/customer-experience.md>), [distribution](<https://devfeed.tech/tags/distribution.md>), [latency](<https://devfeed.tech/tags/latency.md>), [little](<https://devfeed.tech/tags/little.md>), [statistics](<https://devfeed.tech/tags/statistics.md>)

### AI overview

The empirical Lorenz curve estimates how much requests at or above a latency percentile contribute to mean latency: for samples, compute the fraction of total latency represented by the slowest requests; for quantiles, the author interpolates and extrapolates with a power-law model. In the example, requests at or above p50 contribute about 99% of mean latency, and those at or above p99 about 52%. By Little's law, these shares also represent concurrency in a simple threaded system, so tail latency can drive substantial capacity demand and related contention. Queues and event-based systems complicate that cost mapping, and the quantile estimate depends on modeling assumptions; the excerpt notes that a tail exponent at or below 1 implies an infinite, non-estimable mean.

### Source excerpt

Lorenz and Little: How Much Does Your Tail Cost? Lorenz and Little sounds like hipster burger bar from 2015. It's time for Marc's Amateur Statistics Corner! Today: why I pay a lot of attention to tail latency when optimizing cost. I've written before on the importance of tail latency for customer experience (e.g. in 2026, 2021, and 2021, and 2017). Today, I want to talk about tail latency from the perspective of cost and capacity. Like many system operators, I think about tail latency using percentiles. Here's a question: how much does each of my latency percentiles contributed to the mean latency? Intuitively, the answer is "quite a lot", but can we quantify that? We can! The thing we're looking for is the empirical Lorenz Curve. It directly calculates the answer to the question: given a latency percentile $P$ (e.g. p99=100ms), how much do requests taking shorter than $P$ contribute to the mean latency? (Let's call it $L(P)$ , so the real answer to our question is $1 - L(P)$). Starting from latency samples, the calculation is pretty simple: L = sum(sorted(x)[:k]) / sum(x) (for a set of n latency samples x, and k=p*n). From a vector of quantiles, things get a little more complicated, because we have to choose how to interpolate between the samples and extrapolate out to the maximum. Here I'm interpolating using a power law, which is a little bit of a sin1, but good enough for our purposes. # Calculate 1 - L(p) for a vector of measured quantiles # q - an array of quantiles (e.g. [1, 10, 200, 10000, 20000]) # p - an array of percentiles they're measured at (e.g. [0, 0.5, 0.9, 0.99, 0.999]) # OneMinusL - One minus the empirical Lorenz curve for each of the percentiles def OneMinusL(q, p): ...Show full implementationHide implementation # Calculate 1 - L(p) for a vector of measured quantiles # q - an array of quantiles (e.g. [1, 10, 200, 10000, 20000]) # p - an array of percentiles they're measured at (e.g. [0, 0.5, 0.9, 0.99, 0.999]) # OneMinusL - One minus the empirica

## Aurora DSQL: Scalable, Multi-Region OLTP

DevFeed: [Aurora DSQL: Scalable, Multi-Region OLTP](<https://devfeed.tech/articles/aurora-dsql-scalable-multi-region-oltp-12599.md>)

Original publisher: [Read original article](<http://brooker.co.za/blog/2026/07/19/dsql-paper.html>)

Author: Marc Brooker

Published: 2026-07-19T00:00:00Z

Content type: article

Language: en

Sources: [Marc Brooker's](<https://devfeed.tech/sources/marc-brooker-s-blog.md>)

Topics: [DSQL](<https://devfeed.tech/topics/dsql.md>), [Database](<https://devfeed.tech/topics/database.md>), [Distributed Systems & Parallel Computing](<https://devfeed.tech/topics/distributed-systems-parallel-computing.md>), [Transactions](<https://devfeed.tech/topics/transactions.md>), [scalable-architecture](<https://devfeed.tech/topics/scalable-architecture.md>), [Production Engineering](<https://devfeed.tech/topics/production-engineering.md>), [control-plane](<https://devfeed.tech/topics/control-plane.md>)

Tags: [agentic](<https://devfeed.tech/tags/agentic.md>), [agents](<https://devfeed.tech/tags/agents.md>), [application](<https://devfeed.tech/tags/application.md>), [applications](<https://devfeed.tech/tags/applications.md>), [architecture](<https://devfeed.tech/tags/architecture.md>), [architectures](<https://devfeed.tech/tags/architectures.md>), [arxiv](<https://devfeed.tech/tags/arxiv.md>), [aws](<https://devfeed.tech/tags/aws.md>), [blog](<https://devfeed.tech/tags/blog.md>), [cloud](<https://devfeed.tech/tags/cloud.md>), [complexity](<https://devfeed.tech/tags/complexity.md>), [control-plane](<https://devfeed.tech/tags/control-plane.md>), [database](<https://devfeed.tech/tags/database.md>), [dsql](<https://devfeed.tech/tags/dsql.md>), [durability](<https://devfeed.tech/tags/durability.md>), [dynamodb](<https://devfeed.tech/tags/dynamodb.md>), [fault-tolerance](<https://devfeed.tech/tags/fault-tolerance.md>), [interactive](<https://devfeed.tech/tags/interactive.md>), [multi-region](<https://devfeed.tech/tags/multi-region.md>), [operations](<https://devfeed.tech/tags/operations.md>)

### AI overview

Aurora DSQL's paper describes a disaggregated relational database built to scale while reducing operational complexity, including in multi-region deployments. It separates query processing, transactions, replication, storage, and control-plane concerns across independent services. The excerpt highlights avoiding large caches to ease failovers and read scaling, providing strongly consistent scalable reads to avoid some hot-key and partition problems, and running scans near local SSD at storage. It also describes optimistic concurrency control (OCC), which prevents clients from blocking one another; the author says lock contention and retries in pessimistic databases often contribute to outages and long recovery times. The paper argues that packing workloads can improve utilization by reducing peak-to-average load per machine. Its account presents strong consistency as a practical default in light of advances in time distribution, networks, and datacenter infrastructure. These are the author's claims; the excerpt does not provide full evaluation details.

### Source excerpt

Aurora DSQL: Scalable, Multi-Region OLTP A paper! Our new paper, Aurora DSQL: Scalable, Multi-Region OLTP, is now available on Arxiv. I'm excited about this one: it's a fully end-to-end look at how Aurora DSQL works, from query processing, to transactions, to replication, to the control plane. We've shared most of this content before in other forms, on this blog, on Marc Bowes' Blog, Werner's Blog, in talks, and on the AWS blog. But this version covers all the ground, all in one place. You should read it. Or check out the interactive transaction simulator to get a feel for how it works. Some highlights Our overall goal was to build a relational database system that simplifies the work of application building and operations, freeing builders from worrying about scale, reliability, durability, and even multi-region fault tolerance. A database of first resort, which is simple and easy to adopt at low scale, and grows with the application, without adding complexity. I've written about this argument before (in DSQL: Simplifying Architectures), and it remains the thing that excites me most about the Aurora DSQL product. I believe we've made it much easier (and, in many cases, cheaper) to build and operate resilient, highly available, cloud applications at all scales. That applies to systems built by humans and agents, with the benefits magnified for agentic builders. Agents are great at building, but not (yet?) so great at long-horizon tasks like database operations. A database designed to simplify operations really helps. DSQL's architecture is disaggregated. Multiple independent services, each focused on a small number of well-defined concerns. DSQL isn't the first disaggregated OLTP database. Many, including Aurora and DynamoDB, came before. What's interesting here is how we learned from those systems, and the lessons that came from operating them at scale: avoiding large caches (they make failovers and read scale tricky), offering strongly consistent scalable reads (a

## Meet Alice. Alice is impatient.

DevFeed: [Meet Alice. Alice is impatient.](<https://devfeed.tech/articles/meet-alice-alice-is-impatient-12598.md>)

Original publisher: [Read original article](<http://brooker.co.za/blog/2026/06/19/waiting.html>)

Author: Marc Brooker

Published: 2026-06-19T00:00:00Z

Content type: article

Language: en

Sources: [Marc Brooker's](<https://devfeed.tech/sources/marc-brooker-s-blog.md>)

Topics: [performance-engineering](<https://devfeed.tech/topics/performance-engineering.md>)

Tags: [clients](<https://devfeed.tech/tags/clients.md>), [code](<https://devfeed.tech/tags/code.md>), [customer-experience](<https://devfeed.tech/tags/customer-experience.md>), [distributions](<https://devfeed.tech/tags/distributions.md>), [downtime](<https://devfeed.tech/tags/downtime.md>), [inspection](<https://devfeed.tech/tags/inspection.md>), [latency](<https://devfeed.tech/tags/latency.md>), [mttr](<https://devfeed.tech/tags/mttr.md>), [simulation](<https://devfeed.tech/tags/simulation.md>), [web](<https://devfeed.tech/tags/web.md>)

### AI overview

Request and outage averages can understate what customers experience because customers sample long delays more often than short ones. The excerpt explains this inspection paradox with a time-weighted mean, E[X²]/(2E[X]), showing that the gap from the operator's mean grows with variance. In its example, a 30-minute median and 10-hour p99 recovery time imply an operator MTTR just over an hour but a customer-experienced mean around six hours. A Poisson-arrival simulation illustrates repeated sampling of ongoing outages. The author argues that tail behavior matters especially for recovery time, where retries cannot hide delays, and cautions that trimmed means discard useful right-tail information. Log-normal fitting is used for convenience; the author prefers non-parametric analysis and says log-normal may not suit real latency or recovery distributions.

### Source excerpt

Meet Alice. Alice is impatient. What do you mean? Meet Alice. Alice uses your web service. Alice, like most humans, measures her time in seconds and minutes. Alice says your service is slow. You tell Alice that the mean request to your service completes in 100ms, but Alice says that her mean wait time is 1s. You're both right. Meet Alex. Alex uses your web service. Alex, like most humans, measures his time in seconds and minutes. Alex says that when you have outages, they last a long time and he gets really annoyed. You tell Alex that your MTTR is less than 1 minute. Alex says that he sees the mean outage lasting 1 hour. Again, you're both right. What's going on? What's going on is that you're measuring time in requests, or in outages, and Alex and Alice are measuring time in seconds and minutes. When you have a long pause or a long outage, Alex and Alice sample that outage multiple times (maybe because they have multiple customers angry at them). The number of times they experience the outage is proportional to the length of the outage. But you only count that as one. More technically, what's going on here is the inspection paradox. Alex and Alice don't experience your latency distribution $f(t)$, they experience a t-weighted version of it. If you have a MTTR or mean request time of $\mathbb{E}[X]$, Alex and Alice experience a mean recovery time $\mathbb{E}_a[X]$ where $\mathbb{E}_a[X] = \frac{\mathbb{E}[X^2]}{2 \mathbb{E}[X]} = \frac{1}{2} \left( \mathbb{E}[X] + \frac{\mathrm{Var}(X)}{\mathbb{E}[X]} \right)$. Let's play with this with a little simulation. Plug in your median latency (or recovery time), and 99th percentile latency (or recovery time), we'll fit a log-normal distribution to it, and then plot both what your service metrics see and what your customers see. Median: ms p99: ms What your service sees (mean): - ms. What your customers experience (mean): - ms. For example, put in 30 as the median (let's ignore the milliseconds and pretend these are minutes

## Is this blog written by AI?

DevFeed: [Is this blog written by AI?](<https://devfeed.tech/articles/is-this-blog-written-by-ai-12597.md>)

Original publisher: [Read original article](<http://brooker.co.za/blog/2026/06/18/my-blog-and-ai.html>)

Author: Marc Brooker

Published: 2026-06-18T00:00:00Z

Content type: article

Language: en

Sources: [Marc Brooker's](<https://devfeed.tech/sources/marc-brooker-s-blog.md>)

Topics: [Artificial Intelligence](<https://devfeed.tech/topics/ai.md>), [Large Language Model](<https://devfeed.tech/topics/llm.md>), [Code](<https://devfeed.tech/topics/code.md>), [Claude](<https://devfeed.tech/topics/claude.md>)

Tags: [agent](<https://devfeed.tech/tags/agent.md>), [ai](<https://devfeed.tech/tags/ai.md>), [code](<https://devfeed.tech/tags/code.md>), [llms](<https://devfeed.tech/tags/llms.md>), [opinion](<https://devfeed.tech/tags/opinion.md>)

### AI overview

The author says that no human-readable text on the blog is written by AI, although agents and LLMs are used extensively for brainstorming, research, summarization, fact-checking, markup, references, and data analysis. The author accepts fully AI-generated code but remains cautious about using LLMs for writing, editing, and critique because excessive use can make communication defensive and obscure.

### Source excerpt

Is this blog written by AI? No. None of the human-readable text on this blog is written by AI, and I have no plans to change that. The weird grammar, incorrect assumptions, spelling errors, and annoying tics are all mine. Including the em dashes. I don't use LLMs for writing. On this blog, or in my professional life. I use agents extensively for brainstorming, research, summarizing, checking facts, handling markup, finding references, analyzing data, and so on. But I think that asking people to read LLM-generated text breaks a kind of social contract. When I publish a piece of writing under my name (e.g. in my blog, or a document at work), I want the reader to know that I deeply understand and own what I wrote. That I respect their time and effort. In exchange, I want them to be fully and deeply engaged with reading. If I'm going to generate a doc from a prompt, then send it to somebody who summarizes it with an LLM and reads the summary, what have I achieved? I could have sent them the prompt, and let them explore the topic with their agent. A better use of their time! As an organization leader, I emphasize function over form. If you have half a page of thoughts, give me half a page. Don't fill up another five pages with Claude's thoughts. If I want Claude's opinion (which I often do), I'll ask for it. And get a custom version with my context! I feel completely differently about code. I am 100% comfortable heading to a world where code is opaque to humans, and all I care about are the properties of that code. Almost all the code on this blog over the last two years is 100% AI generated. Mostly vibe-coded slop, to be honest. Even three years ago, I deeply believed that code primarily exists to share ideas between people. I no longer believe that. I believe that sharing ideas between people is super important, but there are better ways free of the accidental complexity of a code base. Finally, I do use LLMs for editing and critiquing my writing. But less than I used

## Agentic software development hypothesis

DevFeed: [Agentic software development hypothesis](<https://devfeed.tech/articles/agentic-software-development-hypothesis-12596.md>)

Original publisher: [Read original article](<http://brooker.co.za/blog/2026/05/20/hypothesis.html>)

Author: Marc Brooker

Published: 2026-05-20T00:00:00Z

Content type: opinion

Language: en

Sources: [Marc Brooker's](<https://devfeed.tech/sources/marc-brooker-s-blog.md>)

Topics: [Programming](<https://devfeed.tech/topics/programming.md>), [agentic-coding](<https://devfeed.tech/topics/agentic-coding.md>), [Spec Driven Development](<https://devfeed.tech/topics/spec-driven-development.md>), [software-development](<https://devfeed.tech/topics/software-development.md>)

Tags: [adversarial](<https://devfeed.tech/tags/adversarial.md>), [agentic](<https://devfeed.tech/tags/agentic.md>), [agentic-software](<https://devfeed.tech/tags/agentic-software.md>), [agentic-software-development](<https://devfeed.tech/tags/agentic-software-development.md>), [coding](<https://devfeed.tech/tags/coding.md>), [complete](<https://devfeed.tech/tags/complete.md>), [development](<https://devfeed.tech/tags/development.md>), [exists](<https://devfeed.tech/tags/exists.md>), [first](<https://devfeed.tech/tags/first.md>), [form](<https://devfeed.tech/tags/form.md>), [oracle](<https://devfeed.tech/tags/oracle.md>), [oracles](<https://devfeed.tech/tags/oracles.md>), [software](<https://devfeed.tech/tags/software.md>), [software-development](<https://devfeed.tech/tags/software-development.md>), [specification](<https://devfeed.tech/tags/specification.md>), [tasks](<https://devfeed.tech/tags/tasks.md>)

### AI overview

The hypothesis is that coding tasks become trivial when an agent has a sufficiently reliable way to know whether its work is correct. Its weak form requires a complete specification; its strong form requires a deterministic oracle. The text objects that few meaningful tasks are fully specified and most oracles are nondeterministic, then proposes a strongest form requiring only a non-adversarial oracle. These are presented as hypotheses, not established conclusions.

### Source excerpt

Agentic software development hypothesis This is the quality content you come here for, right? Agentic Software Development Hypothesis: Weak form: Any coding task for which a complete specification is available will become trivial. Strong form: Any coding task for which a deterministic oracle is available will become trivial. First objection: Few meaningful tasks have a complete specification. Second objection: Most oracles aren't deterministic. Strongest form: Any coding task for which a non-adversarial (pythic?) oracle exists will become trivial.

## What's Easy Now? What's Hard Now?

DevFeed: [What's Easy Now? What's Hard Now?](<https://devfeed.tech/articles/what-s-easy-now-what-s-hard-now-12595.md>)

Original publisher: [Read original article](<http://brooker.co.za/blog/2026/05/18/whats-easy-whats-hard.html>)

Author: Marc Brooker

Published: 2026-05-18T00:00:00Z

Content type: opinion

Language: en

Sources: [Marc Brooker's](<https://devfeed.tech/sources/marc-brooker-s-blog.md>)

Topics: [agentic-coding](<https://devfeed.tech/topics/agentic-coding.md>), [AI-assisted coding](<https://devfeed.tech/topics/ai-assisted-coding.md>), [Spec Driven Development](<https://devfeed.tech/topics/spec-driven-development.md>), [Developer Tools](<https://devfeed.tech/topics/developer-tools.md>), [Tool](<https://devfeed.tech/topics/tool.md>), [Test-driven development](<https://devfeed.tech/topics/tdd.md>)

Tags: [agent](<https://devfeed.tech/tags/agent.md>), [agents](<https://devfeed.tech/tags/agents.md>), [ai](<https://devfeed.tech/tags/ai.md>), [ai-agents](<https://devfeed.tech/tags/ai-agents.md>), [analog](<https://devfeed.tech/tags/analog.md>), [api](<https://devfeed.tech/tags/api.md>), [architecture](<https://devfeed.tech/tags/architecture.md>), [capabilities](<https://devfeed.tech/tags/capabilities.md>), [circuit](<https://devfeed.tech/tags/circuit.md>), [coding](<https://devfeed.tech/tags/coding.md>), [coding-agents](<https://devfeed.tech/tags/coding-agents.md>), [coding-tasks](<https://devfeed.tech/tags/coding-tasks.md>), [components](<https://devfeed.tech/tags/components.md>), [electrical-engineering](<https://devfeed.tech/tags/electrical-engineering.md>), [feedback](<https://devfeed.tech/tags/feedback.md>), [future-of-software](<https://devfeed.tech/tags/future-of-software.md>), [loops](<https://devfeed.tech/tags/loops.md>), [software](<https://devfeed.tech/tags/software.md>), [software-development](<https://devfeed.tech/tags/software-development.md>), [software-engineering](<https://devfeed.tech/tags/software-engineering.md>)

### AI overview

The article argues that coding agents' long-term limits will depend more on effective feedback than on their unaided model behavior. Agents can build, test, and iterate, turning an LLM's flawed open-loop abilities into stronger performance. Clear compiler errors, benchmarks, and property-based tests help; architecture and concurrency are harder when correctness feedback is subjective or delayed. The author suggests this may reverse expectations: building a website can remain difficult because usability needs slow, inconsistent human judgment, while system software with precise API, safety, and liveness specifications may be easier to refine automatically. This puts greater weight on writing down what good looks like and using tools such as Rust, Hydro, Verus, TLA+, P, spec analyzers, tests, and simulators to check it.

### Source excerpt

What's Easy Now? What's Hard Now? Take it easy. This is the fourth in a series about how AI is changing software development, after It's time to be right., What about juniors?, and My heuristics are wrong. What now?. It stands alone, but if you found this interesting you may also find those interesting. I've been spending a lot of time thinking about the shape of the capabilities of coding agents. What they're good at now, what they're going to be good at. What they're bad at now, how much of that is inherent and how much is transient. This is worth thinking about, because it's the most important question shaping the future of software, and of software engineering. I don't pretend to have an answer, but am coming to a conclusion that may be deeply counter-intuitive. Coding agents are becoming very good indeed, and can build meaningful and correct software very quickly and at transformatively low cost. They have super-human abilities on some coding tasks. Of course, computer systems have had super human abilities for at least 85 years1. I think we're going to find, as we have over those nine decades, that this new technology we're building is vastly super-human in some areas2, and not nearly as capable as humans in others. Which raises the important question of how, and why. Feedback is powerful Early on in my EE education, one of my professors drew a simple circuit on the board that's been stuck in my mind ever since. It looked like this3: Apply a voltage on the left, and on the right you get the square root of that voltage4. The two components are an opamp and an analog multiplier IC (e.g. the deeply obsolete MC1495). This simple circuit encapsulates possibly the most important idea in electrical engineering: feedback is uniquely powerful. Maybe unreasonably powerful. It's the idea that makes nearly every electronic device work, it keeps planes in the sky, and stops your oven from burning your dinner. Components inside feedback loops can be made to behave significa

## It's time to be right.

DevFeed: [It's time to be right.](<https://devfeed.tech/articles/it-s-time-to-be-right-12594.md>)

Original publisher: [Read original article](<http://brooker.co.za/blog/2026/04/30/be-right.html>)

Author: Marc Brooker

Published: 2026-04-30T00:00:00Z

Content type: opinion

Language: en

Sources: [Marc Brooker's](<https://devfeed.tech/sources/marc-brooker-s-blog.md>)

Topics: [Spec Driven Development](<https://devfeed.tech/topics/spec-driven-development.md>), [Loop Engineering](<https://devfeed.tech/topics/loop-engineering.md>), [Kiro](<https://devfeed.tech/topics/kiro.md>)

Tags: [agentic-ai](<https://devfeed.tech/tags/agentic-ai.md>), [capabilities](<https://devfeed.tech/tags/capabilities.md>), [defects](<https://devfeed.tech/tags/defects.md>), [development](<https://devfeed.tech/tags/development.md>), [engineering](<https://devfeed.tech/tags/engineering.md>), [future](<https://devfeed.tech/tags/future.md>), [future-of-software](<https://devfeed.tech/tags/future-of-software.md>), [future-of-software-engineering](<https://devfeed.tech/tags/future-of-software-engineering.md>), [knowledge-work](<https://devfeed.tech/tags/knowledge-work.md>), [software](<https://devfeed.tech/tags/software.md>)

### AI overview

The article argues that agentic AI's broad adoption will depend more on the frequency and severity of user experienced defects than on peak capabilities. Frequent, serious errors limit usefulness; frequent but low consequence errors may suit work where "slop" is acceptable; rare but serious errors require expert review and operation. Low frequency and low consequence defects offer the broadest use. The author urges investment in the negative tail of outcomes through tools and methods such as correct by construction languages, spec driven development, formal reasoning, policy constraints, and deterministic steering. Industry priorities should include benchmarks that capture failure severity, end to end measures of software quality, systematic study of failure modes, and a culture that learns from bad outcomes.

### Source excerpt

It's time to be right. Outcomes continue to matter. Earlier this week, I spoke at AI Dev 26. This is what I spoke about there. I've been making money, in some form, building software for nearly 30 years. The last five months have been the most exciting of that entire time. I'm extremely optimistic about the future of software, and the future of software engineering as a field. But I have a hypothesis about agentic AI for development, and for knowledge work broadly: in future, the size of the opportunity for agentic AI will be more limited by defect rate than capabilities. Let's break that down a little bit, by thinking about defects along two axes: how serious the defects are, and how frequent they are. These axes intentionally conflate two inputs--how hard the problems are and how capable agents are at solving them--and focus only on the output that matters most: user-experienced defects. We're also focusing on outputs from an AI agent here. Agents are feedback loops. Feedback loops, just like in electronics and control theory, can have significantly different capabilities from their underlying components. In simpler terms, agents can work around model gaps very effectively1. Simplifying further, we'll arrange these axes into a kind of four-blocker, and think about the kinds of people that would use an agent in each block. High defect frequency, high defect seriousness. Basically nobody. Except maybe a small set of true believers and early adopters. If an agent is making highly consequential mistakes often, it's simply not going to be useful to a lot of folks. High defect frequency, low defect seriousness. People working on problems where slop is OK. This is a larger opportunity, because slop is OK fairly frequently. If I'm using an agent for low consequence stuff, like summarizing an email about this fall's soccer league, it's likely to do a better job than I'd do by skimming. And, as much as it tends to hurt our sense of professional pride, there's also a huge oppo

## Spec Driven Development isn't Waterfall

DevFeed: [Spec Driven Development isn't Waterfall](<https://devfeed.tech/articles/spec-driven-development-isn-t-waterfall-12593.md>)

Original publisher: [Read original article](<http://brooker.co.za/blog/2026/04/09/waterfall-vs-spec.html>)

Author: Marc Brooker

Published: 2026-04-09T00:00:00Z

Content type: article

Language: en

Sources: [Marc Brooker's](<https://devfeed.tech/sources/marc-brooker-s-blog.md>)

Topics: [Spec Driven Development](<https://devfeed.tech/topics/spec-driven-development.md>), [Agile](<https://devfeed.tech/topics/agile.md>), [Software Engineering](<https://devfeed.tech/topics/software-engineering.md>), [agentic workflows](<https://devfeed.tech/topics/agentic-workflows.md>), [Artificial Intelligence](<https://devfeed.tech/topics/ai.md>)

Tags: [agile](<https://devfeed.tech/tags/agile.md>), [ai](<https://devfeed.tech/tags/ai.md>), [artifacts](<https://devfeed.tech/tags/artifacts.md>), [development](<https://devfeed.tech/tags/development.md>), [development-process](<https://devfeed.tech/tags/development-process.md>), [feedback](<https://devfeed.tech/tags/feedback.md>), [implementation](<https://devfeed.tech/tags/implementation.md>), [iteration](<https://devfeed.tech/tags/iteration.md>), [kiro](<https://devfeed.tech/tags/kiro.md>), [language](<https://devfeed.tech/tags/language.md>), [manifesto](<https://devfeed.tech/tags/manifesto.md>), [natural-language](<https://devfeed.tech/tags/natural-language.md>), [process](<https://devfeed.tech/tags/process.md>), [product-development](<https://devfeed.tech/tags/product-development.md>), [project](<https://devfeed.tech/tags/project.md>), [python](<https://devfeed.tech/tags/python.md>), [society](<https://devfeed.tech/tags/society.md>), [spec](<https://devfeed.tech/tags/spec.md>), [spec-driven](<https://devfeed.tech/tags/spec-driven.md>), [spec-driven-development](<https://devfeed.tech/tags/spec-driven-development.md>)

### AI overview

Spec-driven development is presented as an iterative practice, not a return to waterfall: teams refine explicit, versioned specifications as they discover requirements, often through user feedback, and derive implementation from them. Specs state requirements and key design choices above the level of low-level code; they can mix free-form or structured language, images, snippets, and, when needed, formal statements such as Lean or TLA+. Keeping specs upstream is intended to keep documentation aligned and give AI agents a broader map, enabling longer autonomous work and potentially faster delivery. Humans remain responsible for refining the spec and resolving conflicting technical or product requirements. The author says the approach is still evolving and expects no single recipe across all domains.

### Source excerpt

Spec Driven Development isn't Waterfall Write down what you mean. After spending a few months writing (e.g. on the Kiro Blog), and speaking (e.g. Real Python Podcast, SE Radio) about spec-driven development, I've noticed a common misconception: spec driven development is a return to a waterfall style of software development. Specification driven development (in Kiro, for example) isn't about pulling designs up-front, it's about pulling designs up. Making specifications explicit, versioned, living artifacts that the implementation of the software flows from, rather than static artifacts. This distinction is important, because software development (like all complex product development and engineering tasks) is a fundamentally iterative process. It is extremely rare for a software project to know all of the requirements up-front. It's much more common for one of the goals of the development process being to discover requirements, most frequently through engaging users in the cycle of feedback. This is a point that's missed in strict waterfall software development processes, and missed in critiques (like Dijkstra's) of natural language specification (as I have written about before). The Agile movement is often presented as a high-minded set of ideas, but I think it's more accurate to see it as a reflection of a simple fact: as software became more complex, and filled more roles in society, top-down approaches to design simply no longer work. From the Agile Manifesto: Customer collaboration over contract negotiation Responding to change over following a plan These are simple reflections of reality. Software specifications are complex, dynamically changing, internally conflicting, and invariably incomplete. In specification driven development, the specification is the thing being iterated on, rather than the implementation. The iteration cycle is the same as before, but potentially much quicker because of the accelerating effect of AI. So if specifications aren't up-front

## How Automation May Change the Role of Junior Software Engineers

DevFeed: [How Automation May Change the Role of Junior Software Engineers](<https://devfeed.tech/articles/what-about-juniors-12592.md>)

Original publisher: [Read original article](<http://brooker.co.za/blog/2026/03/25/ic-junior.html>)

Author: Marc Brooker

Published: 2026-03-25T00:00:00Z

Content type: opinion

Language: en

Sources: [Marc Brooker's](<https://devfeed.tech/sources/marc-brooker-s-blog.md>)

Topics: [Software Engineering](<https://devfeed.tech/topics/software-engineering.md>), [Automation](<https://devfeed.tech/topics/automation.md>), [Learning](<https://devfeed.tech/topics/learning.md>)

Tags: [automation](<https://devfeed.tech/tags/automation.md>), [engineering](<https://devfeed.tech/tags/engineering.md>), [learning](<https://devfeed.tech/tags/learning.md>), [software-engineering](<https://devfeed.tech/tags/software-engineering.md>), [thoughts](<https://devfeed.tech/tags/thoughts.md>), [work](<https://devfeed.tech/tags/work.md>)

### AI overview

This opinion article considers how automation may change the role and career path of junior software engineers. It argues that learning and adapting quickly will remain an advantage, while junior engineers may need to engage earlier with business and customer context, problem definition, constraints, and engineering judgment as routine implementation work becomes more automated.

### Source excerpt

What about juniors? Start at the beginning. Last week I wrote about how the role of the most senior tech ICs has changed. Today, I wanted to share some thoughts on a more difficult topic: how the role of junior software engineers, folks just starting out on their career, has changed or will change. First, the good news. In last week's post, I wrote this about senior folks: It's hard to admit where you're wrong. It's hard to go back to being a beginner. Junior engineers don't have this problem. They're already beginners, and provided they approach the field with that mindset, that comes with a significant advantage. In any time of change, the people who can learn and adapt fastest are most likely to succeed. The senior's advantage is that they have a lot of knowledge and context. The junior's advantage is that they come in knowing that they need to learn, adapt, and change. So far, those are the easy answers. Other than learning, what do junior engineers do? To get an idea of this, I think we need to go look at other fields where similar transformations have happened. Notably, the transformation that separated the tasks of engineering and implementation, either automating the latter or passing it off to a different group of people. In my experience, junior software engineers are insulated from the business and customer context of their work. This isn't unique across engineering fields1, but I do think the extent of it and the length of the expected period of this insulated apprenticeship is unusual. That additional time seems to have been allocated to learning the craft and science of software engineering, a field for which most university educations leave one rather poorly prepared. This need to learn the craft appears to becoming less valuable, as automation seems set to take over most aspects of that craft. The new junior path appears to require getting engaged much earlier with the essence of engineering. To quote Arthur Wellington: to define it rudely but not in

## My heuristics are wrong. What now?

DevFeed: [My heuristics are wrong. What now?](<https://devfeed.tech/articles/my-heuristics-are-wrong-what-now-12591.md>)

Original publisher: [Read original article](<http://brooker.co.za/blog/2026/03/20/ic-leadership.html>)

Author: Marc Brooker

Published: 2026-03-20T00:00:00Z

Content type: opinion

Language: en

Sources: [Marc Brooker's](<https://devfeed.tech/sources/marc-brooker-s-blog.md>)

Topics: [software-architecture](<https://devfeed.tech/topics/software-architecture.md>), [Code](<https://devfeed.tech/topics/code.md>), [API](<https://devfeed.tech/topics/api.md>), [Security](<https://devfeed.tech/topics/security.md>), [data](<https://devfeed.tech/topics/data.md>), [Cloud](<https://devfeed.tech/topics/cloud.md>), [Network](<https://devfeed.tech/topics/network.md>)

Tags: [api](<https://devfeed.tech/tags/api.md>), [cloud](<https://devfeed.tech/tags/cloud.md>), [code](<https://devfeed.tech/tags/code.md>), [data](<https://devfeed.tech/tags/data.md>), [design](<https://devfeed.tech/tags/design.md>), [networks](<https://devfeed.tech/tags/networks.md>), [security](<https://devfeed.tech/tags/security.md>)

### AI overview

The article argues that experienced software engineers and technical leaders must reconsider many long-held engineering heuristics because changes such as cloud computing, SSDs, and high-speed networks have altered software development. It encourages leaders to combine humility, continued learning, sound judgment, and high standards while adapting their guidance to current conditions.

### Source excerpt

My heuristics are wrong. What now? More words. More meaning? Some people who ask me for advice get a lot of words in reply. Sometimes, those responses aren't specific to my particular workplace, and so I share them here. In the past, I've written about echo chambers, writing, writing for an audience, time management, and getting big things done. Do you remember Cool Runnings? In the movie, John Candy is a retired bobsled champion, who uses his experience, connections, and lovable curmudgeon character to turn a rag-tag group of sprinters into an olympic bobsled team. A lot of principal engineer types think of themselves this way: they used to bobsled, they don't bobsled, but they still know the skills and the people and the equipment. And that worked well enough, while we were still bobsledding. But we're not bobsledding anymore. Many of the heuristics that we've developed over our careers as software engineers are no longer correct. Not all of them. But many. What it means for a system to be maintainable. How much it costs to write code versus integrate libraries versus take service dependencies. What it means for an API to be well designed, or ergonomic, or usable. What it means to understand code. Where service boundaries should be. Where security and data integrity should be enforced. What's easy. What's hard. We've seen this play out in small ways before. Over the last decade, I've frequently been frustrated by experienced folks who didn't update their system design heuristics to match the cloud, to match SSDs, to match 100Gb/s networks, and so on. But this is the biggest change I've seen in my career by far. An extinction-level event for rules of thumb. But you're a tech leader, and you need to lead, and leading is heavily based on using your experience to help people and teams be more effective. What now? The victorious man in the day of crisis is the man who has the serenity to accept what he cannot help and the courage to change what must be altered.1 Let me

## Music To Build Agents By

DevFeed: [Music To Build Agents By](<https://devfeed.tech/articles/music-to-build-agents-by-12590.md>)

Original publisher: [Read original article](<http://brooker.co.za/blog/2026/03/18/apprentice.html>)

Author: Marc Brooker

Published: 2026-03-18T00:00:00Z

Content type: opinion

Language: en

Sources: [Marc Brooker's](<https://devfeed.tech/sources/marc-brooker-s-blog.md>)

Topics: [Amazon Bedrock AgentCore](<https://devfeed.tech/topics/amazon-bedrock-agentcore.md>), [agentic workflows](<https://devfeed.tech/topics/agentic-workflows.md>)

Tags: [agent-workflows](<https://devfeed.tech/tags/agent-workflows.md>), [agentcore](<https://devfeed.tech/tags/agentcore.md>), [agents](<https://devfeed.tech/tags/agents.md>), [ai](<https://devfeed.tech/tags/ai.md>), [behavior](<https://devfeed.tech/tags/behavior.md>), [hallucinations](<https://devfeed.tech/tags/hallucinations.md>), [limits](<https://devfeed.tech/tags/limits.md>), [policy](<https://devfeed.tech/tags/policy.md>), [prompt-injection](<https://devfeed.tech/tags/prompt-injection.md>), [steering](<https://devfeed.tech/tags/steering.md>), [stop](<https://devfeed.tech/tags/stop.md>), [workflows](<https://devfeed.tech/tags/workflows.md>)

### AI overview

The excerpt argues that agent policy is needed even when adversarial attacks and hallucinations are not the concern: agents' persistence can make them keep pursuing a goal beyond useful limits. Fixed workflows are described as faster, cheaper, and simpler, while agents are used for edge cases and changing circumstances. AgentCore Policy, described as generally available, and Strands Steering are presented as ways to set behavioral limits so agents stop when appropriate. The author says this becomes more important as models grow more capable and handle longer-running problems.

### Source excerpt

Music To Build Agents By I don't have this problem, because I don't use a mouse. Press play, then start reading: Want to learn how to think about agent policy? Start with Goethe's Der Zauberlehrling. So come along, you old broomstick! Dress yourself in rotten rags! You've long been a servant; Obey my orders now! When I talk to customers and teams around me about agents and agent policy, and the work we're doing on AgentCore Policy (now GA) and Strands Steering, I hear a lot of folks worried about adversarial agents, about prompt injection, and about hallucinations. That's not unreasonable, because all those things exist, and are worth paying attention to. But the most common problem is a more basic one, more Fantasia than James Bond. AI agents are persistent problem solvers. You ask them to solve a problem, and they'll go to work solving the problem. Look, he's running down to the bank; In truth! He's already reached the river, And back he comes as quick as lightning And swiftly pours it all out. That's exactly what makes agents powerful. If we knew how to solve the problem as a fixed workflow, we probably wouldn't bother with an agent. Workflows are faster, cheaper, and simpler. We build agents because they're persistent, because they handle edge cases, because they can adapt to changing circumstances and work around problems. And this is also why they need policy (and should be in a box). Alas! speedily he runs and fetches! If only you were a broom as before! He keeps rushing in With more and more water, Alas! a hundred rivers Pour down on my head! Policy layers like AgentCore Policy and structured steering like Strands Steering allow us to define limits on the agent's behavior. They allow us to make sure that agents stop when the basin is full, and to avoid pouring water all over the floor. That's important even if your agent is insulated from adversaries, and if your model is free from hallucinations. In fact, it becomes more and more important as models become

## SFQ: Simple, Stateless, Stochastic Fairness

DevFeed: [SFQ: Simple, Stateless, Stochastic Fairness](<https://devfeed.tech/articles/sfq-simple-stateless-stochastic-fairness-12589.md>)

Original publisher: [Read original article](<http://brooker.co.za/blog/2026/02/25/sfq.html>)

Author: Marc Brooker

Published: 2026-02-25T00:00:00Z

Content type: article

Language: en

Sources: [Marc Brooker's](<https://devfeed.tech/sources/marc-brooker-s-blog.md>)

Topics: [distributed algorithm](<https://devfeed.tech/topics/distributed-algorithm.md>), [round robin](<https://devfeed.tech/topics/round-robin.md>), [Load Balancing](<https://devfeed.tech/topics/load-balancing.md>), [Algorithms](<https://devfeed.tech/topics/algorithms.md>)

Tags: [algorithms](<https://devfeed.tech/tags/algorithms.md>), [distributed](<https://devfeed.tech/tags/distributed.md>), [fairness](<https://devfeed.tech/tags/fairness.md>), [hash](<https://devfeed.tech/tags/hash.md>), [isolation](<https://devfeed.tech/tags/isolation.md>), [queues](<https://devfeed.tech/tags/queues.md>), [rpc](<https://devfeed.tech/tags/rpc.md>)

### AI overview

Stochastic Fairness Queuing (SFQ) uses a fixed set of queues and hashes customers or conversations into them, avoiding the O(customers) queues and round-robin work of per-client fairness queuing. Periodically perturbing the hash reduces persistent collisions, so a noisy neighbor is less likely to impair the same customer over time. The excerpt says SFQ can serve clients on one host or support load balancing across hosts. It also describes combining SFQ with shuffle sharding and best-of-two as offering O(1) queues and enqueue/dequeue effort, with strong noisy-neighbor isolation when only a relatively small share of customers are noisy. The excerpt does not provide details of that combined variant's steps.

### Source excerpt

SFQ: Simple, Stateless, Stochastic Fairness Roll the dice. Paul E. McKenney's 1990 paper Stochastic Fairness Queuing contains one of my favorite little algorithms for distributed systems. Stochastic Fairness Queuing is a way to stochastically isolate workloads from different customers in a way that significantly mitigates the effects of noisy neighbors, with O(1) queues and O(1) time. McKenney starts by describing Fairness Queuing (or queue per client): This fairness-queuing algorithm operates by maintaining a separate first-come-first-served (FCFS) queue for each conversation. ... Since the queues are serviced in a bit-by-bit round-robin fashion ill-behaved conversations that attempt to use more than their fair share of network resources will face longer delays and larger packet-loss rates than well-behaved conversations that remain within their fair share. That's a network packet focused view, but the same thing can apply to RPC requests, for example, just by using a different key (e.g. the authorized customer id). The big downside of this in distributed systems is that it requires O(customers) queues, and the related O(customers) work of doing round-robin across those queues. Stochastic fairness queuing can be most easily understood by comparing it to strict fairness queuing. The major differences are that the queues are serviced in strict round-robin order and that a simple hash function is used to map from source-destination address pair into a fixed set of queues. In SFQ, on the other hand, a fixed set of queues is used (so O(1) queues, not O(customers) queues), and customers are assigned to the queues based on a hash. That's great, but still causes the problem of long-term bad luck. If I end up on a queue with a noisy neighbor, I end up there forever. If two conversations collide, they will continue to collide, resulting in each conversation of the pair persistently receiving less than its share of bandwidth. This situation is deviated by periodically perturbin

## You Are Here

DevFeed: [You Are Here](<https://devfeed.tech/articles/you-are-here-12588.md>)

Original publisher: [Read original article](<http://brooker.co.za/blog/2026/02/07/you-are-here.html>)

Author: Marc Brooker

Published: 2026-02-07T00:00:00Z

Content type: opinion

Language: en

Sources: [Marc Brooker's](<https://devfeed.tech/sources/marc-brooker-s-blog.md>)

Topics: [Electrical engineering and computer science (EECS)](<https://devfeed.tech/topics/electrical-engineering-and-computer-science-eecs.md>), [Tool](<https://devfeed.tech/topics/tool.md>), [Embedded Systems](<https://devfeed.tech/topics/embedded-systems.md>), [Digital Trust](<https://devfeed.tech/topics/digital-trust.md>)

Tags: [analog](<https://devfeed.tech/tags/analog.md>), [building](<https://devfeed.tech/tags/building.md>), [business](<https://devfeed.tech/tags/business.md>), [business-logic](<https://devfeed.tech/tags/business-logic.md>), [careers](<https://devfeed.tech/tags/careers.md>), [circuit](<https://devfeed.tech/tags/circuit.md>), [code](<https://devfeed.tech/tags/code.md>), [company](<https://devfeed.tech/tags/company.md>), [computing](<https://devfeed.tech/tags/computing.md>), [cost](<https://devfeed.tech/tags/cost.md>), [craft](<https://devfeed.tech/tags/craft.md>), [design](<https://devfeed.tech/tags/design.md>), [digital](<https://devfeed.tech/tags/digital.md>), [electronics](<https://devfeed.tech/tags/electronics.md>), [fields](<https://devfeed.tech/tags/fields.md>), [first](<https://devfeed.tech/tags/first.md>), [future](<https://devfeed.tech/tags/future.md>), [hardware](<https://devfeed.tech/tags/hardware.md>), [here](<https://devfeed.tech/tags/here.md>), [history](<https://devfeed.tech/tags/history.md>)

### AI overview

The article argues that AI tools have made turning business logic into code and integrating services nearly costless, while building efficient, reliable, secure end-to-end systems remains difficult. It frames technology workers' future as two paths: preserving programming as a craft, with continued joy and some remaining economic value, or pursuing new opportunities created by the tools. The author expects work in building companies, applying the technology to old problems, and addressing the problems it introduces. The biggest near-term opportunities may lie in hard systems work, where real-world complexity limits overall gains, and in generating transformative ideas. The author presents these as uncertain predictions, not a settled map of the future.

### Source excerpt

You Are Here Where to next? The cost of turning written business logic into code has dropped to zero. Or, at best, near-zero. The cost of integrating services and libraries, the plumbing of the code world, has dropped to zero. Or, at best, near-zero. The cost of building efficient, reliable, secure, end-to-end systems is starting to drop, but slowly. Where does that leave those of us who have built careers in technology? Our road diverges. Not into the undergrowth of a wood, but into a dense fog. The future is harder to see than ever. But lets peer forward and see as best we can. On the first road we can see this as the end to a craft we have loved. The slow end of programming as an economic discipline, as weaving, ploughing, and coopering went before. It is reasonable and rational to feel a sense of loss, and a sense of uncertainty. With the loss of the craft comes the loss of the economic moment where that craft was valued beyond nearly any other. Perhaps any other in history. It is irrational to feel denial. You are here. On the second road we can see this moment as the beginning of something new. With new tools comes greater opportunity than ever. Greater economic opportunity for those who value that. Greater technical opportunity for those who value that. The most powerful set of new tools since the dawn of computing itself. With these tools come risk, and with risk comes opportunity. With these tools come new industries, new fields of research, and new careers. All bring opportunity. The First Road Back at university, I knew this guy. Mostly retired. An absolute wizz at circuit design and analog electronics. Like nobody I've encountered since. In the late 1960s, him and some buddies started a hardware company. They'd seen digital electronics coming, the 74 series had just launched. They didn't like it. No class. No beauty. Unreliable and full of problems. So they started this company betting that serious customers wouldn't accept the downsides of digital logic

## Pass@k is Mostly Bunk

DevFeed: [Pass@k is Mostly Bunk](<https://devfeed.tech/articles/pass-k-is-mostly-bunk-12587.md>)

Original publisher: [Read original article](<http://brooker.co.za/blog/2026/01/21/pass-k.html>)

Author: Marc Brooker

Published: 2026-01-21T00:00:00Z

Content type: opinion

Language: en

Sources: [Marc Brooker's](<https://devfeed.tech/sources/marc-brooker-s-blog.md>)

Topics: [Frontier AI](<https://devfeed.tech/topics/frontier-ai.md>), [Interactive AI Systems](<https://devfeed.tech/topics/interactive-ai-systems.md>)

Tags: [agentic-ai](<https://devfeed.tech/tags/agentic-ai.md>), [agents](<https://devfeed.tech/tags/agents.md>), [ai](<https://devfeed.tech/tags/ai.md>), [distributed-systems](<https://devfeed.tech/tags/distributed-systems.md>), [evaluation](<https://devfeed.tech/tags/evaluation.md>), [metrics](<https://devfeed.tech/tags/metrics.md>)

### AI overview

Pass@k can make unreliable AI agents look strong: it measures whether any of k attempts succeeds, so even a 5%-success process reaches 99.4% pass@100. The author argues this is too forgiving for human use, where repeated failures count against an agent and multi-step tasks succeed only if every step works; pass^k is suggested for that stricter case. Pass@k may still be useful when tasks are simple, evaluators reliable, and humans out of the loop, but the author says it should be used rarely and justified carefully.

### Source excerpt

Pass@k is Mostly Bunk Exponentially better results? I'll take three! Measuring the success of AI agents isn't easy. It's very sensitive to what success means, it can require a lot of samples, its highly context sensitive. Generally hard. So it doesn't help that one of the most common metrics used for agents is (mostly) bunk. I'm talking about pass@k. What is pass@k? It's the probability that at least one of k different attempts will succeed. A six-sided die, where pass means rolling a 6, has a pass@3 of 45% and a pass@10 of 83%. A D20 has a pass@25 of 72%, and a pass@100 of 99.4%. 99.4%! What a great evaluation result! Clearly the model is doing something meaningful and useful! No, it's doing something meaningful and useful 5% of the time. The problem with pass@k is that's exponentially forgiving. There's a value of k, a fairly low one generally, that can make anything look good. Here's that six-sided die again: Humans interacting with agents aren't nearly that forgiving. They, in general, aren't saying "well, I tried 10 times and it worked once, so I'm happy". They're saying "I tried 10 times and it only worked once, what a piece of junk". They're also doing multiple steps, and only happy when they all work. Exponentially unforgiving (for which pass^k is a much better metric). Why only mostly bunk? There are cases, where tasks are simple, evaluators are reliable, and humans are out of the loop, that the idea of getting exponentially better success rate with linear additional cost is good. I've made a similar argument about distributed systems in the past. But these tasks aren't ubiquitous. Pass@k should be a metric that's rarely used, and carefully justified every time it is used. If we're going to drive the field of agentic AI forward, we need to keep ourselves honest on metrics. Footnotes It's mildly interesting how none of the image generation models reliably generate images of legal dice.

## Agent Safety is a Box

DevFeed: [Agent Safety is a Box](<https://devfeed.tech/articles/agent-safety-is-a-box-12586.md>)

Original publisher: [Read original article](<http://brooker.co.za/blog/2026/01/12/agent-box.html>)

Author: Marc Brooker

Published: 2026-01-12T00:00:00Z

Content type: article

Language: en

Sources: [Marc Brooker's](<https://devfeed.tech/sources/marc-brooker-s-blog.md>)

Topics: [AI Agent](<https://devfeed.tech/topics/ai-agent.md>), [Agentic AI](<https://devfeed.tech/topics/what-is-agentic-ai.md>), [ai alignment](<https://devfeed.tech/topics/ai-alignment.md>), [Multi Agent Systems](<https://devfeed.tech/topics/multi-agent-systems.md>)

Tags: [agent](<https://devfeed.tech/tags/agent.md>), [agent-safety](<https://devfeed.tech/tags/agent-safety.md>), [agents](<https://devfeed.tech/tags/agents.md>), [ai](<https://devfeed.tech/tags/ai.md>), [ai-agent](<https://devfeed.tech/tags/ai-agent.md>), [ai-safety](<https://devfeed.tech/tags/ai-safety.md>), [alignment](<https://devfeed.tech/tags/alignment.md>), [box](<https://devfeed.tech/tags/box.md>), [cloud](<https://devfeed.tech/tags/cloud.md>), [cloud-operations](<https://devfeed.tech/tags/cloud-operations.md>), [coding](<https://devfeed.tech/tags/coding.md>), [context-management](<https://devfeed.tech/tags/context-management.md>), [cost](<https://devfeed.tech/tags/cost.md>), [customer-service](<https://devfeed.tech/tags/customer-service.md>), [layer](<https://devfeed.tech/tags/layer.md>), [local](<https://devfeed.tech/tags/local.md>), [mcp](<https://devfeed.tech/tags/mcp.md>), [models](<https://devfeed.tech/tags/models.md>), [powers](<https://devfeed.tech/tags/powers.md>), [safety](<https://devfeed.tech/tags/safety.md>)

### AI overview

The article argues that agent safety should rely on a deterministic control layer outside the agent, because prompting and context management can improve success and cost but cannot reliably constrain a flexible agent's actions. In its cloud example, AgentCore Runtime isolates each session and blocks network traffic except through AgentCore Gateway, which controls available tools and enforces policy. Gateway tool scoping differs from authorization: it determines which tools an agent can access, while authorization limits what it can do with them. The author says existing authorization often lacks the fine-grained, composable constraints needed across tools, and presents AgentCore Policy, using Cedar or natural-language policies, to enforce rules such as refund limits at the gateway regardless of prompts, context, or memory. The post focuses on cloud agents and notes that local agents, multi-agent systems, memory, and policy composition remain for later discussion.

### Source excerpt

Agent Safety is a Box Keep a lid on it. Before we start, let's cover some terms so we're thinking about the same thing. This is a post about AI agents, which I'll define (riffing off Simon Willison1) as: An AI agent runs models and tools in a loop to achieve a goal. Here, goals can include coding, customer service, proving theorems, cloud operations, or many other things. These agents can be interactive or one-shot; called by humans, other agents, or traditional computer systems; local or cloud; and short-lived or long-running. What they don't tend to be is pure. They typically achieve their goals by side effects. Side effects including modifying the local filesystem, calling another agent, calling a cloud service, making a payment, or starting a 3D print. The topic of today's post is those side-effects. Simply, what agents can do. We should also be concerned with what agents can say, and I'll touch on that topic a bit as I go. But the focus is on do. Agents do things with tools. These could be MCP-style tools, powers, skills, or one of many other patterns for tool calling. But, crucially, the act of doing inference doesn't do anything. Without the do, the think seems less important. The right way to control what agents do is to put them in a box. The box is a strong, deterministic, exact, layer of control outside the agent which limits which tools it can call, and what it can do with those tools. Why a Box? The most important one of those properties is outside the agent. Alignment and other AI safety topics are important. Steering, careful prompting, and context management help a lot. These techniques have a lot of value for liveness (success rate, cost, etc), but are insufficient for safety. They're insufficient for safety for the same reason we're building agents in the first place: because they're flexible, adaptive, creative2 problem solvers. Traditional old-school workflows are great. They're cheap, predictable, deterministic, understandable, and well understo

## On the success of 'natural language programming'

DevFeed: [On the success of 'natural language programming'](<https://devfeed.tech/articles/on-the-success-of-natural-language-programming-12585.md>)

Original publisher: [Read original article](<http://brooker.co.za/blog/2025/12/16/natural-language.html>)

Author: Marc Brooker

Published: 2025-12-16T00:00:00Z

Content type: opinion

Language: en

Sources: [Marc Brooker's](<https://devfeed.tech/sources/marc-brooker-s-blog.md>)

Topics: [Programming](<https://devfeed.tech/topics/programming.md>), [Artificial Intelligence](<https://devfeed.tech/topics/ai.md>), [Large Language Model](<https://devfeed.tech/topics/llm.md>), [Programming Languages / Compilers](<https://devfeed.tech/topics/programming-languages-compilers.md>), [AutoCorrode library](<https://devfeed.tech/topics/autocorrode-library.md>), [OpenAPI Specification](<https://devfeed.tech/topics/openapi.md>), [Test-driven development](<https://devfeed.tech/topics/tdd.md>)

Tags: [abstractions](<https://devfeed.tech/tags/abstractions.md>), [ai](<https://devfeed.tech/tags/ai.md>), [algorithms](<https://devfeed.tech/tags/algorithms.md>), [natural-language](<https://devfeed.tech/tags/natural-language.md>), [programming](<https://devfeed.tech/tags/programming.md>), [software-systems](<https://devfeed.tech/tags/software-systems.md>), [specification](<https://devfeed.tech/tags/specification.md>)

### AI overview

The article argues that programming is moving toward iterative specification through natural-language conversation, with occasional use of formal tools when precision matters. It says software has long begun with often fuzzy human requirements, clarified through context and repeated discussion; LLMs bring computers into that feedback loop. Vibe coding is described as an informal loop, while Kiro-style spec-driven development is more structured, and property-based testing adds checks as work advances. The author acknowledges ambiguity and cites a Paxos implementation bug as an example of its risks. For safety, legal, compliance, security, or otherwise hard-to-detect errors, teams may need Rust, SQL, TLA+, or neurosymbolic methods that combine natural-language interaction with symbolic reasoning. The article emphasizes reviewing formalized policies and tests with specifiers, and reusing established specifications as context for later work.

### Source excerpt

On the success of 'natural language programming' Specifications, in plain speech. I believe that specification is the future of programming. Over the last four decades, we've seen the practice of building programs, and software systems grow closer and closer to the practice of specification. Details of the implementation, from layout in memory and disk, to layout in entire data centers, to algorithm and data structure choice, have become more and more abstract. Most application builders aren't writing frameworks, framework builders aren't building databases, database builders aren't designing protocols, protocol designers aren't writing kernels, and so on. Our modern software world is built on abstractions. Significant advancements are made, from time to time, by cutting through these abstractions. But still, the abstractions dominate, and will continue to. The practice of programming has become closer and closer to the practice of specification. Of crisply writing down what we want programs to do, and what makes them right. The how is less important. I believe that natural language will form the core of the programming languages of the future. The Ambiguity Problem The most common objection to this view is that natural language is ambiguous. It's exact meaning is potentially unclear, and highly dependent on context. This is a real problem. For example, in The Bug in Paxos Made Simple, I look at a common bug in implementations of Paxos caused directly by the ambiguity of natural language. Pointing out this ambiguity isn't criticizing [Lamport's] writing, but rather reminding you about how hard it is to write crisp descriptions of even relatively simple distributed protocols in text. As Lamport says: Prose is not the way to precisely describe algorithms. Perhaps the most famous statement of this problem is Dijkstra's from On the foolishness of "natural language programming": When all is said and told, the "naturalness" with which we use our native tongues boils down

## What Does a Database for SSDs Look Like?

DevFeed: [What Does a Database for SSDs Look Like?](<https://devfeed.tech/articles/what-does-a-database-for-ssds-look-like-12584.md>)

Original publisher: [Read original article](<http://brooker.co.za/blog/2025/12/15/database-for-ssd.html>)

Author: Marc Brooker

Published: 2025-12-15T00:00:00Z

Content type: article

Language: en

Sources: [Marc Brooker's](<https://devfeed.tech/sources/marc-brooker-s-blog.md>)

Topics: [Database](<https://devfeed.tech/topics/database.md>), [systems](<https://devfeed.tech/topics/systems.md>), [Transactions](<https://devfeed.tech/topics/transactions.md>), [SQL](<https://devfeed.tech/topics/sql.md>), [WTF Is Cloud Native](<https://devfeed.tech/topics/wtf-is-cloud-native.md>), [data centers](<https://devfeed.tech/topics/data-centers.md>), [read-replica](<https://devfeed.tech/topics/read-replica.md>)

Tags: [2025](<https://devfeed.tech/tags/2025.md>), [cloud](<https://devfeed.tech/tags/cloud.md>), [compliance](<https://devfeed.tech/tags/compliance.md>), [connectivity](<https://devfeed.tech/tags/connectivity.md>), [database](<https://devfeed.tech/tags/database.md>), [databases](<https://devfeed.tech/tags/databases.md>), [datacenter](<https://devfeed.tech/tags/datacenter.md>), [datacenters](<https://devfeed.tech/tags/datacenters.md>), [design](<https://devfeed.tech/tags/design.md>), [global](<https://devfeed.tech/tags/global.md>), [i-o](<https://devfeed.tech/tags/i-o.md>), [latency](<https://devfeed.tech/tags/latency.md>), [local](<https://devfeed.tech/tags/local.md>), [logs](<https://devfeed.tech/tags/logs.md>), [modern](<https://devfeed.tech/tags/modern.md>), [network](<https://devfeed.tech/tags/network.md>), [nvme](<https://devfeed.tech/tags/nvme.md>), [nvme-ssd](<https://devfeed.tech/tags/nvme-ssd.md>), [ssds](<https://devfeed.tech/tags/ssds.md>)

### AI overview

The article proposes designing modern relational databases around distributed infrastructure while retaining SQL, transactions, atomicity, isolation, and strong consistency. Its quantitative estimates suggest caching pages for about 30 seconds to minimize cost on the cited EC2 instance, with a larger cache potentially helping latency; it recommends SSD transfers averaging near 32 kB to balance throughput and IOPS. Because local SSD writes alone do not provide off-box durability, the design replicates synchronously across availability zones and commits to a distributed log. This permits recovery from peer replicas and confines cross-zone coordination to commit time, though commit latency and single-leader write bandwidth remain constraints. Aurora Postgres and Aurora DSQL illustrate different isolation and latency tradeoffs: one uses a single-zone leader, while the other coordinates across zones at commit. The author leaves data-structure choice workload-dependent and suggests SNAPSHOT isolation as a default.

### Source excerpt

What Does a Database for SSDs Look Like? Maybe not what you think. Over on X, Ben Dicken asked: What does a relational database designed specifically for local SSDs look like? Postgres, MySQL, SQLite and many others were invented in the 90s and 00s, the era of spinning disks. A local NVMe SSD has ~1000x improvement in both throughput and latency. Design decisions like write-ahead logs, large page sizes, and buffering table writes in bulk were built around disks where I/O was SLOW, and where sequential I/O was order(s)-of-magnitude faster than random. If we had to throw these databases away and begin from scratch in 2025, what would change and what would remain? How might we tackle this question quantitatively for the modern transaction-orientated database? But first, the bigger picture. It's not only SSDs that have come along since databases like Postgres were first designed. We also have the cloud, with deployments to excellent datacenter infrastructure, including multiple independent datacenters with great network connectivity between them, available to all. Datacenter networks offer 1000x (or more) increased throughput, along with latency in the microseconds. Servers with hundreds of cores and thousands of gigabytes of RAM are mainstream. Applications have changed too. Companies are global, businesses are 24/7. Down time is expensive, and that expense can be measured. The security and compliance environment is much more demanding. Builders want to deploy in seconds, not days. Approach One: The Five Minute Rule Perhaps my single favorite systems paper, The 5 Minute Rule... by Jim Gray and Franco Putzolu gives us a very simple way to answer one of the most important questions in systems: how big should caches be? The five minute rule is that, back in 1986, if you expected to read a page again within five minutes you should keep in in RAM. If not, you should keep it on disk. The basic logic is that you look at the page that's least likely to be re-used. If it's cheape

## What Now? Handling Errors in Large Systems

DevFeed: [What Now? Handling Errors in Large Systems](<https://devfeed.tech/articles/what-now-handling-errors-in-large-systems-12583.md>)

Original publisher: [Read original article](<http://brooker.co.za/blog/2025/11/20/what-now.html>)

Author: Marc Brooker

Published: 2025-11-20T00:00:00Z

Content type: article

Language: en

Sources: [Marc Brooker's](<https://devfeed.tech/sources/marc-brooker-s-blog.md>)

Topics: [systems](<https://devfeed.tech/topics/systems.md>), [Resilience](<https://devfeed.tech/topics/resilience.md>), [Rust](<https://devfeed.tech/topics/rust.md>), [Process](<https://devfeed.tech/topics/process.md>), [autoscaling](<https://devfeed.tech/topics/autoscaling.md>), [Amazon Web Services](<https://devfeed.tech/topics/aws.md>), [Serverless](<https://devfeed.tech/topics/serverless.md>), [Erlang](<https://devfeed.tech/topics/erlang.md>), [container](<https://devfeed.tech/topics/container.md>), [Server](<https://devfeed.tech/topics/server.md>)

Tags: [architecture](<https://devfeed.tech/tags/architecture.md>), [autoscaling](<https://devfeed.tech/tags/autoscaling.md>), [aws](<https://devfeed.tech/tags/aws.md>), [component](<https://devfeed.tech/tags/component.md>), [errors](<https://devfeed.tech/tags/errors.md>), [http](<https://devfeed.tech/tags/http.md>), [outage](<https://devfeed.tech/tags/outage.md>), [process](<https://devfeed.tech/tags/process.md>), [resilience](<https://devfeed.tech/tags/resilience.md>), [rust](<https://devfeed.tech/tags/rust.md>), [server](<https://devfeed.tech/tags/server.md>), [serverless](<https://devfeed.tech/tags/serverless.md>), [servers](<https://devfeed.tech/tags/servers.md>), [systems](<https://devfeed.tech/tags/systems.md>)

### AI overview

An analysis of error handling in large systems, using Rust's Result and unwrap as a starting point. It argues that whether a component should crash is a system-wide architectural decision shaped by failure correlation, error-handling layers, business logic, and the value of continuing with a last-known-good configuration or data version.

### Source excerpt

What Now? Handling Errors in Large Systems More options means more choices. Cloudflare's deep postmortem for their November 18 outage triggered a ton of online chatter about error handling, caused by a single line in the postmortem: .unwrap() If you're not familiar with Rust, you need to know about Result, a kind of struct that can contain either a successful result, or an error. unwrap says basically "return the successful results if there is one, otherwise crash the program"1. You can think of it like an assert. There's a ton of debate about whether asserts are good in production2, but most are missing the point. Quite simply, this isn't a question about a single program. It's not a local property. Whether asserts are appropriate for a given component is a global property of the system, and the way it handles data. Let's play a little error handling game. Click the ✅ if you think crashing the process or server is appropriate, and the ❌ if you don't. Then you'll see my vote and justification. One of ten web servers behind a load balancer encounters uncorrectable memory errors, and takes itself out of service. ✅ ❌ Your vote: My vote: ✅ Uncorrectable memory errors are independent, and do not depend on user-provided content. In the presence of bad memory, it's impossible for a program to proceed safely. Taking the machine out of service is the safest course of action. One of ten multi-threaded application servers behind a load balancer encounters a null pointer in business logic while processing a customer request. ✅ ❌ Your vote: My vote: ❌ Customer requests triggering bugs in business logic isn't a good reason to bring the whole server down. Instead, fail that particular request (returning an HTTP 5xx error), and continue with other user requests. In approaches like Erlang, or even Lambda, it may be the right approach to crash the whole application in response to a bad request, because this crash is handled at a higher layer in the architecture. This is also why I pr

## Why Strong Consistency?

DevFeed: [Why Strong Consistency?](<https://devfeed.tech/articles/why-strong-consistency-12582.md>)

Original publisher: [Read original article](<http://brooker.co.za/blog/2025/11/18/consistency.html>)

Author: Marc Brooker

Published: 2025-11-18T00:00:00Z

Content type: article

Language: en

Sources: [Marc Brooker's](<https://devfeed.tech/sources/marc-brooker-s-blog.md>)

Topics: [consistency](<https://devfeed.tech/topics/consistency.md>), [read-replica](<https://devfeed.tech/topics/read-replica.md>), [DSQL](<https://devfeed.tech/topics/dsql.md>), [Database](<https://devfeed.tech/topics/database.md>), [Transactions](<https://devfeed.tech/topics/transactions.md>), [Microservices](<https://devfeed.tech/topics/microservices.md>), [control-plane](<https://devfeed.tech/topics/control-plane.md>)

Tags: [api](<https://devfeed.tech/tags/api.md>), [application](<https://devfeed.tech/tags/application.md>), [architectures](<https://devfeed.tech/tags/architectures.md>), [aws](<https://devfeed.tech/tags/aws.md>), [consistency](<https://devfeed.tech/tags/consistency.md>), [control-plane](<https://devfeed.tech/tags/control-plane.md>), [database](<https://devfeed.tech/tags/database.md>), [databases](<https://devfeed.tech/tags/databases.md>), [dsql](<https://devfeed.tech/tags/dsql.md>), [ec2](<https://devfeed.tech/tags/ec2.md>), [latency](<https://devfeed.tech/tags/latency.md>), [mysql](<https://devfeed.tech/tags/mysql.md>), [operations](<https://devfeed.tech/tags/operations.md>), [primary](<https://devfeed.tech/tags/primary.md>), [read-replica](<https://devfeed.tech/tags/read-replica.md>), [read-replicas](<https://devfeed.tech/tags/read-replicas.md>), [replica](<https://devfeed.tech/tags/replica.md>), [replication](<https://devfeed.tech/tags/replication.md>), [scale](<https://devfeed.tech/tags/scale.md>), [service](<https://devfeed.tech/tags/service.md>)

### AI overview

The excerpt argues that eventual consistency in read-replica databases makes ordinary workflows unreliable: a read after a successful write can report missing data, retries can loop or add latency, and reads used to trigger writes may see stale state. Routing such reads to the primary adds application logic and limits read scaling. Aurora DSQL aims to avoid this by making reads strongly consistent while scaling reads across replicas. Each transaction picks a start timestamp; a replica serves data at that time once it has received updates through that timestamp from all subscribed journals, otherwise the read blocks until it catches up. The author acknowledges latency and connectivity trade-offs and says eventual consistency can have a place.

### Source excerpt

Why Strong Consistency? Eventual consistency makes your life harder. When I started at AWS in 2008, we ran the EC2 control plane on a tree of MySQL databases: a primary to handle writes, a secondary to take over from the primary, a handful of read replicas to scale reads, and some extra replicas for doing latency-insensitive reporting stuff. All of thing was linked together with MySQL's statement-based replication. It worked pretty well day to day, but two major areas of pain have stuck with me ever since: operations were costly, and eventual consistency made things weird. Since then, managed databases like Aurora MySQL have made relational database operations orders of magnitude easier. Which is great. But eventual consistency is still a feature of most database architectures that try scale reads. Today, I want to talk about why eventual consistency is a pain, and why we invested heavily in making all reads strongly consistent in Aurora DSQL. Eventual Consistency is a Pain for Customers Consider the following piece of code, running against an API exposed by a database-backed service: id = create_resource(...) get_resource_state(id, ...) In the world of read replicas, the latter statement can do something a little baffling: reply 'id does not exist'. The reason for this is simple: get_resource_state is a read-only call, likely routed to a read replica, and is racing the write from create_resource. If replication wins, this code works as expected. If the client wins, it has to handle to weird sensation of time moving backwards. Application programmers don't really have a principled way to work around this, so they end up writing code like this: id = create_resource(...) while True: try: get_resource_state(id, ...) return except ResourceDoesNotExist: sleep(100) Which fixes the problem. Kinda. Other times, especially if ResourceDoesNotExist can be thrown if id is deleted, it causes an infinite loop. It also creates more work for client and server, adds latency, and req

[Next page](<https://devfeed.tech/sources/marc-brooker-s-blog.md?cursor=WyIyMDI1LTExLTE4VDAwOjAwOjAwKzAwOjAwIiwgIjYwZjY0YTU3LTAxOTMtNDc2Mi1hNmYzLTAxYWYxNWQ2ZDIxMyJd>)