# Marc Brooker's Blog Marc Brooker's Blog This is one page of public article previews, not the complete archive. Follow Next page to continue. Summaries are not the original full articles. ## Lorenz and Little: How Much Does Your Tail Cost? DevFeed: [Lorenz and Little: How Much Does Your Tail Cost?]() Original publisher: [Read original article]() Author: Marc Brooker Published: 2026-07-29T00:00:00Z Content type: article Language: en Sources: [Marc Brooker's Blog](), [Marc Brooker's Blog]() Topics: [Latency](), [Statistics](), [Optimization](), [Concurrency]() Tags: [2026](), [concurrency](), [cost](), [latency](), [optimization](), [statistics]() ### AI overview The article explains how tail-latency percentiles contribute to mean latency, concurrency, and service cost. It introduces the empirical Lorenz Curve and uses Little's law to argue that optimizing the tail can substantially reduce concurrency, capacity demand, and lock contention. ### Source excerpt Lorenz and Little: How Much Does Your Tail Cost? Lorenz and Little sounds like hipster burger bar from 2015. It's time for Marc's Amateur Statistics Corner! Today: why I pay a lot of attention to tail latency when optimizing cost. I've written before on the importance of tail latency for customer experience (e.g. in 2026, 2021, and 2021, and 2017). Today, I want to talk about tail latency from the perspective of cost and capacity. Like many system operators, I think about tail latency using percentiles. Here's a question: how much does each of my latency percentiles contributed to the mean latency? Intuitively, the answer is "quite a lot", but can we quantify that? We can! The thing we're looking for is the empirical Lorenz Curve. It directly calculates the answer to the question: given a latency percentile $P$ (e.g. p99=100ms), how much do requests taking shorter than $P$ contribute to the mean latency? (Let's call it $L(P)$ , so the real answer to our question is $1 - L(P)$). Starting from latency samples, the calculation is pretty simple: L = sum(sorted(x)[:k]) / sum(x) (for a set of n latency samples x, and k=p*n). From a vector of quantiles, things get a little more complicated, because we have to choose how to interpolate between the samples and extrapolate out to the maximum. Here I'm interpolating using a power law, which is a little bit of a sin1, but good enough for our purposes. # Calculate 1 - L(p) for a vector of measured quantiles # q - an array of quantiles (e.g. [1, 10, 200, 10000, 20000]) # p - an array of percentiles they're measured at (e.g. [0, 0.5, 0.9, 0.99, 0.999]) # OneMinusL - One minus the empirical Lorenz curve for each of the percentiles def OneMinusL(q, p): ...Show full implementationHide implementation # Calculate 1 - L(p) for a vector of measured quantiles # q - an array of quantiles (e.g. [1, 10, 200, 10000, 20000]) # p - an array of percentiles they're measured at (e.g. [0, 0.5, 0.9, 0.99, 0.999]) # OneMinusL - One minus the empirica ## Aurora DSQL: Scalable, Multi-Region OLTP DevFeed: [Aurora DSQL: Scalable, Multi-Region OLTP]() Original publisher: [Read original article]() Author: Marc Brooker Published: 2026-07-19T00:00:00Z Content type: article Language: en Sources: [Marc Brooker's Blog](), [Marc Brooker's Blog]() Topics: [DSQL](), [Databases](), [distributed-systems](), [Replication](), [Transactions](), [Amazon Web Services]() Tags: [aws](), [database](), [dsql](), [replication](), [systems](), [transactions]() ### AI overview The article introduces the Aurora DSQL paper, which explains the design of a scalable, multi-region OLTP relational database. It highlights DSQL's disaggregated architecture, transaction processing, replication, strongly consistent reads, operational simplicity, and potential benefits for resilient cloud applications and agentic builders. ### Source excerpt Aurora DSQL: Scalable, Multi-Region OLTP A paper! Our new paper, Aurora DSQL: Scalable, Multi-Region OLTP, is now available on Arxiv. I'm excited about this one: it's a fully end-to-end look at how Aurora DSQL works, from query processing, to transactions, to replication, to the control plane. We've shared most of this content before in other forms, on this blog, on Marc Bowes' Blog, Werner's Blog, in talks, and on the AWS blog. But this version covers all the ground, all in one place. You should read it. Or check out the interactive transaction simulator to get a feel for how it works. Some highlights Our overall goal was to build a relational database system that simplifies the work of application building and operations, freeing builders from worrying about scale, reliability, durability, and even multi-region fault tolerance. A database of first resort, which is simple and easy to adopt at low scale, and grows with the application, without adding complexity. I've written about this argument before (in DSQL: Simplifying Architectures), and it remains the thing that excites me most about the Aurora DSQL product. I believe we've made it much easier (and, in many cases, cheaper) to build and operate resilient, highly available, cloud applications at all scales. That applies to systems built by humans and agents, with the benefits magnified for agentic builders. Agents are great at building, but not (yet?) so great at long-horizon tasks like database operations. A database designed to simplify operations really helps. DSQL's architecture is disaggregated. Multiple independent services, each focused on a small number of well-defined concerns. DSQL isn't the first disaggregated OLTP database. Many, including Aurora and DynamoDB, came before. What's interesting here is how we learned from those systems, and the lessons that came from operating them at scale: avoiding large caches (they make failovers and read scale tricky), offering strongly consistent scalable reads (a ## Meet Alice. Alice is impatient. DevFeed: [Meet Alice. Alice is impatient.]() Original publisher: [Read original article]() Author: Marc Brooker Published: 2026-06-19T00:00:00Z Content type: article Language: en Sources: [Marc Brooker's Blog](), [Marc Brooker's Blog]() Topics: [Latency](), [Web](), [Code](), [Server]() Tags: [code](), [latency](), [metrics](), [outage](), [reasoning](), [recovery](), [server](), [time](), [web]() ### AI overview The article explains why customers may perceive web-service latency and outages as much longer than service-side averages suggest. It uses the inspection paradox to show that people disproportionately encounter longer events, presents a formula for the customer-observed mean recovery time, and describes a simulator based on median and p99 values. ### Source excerpt Meet Alice. Alice is impatient. What do you mean? Meet Alice. Alice uses your web service. Alice, like most humans, measures her time in seconds and minutes. Alice says your service is slow. You tell Alice that the mean request to your service completes in 100ms, but Alice says that her mean wait time is 1s. You're both right. Meet Alex. Alex uses your web service. Alex, like most humans, measures his time in seconds and minutes. Alex says that when you have outages, they last a long time and he gets really annoyed. You tell Alex that your MTTR is less than 1 minute. Alex says that he sees the mean outage lasting 1 hour. Again, you're both right. What's going on? What's going on is that you're measuring time in requests, or in outages, and Alex and Alice are measuring time in seconds and minutes. When you have a long pause or a long outage, Alex and Alice sample that outage multiple times (maybe because they have multiple customers angry at them). The number of times they experience the outage is proportional to the length of the outage. But you only count that as one. More technically, what's going on here is the inspection paradox. Alex and Alice don't experience your latency distribution $f(t)$, they experience a t-weighted version of it. If you have a MTTR or mean request time of $\mathbb{E}[X]$, Alex and Alice experience a mean recovery time $\mathbb{E}_a[X]$ where $\mathbb{E}_a[X] = \frac{\mathbb{E}[X^2]}{2 \mathbb{E}[X]} = \frac{1}{2} \left( \mathbb{E}[X] + \frac{\mathrm{Var}(X)}{\mathbb{E}[X]} \right)$. Let's play with this with a little simulation. Plug in your median latency (or recovery time), and 99th percentile latency (or recovery time), we'll fit a log-normal distribution to it, and then plot both what your service metrics see and what your customers see. Median: ms p99: ms What your service sees (mean): - ms. What your customers experience (mean): - ms. For example, put in 30 as the median (let's ignore the milliseconds and pretend these are minutes ## Is this blog written by AI? DevFeed: [Is this blog written by AI?]() Original publisher: [Read original article]() Author: Marc Brooker Published: 2026-06-18T00:00:00Z Content type: article Language: en Sources: [Marc Brooker's Blog](), [Marc Brooker's Blog]() Topics: [Artificial Intelligence](), [Large Language Model](), [Code](), [Claude]() Tags: [agent](), [ai](), [code](), [llms](), [opinion]() ### AI overview The author says that no human-readable text on the blog is written by AI, although agents and LLMs are used extensively for brainstorming, research, summarization, fact-checking, markup, references, and data analysis. The author accepts fully AI-generated code but remains cautious about using LLMs for writing, editing, and critique because excessive use can make communication defensive and obscure. ### Source excerpt Is this blog written by AI? No. None of the human-readable text on this blog is written by AI, and I have no plans to change that. The weird grammar, incorrect assumptions, spelling errors, and annoying tics are all mine. Including the em dashes. I don't use LLMs for writing. On this blog, or in my professional life. I use agents extensively for brainstorming, research, summarizing, checking facts, handling markup, finding references, analyzing data, and so on. But I think that asking people to read LLM-generated text breaks a kind of social contract. When I publish a piece of writing under my name (e.g. in my blog, or a document at work), I want the reader to know that I deeply understand and own what I wrote. That I respect their time and effort. In exchange, I want them to be fully and deeply engaged with reading. If I'm going to generate a doc from a prompt, then send it to somebody who summarizes it with an LLM and reads the summary, what have I achieved? I could have sent them the prompt, and let them explore the topic with their agent. A better use of their time! As an organization leader, I emphasize function over form. If you have half a page of thoughts, give me half a page. Don't fill up another five pages with Claude's thoughts. If I want Claude's opinion (which I often do), I'll ask for it. And get a custom version with my context! I feel completely differently about code. I am 100% comfortable heading to a world where code is opaque to humans, and all I care about are the properties of that code. Almost all the code on this blog over the last two years is 100% AI generated. Mostly vibe-coded slop, to be honest. Even three years ago, I deeply believed that code primarily exists to share ideas between people. I no longer believe that. I believe that sharing ideas between people is super important, but there are better ways free of the accidental complexity of a code base. Finally, I do use LLMs for editing and critiquing my writing. But less than I used ## Agentic software development hypothesis DevFeed: [Agentic software development hypothesis]() Original publisher: [Read original article]() Author: Marc Brooker Published: 2026-05-20T00:00:00Z Content type: opinion Language: en Sources: [Marc Brooker's Blog](), [Marc Brooker's Blog]() Topics: [coding](), [software-development](), [AI Development]() Tags: [agentic](), [agentic-ai](), [coding](), [development](), [software-development]() ### AI overview The article proposes an agentic software development hypothesis. It argues that coding tasks become trivial when they have a complete specification, a deterministic oracle, or, in its strongest form, a non-adversarial oracle. It also raises objections that meaningful tasks rarely have complete specifications and that most oracles are not deterministic. ### Source excerpt Agentic software development hypothesis This is the quality content you come here for, right? Agentic Software Development Hypothesis: Weak form: Any coding task for which a complete specification is available will become trivial. Strong form: Any coding task for which a deterministic oracle is available will become trivial. First objection: Few meaningful tasks have a complete specification. Second objection: Most oracles aren't deterministic. Strongest form: Any coding task for which a non-adversarial (pythic?) oracle exists will become trivial. ## What's Easy Now? What's Hard Now? DevFeed: [What's Easy Now? What's Hard Now?]() Original publisher: [Read original article]() Author: Marc Brooker Published: 2026-05-18T00:00:00Z Content type: opinion Language: en Sources: [Marc Brooker's Blog](), [Marc Brooker's Blog]() Topics: [AI-assisted coding](), [Artificial Intelligence](), [AI Development](), [Large Language Model](), [developer tooling](), [Software Engineering]() Tags: [ai](), [coding-agents](), [developer-tooling](), [llm](), [software-development](), [software-engineering]() ### AI overview The article examines what coding agents are good and bad at as AI changes software development. It argues that coding agents can rapidly build meaningful, correct software at very low cost and can outperform humans on some tasks, while remaining less capable in others. It explains this behavior through feedback loops: agents combine an LLM's useful but flawed standalone behavior with iterative feedback, enabling capabilities beyond the LLM alone. ### Source excerpt What's Easy Now? What's Hard Now? Take it easy. This is the fourth in a series about how AI is changing software development, after It's time to be right., What about juniors?, and My heuristics are wrong. What now?. It stands alone, but if you found this interesting you may also find those interesting. I've been spending a lot of time thinking about the shape of the capabilities of coding agents. What they're good at now, what they're going to be good at. What they're bad at now, how much of that is inherent and how much is transient. This is worth thinking about, because it's the most important question shaping the future of software, and of software engineering. I don't pretend to have an answer, but am coming to a conclusion that may be deeply counter-intuitive. Coding agents are becoming very good indeed, and can build meaningful and correct software very quickly and at transformatively low cost. They have super-human abilities on some coding tasks. Of course, computer systems have had super human abilities for at least 85 years1. I think we're going to find, as we have over those nine decades, that this new technology we're building is vastly super-human in some areas2, and not nearly as capable as humans in others. Which raises the important question of how, and why. Feedback is powerful Early on in my EE education, one of my professors drew a simple circuit on the board that's been stuck in my mind ever since. It looked like this3: Apply a voltage on the left, and on the right you get the square root of that voltage4. The two components are an opamp and an analog multiplier IC (e.g. the deeply obsolete MC1495). This simple circuit encapsulates possibly the most important idea in electrical engineering: feedback is uniquely powerful. Maybe unreasonably powerful. It's the idea that makes nearly every electronic device work, it keeps planes in the sky, and stops your oven from burning your dinner. Components inside feedback loops can be made to behave significa ## It's time to be right. DevFeed: [It's time to be right.]() Original publisher: [Read original article]() Author: Marc Brooker Published: 2026-04-30T00:00:00Z Content type: opinion Language: en Sources: [Marc Brooker's Blog](), [Marc Brooker's Blog]() Topics: [AI Agent](), [Artificial Intelligence](), [Software Engineering](), [Software](), [debugging]() Tags: [agentic-ai](), [ai](), [ai-agent](), [development](), [software](), [software-engineering]() ### AI overview The article argues that the future opportunity for agentic AI in software development and knowledge work will be constrained more by defect rate than by capability. It examines defects by frequency and seriousness, describing how agents can work around gaps in their underlying models. Lower-consequence uses, including one-off scripts, experiments, tools, and basic UIs, are presented as a substantial opportunity, while consequential work requires much stronger reliability and expert oversight. ### Source excerpt It's time to be right. Outcomes continue to matter. Earlier this week, I spoke at AI Dev 26. This is what I spoke about there. I've been making money, in some form, building software for nearly 30 years. The last five months have been the most exciting of that entire time. I'm extremely optimistic about the future of software, and the future of software engineering as a field. But I have a hypothesis about agentic AI for development, and for knowledge work broadly: in future, the size of the opportunity for agentic AI will be more limited by defect rate than capabilities. Let's break that down a little bit, by thinking about defects along two axes: how serious the defects are, and how frequent they are. These axes intentionally conflate two inputs--how hard the problems are and how capable agents are at solving them--and focus only on the output that matters most: user-experienced defects. We're also focusing on outputs from an AI agent here. Agents are feedback loops. Feedback loops, just like in electronics and control theory, can have significantly different capabilities from their underlying components. In simpler terms, agents can work around model gaps very effectively1. Simplifying further, we'll arrange these axes into a kind of four-blocker, and think about the kinds of people that would use an agent in each block. High defect frequency, high defect seriousness. Basically nobody. Except maybe a small set of true believers and early adopters. If an agent is making highly consequential mistakes often, it's simply not going to be useful to a lot of folks. High defect frequency, low defect seriousness. People working on problems where slop is OK. This is a larger opportunity, because slop is OK fairly frequently. If I'm using an agent for low consequence stuff, like summarizing an email about this fall's soccer league, it's likely to do a better job than I'd do by skimming. And, as much as it tends to hurt our sense of professional pride, there's also a huge oppo ## Spec Driven Development isn't Waterfall DevFeed: [Spec Driven Development isn't Waterfall]() Original publisher: [Read original article]() Author: Marc Brooker Published: 2026-04-09T00:00:00Z Content type: opinion Language: en Sources: [Marc Brooker's Blog](), [Marc Brooker's Blog]() Topics: [Spec Driven Development](), [Agile](), [Kiro](), [Software Engineering](), [Artificial Intelligence]() Tags: [agile](), [ai](), [blog](), [development](), [engineering](), [software-development](), [spec-driven-development](), [writing]() ### AI overview The article argues that spec-driven development is not a return to waterfall development. It presents specifications as explicit, versioned, living artifacts that evolve through user feedback and iterative development, with implementation derived from changing requirements and design choices. In this view, AI can accelerate the iteration cycle without making the process rigid or top-down. ### Source excerpt Spec Driven Development isn't Waterfall Write down what you mean. After spending a few months writing (e.g. on the Kiro Blog), and speaking (e.g. Real Python Podcast, SE Radio) about spec-driven development, I've noticed a common misconception: spec driven development is a return to a waterfall style of software development. Specification driven development (in Kiro, for example) isn't about pulling designs up-front, it's about pulling designs up. Making specifications explicit, versioned, living artifacts that the implementation of the software flows from, rather than static artifacts. This distinction is important, because software development (like all complex product development and engineering tasks) is a fundamentally iterative process. It is extremely rare for a software project to know all of the requirements up-front. It's much more common for one of the goals of the development process being to discover requirements, most frequently through engaging users in the cycle of feedback. This is a point that's missed in strict waterfall software development processes, and missed in critiques (like Dijkstra's) of natural language specification (as I have written about before). The Agile movement is often presented as a high-minded set of ideas, but I think it's more accurate to see it as a reflection of a simple fact: as software became more complex, and filled more roles in society, top-down approaches to design simply no longer work. From the Agile Manifesto: Customer collaboration over contract negotiation Responding to change over following a plan These are simple reflections of reality. Software specifications are complex, dynamically changing, internally conflicting, and invariably incomplete. In specification driven development, the specification is the thing being iterated on, rather than the implementation. The iteration cycle is the same as before, but potentially much quicker because of the accelerating effect of AI. So if specifications aren't up-front ## How Automation May Change the Role of Junior Software Engineers DevFeed: [How Automation May Change the Role of Junior Software Engineers]() Original publisher: [Read original article]() Author: Marc Brooker Published: 2026-03-25T00:00:00Z Content type: opinion Language: en Sources: [Marc Brooker's Blog](), [Marc Brooker's Blog]() Topics: [Software Engineering](), [Automation](), [Learning]() Tags: [automation](), [engineering](), [learning](), [software-engineering](), [thoughts](), [work]() ### AI overview This opinion article considers how automation may change the role and career path of junior software engineers. It argues that learning and adapting quickly will remain an advantage, while junior engineers may need to engage earlier with business and customer context, problem definition, constraints, and engineering judgment as routine implementation work becomes more automated. ### Source excerpt What about juniors? Start at the beginning. Last week I wrote about how the role of the most senior tech ICs has changed. Today, I wanted to share some thoughts on a more difficult topic: how the role of junior software engineers, folks just starting out on their career, has changed or will change. First, the good news. In last week's post, I wrote this about senior folks: It's hard to admit where you're wrong. It's hard to go back to being a beginner. Junior engineers don't have this problem. They're already beginners, and provided they approach the field with that mindset, that comes with a significant advantage. In any time of change, the people who can learn and adapt fastest are most likely to succeed. The senior's advantage is that they have a lot of knowledge and context. The junior's advantage is that they come in knowing that they need to learn, adapt, and change. So far, those are the easy answers. Other than learning, what do junior engineers do? To get an idea of this, I think we need to go look at other fields where similar transformations have happened. Notably, the transformation that separated the tasks of engineering and implementation, either automating the latter or passing it off to a different group of people. In my experience, junior software engineers are insulated from the business and customer context of their work. This isn't unique across engineering fields1, but I do think the extent of it and the length of the expected period of this insulated apprenticeship is unusual. That additional time seems to have been allocated to learning the craft and science of software engineering, a field for which most university educations leave one rather poorly prepared. This need to learn the craft appears to becoming less valuable, as automation seems set to take over most aspects of that craft. The new junior path appears to require getting engaged much earlier with the essence of engineering. To quote Arthur Wellington: to define it rudely but not in ## My heuristics are wrong. What now? DevFeed: [My heuristics are wrong. What now?]() Original publisher: [Read original article]() Author: Marc Brooker Published: 2026-03-20T00:00:00Z Content type: opinion Language: en Sources: [Marc Brooker's Blog](), [Marc Brooker's Blog]() Topics: [Architecture & Design](), [Code](), [API](), [Security](), [data](), [Cloud](), [Networks]() Tags: [api](), [cloud](), [code](), [data](), [design](), [networks](), [security]() ### AI overview The article argues that experienced software engineers and technical leaders must reconsider many long-held engineering heuristics because changes such as cloud computing, SSDs, and high-speed networks have altered software development. It encourages leaders to combine humility, continued learning, sound judgment, and high standards while adapting their guidance to current conditions. ### Source excerpt My heuristics are wrong. What now? More words. More meaning? Some people who ask me for advice get a lot of words in reply. Sometimes, those responses aren't specific to my particular workplace, and so I share them here. In the past, I've written about echo chambers, writing, writing for an audience, time management, and getting big things done. Do you remember Cool Runnings? In the movie, John Candy is a retired bobsled champion, who uses his experience, connections, and lovable curmudgeon character to turn a rag-tag group of sprinters into an olympic bobsled team. A lot of principal engineer types think of themselves this way: they used to bobsled, they don't bobsled, but they still know the skills and the people and the equipment. And that worked well enough, while we were still bobsledding. But we're not bobsledding anymore. Many of the heuristics that we've developed over our careers as software engineers are no longer correct. Not all of them. But many. What it means for a system to be maintainable. How much it costs to write code versus integrate libraries versus take service dependencies. What it means for an API to be well designed, or ergonomic, or usable. What it means to understand code. Where service boundaries should be. Where security and data integrity should be enforced. What's easy. What's hard. We've seen this play out in small ways before. Over the last decade, I've frequently been frustrated by experienced folks who didn't update their system design heuristics to match the cloud, to match SSDs, to match 100Gb/s networks, and so on. But this is the biggest change I've seen in my career by far. An extinction-level event for rules of thumb. But you're a tech leader, and you need to lead, and leading is heavily based on using your experience to help people and teams be more effective. What now? The victorious man in the day of crisis is the man who has the serenity to accept what he cannot help and the courage to change what must be altered.1 Let me ## Music To Build Agents By DevFeed: [Music To Build Agents By]() Original publisher: [Read original article]() Author: Marc Brooker Published: 2026-03-18T00:00:00Z Content type: article Language: en Sources: [Marc Brooker's Blog](), [Marc Brooker's Blog]() Topics: [Artificial Intelligence](), [prompt injection]() Tags: [agents](), [ai](), [ai-agents](), [policy](), [prompt-injection](), [workflow]() ### AI overview The article uses Goethe's The Sorcerer's Apprentice to explain why persistent AI agents need policy. Agents are valuable because they handle edge cases and adapt to changing circumstances, but their persistence can cause them to continue acting beyond intended limits. Policy layers such as AgentCore Policy and structured steering such as Strands Steering can constrain agent behavior, even when adversarial behavior and hallucinations are not concerns. ### Source excerpt Music To Build Agents By I don't have this problem, because I don't use a mouse. Press play, then start reading: Want to learn how to think about agent policy? Start with Goethe's Der Zauberlehrling. So come along, you old broomstick! Dress yourself in rotten rags! You've long been a servant; Obey my orders now! When I talk to customers and teams around me about agents and agent policy, and the work we're doing on AgentCore Policy (now GA) and Strands Steering, I hear a lot of folks worried about adversarial agents, about prompt injection, and about hallucinations. That's not unreasonable, because all those things exist, and are worth paying attention to. But the most common problem is a more basic one, more Fantasia than James Bond. AI agents are persistent problem solvers. You ask them to solve a problem, and they'll go to work solving the problem. Look, he's running down to the bank; In truth! He's already reached the river, And back he comes as quick as lightning And swiftly pours it all out. That's exactly what makes agents powerful. If we knew how to solve the problem as a fixed workflow, we probably wouldn't bother with an agent. Workflows are faster, cheaper, and simpler. We build agents because they're persistent, because they handle edge cases, because they can adapt to changing circumstances and work around problems. And this is also why they need policy (and should be in a box). Alas! speedily he runs and fetches! If only you were a broom as before! He keeps rushing in With more and more water, Alas! a hundred rivers Pour down on my head! Policy layers like AgentCore Policy and structured steering like Strands Steering allow us to define limits on the agent's behavior. They allow us to make sure that agents stop when the basin is full, and to avoid pouring water all over the floor. That's important even if your agent is insulated from adversaries, and if your model is free from hallucinations. In fact, it becomes more and more important as models become ## SFQ: Simple, Stateless, Stochastic Fairness DevFeed: [SFQ: Simple, Stateless, Stochastic Fairness]() Original publisher: [Read original article]() Author: Marc Brooker Published: 2026-02-25T00:00:00Z Content type: opinion Language: en Sources: [Marc Brooker's Blog](), [Marc Brooker's Blog]() Topics: [Algorithms](), [distributed-systems](), [Remote Procedure Call (RPC)](), [Network](), [systems]() Tags: [algorithms](), [customer](), [distributed-systems](), [network](), [queuing](), [rpc]() ### AI overview The article explains Stochastic Fairness Queuing (SFQ), an algorithm for isolating workloads from different customers and mitigating noisy-neighbor effects with O(1) queues and O(1) time. It contrasts SFQ with per-client fairness queuing, explains hash-based queue assignment and periodic hash perturbation, and discusses applying the approach to network traffic, RPC requests, single-host services, and load balancing across hosts. ### Source excerpt SFQ: Simple, Stateless, Stochastic Fairness Roll the dice. Paul E. McKenney's 1990 paper Stochastic Fairness Queuing contains one of my favorite little algorithms for distributed systems. Stochastic Fairness Queuing is a way to stochastically isolate workloads from different customers in a way that significantly mitigates the effects of noisy neighbors, with O(1) queues and O(1) time. McKenney starts by describing Fairness Queuing (or queue per client): This fairness-queuing algorithm operates by maintaining a separate first-come-first-served (FCFS) queue for each conversation. ... Since the queues are serviced in a bit-by-bit round-robin fashion ill-behaved conversations that attempt to use more than their fair share of network resources will face longer delays and larger packet-loss rates than well-behaved conversations that remain within their fair share. That's a network packet focused view, but the same thing can apply to RPC requests, for example, just by using a different key (e.g. the authorized customer id). The big downside of this in distributed systems is that it requires O(customers) queues, and the related O(customers) work of doing round-robin across those queues. Stochastic fairness queuing can be most easily understood by comparing it to strict fairness queuing. The major differences are that the queues are serviced in strict round-robin order and that a simple hash function is used to map from source-destination address pair into a fixed set of queues. In SFQ, on the other hand, a fixed set of queues is used (so O(1) queues, not O(customers) queues), and customers are assigned to the queues based on a hash. That's great, but still causes the problem of long-term bad luck. If I end up on a queue with a noisy neighbor, I end up there forever. If two conversations collide, they will continue to collide, resulting in each conversation of the pair persistently receiving less than its share of bandwidth. This situation is deviated by periodically perturbin ## Technology Careers as Programming Costs Fall DevFeed: [Technology Careers as Programming Costs Fall]() Original publisher: [Read original article]() Author: Marc Brooker Published: 2026-02-07T00:00:00Z Content type: opinion Language: en Sources: [Marc Brooker's Blog](), [Marc Brooker's Blog]() Topics: [Programming](), [Code]() Tags: [programming](), [tools]() ### AI overview An opinion piece argues that the cost of turning business logic into code and integrating services and libraries is now near zero, while building reliable, secure end-to-end systems is becoming cheaper more slowly. It considers the resulting uncertainty and opportunities for technology careers. ### Source excerpt You Are Here Where to next? The cost of turning written business logic into code has dropped to zero. Or, at best, near-zero. The cost of integrating services and libraries, the plumbing of the code world, has dropped to zero. Or, at best, near-zero. The cost of building efficient, reliable, secure, end-to-end systems is starting to drop, but slowly. Where does that leave those of us who have built careers in technology? Our road diverges. Not into the undergrowth of a wood, but into a dense fog. The future is harder to see than ever. But lets peer forward and see as best we can. On the first road we can see this as the end to a craft we have loved. The slow end of programming as an economic discipline, as weaving, ploughing, and coopering went before. It is reasonable and rational to feel a sense of loss, and a sense of uncertainty. With the loss of the craft comes the loss of the economic moment where that craft was valued beyond nearly any other. Perhaps any other in history. It is irrational to feel denial. You are here. On the second road we can see this moment as the beginning of something new. With new tools comes greater opportunity than ever. Greater economic opportunity for those who value that. Greater technical opportunity for those who value that. The most powerful set of new tools since the dawn of computing itself. With these tools come risk, and with risk comes opportunity. With these tools come new industries, new fields of research, and new careers. All bring opportunity. The First Road Back at university, I knew this guy. Mostly retired. An absolute wizz at circuit design and analog electronics. Like nobody I've encountered since. In the late 1960s, him and some buddies started a hardware company. They'd seen digital electronics coming, the 74 series had just launched. They didn't like it. No class. No beauty. Unreliable and full of problems. So they started this company betting that serious customers wouldn't accept the downsides of digital logic ## Pass@k is Mostly Bunk DevFeed: [Pass@k is Mostly Bunk]() Original publisher: [Read original article]() Author: Marc Brooker Published: 2026-01-21T00:00:00Z Content type: opinion Language: en Sources: [Marc Brooker's Blog](), [Marc Brooker's Blog]() Topics: [Artificial Intelligence](), [distributed-systems]() Tags: [agentic-ai](), [agents](), [ai](), [ai-agents](), [distributed-systems](), [evaluation](), [metrics]() ### AI overview The article argues that pass@k, the probability that at least one of k attempts succeeds, is usually a misleading metric for evaluating AI agents. Because it becomes exponentially more forgiving as k increases, it can make unreliable systems appear effective. The author contrasts this with human expectations and multi-step tasks, while acknowledging that pass@k can be appropriate for simple, reliably evaluated tasks without human involvement. ### Source excerpt Pass@k is Mostly Bunk Exponentially better results? I'll take three! Measuring the success of AI agents isn't easy. It's very sensitive to what success means, it can require a lot of samples, its highly context sensitive. Generally hard. So it doesn't help that one of the most common metrics used for agents is (mostly) bunk. I'm talking about pass@k. What is pass@k? It's the probability that at least one of k different attempts will succeed. A six-sided die, where pass means rolling a 6, has a pass@3 of 45% and a pass@10 of 83%. A D20 has a pass@25 of 72%, and a pass@100 of 99.4%. 99.4%! What a great evaluation result! Clearly the model is doing something meaningful and useful! No, it's doing something meaningful and useful 5% of the time. The problem with pass@k is that's exponentially forgiving. There's a value of k, a fairly low one generally, that can make anything look good. Here's that six-sided die again: Humans interacting with agents aren't nearly that forgiving. They, in general, aren't saying "well, I tried 10 times and it worked once, so I'm happy". They're saying "I tried 10 times and it only worked once, what a piece of junk". They're also doing multiple steps, and only happy when they all work. Exponentially unforgiving (for which pass^k is a much better metric). Why only mostly bunk? There are cases, where tasks are simple, evaluators are reliable, and humans are out of the loop, that the idea of getting exponentially better success rate with linear additional cost is good. I've made a similar argument about distributed systems in the past. But these tasks aren't ubiquitous. Pass@k should be a metric that's rarely used, and carefully justified every time it is used. If we're going to drive the field of agentic AI forward, we need to keep ourselves honest on metrics. Footnotes It's mildly interesting how none of the image generation models reliably generate images of legal dice. ## Agent Safety is a Box DevFeed: [Agent Safety is a Box]() Original publisher: [Read original article]() Author: Marc Brooker Published: 2026-01-12T00:00:00Z Content type: opinion Language: en Sources: [Marc Brooker's Blog](), [Marc Brooker's Blog]() Topics: [AI Agent](), [ai safety](), [Responsibility & Safety](), [Model Context Protocol]() Tags: [agent](), [ai-agent](), [ai-agents](), [ai-safety](), [mcp]() ### AI overview The article argues that AI agents should be constrained by a strong, deterministic control layer outside the agent--a "box"--that limits which tools they can call and what they can do. It distinguishes safety from prompting, steering, and context management, which may improve success rates and cost but cannot provide sufficient guarantees for flexible and adaptive agents acting through side effects. ### Source excerpt Agent Safety is a Box Keep a lid on it. Before we start, let's cover some terms so we're thinking about the same thing. This is a post about AI agents, which I'll define (riffing off Simon Willison1) as: An AI agent runs models and tools in a loop to achieve a goal. Here, goals can include coding, customer service, proving theorems, cloud operations, or many other things. These agents can be interactive or one-shot; called by humans, other agents, or traditional computer systems; local or cloud; and short-lived or long-running. What they don't tend to be is pure. They typically achieve their goals by side effects. Side effects including modifying the local filesystem, calling another agent, calling a cloud service, making a payment, or starting a 3D print. The topic of today's post is those side-effects. Simply, what agents can do. We should also be concerned with what agents can say, and I'll touch on that topic a bit as I go. But the focus is on do. Agents do things with tools. These could be MCP-style tools, powers, skills, or one of many other patterns for tool calling. But, crucially, the act of doing inference doesn't do anything. Without the do, the think seems less important. The right way to control what agents do is to put them in a box. The box is a strong, deterministic, exact, layer of control outside the agent which limits which tools it can call, and what it can do with those tools. Why a Box? The most important one of those properties is outside the agent. Alignment and other AI safety topics are important. Steering, careful prompting, and context management help a lot. These techniques have a lot of value for liveness (success rate, cost, etc), but are insufficient for safety. They're insufficient for safety for the same reason we're building agents in the first place: because they're flexible, adaptive, creative2 problem solvers. Traditional old-school workflows are great. They're cheap, predictable, deterministic, understandable, and well understo ## On the success of 'natural language programming' DevFeed: [On the success of 'natural language programming']() Original publisher: [Read original article]() Author: Marc Brooker Published: 2025-12-16T00:00:00Z Content type: opinion Language: en Sources: [Marc Brooker's Blog](), [Marc Brooker's Blog]() Topics: [Programming](), [Code](), [Protocol (disambiguation)](), [Mathematics](), [Refactoring]() Tags: [algorithms](), [programming](), [reasoning](), [software](), [writing]() ### AI overview The article argues that programming has increasingly become specification: developers describe what software should do while implementation details are hidden behind layers of abstraction. It proposes that natural language may become central to future programming languages, while acknowledging that ambiguity makes natural-language specifications difficult to use precisely. The discussion references bugs in distributed-protocol implementations and argues that formal mathematical and symbolic tools remain powerful for precise reasoning. ### Source excerpt On the success of 'natural language programming' Specifications, in plain speech. I believe that specification is the future of programming. Over the last four decades, we've seen the practice of building programs, and software systems grow closer and closer to the practice of specification. Details of the implementation, from layout in memory and disk, to layout in entire data centers, to algorithm and data structure choice, have become more and more abstract. Most application builders aren't writing frameworks, framework builders aren't building databases, database builders aren't designing protocols, protocol designers aren't writing kernels, and so on. Our modern software world is built on abstractions. Significant advancements are made, from time to time, by cutting through these abstractions. But still, the abstractions dominate, and will continue to. The practice of programming has become closer and closer to the practice of specification. Of crisply writing down what we want programs to do, and what makes them right. The how is less important. I believe that natural language will form the core of the programming languages of the future. The Ambiguity Problem The most common objection to this view is that natural language is ambiguous. It's exact meaning is potentially unclear, and highly dependent on context. This is a real problem. For example, in The Bug in Paxos Made Simple, I look at a common bug in implementations of Paxos caused directly by the ambiguity of natural language. Pointing out this ambiguity isn't criticizing [Lamport's] writing, but rather reminding you about how hard it is to write crisp descriptions of even relatively simple distributed protocols in text. As Lamport says: Prose is not the way to precisely describe algorithms. Perhaps the most famous statement of this problem is Dijkstra's from On the foolishness of "natural language programming": When all is said and told, the "naturalness" with which we use our native tongues boils down ## What Does a Database for SSDs Look Like? DevFeed: [What Does a Database for SSDs Look Like?]() Original publisher: [Read original article]() Author: Marc Brooker Published: 2025-12-15T00:00:00Z Content type: article Language: en Sources: [Marc Brooker's Blog](), [Marc Brooker's Blog]() Topics: [Databases](), [systems](), [cloud-infrastructure](), [datacenter](), [Latency]() Tags: [2025](), [cloud](), [database](), [databases](), [datacenter](), [datacenters](), [infrastructure](), [latency](), [mysql](), [network](), [networks](), [postgres](), [servers](), [sqlite](), [storage](), [systems]() ### AI overview The article examines how a relational database designed specifically for local NVMe SSDs might differ from databases created for spinning disks. It questions whether design choices such as write-ahead logging, large pages, bulk writes, and caching should change in light of faster storage, modern datacenter networks, abundant compute and memory, cloud infrastructure, and always-on global applications. It begins a quantitative analysis using the Five Minute Rule and 2025 hardware economics. ### Source excerpt What Does a Database for SSDs Look Like? Maybe not what you think. Over on X, Ben Dicken asked: What does a relational database designed specifically for local SSDs look like? Postgres, MySQL, SQLite and many others were invented in the 90s and 00s, the era of spinning disks. A local NVMe SSD has ~1000x improvement in both throughput and latency. Design decisions like write-ahead logs, large page sizes, and buffering table writes in bulk were built around disks where I/O was SLOW, and where sequential I/O was order(s)-of-magnitude faster than random. If we had to throw these databases away and begin from scratch in 2025, what would change and what would remain? How might we tackle this question quantitatively for the modern transaction-orientated database? But first, the bigger picture. It's not only SSDs that have come along since databases like Postgres were first designed. We also have the cloud, with deployments to excellent datacenter infrastructure, including multiple independent datacenters with great network connectivity between them, available to all. Datacenter networks offer 1000x (or more) increased throughput, along with latency in the microseconds. Servers with hundreds of cores and thousands of gigabytes of RAM are mainstream. Applications have changed too. Companies are global, businesses are 24/7. Down time is expensive, and that expense can be measured. The security and compliance environment is much more demanding. Builders want to deploy in seconds, not days. Approach One: The Five Minute Rule Perhaps my single favorite systems paper, The 5 Minute Rule... by Jim Gray and Franco Putzolu gives us a very simple way to answer one of the most important questions in systems: how big should caches be? The five minute rule is that, back in 1986, if you expected to read a page again within five minutes you should keep in in RAM. If not, you should keep it on disk. The basic logic is that you look at the page that's least likely to be re-used. If it's cheape ## What Now? Handling Errors in Large Systems DevFeed: [What Now? Handling Errors in Large Systems]() Original publisher: [Read original article]() Author: Marc Brooker Published: 2025-11-20T00:00:00Z Content type: article Language: en Sources: [Marc Brooker's Blog](), [Marc Brooker's Blog]() Topics: [systems](), [Resilience](), [Rust](), [Processes](), [autoscaling](), [Amazon Web Services](), [Serverless](), [Erlang](), [Containers](), [servers]() Tags: [architecture](), [autoscaling](), [aws](), [component](), [errors](), [http](), [outage](), [process](), [resilience](), [rust](), [server](), [serverless](), [servers](), [systems]() ### AI overview An analysis of error handling in large systems, using Rust's Result and unwrap as a starting point. It argues that whether a component should crash is a system-wide architectural decision shaped by failure correlation, error-handling layers, business logic, and the value of continuing with a last-known-good configuration or data version. ### Source excerpt What Now? Handling Errors in Large Systems More options means more choices. Cloudflare's deep postmortem for their November 18 outage triggered a ton of online chatter about error handling, caused by a single line in the postmortem: .unwrap() If you're not familiar with Rust, you need to know about Result, a kind of struct that can contain either a successful result, or an error. unwrap says basically "return the successful results if there is one, otherwise crash the program"1. You can think of it like an assert. There's a ton of debate about whether asserts are good in production2, but most are missing the point. Quite simply, this isn't a question about a single program. It's not a local property. Whether asserts are appropriate for a given component is a global property of the system, and the way it handles data. Let's play a little error handling game. Click the ✅ if you think crashing the process or server is appropriate, and the ❌ if you don't. Then you'll see my vote and justification. One of ten web servers behind a load balancer encounters uncorrectable memory errors, and takes itself out of service. ✅ ❌ Your vote: My vote: ✅ Uncorrectable memory errors are independent, and do not depend on user-provided content. In the presence of bad memory, it's impossible for a program to proceed safely. Taking the machine out of service is the safest course of action. One of ten multi-threaded application servers behind a load balancer encounters a null pointer in business logic while processing a customer request. ✅ ❌ Your vote: My vote: ❌ Customer requests triggering bugs in business logic isn't a good reason to bring the whole server down. Instead, fail that particular request (returning an HTTP 5xx error), and continue with other user requests. In approaches like Erlang, or even Lambda, it may be the right approach to crash the whole application in response to a bad request, because this crash is handled at a higher layer in the architecture. This is also why I pr ## Why Strong Consistency? DevFeed: [Why Strong Consistency?]() Original publisher: [Read original article]() Author: Marc Brooker Published: 2025-11-18T00:00:00Z Content type: opinion Language: en Sources: [Marc Brooker's Blog](), [Marc Brooker's Blog]() Topics: [DSQL](), [Databases](), [Replication](), [MySQL](), [Latency](), [API](), [Amazon EC2](), [Microservice](), [Amazon Web Services]() Tags: [api](), [aws](), [database](), [dsql](), [latency](), [microservices](), [mysql](), [replication]() ### AI overview The article explains how eventual consistency in read-replica architectures can cause applications to observe missing or stale data after a write. It argues that strong consistency simplifies client and service logic by avoiding retries, infinite loops, extra work, and latency, and presents Aurora DSQL as a database designed to make all reads strongly consistent. ### Source excerpt Why Strong Consistency? Eventual consistency makes your life harder. When I started at AWS in 2008, we ran the EC2 control plane on a tree of MySQL databases: a primary to handle writes, a secondary to take over from the primary, a handful of read replicas to scale reads, and some extra replicas for doing latency-insensitive reporting stuff. All of thing was linked together with MySQL's statement-based replication. It worked pretty well day to day, but two major areas of pain have stuck with me ever since: operations were costly, and eventual consistency made things weird. Since then, managed databases like Aurora MySQL have made relational database operations orders of magnitude easier. Which is great. But eventual consistency is still a feature of most database architectures that try scale reads. Today, I want to talk about why eventual consistency is a pain, and why we invested heavily in making all reads strongly consistent in Aurora DSQL. Eventual Consistency is a Pain for Customers Consider the following piece of code, running against an API exposed by a database-backed service: id = create_resource(...) get_resource_state(id, ...) In the world of read replicas, the latter statement can do something a little baffling: reply 'id does not exist'. The reason for this is simple: get_resource_state is a read-only call, likely routed to a read replica, and is racing the write from create_resource. If replication wins, this code works as expected. If the client wins, it has to handle to weird sensation of time moving backwards. Application programmers don't really have a principled way to work around this, so they end up writing code like this: id = create_resource(...) while True: try: get_resource_state(id, ...) return except ResourceDoesNotExist: sleep(100) Which fixes the problem. Kinda. Other times, especially if ResourceDoesNotExist can be thrown if id is deleted, it causes an infinite loop. It also creates more work for client and server, adds latency, and req ## DSQL: Simplifying Architectures DevFeed: [DSQL: Simplifying Architectures]() Original publisher: [Read original article]() Author: Marc Brooker Published: 2025-11-02T00:00:00Z Content type: opinion Language: en Sources: [Marc Brooker's Blog](), [Marc Brooker's Blog]() Topics: [DSQL](), [Serverless](), [Database](), [PostgreSQL](), [Concurrency]() Tags: [building](), [complexity](), [concurrency](), [cost](), [database](), [dsql](), [efficiency](), [infrastructure](), [memory](), [postgresql](), [scale](), [serverless]() ### AI overview The article presents Aurora DSQL as a way to simplify database-backed application architectures. It highlights serverless operation, automatic scaling of memory, CPU, and storage, managed software updates, usage-based pricing, strong consistency, and snapshot isolation. These features are described as reducing operational debt, infrastructure management, and application complexity. ### Source excerpt DSQL: Simplifying Architectures Complexity is a choice. While we were designing and building Aurora DSQL, we spent a lot of time thinking about our experience building and running database-backed systems. We saw that building great, fast, cost-effective, highly-available, systems was harder than it needed to be. We wanted to make it easier. Today, I want to discuss some of Aurora DSQL's features, and how I think they come together to make your life, as an application, service, or website developer, easier. We wanted for our customers what we wanted for ourselves: a relational database that allows us to build great systems with simple architectures. Serverless: The most obvious one is serverlessness. With DSQL you don't need to pick hardware, manage clusters, design for failover, think about patching engines or systems, monitor CPU and memory, or any of the many other tasks that come with good cluster management. When you get started with any database, you can basically ignore these things: buying excess CPU and memory at small scale is cheap, and monitoring is easy when you aren't also trying to drive efficiency. But this is a kind of operational debt, which will eventually come due. With DSQL, you can avoid this debt entirely: the system scales memory, CPU, and storage to meet your application's needs, and so you don't need to monitor these things. It keeps the database software and system software up-to-date too. Scale-down: I see a lot of database architectures take on a lot of complexity to reduce cost during nights and weekends: scaling down primaries, removing read replicas, and even removing secondaries. These are smart cost management things with many database products, but you simply don't need them with DSQL. DSQL scales up and down with your application, priced on the amount of work your database is doing, not the scale of your infrastructure. Doing no work? You'll spend no money. Strong consistency and just-right isolation: DSQL offers a single isolation ## Fixing UUIDv7 (for database use-cases) DevFeed: [Fixing UUIDv7 (for database use-cases)]() Original publisher: [Read original article]() Author: Marc Brooker Published: 2025-10-22T00:00:00Z Content type: article Language: en Sources: [Marc Brooker's Blog](), [Marc Brooker's Blog]() Topics: [Databases](), [Caching](), [Security](), [Architecture & Design]() Tags: [cache](), [cryptographic](), [database](), [datacenters](), [performance](), [security]() ### AI overview The article examines UUIDv7 for database use cases, explaining how time ordering can improve index locality, insertion performance, and cache effectiveness while introducing concerns about timestamp leakage, reduced entropy, correlated behavior, and hot spotting. It proposes modifying UUIDv7 with a keyed hash, a tunable entropy parameter, and an infrastructure-specific identifier to balance locality with ID distribution. ### Source excerpt Fixing UUIDv7 (for database use-cases) How do I even balance a V7? RFC9562 defines UUID Version 7. This has made a lot of people very angry and been widely regarded as a bad move1. More seriously, UUIDv7 has received a lot of criticism, despite seemingly achieving what it set out to do. The legitimate criticism seems to be on a few points. V7 UUIDs: Leak information (namely the server timestamp). Are a bad choice for cases where security or operational requirements require UUIDs that are hard to guess, because they have less entropy. Introduce correlated behavior between datacenters, regions, and installations of applications, increasing the probability of triggering bugs across failure boundaries. Are hard to present in UIs, because the E6EE7F40... format doesn't work, because of deterministic first digits. Before thinking about how we might fix these issues, let's understand why folks are drawn to UUIDv7. Most of the use-cases I see are related to increasing database insert performance. To quote the RFC: Time-ordered monotonic UUIDs benefit from greater database-index locality because the new values are near each other in the index. As a result, objects are more easily clustered together for better performance. The real-world differences in this approach of index locality versus random data inserts can be one order of magnitude or more. This effect is very real. Random DB keys like UUIDv4 destroy spatial locality (as I've written about before), making database caches less effective, almost always reducing insert performance, and reducing query performance where queries have substantial temporal locality. The slight upside to this is that they also avoid hot spotting in distributed or sharded architectures. Can we both have good insert performance and avoid the downsides of UUIDv7? Yes, I believe we can. Let's keep the overall format from the RFC: 0 1 2 3 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ ## Is Systems Research Really Just About Making Numbers Bigger? DevFeed: [Is Systems Research Really Just About Making Numbers Bigger?]() Original publisher: [Read original article]() Author: Marc Brooker Published: 2025-10-12T00:00:00Z Content type: opinion Language: en Sources: [Marc Brooker's Blog](), [Marc Brooker's Blog]() Topics: [systems](), [Artificial Intelligence](), [machine learning overfitting](), [Programming](), [Algorithms, Complexity]() Tags: [ai](), [algorithms](), [benchmarks](), [code](), [research](), [software-engineer](), [systems](), [testing](), [verification]() ### AI overview This opinion argues that systems research should broaden beyond making performance numbers bigger. It presents AI-driven solution discovery as especially effective in areas with reliable verifiers, such as software artifacts tested against predefined workloads, while emphasizing that robust testing is essential to prevent overfitting and reward hacking. ### Source excerpt Is Systems Research Really Just About Making Numbers Bigger? The Barbarian F.C. of systems research would be pretty cool. Lots of folks online have been talking about Barbarians at the Gate: How AI is Upending Systems Research by Cheng, Liu, Pan, et al this week. Maybe unsurprisingly, given the fact that I work in AI for my day job, and both consume and produce systems research, I found it super interesting. Perhaps the most interesting discussion, however, isn't about AI at all. It's about the scope of systems research. What systems research is, or aught to be. The paper's core argument is well captured in the abstract: We argue that systems research, long focused on designing and evaluating new performance-oriented algorithms, is particularly well-suited for AI-driven solution discovery. This is because system performance problems naturally admit reliable verifiers: solutions are typically implemented in real systems or simulators, and verification reduces to running these software artifacts against predefined workloads and measuring performance. I, 100%, enthusiastically, agree with this point. I think it's a point that generalizes way beyond systems research to the entire software industry: AI is going to be most effective in problem spaces where there are what the authors call reliable verifiers. Where, in effect, we can do automated hill climbing towards a low-ambiguity solution or Pareto frontier of solutions. Much of the next decade is going to be defined by finding better techniques to build these reliable verifiers where none existed before. If you're a software engineer, what I'm saying here is testing is going to be the most important thing. A bit later, the paper highlights two of the challenges with building these verifiers for systems builders (and software developers generally): Prevent overfitting. Evaluating against narrow workloads lead to algorithm failures like overfitting, where the solutions either hard-code behaviors or overfit to specific tr ## Locality, and Temporal-Spatial Hypothesis DevFeed: [Locality, and Temporal-Spatial Hypothesis]() Original publisher: [Read original article]() Author: Marc Brooker Published: 2025-10-05T00:00:00Z Content type: article Language: en Sources: [Marc Brooker's Blog](), [Marc Brooker's Blog]() Topics: [Databases](), [Caching](), [systems](), [IO](), [Streaming](), [Time Series](), [observability]() Tags: [cache](), [database](), [database-performance](), [io](), [linux](), [metrics](), [observability](), [performance](), [postgres](), [streaming](), [systems](), [time-series]() ### AI overview The article examines locality in database and computer-system performance, focusing on how temporal and spatial locality influence caching, read-ahead, and data layout. It develops the temporal-spatial locality hypothesis and compares its relevance for streaming, time-series, metrics, observability, and hash-based database systems. ### Source excerpt Locality, and Temporal-Spatial Hypothesis Good fences make good neighbors? Last week at PGConf NYC, I had the pleasure of hearing Andres Freund talking about the great work he's been doing to bring async IO to Postgres 18. One particular result caught my eye: a large difference in performance between forward and reverse scans, seemingly driven by read ahead1. The short version is that IO layers (like Linux's) optimize performance by proactively pre-fetching data ahead of the current read point in a file, so it's already cached when needed. Notably, most of these systems don't do this backwards. This leads to a big difference in performance between forward scans (where the pages are already in the cache when they're needed) and backward scans (where the database needs to block on IO to fetch the next page). This lead me to thinking more about a particular hypothesis behind many database designs: a temporal-spatial locality hypothesis2. Before we get there, let's talk about locality more generally, because it might be the single most important idea in database performance (and computer systems performance generally). Temporal locality is the idea that data accessed recently is likely to be accessed again soon. This idea is what's behind CPU caches, database buffer pools, and most caches you'll come across in computer systems. Spatial locality is the idea that when we access data, we're likely to access nearby data soon. Almost all database systems take advantage of these forms of locality, and would lost significant performance without taking advantage of them. Stacks of books could be written about these ideas. Stacks of books have been written about these ideas. We could talk about cache-oblivious algorithms, or non-polluting read and write instructions, or have an argument about linked lists. Instead, I want to zoom in to a particular idea in databases: temporal-spatial hypothesis. The hypothesis I mean has a definition something like this: Temporal-spatial localit ## Seven Years of Firecracker DevFeed: [Seven Years of Firecracker]() Original publisher: [Read original article]() Author: Marc Brooker Published: 2025-09-18T00:00:00Z Content type: article Language: en Sources: [Marc Brooker's Blog](), [Marc Brooker's Blog]() Topics: [Firecracker](), [virtualization](), [Amazon Web Services](), [Security](), [Open Source](), [Amazon Bedrock AgentCore](), [Serverless](), [Artificial Intelligence](), [Multitenancy](), [Large Language Model](), [Python]() Tags: [ai](), [amazon-bedrock-agentcore](), [aws](), [aws-lambda](), [firecracker](), [llm](), [multitenancy](), [open-source](), [python](), [security](), [virtualization]() ### AI overview This article reflects on seven years of Firecracker, an open source technology for creating and managing small virtual machines. It describes Firecracker's role in AWS Lambda, its security and compatibility benefits compared with container and language-level isolation, its use across AWS, and its role in Amazon Bedrock AgentCore for running AI agents. ### Source excerpt Seven Years of Firecracker Time flies like an arrow. Fruit flies like a banana. Back at re:Invent 2018, we shared Firecracker with the world. Firecracker is open source software that makes it easy to create and manage small virtual machines. At the time, we talked about Firecracker as one of the key technologies behind AWS Lambda, including how it'd allowed us to make Lambda faster, more efficient, and more secure. A couple years later, we published Firecracker: Lightweight Virtualization for Serverless Applications (at NSDI'20). Here's me talking through the paper back then: The paper went into more detail into how we're using Firecracker in Lambda, how we think about the economics of multitenancy (more about that here), and how we chose virtualization over kernel-level isolation (containers) or language-level isolation for Lambda. Despite these challenges, virtualization provides many compelling benefits. From an isolation perspective, the most compelling benefit is that it moves the security-critical interface from the OS boundary to a boundary supported in hardware and comparatively simpler software. It removes the need to trade off between kernel features and security: the guest kernel can supply its full feature set with no change to the threat model. VMMs are much smaller than general-purpose OS kernels, exposing a small number of well-understood abstractions without compromising on software compatibility or requiring software to be modified. Firecracker has really taken off, in all three ways we hoped it would. First, we use it in many more places inside AWS, backing the infrastructure we offer to customers across multiple services. Second, folks use the open source version directly, building their own cool products and businesses on it. Third, it was the motivation for a wave of innovation in the VM space. In this post, I wanted to write a bit about two of the ways we're using Firecracker at AWS that weren't covered in the paper. Bedrock AgentCore Back in J [Next page]()