# Nelson Elhage

Recent content in Posts on Made of Bugs

This is one page of public article previews, not the complete archive. Follow Next page to continue. Summaries are not the original full articles.

## From error-handling to structured concurrency

DevFeed: [From error-handling to structured concurrency](<https://devfeed.tech/articles/from-error-handling-to-structured-concurrency-21946.md>)

Original publisher: [Read original article](<https://blog.nelhage.com/post/concurrent-error-handling/>)

Author: Nelson Elhage

Published: 2026-03-23T15:30:00Z

Content type: opinion

Language: en

Sources: [Nelson Elhage](<https://devfeed.tech/sources/nelson-elhage.md>)

Topics: [Concurrency](<https://devfeed.tech/topics/concurrency.md>), [Concurrent Programming](<https://devfeed.tech/topics/concurrent-programming.md>), [Error Handling](<https://devfeed.tech/topics/error-handling.md>), [Exception](<https://devfeed.tech/topics/exception.md>)

Tags: [concurrency](<https://devfeed.tech/tags/concurrency.md>), [error-handling](<https://devfeed.tech/tags/error-handling.md>), [examples](<https://devfeed.tech/tags/examples.md>), [exception](<https://devfeed.tech/tags/exception.md>), [exception-handling](<https://devfeed.tech/tags/exception-handling.md>), [structured-concurrency](<https://devfeed.tech/tags/structured-concurrency.md>)

### AI overview

This article examines how error-handling patterns based on stack unwinding and cleanup should be adapted for concurrent programs with multiple tasks. It introduces the problem of unhandled errors in concurrent execution and compares possible behaviors when one task fails.

### Source excerpt

How should we think about error-handling in concurrent programs? In single-threaded programs, we've mostly converged on a standard pattern, with a diverse zoo of implementations and concrete patterns. When an error occurs, it is propagated up the stack until we find a stack frame which is prepared to handle it. As we do so, we unwind the stack frames in-order, giving each frame the opportunity to clean up or destroy resources as appropriate.

## Solving regex crosswords with Z3

DevFeed: [Solving regex crosswords with Z3](<https://devfeed.tech/articles/solving-regex-crosswords-with-z3-21962.md>)

Original publisher: [Read original article](<https://blog.nelhage.com/post/regex-crosswords-z3/>)

Author: Nelson Elhage

Published: 2025-10-21T14:00:00Z

Content type: tutorial

Language: en

Sources: [Nelson Elhage](<https://devfeed.tech/sources/nelson-elhage.md>)

Topics: [Regular expression](<https://devfeed.tech/topics/regular-expression.md>), [Automaton](<https://devfeed.tech/topics/automaton.md>), [Finite-state machine](<https://devfeed.tech/topics/finite-state-machine.md>), [Python](<https://devfeed.tech/topics/python.md>)

Tags: [code](<https://devfeed.tech/tags/code.md>), [dfa](<https://devfeed.tech/tags/dfa.md>), [experimentation](<https://devfeed.tech/tags/experimentation.md>), [performance](<https://devfeed.tech/tags/performance.md>), [python](<https://devfeed.tech/tags/python.md>), [regex](<https://devfeed.tech/tags/regex.md>)

### AI overview

This technical post explains how to solve regular-expression crossword puzzles with a Z3-backed solver. It describes encoding regular expressions as deterministic finite automata, using a Python library to convert regex syntax to finite-state machines, and exploring solver performance and implementation improvements.

### Source excerpt

For a while now, I've been fascinated by Z3 and by SMT solving more broadly. While on pat leave recently, I was reminded of the existence of regular-expression crossword puzzles, and allowed myself to get nerdsniped by writing a Z3-backed solver. I expected to spend perhaps an afternoon cranking out a quick solver; I ended up getting sucked into understanding and debugging Z3 performance, and learning far more about Z3 and about SMT than I expected.

## The ITTAGE indirect branch predictor

DevFeed: [The ITTAGE indirect branch predictor](<https://devfeed.tech/articles/the-ittage-indirect-branch-predictor-21953.md>)

Original publisher: [Read original article](<https://blog.nelhage.com/post/ittage-branch-predictor/>)

Author: Nelson Elhage

Published: 2025-07-04T21:30:00Z

Content type: article

Language: en

Sources: [Nelson Elhage](<https://devfeed.tech/sources/nelson-elhage.md>)

Topics: [cpu](<https://devfeed.tech/topics/cpu.md>), [Python](<https://devfeed.tech/topics/python.md>), [Algorithm](<https://devfeed.tech/topics/algorithm.md>), [Python 3.14](<https://devfeed.tech/topics/python-3-14.md>)

Tags: [algorithm](<https://devfeed.tech/tags/algorithm.md>), [bytecode](<https://devfeed.tech/tags/bytecode.md>), [cpu](<https://devfeed.tech/tags/cpu.md>), [performance](<https://devfeed.tech/tags/performance.md>), [python](<https://devfeed.tech/tags/python.md>), [python-3-14](<https://devfeed.tech/tags/python-3-14.md>)

### AI overview

An explanatory article about ITTAGE indirect branch prediction and how modern CPUs predict bytecode-dispatch jumps. It connects the topic to Python 3.14's tail-calling interpreter and describes ITTAGE as a TAGE variant that predicts indirect jump destinations using program-counter history and multiple history-length tables.

### Source excerpt

While investigating the performance of the new Python 3.14 tail-calling interpreter, I learned (via this very informative comment from Sam Gross) new (to me) piece of performance trivia: Modern CPUs mostly no longer struggle to predict the bytecode-dispatch indirect jump inside a "conventional" bytecode interpreter loop. In steady-state, assuming the bytecode itself is reasonable stable, modern CPUs achieve very high accuracy predicting the dispatch, even for "vanilla" while / switch-style interpreter loops1!

## Performance of the Python 3.14 tail-call interpreter

DevFeed: [Performance of the Python 3.14 tail-call interpreter](<https://devfeed.tech/articles/performance-of-the-python-3-14-tail-call-interpreter-21947.md>)

Original publisher: [Read original article](<https://blog.nelhage.com/post/cpython-tail-call/>)

Author: Nelson Elhage

Published: 2025-03-09T22:00:00Z

Content type: article

Language: en

Sources: [Nelson Elhage](<https://devfeed.tech/sources/nelson-elhage.md>)

Topics: [Python 3.14](<https://devfeed.tech/topics/python-3-14.md>), [benchmarking](<https://devfeed.tech/topics/benchmarking.md>), [gcc](<https://devfeed.tech/topics/gcc.md>), [LLVM](<https://devfeed.tech/topics/llvm.md>), [Software Engineering](<https://devfeed.tech/topics/software-engineering.md>)

Tags: [benchmark](<https://devfeed.tech/tags/benchmark.md>), [benchmarking](<https://devfeed.tech/tags/benchmarking.md>), [benchmarks](<https://devfeed.tech/tags/benchmarks.md>), [bytecode](<https://devfeed.tech/tags/bytecode.md>), [clang](<https://devfeed.tech/tags/clang.md>), [compilers](<https://devfeed.tech/tags/compilers.md>), [cpu](<https://devfeed.tech/tags/cpu.md>), [gcc](<https://devfeed.tech/tags/gcc.md>), [intel](<https://devfeed.tech/tags/intel.md>), [llvm](<https://devfeed.tech/tags/llvm.md>), [performance](<https://devfeed.tech/tags/performance.md>), [performance-engineering](<https://devfeed.tech/tags/performance-engineering.md>), [python](<https://devfeed.tech/tags/python.md>), [python-3-14](<https://devfeed.tech/tags/python-3-14.md>), [software-engineering](<https://devfeed.tech/tags/software-engineering.md>)

### AI overview

This article examines the performance gains attributed to CPython's Python 3.14 tail-call bytecode interpreter. It finds that the initial 10-15% improvement was primarily caused by inadvertently working around an LLVM 19 regression; with better baselines, the gain is closer to 1-5%, depending on the setup.

### Source excerpt

About a month ago, the CPython project merged a new implementation strategy for their bytecode interpreter. The initial headline results were very impressive, showing a 10-15% performance improvement on average across a wide range of benchmarks across a variety of platforms. Unfortunately, as I will document in this post, these impressive performance gains turned out to be primarily due to inadvertently working around a regression in LLVM 19. When benchmarked against a better baseline (such GCC, clang-18, or LLVM 19 with certain tuning flags), the performance gain drops to 1-5% or so depending on the exact setup.

## Building personal software with Claude

DevFeed: [Building personal software with Claude](<https://devfeed.tech/articles/building-personal-software-with-claude-21955.md>)

Original publisher: [Read original article](<https://blog.nelhage.com/post/personal-software-with-claude/>)

Author: Nelson Elhage

Published: 2025-01-27T20:00:00Z

Content type: article

Language: en

Sources: [Nelson Elhage](<https://devfeed.tech/sources/nelson-elhage.md>)

Topics: [Claude](<https://devfeed.tech/topics/claude.md>), [Rust](<https://devfeed.tech/topics/rust.md>), [Emacs](<https://devfeed.tech/topics/emacs.md>), [Obsidian](<https://devfeed.tech/topics/obsidian-md.md>), [Software Engineering](<https://devfeed.tech/topics/software-engineering.md>), [JSON](<https://devfeed.tech/topics/json.md>), [Markdown](<https://devfeed.tech/topics/markdown.md>), [Org mode](<https://devfeed.tech/topics/orgmode.md>)

Tags: [app](<https://devfeed.tech/tags/app.md>), [claude](<https://devfeed.tech/tags/claude.md>), [code](<https://devfeed.tech/tags/code.md>), [emacs](<https://devfeed.tech/tags/emacs.md>), [json](<https://devfeed.tech/tags/json.md>), [markdown](<https://devfeed.tech/tags/markdown.md>), [profiling](<https://devfeed.tech/tags/profiling.md>), [rust](<https://devfeed.tech/tags/rust.md>), [software](<https://devfeed.tech/tags/software.md>)

### AI overview

The author describes using Claude, under supervision, to port parts of an Emacs package from Elisp to Rust. The resulting approach reduced one measured execution time from about 90 seconds to 15 milliseconds and changed the author's view of LLMs' role in software engineering.

### Source excerpt

Earlier this month, I used Claude to port (parts of) an Emacs package into Rust, shrinking the execution time by a factor of 1000 or more (in one concrete case: from 90s to about 15ms). This is a variety of yak-shave that I do somewhat routinely, both professionally and in service of my personal computing environment. However, this time, Claude was able to execute substantially the entire project under my supervision without me writing almost-any lines of code, speeding up the project substantially compared to doing it by hand.

## Finding near-duplicates with Jaccard similarity and MinHash

DevFeed: [Finding near-duplicates with Jaccard similarity and MinHash](<https://devfeed.tech/articles/finding-near-duplicates-with-jaccard-similarity-and-minhash-21952.md>)

Original publisher: [Read original article](<https://blog.nelhage.com/post/fuzzy-dedup/>)

Author: Nelson Elhage

Published: 2024-07-03T23:00:00Z

Content type: tutorial

Language: en

Sources: [Nelson Elhage](<https://devfeed.tech/sources/nelson-elhage.md>)

Topics: [dataset](<https://devfeed.tech/topics/dataset.md>), [Website](<https://devfeed.tech/topics/website.md>)

Tags: [dataset](<https://devfeed.tech/tags/dataset.md>), [pairs](<https://devfeed.tech/tags/pairs.md>), [pipeline](<https://devfeed.tech/tags/pipeline.md>), [range](<https://devfeed.tech/tags/range.md>), [scale](<https://devfeed.tech/tags/scale.md>)

### AI overview

This article explains approximate document deduplication using Jaccard similarity and the MinHash approximation technique. It discusses defining similarity between document pairs, setting a threshold for approximate duplicates, and why approximate similarity is not transitive.

### Source excerpt

Suppose we have a large collection of documents, and we wish you identify which documents are approximately the same as each other. For instance, we may have crawled the web over some period of time, and expect to have fetched the "same page" several times, but to see slight differences in metadata, or that we have several revisions of a page following small edits. In this post I want to explore the method of approximate deduplication via Jaccard similarity and the MinHash approximation trick.

## Stripe's monorepo developer environment

DevFeed: [Stripe's monorepo developer environment](<https://devfeed.tech/articles/stripe-s-monorepo-developer-environment-21966.md>)

Original publisher: [Read original article](<https://blog.nelhage.com/post/stripe-dev-environment/>)

Author: Nelson Elhage

Published: 2024-05-21T17:00:00Z

Content type: article

Language: en

Sources: [Nelson Elhage](<https://devfeed.tech/sources/nelson-elhage.md>)

Topics: [monorepo](<https://devfeed.tech/topics/monorepo.md>), [Developer experience](<https://devfeed.tech/topics/developer-experience.md>), [code productivity](<https://devfeed.tech/topics/code-productivity.md>), [Tooling](<https://devfeed.tech/topics/tooling.md>), [Ruby](<https://devfeed.tech/topics/ruby.md>), [Development](<https://devfeed.tech/topics/development.md>)

Tags: [developer](<https://devfeed.tech/tags/developer.md>), [developer-experience](<https://devfeed.tech/tags/developer-experience.md>), [developer-productivity](<https://devfeed.tech/tags/developer-productivity.md>), [development](<https://devfeed.tech/tags/development.md>), [monorepo](<https://devfeed.tech/tags/monorepo.md>), [ruby](<https://devfeed.tech/tags/ruby.md>), [stripe](<https://devfeed.tech/tags/stripe.md>), [tooling](<https://devfeed.tech/tags/tooling.md>)

### AI overview

A former Stripe engineer recalls the company's developer environment from 2012 to 2019, focusing on tooling used to develop and test code in a large Ruby monorepo. The account explains that these choices were shaped by Stripe's business and technical context and may not reflect the company's current environment.

### Source excerpt

I worked at Stripe for about seven years, from 2012 to 2019. Over that time, I used and contributed to many generations of Stripe's developer environment - the tools that engineers used daily to write and test code. I think Stripe did a pretty good job designing and building that developer experience, and since leaving, I've found myself repeatedly describing features of that environment to friends and colleagues. This post is an attempt to record the salient features of that environment as I remember it.

## Performance engineering, profilers, and seeing the invisible

DevFeed: [Performance engineering, profilers, and seeing the invisible](<https://devfeed.tech/articles/performance-engineering-profilers-and-seeing-the-invisible-21957.md>)

Original publisher: [Read original article](<https://blog.nelhage.com/post/profilers-seeing-the-invisible/>)

Author: Nelson Elhage

Published: 2023-12-18T16:00:00Z

Content type: opinion

Language: en

Sources: [Nelson Elhage](<https://devfeed.tech/sources/nelson-elhage.md>)

Topics: [Software](<https://devfeed.tech/topics/software.md>), [Tool](<https://devfeed.tech/topics/tool.md>), [coding](<https://devfeed.tech/topics/coding.md>)

Tags: [engineering](<https://devfeed.tech/tags/engineering.md>), [paper](<https://devfeed.tech/tags/paper.md>), [performance](<https://devfeed.tech/tags/performance.md>), [performance-engineering](<https://devfeed.tech/tags/performance-engineering.md>), [software](<https://devfeed.tech/tags/software.md>), [tools](<https://devfeed.tech/tags/tools.md>)

### AI overview

The article examines performance engineering through the idea that experts can perceive information absent from direct observation. It argues that profilers show where a particular program execution spends time, but effective optimization also requires understanding context and information not present in the profile.

### Source excerpt

I was recently introduced to the paper "Seeing the Invisible: Perceptual-Cognitive Aspects of Expertise" by Gary Klein and Robert Hoffman. It's excellent and I recommend you read it when you have a chance. Klein and Hoffman discuss the ability of experts to "see what is not there": in addition to observing data and cues that are present in the environment, experts perceive implications of these cues, such as the absence of expected or "typical" information, the typicality or atypicality of observed data, and likely/possible past and future time trajectories of a system based on a point-in-time snapshot or limited duration of observation.

## Advent of Code in C++ Template Metaprogramming

DevFeed: [Advent of Code in C++ Template Metaprogramming](<https://devfeed.tech/articles/advent-of-code-in-c-template-metaprogramming-21942.md>)

Original publisher: [Read original article](<https://blog.nelhage.com/post/advent-of-templates/>)

Author: Nelson Elhage

Published: 2023-12-08T15:30:00Z

Content type: tutorial

Language: en

Sources: [Nelson Elhage](<https://devfeed.tech/sources/nelson-elhage.md>)

Topics: [Advent of Code](<https://devfeed.tech/topics/advent-of-code.md>), [C++](<https://devfeed.tech/topics/c-plus-plus.md>), [Code](<https://devfeed.tech/topics/code.md>), [Compiler](<https://devfeed.tech/topics/compiler.md>), [polyfill](<https://devfeed.tech/topics/polyfill.md>)

Tags: [advent-of-code](<https://devfeed.tech/tags/advent-of-code.md>), [c-plus-plus](<https://devfeed.tech/tags/c-plus-plus.md>), [code](<https://devfeed.tech/tags/code.md>), [compiler](<https://devfeed.tech/tags/compiler.md>), [polyfill](<https://devfeed.tech/tags/polyfill.md>), [writeup](<https://devfeed.tech/tags/writeup.md>)

### AI overview

An annotated walkthrough of solving the first Advent of Code problem using purely compile-time C++ template metaprogramming. It covers compile-time input handling, folds, helper types, and state tracking to calculate calibration values.

### Source excerpt

This December, the imp of the perverse struck me, and I decided to see how many days of Advent of Code I could do purely in compile-time C++ metaprogramming. As of this writing, I've done two days, and I'm not sure I'll make it any further. However, that's one more day than I planned to do as of yesterday, which is in turn further than I thought I'd make it after my first attempt.

## Why Python pickle remains common in machine-learning software

DevFeed: [Why Python pickle remains common in machine-learning software](<https://devfeed.tech/articles/what-s-with-ml-software-and-pickles-21956.md>)

Original publisher: [Read original article](<https://blog.nelhage.com/post/pickles-and-ml/>)

Author: Nelson Elhage

Published: 2023-11-08T05:00:00Z

Content type: opinion

Language: en

Sources: [Nelson Elhage](<https://devfeed.tech/sources/nelson-elhage.md>)

Topics: [Machine Learning & Artificial Intelligence](<https://devfeed.tech/topics/machine-learning-artificial-intelligence.md>), [Python](<https://devfeed.tech/topics/python.md>), [Software](<https://devfeed.tech/topics/software.md>), [Security](<https://devfeed.tech/topics/security.md>), [upgrade](<https://devfeed.tech/topics/upgrade.md>)

Tags: [machine-learning](<https://devfeed.tech/tags/machine-learning.md>), [ml](<https://devfeed.tech/tags/ml.md>), [python](<https://devfeed.tech/tags/python.md>), [security](<https://devfeed.tech/tags/security.md>), [software](<https://devfeed.tech/tags/software.md>), [upgrade](<https://devfeed.tech/tags/upgrade.md>)

### AI overview

The author examines why Python pickle is widely used in machine-learning software despite security concerns and brittle serialization. The article argues that ML software is shaped by research workflows, where teams prioritize exploring ideas and producing knowledge over maintaining durable software artifacts.

### Source excerpt

I have spent many years as an software engineer who was a total outsider to machine-learning, but with some curiosity and occasional peripheral interactions with it. During this time, a recurring theme for me was horror (and, to be honest, disdain) every time I encountered the widespread usage of Python pickle in the Python ML ecosystem. In addition to their major security issues1, the use of pickle for serialization tends to be very brittle, leading to all kinds of nightmares as you evolve your code and upgrade libraries and Python versions.

## Service Performance at Capacity: Request Rate, Throughput, and Graceful Behavior

DevFeed: [Service Performance at Capacity: Request Rate, Throughput, and Graceful Behavior](<https://devfeed.tech/articles/graceful-behavior-at-capacity-21967.md>)

Original publisher: [Read original article](<https://blog.nelhage.com/post/systems-at-capacity/>)

Author: Nelson Elhage

Published: 2023-08-07T16:00:00Z

Content type: article

Language: en

Sources: [Nelson Elhage](<https://devfeed.tech/sources/nelson-elhage.md>)

Topics: [systems](<https://devfeed.tech/topics/systems.md>), [servers](<https://devfeed.tech/topics/servers.md>), [web applications](<https://devfeed.tech/topics/web-applications.md>), [Back end](<https://devfeed.tech/topics/backend.md>), [TCP/IP](<https://devfeed.tech/topics/tcp-ip.md>), [Firewall](<https://devfeed.tech/topics/firewall.md>), [Networks](<https://devfeed.tech/topics/networks.md>), [Python](<https://devfeed.tech/topics/python.md>)

Tags: [architecture](<https://devfeed.tech/tags/architecture.md>), [backend](<https://devfeed.tech/tags/backend.md>), [capacity](<https://devfeed.tech/tags/capacity.md>), [firewall](<https://devfeed.tech/tags/firewall.md>), [http](<https://devfeed.tech/tags/http.md>), [network](<https://devfeed.tech/tags/network.md>), [performance](<https://devfeed.tech/tags/performance.md>), [python](<https://devfeed.tech/tags/python.md>), [tcp-ip](<https://devfeed.tech/tags/tcp-ip.md>)

### AI overview

The article examines service performance by relating incoming request rate to successful throughput. It introduces these metrics through examples involving an HTTP web application and a TCP/IP router or firewall.

### Source excerpt

Suppose we've got a service. We'll gloss over the details for now, but let's stipulate that it accepts requests from the outside world, and takes some action in response. Maybe those requests are HTTP requests, or RPCs, or just incoming packets to be routed at the network layer. We can get more specific later. What can we say about its performance? All we know is that it receives requests, and that it acts on them.

## Efficiency trades off against resiliency

DevFeed: [Efficiency trades off against resiliency](<https://devfeed.tech/articles/efficiency-trades-off-against-resiliency-21950.md>)

Original publisher: [Read original article](<https://blog.nelhage.com/post/efficiency-vs-resiliency/>)

Author: Nelson Elhage

Published: 2023-04-15T23:00:00Z

Content type: opinion

Language: en

Sources: [Nelson Elhage](<https://devfeed.tech/sources/nelson-elhage.md>)

Topics: [Resilience](<https://devfeed.tech/topics/resilience.md>), [Monitoring](<https://devfeed.tech/topics/monitoring.md>), [Server](<https://devfeed.tech/topics/server.md>)

Tags: [cpu](<https://devfeed.tech/tags/cpu.md>), [efficiency](<https://devfeed.tech/tags/efficiency.md>), [monitoring](<https://devfeed.tech/tags/monitoring.md>), [performance](<https://devfeed.tech/tags/performance.md>), [resiliency](<https://devfeed.tech/tags/resiliency.md>), [work](<https://devfeed.tech/tags/work.md>)

### AI overview

The article examines how maximizing CPU utilization can leave servers without capacity to absorb unexpected traffic or added workload. It argues that improving efficiency often requires tradeoffs with resilience, while noting that some performance fixes can improve both.

### Source excerpt

What's the "right" level of CPU utilization for a server? If you look at a monitoring dashboard from a well-designed and well-run service, what CPU utilization should we hope to see, averaged over a day or two? It's a very general question, and it's not clear it should have a single answer. That said, for a long time, I generally believed that higher is always better: we should aim for as close to 100% utilization as we can.

## Transformers for software engineers

DevFeed: [Transformers for software engineers](<https://devfeed.tech/articles/transformers-for-software-engineers-21973.md>)

Original publisher: [Read original article](<https://blog.nelhage.com/post/transformers-for-software-engineers/>)

Author: Nelson Elhage

Published: 2022-04-01T20:00:00Z

Content type: tutorial

Language: en

Sources: [Nelson Elhage](<https://devfeed.tech/sources/nelson-elhage.md>)

Topics: [Transformer architecture](<https://devfeed.tech/topics/transformer-architecture.md>), [Programming](<https://devfeed.tech/topics/programming.md>), [Software Engineering](<https://devfeed.tech/topics/software-engineering.md>), [Reverse Engineering](<https://devfeed.tech/topics/reverse-engineering.md>), [Transformers](<https://devfeed.tech/topics/transformers.md>)

Tags: [anthropic](<https://devfeed.tech/tags/anthropic.md>), [architecture](<https://devfeed.tech/tags/architecture.md>), [codex](<https://devfeed.tech/tags/codex.md>), [deep-learning](<https://devfeed.tech/tags/deep-learning.md>), [gpt-3](<https://devfeed.tech/tags/gpt-3.md>), [model-architecture](<https://devfeed.tech/tags/model-architecture.md>), [programming](<https://devfeed.tech/tags/programming.md>), [reverse-engineering](<https://devfeed.tech/tags/reverse-engineering.md>), [software-engineering](<https://devfeed.tech/tags/software-engineering.md>), [storm](<https://devfeed.tech/tags/storm.md>), [transformer-architecture](<https://devfeed.tech/tags/transformer-architecture.md>), [transformers](<https://devfeed.tech/tags/transformers.md>)

### AI overview

This tutorial explains Transformer architecture for software engineers, using software engineering and programming perspectives to discuss how GPT-style Transformer models work. It also connects the architecture to interpretability and reverse-engineering efforts.

### Source excerpt

Ever since its introduction in the 2017 paper, Attention is All You Need, the Transformer model architecture has taken the deep-learning world by storm. Initially introduced for machine translation, it has become the tool of choice for a wide range of domains, including text, audio, video, and others. Transformers have also driven most of the massive increases in model scale and capability in the last few years. OpenAI's GPT-3 and Codex models are Transformers, as are DeepMind's Gopher models and many others.

## A Rare, Nondeterministic RDMA Bug in Large Distributed Training Jobs

DevFeed: [A Rare, Nondeterministic RDMA Bug in Large Distributed Training Jobs](<https://devfeed.tech/articles/a-cursed-bug-21940.md>)

Original publisher: [Read original article](<https://blog.nelhage.com/post/a-cursed-bug/>)

Author: Nelson Elhage

Published: 2022-02-23T03:03:48Z

Content type: opinion

Language: en

Sources: [Nelson Elhage](<https://devfeed.tech/sources/nelson-elhage.md>)

Topics: [bug](<https://devfeed.tech/topics/bug.md>), [distributed-systems](<https://devfeed.tech/topics/distributed-systems.md>), [distributed-training](<https://devfeed.tech/topics/distributed-training.md>), [Processes](<https://devfeed.tech/topics/processes.md>), [coding](<https://devfeed.tech/topics/coding.md>)

Tags: [bug](<https://devfeed.tech/tags/bug.md>), [code](<https://devfeed.tech/tags/code.md>), [debugging](<https://devfeed.tech/tags/debugging.md>), [distributed-systems](<https://devfeed.tech/tags/distributed-systems.md>), [distributed-training](<https://devfeed.tech/tags/distributed-training.md>), [process](<https://devfeed.tech/tags/process.md>), [processes](<https://devfeed.tech/tags/processes.md>), [python](<https://devfeed.tech/tags/python.md>), [rdma](<https://devfeed.tech/tags/rdma.md>)

### AI overview

An Anthropic engineer describes a rare, nondeterministic bug in very large distributed training jobs. The bug originated in the RDMA software stack and caused subprocess launches to fail on a small fraction of nodes, sometimes producing Python exceptions or segmentation faults.

### Source excerpt

In my day job at Anthropic, we run relatively large distributed systems to train large language models. One of the joys of using a lot of computing resources, especially on somewhat niche software stacks, is that you spend a lot of time running into the long-tail of bugs which only happen rarely or in very unusual configurations, which you happen to be the first to encounter. These bugs are frustrating, but I also often enjoy them.

## Distributed cloud builds for everyone

DevFeed: [Distributed cloud builds for everyone](<https://devfeed.tech/articles/distributed-cloud-builds-for-everyone-21949.md>)

Original publisher: [Read original article](<https://blog.nelhage.com/post/distributed-builds-for-everyone/>)

Author: Nelson Elhage

Published: 2021-05-31T23:05:17Z

Content type: opinion

Language: en

Sources: [Nelson Elhage](<https://devfeed.tech/sources/nelson-elhage.md>)

Topics: [cloud-computing](<https://devfeed.tech/topics/cloud-computing.md>), [AWS Lambda](<https://devfeed.tech/topics/aws-lambda.md>), [Development](<https://devfeed.tech/topics/development.md>), [Open Source](<https://devfeed.tech/topics/open-source.md>)

Tags: [aws-lambda](<https://devfeed.tech/tags/aws-lambda.md>), [build](<https://devfeed.tech/tags/build.md>), [cloud](<https://devfeed.tech/tags/cloud.md>), [cloud-computing](<https://devfeed.tech/tags/cloud-computing.md>), [compilation](<https://devfeed.tech/tags/compilation.md>), [compute](<https://devfeed.tech/tags/compute.md>), [distributed](<https://devfeed.tech/tags/distributed.md>), [ec2](<https://devfeed.tech/tags/ec2.md>), [github-actions](<https://devfeed.tech/tags/github-actions.md>), [open-source](<https://devfeed.tech/tags/open-source.md>), [software-development](<https://devfeed.tech/tags/software-development.md>)

### AI overview

This opinion article presents Llama as a design for making distributed cloud builds more accessible to individual developers and smaller open-source projects. It discusses AWS Lambda, cost scaling, and the potential for cloud compilation to become a common development workflow.

### Source excerpt

CPU cycles are cheaper than they have ever been, and cloud computing has never been more ubiquitous. All the major cloud providers offer generous free tiers, and services like GitHub Actions offer free compute resources to open-source repositories. So why do so many developers still build software on their laptops? Despite the embarrassment of riches of cheap or even free cloud compute, most projects I know of, and most developers, still do most of their software development -- building and running code -- directly on their local machines.

## Building LLVM in 90 seconds using Amazon Lambda

DevFeed: [Building LLVM in 90 seconds using Amazon Lambda](<https://devfeed.tech/articles/building-llvm-in-90-seconds-using-amazon-lambda-21944.md>)

Original publisher: [Read original article](<https://blog.nelhage.com/post/building-llvm-in-90s/>)

Author: Nelson Elhage

Published: 2021-05-21T02:00:28Z

Content type: article

Language: en

Sources: [Nelson Elhage](<https://devfeed.tech/sources/nelson-elhage.md>)

Topics: [LLVM](<https://devfeed.tech/topics/llvm.md>), [llama](<https://devfeed.tech/topics/llama.md>), [amazon](<https://devfeed.tech/topics/amazon.md>), [Amazon S3](<https://devfeed.tech/topics/amazon-s3.md>), [C++](<https://devfeed.tech/topics/c-plus-plus.md>), [C](<https://devfeed.tech/topics/c.md>), [Compiler](<https://devfeed.tech/topics/compiler.md>)

Tags: [amazon](<https://devfeed.tech/tags/amazon.md>), [c](<https://devfeed.tech/tags/c.md>), [c-plus-plus](<https://devfeed.tech/tags/c-plus-plus.md>), [compilation](<https://devfeed.tech/tags/compilation.md>), [compiler](<https://devfeed.tech/tags/compiler.md>), [compute](<https://devfeed.tech/tags/compute.md>), [llama](<https://devfeed.tech/tags/llama.md>), [llvm](<https://devfeed.tech/tags/llvm.md>), [optimization](<https://devfeed.tech/tags/optimization.md>), [s3](<https://devfeed.tech/tags/s3.md>)

### AI overview

The article describes Llama, a tool that offloads C and C++ compilation to Amazon Lambda. In the reported test, a stripped-down clang and LLVM build completed in about 80 seconds on a 12-core, 24-thread AMD Ryzen 9 3900 system, at an estimated cost of around forty cents.

### Source excerpt

Last week, Frederic Cambus wrote about building LLVM quickly on some very large machines, culminating in a 2m37s build on a 160-core ARM machine. I don't have a giant ARM behemoth, but I have been working on a tool I call Llama, which lets you offload computational work - including C and C++ builds - onto Amazon Lambda. I decided to see how good it could do at a similar build.

## Some opinionated thoughts on SQL databases

DevFeed: [Some opinionated thoughts on SQL databases](<https://devfeed.tech/articles/some-opinionated-thoughts-on-sql-databases-21965.md>)

Original publisher: [Read original article](<https://blog.nelhage.com/post/some-opinionated-sql-takes/>)

Author: Nelson Elhage

Published: 2021-03-30T17:32:31Z

Content type: opinion

Language: en

Sources: [Nelson Elhage](<https://devfeed.tech/sources/nelson-elhage.md>)

Topics: [Databases](<https://devfeed.tech/topics/databases.md>), [SQL](<https://devfeed.tech/topics/sql.md>), [MySQL](<https://devfeed.tech/topics/mysql.md>), [PostgreSQL](<https://devfeed.tech/topics/postgresql.md>), [SQLite](<https://devfeed.tech/topics/sqlite.md>), [backends](<https://devfeed.tech/topics/backends.md>), [Web APIs](<https://devfeed.tech/topics/web-apis.md>), [Latency](<https://devfeed.tech/topics/latency.md>)

Tags: [acid](<https://devfeed.tech/tags/acid.md>), [backends](<https://devfeed.tech/tags/backends.md>), [consistency](<https://devfeed.tech/tags/consistency.md>), [databases](<https://devfeed.tech/tags/databases.md>), [durability](<https://devfeed.tech/tags/durability.md>), [high-availability](<https://devfeed.tech/tags/high-availability.md>), [low-latency](<https://devfeed.tech/tags/low-latency.md>), [mysql](<https://devfeed.tech/tags/mysql.md>), [open-source](<https://devfeed.tech/tags/open-source.md>), [postgresql](<https://devfeed.tech/tags/postgresql.md>), [sql](<https://devfeed.tech/tags/sql.md>), [sqlite](<https://devfeed.tech/tags/sqlite.md>), [web-apis](<https://devfeed.tech/tags/web-apis.md>)

### AI overview

An opinionated discussion of SQL databases, focused mainly on MySQL and PostgreSQL, with some coverage of SQLite. Drawing on experience using databases as backends for web applications and APIs, the article argues that open-source SQL databases provide excellent storage engines, throughput, durability, and transactional guarantees when properly tuned.

### Source excerpt

People who work with me tend to realize that I have Opinions about databases, and SQL databases in particular. Last week, I wrote about a Postgres debugging story and tweeted about AWS' policy ban on internal use of SQL databases, and had occasion to discuss and debate some of those feelings on Twitter; this article is an attempt to write up more of them into a single place I can refer to.

## Towards solving Ultimate Tic Tac Toe

DevFeed: [Towards solving Ultimate Tic Tac Toe](<https://devfeed.tech/articles/towards-solving-ultimate-tic-tac-toe-21964.md>)

Original publisher: [Read original article](<https://blog.nelhage.com/post/solving-ultimate-ttt/>)

Author: Nelson Elhage

Published: 2020-07-15T17:15:21Z

Content type: article

Language: en

Sources: [Nelson Elhage](<https://devfeed.tech/sources/nelson-elhage.md>)

Topics: [Algorithms](<https://devfeed.tech/topics/algorithms.md>), [Rust](<https://devfeed.tech/topics/rust.md>), [Programming](<https://devfeed.tech/topics/programming.md>)

Tags: [algorithms](<https://devfeed.tech/tags/algorithms.md>), [experience](<https://devfeed.tech/tags/experience.md>), [game-theory](<https://devfeed.tech/tags/game-theory.md>), [games](<https://devfeed.tech/tags/games.md>), [parallel](<https://devfeed.tech/tags/parallel.md>), [performance](<https://devfeed.tech/tags/performance.md>), [programming](<https://devfeed.tech/tags/programming.md>), [rust](<https://devfeed.tech/tags/rust.md>), [strategy](<https://devfeed.tech/tags/strategy.md>)

### AI overview

The article describes an ongoing effort to solve Ultimate Tic Tac Toe by developing a minimax AI and investigating algorithms from combinatorial game theory. The project also provided experience with Rust and high-performance parallel programming, but the game had not yet been solved.

### Source excerpt

Summary: Read about my efforts to solve the game of Ultimate Tic Tac Toe. It's been a fun journey into interesting algorithms and high-performance parallel programming in Rust. Backstory Starting around the beginning of the COVID-19 lockdown, I've gotten myself deeply nerdsniped by an attempt to solve the game of Ultimate Tic Tac Toe, a two-level Tic Tac Toe variant which is (unlike Tic Tac Toe) nontrivial and contains some interesting strategic elements.

## Write testable code by writing generic code

DevFeed: [Write testable code by writing generic code](<https://devfeed.tech/articles/write-testable-code-by-writing-generic-code-21978.md>)

Original publisher: [Read original article](<https://blog.nelhage.com/post/write-testable-code-by-writing-generic-code/>)

Author: Nelson Elhage

Published: 2020-03-12T01:30:17Z

Content type: article

Language: en

Sources: [Nelson Elhage](<https://devfeed.tech/sources/nelson-elhage.md>)

Topics: [Testing](<https://devfeed.tech/topics/testing.md>), [Amazon S3](<https://devfeed.tech/topics/amazon-s3.md>), [Code](<https://devfeed.tech/topics/code.md>), [Software Engineering](<https://devfeed.tech/topics/software-engineering.md>)

Tags: [code](<https://devfeed.tech/tags/code.md>), [s3](<https://devfeed.tech/tags/s3.md>), [software](<https://devfeed.tech/tags/software.md>), [software-testing](<https://devfeed.tech/tags/software-testing.md>), [testing](<https://devfeed.tech/tags/testing.md>)

### AI overview

The article examines how to make difficult error-handling behavior easier to test by generalizing a specific S3 connection-reset problem into a generic retryable stream abstraction. The supplied text describes restarting reads from the current byte offset using S3 range requests.

### Source excerpt

Alex Gaynor recently asked this question in an IRC channel I hang out in (a channel which contains several software engineers nearly as obsessed with software testing as I am): uhh, so I'm writing some code to handle an econnreset... how do I test this? This is a good question! Testing ECONNRESET is one of those fiddly problems that exists at the interface between systems -- in his case, with S3, not even a system under his control -- that can be infuriatingly tricky to reproduce and test.

## Test suites as classifiers

DevFeed: [Test suites as classifiers](<https://devfeed.tech/articles/test-suites-as-classifiers-21970.md>)

Original publisher: [Read original article](<https://blog.nelhage.com/post/test-suites-as-classifiers/>)

Author: Nelson Elhage

Published: 2020-03-01T20:34:00Z

Content type: opinion

Language: en

Sources: [Nelson Elhage](<https://devfeed.tech/sources/nelson-elhage.md>)

Topics: [test](<https://devfeed.tech/topics/test.md>), [Code](<https://devfeed.tech/topics/code.md>), [Development](<https://devfeed.tech/topics/development.md>), [bug](<https://devfeed.tech/topics/bug.md>), [incident](<https://devfeed.tech/topics/incident.md>)

Tags: [bug](<https://devfeed.tech/tags/bug.md>), [code](<https://devfeed.tech/tags/code.md>), [development](<https://devfeed.tech/tags/development.md>), [incident](<https://devfeed.tech/tags/incident.md>), [patches](<https://devfeed.tech/tags/patches.md>), [test](<https://devfeed.tech/tags/test.md>), [tests](<https://devfeed.tech/tags/tests.md>)

### AI overview

The article presents a test suite as a classifier that evaluates whether code changes are acceptable to apply and deploy. It explains that test results can produce false alarms or missed alarms, whose costs differ: missed alarms may allow bugs to reach production, while false alarms slow development.

### Source excerpt

Suppose we have some codebase we're considering applying some patch to, and which has a robust and maintained test suite. Considering the patch, we may ask, is this patch acceptable to apply and deploy. By this we mean to ask if the patch breaks any important functionality, violates any key properties or invariants of the codebase, or would otherwise cause some unacceptable risk or harm. In principle, we can divide all patches into "acceptable" or "unacceptable" relative to some project-specific notion of what we're willing to allow.

## Systems that defy detailed understanding

DevFeed: [Systems that defy detailed understanding](<https://devfeed.tech/articles/systems-that-defy-detailed-understanding-21968.md>)

Original publisher: [Read original article](<https://blog.nelhage.com/post/systems-that-defy-understanding/>)

Author: Nelson Elhage

Published: 2020-02-22T20:00:00Z

Content type: opinion

Language: en

Sources: [Nelson Elhage](<https://devfeed.tech/sources/nelson-elhage.md>)

Topics: [systems](<https://devfeed.tech/topics/systems.md>), [distributed-systems](<https://devfeed.tech/topics/distributed-systems.md>), [debugging](<https://devfeed.tech/topics/debugging.md>), [incident](<https://devfeed.tech/topics/incident.md>)

Tags: [debugging](<https://devfeed.tech/tags/debugging.md>), [distributed-systems](<https://devfeed.tech/tags/distributed-systems.md>), [incident](<https://devfeed.tech/tags/incident.md>), [software](<https://devfeed.tech/tags/software.md>), [stripe](<https://devfeed.tech/tags/stripe.md>), [systems](<https://devfeed.tech/tags/systems.md>)

### AI overview

The article argues that pursuing detailed explanations is not always the best strategy for dealing with complex systems. It focuses on distributed systems, where component and network failures make complete understanding difficult or impractical, and contrasts investigating root causes with improving fault-tolerance mechanisms such as retries and timeouts.

### Source excerpt

Last week, I wrote about the mindset that computer systems can be understood, and behaviors can be explained, if we're willing to dig deep enough into the stack of abstractions our software is built atop. Some of the ensuing discussion on Twitter and elsewhere lead me to write this followup, in which I want to run through a few classes of systems where I've found pursuing in-detail understanding of the system wasn't the right answer.

## Computers can be understood

DevFeed: [Computers can be understood](<https://devfeed.tech/articles/computers-can-be-understood-21945.md>)

Original publisher: [Read original article](<https://blog.nelhage.com/post/computers-can-be-understood/>)

Author: Nelson Elhage

Published: 2020-02-16T20:00:00Z

Content type: opinion

Language: en

Sources: [Nelson Elhage](<https://devfeed.tech/sources/nelson-elhage.md>)

Topics: [Software](<https://devfeed.tech/topics/software.md>), [systems](<https://devfeed.tech/topics/systems.md>), [coding](<https://devfeed.tech/topics/coding.md>)

Tags: [abstraction](<https://devfeed.tech/tags/abstraction.md>), [architecture](<https://devfeed.tech/tags/architecture.md>), [browser](<https://devfeed.tech/tags/browser.md>), [compilers](<https://devfeed.tech/tags/compilers.md>), [complexity](<https://devfeed.tech/tags/complexity.md>), [concepts](<https://devfeed.tech/tags/concepts.md>), [cpu](<https://devfeed.tech/tags/cpu.md>), [exploration](<https://devfeed.tech/tags/exploration.md>), [hardware](<https://devfeed.tech/tags/hardware.md>), [javascript](<https://devfeed.tech/tags/javascript.md>), [kernel](<https://devfeed.tech/tags/kernel.md>), [learning](<https://devfeed.tech/tags/learning.md>), [software](<https://devfeed.tech/tags/software.md>), [systems](<https://devfeed.tech/tags/systems.md>)

### AI overview

The author describes a mindset centered on the belief that computers and software systems can be understood through determined exploration and learning. Although modern systems are highly complex and layered, each layer can be understood at an appropriate level of abstraction, with deeper investigation possible when needed.

### Source excerpt

Introduction This post attempts to describe a mindset I've come to realize I bring to essentially all of my work with software. I attempt to articulate this mindset, some of its implications and strengths, and some of the ways in which it's lead me astray. Software can be understood I approach software with a deep-seated belief that computers and software systems can be understood. This belief is, for me, not some abstruse theoretical assertion, but a deeply felt belief that essentially any question I might care to ask (about computers) has a comprehensible answer which is accessible with determined exploration and learning.

## Reflections on software performance

DevFeed: [Reflections on software performance](<https://devfeed.tech/articles/reflections-on-software-performance-21961.md>)

Original publisher: [Read original article](<https://blog.nelhage.com/post/reflections-on-performance/>)

Author: Nelson Elhage

Published: 2020-02-03T01:00:00Z

Content type: opinion

Language: en

Sources: [Nelson Elhage](<https://devfeed.tech/sources/nelson-elhage.md>)

Topics: [Software](<https://devfeed.tech/topics/software.md>), [coding](<https://devfeed.tech/topics/coding.md>)

Tags: [blog](<https://devfeed.tech/tags/blog.md>), [performance](<https://devfeed.tech/tags/performance.md>), [reflections](<https://devfeed.tech/tags/reflections.md>), [software](<https://devfeed.tech/tags/software.md>)

### AI overview

A reflection on lessons learned from building and using performant software. The article argues that performance is a feature that affects how software is perceived and used, and that making tools faster enables users to pursue tasks more frequently and in new ways.

### Source excerpt

At this point in my career, I've worked on at least three projects where performance was a defining characteristic: Livegrep, Taktician, and Sorbet (I discussed sorbet in particular last time, and livegrep in an earlier post). I've also done a lot of other performance work on the tools I use, some of which ended up on my other blog, Accidentally Quadratic. In this post, I want to reflect on some of the lessons I've learned while writing performant software, and working with rather a lot more not-so-performant software.

## Why the Sorbet typechecker is fast

DevFeed: [Why the Sorbet typechecker is fast](<https://devfeed.tech/articles/why-the-sorbet-typechecker-is-fast-21977.md>)

Original publisher: [Read original article](<https://blog.nelhage.com/post/why-sorbet-is-fast/>)

Author: Nelson Elhage

Published: 2020-01-24T01:00:00Z

Content type: article

Language: en

Sources: [Nelson Elhage](<https://devfeed.tech/sources/nelson-elhage.md>)

Topics: [Ruby](<https://devfeed.tech/topics/ruby.md>), [C++](<https://devfeed.tech/topics/c-plus-plus.md>), [Data structures](<https://devfeed.tech/topics/data-structures.md>), [Architecture & Design](<https://devfeed.tech/topics/architecture-design.md>), [cpu](<https://devfeed.tech/topics/cpu.md>), [Software Engineering](<https://devfeed.tech/topics/software-engineering.md>)

Tags: [c-plus-plus](<https://devfeed.tech/tags/c-plus-plus.md>), [cpu](<https://devfeed.tech/tags/cpu.md>), [data-structures](<https://devfeed.tech/tags/data-structures.md>), [performance](<https://devfeed.tech/tags/performance.md>), [ruby](<https://devfeed.tech/tags/ruby.md>), [software-design](<https://devfeed.tech/tags/software-design.md>)

### AI overview

This article explains why the Sorbet Ruby typechecker is fast. It attributes the performance to C++, native-code compilation, explicit data-structure and allocation control, and designs that improve CPU cache locality. Informal benchmarks on Stripe's codebase measured about 100,000 lines of code typechecked per second per core.

### Source excerpt

This is the second in an indefinite series of posts about things that I think went well in the Sorbet project. The previous one covered our testing approach. Sorbet is fast. Numerous of our early users commented specifically on how fast it was, and how much they appreciated this speed. Our informal benchmarks on Stripe's codebase clocked it as typechecking around 100,000 lines of code per second per core, making it one of the fastest production typecheckers we are aware of.

[Next page](<https://devfeed.tech/sources/nelson-elhage.md?cursor=WyIyMDIwLTAxLTI0VDAxOjAwOjAwKzAwOjAwIiwgIjE2NWQwNjlmLTY4MTctNDMzNi04MDQzLTkwYzlhMzljZTM1NyJd>)