# Evan Jones I'm a software engineer at Datadog in New York. I previously worked at Bluecore, fixed interesting bugs at Twitter, and taught a database class at Columbia as an adjunct. I was a co-founder and CTO of Mitro, a password manager for groups and organizations, along with Vijay Pandurangan and Adam Hilss. Before that, I earned a Ph. D. from MIT, researching distributed OLTP databases with Sam Madden. Even earlier in my life, I worked at Google in New York for a bit more than a year, and I was a gradu This is one page of public article previews, not the complete archive. Follow Next page to continue. Summaries are not the original full articles. ## Why C's setenv() and unsetenv() Are Not Thread-Safe DevFeed: [Why C's setenv() and unsetenv() Are Not Thread-Safe]() Original publisher: [Read original article]() Published: 2023-11-19T14:13:23Z Content type: opinion Language: en Sources: [Evan Jones]() Topics: [C](), [POSIX](), [bug](), [Concurrency](), [Concurrent Programming](), [Go](), [Go Language](), [Rust]() Tags: [bug](), [c](), [crash](), [dns](), [go](), [posix](), [rust](), [thread]() ### AI overview The article explains that C's setenv() and unsetenv() modify global environment state and can race with getenv(), causing crashes in multithreaded programs. It argues that the POSIX interface is difficult to use safely and has affected software such as Go and Rust. ### Source excerpt You can't safely use the C setenv() or unsetenv() functions in a program that uses threads. Those functions modify global state, and can cause other threads calling getenv() to crash. This also causes crashes in other languages that use those C standard library functions, such as Go's os.Setenv (Go issue) and Rust's std::env::set_var() (Rust issue). I ran into this in a Go program, because Go's built-in DNS resolver can call C's getaddrinfo(), which uses environment variables. This cost me 2 days to track down and file the Go bug. Sadly, this problem has been known for decades. For example, an article from January 2017 said: "None of this is new, but we do re-discover it roughly every five years. See you in 2022." This was only one year off! (She wrote an update in October 2023 after I emailed her about my Go bug.) This is a flaw in the POSIX standard, which extends the C Standard to allow modifying environment varibles. The most infuriating part is that many people who could influence the standard or maintain the C libraries don't see this as a problem. The argument is that the specification clearly documents that setenv() cannot be used with threads. Therefore, if someone does this, the crashes are their fault. We should apparently read every function's specification carefully, not use software written by others, and not use threads. These are unrealistic assumptions in modern software. I think we should instead strive to create APIs that are hard to screw up, and evolve as the ecosystem changes. The C language and standard library continue to play an important role at the base of most software. We either need to figure out how to improve it, or we need to figure out how to abandon it. Why is setenv() not thread-safe? The biggest problem is that getenv() returns a char*, with no need for applications to free it later. One thread could be using this pointer when another thread changes the same environment variable using setenv() or unsetenv(). The getenv() function ## Random Load Balancing is Unevenly Distributed DevFeed: [Random Load Balancing is Unevenly Distributed]() Original publisher: [Read original article]() Published: 2023-08-29T13:13:23Z Content type: article Language: en Sources: [Evan Jones]() Topics: [Load Balancing](), [distributed-systems](), [Simulation](), [Server]() Tags: [capacity](), [distributed](), [load-balancing](), [simulation]() ### AI overview Randomly distributing work across servers creates load imbalance because the most-loaded server, rather than the average server, determines required capacity. A simulation illustrates how this can waste capacity and cause worse-than-linear scaling. ### Source excerpt This is a reminder that random load balancing is unevenly distributed. If we distribute a set of items randomly across a set of servers (e.g. by hashing, or by randomly selecting a server), the average number of items on each server is num_items / num_servers. It is easy to assume each server has close to the same number of items. However, since we are selecting servers at random, they will have different numbers of items, and the imbalance can be important. For load balancing, a reasonable model is that each server has fixed capacity (e.g. it can serve 3000 requests/second, or store 100 items, etc.). We need to divide the total workload over the servers, so that each server stays below its capacity. This means the number of servers is determined by the most loaded server, not the average. This is a classic balls in bins problem that has been well studied, and there are some interesting theoretical results. However, I wanted some specific numbers, so I wrote a small simulation. The summary is that the imbalance varies with the expected number of items per server (that is, num_items / num_servers). A workload is more balanced with more items or with fewer servers. Most interestingly, this means that scaling a system by adding more servers makes the distribution more unfair. This is one reason we can get worse than linear scaling of some distributed systems. Let's make this more concrete with an example. Let's assume we have a workload of 1000 items, and each server can hold a maximum of 100 items. If we place the exact same number of items on each server, we only need 10 servers, and each of them is completely busy. However, if we place the items randomly, then the median (p50) number of items is 100 items. This means half the servers will have more than 100 items, and will be overloaded. If we want less than a 1% chance of an overloaded server, we need to look at the 99th percentile (p99) server load. We need to use at least 13 servers, which has a p99 load of 97 it ## Nanosecond timestamp collisions are common DevFeed: [Nanosecond timestamp collisions are common]() Original publisher: [Read original article]() Published: 2023-07-20T21:39:42Z Content type: article Language: en Sources: [Evan Jones]() Topics: [DateTime](), [Go Language]() Tags: [go](), [threads](), [time]() ### AI overview The article reports tests of timestamp collisions using Go's absolute and monotonic clocks. It finds that raw nanosecond timestamps can collide frequently across threads and that collision behavior varies between Linux and Mac OS X. ### Source excerpt I was wondering: how often do nanosecond timestamps collide on modern systems? The answer is: very often, like 5% of all samples, when reading the clock on all 4 physical cores at the same time. As a result, I think it is unsafe to assume that a raw nanosecond timestamp is a unique identifier. I wrote a small test program to test this. I used Go, which records both the "absolute" time and the "monotonic clock" relative time on each call to time.Now(), so I compared both the relative difference between consecutive timestamps, as well as just the absolute timestamps. As expected, the behavior depends on the system, so I observe very different results on Mac OS X and Linux. On Linux, within a single thread, both the absolute and monotonic times always increase. On my system, the minimum increment was 32 ns. Between threads, approximately 5% of the absolute times were exactly the same as other threads. Even with 2 threads on a 4 core system, approximately 2% of timestamps collided. On Mac OS X: the absolute time has microsecond resolution, so there are an astronomical number of collisions when I repeat this same test. Even within a thread I often observe the monotonic clock not increment. See the test program on Github if you are curious. ## How much does the read/write buffer size matter for socket throughput? DevFeed: [How much does the read/write buffer size matter for socket throughput?]() Original publisher: [Read original article]() Published: 2023-07-16T16:04:32Z Content type: article Language: en Sources: [Evan Jones]() Topics: [IO](), [systems](), [Linux](), [Benchmark](), [networking](), [Rust](), [Ubuntu](), [Google Cloud Platform (GCP)]() Tags: [amd](), [benchmark](), [blocking](), [cloud](), [cpu](), [kernel](), [linux](), [networking](), [performance](), [processors](), [rust](), [tcp](), [test](), [ubuntu](), [unix]() ### AI overview This article reports experiments on how blocking I/O buffer sizes affect read() and write() throughput. It finds that approximately 32 KiB is a useful starting point, while large writes may benefit from buffers around 256 KiB to 1 MiB. Results vary by hardware and software. ### Source excerpt The read() and write() system calls take a variable-length byte array as an argument. As a simplified model, the time for the system call should be some constant "per-call" time, plus time directly proportional to the number of bytes in the array. That is, the time for each call should be time = (per_call_minimum_time) + (array_len) x (per_byte_time). With this model, using a larger buffer should increase throughput, asymptotically approaching 1/per_byte_time. I was curious: do real system calls behave this way? What are the ideal buffer sizes for read() and write() if we want to maximize throughput? I decided to do some experiments with blocking I/O. These are not rigorous, and I suspect the results will vary significantly if the hardware and software are different than one the system I tested. The really short answer is that a buffer of 32 KiB is a good starting point on today's systems, and I would want to measure the performance to go beyond that. However, for large writes, performance can increase. On Linux, the simple model holds for small buffers (≤ 4 KiB), but once the program approaches the maximum throughput, the throughput becomes highly variable and in many cases decreases as the buffers get larger. For blocking I/O, approximately 32 KiB is large enough to hit the maximum throughput for read(), but write() throughput improves with buffers up to around 256 KiB - 1 MiB. The reason for the asymmetry is that the Linux kernel will only write less than the entire buffer (a "short write") if there is an error (e.g. a signal causing EINTR). Thus, larger write buffers means the operating system needs to switch to the process less often. On the other head, "short reads", where a read() returns less than the maximum length, become increasingly common as the buffer size increases, which diminishes the benefit. There is a SO_RCVLOWAT socket option to change this that I did not test. The experiments were run on two 16 CPU Google Cloud T2D instances, which use AMD EPYC ## The C Standard Library Function isspace() Depends on Locale DevFeed: [The C Standard Library Function isspace() Depends on Locale]() Original publisher: [Read original article]() Published: 2023-06-06T13:41:23Z Content type: article Language: en Sources: [Evan Jones]() Topics: [C](), [ASCII](), [bug](), [Library]() Tags: [ascii](), [bug](), [c](), [function](), [implementation](), [library](), [locale](), [mac](), [mac-os](), [standard](), [standard-library]() ### AI overview The article explains that C's isspace() behavior depends on the active locale. In the default C locale it recognizes six ASCII whitespace characters, while other locales may also recognize Unicode whitespace. The author connects this behavior to a parsing bug involving PostgreSQL Hstore values on Mac OS X. ### Source excerpt This is a post for myself, because I wasted a lot of time understanding this bug, and I want to be able to remember it in the future. I expect close to zero others to be interested. The C standard library function isspace() returns a non-zero value (true) for the six "standard" ASCII white-space characters ('\t', '\n', '\v', '\f', '\r', ' '), and any locale-specific characters. By default, a program starts in the "C" locale, which will only return true for the six ASCII white-space characters. However, if the program changes locales, it can return true for other values. As a result, unless you really understand locales, you should use your own version of this function, or ICU4C's u_isspace() function. An implementation of isspace() for ASCII is one line: /* Returns true for the 6 ASCII white-space characters: \t \n \v \f \r ' '. */ int isspace_ascii(int c) { return c == '\t' || c == '\n' || c == '\v' || c == '\f' || c == '\r' || c == ' '; } I ran into this because On Mac OS X, Postgres switches to the system's default locale, which is something that uses UTF-8 (e.g. en_US.UTF-8, fr_CA.UTF-8, etc). In this case, isspace() returns true for Unicode white-space values, which includes 0x85 = NEL = Next Line, and 0xA0 = NBSP = No-Break Space. This caused a bug in parsing Postgres Hstore values that use Unicode. I have attempted to submit a patch to fix this (mailing list post, commitfest entry). For a program to demonstrate the behaviour on different systems, see isspace_locale on Github. ## Huge pages can reduce virtual-memory translation overhead DevFeed: [Huge pages can reduce virtual-memory translation overhead]() Original publisher: [Read original article]() Published: 2023-01-16T16:46:39Z Content type: opinion Language: en Sources: [Evan Jones]() Topics: [Kernel](), [x86](), [cpu](), [Linux](), [systems]() Tags: [cpu](), [kernel](), [linux](), [memory](), [performance](), [systems](), [x86]() ### AI overview This article explains how huge pages can reduce virtual-memory page mapping overhead caused by limited CPU Translation Lookaside Buffers. A memory-access experiment found that 2 MiB huge pages were 2.9x faster than 4 kiB pages on an Intel 11th generation Core i5-1135G7, while 1 GiB pages were 3.1x faster. ### Source excerpt Nearly all programs are written to access virtual memory addresses, which the CPU must translate to physical addresses. These translations are usually fast because the mappings are cached in the CPU's Translation Lookaside Buffer (TLB). Unfortunately, virtual memory on x86 has used a 4 kiB page size since the 386 was released in 1985, when computers had a bit less memory than they do today. Also unfortunately, TLBs are pretty small because they need to be fast. For example, AMD's Zen 4 Microarchitecture, which first shipped in September 2022, has a first level data TLB with 72 entries, and a second level TLB with 3072 entries. This means when an application's working set is larger than approximately 4 kiB x 3072 = 12 MiB, some memory accesses will require page table lookups, multiplying the number of memory accesses required. This is a brand-new CPU, with one of the biggest TLBs on the market, so most systems will be worse. Using larger virtual memory page sizes (aka huge pages) can reduce page mapping overhead substantially. Since RAM is so much larger than it was in 1985, a larger page size seems like obviously a good idea to me. In 2021, Google published a paper about making their malloc implementation (TCMalloc) huge page aware (called Temeraire). They report this improved average requests-per-second throughput across their fleet by 7%, by increasing the amount of memory that is backed by huge pages. This made me curious about the "best case" performance benefits. I wrote a small program that allocates 4 GiB, then randomly reads uint64 values from it. On my Intel 11th generation Core i5-1135G7 (Tiger Lake) from 2020, using 2 MiB huge pages is 2.9x faster. I also tried 1 GiB pages, which is 3.1x faster than 4 kiB pages, but only 8% faster than 2 MiB pages. My conclusion: Using madvise() to get the kernel to use huge pages seems like a relatively easy performance win for applications that use a large amount of RAM. Unfortunately, using larger pages is not without ## Replicating Database Changes to a Message Queue is Tricky DevFeed: [Replicating Database Changes to a Message Queue is Tricky]() Original publisher: [Read original article]() Published: 2022-12-13T02:01:42Z Content type: tutorial Language: en Sources: [Evan Jones]() Topics: [Database](), [Kafka](), [systems]() Tags: [database](), [kafka](), [message-queue](), [systems]() ### AI overview This article explains why replicating database changes to a message queue can produce missing or extra updates when components fail. It compares parallel and sequential approaches and recommends using a database transaction to update application state and record pending messages at a single commit point. ### Source excerpt Let's imagine we have an program that stores its state in a database, and we want other programs to do things when changes occur. For example, we might want to send email notifications if a bank balance drops below a threshold. This is a very common reason applications use message queues like Kafka. Unfortunately, the "trivial" implementation does not work when components fail. I suspect there are many real applications that get this wrong. Most of the time, these applications work correctly, and the changes are replicated across multiple systems. However, when things restart, updates can go missing, or extra updates can appear. In this article, I'm going to try to explain how this can go wrong, and some ways to fix it. Attempt one: Update both in parallel The application performs the following operations: Write the change to the database. At the same time, publish the message to the message queue. The problem in this case is that the database update could fail, but publishing the message succeeds. This means applications consuming the stream receive an "extra" update that does not exist in the database. Attempt two: Update database, then publish message Okay, let's try again, and make sure that updating the database succeeds: Write the change to the database. Wait for the database to confirm the write occurred. Publish the message to the message queue. We fixed the "extra" message update problem! However, we still have a problems: If the application crashes after writing to the database, but before publishing the message, the stream is missing an update. This can be particularly bad if the message queue is unavailable. The application can retry publishing the message for a while. However, if the message queue is down for long enough, it is likely the application will run out of memory, or be restarted. In this case, all the pending updates are lost. So now what? We can't do the operations sequentially, and we can't do them in parallel. The trick is to order the wor ## Go: Functional options are slow DevFeed: [Go: Functional options are slow]() Original publisher: [Read original article]() Published: 2022-05-23T14:19:42Z Content type: article Language: en Sources: [Evan Jones]() Topics: [Go Language](), [configuration](), [Benchmark](), [Code](), [Compiler]() Tags: [analysis](), [benchmark](), [compiler](), [configuration](), [function](), [functional](), [go](), [memory](), [performance]() ### AI overview This article examines Go's functional options pattern through a microbenchmark. It reports that functional options can be slower than configuration structs, may require more instructions and memory allocations, and are less likely to be inlined, especially when calls go through an interface. The author recommends configuration structs for performance-critical code. ### Source excerpt The Go "functional options" pattern is a way of passing options to a function. The function takes a variable number of arguments, which are themselves functions (a type like ...func(*config). I think it was first introduced by Rob Pike in a 2014 blog post. It is now used by many APIs. For example, gRPC's DialContext(), AWS's LoadDefaultConfig(), and OpenTelemetry's Tracer.Start(). This style should be avoided when performance is critical. This article describes a microbenchmark that shows functional options are slower, require more instructions, are less likely to be inlined, and may require memory allocations. In the worst case, Go must allocate a slice for the ... argument, and an additional object for each option in the slice. Often, the compiler's escape analysis can optimize these away, but not always. In particular, the compiler can never optimize an interface method call. The alternative is to pass a struct with the configuration values. This does not rely on the compiler to be efficient. In my opinion, performance-critical code should be efficient "by design," and not rely on compiler optimizations that may or may not apply. This article presents a brief experiment to try and demonstrate the performance differences. Functional options versus configuration structs The alternative to functional options is passing a configuration struct, like the standard library's http.Server or tls.Config. To make this concrete, let's consider a constructor function that creates a type *Foo with two options: a boolean (enabling a feature that is disabled by default), and an integer (taking a numeric value). With functional options, the code would look like the following: Functional options creating *Foo func NewFoo(options ...FooOption) *Foo { // ... implementation } func CallNewFoo() *Foo { return NewFoo(WithBoolOption(), WithIntOption(42)) } The configuration struct version would look something like the following: Configuration struct creating *Foo func NewFooStruct(config ## Postgres large sub-string query performance DevFeed: [Postgres large sub-string query performance]() Original publisher: [Read original article]() Published: 2022-02-27T21:47:01Z Content type: article Language: en Sources: [Evan Jones]() Topics: [benchmarking](), [Benchmark](), [Query (disambiguation)]() Tags: [benchmark](), [performance](), [postgres]() ### AI overview A benchmark of PostgreSQL substring queries on large variable-length strings finds that substring searches are about six times slower than HSTORE or JSONB key lookups. Regular expressions are about three times slower than LIKE, BYTEA queries are faster than TEXT queries, and inline compressed TOAST storage outperforms uncompressed and out-of-line storage in the tested workload. ### Source excerpt Following up on my last post about large JSON queries, I also benchmarked sub-string queries on large variable-length strings. I wanted to check if sub-string queries might be faster than HSTORE or JSONB key lookups. I tested both binary (BYTEA) and Unicode text (TEXT). Unfortunately, Postgres sub-string queries are about 6x slower than HSTORE or JSONB key queries. I also learned that Postgres regular expressions are extremely slow: about 3x slower than using LIKE. Queries on BYTEA are faster than queries on TEXT. Perhaps the most interestingly, TOAST inline compressed storage was faster than the uncompressed storage. However, similar to my results with JSON values, out-of-line storage using a separate TOAST table is quite a bit slower than inline storage. I spent a bit of time looking at the code that implements Postgres's LIKE operator and the implementation of the POSITION function. I'm pretty sure they could be made quite a bit faster, at least for UTF-8 or binary strings. For example, the BYTEA implementation of POSITION is a function called byteapos. It implements the simple O(nm) implementation, where for each byte of the string, you compare it to the substring. There are much faster implementations that can use SIMD instructions. I think there is an opportunity to make POSITION() and LIKE substantially faster. Query performance benchmark For details on the benchmark setup, see my previous article. In this case, I tested TEXT and BLOB columns that are near the 2004 byte limit for inline uncompressed tuples. I measured querying substrings that either did not match, or matched at the very end, which should represent the "worst case" search times. I tested LIKE, regular expressions (~ operator), and the POSITION() function. The entire workload was in memory, and parallel queries were disabled. For more details, see the benchmark source code in Github, which links to a Google sheet with the raw results. Fastest query times (ms) This table shows the fastest query ## Postgres large JSON value query performance DevFeed: [Postgres large JSON value query performance]() Original publisher: [Read original article]() Published: 2022-02-01T14:19:27Z Content type: article Language: en Sources: [Evan Jones]() Topics: [JSON](), [Benchmark](), [Database](), [data]() Tags: [benchmark](), [database](), [json](), [performance](), [postgres]() ### AI overview This article presents benchmark results on query performance for large JSON, JSONB, and HSTORE values in Postgres. It reports a 2-10x slowdown once rows exceed about 2 KiB, discusses the effects of compression and external TOAST storage, and suggests JSONB, HSTORE, splitting large values across rows, and LZ4 compression as relevant considerations. ### Source excerpt Postgres supports three types for "schemaless" data: JSON (added in 9.2), JSONB (added in 9.4), and HSTORE (added in 8.2 as an extension). Unfortunately, the performance of queries of all three gets substantially slower (2-10x) for values larger than about 2 kiB, due to how Postgres stores long variable-length data (TOAST). The same performance cliff applies to any variable-length types, like TEXT and BYTEA. This article contains some quick-and-dirty benchmark results to explore how Postgres's performance changes for the "schemaless" data types when they become large. My conclusion is that you should expect a 2-10x slower queries once a row gets larger than Postgres's 2 kiB limit. Most applications should use JSONB for schemaless data. It stores parsed JSON in a binary format, so queries are efficient. Accessing JSONB values is about 2x slower than accessing a BYTEA column. Queries on HSTORE values are slightly faster (~10-20%), so if performance is critical and string key/value pairs are sufficient, it is worth considering. Never use JSON because the performance is terrible. Compressed values makes queries take about 2x more time, and queries for values stored in external TOAST tables take about 5x more time. In cases where you need excellent query performance, you may want to consider trying to split large JSON values across multiple rows. If you are using Postgres 14 or later, you should use LZ4 compression. I didn't test it, but others have found it to use a bit more space but be signficantly faster (1, 2). This should reduce the performance penalty for compressed values. This is a general database problem and not a Postgres problem: MySQL and others have their own performance cliffs for large values. However, Postgres's row length limit is pretty low. I suspect the 8 kiB page may be the wrong default these days. A rough rule of thumb is that smaller pages are better for workloads that read and write small values, but larger pages are likely better for queries t