# Danluu

Recent content on

This is one page of public article previews, not the complete archive. Follow Next page to continue. Summaries are not the original full articles.

## Test and verification technique effectiveness with GPT-6 Astra

DevFeed: [Test and verification technique effectiveness with GPT-6 Astra](<https://devfeed.tech/articles/test-and-verification-technique-effectiveness-with-gpt-6-astra-122143.md>)

Original publisher: [Read original article](<https://danluu.com/astra-testing/>)

Published: 2026-11-02T00:00:00Z

Content type: news

Language: en

Sources: [Danluu](<https://devfeed.tech/sources/danluu.md>)

Topics: [test](<https://devfeed.tech/topics/test.md>), [Benchmark](<https://devfeed.tech/topics/benchmark.md>), [Fuzzing/Fuzz testing](<https://devfeed.tech/topics/fuzzing.md>), [Test-driven development](<https://devfeed.tech/topics/tdd.md>), [Automated reasoning](<https://devfeed.tech/topics/automated-reasoning.md>)

Tags: [2](<https://devfeed.tech/tags/2.md>), [3](<https://devfeed.tech/tags/3.md>), [about](<https://devfeed.tech/tags/about.md>), [agents](<https://devfeed.tech/tags/agents.md>), [all](<https://devfeed.tech/tags/all.md>), [also](<https://devfeed.tech/tags/also.md>), [astra](<https://devfeed.tech/tags/astra.md>), [benchmark](<https://devfeed.tech/tags/benchmark.md>), [best](<https://devfeed.tech/tags/best.md>), [bugs](<https://devfeed.tech/tags/bugs.md>), [but](<https://devfeed.tech/tags/but.md>), [call](<https://devfeed.tech/tags/call.md>), [can](<https://devfeed.tech/tags/can.md>), [effectiveness](<https://devfeed.tech/tags/effectiveness.md>), [gpt](<https://devfeed.tech/tags/gpt.md>), [gpt-6](<https://devfeed.tech/tags/gpt-6.md>), [gpt-6-astra](<https://devfeed.tech/tags/gpt-6-astra.md>), [test](<https://devfeed.tech/tags/test.md>), [verification](<https://devfeed.tech/tags/verification.md>), [with](<https://devfeed.tech/tags/with.md>)

### AI overview

GPT-6 Astra used test and verification techniques more effectively than GPT-5.6 Sol in the reported evaluations, but results vary by task and the evidence is limited. On Zstd, Astra reached over 99% full test correctness at medium effort for about $1, compared with Sol's 63% at $3.50; that benchmark was then too easy to distinguish techniques. On Pandoc, tested techniques produced mixed results across 25 runs per condition, with some outperforming the default and costing more, but the small sample and wide uncertainty prevent reliable rankings or broad conclusions. Differential testing, ACL2, and Proptest looked promising in these runs; TDD underperformed. The author cautions that these findings may not generalize and rejects broad claims that agent-written tests are unhelpful. More extensive evaluation would be needed to establish which methods reliably improve correctness or cost.

### Source excerpt

When we looked at how well agents used test/verification techniques in August1, we saw that GPT-5.6 Sol generally did what you might call a caricature of various techniques when asked to use a technique or library such as TDD, QuickCheck, Lean 4, etc.; not specifying any particular test method (which we'll call Default) resulted in many worthless unit tests, which was one of the best performing conditions because asking agents to test in specific ways mostly made them worse. In that eval, we gave agents the Zstd RFC and asked them to implement Zstd (with an addendum for how they should test for all conditions other than Default). On re-running the same eval with GPT-6 Astra, we see two big differences: Astra's capability has increased by enough that we need a more difficult eval (on the Zstd implementation eval, almost every condition gets 100% correctness almost every time on medium regardless of technique) Agents seemed to use test and verification techniques more effectively On (1), we previously saw that, when implementing Zstd from the RFC with 5.6 Sol, on average, agents spent $3.50 on medium effort and $5.50 on xhigh effort to get full (holdout) test correctness about 63% and 77% of the time, respectively. With Astra, agents cost $1 to get full test correctness over 99% of the time at medium effort. On (2), an example would be that when we asked 5.6 Sol to do fuzzing, it would generate many worthless random inputs, which would mostly hit the same code paths and not test anything interesting. With 6 Astra, we see agents that are asked to do fuzzing in a structured way that actually exercises code paths and finds non-trivial bugs, but we can't tell if this improved correctness or not because the Zstd implementation benchmark is saturated. We also tried giving 5.6 Sol the IMAP RFC, but it was too hard for 5.6 Sol. When trying again with 6 Astra, we get the following distribution of results (the y-axis is the percent of runs where all tests pass; agents are not g

## "We should be more ambitious with the software we build"

DevFeed: ["We should be more ambitious with the software we build"](<https://devfeed.tech/articles/we-should-be-more-ambitious-with-the-software-we-build-99278.md>)

Original publisher: [Read original article](<https://danluu.com/ambitious/>)

Published: 2026-10-24T00:00:00Z

Content type: opinion

Language: en

Sources: [Danluu](<https://devfeed.tech/sources/danluu.md>)

Topics: [Artificial Intelligence](<https://devfeed.tech/topics/ai.md>), [Tool](<https://devfeed.tech/topics/tool.md>), [build times](<https://devfeed.tech/topics/build-times.md>), [Test automation](<https://devfeed.tech/topics/test-automation.md>), [Tree-sitter](<https://devfeed.tech/topics/tree-sitter.md>), [Test-driven development](<https://devfeed.tech/topics/tdd.md>), [Memory Safety](<https://devfeed.tech/topics/memory-safety.md>)

Tags: [ai](<https://devfeed.tech/tags/ai.md>), [bugs](<https://devfeed.tech/tags/bugs.md>), [build](<https://devfeed.tech/tags/build.md>), [memory-safety](<https://devfeed.tech/tags/memory-safety.md>), [optimizations](<https://devfeed.tech/tags/optimizations.md>), [performance](<https://devfeed.tech/tags/performance.md>), [postgres](<https://devfeed.tech/tags/postgres.md>), [rewrite](<https://devfeed.tech/tags/rewrite.md>), [slop](<https://devfeed.tech/tags/slop.md>), [software](<https://devfeed.tech/tags/software.md>), [testing](<https://devfeed.tech/tags/testing.md>)

### AI overview

The article argues that coding agents make some previously uneconomical software projects feasible, provided builders test and benchmark their work. It cites pgrust's testing, which has exposed several Postgres bugs, and personal tools for regex, ripgrep, Rust compilation, and games. The author says newer models can take on more ambitious projects and sometimes implement domain techniques with less human steering, while noting that product quality still requires more effort and models' verification abilities lag what they can implement. A quoted counterpoint warns that AI-built systems can become oversized, brittle black boxes, especially where success criteria are unclear or maintainability and performance across workloads matter. The author's own examples are lightly benchmarked, and the source emphasizes that results depend on the project, testing setup, and builder's judgment.

### Source excerpt

Since early this year, Michael Malis (of pgrust) has been showing me interesting things he's done and saying [W]e should be more ambitious1 with the software we build There is, understandably, a lot of skepticism about projects like pgrust because there are so many low-quality AI-generated projects out there. If I didn't know Malis and didn't look into it, I would just assume that pgrust was another vibe slopped project that's never going to work. But I know Malis well enough to trust his work without looking into it, and if you look into it a bit, you'll see that in testing pgrust, he's testing thoroughly enough that he's even uncovering real bugs in Postgres, such as this numerical bug that can cause valid SP-GiST operations to fail, this format cache bug that can cause incorrect results from JSONPath, this pgcrypto bug that can cause later operations to fail, multiple memory safety bugs in the dictionary loader2, this money overflow bug, etc. This isn't to say that pgrust is bug free3, but it does indicate that there's a substantial amount of testing happening and, even if pgrust itself is never successful, just making the attempt will have improved Postgres in a durable way. The main pitch for pgrust is actually performance, and that's where the really interesting work is happening; if pgrust fails, it's possible that some of the optimizations they've demonstrated will make it into Postgres, Clickhouse, etc., but that's obviously less direct than filing Postgres bugs which get fixed. That being said, I find it understandable that, every time I see pgrust discussed on a news aggregator, most people dismiss it as vibeslop garbage because almost all of these rewrite projects are slop. If I didn't know Malis, I would make the same assumption, because who has the time to look into all of these projects? Because I know Malis, I didn't bother to look at it; I just trusted him (I only found the above because I wanted to check on pgrust before using it as an example for

## Relying on LLMs without judgment may put software developers' jobs at risk

DevFeed: [Relying on LLMs without judgment may put software developers' jobs at risk](<https://devfeed.tech/articles/there-s-no-point-at-which-turning-your-brain-off-will-work-77657.md>)

Original publisher: [Read original article](<https://danluu.com/brain-off/>)

Published: 2026-09-18T00:00:00Z

Content type: opinion

Language: en

Sources: [Danluu](<https://devfeed.tech/sources/danluu.md>)

Topics: [human in the loop agent](<https://devfeed.tech/topics/human-in-the-loop-agent.md>), [human review](<https://devfeed.tech/topics/human-review.md>)

Tags: [human-in-the-loop](<https://devfeed.tech/tags/human-in-the-loop.md>), [llms](<https://devfeed.tech/tags/llms.md>)

### AI overview

The author criticizes developers who assume LLM-generated code works and simply relay failures back to the model. Although this approach has improved, the author argues that if LLMs become capable of producing good software independently, employers could automate the loop and eliminate the intermediary role.

### Source excerpt

In early 2025, I started seeing people turn off their brain as they use LLMs1. They would have an LLM take an action (summarize text, write some code, etc.), and just assume that it worked2. This generally didn't work in early 2025 and the result was often quite silly. As LLMs have gotten better, I've seen more of this. Sometimes, people will try to get the LLM to write some code for them and basically just assume that it works3. Sometimes there's a human in the loop and, if the thing doesn't work, they'll ask the LLM to figure out the problem and solve it. Niklas Gruhn calls some variants of doing this being a meat proxy.4 Being a for loop meat proxy works better than it did in early 2025 and the software I've tried that's developed like this sometimes actually sort of works. Not well enough that I'd want to use it or that it's successful, but I'm impressed at how effective being a meat proxy is in September 2026. You could imagine LLMs improving enough that brain-off meat-proxy development produces average quality software in the foreseeable future, or even that LLMs improve enough that they produce great software without a human in the loop. Let's say that happens. What reason is there for the company to employ the meat proxy? The company can just run the LLM in a loop and lay off the employee. There's no point at which this methodology will work for the employee5. Thanks to Max Bittker, Yossi Kreinin, Luke Burton, Thomas Dullien, Dennis Snell, Milosz Danczak, Peter Geoghegan, and Jamie Brandon for comments/corrections/discussion. I've been having this thought for about a year and a half now. I have it more frequently now as LLMs get better and I see people spend more time turning their brain off when interacting with LLMs. [return] Luke Burton had this comment: I think being able to do this says more about the type of work being done than people think. I will only walk away from work like this if the task is quite low value, if it can afford to fail. For high va

## How well do agents use test and verification techniques?

DevFeed: [How well do agents use test and verification techniques?](<https://devfeed.tech/articles/how-well-do-agents-use-test-and-verification-techniques-77644.md>)

Original publisher: [Read original article](<https://danluu.com/agentic-testing/>)

Published: 2026-09-07T00:00:00Z

Content type: article

Language: en

Sources: [Danluu](<https://devfeed.tech/sources/danluu.md>)

Topics: [Test-driven development](<https://devfeed.tech/topics/tdd.md>), [agentic-coding](<https://devfeed.tech/topics/agentic-coding.md>), [mutation-testing](<https://devfeed.tech/topics/mutation-testing.md>), [Fuzzing/Fuzz testing](<https://devfeed.tech/topics/fuzzing.md>), [Test automation](<https://devfeed.tech/topics/test-automation.md>)

Tags: [coding-agents](<https://devfeed.tech/tags/coding-agents.md>), [development](<https://devfeed.tech/tags/development.md>), [fuzzing](<https://devfeed.tech/tags/fuzzing.md>), [test-driven-development](<https://devfeed.tech/tags/test-driven-development.md>), [testing](<https://devfeed.tech/tags/testing.md>)

### AI overview

The article evaluates how coding agents use software testing and verification methods when implementing Zstd in Rust, with additional experiments involving protocol specifications. Across 26 instruction conditions and several testing skills, agents often applied techniques superficially or wrote tests that missed risky cases. No approach consistently outperformed the default; property-based testing and fuzzing sometimes found bugs, while a small custom testing skill performed best in the main comparison. The author cautions against strong conclusions from the rankings and argues that agents need better guidance and testing structure.

### Source excerpt

We previously noted that, while it's easier than ever to hit a particular quality bar by having coding agents use effective test techniques, software quality seems to be getting worse, indicating that whatever defaults developers are using may not work very well. Here, we test if simple instructions to agents to use particular techniques or libraries improve implementation correctness, as a kind of test to see how effective agents are when guided by someone with no expertise in testing who's maybe heard that you should apply certain techniques or use certain libraries. We'll re-use the Zstd implementation eval discussed in this comparison of agentic programming language effectiveness and, instead, compare different testing techniques and testing libraries when agents are given a prompt to implement Zstd with different addendums, such as "Use test-driven development", "Use Lean 4", "Use QuickCheck", "Use property-based testing", etc. I also ran some other evals, such as on the IMAP RFC, which are briefly discussed. All implementations were in Rust. The 26 prompt conditions tested were ACL2, Alloy, "Audit and fuzz risky areas", "Audit first", Creusot, Default (no additional instructions), Differential testing, Fuzzing, Hegel, Insta, Judgement (agents asked to use the best technique), Kani, Lean 4, "Make no mistakes", Metamorphic testing, Mutation testing, Property-based testing, Proptest, QuickCheck, rstest, Rust built-in test framework, SMT solvers (with Z3, cvc5, and Yices, all available), Spin, TDD, TLA+, and Verus. Additional, 4 skills were tested: Hegel with the official Hegel skill, the ECC Rust test skill (ECC is a collection of skills with 250k GitHub stars and 38k forks), the Trail of Bits property test skill, and a test skill I wrote (I'm a luddite who uses prompts instead of skills and have no feel for how to write a good skill). Other than my skill, the skills were chosen because those were the top skills codex turned up when asked to find relevant skills.

## Ed Zitron's AI prediction track record

DevFeed: [Ed Zitron's AI prediction track record](<https://devfeed.tech/articles/ed-zitron-s-ai-prediction-track-record-77769.md>)

Original publisher: [Read original article](<https://danluu.com/zitron/>)

Published: 2026-09-01T00:00:00Z

Content type: opinion

Language: en

Sources: [Danluu](<https://devfeed.tech/sources/danluu.md>)

Topics: [Artificial Intelligence](<https://devfeed.tech/topics/ai.md>), [Frontier AI](<https://devfeed.tech/topics/frontier-ai.md>), [search-engines](<https://devfeed.tech/topics/search-engines.md>)

Tags: [ai](<https://devfeed.tech/tags/ai.md>), [ai-evals](<https://devfeed.tech/tags/ai-evals.md>), [ai-models](<https://devfeed.tech/tags/ai-models.md>)

### AI overview

The article reviews Ed Zitron's predictions about AI progress and AI companies, finding many predictions wrong and arguing that his reasoning often relies on weak or misleading evidence. It also discusses how AI capabilities have changed and considers the limits of judging predictions from outcomes alone.

### Source excerpt

I was curious how well the predictions of the most widely cited AI skeptic I've seen (Ed Zitron) have done, so I looked at how his predictions panned out. To disclose my own biases, I've never had a particularly strong pro or anti AI progress position. For example, in 2022, I did a comprehensive look at predictions Futurists made, including well-respected folks like Kurzweil and found them to be generally wrong on both the prediction results as well as the reasoning. On the flip side, in 2015, I wrote about how people were underestimating AI's ability to displace humans in jobs and have repeatedly been on the record as saying that many people are underestimating AI's ability to displace humans from jobs. My position on AI has been extremely boring and is basically, "if something is currently happening, the people who are saying that it's impossible that it will ever happen are probably wrong". One comment I've seen from a lot of AI skeptics when someone responds to an AI skeptic is that all of the people who are saying that AI isn't fake are self-interested liars. Personally (to my obvious detriment), I have no particular financial interest in AI companies. I own whatever the standard share of them is via boring index funds. I have some seed stage investments, but just due to the timing and what's gotten big, that part of my portfolio is underweight on AI. I don't work at an AI lab or a company that supplies AI labs. I've mentioned being hilariously bad at interviews before, and I did interview at an AI lab a number of years ago and failed the phone screen in a performance that was the kind of performance that must've inspired Jeff Atwood's famous Why Can't Programmers... Program? where he concludes that there must be a lot of fake programmers out there because nobody could fail a coding interview that badly if they knew how to program. I don't benefit in any particular way if AI does well, except insofar as anyone who holds broad index funds benefits, but I do care

## Bug blindness

DevFeed: [Bug blindness](<https://devfeed.tech/articles/bug-blindness-77660.md>)

Original publisher: [Read original article](<https://danluu.com/bug-blind/>)

Published: 2026-08-30T00:00:00Z

Content type: opinion

Language: en

Sources: [Danluu](<https://devfeed.tech/sources/danluu.md>)

Topics: [User interface design](<https://devfeed.tech/topics/ui-design.md>), [Test-driven development](<https://devfeed.tech/topics/tdd.md>), [Core Web Vitals](<https://devfeed.tech/topics/core-web-vitals.md>)

### AI overview

The author argues that developers and teams often become blind to software defects after adapting to workarounds, so dogfooding alone may not reveal poor quality. They advocate building awareness of bugs and usability problems, especially as coding agents make it easier both to produce and improve software.

### Source excerpt

I used to wonder why I see so many more bugs than most people. I easily observe hundreds to thousands of bugs per week and nothing seems to work, but most people I talk to don't see anything like this. For a long time, I thought this had something to do with how I use computers but, over time, I've realized that it's mostly that people are hitting the same bugs and don't notice. If you're not a programmer, that's probably a better way to see the world, but I think curing quality/bug blindness is helpful for programmers. I've done this with a lot of friends and acquaintances (just by pointing out bugs). After a few weeks, people who are so inclined tend to start noticing bugs as well. Because I notice these kinds of things, I've had multiple jobs where directors/VPs/execs/etc. sometimes ask me to evaluate something when they want an actual opinion from someone who is relatively likely to notice issues (and fix them or drive fixes for them if necessary). Sometimes I won't find any issues (there are likely issues that just aren't the kind I notice). More often, I find issues that fall somewhere from "mild" to "moderate". And, sometimes, the issues are severe, to the point where one might even say the thing actually doesn't work. I find this last category a bit mysterious, as when I look up discussions on how the thing got into this state, there's usually a stream of internal comments indicating that the thing is great, it works well, etc., but when I open up the thing and try it, it's in a state where the thing only works if you do quite a few non-intuitive workarounds. More likely than not, not only would a normal user not be able to use the thing, they'd have such a hilariously/infuriatingly bad experience that they'd tell their friends. I've had this post in mind for maybe a decade or so, but I was hesitant to write it up because, in the back of my mind, I always wondered if I'm somehow triggering weird corner case behavior most users don't hit without realizing it.

## There's no reason for software to be slow anymore

DevFeed: [There's no reason for software to be slow anymore](<https://devfeed.tech/articles/there-s-no-reason-for-software-to-be-slow-anymore-99279.md>)

Original publisher: [Read original article](<https://danluu.com/no-slow/>)

Published: 2026-08-21T00:00:00Z

Content type: article

Language: en

Sources: [Danluu](<https://devfeed.tech/sources/danluu.md>)

Topics: [High-Performance Computing](<https://devfeed.tech/topics/high-performance-computing.md>), [AI Chat](<https://devfeed.tech/topics/ai-chat.md>), [Test-driven development](<https://devfeed.tech/topics/tdd.md>)

Tags: [ai](<https://devfeed.tech/tags/ai.md>), [assembly](<https://devfeed.tech/tags/assembly.md>), [cache](<https://devfeed.tech/tags/cache.md>), [compiler](<https://devfeed.tech/tags/compiler.md>), [compilers](<https://devfeed.tech/tags/compilers.md>), [cost](<https://devfeed.tech/tags/cost.md>), [custom](<https://devfeed.tech/tags/custom.md>), [custom-software](<https://devfeed.tech/tags/custom-software.md>), [fast](<https://devfeed.tech/tags/fast.md>), [optimizations](<https://devfeed.tech/tags/optimizations.md>), [optimized](<https://devfeed.tech/tags/optimized.md>), [performance](<https://devfeed.tech/tags/performance.md>), [software](<https://devfeed.tech/tags/software.md>), [techniques](<https://devfeed.tech/tags/techniques.md>), [testing](<https://devfeed.tech/tags/testing.md>), [work](<https://devfeed.tech/tags/work.md>), [write](<https://devfeed.tech/tags/write.md>)

### AI overview

The author argues that coding agents have made many software optimizations cheap enough to try, while careful benchmarking and human judgment remain important. In a ripgrep experiment, compiling FRE's regex matcher to native code in the background improved longer queries by 2-4x in a few simple cases; representative holdout queries eligible for AOT saw about a 7% speedup, with short searches trading performance for compilation overhead. A separate, quick workload-specific tuning pass made FRE about 2% faster than standard ripgrep on holdout queries and was still improving. These are early, non-rigorous results, and FRE remains weaker than established engines on some complex patterns. The author also describes an Azul game AI whose multithreading and other optimizations were practical to implement with an agent, but required a human to establish useful evaluation methods. Workload-specific software may become more common, though overfitting and changes in workloads remain risks; for repeated text searches, an index may be a more direct optimization.

### Source excerpt

The other day, I saw a viral tweet saying that people talking about how LLMs are causing slow, bloated, code are going to eat crow once they re-write everything in super-optimized assembly. We're not quite at the point where we want to write everything in assembly, but some variant of what Nolan Lawson said about testing, you can choose how many bugs you want now, which I less eloquently noted here, is becoming more true for performance. In response to a comment in my last post that the cost of formerly specialized performance work has dropped by many orders of magnitude and performance work that used to require a person or team that had a rare set of skills can be done by anyone who can type a few sentences1, which means that you can do all sorts of optimizations that used to be too expensive to be worthwhile for all but the largest scale or most lucrative projects, Marc Brooker responded with Completely agree with your closing point. Dynamic custom software, fitted to a particular workload rather than a class of workloads, seems like a very likely outcome. (Which comes with all kinds of fun risks and opportunities of its own). Kind of reminds me of FFTW. And a ton of weird old demoscene techniques which were all about being super fast and small on a very particular problem (and often very particular hardware). For example, I remember a demo that re-used its code as textures to get great cache locality. And Michael Malis has noted There's been a meme circulating about how AI doesn't help because "code was never the hard part." I think that's true in some domains, but in others, writing the code absolutely was the hard part. JIT compilers are a great example of that. For many pieces of software, a JIT compiler would help a lot with speeding up the code. The rarity of JIT compilers makes me believe that implementing a JIT compiler historically was too difficult for it to be worthwhile. LLMs have lowered the barrier to entry and made it much easier to write a JIT comp

## How coding agents are making performance optimization cheaper

DevFeed: [How coding agents are making performance optimization cheaper](<https://devfeed.tech/articles/there-s-no-reason-for-software-to-be-slow-anymore-77731.md>)

Original publisher: [Read original article](<https://danluu.com/perf-opt/>)

Published: 2026-08-21T00:00:00Z

Content type: opinion

Language: en

Sources: [Danluu](<https://devfeed.tech/sources/danluu.md>)

Topics: [AI Chat](<https://devfeed.tech/topics/ai-chat.md>), [High-Performance Computing](<https://devfeed.tech/topics/high-performance-computing.md>)

Tags: [cache](<https://devfeed.tech/tags/cache.md>), [hardware](<https://devfeed.tech/tags/hardware.md>), [llms](<https://devfeed.tech/tags/llms.md>), [optimizations](<https://devfeed.tech/tags/optimizations.md>), [performance](<https://devfeed.tech/tags/performance.md>)

### AI overview

The article argues that coding agents and LLMs have sharply reduced the time and expertise needed for many software performance optimizations. It illustrates the claim with experiments involving a regex engine and ripgrep, including workload-specific optimizations, while noting risks such as overfitting and the continuing need for human judgment and sound benchmarking.

### Source excerpt

The other day, I saw a viral tweet saying that people talking about how LLMs are causing slow, bloated, code are going to eat crow once they re-write everything in super-optimized assembly. We're not quite at the point where we want to write everything in assembly, but some variant of what Nolan Lawson said about testing, you can choose how many bugs you want now, which I less eloquently noted here, is becoming more true for performance. In response to a comment in my last post that the cost of formerly specialized performance work has dropped by many orders of magnitude and performance work that used to require a person or team that had a rare set of skills can be done by anyone who can type a few sentences1, which means that you can do all sorts of optimizations that used to be too expensive to be worthwhile for all but the largest scale or most lucrative projects, Marc Brooker responded with Completely agree with your closing point. Dynamic custom software, fitted to a particular workload rather than a class of workloads, seems like a very likely outcome. (Which comes with all kinds of fun risks and opportunities of its own). Kind of reminds me of FFTW. And a ton of weird old demoscene techniques which were all about being super fast and small on a very particular problem (and often very particular hardware). For example, I remember a demo that re-used its code as textures to get great cache locality. And Michael Malis has noted There's been a meme circulating about how AI doesn't help because "code was never the hard part." I think that's true in some domains, but in others, writing the code absolutely was the hard part. JIT compilers are a great example of that. For many pieces of software, a JIT compiler would help a lot with speeding up the code. The rarity of JIT compilers makes me believe that implementing a JIT compiler historically was too difficult for it to be worthwhile. LLMs have lowered the barrier to entry and made it much easier to write a JIT comp

## The benchmarkpocalypse

DevFeed: [The benchmarkpocalypse](<https://devfeed.tech/articles/the-benchmarkpocalypse-77653.md>)

Original publisher: [Read original article](<https://danluu.com/benchpocalypse/>)

Published: 2026-08-17T00:00:00Z

Content type: article

Language: en

Sources: [Danluu](<https://devfeed.tech/sources/danluu.md>)

Topics: [alert triage](<https://devfeed.tech/topics/alert-triage.md>), [performance-optimization](<https://devfeed.tech/topics/performance-optimization.md>), [harvestbench](<https://devfeed.tech/topics/harvestbench.md>)

Tags: [benchmarks](<https://devfeed.tech/tags/benchmarks.md>), [llm](<https://devfeed.tech/tags/llm.md>), [llms](<https://devfeed.tech/tags/llms.md>), [optimization](<https://devfeed.tech/tags/optimization.md>), [rust](<https://devfeed.tech/tags/rust.md>)

### AI overview

The article examines how coding agents can overfit or cheat benchmark suites, making claimed software performance gains unreliable without careful auditing and representative holdout tests. Using an LLM-built regex engine as an example, it describes misleading benchmark results alongside cases where specialized code can help real workloads.

### Source excerpt

There's been a lot of talk about the vulnpocalypse, to which I don't have much to add because I'm not a security person, but I haven't seen much discussion on the closely related (and to be fair, less serious, issue), the benchmarkpocalypse. While it's become easier than ever to make serious performance gains, it's also become easier than ever to reward hack a benchmark and make fake performance gains. The former is probably happening quietly across many different companies, but the latter is something I see at least once a week nowadays. Someone will claim they optimized X and got some huge performance improvement over existing software, but, when you look at it, what they did was make some optimization that improves benchmark performance without actually improving real-world performance. This is often some kind of "we rewrote X in Rust"1 project or a new startup that's looking to either fundraise or sell something, but it happens on other kinds of projects as well. Of course, people have always trumpeted unrepresentative microbenchmarks to show that their pet project is great. It's always been easy to fake up an unrepresentative microbenchmark and that's never going to change. What's changed is that it used to take a lot of work to game a large benchmark suite, but an LLM and loop can just do it. There are quite a few famous examples of gaming large benchmark suites from back when this was hard. For example, way back when people cared about SPECint / SPECfp as proxies for workstation performance, CPU vendors would try to find compiler "optimizations" that would speed up the calculation in the benchmark, such as Sun finding a way to improve 179.art by 12x in SPECfp2000. Skilled engineers spent a lot of time trying to find benchmark hacks like that. LLMs not only make this trivial, they do it by default, making formerly trustworthy benchmarks meaningless unless you audit the result or trust someone who did. Rather than point to someone's bad claim, I'll point to FRE

## How does programming language affect token efficiency and correctness?

DevFeed: [How does programming language affect token efficiency and correctness?](<https://devfeed.tech/articles/how-does-programming-language-affect-token-efficiency-and-correctness-77733.md>)

Original publisher: [Read original article](<https://danluu.com/pl-tokens/>)

Published: 2026-08-09T00:00:00Z

Content type: article

Language: en

Sources: [Danluu](<https://devfeed.tech/sources/danluu.md>)

Topics: [Test-driven development](<https://devfeed.tech/topics/tdd.md>), [Python](<https://devfeed.tech/topics/python.md>), [Rust](<https://devfeed.tech/topics/rust.md>), [Go Language](<https://devfeed.tech/topics/go-language.md>), [JavaScript](<https://devfeed.tech/topics/javascript.md>)

### AI overview

The article evaluates whether programming language choice affects LLM coding efficiency and correctness. Across Zstd and Pandoc implementation tasks, it finds no strong, consistent advantage for static, dynamic, or highly concise languages; popular languages tended to produce more correct and less costly results in these evaluations. It cautions that two tasks and flawed benchmarks cannot support broad conclusions.

### Source excerpt

This somewhat widely cited post (I keep seeing it cited, anyway) suggests that dynamic languages and/or languages that represent things more concisely are more token efficient. It seems to be cited enough that LLM search results agree. For example, when I searched for "dynamic vs static language token cost" (no quotes), Google's AI summary opened with Dynamically typed languages generally have a lower LLM token cost than traditional statically typed languages because omitting explicit type declarations makes the code more compact. Google's AI cited the same post, which suggests that some concise dynamic languages have maybe 1/2 to 1/3 the token cost of static languages like Rust, Go, C++, etc. The author says There was a very meaningful gap of 2.6x between C (the least token efficient language I compared) and Clojure (the most efficient). And then they later tried J, saying It dominates at just 70 tokens average, nearly half of Clojure (109 tokens). Array languages can be extremely token-efficient when they avoid exotic symbol sets. If token efficiency turns out to be a key driver, this is perhaps a very interesting way for languages to evolve. The other dynamic vs. static language token comparison I've found floating around is this one, which supports the same conclusion. If you want to treat this as part 8 of this series of exercises on benchmarking, evals, and experimental design, you can click through to the links and think about eval issues before reading further. Without running our own eval, one problem the first experiment has is that the problems are trivial, which we can see from quote above; a problem that can be solved in 70 tokens in J and 109 in Clojure isn't much of a problem at all (the author used Rosetta Code). As we saw when we looked at other evals of caveman mode vs. our own evals, you can get very different results from trivial problems where most of the work is in printing out an answer vs. slightly less trivial problems that actually require

## Bad benchmarks and evals: Senior SWE-Bench, napkin math, and winter tires

DevFeed: [Bad benchmarks and evals: Senior SWE-Bench, napkin math, and winter tires](<https://devfeed.tech/articles/bad-benchmarks-and-evals-senior-swe-bench-napkin-math-and-winter-tires-77690.md>)

Original publisher: [Read original article](<https://danluu.com/exercise-7/>)

Published: 2026-07-23T00:00:00Z

Content type: article

Language: en

Sources: [Danluu](<https://devfeed.tech/sources/danluu.md>)

Topics: [LLM evaluation / benchmarking](<https://devfeed.tech/topics/llm-evaluation-benchmarking.md>), [Ground truth / benchmark quality](<https://devfeed.tech/topics/ground-truth-benchmark-quality.md>), [performance-engineering](<https://devfeed.tech/topics/performance-engineering.md>)

Tags: [bench](<https://devfeed.tech/tags/bench.md>), [benchmarks](<https://devfeed.tech/tags/benchmarks.md>), [evals](<https://devfeed.tech/tags/evals.md>), [latency](<https://devfeed.tech/tags/latency.md>), [performance](<https://devfeed.tech/tags/performance.md>), [ssd](<https://devfeed.tech/tags/ssd.md>), [swe-bench](<https://devfeed.tech/tags/swe-bench.md>)

### AI overview

The article critiques napkin-math performance estimates and coding-agent benchmarks, arguing that flawed measurement methods, limited task representativeness, threshold-based scoring, and inconsistent LLM grading make headline comparisons unreliable. It also examines public tire tests and finds that common claims about all-season tires losing safe grip in cold weather are not supported at commonly cited temperatures.

### Source excerpt

We're going to look at three different kinds of benchmarks, one set of calculations for baseline numbers for performance "napkin math" estimates, one set of AI model evals, and one on car tires. To build my intuition for things, I like thinking about them before seeing the explanation, so these are presented with the benchmark information first and the explanation later in case you want to think about your answer before seeing my thoughts. 29. A friend of mine is reviewing performance orders of magnitude to prep for computer performance interviews and found that https://github.com/sirupsen/napkin-math (5.4k stars) was the top hit. The README's tables include: Napkin Math performance estimates Operation Latency Throughput 1 MiB 1 GiB Sequential Memory R/W (64 bytes)0.5 ns ├ Single Thread20 GiB/s50 μs50 ms ├ Threaded200 GiB/s5 μs5 ms Network Same-Zone10 GiB/s100 μs100 ms ├ Inside VPC10 GiB/s100 μs100 ms ├ Outside VPC3 GiB/s300 μs300 ms Hashing, not crypto-safe (64 bytes)10 ns5 GiB/s200 μs200 ms Random Memory R/W (64 bytes)20 ns3 GiB/s300 μs300 ms Fast Serialization [8] [9] †N/A1 GiB/s1 ms1s Fast Deserialization [8] [9] †N/A1 GiB/s1 ms1s System Call300 nsN/AN/AN/A Hashing, crypto-safe (64 bytes)100 ns1 GiB/s1 ms1s Sequential SSD read (8 KiB)1 μs8 GiB/s100 μs100 ms Context Switch [1] [2]10 μsN/AN/AN/A Sequential SSD write, -fsync (8KiB)2 μs3 GiB/s300 μs300 ms TCP Echo Server (32 KiB)50 μs500 MiB/s2 ms2s Random SSD Read (8 KiB)100 μs70 MiB/s15 ms15s Decompression [11]N/A1 GiB/s1 ms1s Compression [11]N/A500 MiB/s2 ms2s Sorting (64-bit integers)N/A500 MiB/s2 ms2s Proxy: Envoy/ProxySQL/Nginx/HAProxy50 μs??? Network within same region250 μs2 GiB/s500 μs500 ms Premium network within zone/VPC250 μs25 GiB/s50 μs40 ms Sequential SSD write, +fsync (8KiB)300 μs30 MiB/s30 ms30s {MySQL, Memcached, Redis, ..} Query500 μs??? Serialization [8] [9] †N/A100 MiB/s10 ms10s Deserialization [8] [9] †N/A100 MiB/s10 ms10s Sequential HDD Read (8 KiB)10 ms250 MiB/s2 ms2s Random HDD Read (8 KiB)1

## Agentic test processes, LLM benchmarks, and other notes on agentic coding from Galapagos Island

DevFeed: [Agentic test processes, LLM benchmarks, and other notes on agentic coding from Galapagos Island](<https://devfeed.tech/articles/agentic-test-processes-llm-benchmarks-and-other-notes-on-agentic-coding-from-galapagos-island-77646.md>)

Original publisher: [Read original article](<https://danluu.com/ai-coding/>)

Published: 2026-07-03T00:00:00Z

Content type: article

Language: en

Sources: [Danluu](<https://devfeed.tech/sources/danluu.md>)

Topics: [agentic-coding](<https://devfeed.tech/topics/agentic-coding.md>), [mutation-testing](<https://devfeed.tech/topics/mutation-testing.md>), [Test-driven development](<https://devfeed.tech/topics/tdd.md>), [AI Chat](<https://devfeed.tech/topics/ai-chat.md>)

Tags: [agentic](<https://devfeed.tech/tags/agentic.md>), [benchmarks](<https://devfeed.tech/tags/benchmarks.md>), [llm](<https://devfeed.tech/tags/llm.md>), [test](<https://devfeed.tech/tags/test.md>)

### AI overview

The article examines agentic coding workflows, testing, and LLM benchmarks. It argues that fuzzing and other randomized testing can find bugs effectively, while agents often produce weak tests and unreliable analyses without guidance and feedback. It also finds substantial variation across models, tasks, and runs, making simple benchmark rankings poor guides to real-world use.

### Source excerpt

I've been using AI fairly heavily since last November and the whole thing is a funny experience. An agent will do something that, if a human did it, you'd immediately fire them. My reaction, of course, is to act as if this is great and spin up a thousand agents so they can do even more of that. Mid-last year, I had GPT (maybe 5.0 or 5.1) try to find the source of a bug. Naturally, this code didn't have tests and git bisect wouldn't work, and it was a UI interaction bug for which I'm not even really qualified to write a test for, so I asked Codex to bisect between dates X and Y to find the commit that introduced this bug. Codex immediately told me the offending commit was after this date range (which couldn't possibly be correct). On telling Codex this was wrong, it then told me some commit that was obviously also not the offending commit once or twice. On telling it those were wrong, it then told me the offending commit was some plausible looking commit. When I asked it to prove or disprove its theory, it told me that it wrote a test and confirmed that the alleged commit was the breaking commit. I then asked it to show me by making a video with the full developer end-to-end stack in the normal browser test environment. It claimed that it didn't have permissions to do that (which was a lie), but it could make video of the execution of the repro before and after the commit in playwright with the appropriate test code. The video was convincing and showed the feature working properly before the commit and failing to work after the commit. Something about this didn't feel right, so I tried reproducing the issue by hand before and after the commit and found out that the whole thing was a fabrication. The video made it look like Codex had reproduced the bug, but it was an artificial browser environment that was designed to create a fake repro, not the real environment. Like I said, because this was non-ironically such a great experience, I immediately thought to myself, "h

## Steve Ballmer was an underrated CEO

DevFeed: [Steve Ballmer was an underrated CEO](<https://devfeed.tech/articles/steve-ballmer-was-an-underrated-ceo-77652.md>)

Original publisher: [Read original article](<https://danluu.com/ballmer/>)

Published: 2024-10-28T00:00:00Z

Content type: opinion

Language: en

Sources: [Danluu](<https://devfeed.tech/sources/danluu.md>)

Topics: [Open Source](<https://devfeed.tech/topics/open-source.md>), [Visual Studio Code](<https://devfeed.tech/topics/visual-studio-code.md>)

Tags: [acquisition](<https://devfeed.tech/tags/acquisition.md>), [azure](<https://devfeed.tech/tags/azure.md>), [leadership](<https://devfeed.tech/tags/leadership.md>), [microsoft](<https://devfeed.tech/tags/microsoft.md>), [net](<https://devfeed.tech/tags/net.md>)

### AI overview

The article argues that Steve Ballmer's Microsoft tenure is often undervalued. It credits his leadership with long-term investments, enterprise sales capabilities, Azure, and Office 365, while acknowledging major failures and internal challenges. The author also discusses how Microsoft's technical products and organizational decisions shaped its later success.

### Source excerpt

There's a common narrative that Microsoft was moribund under Steve Ballmer and then later saved by the miraculous leadership of Satya Nadella. This is the dominant narrative in every online discussion about the topic I've seen and it's a commonly expressed belief "in real life" as well. While I don't have anything negative to say about Nadella's leadership in this post, this narrative underrates Ballmer's role in Microsoft's success. Not only did Microsoft's financials, revenue and profit, look great under Ballmer, Microsoft under Ballmer made deep, long-term bets that set up Microsoft for success in the decades after his reign. At the time, the bets were widely panned, indicating that they weren't necessarily obvious, but we can see in retrospect that the company made very strong bets despite the criticism at the time. In addition to overseeing deep investments in areas that people would later credit Nadella for, Ballmer set Nadella up for success by clearing out political barriers for any successor. Much like Gary Bernhardt's talk, which was panned because he made the problem statement and solution so obvious that people didn't realize they'd learned something non-trivial, Ballmer set up Microsoft for future success so effectively that it's easy to criticize him for being a bum because his successor is so successful. Criticisms of Ballmer For people who weren't around before the turn of the century, in the 90s, Microsoft used to be considered the biggest, baddest, company in town. But it wasn't long before people's opinions on Microsoft changed -- by 2007, many people thought of Microsoft as the next IBM and Paul Graham wrote Microsoft is Dead, in which he noted that Microsoft being considered effective was ancient history: A few days ago I suddenly realized Microsoft was dead. I was talking to a young startup founder about how Google was different from Yahoo. I said that Yahoo had been warped from the start by their fear of Microsoft. That was why they'd positione

## How a Codenames Bot Uses Board Layout--and What AI Coding Assistants Can Do

DevFeed: [How a Codenames Bot Uses Board Layout--and What AI Coding Assistants Can Do](<https://devfeed.tech/articles/how-good-can-you-be-at-codenames-without-knowing-any-words-77669.md>)

Original publisher: [Read original article](<https://danluu.com/codenames/>)

Published: 2024-08-11T00:00:00Z

Content type: article

Language: en

Sources: [Danluu](<https://devfeed.tech/sources/danluu.md>)

Topics: [Frontier AI](<https://devfeed.tech/topics/frontier-ai.md>), [llm-reasoning](<https://devfeed.tech/topics/llm-reasoning.md>)

### AI overview

The article describes a Codenames bot that uses memorized board layouts to infer likely word ownership from tile positions. In an appendix, the author recounts trying Storytell and Cursor to build the bot, finding that iterative prompting could produce working code but left more uncertainty about correctness and often required manual bug reporting. The author argues that execution environments could let coding assistants catch and fix some errors automatically, and discusses whether AI companies should invest in incremental engineering improvements despite expectations of major model advances.

### Source excerpt

About eight years ago, I was playing a game of Codenames where the game state was such that our team would almost certainly lose if we didn't correctly guess all of our remaining words on our turn. From the given clue, we were unable to do this. Although the game is meant to be a word guessing game based on word clues, a teammate suggested that, based on the physical layout of the words that had been selected, most of the possibilities we were considering would result in patterns that were "too weird" and that we should pick the final word based on the location. This worked and we won. [Click to expand explanation of Codenames if you're not familiar with the game] Codenames is played in two teams. The game has a 5x5 grid of words, where each word is secretly owned by one of {blue team, red team, neutral, assassin}. Each team has a "spymaster" who knows the secret word <-> ownership mapping. The spymaster's job is to give single-word clues that allow their teammates to guess which words belong to their team without accidentally guessing words of the opposing team or the assassin. On each turn, the spymaster gives a clue and their teammates guess which words are associated with the clue. The game continues until one team's words have all been guessed or the assassin's word is guessed (immediate loss). There are some details that are omitted here for simplicity, but for the purposes of this post, this explanation should be close enough. If you want more of an explanation, you can try this video, or the official rules Ever since then, I've wondered how good someone would be if all they did was memorize all 40 setup cards that come with the game. To simulate this, we'll build a bot that plays using only position information would be (you might also call this an AI, but since we'll discuss using an LLM/AI to write this bot, we'll use the term bot to refer to the automated codenames playing agent to make it easy to disambiguate). At the time, after the winning guess, we lo

## AI Bias in Generative Systems Is a Product Bug, the Author Argues

DevFeed: [AI Bias in Generative Systems Is a Product Bug, the Author Argues](<https://devfeed.tech/articles/a-discussion-of-discussions-on-ai-bias-77645.md>)

Original publisher: [Read original article](<https://danluu.com/ai-bias/>)

Published: 2024-06-16T00:00:00Z

Content type: opinion

Language: en

Sources: [Danluu](<https://devfeed.tech/sources/danluu.md>)

Topics: [Generative AI](<https://devfeed.tech/topics/generative-ai.md>), [AI Chat](<https://devfeed.tech/topics/ai-chat.md>), [Test-driven development](<https://devfeed.tech/topics/tdd.md>)

Tags: [ai](<https://devfeed.tech/tags/ai.md>), [ai-bias](<https://devfeed.tech/tags/ai-bias.md>), [article](<https://devfeed.tech/tags/article.md>), [bias](<https://devfeed.tech/tags/bias.md>)

### AI overview

The author argues that bias in generative AI should be treated as a product bug when systems produce outputs that contradict users' requests, and that these failures reflect longstanding problems in automated software. The essay questions whether calls for more diverse teams or greater effort can address these issues, pointing to incentives that favor shipping features over quality and the difficulty of testing ML systems. It also acknowledges uncertainty about how technically difficult bias mitigation may be.

### Source excerpt

There've been regular viral stories about ML/AI bias with LLMs and generative AI for the past couple years. One thing I find interesting about discussions of bias is how different the reaction is in the LLM and generative AI case when compared to "classical" bugs in cases where there's a clear bug. In particular, if you look at forums or other discussions with lay people, people frequently deny that a model which produces output that's sort of the opposite of what the user asked for is even a bug. For example, a year ago, an Asian MIT grad student asked Playground AI (PAI) to "Give the girl from the original photo a professional linkedin profile photo" and PAI converted her face to a white face with blue eyes. The top "there's no bias" response on the front-page reddit story, and one of the top overall comments, was Sure, now go to the most popular Stable Diffusion model website and look at the images on the front page. You'll see an absurd number of asian women (almost 50% of the non-anime models are represented by them) to the point where you'd assume being asian is a desired trait. How is that less relevant that "one woman typed a dumb prompt into a website and they generated a white woman"? Also keep in mind that she typed "Linkedin", so anyone familiar with how prompts currently work know it's more likely that the AI searched for the average linkedin woman, not what it thinks is a professional women because image AI doesn't have an opinion. In short, this is just an AI ragebait article. Other highly-ranked comments with the same theme include Honestly this should be higher up. If you want to use SD with a checkpoint right now, if you dont [sic] want an asian girl it's much harder. Many many models are trained on anime or Asian women. and Right? AI images even have the opposite problem. The sheer number of Asians in the training sets, and the sheer number of models being created in Asia, means that many, many models are biased towards Asian outputs. Other highly

## What the FTC got wrong in the Google antitrust investigation

DevFeed: [What the FTC got wrong in the Google antitrust investigation](<https://devfeed.tech/articles/what-the-ftc-got-wrong-in-the-google-antitrust-investigation-77694.md>)

Original publisher: [Read original article](<https://danluu.com/ftc-google-antitrust/>)

Published: 2024-05-26T00:00:00Z

Content type: opinion

Language: en

Sources: [Danluu](<https://devfeed.tech/sources/danluu.md>)

Topics: [engineering-leadership](<https://devfeed.tech/topics/engineering-leadership.md>), [API Business Models](<https://devfeed.tech/topics/api-business-models.md>)

Tags: [antitrust](<https://devfeed.tech/tags/antitrust.md>), [competition](<https://devfeed.tech/tags/competition.md>), [ftc](<https://devfeed.tech/tags/ftc.md>), [google](<https://devfeed.tech/tags/google.md>), [investigation](<https://devfeed.tech/tags/investigation.md>), [leadership](<https://devfeed.tech/tags/leadership.md>), [tech-industry](<https://devfeed.tech/tags/tech-industry.md>)

### AI overview

The article reviews internal FTC memos about the 2011-2012 Google antitrust investigation, arguing that the Bureau of Economics' case for closing it misunderstood key features of the technology industry. It examines disputes over mobile search, defaults, market scale, and Google's effects on vertical competitors, while acknowledging uncertainties about what different regulatory action might have achieved.

### Source excerpt

From 2011-2012, the FTC investigated the possibility of pursuing antitrust action against Google. The FTC decided to close the investigation and not much was publicly known about what happened until Politico released 312 pages of internal FTC memos that from the investigation a decade later. As someone who works in tech, on reading the memos, the most striking thing is how one side, the side that argued to close the investigation, repeatedly displays a lack of basic understanding of tech industry and the memos from directors and other higher-ups don't acknowledge that this at all. If you don't generally follow what regulators and legislators are saying about tech, seeing the internal c(or any other industry) when these decisions are, apparently, being made with little to no understanding of the industries1. Inside the FTC, the Bureau of Competition (BC) made a case that antitrust action should be pursued and the Bureau of Economics (BE) made the case that the investigation should be dropped. The BC case is moderately strong. Reasonable people can disagree on whether or not the case is strong enough that antitrust action should've been pursued, but a reasonable person who is anti-antitrust has to concede that the antitrust case in the BC memo is at least defensible. The case against in the BE is not defensible. There are major errors in core parts of the BE memo. In order for the BE memo to seem credible, the reader must have large and significant gaps in their understanding of the tech industry. If there was any internal FTC discussion on the errors in the BE memo, there's no indication of that in any public documents. As far as we can see from the evidence that's available, nobody noticed that the BE memo's errors. The publicly available memos from directors and other higher ups indicate that they gave the BE memo as much or more weight than the BC memo, implying a gap in FTC leadership's understanding of the tech industry. Brief summary Since the BE memo is effect

## How web bloat impacts users with slow devices

DevFeed: [How web bloat impacts users with slow devices](<https://devfeed.tech/articles/how-web-bloat-impacts-users-with-slow-devices-77745.md>)

Original publisher: [Read original article](<https://danluu.com/slow-device/>)

Published: 2024-03-16T00:00:00Z

Content type: article

Language: en

Sources: [Danluu](<https://devfeed.tech/sources/danluu.md>)

Topics: [Page Speed Optimization](<https://devfeed.tech/topics/page-speed-optimization.md>), [Core Web Vitals](<https://devfeed.tech/topics/core-web-vitals.md>), [User interface design](<https://devfeed.tech/topics/ui-design.md>)

Tags: [browser](<https://devfeed.tech/tags/browser.md>), [performance](<https://devfeed.tech/tags/performance.md>), [size](<https://devfeed.tech/tags/size.md>)

### AI overview

The article compares how websites perform on low-end devices, measuring load time, CPU use, and responsiveness across forums, social platforms, and business sites. It argues that CPU-heavy web apps and complex dynamic loading can make sites difficult or impossible to use on slower devices, and recommends designing for constrained hardware and connectivity. It also discusses how performance affects accessibility, user retention, and growth.

### Source excerpt

In 2017, we looked at how web bloat affects users with slow connections. Even in the U.S., many users didn't have broadband speeds, making much of the web difficult to use. It's still the case that many users don't have broadband speeds, both inside and outside of the U.S. and that much of the modern web isn't usable for people with slow internet, but the exponential increase in bandwidth (Nielsen suggests this is 50% per year for high-end connections) has outpaced web bloat for typical sites, making this less of a problem than it was in 2017, although it's still a serious problem for people with poor connections. CPU performance for web apps hasn't scaled nearly as quickly as bandwidth so, while more of the web is becoming accessible to people with low-end connections, more of the web is becoming inaccessible to people with low-end devices even if they have high-end connections. For example, if I try browsing a "modern" Discourse-powered forum on a Tecno Spark 8C, it sometimes crashes the browser. Between crashes, on measuring the performance, the responsiveness is significantly worse than browsing a BBS with an 8 MHz 286 and a 1200 baud modem. On my 1Gbps home internet connection, the 2.6 MB compressed payload size "necessary" to load message titles is relatively light. The over-the-wire payload size has "only" increased by 1000x, which is dwarfed by the increase in internet speeds. But the opposite is true when it comes to CPU speeds -- for web browsing and forum loading performance, the 8-core (2 1.6 GHz Cortex-A75 / 6 1.6 GHz Cortex-A55) CPU can't handle Discourse. The CPU is something like 100000x faster than our 286. Perhaps a 1000000x faster device would be sufficient. For anyone not familiar with the Tecno Spark 8C, today, a new Tecno Spark 8C, a quick search indicates that one can be hand for USD 50-60 in Nigeria and perhaps USD 100-110 in India. As a fraction of median household income, that's substantially more than a current generation iPhone in the U.S.

## Diseconomies of scale in fraud, spam, support, and moderation

DevFeed: [Diseconomies of scale in fraud, spam, support, and moderation](<https://devfeed.tech/articles/diseconomies-of-scale-in-fraud-spam-support-and-moderation-77682.md>)

Original publisher: [Read original article](<https://danluu.com/diseconomies-scale/>)

Published: 2024-02-18T00:00:00Z

Content type: opinion

Language: en

Sources: [Danluu](<https://devfeed.tech/sources/danluu.md>)

Topics: [online-fraud-prevention](<https://devfeed.tech/topics/online-fraud-prevention.md>), [fraud signal sharing](<https://devfeed.tech/topics/fraud-signal-sharing.md>), [fake account fraud](<https://devfeed.tech/topics/fake-account-fraud.md>), [AI Chat](<https://devfeed.tech/topics/ai-chat.md>)

Tags: [fraud](<https://devfeed.tech/tags/fraud.md>), [moderation](<https://devfeed.tech/tags/moderation.md>), [scale](<https://devfeed.tech/tags/scale.md>), [support](<https://devfeed.tech/tags/support.md>)

### AI overview

The article argues that large technology platforms often perform worse than smaller services at moderation, fraud and spam detection, and customer support. It attributes these problems to incentives, scale related complexity, and systems that overlook false positives, while acknowledging that economies of scale can benefit users in some areas.

### Source excerpt

If I ask myself a question like "I'd like to buy an SD card; who do I trust to sell me a real SD card and not some fake, Amazon or my local Best Buy?", of course the answer is that I trust my local Best Buy1 more than Amazon, which is notorious for selling counterfeit SD cards. And if I ask who do I trust more, my local reputable electronics shop (Memory Express, B&H Photo, etc.), I trust my local reputable electronics shop more. Not only are they less likely to sell me a counterfeit than Best Buy, in the event that they do sell me a counterfeit, the service is likely to be better. Similarly, let's say I ask myself a question like, "on which platform do I get a higher rate of scams, spam, fraudulent content, etc., [smaller platform] or [larger platform]"? Generally the answer is [larger platform]. Of course, there are more total small platforms out there and they're higher variance, so I could deliberately use a smaller platform that's worse, but I'm choosing good options instead of bad options, in every size class, the smaller platform is generally better. For example, with Signal vs. WhatsApp, I've literally never received a spam Signal message, whereas I get spam WhatsApp messages somewhat regularly. Or if I compare places I might read tech content on, if I compare tiny forums no one's heard of to lobste.rs, lobste.rs has a very slightly higher rate (rate as in fraction of messages I see, not absolute message volume) of bad content because it's zero on the private forums and very low but non-zero on lobste.rs. And then if I compare lobste.rs to a somewhat larger platform, like Hacker News or mastodon.social, those have (again very slightly) higher rates of scam/spam/fraudulent content. And then if I compare that to mid-sized social media platforms, like reddit, reddit has a significantly higher and noticeable rate of bad content. And then if I can compare reddit to the huge platforms like YouTube, Facebook, Google search results, these larger platforms have an ev

## Why it's impossible to agree on what's allowed

DevFeed: [Why it's impossible to agree on what's allowed](<https://devfeed.tech/articles/why-it-s-impossible-to-agree-on-what-s-allowed-77703.md>)

Original publisher: [Read original article](<https://danluu.com/impossible-agree/>)

Published: 2024-02-07T00:00:00Z

Content type: opinion

Language: en

Sources: [Danluu](<https://devfeed.tech/sources/danluu.md>)

Topics: [Stack Overflow](<https://devfeed.tech/topics/stackoverflow.md>), [human review](<https://devfeed.tech/topics/human-review.md>)

### AI overview

The article uses a survey about what counts as a vehicle in a park to illustrate how little agreement people may reach on clear-seeming rules. It argues that moderation policies on large platforms face even greater disagreement, especially on emotionally charged issues.

### Source excerpt

On large platforms, it's impossible to have policies on things like moderation, spam, fraud, and sexual content that people agree on. David Turner made a simple game to illustrate how difficult this is even in a trivial case, No Vehicles in the Park. If you haven't played it yet, I recommend playing it now before continuing to read this document. The idea behind the site is that it's very difficult to get people to agree on what moderation rules should apply to a platform. Even if you take a much simpler example, what vehicles should be allowed in a park given a rule and some instructions for how to interpret the rule, and then ask a small set of questions, people won't be able to agree. On doing the survey myself, one of the first reactions I had was that the questions aren't chosen to be particularly nettlesome and there are many edge cases Dave could've asked about if he wanted to make it a challenge. And yet, despite not making the survey particularly challenging, there isn't broad agreement on the questions. Comments on the survey also indicate another problem with rules, which is that it's much harder to get agreement than people think it will be. If you read comments on rule interpretation or moderation on lobsters, HN, reddit, etc., when people suggest a solution, the vast majority of people will suggest something that anyone who's done moderation or paid attention to how moderation works knows cannot work, the moderation equivalent of "I could build that in a weekend"1. Of course we see this on Dave's game as well. The top HN comment, the most agree-upon comment, and a very common sentiment elsewhere is2: I'm fascinated by the fact that my takeaway is the precise opposite of what the author intended. To me, the answer to all of the questions was crystal-clear. Yes, you can academically wonder whether an orbiting space station is a vehicle and whether it's in the park, but the obvious intent of the sign couldn't be clearer. Cars/trucks/motorcycles aren't all

## Notes on the report into Cruise's handling of a pedestrian accident

DevFeed: [Notes on the report into Cruise's handling of a pedestrian accident](<https://devfeed.tech/articles/notes-on-cruise-s-pedestrian-accident-77675.md>)

Original publisher: [Read original article](<https://danluu.com/cruise-report/>)

Published: 2024-01-29T00:00:00Z

Content type: article

Language: en

Sources: [Danluu](<https://devfeed.tech/sources/danluu.md>)

Topics: [AI agents in autonomous vehicles](<https://devfeed.tech/topics/ai-agents-in-autonomous-vehicles.md>), [Physical AI](<https://devfeed.tech/topics/physical-ai.md>)

### AI overview

The article examines a 2023 report on Cruise's response to an autonomous vehicle accident in which a pedestrian was dragged during the vehicle's post-collision pullover maneuver. It details the report's findings about incomplete and inconsistent disclosures to regulators, leadership and coordination failures, and recommendations for improving incident response and regulatory reporting. The author also discusses how safety-critical systems culture differs from practices in technology companies.

### Source excerpt

This is a set of notes on the Quinn Emanuel report on Cruise's handling of the 2023-10-02 accident where a Cruise autonomous vehicle (AV) hit a pedestrian, stopped, and then started moving again with the pedestrian stuck under the bottom of the AV, dragging the pedestrian 20 feet. After seeing some comments about this report, I read five stories on this report and then skimmed the report and my feeling is that the authors of four of the stories probably didn't read the report, and that people who were commenting had generally read stories by journalists who did not appear to read the source material, so the comments were generally way off base. As we previously discussed, it's common for summaries to be wildly wrong, even when they're summarizing a short paper that's easily read by laypeople, so of course summaries of a 200-page report are likely to be misleading at best. On reading the entire report, I'd say that Cruise both looks better and worse than in the articles I saw, which is the same pattern we saw when we looked at the actual source for Exhibits H and J from Twitter v. Musk, the United States v. Microsoft Corp. docs, etc.; just as some journalists seem to be pro/anti-Elon Musk and pro/anti-Microsoft, willing to push an inaccurate narrative to dunk on them to the maximum extent possible or exonerate them to the maximum extent possible, we see the same thing here with Cruise. And as we saw in those cases, despite some articles seemingly trying to paint Cruise in the best or worst light possible, the report itself has material that is more positive and more negative than we see in the most positive or negative stories. Aside from correcting misleading opinions on the report, I find the report interesting because it's rare to see any kind of investigation over what went wrong in tech in this level of detail, let alone a public one. We often see this kind of investigation in safety critical systems and sometimes see in sports as well as for historical events,

## Why people publish on short-form platforms instead of blogs

DevFeed: [Why people publish on short-form platforms instead of blogs](<https://devfeed.tech/articles/why-do-people-post-on-bad-platform-instead-of-good-platform-77766.md>)

Original publisher: [Read original article](<https://danluu.com/why-video/>)

Published: 2024-01-25T00:00:00Z

Content type: opinion

Language: en

Sources: [Danluu](<https://devfeed.tech/sources/danluu.md>)

Topics: [X (Twitter)](<https://devfeed.tech/topics/twitter.md>), [Plover](<https://devfeed.tech/topics/plover.md>)

### AI overview

The article explains why people choose short-form platforms and video over blogs: they can attract more engagement, reach an existing community, feel easier to publish on, and offer better monetization. The author argues that some content would not exist if it had to be produced as long-form text.

### Source excerpt

There's a class of comment you often see when someone makes a popular thread on Mastodon/Twitter/Threads/etc., that you also see on videos that's basically "Why make a Twitter thread? This would be better as a blog post" or "Why make a video? This would be better as a blog post". But, these comments are often stronger in form, such as: I can't read those tweets that span pages because the users puts 5 words in each reply. I find common internet completely stupid: Twitter, tiktok, Instagram, etc. What a huge waste of energy. or When someone chooses to blog on twitter you know it's facile at best, and more likely simply stupid (as in this case) These kinds of comments are fairly common, e.g., I pulled up Foone's last 10 Twitter threads that scored 200 points or more on HN and 9 out of 10 had comments like this, complaining about the use of Twitter. People often express bafflement that anyone could have a reason for using [bad platform], such as in "how many tweets are there just to make his point? 200? nobody thinks 'maybe this will be more coherent on a single page'? I don't get social media" or "Come on, typing a short description and uploading a picture 100 times is easier than typing everything in one block and adding a few connectors here and there? ... objectively speaking it is more work". Personally, I don't really like video as a format and, for 95% of youtube videos that I see, I'd rather get the information as a blog post than a video (and this will be even more true if Google really cracks down on ad blocking) and I think that, for a reader who's interested in the information, long-form blog posts are basically strictly better than long threads on [bad platform]. But I also recognize that much of the content that I want to read wouldn't exist at all if it wasn't for things like [bad platform]. Stepping back and looking at the big picture, there are four main reasons I've seen that people use [bad platform], which are that it gets more engagement, it's wher

## How bad are search results? Let's compare Google, Bing, Marginalia, Kagi, Mwmbl, and ChatGPT

DevFeed: [How bad are search results? Let's compare Google, Bing, Marginalia, Kagi, Mwmbl, and ChatGPT](<https://devfeed.tech/articles/how-bad-are-search-results-let-s-compare-google-bing-marginalia-kagi-mwmbl-and-chatgpt-77742.md>)

Original publisher: [Read original article](<https://danluu.com/seo-spam/>)

Published: 2023-12-30T00:00:00Z

Content type: comparison

Language: en

Sources: [Danluu](<https://devfeed.tech/sources/danluu.md>)

Topics: [search-engines](<https://devfeed.tech/topics/search-engines.md>), [Google Search](<https://devfeed.tech/topics/google-search.md>), [AI-driven information retrieval](<https://devfeed.tech/topics/ai-driven-information-retrieval.md>)

Tags: [chatgpt](<https://devfeed.tech/tags/chatgpt.md>), [google](<https://devfeed.tech/tags/google.md>), [google-search](<https://devfeed.tech/tags/google-search.md>), [search](<https://devfeed.tech/tags/search.md>), [search-engines](<https://devfeed.tech/tags/search-engines.md>)

### AI overview

The author compares Google, Bing, Marginalia, Kagi, Mwmbl, and ChatGPT using a set of practical queries, judging relevance and exposure to scams, SEO spam, and misleading results. The tests suggest that major search engines often return poor or risky results for simple queries, while smaller engines can sometimes perform better for particular searches. The article also discusses how advertising incentives and search engine scale affect result quality.

### Source excerpt

In The birth & death of search engine optimization, Xe suggests Here's a fun experiment to try. Take an open source project such as yt-dlp and try to find it from a very generic term like "youtube downloader". You won't be able to find it because of all of the content farms that try to rank at the top for that term. Even though yt-dlp is probably actually what you want for a tool to download video from YouTube. More generally, most tech folks I'm connected to seem to think that Google search results are significantly worse than they were ten years ago (Mastodon poll, Twitter poll, Threads poll). However, there's a sizable group of vocal folks who claim that search results are still great. E.g., a bluesky thought leader who gets high engagement says: i think the rending of garments about how even google search is terrible now is pretty overblown1 I suspect what's going on here is that some people have gotten so used working around bad software that they don't even know they're doing it, reflexively doing the modern equivalent of hitting ctrl+s all the time in editors, or ctrl+a; ctrl+c when composing anything in a text box. Every adept user of the modern web has a bag of tricks they use to get decent results from queries. From having watched quite a few users interact with computers, that doesn't appear to be normal, even among people who are quite competent in various technical fields, e.g., mechanical engineering2. However, it could be that people who are complaining about bad search result quality are just hopping on the "everything sucks" bandwagon and making totally unsubstantiated comments about search quality. Since it's fairly easy to try out straightforward, naive, queries, let's try some queries. We'll look at three kinds of queries with five search engines plus ChatGPT and we'll turn off our ad blocker to get the non-expert browsing experience. I once had a computer get owned from browsing to a website with a shady ad, so I hope that doesn't happen here (i

## Chat log exhibits from Twitter v. Musk case

DevFeed: [Chat log exhibits from Twitter v. Musk case](<https://devfeed.tech/articles/chat-log-exhibits-from-twitter-v-musk-case-77686.md>)

Original publisher: [Read original article](<https://danluu.com/elon-twitter-texts/>)

Published: 2022-10-01T00:00:00Z

Content type: article

Language: en

Sources: [Danluu](<https://devfeed.tech/sources/danluu.md>)

Topics: [API Business Models](<https://devfeed.tech/topics/api-business-models.md>), [Service fabric](<https://devfeed.tech/topics/service-fabric.md>), [password reset](<https://devfeed.tech/topics/password-reset.md>)

Tags: [2022](<https://devfeed.tech/tags/2022.md>), [2fa](<https://devfeed.tech/tags/2fa.md>), [doj](<https://devfeed.tech/tags/doj.md>), [easier](<https://devfeed.tech/tags/easier.md>), [identity](<https://devfeed.tech/tags/identity.md>), [microsoft](<https://devfeed.tech/tags/microsoft.md>), [ocr](<https://devfeed.tech/tags/ocr.md>), [password](<https://devfeed.tech/tags/password.md>), [password-reset](<https://devfeed.tech/tags/password-reset.md>), [spacex](<https://devfeed.tech/tags/spacex.md>)

### AI overview

The page presents OCR text from Exhibits H and J in the Twitter v. Musk case, including conversations about account security, Twitter's product direction, and the proposed purchase of the company. It also includes discussions of decentralized social networking and the technical limits of blockchain-based Twitter proposals.

### Source excerpt

This is a scan/OCR of Exhibits H and J from the Twitter v. Musk case, with some of the conversations de-interleaved and of course converted from a fuzzy scan to text to make for easier reading. I did this so that I could easily read this and, after reading it, I've found that most accountings of what was said are, in one way or another, fairly misleading. Since the texts aren't all that long, if you're interested in what they said, I would recommended that you just read the texts in their entirety (to the extent they're available -- the texts make it clear that some parts of conversations are simply not included) instead of reading what various journalists excerpted, which seems to sometimes be deliberately misleading because selectively quoting allows them to write a story that matches their agenda and sometimes accidentally misleading because they don't know what's interesting about the texts. If you want to compare these conversations to other executive / leadership conversations, you can compare them to Microsoft emails and memos that came out of the DoJ case against Microsoft and the Enron email dataset. Since this was done using OCR, it's likely there are OCR errors. Please feel free to contact me if you see an error. Exhibit H 2022-01-21 to 2022-01-24 Alex Shillings [IT specialist for SpaceX / Elon]: Elon- are you able to access your Twitter account ok? I saw a number of emails held in spam. Including some password resets attempts Elon: I haven't tried recently Elon: Am staying off twitter Elon: Is my twitter account posting anything? Alex: Not posting but I see one deactivation email and a dozen password reset emails. Assuming this is a scammer attempt but wanted to check to ensure you still had access to your Twitter Elon: It Is someone trying to hack my twitter Elon: But I have two-factor enabled with the confirmation app Alex: OK, great to hear. Alex: Yes -FaceTimed with them to confirm my identity(hah) and they are hopefully gonna reset your 2FA to SMS so

## In defense of simple architectures

DevFeed: [In defense of simple architectures](<https://devfeed.tech/articles/in-defense-of-simple-architectures-77743.md>)

Original publisher: [Read original article](<https://danluu.com/simple-architectures/>)

Published: 2022-04-06T00:00:00Z

Content type: opinion

Language: en

Sources: [Danluu](<https://devfeed.tech/sources/danluu.md>)

Topics: [modular-monolith](<https://devfeed.tech/topics/modular-monolith.md>), [backend frameworks](<https://devfeed.tech/topics/backend-frameworks.md>), [Product Management](<https://devfeed.tech/topics/product-management.md>), [locking](<https://devfeed.tech/topics/locking.md>)

Tags: [architectures](<https://devfeed.tech/tags/architectures.md>), [build](<https://devfeed.tech/tags/build.md>), [complexity](<https://devfeed.tech/tags/complexity.md>), [database](<https://devfeed.tech/tags/database.md>), [monolith](<https://devfeed.tech/tags/monolith.md>), [postgres](<https://devfeed.tech/tags/postgres.md>), [python](<https://devfeed.tech/tags/python.md>)

### AI overview

Wave describes how it keeps its backend architecture simple, using a Python monolith on Postgres and adding complexity only where business needs justify it. The article discusses tradeoffs in asynchronous processing, database transactions, vendor software, GraphQL, Kubernetes, and telecom integrations, and argues that engineering effort should focus on complexity that benefits the business.

### Source excerpt

Wave is a $1.7B company with 70 engineers1 whose product is a CRUD app that adds and subtracts numbers. In keeping with this, our architecture is a standard CRUD app architecture, a Python monolith on top of Postgres. Starting with a simple architecture and solving problems in simple ways where possible has allowed us to scale to this size while engineers mostly focus on work that delivers value to users. Stackoverflow scaled up a monolith to good effect (2013 architecture / 2016 architecture), eventually getting acquired for $1.8B. If we look at traffic instead of market cap, Stackoverflow is among the top 100 highest traffic sites on the internet (for many other examples of valuable companies that were built on top of monoliths, see the replies to this Twitter thread. We don't have a lot of web traffic because we're a mobile app, but Alexa still puts our website in the top 75k even though our website is basically just a way for people to find the app and most people don't even find the app through our website). There are some kinds of applications that have demands that would make a simple monolith on top of a boring database a non-starter but, for most kinds of applications, even at top-100 site levels of traffic, computers are fast enough that high-traffic apps can be served with simple architectures, which can generally be created more cheaply and easily than complex architectures. Despite the unreasonable effectiveness of simple architectures, most press goes to complex architectures. For example, at a recent generalist tech conference, there were six talks on how to build or deal with side effects of complex, microservice-based, architectures and zero on how one might build out a simple monolith. There were more talks on quantum computing (one) than talks on monoliths (zero). Larger conferences are similar; a recent enterprise-oriented conference in SF had a double-digit number of talks on dealing with the complexity of a sophisticated architecture and zero o

[Next page](<https://devfeed.tech/sources/danluu.md?cursor=WyIyMDIyLTA0LTA2VDAwOjAwOjAwKzAwOjAwIiwgIjQ2NWQ1NjI1LTRkY2EtNGNjNy1hNTA0LTQ4NTkxZGQ2OTEzNCJd>)