# nondeterminism Published articles for nondeterminism. This is one page of public article previews, not the complete archive. Follow Next page to continue. Summaries are not the original full articles. ## One AI Output Is an Example, Not an Evaluation DevFeed: [One AI Output Is an Example, Not an Evaluation]() Original publisher: [Read original article]() Author: Raluca Budiu Published: 2026-08-14T17:00:00Z Content type: article Language: en Sources: [NN/g latest articles and announcements]() Topics: [Artificial Intelligence](), [Human-AI evaluation](), [LLM evaluation / benchmarking](), [benchmarking](), [User experience (UX)]() Tags: [ai](), [article](), [benchmarking](), [confidence-interval](), [eval](), [evaluation](), [language-models](), [llm](), [metrics](), [nondeterminism](), [performance](), [research](), [statistical-significance](), [usability]() ### AI overview One AI output is only an example, not a reliable evaluation. Because AI systems can produce different results from the same input, teams should assess them with multiple representative inputs, repeated runs, quantitative metrics, and confidence intervals. ### Source excerpt One output cannot establish how well an AI system performs. Evaluate with multiple representative inputs, repeated runs, and confidence intervals. ## Scaling security reviews at 1Password: Solving the context and nondeterminism problems DevFeed: [Scaling security reviews at 1Password: Solving the context and nondeterminism problems]() Original publisher: [Read original article]() Author: info@1password.com (Megan Barker) Published: 2026-07-30T00:00:00Z Content type: article Language: en Sources: [Blog on 1Password Blog]() Topics: [Security](), [Code review](), [Large Language Model](), [long-context](), [GitHub](), [Python](), [Go Language](), [pull-requests]() Tags: [ai](), [building-1password](), [code](), [context-window](), [developers](), [github](), [go](), [llm](), [nondeterminism](), [python](), [review](), [security]() ### AI overview 1Password describes how it evolved SAGE to scale security code reviews across large repositories. The v2 design addresses limited LLM context and nondeterministic outputs by generating directory-level scaffolding that compresses structural security context, with a Python proof of concept being productionized in Go. ### Source excerpt In our last post, we shared how we began to scale our security code review process with SAGE. We discussed how we gathered historical Product Security (ProdSec) review records to create a 1Password-specific ruleset, the three-stage Finder/Critic/Judge pipeline, and the limitations of our v1 implementation. Above all, human ProdSec reviewers still had to bring full context to the findings: where the trust boundaries lie, which directories are sensitive, and whether mitigations exist elsewhere in the codebase. Our goal for v2 was to help SAGE understand our entire codebase. Many of our GitHub repositories are huge, including our client and server monorepos. That means we have way too much information to fit within any LLM's context window. We had to find a way to let SAGE perform deeper reasoning about the PR diffs it reviews without the codebase itself. There was another hurdle. As we built v2, we ran into a fundamental LLM trait: they can't reliably produce the same output twice. We knew we had to do our best to manage this nondeterminism so we could trust SAGE to be a relatively consistent security reviewer. We had two things to figure out: how to fit a lot of data into a context window, and how to get consistent output from inherently inconsistent tools. If we could solve those riddles, SAGE wouldn't just know 1Password, it would finally understand it. And it would earn the name SuperSAGE. Compressing context with scaffolding As it turns out, our Security Research team had already developed a Python proof of concept designed to compress our code context. It was a set of LLM prompts that generated one SCAFFOLDING.md file per source directory. Those scaffolding files carried compressed structural context like sensitivity ratings, attack surfaces, trust boundaries, and file summaries. It was a great foundation; we just had to productionize it as a Go rewrite on top of SAGE v1's model-agnostic llm.Client harness. To start, the PoC took inventory of our code structure. ## LLMs are bad at vibing specifications DevFeed: [LLMs are bad at vibing specifications]() Original publisher: [Read original article]() Author: Hillel Wayne Published: 2026-03-10T17:12:30Z Content type: article Language: en Sources: [Newsletter feed for Hillel Wayne's Newsletter]() Topics: [Formal methods](), [Large Language Model](), [Specifications](), [Artificial Intelligence]() Tags: [ai](), [formal-methods](), [llms](), [nondeterminism](), [specifications](), [verification]() ### AI overview The article examines AI-generated TLA+ and Alloy specifications through a case study. It argues that these specifications may fail to compile or model-check and often contain tautological or obvious properties rather than subtle properties involving concurrency, nondeterminism, or multi-step bad behavior. ### Source excerpt No newsletter next week I'll be speaking at InfoQ London. But see below for a book giveaway! LLMs are bad at vibing specifications About a year ago I wrote AI is a gamechanger for TLA+ users, which argued that AI are a "specification force multiplier". That was written from the perspective an TLA+ expert using these tools. A full 4% of Github TLA+ specs now have the word "Claude" somewhere in them. This is interesting to me, because it suggests there was always an interest in formal methods, people just lacked the skills to do it. It's also interesting because it gives me a sense of what happens when beginners use AI to write formal specs. It's not good. As a case study, we'll use this project, which is kind of enough to have vibed out TLA+ and Alloy specs. Looking at a project Starting with the Alloy spec. Here it is in its entirety: module ThreatIntelMesh sig Node {} one sig LocalNode extends Node {} sig Snapshot { owner: one Node, signed: one Bool, signatures: set Signature } sig Signature {} sig Policy { allowUnsignedImport: one Bool } pred canImport[p: Policy, s: Snapshot] { (p.allowUnsignedImport = True) or (s.signed = True) } assert UnsignedImportMustBeDenied { all p: Policy, s: Snapshot | p.allowUnsignedImport = False and s.signed = False implies not canImport[p, s] } assert SignedImportMayBeAccepted { all p: Policy, s: Snapshot | s.signed = True implies canImport[p, s] } check UnsignedImportMustBeDenied for 5 check SignedImportMayBeAccepted for 5 Couple of things to note here: first of all, this doesn't actually compile. It's using the Boolean standard module so needs open util/boolean to function. Second, Boolean is the wrong approach here; you're supposed to use subtyping. sig Snapshot { owner: one Node, - signed: one Bool, signatures: set Signature } + sig SignedSnapshot in Snapshot {} pred canImport[p: Policy, s: Snapshot] { - s.signed = True + s in SignedSnapshot } So we know the person did not actually run these specs. This is somewhat less of a probl ## Using Dependency Injection in Python to Make Date-Dependent Code Deterministic and Testable DevFeed: [Using Dependency Injection in Python to Make Date-Dependent Code Deterministic and Testable]() Original publisher: [Read original article]() Author: Haki Benita Published: 2020-05-31T21:00:00Z Content type: article Language: en Sources: [Haki Benita]() Topics: [Dependency injection](), [Python](), [DateTime](), [Testing]() Tags: [articles](), [dependency-injection](), [django](), [nondeterminism](), [python](), [test](), [testing]() ### AI overview This article explains how dependency injection can make Python code that depends on the current date deterministic and easier to test. It uses a function returning tomorrow's date to show why hard-coded time references create fragile tests, and presents passing the reference date as an argument as an alternative to mocking or external libraries. ### Source excerpt If you ever had a test that one day just started to fail, unprovoked, or a test that fails once every blue moon for no apparent reason, it's possible your code is relying on something that is not deterministic. In this article I describe a practical approach to dependency injection in Python that when used correctly, can eliminate nondeterminism and make your code easier to maintain and to test. ## Other Complexity Classes DevFeed: [Other Complexity Classes]() Original publisher: [Read original article]() Published: 2012-02-29T17:12:35Z Content type: article Language: en Sources: [Jeremy Kun]() Topics: [Algorithms, Complexity](), [Computing](), [Math and Logic]() Tags: [complexity](), [computational-complexity](), [halting-problem](), [kolmogorov-complexity](), [nondeterminism](), [open-questions](), [oracle](), [primer](), [pspace](), [turing-machine](), [turing-machines]() ### AI overview This article introduces complexity classes beyond P and NP, including PSPACE and NPSPACE, and discusses oracle-based classes. It explains that PSPACE equals NPSPACE while many relationships among complexity classes remain open. ### Source excerpt Not Just Time, But Space Too! So far on this blog we've introduced models for computation, focused on Turing machines and given a short overview of the two most fundamental classes of problems: P and NP. While the most significant open question in the theory of computation is still whether P = NP, it turns out that there are hundreds (almost 500, in fact!) other "classes" of problems whose relationships are more or less unknown.