# Off-by-1 Labs: Why AI-generated vulnerability patches still require expert human review

DevFeed: [Off-by-1 Labs: Why AI-generated vulnerability patches still require expert human review](<https://devfeed.tech/articles/off-by-1-labs-why-ai-generated-vulnerability-patches-still-require-expert-human-review-1974.md>)

Original publisher: [Read original article](<https://1password.com/blog/why-ai-generated-patches-still-require-human-review>)

Author: info@1password.com (Keith Hoodlet)

Published: 2026-08-06T00:00:00Z

Content type: article

Language: en

Sources: [Blog on 1Password Blog](<https://devfeed.tech/sources/blog-on-1password-blog.md>)

Topics: [Large Language Model](<https://devfeed.tech/topics/llm.md>), [Vulnerabilities](<https://devfeed.tech/topics/vulnerabilities.md>), [Security](<https://devfeed.tech/topics/security.md>), [LLM evaluation / benchmarking](<https://devfeed.tech/topics/llm-evaluation-benchmarking.md>), [Artificial Intelligence](<https://devfeed.tech/topics/ai.md>), [datasets](<https://devfeed.tech/topics/datasets.md>)

Tags: [agents](<https://devfeed.tech/tags/agents.md>), [ai](<https://devfeed.tech/tags/ai.md>), [data](<https://devfeed.tech/tags/data.md>), [developers](<https://devfeed.tech/tags/developers.md>), [large-language-models-llms](<https://devfeed.tech/tags/large-language-models-llms.md>), [openai](<https://devfeed.tech/tags/openai.md>), [research](<https://devfeed.tech/tags/research.md>), [review](<https://devfeed.tech/tags/review.md>), [security](<https://devfeed.tech/tags/security.md>), [vulnerability](<https://devfeed.tech/tags/vulnerability.md>)

## AI overview

A 1Password security research study finds that large language models generated vulnerability patches with embedded defects 53.9% of the time across 6,080 patches for six recently disclosed CVEs. Only 26.0% fully resolved the vulnerability without materially changing application behavior, supporting the need for expert human review.

## Source excerpt

We studied what happens when Large Language Models (LLMs) generate vulnerability patches for recently disclosed, complex vulnerabilities. Our data shows that LLMs produce Fix-Like Artifacts with Embedded Defects (FLAWED) 53.9% of the time when complex patches are required. By sharing the results of our research, our goal is to provide defenders with the tooling and methodology necessary to improve vulnerability remediation outcomes at scale. Along with this blog, we are releasing our tooling, datasets, and an in-depth research paper to share what we've learned. With models and agentic harnesses now performing impactful vulnerability discovery at scale, as recently witnessed with Anthropic's Project Glasswing, defenders are naturally turning to AI agents to generate vulnerability patches. Indeed, this exact response made headlines in June with OpenAI's announcement of Project Daybreak in collaboration with a number of partners who aim to "Patch the Planet". But how effective are LLMs at producing patches without altering the application's behavior? Do the patches they generate actually mitigate the vulnerabilities in question? And how frequently might those patches introduce new vulnerabilities? We set out to answer these questions as the inaugural research project for 1Password's brand-new security research team, Off-by-1 Labs. The paper's title is Frontier Models' Vulnerability Patches are Often F.L.A.W.E.D., and unlike other research in this space, this study targets novel vulnerabilities not likely to be found in the training data of frontier models, and then exercises frontier models to determine their efficacy at successfully producing patches. Across six recently-disclosed CVEs, we produced 6,080 patches using two frontier, cyber-capable reasoning models. The average success rate for generating a patch that fully resolved the vulnerability (without materially changing application behavior) was just 26.0%. Patches that successfully resolved the vulnerability, b