# DeepsecBench: evaluating model performance in finding cybersecurity vulnerabilities

DevFeed: [DeepsecBench: evaluating model performance in finding cybersecurity vulnerabilities](<https://devfeed.tech/articles/deepsecbench-evaluating-model-performance-in-finding-cybersecurity-vulnerabilities-730.md>)

Original publisher: [Read original article](<https://vercel.com/blog/deepsecbench-evaluating-model-performance-in-finding-cybersecurity-vulnerabilities>)

Author: Eric Dodds

Published: 2026-07-27T04:00:00Z

Content type: article

Language: en

Sources: [Vercel News](<https://devfeed.tech/sources/vercel-news.md>)

Topics: [Benchmark](<https://devfeed.tech/topics/benchmark.md>), [Vulnerabilities](<https://devfeed.tech/topics/vulnerabilities.md>), [AI Models](<https://devfeed.tech/topics/ai-models.md>), [Cybersecurity](<https://devfeed.tech/topics/cybersecurity.md>), [Code](<https://devfeed.tech/topics/code.md>), [Open Source](<https://devfeed.tech/topics/open-source.md>), [OpenAI](<https://devfeed.tech/topics/openai.md>), [hugging face](<https://devfeed.tech/topics/hugging-face.md>)

Tags: [ai-models](<https://devfeed.tech/tags/ai-models.md>), [analysis](<https://devfeed.tech/tags/analysis.md>), [benchmark](<https://devfeed.tech/tags/benchmark.md>), [breach](<https://devfeed.tech/tags/breach.md>), [code](<https://devfeed.tech/tags/code.md>), [cost](<https://devfeed.tech/tags/cost.md>), [cybersecurity](<https://devfeed.tech/tags/cybersecurity.md>), [false-positives](<https://devfeed.tech/tags/false-positives.md>), [hugging-face](<https://devfeed.tech/tags/hugging-face.md>), [model](<https://devfeed.tech/tags/model.md>), [open-source](<https://devfeed.tech/tags/open-source.md>), [openai](<https://devfeed.tech/tags/openai.md>), [testing](<https://devfeed.tech/tags/testing.md>), [time](<https://devfeed.tech/tags/time.md>), [vulnerabilities](<https://devfeed.tech/tags/vulnerabilities.md>)

## AI overview

DeepsecBench evaluates how well AI models find cybersecurity vulnerabilities in application code. It uses an open-source codebase, 50 entry-point files, and 231 human-judged findings, reporting recall, precision, cost, total time, and a recall-weighted F2 score. The article explains that the benchmark is kept secret to prevent models from training on its contents and describes how its results can guide security scanning programs.

## Source excerpt

Last week, OpenAI evaluated two models on an exploit benchmark within an isolated sandbox. Guardrails were reduced for testing, and the models found a vulnerability in their environment, accessed the internet, and reached Hugging Face's production database. No human directed the action, but the breach is a clear example of how much more capable malicious attackers are when equipped with powerful AI models. But defenders have the same tools, and a clear advantage: knowledge of their own codebase. Hacks are initiated from the outside, so the single best defense is finding vulnerabilities from the inside before attackers do. Today we're releasing DeepsecBench, a benchmark that evaluates how well different models find cybersecurity vulnerabilities in application code. For each model the report includes recall, precision, cost, and total time, and combines recall and precision into a single benchmark score. Here is a sample of model performance from the leaderboard: Rank Model Level Score Cost Total time 1 GPT-5.6 Sol xhigh 35.58 $55.98 03:39:00 3 Claude Opus 5 medium 28.36 $31.96 00:47:01 8 Kimi K3 high 17.56 $12.38 01:59:00 10 Grok 4.5 high 15.58 $5.60 01:24:00 We built deepsec to make scanning as easy as possible. Now you can use the benchmark report to build a security scanning program that fits your budget and the complexity of your codebase, choosing the right mix of models to run and how often to run them. How the benchmark works DeepsecBench runs on an open-source codebase at a commit state just before a large number of vulnerabilities were fixed. We selected 50 entry-point files and built a golden set of 231 human-judged findings. Each model's score is a recall-weighted F2 (Score = 100 x 5PR/(4P+R)), weighting recall (R) twice as much as precision (P), because missed vulnerabilities will go unfixed, while false positives don't make your codebase less secure. Findings beyond the golden set are classified by a judge model as real or false, and count for or against