# Catching Bugs Before They Land: JiTTesting's Human-on-the-Loop Approach at Full Scale

DevFeed: [Catching Bugs Before They Land: JiTTesting's Human-on-the-Loop Approach at Full Scale](<https://devfeed.tech/articles/catching-bugs-before-they-land-jittesting-s-human-on-the-loop-approach-at-full-scale-106789.md>)

Original publisher: [Read original article](<https://engineering.fb.com/2026/10/29/security/catching-bugs-before-they-land-jittestings-human-on-the-loop-approach-at-full-scale/>)

Author: Sreeja Thummala; Sophie Zeng; Mark Harman; Weini Liu; Eric Liu

Published: 2026-10-29T16:00:12Z

Content type: article

Language: en

Sources: [Engineering at Meta](<https://devfeed.tech/sources/engineering-at-meta.md>), [Meta AI Research](<https://devfeed.tech/sources/meta-ai-research.md>), [Meta ML Applications](<https://devfeed.tech/sources/meta-ml-applications.md>)

Topics: [Test automation](<https://devfeed.tech/topics/test-automation.md>), [ai code review](<https://devfeed.tech/topics/ai-code-review.md>), [human review](<https://devfeed.tech/topics/human-review.md>), [Code review](<https://devfeed.tech/topics/code-review.md>)

Tags: [agentic](<https://devfeed.tech/tags/agentic.md>), [agentic-ai](<https://devfeed.tech/tags/agentic-ai.md>), [agentic-software-development](<https://devfeed.tech/tags/agentic-software-development.md>), [ai](<https://devfeed.tech/tags/ai.md>), [ai-research](<https://devfeed.tech/tags/ai-research.md>), [approach](<https://devfeed.tech/tags/approach.md>), [bugs](<https://devfeed.tech/tags/bugs.md>), [generation](<https://devfeed.tech/tags/generation.md>), [human-on-the-loop](<https://devfeed.tech/tags/human-on-the-loop.md>), [language-models](<https://devfeed.tech/tags/language-models.md>), [large-language-models](<https://devfeed.tech/tags/large-language-models.md>), [meta](<https://devfeed.tech/tags/meta.md>), [ml-applications](<https://devfeed.tech/tags/ml-applications.md>), [production](<https://devfeed.tech/tags/production.md>), [scale](<https://devfeed.tech/tags/scale.md>), [security-privacy](<https://devfeed.tech/tags/security-privacy.md>), [tests](<https://devfeed.tech/tags/tests.md>)

## AI overview

Meta's Catching JiTTests generates executable tests to expose behavioral differences introduced by code changes, aiming to catch bugs that code-review-based LLM approaches may miss. It runs overnight on spare capacity against diffs ranked as risky. Two workflows generate candidate tests: one treats each diff like a mutant, while the other infers the change's intent and creates tests for plausible implementation risks. Rule-based RubFake and an ensemble of LLM judges rank likely false positives; engineers set review criteria and assess outcomes in aggregate rather than reviewing every test. Meta reports that 589 engineers engaged with the system, with a 60% true-positive rate among reported findings. In the first half of 2026, it helped catch 74 serious privacy bugs before production, and Meta reports a 20% reduction in privacy SEVs during that period.

## Source excerpt

Over the last year we've been sharing our work leveraging large language models to catch bugs on the fly -- a novel approach we call Catching Just-in-Time Tests (JiTTests). Now, we're sharing the results and positive impact we've seen with Catching JiTTests deployed into production. Our paper, "Just-in-Time Catching Test Generation at Meta," fully details [...] Read More... The post Catching Bugs Before They Land: JiTTesting's Human-on-the-Loop Approach at Full Scale appeared first on Engineering at Meta.