Meta has deployed its Just-in-Time Catching Tests (JiTTests) system into production, an LLM-based pipeline that generates executable tests designed to fail when a code change introduces a bug, providing concrete behavioral evidence before merge. Operating overnight on spare capacity across Meta’s backend codebase of hundreds of millions of lines, the pipeline targets risky diffs using a Diff Risk Score-like mechanism and employs two diff-aware generation workflows: "dodgy diff," which treats the diff as a mutant to differentiate from its parent, and "intent-aware," which infers the change’s purpose, enumerates risks, builds mutants for each, and synthesizes tests to kill them. Candidate failures are filtered by an LLM judge ensemble (Llama 3.3 70B, Gemini 3 Pro, Claude Sonnet 4) and RubFake, a deterministic rule engine flagging reflection, broken mocks, and NotImplemented exceptions. In the research evaluation spanning 22,126 generated tests, the intent-aware workflow achieved weak catches in 6.4% of tests and on 7.9% of diffs, outperforming dodgy diff (2.5% and 4.0%) and the prior ACH mutation-guided hardening (0.8% and 1.6%). The LLM and rule-based assessors reduced human review load by 70%. Initial deployment confirmed only 8 of 41 reported catches (19.5% true-positive rate), but subsequent production scaling yielded a self-reported 60% true-positive rate across 589 engaged engineers. Critically, JiTTests caught 74 privacy SEVs at the diff stage in H1 2026, correlating with a 20% reduction in overall privacy SEV occurrence. The system remains internal with no public code, models, or API release.
Read original
marktechpost