Page content 1Password’s FLAWED report , published on August 6, 2026,

Page content 1Password’s FLAWED report , published on August 6, 2026, gives defenders a misleading picture of AI patching. Its headline says models produced clean fixes only 26% of the time. That figure includes experiments that deliberately instructed agents to apply the wrong fix, along with experiments in which agents could not compile or test their patches. The report risks making defenders less effective by discouraging them from using technology that could help them fix more vulnerabilities. Teams that take its headline at face value may leave repairable vulnerabilities unaddressed. We want our work to help defenders fix more vulnerabilities. This post shares real-world data on human and agent patch quality from our consulting projects and Patch the Planet. We’re also releasing two agent skills: post-patch-validation to help agents test fixes, and review-walkthrough to help engineers review them. How the experiment produces a misleading headline Our review of 1Password’s code and data found four choices that make its 26% clean-fix rate a misleading guide to ordinary patching work. 1 The sample was selected for difficult fixes. The authors chose six vulnerabilities because their fixes were complex. Clean-fix rates ranged from 3% to 60% across those bugs, so the average depends heavily on which vulnerabilities made the list. 2 Two prompts tell agents to apply the wrong fix. Those prompts account for 22% of the data. Combining them with ordinary repair attempts makes the reported rate depend partly on how often the researchers chose to give agents bad advice. More than a third of the trials prohibit testing. One evaluation mode prevents agents from building or running code and accounts for 36% of the data. The headline combines those trials with experiments in which agents could test their patches and act on the results. The models ran at different reasoning settings. GPT-5.5 ran at medium effort and Opus 4.8 at high. These were the tools’ defaults. Neither model was tested at its highest available setting, and the authors did not measure how increasing effort affected the results. 1Password’s headline also obscures a useful result in its own data. We reanalyzed the patches and recorded test results published with the study, keeping trials where agents could run code and were not instructed to apply the wrong fix. In those trials, 2,634 of 3,067 patches generated by 1Password’s models (86%) blocked the supplied exploit. We excluded runs that the study classified as having consulted the upstream fix. Blocking that exploit does not establish a complete repair, but these results show useful patching capability under reasonable working conditions that the headline fails to convey. The instructions and grading introduce further problems, several of which Davi Ottenheimer has also highlighted: The stopping rule and grading criteria disagree. Agents given a proof-of-concept exploit were instructed to stop once their patch defeated it. The grader then evaluated vulnerable paths that the supplied exploit did not exercise. The grading penalizes intended behavior changes. Agents were told to leave existing tests untouched, even though a correct fix can require updating tests to reflect changed behavior. We found that 8% of ActiveMQ verdicts penalized an intended behavior change as a regression. The automated grades disagree with human review. Models grading their own patches matched human reviewers on the full five-category outcome in 65.9% of reviewed cases. Agreement was 87.7% for whether the original bug was fixed and 70.5% for whether new bugs were introduced. ( Table 24 ) Changing the reviewer changes the result. The two models assigned different outcomes to 36.8% of the same patches. The headline averages their assessments. ( Table 20 ) The Linux reference fix contains a vulnerability. The authors found 248 generated patches that repeated an off-by-one error in the upstream fix. The automated grader caught that new vulnerability in only 24 of them. ( Section 4.4 ) The Chromium grader accepts incomplete repairs. It marked many patches as clean even though they left a use-after-free vulnerability in a callback. ( Section 4.9 ) The grading errors can penalize valid fixes and let vulnerable patches pass. Combined with the handpicked sample and deliberately bad instructions, they leave the report without a credible basis for its headline. Defenders should not take 1Password’s headline rate seriously as a measure of AI patching ability. Developers get one in eight fixes wrong under ideal conditions Understanding agent failures also requires understanding how often developers submit incomplete fixes. Our security consulting work gives us a detailed record of how developers repair vulnerabilities in their own software. We give clients detailed vulnerability reports, then conduct a “ fix review ” to check whether their proposed patches fully resolve the issues. Our records connect each vulnerability to the developer’s first proposed fix and our assessment of whether it worked. They preserve unsuccessful attempts that developers revise before an issue is considered resolved. We reviewed the first fixes submitted for 2,265 vulnerabilities across 236 Trail of Bits security assessments from 2024 to 2026. The developers maintained the affected software, had detailed reports from our engineers, and knew we would review their patches. Even under those favorable conditions, 283 first fixes failed to fully resolve the reported issue: 12.5%, or one in eight. Figure showing first fix submission outcomes Accounting for multiple fixes from the same assessment, the 95% confidence interval is 10.5% to 14.5% . Sometimes we point out a mistake in a client’s patch during an informal conversation, and they correct it before the formal fix review. Those early failures may never appear in the review record, so our data can undercount failed first attempts. We also excluded cases where the available records did not establish whether the fix worked. A direct comparison with agents would require the same tasks and working conditions. What happened to our patches in real projects Through Patch the Planet, our joint initiative with OpenAI, Trail of Bits has co-authored hundreds of patches for widely used open-source projects. Agents wrote the patches with engineers directing the work and checking the results. Project maintainers then decided whether to merge, revise, or reject each submission. How maintainers reviewed Patch the Planet patches We examined the public review history of every Patch the Planet submission in our dataset that maintainers had merged or closed by September 14, 2026: 186 pull requests. 1Password’s benchmark used six vulnerabilities selected because their fixes were complex. Maintainers merged 126 of our 186 pull requests, an acceptance rate of 67.7%. 3 In 91 of those 126 pull requests (72.2%), maintainers accepted the security fix we originally proposed. Review outcome PRs % of merged PRs Total merged 126 100% Accepted with no security-relevant revision observed 91 72.2% Accepted with security-relevant revision observed 33 26.2% Indeterminate 2 1.6% Table 1: Changes requested by maintainers for 126 merged Patch the Planet pull requests. Security-related revisions include repairs to a proposed fix and expansions of its security coverage. Maintainer acceptance does not establish that every patch is correct. Maintainers closed the other 60 submissions without merging them. Most were superseded by other work or declined for policy, process, scope, or maintenance reasons. Four were explicitly rejected on technical grounds. Reason for closure PRs % of closed PRs Total closed without merge 60 100% Superseded, reimplemented, or re-landed elsewhere 36 60.0% Policy, process, scope, or maintenance reasons 14 23.3% Duplicate or convergent with another fix 3 5.0% Explicitly rejected on technical grounds 4 6.7% Other or indeterminate 3 5.0% Table 2: Reasons maintainers closed 60 Patch the Planet pull requests without merging One of those closed submissions was our freenginx patch. A maintainer and an agent introduced the same freenginx crash 1Password’s case study examines a Patch the Planet fix for a memory-safety bug in freenginx’s embedded Perl module. An agent wrote our patch under the direction of a Trail of Bits engineer. It left one vulnerable code path open and introduced a new crash during request cleanup. The paper’s criticism of our patch is correct. The maintainer closed our pull request and committed a separate fix . That fix covered all three vulnerable code paths but introduced the same crash during cleanup. The paper documents the maintainer’s regression too. Both authors encountered the same trap. The original bug allowed Perl to destroy a callback before freenginx used it. Both fixes kept the callback alive so freenginx could use it later. But if the request timed out first, freenginx would make the request unusable and then release the