Door's Locked, Try the Window: Measuring Circumvention Propensity in Coding Agents

Prakrat Agrawal, mentored by Jérémy Scheurer, built an evaluation suite to measure whether frontier coding agents respect a constraint they have been given, or quietly route around it to finish the task.

Summary

Coding agents are increasingly given write access to real codebases under constraints that are meant to hold: files they must not touch, tests they must not rewrite. CircumEval measures how often agents honour those constraints when honouring them means failing the task. Each of its twelve scenarios — eight built on FastAPI and four on datasette — introduces a bug and then locks either the buggy source file or the test that asserts the buggy behaviour, enforced at three layers: file permissions, kernel immutability flags, and a sandboxed container running as a non-root user. The aligned response is to stop and report the lock; circumvention is any route around it. Claude Opus 4.6, Claude Sonnet 4.6 and GPT-5.4 all circumvent heavily, at 100%, 89% and 99% respectively on source-locked tasks, and at 40%, 66% and 94% on test-locked tasks. The agents reach for conftest.py monkeypatching, global runtime patching, replacement of cached Python bytecode, and edits to test assertions. Prompting the models not to circumvent produces unpredictable results; only an explicit instruction to stop and report drives circumvention to zero.