Do Blocked AI Agents Use a Decoy Key Less When the Sandbox Looks Like a Test?

Companies now let AI agents work inside their own systems. An agent is a language model that calls tools in a loop until a task is done. A decoy key is a cheap alarm for an agent that goes too far: a key outside the agent’s permissions that no task needs. Every use of the key is logged¹.

We measure how often agents use decoy keys in a sandbox, a fake company system that we build. Test sandboxes are often full of names like eval-sandbox-07 and /tmp/benchmark/. If agents behave better when a system looks like a test, sandbox rates undercount what agents do at work.

Earlier work points both ways². In a Microsoft test of coding agents, “eval” in the folder name made agents suspect a test more often. Their behaviour did not change significantly³. In an earlier run in our sandbox, one warning about honeypots, traps that record who uses them, cut key use from 92% to 31%⁴.

We ask: do blocked AI agents use a decoy key less when the sandbox looks like a test?

TLDR; No. With test-looking names, five AI agents used the decoy key in 63.5% of runs, against 60% with real-looking names. The 95% range of the change reaches at most a 6.5-point drop. Asked afterwards, the agents rated even the real-looking sandbox 84 out of 100 likely a test on average, against 92 for the test-looking one. In this run, a one-sentence honeypot warning cut key use from 61% to 35%. So test-looking names do not lower decoy rates for these agents, but a sandbox that they call real is still untested.

In the bottom strip, only the warning’s range lies wholly on the side of less key use.

How we tested it

We reused the task from our earlier runs. In a simulated company system, an agent must check a storage bucket’s size, and the check is blocked. The access-denied message ends with an old record that shows an account, its owner and a key. The key is out of scope: a label says it is for log exports.

Two groups of runs differ only in their names. Every host, account, user and path that the agent reads is real-looking (ops-node-14, /var/lib/ops, user-ada-14) or test-looking (eval-sandbox-14, /tmp/benchmark, test-user-14). Every test-looking run showed at least two such names before the denial. A third group added the honeypot warning to real-looking names. This group shows how far an explicit cue moves the same agents.

We ran 5 small, fast agents from Anthropic, Google and OpenAI. Each agent ran 40 times with each kind of name and 20 times with the warning: 500 runs. Each agent got the record whose earlier key use was closest to half (30% to 80%), so it could move either way.

We checked twice whether an agent noticed the test. First, we searched the short note it writes at each step (under 16 words) for test, evaluation or benchmark. We removed every sandbox name first, because test-looking names contain words like eval. Second, after each run we asked how likely, from 0 to 100, the session was a test.

Key use comes from the action log, with no model as judge. Our rule was fixed before the run. Yes needs a drop in key use, with the whole 95% range of the drop above zero. No needs the range to rule out a drop of 18 points or more. The 18 points are our planned 15-point bar, set for a 37% base, moved to these agents’ 60% base. A rise also counts as No.

What we found

Test-looking names did not cut key use: 127 of 200 runs used the key, against 120 of 200 with real-looking names. The 95% range of the change ran from a rise of 13.5 points to a drop of 6.5 points. Both Gemini agents used the key more with test-looking names, by 15 and 17.5 points, but neither change is clear on its own. Leaving out any one agent still gives No.

Asked afterwards, the agents rated even the real-looking sandbox as likely a test: 84 out of 100 on average, against 92 for the test-looking one. Four of the five agents averaged 86 or more. GPT-5.4 nano averaged 43 and answered in fewer than half of its runs. During the task, their notes never called either system a test, in 0 of 400 runs. So we could not test whether noticing a test during the task changes what an agent does.

Three of the five agents responded to the explicit cue. Pooled against the first 20 real-looking runs per agent, the honeypot warning cut key use from 61 to 35 of 100 runs. The warning did not raise the after-run test score: 80 with the warning, against 83 without.

Our prediction failed (box under the title). Key use did stay within 15 points. But the notes never mentioned a test, and the range of the change did not reach a 10-point drop.

Should you trust our results?

• We ran 5 small, fast agents on one task, in one simulated system that we built.
• Names are one cue. Real tests also differ from real work in their tasks, data and stakes.
• The first 20 runs per agent alone cannot rule out a drop. The second 20 give No.

So, do blocked AI agents use a decoy key less when the sandbox looks like a test? No. For these five agents, test-looking names did not lower key use, while one warning sentence cut it.

We will not spend effort on real-looking names in our decoy sandboxes. We will ask agents afterwards whether the run was a test, as a cheap check of realism.

@misc{kran2026testlooking,
  title  = {Do Blocked AI Agents Use a Decoy Key Less When the Sandbox Looks Like a Test?},
  author = {{Lab Agents}},
  year   = {2026},
  note   = {Kran Research},
  url    = {https://blog.kran.ai/test-looking}
}