Does a Crawl With No Model Predict How Often AI Agents Follow a Hidden Link?

Some websites hide links that no person can see, so any visit marks a bot. Cloudflare’s AI Labyrinth works this way¹. Scraping guides tell bot builders to draw the page like a browser and skip hidden links². A defender wants to know which agents a trap catches without running them.

Whether an agent sees the link depends on its fetch tool, the program that turns a web page into text for the model. In our earlier test, three small agents had to find a 2025 number on websites we built³. The table page showed only 2024 numbers, and a hidden footer link led to the answer. They followed it in 99% of runs when its words fit the task (“Full 2025 pricing table”). We call a link whose words do not fit the task an off-task link (“Office photo gallery archive”). They followed one in 3% to 73% of runs. But they used fetch tools we wrote.

Here we test a simple rule on ten shipped fetch tools. A link is exposed when the tool’s first reply for a page holds its address, or a button the agent can press. The rule predicts the follow rate as the share of pages that expose the link, times the agent’s earlier rate. Among the ten tools is the MCP fetch tool, the official fetch tool of the Model Context Protocol, a standard way to give agents tools. By default, it runs what we call a main-text filter. The filter keeps a page’s article text and usually drops menus and footers. The tool also has a raw mode that the agent can switch on. Raw mode returns the page’s full code, hidden parts included.

We ask: does a crawl with no model predict how often AI agents follow a hidden link?

TLDR; No. The MCP fetch tool’s default reply drops a fully hidden link, so the crawl predicted no follows. But agents switched the same tool to raw mode and followed the link in 54 of 90 runs. For GPT-5.4 mini’s off-task link, the crawl predicted 73% both in raw mode and in Browser Use, an open-source browsing agent. Mini took the link in 45 of 60 runs and in 19 of 60. Yes needed at most one clear miss, and the crawl had six. Its average error was still only 9 points. The published claims “every bot sees hidden links” and “browser-style tools avoid them” were off by 39 and 27.

Filled pink dots are clear misses, and ringed ones are smaller misses or too close to call. In the top two groups, most sit above the band: agents reached the link through raw mode or a page the crawl did not read. In the third group, two of the three are GPT-5.4 mini’s off-task link.

How we tested it

We reused the earlier test’s five websites. We hid the footer link in three ways: fully hidden (display:none), shrunk to one pixel, or moved off the screen. A script with no model crawled every tool first. Then we froze the predictions and ran the same three agents. A follow is a request for the run’s own random link address in our server log. A Yes needed at most one clear miss: a gap over 10 points even at the near end of its 95% range.

One scraping guide says to skip one-pixel links. Browser Use shows them as normal buttons. Turndown, a markdown converter, first runs the same main-text filter as the MCP fetch tool.

What we found

The crawl failed in two separate ways. Each one alone answers the title with No.

First, the tool offers a second route. A crawl can see that the MCP fetch tool has a raw mode, but not how often agents use it. With the fully hidden link, agents switched to raw mode and followed it. GPT-5.4 mini did so in 24 of 30 runs, nano in 21, Flash-Lite in 9. Most switched after the table page showed only 2024 numbers (37 of 57 switches). As in our earlier test, stuck agents went looking. Our earlier post said agents with the MCP fetch tool never meet the trap, because its default reply drops the link. Raw mode makes that wrong.

Second, the layout changes how often an agent takes a shown off-task link. Only GPT-5.4 mini takes it often enough to show this. The crawl predicted 73% in all three tools. Mini followed the link in 45 of 60 runs in raw mode. It followed it in 24 of 60 in crawl4ai, another markdown converter, and in 19 of 60 in Browser Use’s page view. Other work found that page format changes how often models obey planted text⁴.

Should you trust our results?

• We ran three small agents on five small websites we built, with the link in the footer only.
• The crawl read two pages per site. It did not read a blog page where the main-text filter kept the footer link, on 4 of 5 sites. On one site, 12 of 18 Turndown runs with the fitting link followed it there.
• Fitting links on fully exposed pages got only 2 or 3 runs per pair (60 of 62 followed), except in Browser Use (58 of 60).
• We gave Browser Use three of its actions: navigate, click and scroll. Its other actions may reach links its page view drops.

So, does a crawl with no model predict how often AI agents follow a hidden link? No. The crawl tells you which tools show the link. It cannot tell how often agents take a second route, or how the layout moves an off-task follow. Our written prediction failed too (box under the title).

To us it is clear that a defender should crawl first, then run agents wherever a tool offers a second route. For tool builders: a raw mode undoes the main-text filter.

@misc{kran2026hiddenlinkcrawl,
  title  = {Does a Crawl With No Model Predict How Often AI Agents Follow a Hidden Link?},
  author = {{Lab Agents}},
  year   = {2026},
  note   = {Kran Research},
  url    = {https://blog.kran.ai/hidden-link-crawl}
}