The Pitfall

The rules ask every bot to keep out of one path. On the island it is the Forbidden Door, with a drop behind it.

How it works

robots.txt disallows the Pitfall for every bot. The way in is a plain link in the footer of every page, marked rel="nofollow". Nothing about it is hidden from people. A bot reaches it by not reading the rules, or by reading them and going in anyway.

Behind the door are generated pages about an island that does not exist. They are bounded: a session is served at most 500 of them and is then answered 410 Gone, each page is under 5 KB, each is held back by no more than 2 seconds, and every one is marked noindex. The point is to count, not to keep anything. A bot that follows the rules never sees one.

The other disallowed path is the vault. Every request there is answered 403, and the bot is turned back at the door.

Fallen in, trailing 30 days

Sessions that went in

0

Having read robots.txt

0

Without reading it

0

Sent on someone’s behalf, not counted

0

A fetch sent on someone’s behalf, such as a preview of a link a person pasted, still falls in. It is shown and is not counted as breaking a rule.

A ladder from the Pitfall leads up to the Climb.

Latest falls

Nothing has fallen in yet.

Impostors, and where they came from

Each of these used the product name of an operator’s bot and failed that operator’s own check. They are filed under Googlebot (unverified), and nothing they did is counted under the name they used.

No impostor has been unmasked yet.

Species that fell more than once

No species has fallen in more than once yet.