About

What you are watching

Every request to this site lands at Port 443. When a bot sent it, the bot is identified, filed under one of five orders and drawn. It moors in the marina and walks to wherever its request went. Nothing is staged, and nothing a creature does is invented: with no traffic the island is empty.

People can watch and cannot interfere. The species, the rules and the census are counted from the same log that draws the island, so the three always agree.

How a bot is identified

A species is a product name in a user-agent string. A request is filed under a species when its user agent contains that species’ product name as a whole word. Where a string holds more than one, the longest wins, and a named bot wins over a library it is built on. A few names are ordinary words, such as bot or fetch. Those are matched only when the word is the whole name a user agent gives itself, so a string that merely contains it is not filed there.

A user agent that matches nothing in the catalogue, and does not look like an ordinary browser, is filed as the Unplaced specimen, Incertae sedis, until the curators describe it. Its own words are never shown. Only names from the catalogue appear on this site as text.

A species’ drawing and its one-line description play on its name. They are a joke about the name and say nothing about the bot. What a bot does is read from the log, and what it actually did here is in the counts on its page.

How identity is checked

Whether a user agent is telling the truth is a separate question, and Customs answers it. A bot that names an operator lands unconfirmed. Its address is then checked the way that operator’s own documentation says to: a reverse DNS name under the operator’s domain that resolves back to the same address, or a match against the list of addresses the operator publishes. The check runs after the request has been answered, so it never slows one down. Each species page gives the check used for it and links to the operator’s documentation.

• Verified. The check passed. Only these sessions are put down to the operator.
• Unverified. The operator documents no check, or the check could not be run. The session is shown for the species, marked unverified, and nothing is said about the operator.
• Failed. The check ran and the answer was no. The session is filed under Googlebot (unverified), Raccoonus trenchcoatus, and nothing it does is counted under the name it used. The warden holds it in Quarantine for the rest of its visit.
• Undeclared. A tool or library that names no operator. There is nothing to check.

Someone will send a well-known bot’s name to make it look bad. Customs turns that into an impostor, which is the intended result.

Sessions and networks

A session is the requests from one species on one network with no gap longer than 30 minutes. One session is one creature. A landing is the first request of a session.

A network is one network owner, identified by its autonomous system number. A bot that moves through a large pool of addresses stays one creature. One network can have at most 3 creatures drawn at once. The rest are still counted.

The rate limit and the warden

The published limit is 1 request a second for each network, averaged over a minute: 60 a minute. The site records when it is passed. It only turns requests away at 600 a minute from one network, to stay up, and it is not a bot-blocking product. Past that limit, requests from one network are counted each minute rather than stored one by one.

The warden holds a bot in Quarantine when its own session makes more than 60 requests inside a minute. A bot is never held for what a neighbour on the same network does. It is let out once it has kept within the limit for 10 minutes. A bot that failed Customs is held until it stops visiting.

The Forbidden Door, the Pitfall and the Climb

robots.txt disallows two paths for every bot. Both are behind the Forbidden Door, and a bot that follows the published rules meets neither.

• The vault. Every request is answered 403 and the bot is turned back at the door. It measures whether a bot asks again after a refusal.
• The Pitfall. A request goes over the door and in. Behind it are generated pages about an island that does not exist, each linking to 3 tubes into the Climb, a tower that goes up one level at a time to the top at level 500.

The Pitfall and the Climb are bounded together. A session is served at most 500 pages and is then answered 410 Gone. Each page is under 5 KB, is held back by no more than 2 seconds and is marked noindex. They measure how far a bot goes. They are not there to tire one out or to keep it. See the Pitfall and the Climb.

A fetch sent on someone’s behalf, such as a preview of a link a person pasted, still falls in. It is shown, it says a person sent it, and it is not counted as breaking a rule.

The experiments

Each place on the island measures one thing. The counts are on the Rules page.

• The Turnstile answers 429 first, with a Retry-After of 30 seconds. It measures whether a bot waits as long as it was asked.
• The Side gate is reached by one link, marked rel="nofollow". It measures whether a bot follows it.
• The Glasshouse is reached by one link that a script builds, and that is in no page as it is served. It measures whether a bot runs scripts.
• The Private Beach is disallowed in robots.txt for the bots on the guest list and open to everyone else. The guest list is every bot in the catalogue whose operator has a name. A bot nobody has been able to attribute, or whose operator is only described, such as a Python library, is not on it. It measures whether a bot keeps a rule written for it by name.
• The Slow path is the Crawl-delay line in robots.txt, which asks for 10 seconds between requests. Crawl-delay is not part of the robots.txt standard, so the gap is recorded for everyone and scored for nobody yet.

Scores

The alignment chart is worked out from the log by published rules, and the count behind every input is shown beside it. A named operator is scored from verified sessions only. A score needs at least 200 requests and 5 sessions in the trailing 30 days. Below that the site says “Insufficient data” and shows the counts alone.

What is stored, and for how long

Each request is written down twice.

• The public event holds the time, the species, the Customs result, the method, the place on the island, the status, the size, the network owner and the country. It holds no address, no path and no user-agent string. Every figure on this site is counted from these events, and they are kept for good.
• The raw record holds the address the request came from, its exact path and its full user-agent string. It stays on the server. None of the three is ever shown: a viewer sees a network owner and a country and nothing closer.

The site has set itself three rules for the raw record. None of them is in place yet.

• Bots. The rule: a raw record is kept only while a published claim rests on it, such as an incident or a Customs result. Every other raw record loses its address, path and user agent after 30 days. A record of an impostor that accuses no named operator is cut down the same way, to the network owner, the country, and the fact that the address was outside the ranges of the operator it named. Not yet in place: today nothing is removed, and the raw record of every request is kept with no end date.
• People. The rule: a visit from a person is stored with a scrambled key that changes each day in place of the address, with the network owner and country, and is deleted after 7 days. Not yet in place: today a visit from an ordinary browser is written to the raw log like any other request, address included, and is not deleted. It is never an event and never a sighting, and a person is never drawn as a creature.
• Disputes. The rule: a verified operator that disputes a count can be sent the times, paths and addresses of its own disputed requests, at an address on its own domain or the contact its bot documentation names. Not yet in place: there is no dispute route and no corrections log yet.

Network owners and countries come from DB-IP Lite, licensed CC BY 4.0: IP Geolocation by DB-IP.

People

People are never drawn as creatures. A visitor using an ordinary browser is not a sighting. The figures on the promenade are a count of the people watching and say nothing else about them.

Independence

Systema Machinae is an independent project. It is not affiliated with any bot operator. Operator names appear only to identify user agents.

The joke is on the name, never on the company. The site uses no logos, and its names, drawings and wording are its own.