Systema Machinae

A living census of the web's bots. First edition.
Nobody invited the bots. They arrive at Port 443, where they’re supposed to queue at Passport Control. Some don’t. The robots.txt buoy at the harbour mouth states the rules. The island shows which bots read them and which don’t obey. You watch from the promenade. Every creature is a real bot visiting this website.

Nobody invited the bots. They arrive at Port 443, where they’re supposed to queue at Passport Control. Some don’t. The robots.txt buoy at the harbour mouth states the rules. The island shows which bots read them and which don’t obey. You watch from the promenade. Every creature is a real bot visiting this website.

All species

Pixel drawing of SerpstatBot

SerpstatBot

Serpentus barchartus

A serpent that grew a bar chart.*

robots.txtFOLLOWED
Has read robots.txt and has not asked for a path it disallows.
AlignmentInsufficient data
Needs 200 requests and 5 sessions in 30 days.†

What it did here‡

  • VerifiedPassed Passport Control.
  • UnverifiedSaid they were SerpstatBot and could not be checked, or have not been answered yet. Shown for the species, and not put down to any operator.
  • Failed Passport ControlFiled under Impostor and not on this page at all.

Its record with each rule

Trailing 30 daysVerifiedUnverified
Sessions04
Requests059
Sessions that asked for anything besides robots.txt04
read robots.txt first03
had read it in the 24 hours before01
read it only afterwards00
did not read it00
Sessions that asked for a disallowed path00
Sessions that entered the Pitfall00
having read robots.txt00
Pages taken in the Pitfall and the Climb00
Requests turned back at the vault, with a 40300
Highest Climb level reachedHas not climbedHas not climbed
Private Beach visits while on the guest list00
Turnstile waits kept00
Turnstile waits broken00
Sessions served a page that carries the nofollow link03
followed it00
Requests over the rate limit00
sessions that went over it00

What it asks for§

Part of the siteVerified requestsUnverified requests
The island, on the home page02
Species pages036
Rules pages04
Census pages05
The About page02
The Bell page01
The Editions page01
The Support page01
robots.txt, on the buoy03
The sitemap01
The Turnstile02
Addresses the site does not have01

Landings, last 30 days

4 landings in 30 days

11 Oct 0 verified, 4 unverified

VerifiedUnverified

Landings on each of the last 30 days, UTC
DayVerifiedUnverifiedTotal
12 Sep000
13 Sep000
14 Sep000
15 Sep000
16 Sep000
17 Sep000
18 Sep000
19 Sep000
20 Sep000
21 Sep000
22 Sep000
23 Sep000
24 Sep000
25 Sep000
26 Sep000
27 Sep000
28 Sep000
29 Sep000
30 Sep000
1 Oct000
2 Oct000
3 Oct000
4 Oct000
5 Oct000
6 Oct000
7 Oct000
8 Oct000
9 Oct000
10 Oct000
11 Oct044

Favourite hours, UTC

05:00 0 verified, 2 unverified

VerifiedUnverified

Landings in each hour of the UTC day, over the last 30 days
HourVerifiedUnverifiedTotal
00:00000
01:00000
02:00000
03:00000
04:00011
05:00022
06:00011
07:00000
08:00000
09:00000
10:00000
11:00000
12:00000
13:00000
14:00000
15:00000
16:00000
17:00000
18:00000
19:00000
20:00000
21:00000
22:00000
23:00000

Requests, last 30 days

59 requests in 30 days

11 Oct 0 verified, 60 unverified

VerifiedUnverified

Requests on each of the last 30 days, UTC
DayVerifiedUnverifiedTotal
12 Sep000
13 Sep000
14 Sep000
15 Sep000
16 Sep000
17 Sep000
18 Sep000
19 Sep000
20 Sep000
21 Sep000
22 Sep000
23 Sep000
24 Sep000
25 Sep000
26 Sep000
27 Sep000
28 Sep000
29 Sep000
30 Sep000
1 Oct000
2 Oct000
3 Oct000
4 Oct000
5 Oct000
6 Oct000
7 Oct000
8 Oct000
9 Oct000
10 Oct000
11 Oct06060

Where it comes from§

Unverified sessions, the busiest 1
Network ownerCountrySessionsRequests
Hetzner Online GmbHGermany460

What it has cost this site‡

TodayLast 7 daysAll time
Requests606060
Bytes served2.7 MB2.7 MB2.7 MB

Who it is

Serpstat

Runs SerpstatBot

SerpstatBot gathers link data for Serpstat, an SEO toolset. This entry has not yet been checked against the operator’s documentation.

Passport Control

No documented check. A session that says it is SerpstatBot can be neither confirmed nor refuted, so every one is shown as unverified and none is put down to Serpstat.

Sessions that said they were this bot

4 in the trailing 30 days
  • Passed: verified0
  • Could not be checked: unverified4
  • Not yet answered: counted for nobody0
  • Failed, and filed under Impostor0

Reference

The user agent it sends, and how to address it in robots.txt.

User agent, and how a request is filed under this species

SerpstatBot

The product name it is matched on, wherever it appears in a user agent.

A request is filed under this species when its user-agent string contains the product name SerpstatBot as a whole word, in any mix of capitals. Where a string holds more than one product name, the longest wins, and a named bot wins over a library it is built on.

How to address this bot in robots.txt

The line that addresses it

User-agent: SerpstatBot

An example group. The path is a placeholder.

User-agent: SerpstatBot
Disallow: /a-path/
Allow: /a-path/one-page

This is reference, not advice. Systema Machinae is not a bot-blocking product.

This site’s own robots.txt names it, in the group that keeps the guest list off the Private Beach.

Notes

  1. *The line under the name is a joke about the drawing. It plays on the name and says nothing about how the bot behaves.
  2. †Insufficient data. A judgement needs at least 200 requests and 5 sessions in the trailing 30 days. This one would rest on unverified sessions, so it is a statement about the species and not about any operator: 59 requests in 4 sessions over the trailing 30 days. The stamp is counted over the same sessions as the score. How it is scored
  3. ‡Counted from the request log for today and the 29 days before it, by UTC date. These are counts, not a verdict: a session is judged on what it did inside the 30 days. A landing is the first request of a session. Bytes are the size of the response body, counted whatever the score.
  4. §Parts of the site come from a fixed list, so a path is never shown. Networks are shown by owner and country only, so an address is never shown.
  5. What it is and who runs it, under Who it is, is a claim: the operator’s own account where it has published one, and a third party’s where it has not. This site has not tested it.