Systema Machinae

A living census of the web's bots. First edition.
Nobody invited the bots. They arrive at Port 443, where they’re supposed to queue at Passport Control. Some don’t. The robots.txt buoy at the harbour mouth states the rules. The island shows which bots read them and which don’t obey. You watch from the promenade. Every creature is a real bot visiting this website.

Nobody invited the bots. They arrive at Port 443, where they’re supposed to queue at Passport Control. Some don’t. The robots.txt buoy at the harbour mouth states the rules. The island shows which bots read them and which don’t obey. You watch from the promenade. Every creature is a real bot visiting this website.

All species

Pixel drawing of Bingbot

Bingbot

Bellus dingdongus

A brass bell on legs. You hear it before you see it.*

robots.txtFOLLOWED
Has read robots.txt and has not asked for a path it disallows.
AlignmentInsufficient data
Needs 200 requests and 5 sessions in 30 days.†

What it did here‡

  • VerifiedPassed Passport Control.
  • UnverifiedSaid they were Bingbot and could not be checked, or have not been answered yet. Shown for the species, and not put down to Microsoft Bing.
  • Failed Passport ControlFiled under Impostor and not on this page at all.

Its record with each rule

Trailing 30 daysVerifiedUnverified
Sessions10
Requests90
Sessions that asked for anything besides robots.txt10
read robots.txt first10
had read it in the 24 hours before00
read it only afterwards00
did not read it00
Sessions that asked for a disallowed path00
Sessions that entered the Pitfall00
having read robots.txt00
Pages taken in the Pitfall and the Climb00
Requests turned back at the vault, with a 40300
Highest Climb level reachedHas not climbedHas not climbed
Private Beach visits while on the guest list00
Turnstile waits kept00
Turnstile waits broken00
Sessions served a page that carries the nofollow link10
followed it00
Requests over the rate limit00
sessions that went over it00

What it asks for§

Part of the siteVerified requestsUnverified requests
The island, on the home page20
Rules pages10
Census pages10
The About page10
The Support page10
robots.txt, on the buoy20
The Turnstile10

Landings, last 30 days

1 landing in 30 days

10 Oct 1 verified, 0 unverified

VerifiedUnverified

Landings on each of the last 30 days, UTC
DayVerifiedUnverifiedTotal
11 Sep000
12 Sep000
13 Sep000
14 Sep000
15 Sep000
16 Sep000
17 Sep000
18 Sep000
19 Sep000
20 Sep000
21 Sep000
22 Sep000
23 Sep000
24 Sep000
25 Sep000
26 Sep000
27 Sep000
28 Sep000
29 Sep000
30 Sep000
1 Oct000
2 Oct000
3 Oct000
4 Oct000
5 Oct000
6 Oct000
7 Oct000
8 Oct000
9 Oct000
10 Oct101

Favourite hours, UTC

16:00 1 verified, 0 unverified

VerifiedUnverified

Landings in each hour of the UTC day, over the last 30 days
HourVerifiedUnverifiedTotal
00:00000
01:00000
02:00000
03:00000
04:00000
05:00000
06:00000
07:00000
08:00000
09:00000
10:00000
11:00000
12:00000
13:00000
14:00000
15:00000
16:00101
17:00000
18:00000
19:00000
20:00000
21:00000
22:00000
23:00000

Requests, last 30 days

9 requests in 30 days

10 Oct 9 verified, 0 unverified

VerifiedUnverified

Requests on each of the last 30 days, UTC
DayVerifiedUnverifiedTotal
11 Sep000
12 Sep000
13 Sep000
14 Sep000
15 Sep000
16 Sep000
17 Sep000
18 Sep000
19 Sep000
20 Sep000
21 Sep000
22 Sep000
23 Sep000
24 Sep000
25 Sep000
26 Sep000
27 Sep000
28 Sep000
29 Sep000
30 Sep000
1 Oct000
2 Oct000
3 Oct000
4 Oct000
5 Oct000
6 Oct000
7 Oct000
8 Oct000
9 Oct000
10 Oct909

Where it comes from§

Verified sessions, the busiest 1
Network ownerCountrySessionsRequests
Microsoft CorporationUnited States19

What it has cost this site‡

TodayLast 7 daysAll time
Requests999
Bytes served277.5 KB277.5 KB277.5 KB

Who it is

Microsoft

Operator: Microsoft Bing

Bingbot is the bot behind Microsoft Bing. It fetches pages so they can appear in Bing’s results.

How Microsoft Bing says to check it

Reverse lookup to a name ending in search.msn.com, then forward lookup back to the same address. Or match the list of Bingbot addresses, refreshed daily. The operator’s documentation

Passport Control

A session that says it is Bingbot is checked in the way Microsoft Bing documents. Passing any one check is enough.

  • Address list A match against the list of addresses the operator publishes. The published list.
  • Reverse DNS Forward-confirmed reverse DNS: the address has a reverse DNS name under the operator’s domain, and that name resolves back to the same address. Names must end in search.msn.com.

Sessions that said they were this bot

1 in the trailing 30 days
  • Passed: verified1
  • Could not be checked: unverified0
  • Not yet answered: counted for nobody0
  • Failed, and filed under Impostor0

Reference

The user agent it sends, and how to address it in robots.txt.

User agent, and how a request is filed under this species

Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)

A string it is known to send. Parts that vary are shown as an ellipsis.

A request is filed under this species when its user-agent string contains the product name bingbot as a whole word, in any mix of capitals. Where a string holds more than one product name, the longest wins, and a named bot wins over a library it is built on.

How to address this bot in robots.txt

The line that addresses it

User-agent: bingbot

An example group. The path is a placeholder.

User-agent: bingbot
Disallow: /a-path/
Allow: /a-path/one-page

This is reference, not advice. Systema Machinae is not a bot-blocking product.

This site’s own robots.txt names it, in the group that keeps the guest list off the Private Beach.

Notes

  1. *The line under the name is a joke about the drawing. It plays on the name and says nothing about how the bot behaves.
  2. †Insufficient data. A judgement needs at least 200 requests and 5 sessions in the trailing 30 days. This one would rest on verified sessions only, so it is a statement about Microsoft Bing’s bot: 9 requests in 1 session over the trailing 30 days. The stamp is counted over the same sessions as the score. How it is scored
  3. ‡Counted from the request log for today and the 29 days before it, by UTC date. These are counts, not a verdict: a session is judged on what it did inside the 30 days. A landing is the first request of a session. Bytes are the size of the response body, counted whatever the score.
  4. §Parts of the site come from a fixed list, so a path is never shown. Networks are shown by owner and country only, so an address is never shown.
  5. What it is and who runs it, under Who it is, is a claim: the operator’s own account where it has published one, and a third party’s where it has not. This site has not tested it.