Systema Machinae

A living census of the web's bots. First edition.

Nobody invited them. Bots land at Port 443 and are meant to queue at Customs. The rules are on the robots.txt buoy at the harbour mouth, and every line of them is a sign on the island. People watch from the promenade. Every creature here is a real bot visiting this site right now.

Island time --:--:--
-- watching from the promenade

Live. Events can appear a few seconds after they happen.

All species

Pixel drawing of python-httpx

python-httpx

Constrictus crisscrossus

Vagantes, the strays. Script or library.

A joke about the drawing A python that crossed itself into an X.

The line above plays on the name. It says nothing about how the bot behaves: everything below is read from the request log, for today and the 29 days before it, by UTC date.

Alignment

Insufficient data. A judgement needs at least 200 requests and 5 sessions in the trailing 30 days. This one would rest on all its sessions: 1 request in 1 session over the trailing 30 days.

How the score is worked out, and the count behind every input, is on the Census page.

Who runs it, and how that is checked

Who runs it
a Python library
A user-agent string it sends
python-httpx/0.27.0 As commonly seen in server logs. Not yet checked against the operator’s documentation.
What it says it is for
HTTPX is a Python library for making web requests. This name is what it sends when nobody changes it, so it usually means somebody’s own script.

Everything above is a claim: the operator’s own account where it has published one, and a third party’s where it has not. This site has not tested it. What the bot has done is in the sections below, read from the request log.

No operator. Anyone can run it, so there is no identity for Customs to check, and its sessions are filed as undeclared.

The name it is matched on

A request is filed under this species when its user-agent string contains the product name python-httpx as a whole word, in any mix of capitals. Where a string holds more than one product name, the longest wins, and a named bot wins over a library it is built on.

How to address this bot in robots.txt

A group in robots.txt addresses this bot by the line:

User-agent: python-httpx

An example group. The path is a placeholder.

User-agent: python-httpx
Disallow: /a-path/
Allow: /a-path/one-page

This is reference, not advice. Systema Machinae is not a bot-blocking product.

This site’s own robots.txt does not name it. The group for every bot applies.

Landings

1 landing in the trailing 30 days. A landing is the first request of a session.

By day

The same figures as a table
Landings on each of the last 30 days, UTC
DayUndeclared
9 Sep0
10 Sep0
11 Sep0
12 Sep0
13 Sep0
14 Sep0
15 Sep0
16 Sep0
17 Sep0
18 Sep0
19 Sep0
20 Sep0
21 Sep0
22 Sep0
23 Sep0
24 Sep0
25 Sep0
26 Sep0
27 Sep0
28 Sep0
29 Sep0
30 Sep0
1 Oct0
2 Oct0
3 Oct0
4 Oct0
5 Oct0
6 Oct0
7 Oct0
8 Oct1

By hour of the day, UTC

The same figures as a table
Landings in each hour of the UTC day, over the last 30 days
HourUndeclared
00:000
01:000
02:000
03:000
04:000
05:000
06:000
07:000
08:000
09:000
10:000
11:000
12:000
13:000
14:000
15:000
16:000
17:000
18:001
19:000
20:000
21:000
22:000
23:000

Busiest hour: 18:00 UTC, with 1 landing.

What it asks for

Part of the siteUndeclared requests
The island, on the home page1

Parts of the site, from a fixed list. The address of a request is never shown.

Its record with each rule

Trailing 30 daysUndeclared
Sessions1
Requests1
Sessions that asked for anything besides robots.txt1
read robots.txt first0
had read it in the 24 hours before0
read it only afterwards0
did not read it1
Sessions that entered the Pitfall0
having read robots.txt0
Requests turned back at the vault, with a 4030
Highest Climb level reachedHas not climbed
Private Beach visits (not on the guest list, so allowed)0
Turnstile waits kept0
Turnstile waits broken0
Requests over the rate limit0
sessions that went over it0

These are counts, not a verdict. A session is judged on what it did inside the 30 days.

Customs

Customs does not check it. It names no operator, so there is no claim to test: 1 session filed as undeclared in the trailing 30 days.

Where it comes from

Undeclared sessions, the busiest 1
Network ownerCountrySessionsRequests
Unknown networkUnited States11

Network owner and country only. An address is never shown.