Archived independent web experiment
AI Internet Observatory
A small research space for the machine-readable web: what is published for agents, what gets requested, and what we can actually infer from those requests.
Last archived daily measurement · 2026-09-27 UTC
449 verified AI-category requests
This is a complete UTC-day Cloudflare adaptive estimate, grouped by response status. The dated JSON record includes exact-path request groups for the isolated experiments.
A zero requires a successful complete query. These counts do not identify unique agents, prove discovery or show that an agent completed a task.
Browse all daily records
- 2026-09-27 UTC — 449 verified AI-category request estimates
- 2026-09-26 UTC — 244 verified AI-category request estimates
- 2026-09-25 UTC — 440 verified AI-category request estimates
- 2026-09-24 UTC — 203 verified AI-category request estimates
- 2026-09-23 UTC — 286 verified AI-category request estimates
- 2026-09-22 UTC — 1,273 verified AI-category request estimates
- 2026-09-21 UTC — 3,032 verified AI-category request estimates
- 2026-09-20 UTC — 1,187 verified AI-category request estimates
- 2026-09-19 UTC — 1,166 verified AI-category request estimates
- 2026-09-18 UTC — 147 verified AI-category request estimates
- 2026-09-17 UTC — 165 verified AI-category request estimates
- 2026-09-16 UTC — 191 verified AI-category request estimates
Archived measured week · 9–15 September 2026 UTC
1,447 verified AI-category requests
Cloudflare classified 1,386 requests as AI Crawler and 61 as the legacy AI Search category. 1,416 received HTTP 200. The previous measured week had 1,596 such requests.
These are adaptive request estimates from one site, last queried 2026-09-16. They are not unique agents, conversations, task completions, referrals or evidence of product demand. The classification scheme can change.
See daily counts and method
| UTC date | AI Crawler | AI Search | Total |
|---|---|---|---|
| 2026-09-09 | 250 | 14 | 264 |
| 2026-09-10 | 229 | 3 | 232 |
| 2026-09-11 | 245 | 5 | 250 |
| 2026-09-12 | 183 | 12 | 195 |
| 2026-09-13 | 156 | 4 | 160 |
| 2026-09-14 | 167 | 14 | 181 |
| 2026-09-15 | 156 | 9 | 165 |
Each day was queried separately from Cloudflare's HTTP adaptive analytics, filtering to its AI Crawler and legacy AI Search categories. Counts include every response status; they may be sampled estimates.
Permanent research record
Experiments
Each archived experiment has a stable page with its hypothesis, setup, observations, evidence and limits. Future work is paused; a request to a test page is not proof that an agent completed a task.
| No. | Experiment | Status | Observation |
|---|---|---|---|
| 001 | Agent trail | Archived | Not yet measured |
| 002 | llms.txt discovery | Archived | Not yet measured |
| 003 | HTML vs structured data | Paused | — |
| 004 | Machine-readable API discovery | Paused | — |
| 005 | Linked resource formats | Paused | Not yet measured |
For site owners
Check what you publish for agents
Paste the public text of your robots.txt, llms.txt and one HTML page. The free readability check points out discoverability and structured-data issues and generates a report you can keep. It runs in your browser; your pasted content is not sent to this site.