AI & Searchgrowth
AI Crawler Traffic
See which AI bots and agents actually visit your site
What it does
AI Crawler Traffic reads your server log (Apache, Nginx, Vercel or JSON) and finds visits from AI bots by their User-Agent. The bot list follows the providers' own documentation, and each bot is sorted by what it does: user-triggered agents such as ChatGPT-User and Claude-User fetch a page because a person asked something, AI search indexes for citations, and AI training collects content for models. You see requests per bot, channel, page and day, how many fetched llms.txt, and a check of the agent infrastructure on the domain (llms.txt, robots.txt, structured data and API). The analysis runs in your browser and the log is not stored by us. GA4 does not see bots, because they do not run JavaScript; that is why the log is needed.
The Problem
AI bots and agents do not show up in Google Analytics.
- Bots do not run JavaScript, so GA4 does not count them
- Unclear whether ChatGPT, Claude and Perplexity actually fetch your pages
- Hard to tell user-triggered visits from crawling for training
The Solution
Your server log records every visit, bots included.
- Recognises AI bots by the providers' documented User-Agent
- Channels: user-triggered agents, AI search, AI training and search engine
- Which pages the bots visit, and who fetches llms.txt
- A check of the agent infrastructure on the domain in the same run
Features
- User-triggered agents: Visits from ChatGPT-User, Claude-User, Perplexity-User and others that fetch a page because a person asked something.
- Channel breakdown: Requests and share per channel, with the bots behind them.
- Pages and llms.txt: The most visited pages per bot, and how many fetches of llms.txt and llms-full.txt.
- Agent infrastructure: llms.txt, robots.txt, structured data and API checked against your domain.
How it works
- Upload the log: Upload or paste an access log from Apache, Nginx, Vercel or in JSON format, and enter the domain.
- Bots are recognised: Each line is checked against a bot list verified against the providers' documentation. Ordinary browsers and generic crawlers are not counted.
- Report: You see requests per bot, channel, page and day, llms.txt fetches and the infrastructure check.
Use cases
- See whether AI assistants use your content: User-triggered agents show that someone actually asked an AI something that led to your site.
- Before changing robots.txt: See which bots come before you block or allow them.
FAQ
- Why is the server log needed?
- AI bots do not run JavaScript, so they do not show up in Google Analytics. The server log records every visit, bots included.
- Is the log stored?
- No. The log is read in your browser. Only the report is saved, in this browser, so it survives a page refresh.
- Which bots are recognised?
- The bots of OpenAI, Anthropic, Perplexity, Meta, Mistral, DuckDuckGo, Apple, Google (Vertex), ByteDance, Common Crawl and Amazon, plus Bingbot, according to the providers' own documentation.
- How many credits does it cost?
- 20 credits per analysis, including the agent infrastructure check.
Try AI Crawler Traffic free