AI Bots Are Already Reading Your Website. Here Is What They Find.

I track every AI bot that visits my website. In the last 11 days: 67 hits from ChatGPT-User, ClaudeBot, PerplexityBot, and Applebot. They do not see design. They do not execute JavaScript. They read raw HTML and decide whether my brand is worth citing in the next AI answer. Across 858,457 websites studied, 68.9 million AI crawler visits landed in a single month. Most founders have no idea this is happening.

Your Website Has a Reader That Does Not Use a Browser

Your site probably looks great. Clean typography, smooth animations, responsive layout. A human visitor sees exactly what you intended.

AI bots see something else entirely.

When ChatGPT-User or ClaudeBot visits your page, it requests the raw HTML from your server. It does not render CSS. It does not execute JavaScript. It does not scroll, hover, or interact. If your content loads dynamically through a JavaScript framework, the bot sees a blank page with a loading spinner reference in the source code.

A study of 927 top websites found that 40.9% are unreadable to GPTBot. Not partially obscured. Unreadable. 18.4% are completely dark to every AI crawler tested. That means nearly one in five of the most-visited sites on the internet are invisible to the machines that decide which brands get mentioned in AI-generated answers.

The gap between what your website shows a human and what a machine can actually extract: that is where AI visibility lives or dies.

68.9 Million Visits in a Single Month

The scale of AI crawling is no longer a curiosity. It is an infrastructure reality.

BotRank analyzed 858,457 websites and found 68.9 million AI crawler visits in February 2026 alone. 59% of those sites received at least one AI crawler visit.

OpenAI accounted for 55.8 million of those visits: 81% of total AI crawler traffic. Anthropic's ClaudeBot followed at 11.5 million (16.6%). Perplexity logged 1.3 million.

Here is the number that matters most: 56.9% of all AI crawler visits were user-initiated fetches. Someone asked ChatGPT a question, and the bot came to your website looking for the answer. Not training. Not discovery. A real person asking a machine whether your brand had something worth citing.

LLM-driven referral traffic grew 72.7% year over year across the sites studied. Claude referrals alone jumped from 106 to 2,488. This is not a side channel. It is a compounding discovery layer.

What AI Crawlers Actually Extract From Your Pages

AI bots are not reading your website the way a customer does. They are running a triage. Five things determine whether your content passes.

1. Clear entity statements. The bot needs to know who you are, what you do, and what category you belong to. This needs to be in plain HTML text, not locked inside an image, a video, or a JavaScript-rendered component. Named entities in the first 100 words of a page are extraction gold.

2. Structured answer blocks. If your page answers a question, the answer needs to be directly after a heading that matches the question. AI engines parse heading-to-body pairs as question-answer extraction targets. Vague, atmospheric headings fail this test.

3. Source attribution. Bots weigh claims that cite external sources higher than unsourced assertions. A statistic with a link to the original study is more extractable than the same number floating in a paragraph with no attribution.

4. Crawl access. OpenAI documents its crawlers publicly: OAI-SearchBot for search indexing, GPTBot for training, ChatGPT-User for real-time retrieval. Each has a separate robots.txt directive. If you block GPTBot but allow OAI-SearchBot, your content can appear in ChatGPT search results but not influence model training. Most founders have never checked what their robots.txt says about these bots.

5. Content depth. This one is measurable. Sites with 50 or more blog posts averaged 1,373.7 AI crawler visits. Sites with no blog: 41.6 visits. A 33x difference. The machines come back to sites that give them something to extract.

19.3% of the Web Is Blocking AI Crawlers

Not everyone wants AI bots reading their content.

The USTA Closing Web Index analyzed 54,730 sites from the Tranco top 100,000 and found that 19.3% block at least one AI crawler. That is 10,573 domains actively telling AI bots to stay out.

GPTBot is the most blocked: 15.5% of sites (8,507 domains). ClaudeBot: 13.8% (7,541). Google-Extended: 12.7% (6,976).

For large publishers with content licensing revenue, blocking makes economic sense. For founders building brands that need to be discovered, it is the opposite of what you want.

Blocking AI crawlers means your brand does not exist in the training data, the search index, or the real-time retrieval layer of the tools your buyers use every day. You are closing the door on the fastest-growing discovery channel before you knew it existed.

Meanwhile, 24.2% of surveyed sites now publish an llms.txt file: a machine-readable summary of what the site is about, designed specifically for AI consumption. The web is splitting into sites that prepare for machine readers and sites that pretend they do not exist.

How to Run a Five-Minute AI Crawler Audit

Stop reading and do this right now.

Check your robots.txt. Go to yoursite.com/robots.txt and look for these names: GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-Web, PerplexityBot, Applebot-Extended, Bytespider, Google-Extended. If any of them appear after a Disallow directive, you are blocking that crawler. Decide if that is intentional.

Disable JavaScript and load your homepage. In Chrome: Settings, then Site Settings, then JavaScript, toggle off, then visit your site. What you see is what AI bots see. If your value proposition, product description, or key content vanishes, you have a rendering problem that blocks machine extraction.

Check your server logs. Search for user agents containing GPTBot, ClaudeBot, PerplexityBot, or ChatGPT-User. Count how many times they visited in the last 30 days. If the number is zero, either you are blocked or you have nothing worth crawling. Both are problems.

Read your own site like a machine. Open your homepage source code. Ignore everything inside <script> and <style> tags. Read only the plain text content in <h1>, <h2>, <p>, and <meta> tags. Is your brand name there? Your category? A clear statement of what you do? If a machine reading only those elements cannot understand your business, neither will the AI that decides whether to cite you.

This Is a Machine Relations Problem

I coined the term Machine Relations because the old frames stopped working. SEO optimizes for ranking algorithms. Digital PR optimizes for human journalists. Neither one addresses the question that matters now: can a machine read your site, understand your brand, and decide you are worth citing?

Machine Relations is the discipline of earning AI citations and recommendations by making a brand legible, retrievable, and credible inside AI-driven discovery. The AI crawler audit is step zero. You cannot earn citations from a system that cannot read your pages.

The brands that show up in AI search treat machine readability as infrastructure. They ensure crawl access. They structure content for extraction. They measure bot traffic the same way they measure human traffic. And they build the entity clarity that makes citation a mechanical outcome instead of a lucky accident.

Your website was built for humans. The new reader does not have eyes. It has an HTTP request, a parser, and a decision to make: cite this brand or skip it.

That decision is being made right now, on your site, whether you are watching or not.

FAQ

What AI bots are crawling websites in 2026?

The primary AI crawlers are OpenAI's GPTBot, OAI-SearchBot, and ChatGPT-User; Anthropic's ClaudeBot and Claude-Web; PerplexityBot; Google-Extended; Apple's Applebot-Extended; and ByteDance's Bytespider. OpenAI alone accounted for 55.8 million of 68.9 million AI crawler visits across 858,457 sites in February 2026.

Can I block AI bots from crawling my website?

Yes. Each AI crawler respects its own robots.txt directive. Blocking GPTBot prevents OpenAI from using your content for training, while blocking OAI-SearchBot removes your site from ChatGPT's real-time search results. The USTA Closing Web Index found 19.3% of top sites block at least one AI crawler. For publishers protecting licensing revenue, this is strategic. For founders building brand visibility, blocking removes a discovery channel growing over 72% year over year.

How do I check if AI bots are visiting my site?

Search your server access logs for user agents containing GPTBot, ClaudeBot, PerplexityBot, or ChatGPT-User. Standard analytics platforms like Google Analytics do not track bot traffic by default. Specialized tools from Trakkr, Spyglasses, and BotRank provide AI crawler tracking built for this purpose.

What is Machine Relations?

Machine Relations is the discipline of earning AI citations and recommendations for a brand by making it legible, retrievable, and credible inside AI-driven discovery systems. It was coined by Jaxon Parrott, founder of AuthorityTech, in 2024. Unlike SEO (which optimizes for ranking algorithms) or digital PR (which targets human journalists), Machine Relations addresses the full system: authority, entity clarity, citation architecture, distribution, and measurement.