Want to stop AI companies from using your website to train their models? You can block AI crawlers with a few lines in your robots.txt file. This guide shows the rules for the most common AI bots, how to add them in WordPress, and the mistakes that can hurt your Google rankings.
Why block AI crawlers?
Some site owners are happy for AI tools to learn from their content, while others want to keep control over how their work is used. The big AI companies have now agreed to back a shared robots.txt signal for training preferences, which makes this topic more important. You can read the background in our article on the new AI training opt-out signal.
Common AI crawlers and their user agents
- GPTBot: OpenAI’s crawler used to gather training data.
- ClaudeBot: Anthropic’s crawler.
- Google-Extended: a Google control token that tells Google whether your content can be used for its AI models. It is separate from Google Search.
- CCBot: the Common Crawl crawler, whose data is widely used to train models.
- PerplexityBot: the crawler of the Perplexity answer engine.
- Applebot-Extended: Apple’s control for AI training use.
Bot names change, so always check each company’s documentation for the current list before you publish your rules.
Copy-paste robots.txt rules
Add these lines to block the crawlers above from your whole site:
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: Google-Extended
Disallow: /
User-agent: CCBot
Disallow: /
User-agent: PerplexityBot
Disallow: /
User-agent: Applebot-Extended
Disallow: /To block only part of your site, replace the slash with a folder, for example Disallow: /members/.
How to add the rules in WordPress
- If you use Rank Math, go to Rank Math, then General Settings, then Edit robots.txt. Yoast and other SEO plugins have a similar editor.
- Paste the new rules below your existing ones and keep your sitemap line.
- Save the changes and open yourdomain.com/robots.txt to check that the new lines appear.
- If your changes do not show, look for a physical robots.txt file in your site’s root folder. A real file overrides the plugin’s version, so edit or delete that file instead.
Mistakes to avoid
- Do not block Googlebot or Bingbot. Blocking search crawlers removes your pages from search results.
- Do not confuse training with search. Some companies use separate bots for AI search and AI training, so read each company’s documentation if you still want to appear in AI search answers.
- Do not use robots.txt for private data. It is a polite request that well-behaved bots follow. Protect sensitive pages with passwords or server rules.
- Do not copy rules blindly. A small typo, such as a missing slash, can block more than you intended.
How to test your robots.txt
- Open your robots.txt in a browser and read it line by line.
- Use the robots.txt report in Google Search Console to confirm Google can read the file.
- Check your server logs a week later to see whether the blocked bots have stopped visiting.
Frequently asked questions
Will blocking AI crawlers hurt my SEO?
Blocking AI training crawlers such as GPTBot or Google-Extended does not remove you from Google Search. Blocking Googlebot would.
Does every AI company obey robots.txt?
No. Major companies say they respect it, but robots.txt cannot force a bot to comply.
Is there one rule that covers every AI bot?
Not yet. A shared signal has been proposed and is being adopted by some companies, but for now separate rules are still the safest approach.
Sources
Published 12 November 2026. Bot names and company policies change, so confirm details with each company before relying on them.
