Robots.txt Builder for SEO and AI Crawlers
Build a valid robots.txt with presets, sitemap lines and one-click rules for AI crawlers like GPTBot and ClaudeBot, then test any URL against it.
Rules for all crawlers
Start from a presetAI crawlers
Quick choicesTicked crawlers get Disallow: / for the whole site.
robots.txt
Test a URL
Why robots.txt matters more than ever
robots.txt is a plain text file at the root of your domain that tells crawlers which parts of your site they may visit. It has always been a technical SEO basic for keeping admin areas, carts and endless filter URLs out of the crawl. Today it is also your main way to say yes or no to AI crawlers, from OpenAI, Anthropic, Perplexity, Google, Apple and others.
Training crawlers versus search crawlers
AI companies increasingly run separate crawlers for different jobs. Training crawlers such as GPTBot, ClaudeBot and Google-Extended collect content to train models. Search crawlers such as OAI-SearchBot, Claude-SearchBot and PerplexityBot index pages so the assistant can cite and link to them in answers. That split lets you choose: the "Block AI training only" option keeps you visible in AI search while opting out of model training.
Blocking Google-Extended does not affect Google Search rankings or your eligibility for AI Overviews, which use the regular Googlebot. Some providers also say their user-triggered fetchers, the ones that open a page because a person asked, may not follow robots.txt rules.
How to use the builder
- Pick a preset, then adjust the Disallow and Allow paths for your site.
- Add every sitemap URL, one per line.
- Choose which AI crawlers to block.
- Test a few important URLs with the tester. It follows the standard rules: the longest matching rule wins, and Allow wins a tie.
- Upload the file to the root of your domain so it opens at yourdomain.com/robots.txt.
Frequently asked questions
Should I block AI crawlers?
It is a business decision. Blocking training crawlers keeps your content out of future model training. Blocking search crawlers can remove you from AI answers and citations. For most marketing sites, allowing AI search crawlers is the better choice for visibility.
What is the difference between Disallow and noindex?
Disallow stops crawling. Noindex, set in a meta tag or HTTP header, stops indexing. A page blocked in robots.txt can still appear in search as a bare URL if other sites link to it, and Google cannot see a noindex tag on a page it is not allowed to crawl.
Where does robots.txt go?
At the root of each host, for example https://example.com/robots.txt. A subdomain like shop.example.com needs its own file. Most platforms, including WordPress and Shopify, create one you can edit or replace.
Do all bots obey robots.txt?
Reputable crawlers do. It is a voluntary standard, though, so it will not stop scrapers that ignore it. Use authentication or server rules for anything truly private.

