How to use the Robots.txt Generator
- 1
Choose a preset, or set the rules for each crawler yourself.
- 2
Add any folders to block (for example /wp-admin/ or /cart/) and your sitemap URL.
- 3
Download the file and upload it to the root of your domain, so it opens at yoursite.com/robots.txt.
AI search bots and AI training bots are different
Some AI crawlers fetch pages to answer a user's question live and cite the source, such as OAI-SearchBot and ChatGPT-User (ChatGPT), Claude-SearchBot and Claude-User (Claude) and PerplexityBot (Perplexity). Blocking these can remove you from AI answers.
Others collect content to train future models, such as GPTBot, ClaudeBot, CCBot and Google-Extended. Blocking these keeps your content out of training data and does not affect your Google rankings. If you want to be recommended by AI assistants, a sensible default is to allow AI search bots and decide on training bots separately.
What robots.txt can and can't do
Robots.txt is a request, not a lock. Reputable crawlers follow it, but it does not protect private content, and it does not remove pages that are already indexed. To keep a page out of Google, use a noindex tag instead, and don't block that page in robots.txt, or Google can't see the noindex.
- Never block your CSS or JavaScript files: Google needs them to render the page
- Don't block pages you want indexed, even "thin" ones
- Each crawler follows only the most specific group that names it
Check it after you upload
Open yoursite.com/robots.txt in a browser to confirm it's live, then check the robots.txt report in Google Search Console for errors.
Frequently asked questions
Will blocking GPTBot remove me from ChatGPT?
Blocking GPTBot stops OpenAI from using your pages for training. ChatGPT search uses OAI-SearchBot and ChatGPT-User, which have their own rules, so you can block training and still appear in ChatGPT answers.
Does Google-Extended affect my Google rankings?
No. Google-Extended only controls whether your content is used to train Gemini models. Googlebot crawling and your search rankings are unaffected.
Where does robots.txt go?
In the root folder of your domain, so it opens at https://yoursite.com/robots.txt. Each subdomain needs its own file.
Is anything I enter stored?
No. The file is generated in your browser and nothing is sent to a server.