Skip to main content

robots.txt Generator

Build a robots.txt from a form: per-agent Allow / Disallow, Sitemap and Crawl-delay, with a one-click preset to block GPTBot, ClaudeBot and other AI crawlers.

robots.txt Generator
Presets
Rules per User-agent
Sitemap
⚠ Warnings (1)
Disallow means "do not crawl", not "remove from search results". To drop an already-indexed URL, use a noindex meta tag or a removal request.
Output
User-agent: *
Allow: /wp-admin/admin-ajax.php
Disallow: /wp-admin/
Disallow: /?s=
Disallow: /search/

Sitemap: https://example.com/sitemap.xml

Define a rule group per user-agent with Allow, Disallow and Crawl-delay lines, add Sitemap URLs, and get a ready-to-upload robots.txt. Presets cover allow-all, block-all and a typical WordPress layout, plus two for AI crawlers: one blocks training use by agents such as GPTBot, ClaudeBot, CCBot and Google-Extended, and one also blocks AI search agents such as OAI-SearchBot. Paths without a leading slash, invalid patterns like Disallow: * and the all-too-common Disallow: / against every crawler are flagged before you deploy.

robots.txt Generator: Features

How a robots.txt is structured

The file is a sequence of groups. Each begins with one or more User-agent lines naming the crawlers it applies to, followed by Disallow and Allow lines giving path prefixes. User-agent: * matches any crawler not named more specifically. A crawler reads only the group that matches it best, so if there is a Googlebot group, Googlebot ignores the * group entirely; rules you want applied to both have to appear in both. The file lives at the site root, https://example.com/robots.txt, encoded as UTF-8.

robots.txt vs noindex: Disallow stops crawling, not indexing

This is the misunderstanding that causes the most trouble. Disallow: /private/ asks the crawler not to fetch those URLs. It does not remove them from search results: a URL linked from elsewhere can still be indexed, typically shown with no description because the content was never read. To keep a page out of results, allow crawling and add a noindex robots meta tag, which the crawler must be able to fetch to obey. The Meta Tag & Open Graph Generator emits that tag.

Blocking AI crawlers (GPTBot, ClaudeBot, CCBot) in robots.txt

AI crawlers fall into three groups by purpose. Training crawlers include OpenAI's GPTBot, Anthropic's ClaudeBot, Meta's meta-externalagent, Amazon's Amazonbot, ByteDance's Bytespider and Mistral AI's MistralAI-Training, and Common Crawl's CCBot, which builds a public dataset, belongs here too. Google-Extended and Applebot-Extended do not crawl at all: they are tokens that control use for purposes such as training. AI search crawlers include OAI-SearchBot, Claude-SearchBot, PerplexityBot, Meta-WebIndexer, Amzn-SearchBot, DuckAssistBot, MistralAI-Index and YouBot. The third group fetches pages on a user's request: ChatGPT-User, Claude-User, Perplexity-User, Meta-ExternalFetcher, Amzn-User and MistralAI-User. The "Block AI training" preset blocks only the first group, so citations and visits from AI search remain. "Block AI search too" blocks the other two groups as well, at the cost of visibility in AI search. Both leave ordinary search crawlers alone, and each list is also a one-click addition to any configuration. To see which of them are hitting your server, the User-Agent Parser identifies them from log lines, and the User-Agent List gives each one's string, token and verification method. Bear in mind that robots.txt is advisory, and agents acting on a user's request may not apply it, so blocking with certainty takes the CDN or firewall.

Wildcards (*) and the $ end anchor in robots.txt

Google, Bing and the other major crawlers understand * for any sequence of characters and $ for end of URL. Disallow: /*.pdf$ blocks URLs ending in .pdf; Disallow: /*? blocks anything with a query string. Paths must still begin with a slash, so Disallow: * and Disallow: *.pdf are invalid and silently ignored. An empty Disallow: means "block nothing" and is the conventional way to spell out allow-all. The generator warns about missing leading slashes and patterns that start with *.

The usual WordPress robots.txt

WordPress sites typically disallow /wp-admin/ but explicitly allow /wp-admin/admin-ajax.php, which front-end scripts call. Internal search results at /?s= and /search/ are commonly disallowed as thin duplicate content. Assets under /wp-content/ and /wp-includes/ should not be blocked, because Google needs the CSS and JavaScript to render pages correctly. The "WordPress" preset produces exactly this configuration as a starting point.

robots.txt Generator: FAQ

Where does robots.txt go?

At the root of the host, so that it is served at https://example.com/robots.txt. A copy in a subdirectory is never read. Each subdomain is a separate site with its own file, and http and https are treated separately too, so check that the https origin serves it after migrating.

Do I need a robots.txt if I am not blocking anything?

Not strictly. A missing file (a 404) is treated as allow-all. A server error (5xx) is not, however: crawlers may treat it as block-all. A minimal file with User-agent: * and an empty Disallow: avoids that risk and gives you somewhere to list the sitemap.

What does Disallow: / do?

It tells the matching crawler not to fetch anything on the site. Against User-agent: * it removes the whole site from every search engine over time. It is the right setting for a staging server and a disastrous one to ship to production, which happens often enough that the generator flags the combination.

Does Crawl-delay work?

Google ignores it. Bing and some others treat the value as a minimum number of seconds between requests, but support varies: Yandex dropped it in 2018. The crawl rate setting in Search Console was retired in January 2024, so to slow Googlebot down, return 503 or 429 responses when the server is under load. The generator notes the Google limitation whenever a Crawl-delay is set.

Where do Sitemap lines go?

Anywhere; they are independent of the groups. The convention is to put them at the end. Each must be an absolute URL such as https://example.com/sitemap.xml, one per line. If you use a sitemap index, listing just the index is enough.

How do I allow a directory for Google but block it for everyone else?

Create two groups: one for User-agent: Googlebot with the Allow and Disallow lines you want, and one for User-agent: * with the block. Each crawler reads only its best-matching group, so Googlebot sees only its own. Use "+ Add group" to create as many groups as you need.

Will blocking AI crawlers hurt my search ranking?

Blocking GPTBot, ClaudeBot, CCBot and similar agents has no effect on Google or Bing rankings, because those are separate from the search crawlers. Google-Extended does not affect search either, but it only controls use for Gemini training and grounding, so it does not opt you out of AI features built on search. Blocking AI search agents such as OAI-SearchBot makes the site less visible in AI search such as ChatGPT and reduces the visits it sends. To stop training use only, choose the "Block AI training" preset.

How quickly do changes take effect?

Google refetches robots.txt roughly every 24 hours; you can request an immediate refetch from the robots.txt report in Search Console. Pages that were already crawled are subject to the new rules the next time they are visited.

A disallowed page is still showing in search results

That is expected. Disallow prevents crawling but does not remove indexed URLs. To get rid of the result, temporarily allow crawling so that a noindex tag can be read, or use the Removals tool in Search Console.

How do I block URLs with query parameters or a specific file type?

Use the wildcard. Disallow: /*? blocks every URL containing a question mark, which covers faceted navigation, internal search and tracking parameters in one line. Disallow: /*.pdf$ blocks only URLs that end in .pdf, with the $ preventing a match on something like /report.pdf.html. Both forms are understood by Google and Bing. Remember that the path still has to begin with a slash, and that blocking parameterised URLs also stops Google reading any canonical tag on them, so make sure the parameter-free version is linked from somewhere.

Is anything I type sent to a server?

No. The file is generated entirely in your browser. Note, though, that robots.txt itself is public: listing a path under Disallow advertises its existence to anyone who looks, so it is not a way to hide anything.

How to Use robots.txt Generator

  1. Pick a preset

    Start from "Allow all", "WordPress" or "Block AI training", whichever is closest to what you need. Loading a preset and adjusting it is less error-prone than writing the file from nothing.

  2. Adjust the per-agent rules

    For each group, set the user-agents and the Disallow and Allow paths, one per line and each starting with a slash. Add the AI training and AI search block groups with their own buttons and list sitemap URLs as absolute addresses.

  3. Check the warnings and save

    Resolve any warnings about missing slashes or block-all rules, then download the file or copy it and place it at the site root. After deploying, confirm that Search Console can fetch it.

Category SEO