robots.txt Generator & Tester
🔒 Runs in your browser
Rules
Test a path
What robots.txt is
robots.txt is a plain-text file you place at the root of your site
(https://example.com/robots.txt) to tell well-behaved crawlers which parts of the
site they may request. It is organised into groups: each group starts with one or more
User-agent lines naming the crawlers it targets, followed by Disallow
and Allow rules. A Disallow path blocks crawling of anything that
starts with that path; an Allow path carves an exception back out. This builder
keeps the groups structured for you, so you never have to remember the exact syntax, and writes
the Sitemap lines at the end where search engines expect them.
How the matching works
Google and Bing follow a specific algorithm, and the tester on this page reproduces it. First
the crawler picks the single most specific group for its name — an exact
User-agent match beats the catch-all * group, and only that group's
rules apply. Then, among the rules in that group, the longest matching path wins;
when an Allow and a Disallow are the same length, Allow
wins. Patterns support * (any run of characters) and a trailing $
(end of the URL), so /*.pdf$ blocks every PDF. If no rule matches, the path is
allowed by default. Type a path and a user-agent into the tester to see the exact rule that
decides it.
Crawling is not indexing
A common trap: Disallow stops a crawler from fetching a URL, but it does
not reliably keep that URL out of search results. If other pages link to a disallowed URL,
Google can still list it (without a snippet, since it never read the page). To keep a page out
of the index, let it be crawled and add a noindex meta tag, or protect it behind
authentication. Use robots.txt for crawl budget and to hide low-value or
duplicate paths — not as a security or privacy control.
Frequently asked questions
What is a robots.txt file?
It is a plain-text file at the root of your site that tells search-engine crawlers which URLs they may request. Rules are grouped by user-agent, then you list Allow and Disallow paths, with Sitemap lines at the end.
Does robots.txt stop a page from being indexed?
No. Disallow blocks crawling, not indexing. A blocked URL can still appear in results if other pages link to it. To keep a page out of the index, allow crawling and add a noindex meta tag, or protect it with authentication.
How do Allow and Disallow take precedence?
Within the group that best matches the crawler, the longest matching path wins. When an Allow and a Disallow are the same length, Allow wins. If no rule matches the path, it is allowed by default.
Where do I put the file and what do * and $ mean?
Place it at the domain root as /robots.txt, for example https://example.com/robots.txt; it applies only to that host and protocol. In paths, * matches any run of characters and a trailing $ anchors to the end of the URL, so /*.pdf$ blocks every PDF.