Robots.txt Tester (Paste)
Check whether a URL is allowed or blocked for any crawler by pasted robots.txt rules — with the deciding rule quoted, entirely on this device.
Press ? for shortcuts
Do it offline:$ python3 -c "import urllib.robotparser as r; p=r.RobotFileParser(); p.parse(open('robots.txt').read().splitlines()); print(p.can_fetch('Googlebot','https://example.com/private/x'))"
How to use it
- Paste your rules into the left pane, or drop the file onto it; the file is read in this tab and never uploaded.
- Enter the full URL to check, scheme and host included — a bare path is refused rather than guessed at.
- Choose one or more crawler tokens, or type the exact user-agent string you care about; up to eight are compared side by side.
- Read the verdict, then the trace below it: every rule that matched, most specific first, with the winner marked.
Paste a robots.txt file, enter a full URL and choose a crawler: this page answers allowed or blocked, quotes the line that decided it, and lists every rule that matched with the length that ranked it. Group selection, wildcards and percent-encoding follow RFC 9309. Your site is never contacted — the rules are the ones you paste, and nothing you paste leaves this device.
How it works
A robots.txt file is a sequence of groups. Each group opens with one or more User-agent lines and then carries its Allow and Disallow rules until the next User-agent line that follows a rule (RFC 9309 §2.2.1). Exactly one group applies to a given crawler: the one whose token best matches the product token you entered, compared without regard to letter case. The longest matching token wins, so Googlebot-Image takes its own group rather than Googlebot's. Where more than one group names the token that selected it, §2.2.1 has their rules combined into one group, however far apart the records sit. When nothing names your crawler the * group applies, and when there is no * group either, no rule can apply and the URL is allowed.
Inside the chosen group every rule is tested and the most specific match decides — not the first one written (RFC 9309 §2.2.2). Specificity is measured in octets of the rule's path pattern, wildcards and the end anchor counted, which is why Allow: /folder/page beats Disallow: /folder for a URL under that folder. If an Allow and a Disallow tie at the same length, the Allow wins. Ties beyond that are settled by the earlier line, so a verdict here never depends on the order rules happen to be listed in.
What gets compared is the path plus the query string, with the fragment discarded, and the comparison respects letter case: /Fish and /fish are different URLs to a crawler even where your server treats them alike. Two characters are special (RFC 9309 §2.2.3). An asterisk stands for any run of characters, and a dollar sign anchors the end of the path — but only as the pattern's final character; written anywhere else it is an ordinary character. A pattern with no end anchor matches any URL it prefixes, and a rule with an empty value matches nothing at all, which is how the classic empty Disallow line opens a site up rather than closing it.
Before either side is compared, both are normalized the way RFC 3986 describes: escapes of unreserved characters are decoded, remaining escapes are uppercased, and anything outside printable US-ASCII is re-encoded as UTF-8 bytes. That is why /%7Ejoe/ and /~joe/ are one path and not two. Size has a rule of its own: §2.5 obliges a crawler to read only the first 500 kibibytes, so this page stops at 512,000 bytes on the last whole line and says so, rather than pretending rules below that point are enforced.
Examples
User-agent: *
Disallow: /folder
Allow: /folder/page
Allowed
Decided by line 3: Allow: /folder/page
Group matched by *
Compared: /folder/page
line 3 · Allow: /folder/page · matched 12 characters
line 2 · Disallow: /folder · matched 7 charactersBoth rules match this URL, and the trace shows why the Allow carries it: twelve characters of pattern against seven. A parser that stopped at the first matching line would report the opposite verdict for the same file.
User-agent: *
Disallow: /*.php$
Allowed
No rule in the matched group matches this URL, so nothing blocks it.
Group matched by *
Compared: /a.php?x=1The anchor demands that the compared string end at .php, and this one ends at x=1, so the rule cannot reach it. Drop the query and the same file blocks the same file name — the surprise is in the anchor, not in the wildcard.
User-agent: Googlebot
Allow: /images/
User-agent: Googlebot-Image
Disallow: /images/
Googlebot — Allowed · Decided by line 2: Allow: /images/ · Group matched by Googlebot
Googlebot-Image — Blocked · Decided by line 5: Disallow: /images/ · Group matched by Googlebot-Image
Bingbot — Allowed · No group in this file applies to this crawler, so nothing blocks this URL.One file, one URL, three answers. The image crawler takes the group that names it rather than the shorter token it starts with, and Bingbot finds nothing addressed to it — with no * group in the file, that is a permission, not an oversight.
User-agent: *
Disallow: /
Allow: /$
Allowed
Decided by line 3: Allow: /$
Group matched by *
Compared: /
line 3 · Allow: /$ · matched 2 characters
line 2 · Disallow: / · matched 1 charactersThe anchored pattern is two characters long and the bare slash is one, so the home page stays crawlable while every deeper URL in the same file is blocked. Test any second address to watch the one-character rule take over.
Every example on this page is executed against the same engine that powers the tool, on every build.
Gotchas worth knowing
Longest match wins — first match is the older rule
Parsers written before RFC 9309 often took the first line that matched, and a few tools still behave that way, which is the usual reason two checkers disagree about one file. The standard settles it by pattern length, and this page shows the measurement instead of asking you to trust the answer.
A dollar sign only anchors at the very end
Written last it means the path stops here; written anywhere else it is an ordinary character with no special power, and the trace says so when it sees one. Because a literal dollar sign is normalized before comparison, the compared path shows it as %24 — the same character, spelled the way the matcher reads it.
The query string is compared, the host is not
Everything after the question mark counts as part of the string a rule is matched against, while the scheme, host and port are used only to confirm you gave a real address. The fragment after a hash never reaches a server and is dropped before matching.
Case matters in paths, not in crawler names
A rule naming /Fish leaves /fish untouched, and the page raises a note when letter case alone changes the outcome, because that is rarely what the author meant. Crawler tokens are the opposite: user-agent is matched without regard to case, so googlebot and Googlebot select the same group.
Crawl-delay, Host, Clean-param and Noindex never move a verdict
None of them is part of RFC 9309. Google ignores Crawl-delay and has said plainly that a Noindex line in this file is unsupported; Host and Clean-param were Yandex extensions. They are listed here when present, and they change nothing in the answer above them.
Blocking a URL is not the same as removing it from search
A disallowed address can still be listed by a search engine that learned about it from links elsewhere — it simply appears without a description, because the page was never read. Keeping something out of results is the job of a noindex meta tag or an X-Robots-Tag header on a page crawlers are allowed to fetch.
The /robots.txt address itself
RFC 9309 §2.2.2 closes by declaring that the robots.txt URI is implicitly allowed, and some parsers special-case it. This one does not: it reports what your file literally says, so a blanket Disallow: / shows /robots.txt as blocked. That is a deliberate choice to keep the trace honest, and it is recorded rather than quietly handled.
Percent-encoding is normalized on both sides
A rule spelled /%7Ejoe/ and a URL spelled /~joe/ describe the same place, so both are folded to one form before comparison. Not every implementation folds the same set of characters — CPython re-encodes reserved punctuation such as a comma, so it can match a rule this page leaves unmatched. When they differ, the trace tells you which spelling was compared.
Common errors
"/private/page" is not an absolute URL — robots.txt rules are matched against a full URL, so include the scheme and host, as in https://example.com/page.- A path on its own has no site attached, and guessing one would make the verdict fiction. Paste the address exactly as it appears in a browser or a log line and the check runs unchanged.
No "User-agent" line in this file at line 1, column 1 — rules only exist inside a group, so nothing is blocked and every URL is allowed. Add "User-agent: *" above the rules.- A file of comments and a sitemap reference is valid and restricts nothing. The verdict is still reported, with this reason attached, so an empty result is never mistaken for a strict one.
"Disallow" appears before any "User-agent" line at line 1, column 1 — a rule outside a group belongs to no crawler and is ignored. Add a "User-agent" line above it.- Rules take effect only inside a group, so anything above the first crawler line is dead text. It is reported at its line number and left exactly as you pasted it.
Input is 1,048,577 characters at line 1 — this tool parses at most 1,048,576 characters (1 MB) in one run. Split the file, or check it locally with the command on this page.- The whole input is declined rather than partly parsed, because line numbers taken from a truncated file would point at the wrong rules. Note the separate 512,000-byte mark, which is where a compliant crawler stops reading a file this page still accepts.
9 user-agents were requested — this tool compares at most 8 in one run. Remove 1 of them and test again.- Eight columns is as much as one comparison can show without the trace becoming unreadable. The limit is on the crawlers per run, not on how many runs you make.
FAQ
Does this page fetch my robots.txt file?
No. It reads only the text you paste or drop in, and it issues no network requests at all — you can confirm that in your browser's network panel. Checking a live file over the network is a separate tool that is not built yet, and pretending otherwise would make the offline claim on this page false.
Where did Google's own robots.txt tester go?
It was removed from Search Console in December 2023. What replaced it is a report that shows which files Google fetched for your site and whether they parsed, not an interactive check of a URL against rules you are still drafting — which is the gap this page fills.
Why does another checker give a different answer?
Usually precedence. Older implementations take the first matching line while the standard takes the longest pattern, and the two disagree exactly when a broad block is narrowed by a specific exception. Read the trace: it prints each matching rule with its measured length, so the disagreement becomes checkable rather than a matter of opinion.
Is this Google's parser?
No, and it does not embed one. The matcher here is written from RFC 9309 and each rule it applies is named in the text above. Google publishes its own parser as open source, and the command printed on this page runs a third implementation from the Python standard library — comparing them is a reasonable thing to do before you deploy a file.
Will a Disallow rule keep a page out of search results?
Not reliably. It asks a crawler not to fetch the address, which means the content is never read, but a URL can still be listed from links pointing at it. Removal from results comes from a noindex signal on a page that crawlers are permitted to fetch.
Does Crawl-delay slow a crawler down?
Not for every crawler, and not for Google, which ignores the directive; it appears in no part of RFC 9309. Some other crawlers do honor it. It is shown here when your file contains it, and it is never allowed to affect the allowed or blocked answer.
What happens when no group names my crawler?
The group whose token is * applies, and the note above the trace says so. If the file has no such group, nothing in it addresses that crawler and the URL is allowed — a robots.txt restricts only what it names.
Keyboard shortcuts
- ⌘/Ctrl + Enter
- Run the test again against the current rules.
- ⌘/Ctrl + S
- Download the decision trace as .txt — downloads rather than saving the page.
- ⌘/Ctrl + C
- With nothing selected, copy the verdict and the deciding rule; a selection still copies itself.
- ⌘/Ctrl + Shift + C
- Force-copy the verdict and deciding rule, selection or not.
- t
- Next theme (single-key; idle in text fields, honors the kill switch).
- ? or ⌘/Ctrl + /
- Open the shortcut help, including the ⌘S conflict note.
- Esc
- Close the topmost overlay; focus returns to its opener — never the input.