How is a path judged allowed or blocked?

We use the longest matching Allow/Disallow rule in the most specific user-agent group. If two rules have the same length, Allow wins. That matches how Google documents robots.txt matching. A missing or empty file allows all paths.

Does robots.txt stop a page from being indexed?

No. robots.txt controls crawling, not indexing. A URL can still appear in search results if it is linked elsewhere, often without a snippet. Use noindex (meta or X-Robots-Tag) when you need to keep a page out of the index, and make sure the page is crawlable so Google can see that tag.

Can I test Googlebot specifically?

Yes. Set the user-agent to Googlebot (the default). If the file has a Googlebot group, those rules apply; otherwise the * group is used. Other crawlers can be tested the same way.

My production file 404s — is that bad?

A 404 robots.txt is treated as allow-all. That is valid. Add a file when you need to disallow paths or advertise a sitemap.

Why does Googlebot still fetch a disallowed URL sometimes?

Google may still probe disallowed URLs occasionally, and other crawlers ignore robots.txt. Treat Disallow as a hint to well-behaved bots, not a firewall.

Can I test a path that is not on this host?

The fetch is always origin/robots.txt. The path field is only matched against that file; it is not fetched itself.

robots.txt Tester — Fetch, Parse & Test Paths

How to use robots.txt Tester

The robots.txt Tester fetches /robots.txt from a public origin, parses User-agent groups, and tells you whether a given crawler is allowed to request a path. It is the live counterpart to the robots.txt Generator. A missing file is treated as allow-all, which is how Google and Bing behave. Sitemap directives are listed so you can jump to the Sitemap Validator next.

Enter the site URL or origin. We request https://host/robots.txt.
Set the user-agent (Googlebot by default) and the path to test.
Click Test. The verdict shows Allowed or Disallowed plus the matching rule.
Read each group’s rules and the raw file if you need to edit it.
Open the generator to draft a replacement, or the Sitemap Validator for listed sitemaps.

How matching works

Crawlers pick the most specific user-agent group (Googlebot beats * when both exist). Inside that group, the longest matching path pattern wins. If two patterns have the same length, Allow wins over Disallow. An empty Disallow means “allow all” for that group. Patterns may include * (any characters) and $ (end of URL). That is the model Google documents; other bots can differ, which is why you can type any user-agent string.

Crawling is not indexing

Disallowing a path stops well-behaved crawlers from fetching it. It does not reliably keep the URL out of the index if other pages link to it — Google may still list the URL with no snippet. To keep a URL out of search results, allow the crawl and send noindex (meta robots or X-Robots-Tag) so the crawler can see the instruction. Blocking a URL that you also want to noindex is a common own-goal. Use this tester together with the Meta Tag Analyzer to confirm both layers.

Crawl vs index
Goal	Mechanism
Save crawl budget / hide params	robots.txt Disallow
Keep a URL out of the index	noindex (and allow the crawl)
Canonicalize duplicates	rel=canonical
Discover URLs	XML sitemaps

Worked examples

Googlebot vs /admin

Inputs: URL + UA Googlebot + path /admin

Result: Disallowed if a matching Disallow rule wins

Missing robots.txt

Inputs: A host that returns 404 for /robots.txt

Result: Treated as allow-all for every user-agent

Sitemap lines listed

Inputs: A file with Sitemap: URLs

Result: Each sitemap URL shown for the validator next

Glossary

User-agent group: A block of robots.txt rules that applies to one or more crawler names, or to *.
Disallow: A rule that asks matching crawlers not to request URLs under that path prefix.
Allow: A more specific exception that permits a path even when a shorter Disallow exists.
Crawl-delay: An unofficial directive some crawlers honor; Google ignores it.
Sitemap directive: A robots.txt line pointing crawlers at an XML sitemap URL.

Related Developer Tools

robots.txt Generator

Generate a robots.txt file for your website. Set user-agents, allow/disallow rules, crawl-delay, and sitemap URL. Free, private robots.txt builder.

XML Sitemap Validator

Validate any XML sitemap online. Check structure, namespace, URL count, lastmod dates, and size limits, and list child sitemaps - enter a URL for a full report.

Website Analyzer

Analyze any public website for technical SEO, HTTP status, security headers, SSL, robots.txt, sitemap signals, and DNS — one URL, an actionable report.

Meta Tag Analyzer

Fetch any public page and extract its title, meta description, canonical, robots, Open Graph, Twitter Card, hreflang, and JSON-LD types. Free live checker.

.htaccess Redirect Generator

Generate .htaccess redirect rules online. Create 301/302 redirects, www and HTTPS forcing, and RewriteRule snippets for Apache. Free and private.

JSON Formatter & Validator

Format, validate, and minify JSON instantly. Includes syntax highlighting, error detection, and a collapsible tree view — free, private, in-browser.

Explore all Developer Tools.

robots.txt Tester

How to use robots.txt Tester

How matching works

Crawling is not indexing

Worked examples

Glossary

Related reading

Frequently Asked Questions

Why use robots.txt Tester?

Common use cases

Get weekly tool tips & updates

Related Developer Tools