Skip to main content
ToolsHub

robots.txt Generator

Build a valid robots.txt by adding allow/disallow rules per user-agent, a crawl delay, and your sitemap URL — then copy or download. No guesswork on syntax.

Updated

Files never leave your browser

Group 1

Generated robots.txt

User-agent: *
Disallow: /admin
Disallow: /private

Sitemap: https://example.com/sitemap.xml

How to use robots.txt Generator

The robots.txt Generator builds a valid robots.txt file for your website by letting you configure allow and disallow rules per user-agent, set a crawl delay, and declare your sitemap URL — without memorising the exact syntax. A correctly formed robots.txt controls which pages search engine bots access, protects admin and staging areas from indexing, and communicates your sitemap location to all compliant crawlers. Generated entirely client-side.

  1. Add a user-agent block — enter the bot name (e.g. Googlebot, Bingbot, or * for all bots).
  2. Add Disallow rules for paths you want to block (e.g. /admin/, /staging/, /private/).
  3. Add Allow rules for paths nested inside a disallowed directory that should be accessible (more specific rules take precedence).
  4. Optionally set a Crawl-delay value in seconds for bots that honour it.
  5. Enter your sitemap URL in the Sitemap field (e.g. https://example.com/sitemap.xml).
  6. Copy the generated robots.txt content and deploy it to your website root at https://yourdomain.com/robots.txt.

Your data never leaves your device — 100% private processing.

robots.txt syntax and rule precedence

A robots.txt file is a plain-text file where each User-agent: block defines rules for one or more crawlers. The wildcard User-agent: * applies to all bots not covered by a more specific block. Within a block, Disallow: /path/ blocks the path and all sub-paths; an empty Disallow: value means allow everything. Allow: rules override Disallow: when both match the same path — Google and Bing use the most specific matching rule (longest match wins). The file is read top-to-bottom but specificity, not order, determines which rule wins for a given URL. A maximum file size of 500 KB is enforced by Google.

Common search engine user-agents
User-agentSearch engineNotes
GooglebotGoogle (web)Primary Google crawler
Googlebot-ImageGoogle ImagesSeparate from web crawler
Googlebot-VideoGoogle VideoSeparate from web crawler
BingbotMicrosoft BingAlso crawls for Yahoo and DuckDuckGo
SlurpYahooLegacy; now mostly Bingbot
DuckDuckBotDuckDuckGoSeparate DuckDuckGo crawler
facebookexternalhitMeta / FacebookOpen Graph link preview crawler
*All botsWildcard fallback for unspecified bots

What robots.txt cannot do

Robots.txt is an advisory protocol — it prevents well-behaved crawlers from accessing pages, but it does not prevent indexing of pages that have external links pointing to them. If another site links to your /admin/ page, Google may still show it in search results based on anchor text alone, even without crawling it. To prevent a page from appearing in search results, use a noindex meta tag or X-Robots-Tag HTTP header instead. Robots.txt also provides no security: malicious crawlers routinely ignore it, so it must never be relied upon to protect sensitive content. Use server-level authentication for genuinely private resources.

Worked examples

Block all bots from /admin

Inputs: User-agent *, disallow /admin/

Result: User-agent: * / Disallow: /admin/

Declare the sitemap

Inputs: sitemap at /sitemap.xml

Result: Sitemap: https://example.com/sitemap.xml

Block one bot

Inputs: block GPTBot only

Result: User-agent: GPTBot / Disallow: /

Glossary

robots.txt
A plain-text file at a website's root that communicates crawling instructions to compliant web crawlers per the Robots Exclusion Protocol.
User-agent
The name of a specific crawler (e.g. Googlebot) or * for a wildcard block covering all unspecified bots.
Disallow
A robots.txt directive that instructs the named bot not to access the specified path or any sub-paths.
Crawl delay
A robots.txt directive requesting a minimum number of seconds between successive requests; respected by Bingbot but ignored by Googlebot.
noindex
An HTML meta tag or HTTP header that instructs search engines not to include the page in their index, regardless of robots.txt.

Related reading

Free · No spam

Get weekly tool tips & updates

New tools, power-user tips, and productivity hacks — delivered free every Friday.

No spam, ever. Unsubscribe with one click.

Related Developer Tools

.gitignore Generator

Generate a .gitignore file for your project. Pick languages, frameworks, and tools to build a ready-to-use git ignore file. Free and private.

.htaccess Redirect Generator

Generate .htaccess redirect rules online. Create 301/302 redirects, www and HTTPS forcing, and RewriteRule snippets for Apache. Free and private.

Open Graph Tag Generator

Generate Open Graph and Twitter Card meta tags for better social media sharing. Preview how your links will look on Facebook, Twitter, and LinkedIn.

cURL Converter

Convert a curl command to JavaScript fetch, Node, Python requests, or PHP. Paste your curl and get clean client code instantly. Free and private.

Cron Expression Generator

Build and decode cron expressions online. Get a plain-English description of any crontab schedule and a field-by-field breakdown. Free cron generator.

JSON Formatter & Validator

Format, validate, and minify JSON instantly. Includes syntax highlighting, error detection, and a collapsible tree view — free, private, in-browser.

Explore all Developer Tools.