Robots.txt Guide: Syntax Rules, Best Practices, and 10 Working Examples

Every website connected to the public internet encounters automated web robots. Search engines like Google and Bing crawl your pages to populate search results, commercial data collectors scrape pricing and directory listings, and AI companies ingest articles to train language models. The primary mechanism for managing these automated visitors is the Robots Exclusion Protocol, standardized by the Internet Engineering Task Force (IETF) in RFC 9309.

If you misconfigure your file, you risk removing your entire site from search engine results or wasting crawl budget on duplicate administrative pages. Studying real-world robots.txt examples and understanding the underlying RFC 9309 parsing rules ensures your directives work as intended.

In this comprehensive guide, we cover the core syntax of robots.txt, debunk common crawling myths, examine verified AI crawler tokens, provide 10 copy-paste templates, and show you how to generate compliant configurations using our free Robots.txt Generator.

Robots.txt Core Syntax and RFC 9309 Rules

A valid robots.txt file must be served from the root of your domain over HTTP or HTTPS at /robots.txt (for example, https://example.com/robots.txt). It is case-sensitive, encoded in UTF-8 text, and consists of directive groups separated by blank lines.

The table below outlines the core directives defined by RFC 9309 and major search engines:

Wildcard Matching: Asterisk and End-of-Pattern Dollar Sign

RFC 9309 Section 2.2.3 formalizes two special pattern characters:

• * (Asterisk): Designates zero or more instances of any valid character. For instance, Disallow: /*?* matches any URL containing a query string with a question mark.
• $ (Dollar sign): Designates the exact end of the URL path pattern. For example, Disallow: /*.pdf$ blocks any URL ending strictly with .pdf, while leaving /download-pdf-guide or /file.pdf?id=12 accessible.

Rule Precedence: The Most Specific Match Wins

When a crawler encounters overlapping Allow and Disallow rules, it does not simply apply the rule written first. According to RFC 9309 Section 2.2.2:

• The crawler matches the pattern from the first character of the URI path.
• The crawler uses the most specific match, defined as the rule matching the greatest number of path characters (octets).
• If an Allow rule and a Disallow rule match the exact same number of characters, the Allow rule takes precedence.

For example, consider these two rules:

Disallow: /blog/
Allow: /blog/case-studies/

For the URL /blog/case-studies/client-a, the Allow rule matches 19 characters, while the Disallow rule matches 6 characters. The crawler selects the longer match and allows access to the page.

5 Critical Robots.txt Realities Every Webmaster Must Know

Misconceptions about robots.txt lead to severe SEO issues. Review these documented operational facts before deploying rules:

1. Robots.txt Does Not Prevent Indexing

Blocking a page in robots.txt only stops search engines from crawling the page content. It does not stop search engines from indexing the URL.

According to Google Search Central documentation on blocking indexing: if an external or internal link points to a disallowed URL with descriptive anchor text, Google can index that URL based on link signals alone. The result is an indexed search snippet showing only the URL title without a page description. To guarantee a page is removed from search results, you must leave the page crawlable and serve an on-page <meta name="robots" content="noindex"> tag or X-Robots-Tag: noindex HTTP response header.

2. Google Completely Ignores Crawl-Delay

Many older articles recommend adding Crawl-delay: 5 to reduce server load. Google Search Central documentation explicitly confirms that Google does not support crawl-delay. Adding this directive does not affect Googlebot crawl speed. To manage Googlebot crawl rate, use the crawl rate settings in your server infrastructure or return HTTP 429 (Too Many Requests) status codes.

3. Noindex Inside Robots.txt Is Unsupported

In July 2019, Google officially deprecated unsupported experimental rules, including specifying noindex inside robots.txt. If you write Noindex: /private/, Google treats it as an invalid line and ignores it.

4. Blocking Prevents Noindex Discovery

If you place <meta name="robots" content="noindex"> on a webpage and simultaneously add Disallow: /my-page/ in robots.txt, Googlebot cannot fetch the page HTML. Because the bot cannot read the HTML, it never sees your noindex tag. Keep URLs allowed in robots.txt while you wait for search engines to process removal tags.

5. Only One User-Agent Group Applies

When Googlebot reads a robots.txt file, it looks for the most specific user-agent group matching its name (Googlebot). If it finds a specific User-agent: Googlebot block, it completely ignores the generic User-agent: * block. Directives from User-agent: * are not merged into specific bot blocks.

AI Crawler Tokens: Managing Scrapers and LLM Bots

Generative AI platforms maintain specialized crawlers. You can control whether AI models ingest your site for training, real-time search grounding, or live user queries by addressing their verified user-agent tokens.

The table below lists verified bot tokens documented by OpenAI, Anthropic, and Google:

To complement your robots.txt controls with structured content summaries for AI assistants, create one with our llms.txt Generator.

10 Copy-Paste Robots.txt Examples

Use the tested configuration templates below to implement secure, optimized crawling rules for your platform.

Example 1: Standard Public Website (Allow All)

Ideal for standard content websites that want full discovery across search engines.

User-agent: *
Disallow:

Sitemap: https://example.com/sitemap.xml

Example 2: Staging and Development Environments (Block All)

Blocks all compliant crawlers from indexing private staging or testing domains.

User-agent: *
Disallow: /

Example 3: WordPress Production Setup

Blocks administrative dashboards and sensitive system files while preserving access to Ajax endpoints and public assets.

User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php
Disallow: /wp-includes/
Allow: /wp-includes/js/
Allow: /wp-includes/css/
Disallow: /readme.html
Disallow: /license.txt

Sitemap: https://example.com/sitemap_index.xml

Example 4: Shopify E-Commerce Store

Protects cart sessions, internal search queries, and customer account checkout pages from wasting crawl resources.

User-agent: *
Disallow: /admin
Disallow: /cart
Disallow: /orders
Disallow: /checkout
Disallow: /account
Disallow: /*?*sort_by=*

Sitemap: https://example.com/sitemap.xml

Example 5: Next.js and Headless Web Applications

Keeps crawlers out of API routes and the admin area. Do not block /_next/: Google needs those JavaScript, CSS and optimized image files to render your pages.

User-agent: *
Disallow: /api/
Disallow: /admin/

Sitemap: https://example.com/sitemap.xml

Example 6: Block Specific File Types with End-of-Pattern Dollar Sign

Prevents search engine crawlers from indexing private documents or downloadable archives.

User-agent: *
Disallow: /*.pdf$
Disallow: /*.doc$
Disallow: /*.docx$
Disallow: /*.zip$

Example 7: Block Internal Search and Filtering Query Parameters

Conserves crawl budget by blocking endless filter and sorting permutations while leaving clean canonical URLs open.

User-agent: *
Disallow: /*?*
Allow: /products$
Allow: /categories$

Sitemap: https://example.com/sitemap.xml

Example 8: Block AI Model Training While Allowing AI Search

Allows conversational engines to index your pages for cited search results while preventing your content from training foundation models.

User-agent: *
Disallow:

User-agent: GPTBot
Disallow: /

User-agent: OAI-SearchBot
Allow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Claude-SearchBot
Allow: /

User-agent: Google-Extended
Disallow: /

Sitemap: https://example.com/sitemap.xml

Example 9: Block the Main AI Training Crawlers

Blocks the training crawlers from OpenAI, Anthropic and Google, plus Common Crawl's CCBot, whose public dataset is widely used to train AI models. AI search and user-triggered fetchers are not blocked by this example.

User-agent: *
Disallow:

User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: CCBot
Disallow: /

Sitemap: https://example.com/sitemap.xml

Example 10: Multi-User-Agent Group Rules

Applies tailored rules for Googlebot while keeping standard guidelines for general crawlers.

User-agent: Googlebot
Disallow: /private/
Allow: /private/press-release/

User-agent: Bingbot
Disallow: /private/
Crawl-delay: 5

User-agent: *
Disallow: /private/

Sitemap: https://example.com/sitemap.xml

RFC 9309 Path Matching Logic: URL Test Table

To demonstrate how compliant crawlers evaluate pattern rules, we tested our example directives against real URL paths. The table below displays rule evaluation based on RFC 9309 path length precedence:

How to Test and Deploy Your Robots.txt

A wrong robots.txt rule can stop search engines from crawling important pages. Follow this verification sequence before publishing changes:

1. Build Your Configuration: Use our free Robots.txt Generator: start from a preset (Allow all, Block all for staging, WordPress or Next.js), type any user-agent or pick one from the suggestions, add paths and sitemaps, and choose whether to block AI training crawlers or all listed AI crawlers. Before downloading, enter a crawler and a URL in the built-in tester to see whether it is allowed or blocked and which rule decided it.
2. Download and Place at Root: Save the output as robots.txt and upload it directly to your server root directory so it resolves at https://example.com/robots.txt.
3. Verify HTTP Headers: Ensure your server returns an HTTP 200 OK status and serves the file with Content-Type: text/plain; charset=utf-8.
4. Check in Google Search Console: Open the robots.txt report to confirm Google fetched your new file without errors. To check a specific page, run it through the URL Inspection tool, which shows whether robots.txt blocks crawling.
5. Inspect Server Redirects: If using Apache or Nginx redirects, ensure /robots.txt does not redirect across multiple hops or redirect to a localized subdomain. For managing redirect rules, use our .htaccess Generator.
6. Add Schema Markup: Ensuring crawlability works hand-in-hand with search enhancement. Generate structured rich snippets for your allowed pages using our Schema Markup Generator.

Conclusion and Next Steps

A clean robots.txt file is fundamental to search engine optimization and crawl management. By observing RFC 9309 rule precedence, leaving private pages open for noindex discovery, and targeting AI crawlers with dedicated tokens, you keep your site secure and discoverable.

To configure and validate your crawler directives:

• Create your rules with the browser-based Robots.txt Generator.
• If your server runs Apache, generate matching rewrite rules with the .htaccess Generator.
• Define assistant-specific guidelines using the llms.txt Generator.
• Win rich snippet search listings on your allowed pages using the Schema Markup Generator.