Search & Technical SEO

Robots.txt Guide: Crawl Rules, User Agents, and Sitemap Directives

Understand robots.txt syntax, user-agent groups, crawl directives, sitemap lines, and the important limits of crawler rules.

A
Ali Raza
Nov 16, 20263 min read
Share:
Robots.txt Guide: Crawl Rules, User Agents, and Sitemap Directives

A robots.txt file gives compliant crawlers instructions about which URL paths they may request. It is useful for crawl management, but it is public and does not protect confidential content. This guide explains its basic structure and common pitfalls.

Open the Robots.txt Generator →

Connected digital network representing web crawlers
Robots rules guide crawler requests across a site.

User agents and directives

Rules are grouped under a User-agent line naming a crawler or using an asterisk for all crawlers. Disallow identifies paths a matching crawler should not fetch; Allow can clarify exceptions. Directives are evaluated within the relevant group, and crawler support can vary. Use exact paths and test the resulting file carefully.

A small illustrative example is:

User-agent: *
Disallow: /private-preview/

Sitemap: https://example.com/sitemap.xml

Replace example paths and domains with values for your own site. Do not assume that a folder is private because it appears in a Disallow rule.

Computer hardware representing technical site configuration
Review the deployed file at the site root after generating it.

Robots.txt is not access control

The file is publicly readable, and well-behaved crawlers may still index a URL discovered elsewhere even if they cannot fetch its contents. Use authentication or server permissions for sensitive material. If you want a crawled page excluded from search, use the appropriate indexing controls rather than relying on crawl blocking alone.

Validate before deploying

  1. Choose the intended user-agent group.
  2. Add only crawl rules that match your indexing plan.
  3. Include the current sitemap URL, publish at the domain root, and check the live response.

Generate a robots.txt file →


Written by Dr. Sarah Chen
Senior Web Performance Engineer and browser runtime researcher. Ph.D. in Computer Science with a focus on compiler optimization and sandboxed execution. Research contributor on WebAssembly, HTML5 Canvas, and modern browser standards at IT Market Plus.

A
Written by

Ali Raza

Principal Software Engineer & Technical Architect with over a decade of experience in client-side web technologies, cryptography, and Core Web Vitals optimization. Founder & Editorial Lead at IT Market Plus.