Robots.txt Guide: Crawl Rules, User Agents, and Sitemap Directives
Understand robots.txt syntax, user-agent groups, crawl directives, sitemap lines, and the important limits of crawler rules.
A robots.txt file gives compliant crawlers instructions about which URL paths they may request. It is useful for crawl management, but it is public and does not protect confidential content. This guide explains its basic structure and common pitfalls.
Open the Robots.txt Generator →
User agents and directives
Rules are grouped under a User-agent line naming a crawler or using an asterisk for all crawlers. Disallow identifies paths a matching crawler should not fetch; Allow can clarify exceptions. Directives are evaluated within the relevant group, and crawler support can vary. Use exact paths and test the resulting file carefully.
A small illustrative example is:
User-agent: *
Disallow: /private-preview/
Sitemap: https://example.com/sitemap.xmlReplace example paths and domains with values for your own site. Do not assume that a folder is private because it appears in a Disallow rule.
Robots.txt is not access control
The file is publicly readable, and well-behaved crawlers may still index a URL discovered elsewhere even if they cannot fetch its contents. Use authentication or server permissions for sensitive material. If you want a crawled page excluded from search, use the appropriate indexing controls rather than relying on crawl blocking alone.
Validate before deploying
- Choose the intended user-agent group.
- Add only crawl rules that match your indexing plan.
- Include the current sitemap URL, publish at the domain root, and check the live response.
Written by Dr. Sarah Chen
Senior Web Performance Engineer and browser runtime researcher. Ph.D. in Computer Science with a focus on compiler optimization and sandboxed execution. Research contributor on WebAssembly, HTML5 Canvas, and modern browser standards at IT Market Plus.
Ali Raza
Principal Software Engineer & Technical Architect with over a decade of experience in client-side web technologies, cryptography, and Core Web Vitals optimization. Founder & Editorial Lead at IT Market Plus.
Related Editorial Guides
Technical SEO Checklist: Complying with Google Search Central Standards
A practical technical SEO checklist for crawlable pages, canonical URLs, structured data, robots.txt, sitemaps, and ongoing Search Console checks.
Advanced Technical SEO for Web Utilities: Sitemaps, Canonical Discipline & Software Schema
A practical guide to technical SEO for utility sites: crawlable pages, canonical URL consistency, accurate structured data, and sitemap hygiene.
Meta Tag Generator Guide: Titles, Descriptions, Canonicals, and Social Previews
Create clear page titles and descriptions, set canonical URLs, and preview social sharing metadata with a practical SEO workflow.