Skip to content
SEOL

    Robots.txt

    robots.txt controls crawling, not indexing. Learn the rules Google follows, how conflicts are resolved, and the mistakes that hide whole sites.

    Time
    18 min
    Level
    intermediate
    Reviewed
    5 November 2026

    After this lesson you can

    • Write robots.txt rules with wildcards correctly.
    • Predict which rule wins when allow and disallow conflict.
    • Avoid blocking resources and using robots.txt for de-indexing.

    robots.txt is a plain text file at the root of a host that tells crawlers which URLs they may request. It manages crawling. It is not a way to keep pages out of search results.

    Syntax

    # Comments start with #
User-agent: *
Disallow: /cart
Disallow: /checkout
Disallow: /*?sort=
Allow: /cart/help

User-agent: Googlebot-Image
Disallow: /private-images/

Sitemap: https://example.com/sitemap.xml

    Which rule wins

    1. Which group? A crawler follows the single most specific group whose User-agent matches it, and falls back to *. Groups are not combined. If you write a Googlebot group, Googlebot ignores the * group completely.
    2. Which rule? Within the group, the rule with the longest matching path wins. On an exact tie, the less restrictive rule (allow) wins.
    RulesURLResult
    Disallow: /shop, Allow: /shop/sale/shop/sale/bootsAllowed (longer rule)
    Disallow: /*.pdf$/files/price.pdfBlocked
    Disallow: /*.pdf$/files/price.pdf?v=2Allowed ($ requires the URL to end there)
    Disallow: /folder, Allow: /folder/folder/pageAllowed (tie, allow wins)

    When robots.txt itself fails

    • 4xx (except 429): Google treats the site as having no restrictions.
    • 5xx: Google stops crawling for a while, then uses its last good copy for up to 30 days.
    • Redirects: followed for a few hops.

    A misconfigured server that returns 500 for robots.txt can pause crawling of the whole site.

    Common mistakes

    • Launching with the staging rule Disallow: /. Check robots.txt as part of every launch.
    • Blocking CSS or JavaScript, so Google cannot render pages properly.
    • Using robots.txt to de-index pages, which hides the noindex rule.
    • Forgetting that groups do not combine, so a new Googlebot group silently drops the general rules.
    • Blocking parameter URLs that carry canonicals, so Google cannot see where they point. For crawl-heavy filters on large sites, blocking is fine. For a few duplicates, canonicals work better.

    Platform notes

    • PrestaShop: generated in Shop Parameters, then Traffic & SEO. Regenerate after changing friendly URLs or languages, and review the result, which includes many rules.
    • Shopify: edit only through the robots.txt.liquid template.
    • WordPress: a virtual file that SEO plugins can edit. A physical file in the web root overrides it.

    Practice

    Read the lesson, pass the quiz and do the exercise to complete it.

    Knowledge check

    5 questions. Get at least 4 right to pass.

    1. 1. Where must robots.txt be placed?
    2. 2. The rules are "Disallow /shop" and "Allow /shop/sale". Is /shop/sale/boots allowed for that group?
    3. 3. Does Google support crawl-delay?
    4. 4. robots.txt returns 404. What does Google do?
    5. 5. There is a group for "Googlebot" and a group for "*". Which rules does Googlebot follow?

    Exercise: Review and test your robots.txt

    Open your robots.txt. For each rule, write what it blocks and why it exists. Test five important URLs and five URLs you expect to be blocked, using the SEO-L page inspector (it reports whether robots.txt allows Googlebot) or Search Console's robots.txt report. Check that CSS and JavaScript files are not blocked.

    Deliverable: Your robots.txt with a comment for each rule, plus a table of the ten test URLs with the expected and actual result.

    Check your work honestly against each point
    Write up your work and meet every point to finish.

    Sources

    Checked against these sources on 5 November 2026. Next review by 5 May 2027.

    Terms

    Checklist items

    Tools