Reader-Submitted Questions

Blocking /search/ from robots.txt prevents Googlebot from crawling those search result pages — but it doesn't prevent Google from following links that appear on other pages that lead to those URLs. If a product page links to a related search, Google can follow that link and discover the URL. It just won't crawl the search result page itself.

The more important question is whether your internal search result pages carry unique content worth indexing. Usually they don't — they're dynamically generated lists that Google can't process well anyway. Blocking /search/ is generally correct. But make sure your valuable product pages are linked directly from crawlable pages (category pages, sitemaps, navigation), not only discoverable through search result pages.

Google uses specificity to resolve conflicts. The more specific rule wins. If two rules are equally specific, the Allow rule takes precedence. Here's how that plays out:

Disallow: /admin/
Allow: /admin/public/

In this case, /admin/public/ is more specific than /admin/, so it wins. Anything in /admin/public/ is allowed. Everything else in /admin/ is blocked.

Disallow: /admin/
Allow: /admin/

These are equally specific. Google's interpretation: Allow wins when specificity is equal. This path would be allowed.

Note: Bing's parser uses a different conflict resolution approach. If you're concerned about multi-engine consistency, use separate User-agent blocks for each crawler rather than relying on conflict resolution behavior.

For Google's parser, order within a User-agent block doesn't determine which rule wins — specificity does. However, order matters for readability and for parsers that process rules sequentially (some non-Google crawlers do this).

Best practice: write rules from most specific to least specific within each block. This makes the file easier to audit and reduces confusion about which rule applies to a given URL. It also provides consistent behavior across different crawlers with different parsing approaches.

Search Console's coverage report reflects the state of Google's index, not the current state of your site. After fixing robots.txt, Googlebot needs to re-crawl those URLs to update their status. This process depends on crawl frequency — high-traffic sites with frequent crawling may see updates within days. Lower-frequency sites can take several weeks.

Steps to accelerate: Submit your sitemap again via Search Console (signals Googlebot that content needs re-evaluation). Use the URL Inspection tool on specific important URLs and click "Request Indexing." Check that the corrected robots.txt is actually live by visiting yourdomain.com/robots.txt directly. Confirm no server caching is serving the old version.

Three weeks is within the normal range. If it extends beyond six weeks with no improvement, investigate whether there's a secondary issue — a redirect loop, a noindex tag that was added alongside the disallow, or a server caching problem.

Two weeks of complete blocking is significant but recoverable. The damage depends on several factors: how many pages were previously indexed, how high your crawl frequency was before the block, and whether external links to your pages remained active during the period.

Pages don't immediately drop from Google's index when crawling stops. Google retains indexed content for some period after the last successful crawl. But after two weeks, Google begins to mark pages as potentially stale. After the block is lifted, expect the following timeline:

Week 1-2 after fix: Googlebot begins re-crawling. Coverage report starts showing status changes. Week 3-4: Crawled pages return to indexed status. Ranking recovery begins for re-indexed pages. Week 6-10: Full recovery for most pages. Some pages that had dropped from the index due to inferred low quality may require additional signals to recover.

Submit your sitemap immediately after fixing. Check the Coverage report weekly. Don't make any other major changes to the site during recovery — isolate the variable.

Robots.txt is the wrong tool for this. Blocking duplicate URLs prevents crawling — it doesn't prevent indexing. If Google discovered those URLs before you added the block, they may remain in the index for weeks or months after the block is in place, because Google can't crawl them to receive a removal signal.

Canonical tags are the correct mechanism. They tell Google which version of a URL to treat as the authoritative one, and they work while the duplicate pages remain crawlable. You can also use noindex on the duplicate pages alongside a canonical tag for belt-and-suspenders coverage.

The only case where robots.txt blocking makes sense for duplicate content is infinite scroll or parameter-generated URLs that were never important enough to be indexed — blocking them prevents wasting crawl budget on content that shouldn't be indexed. But the proper deduplication signal is canonical, not robots.txt.

Yes. This is an underappreciated issue. If your CDN caches robots.txt with a long TTL (time-to-live), Googlebot may receive the cached version for hours or days after you've updated the origin file. The CDN serves the old version. Googlebot caches that. You're now dealing with two layers of caching.

Recommendations: Set robots.txt to have a short cache TTL at the CDN level — no more than one hour, ideally shorter. After updating robots.txt, purge the CDN cache for that specific URL. Verify the live robots.txt by fetching it with a tool that bypasses cache (curl with appropriate headers, or a fresh browser session). Some CDN providers allow you to set cache bypass rules for specific paths — apply this to /robots.txt.

Google's documentation states that Googlebot processes up to 500 kibibytes of a robots.txt file. Content beyond that limit is ignored. For most sites, this is not a practical constraint — a robots.txt with hundreds of rules is still typically under 50 KB.

However, a robots.txt with hundreds of rules is often a symptom of a structural problem. If you're adding individual URL exceptions rather than directory-level rules, the file will grow quickly and become difficult to maintain. A robots.txt that requires hundreds of rules to manage is usually better addressed through URL structure changes or parameter handling configuration rather than ever-growing block lists.

If your file genuinely needs to be large, audit it periodically to remove rules for URLs that no longer exist, consolidate multiple specific rules into broader directory rules where appropriate, and ensure you're not duplicating rules across multiple User-agent blocks unnecessarily.

If robots.txt returns a 404, Google treats it as if the file exists but is empty — meaning all crawling is allowed. This is actually the correct behavior for most sites. You only need a robots.txt file if you want to restrict or direct crawling behavior in some way.

A missing robots.txt is not a problem in itself. But there is one practical reason to have one even if it contains only the sitemap directive: pointing crawlers to your sitemap from robots.txt provides an additional discovery path that doesn't depend on sitemap submission through Search Console. This is useful for sites that want to ensure any crawler, not just Google, can find their sitemap.