Robots.txt Blocked
Your robots.txt file is preventing Google's crawler from accessing one or more of your product pages or the resources those pages depend on (images, stylesheets, JavaScript). Google needs to successfully crawl and render your landing pages to verify product details, check availability, and confirm prices. If robots.txt blocks Google's crawler bot (Googlebot) from the product URL or its critical resources, products are disapproved.
Pages blocked by robots.txt [robots.txt] / Crawl rate limitedImpact: Robots.txt blocking is a silent killer for GMC. When Google's crawler can't access your pages due to robots.txt rules, every affected product is disapproved — but the cause isn't obvious from the product disapproval message. All disapproved products remain off Shopping until the robots.txt rules are updated. This can take down your entire catalog if the rules are too broad.
Root Causes
- 1Your robots.txt has 'Disallow: /' which blocks the entire site — common when a staging robots.txt accidentally got promoted to production.
- 2Your robots.txt disallows specific directories like '/products/' or '/shop/' — typically added during SEO experiments or by a developer without realizing it affects GMC.
- 3Your robots.txt blocks assets like '/wp-content/' or '/assets/' — Google needs to fetch images, CSS, and JS to render the page for verification.
- 4Your robots.txt has a specific rule blocking 'Googlebot' while allowing other bots — sometimes done to restrict AI scrapers but inadvertently affects Google Shopping.
- 5Your feed URL (the XML or CSV feed file) is blocked by robots.txt — Google can't fetch the feed itself, resulting in 'feed unavailable' errors.
Fix by Platform
- 1Shopify manages robots.txt automatically. To view yours: visit yourdomain.com/robots.txt in a browser. The default Shopify robots.txt allows Googlebot and most crawlers.
- 2If you've customized robots.txt via Shopify's robots.txt.liquid template (Online Store → Themes → Actions → Edit code → templates → robots.txt.liquid): review your customizations for any rules that block /products/, /collections/, or the Googlebot user agent.
- 3To reset: delete your robots.txt.liquid customization to revert to Shopify's default.
- 4If you use a third-party SEO app that modifies robots.txt: check the app settings. Some apps add overly aggressive blocks.
- 5Verify with Google Search Console → Settings → Crawling → Crawl stats to see if Googlebot is successfully accessing your site.
{%- comment -%} Shopify's default robots.txt.liquid — safe for GMC {%- endcomment -%}
User-agent: *
Disallow: /admin
Disallow: /cart
Disallow: /checkout
Disallow: /orders
Allow: /products
Allow: /collectionsWhen This Doesn't Apply
Check your store now
Free compliance scan — 47 rules checked in minutes.
Frequently Asked Questions
How do I check if my robots.txt is blocking Googlebot?+
Use Google Search Console's robots.txt Tester (Settings → Crawling → robots.txt): enter a specific product URL and see if Googlebot is allowed. Alternative: visit yourdomain.com/robots.txt and look for 'Disallow' rules matching your product paths. Also run the URL through Google's 'URL Inspection' tool in Search Console — it tells you explicitly whether the URL is crawlable.
Should I block Googlebot from any pages?+
For GMC purposes: don't block product pages, category pages, or their resources (images, CSS, JS). It's fine to block: /admin/, /checkout/, /cart/, /my-account/, /wp-admin/, search result pages (/?s=), and any dev/staging paths. If you have faceted navigation that creates infinite URL variations, you can block those to save crawl budget — but product pages must remain accessible.
I use Cloudflare — could it be blocking Googlebot?+
Yes. Cloudflare's bot protection, 'Under Attack' mode, or custom firewall rules can block legitimate Googlebot requests. Check Cloudflare → Security → Events for blocked Googlebot requests. Whitelist Googlebot by IP range (Google publishes them at support.google.com/webmasters/answer/80553) or use Cloudflare's 'Verified Bots' setting to allow all Google-verified crawlers automatically.
How quickly does GMC recheck after I fix robots.txt?+
Google re-crawls your robots.txt typically within 24 hours of a change. Once Googlebot can access product pages again, products re-verify within 2–3 days and are re-approved within 5–7 business days. Expedite by triggering a manual feed fetch in GMC and requesting a recrawl in Search Console.