Fixing Index Bloat: A Robots.txt & Sitemap Strategy for 1M+ Pages
Client: A large online marketplace with over 1 million product listings.
Challenges We Faced
The client’s site was suffering from severe index bloat and crawl waste, costing them organic visibility:
Wasted Crawl Budget
Googlebot was spending 90% of its time crawling low-value search result pages (e.g., ?search=red+shoes) instead of actual products.
Outdated Sitemaps
Their XML sitemaps were static and manual, missing thousands of new products.
Index Bloat
Google Search Console (GSC) showed 2.5 million pages indexed, but only 500,000 were valuable products.
Robots.txt Errors
Their robots.txt file was blocking CSS files, preventing Google from rendering their pages correctly.
Save Your Crawl Budget
“Is Google crawling the wrong pages? Our Robots.txt Optimization services will guide bots to your money pages.”
Our Approach – How We Solved These Challenges
Robots.txt Optimization
Our robot.txt configuration SEO company team audited their logs and updated the robots.txt file to Disallow: /search? and other low-value parameters.
We unblocked the CSS/JS resources to fix rendering issues.
XML Sitemap Creation & Optimization
We built a new, dynamic XML sitemap system that auto-updated daily.
We split the sitemap into smaller “child” sitemaps (e.g., sitemap-products-1.xml) to help Google process them faster.
Crawl Budget Analysis
We used Log File Analysis to confirm that after the robots.txt update, Googlebot shifted its attention to the product pages.
Pruning & Index Bloat Removal
We submitted a removal request for the 2 million low-value search URLs.
Results
| Metric | Before | After | Growth |
|---|---|---|---|
| Valid Pages Indexed (GSC) | 20% (of total index) | 95% (of total index) | +375% |
| Crawl Frequency (Product Pages) | Once every 30 days | Once every 3 days | +900% |
| Organic Traffic | 850,000/mo | 1,400,000/mo | +64% |
Free Crawl Budget Audit
“Not sure if you’re wasting crawl budget? Claim a Free Log File Analysis from our sitemap & robots.txt agency!”
Advice for Marketers & Brand Owners
- Stop Google from crawling your search results. Use Robots.txt Optimization to block internal search pages. It’s the #1 cause of index bloat.
- Dynamic sitemaps are mandatory for large sites. If you’re manually updating XML files, you’re already behind.
- Monitor your “Coverage” report in GSC. It tells you exactly which pages are excluded and why.
Extra Factors That Made It Work
- The Robots.txt Optimization was the single biggest factor in saving their crawl budget.
- Splitting the sitemaps allowed us to pinpoint exactly which product categories had indexing issues.
- Unblocking the CSS files fixed “Mobile Usability” errors in GSC, giving an additional ranking boost.
