Indexing large e-commerce catalogues
On a large catalogue the constraint is almost never the individual product page. It is that the site generates far more URLs than it earns crawls, and the products compete with filter combinations for the same budget.
A 400-page shop and a 40,000-page shop have different problems with the same name.
The arithmetic that decides everything
Take your average daily crawl requests from Crawl Stats and divide your indexable URL count by it. That is the theoretical number of days for a single full pass — assuming every crawl went to a page you care about, which none of them do. Sites that discover this number is measured in months usually stop asking why new products take six weeks to appear.
Where the budget actually goes
- Faceted navigation. Four filters with five options each is 625 combinations per category, most of which are near-duplicates of each other.
- Sort and view parameters.
?sort=price&view=gridmultiplies every listing page. - Variant URLs. Separate URLs per size or colour, each near-identical.
- Internal search results. Crawlable search pages generate URLs without limit.
- Out-of-stock and discontinued pages that were never retired.
Google's crawl budget guidance addresses this directly, and its practical implication is uncomfortable: reducing the URL space usually helps the products more than promoting the products does.
Segment before you submit
| Segment | Priority | Approach |
|---|---|---|
| Best sellers, high-margin lines | Highest | Submit immediately on publish; link from category hubs and homepage |
| New arrivals | High | Batch daily, drip-feed if the batch is large |
| Long tail, low volume | Low | Rely on category linking; do not spend submissions here |
| Variants | None | Canonical to the parent product |
| Filter combinations | None | Robots rules or parameter handling |
The value of the table is the bottom half. Most catalogue indexing campaigns fail because they treat all 40,000 URLs as equally deserving, which guarantees that the 300 that pay the bills get the same attention as the filter pages.
The quality gate still applies
Product pages carrying only the manufacturer's description are duplicates of every other retailer selling the same item. They will typically land in crawled – currently not indexed regardless of how you schedule them. Original photography, real specifications, genuine reviews and stock information are what distinguishes the page — and no submission strategy substitutes for that.
A working cadence
- Cut the URL space first. Measure the change in Crawl Stats over four weeks.
- Fix category-to-product link depth so nothing sits more than three clicks from the homepage.
- Only then start submitting, segmented, at a daily volume anchored to the crawl baseline — see how many URLs per day.
Questions
Should out-of-stock product pages be removed?
Usually not, if the item is returning — removing and restoring URLs wastes the discovery work already done. If it is discontinued permanently, redirect to the closest equivalent or the parent category.
Do product variants need separate indexed URLs?
Rarely. Variants that differ only by size or colour are near-duplicates and normally canonicalise to the parent, which frees crawl budget for pages that can actually rank.
Segment first, then submit the segment that pays
One task takes up to 25,000 URLs. Point it at the products that matter, not at the filter combinations.
Index a product segment