Crawling and indexing in practice

The fundamentals module explained crawling, indexing and serving. This lesson is about reading the evidence: what Search Console tells you about your pages, and what to do with it.

The Page indexing report

Search Console's Page indexing report shows how many of your known URLs are indexed, and groups the rest by reason.

Reason	What it means	Usually
Page with redirect	The URL redirects elsewhere	Expected
Alternate page with proper canonical tag	A variant correctly points to its canonical	Expected
Excluded by "noindex" tag	The page asks not to be indexed	Expected if intended. Check for mistakes
Duplicate without user-selected canonical	Google sees a duplicate and chose another URL	Check that Google chose the right one
Blocked by robots.txt	Google may not crawl it	Check that the block is intended
Soft 404	Looks like an error page but returns 200	A problem. Return a real 404 or add real content
Not found (404)	The URL returns 404	Fine for removed pages. A problem if linked internally
Server error (5xx)	The server failed	A problem
Discovered – currently not indexed	Known but not crawled yet	Watch it. Many such URLs on large sites can mean crawl budget issues
Crawled – currently not indexed	Crawled, but Google chose not to index it for now	Check important pages. Improve or consolidate thin ones

Method: for each reason, open a handful of example URLs and ask whether the page should be indexed at all. If not, the exclusion is correct. If yes, investigate with URL Inspection.

URL Inspection for single pages

For a specific page, URL Inspection shows whether it is indexed, when it was last crawled, whether crawling and indexing are allowed, and which canonical Google selected. Test live URL shows what Googlebot sees right now, including the rendered HTML.

Crawl budget

Google can and wants to crawl only a certain number of URLs on a site. Google calls this the crawl budget. It depends on:

• Crawl capacity: how much crawling the server handles without slowing down. Slow responses, 5xx errors and 429 responses reduce it.
• Crawl demand: how many URLs Google knows about, how popular they are, and how often they change.

For large sites, Google's recommendations include consolidating duplicates, keeping unimportant URLs (such as endless filter combinations) out of crawling with robots.txt, returning 404 or 410 for removed pages, fixing soft 404s, avoiding long redirect chains, and keeping pages fast.

Size limits

Googlebot processes only the first 2 MB of an HTML file (uncompressed), and each resource it fetches separately, such as CSS and JavaScript files, has the same limit. Content beyond that is ignored. Very large pages, often with huge inline scripts or data, can lose content at the end.

Crawl stats

Search Console's Crawl stats report (under Settings) shows how many requests Googlebot makes, response times and status codes over time. A sudden rise in server errors or response time is an early warning before indexing problems appear.