How search engines work

Google describes its work in three stages. Not every page reaches every stage.

1. Stage 1

   Crawling

   Googlebot discovers URLs through links and sitemaps, downloads them and renders JavaScript.

   Stops here if: robots.txt rules, logins, server errors, no links to the page

2. Stage 2

   Indexing

   Google analyses the content, groups duplicates, picks a canonical URL and stores the page.

   Stops here if: Non-200 status, noindex, duplicates, low-quality content

3. Stage 3

   Serving

   For each search, Google orders the indexed pages by relevance and usefulness to that searcher.

   Stops here if: Weaker relevance or quality than competing pages

1. Crawling: finding pages

There is no central list of every web page. Google's crawler, Googlebot, discovers URLs mainly by following links from pages it already knows, and from sitemaps that site owners submit.

When Googlebot fetches a page, it also renders it with a recent version of Chrome, which means it runs the JavaScript. Rendering takes resources and can fail, so content that only appears after complex scripts run is a common source of problems.

What blocks crawling:

• A rule in robots.txt that disallows the URL.
• Pages behind a login.
• Server errors or very slow responses, which make Googlebot crawl less.
• No links pointing to the page. These are called orphan pages.

2. Indexing: understanding pages

After crawling, Google analyses the page's text, images, video and key tags, such as the <title> element and alt attributes. It stores what it learns in the Google index.

During indexing Google also decides whether the page is a duplicate. When several URLs show the same or very similar content, Google groups them and picks one canonical URL to represent the group.

What prevents indexing:

• The page does not return HTTP status 200. Google's technical requirements say error pages are not indexed.
• A noindex rule in a meta tag or HTTP header.
• Content Google judges low quality, or a page that is mostly a duplicate.

Indexing is not guaranteed even when nothing is blocking it.

3. Serving: showing results

When someone searches, Google looks up matching pages in its index and orders them by what it considers most relevant and useful. Hundreds of factors play a part. Some, such as the searcher's location, language and device, come from the person searching rather than from your page.

Diagnosing a missing page

When a page does not show up in search, work through the stages in order:

1. Can it be crawled? Check robots.txt, whether links to it exist, and whether the server responds.
2. Is it indexed? Use Search Console's URL Inspection tool, which you will set up in a later lesson.
3. Does it rank for anything? If it is indexed but gets no impressions, the problem is relevance or competition, not access.

This order saves a lot of time. There is no point rewriting a page's content if Google has never been allowed to crawl it.