Google Search is a fully-automated search engine that uses software known as web crawlers that explore the web regularly to find pages to add to our index.
In fact, the vast majority of pages listed in our results aren't manually submitted for inclusion, but are found and added automatically when our web crawlers explore the web.
This document explains the stages of how Search works in the context of your website. Having this base knowledge can help you fix crawling issues, get your pages indexed, and learn how to optimize how your site appears in Google Search.
Before we get into the details of how Search works, it's important to note that Google doesn't accept payment to crawl a site more frequently, or rank it higher. If anyone tells you otherwise, they're wrong.
Google doesn't guarantee that it will crawl, index, or serve your page, even if your page follows the Google Search Essentials.
Introducing the three stages of Google Search:
Google Search works in three stages, and not all pages make it through each stage:
1. Crawling: Google downloads text, images, and videos from pages it found on the internet with automated programs called crawlers.
The first stage is finding out what pages exist on the web. There isn't a central registry of all web pages, so Google must constantly look for new and updated pages and add them to its list of known pages.
This process is called "URL discovery". Some pages are known because Google has already visited them. Other pages are discovered when Google follows a link from a known page to a new page: for example, a hub page, such as a category page, links to a new blog post.
Once Google discovers a page's URL, it may visit (or "crawl") the page to find out what's on it. We use a huge set of computers to crawl billions of pages on the web.
During the crawl, Google renders the page and runs any JavaScript it finds using a recent version of Chrome, similar to how your browser renders pages you visit. Rendering is important because websites often rely on JavaScript to bring content to the page, and without rendering Google might not see that content.
Crawling depends on whether Google's crawlers can access the site. Some common issues with Googlebot accessing sites include
a) Problems with the server handling the site(b) Network issues© robots.txt rules preventing Googlebot's access to the page.
2. Indexing: Google analyzes the text, images, and video files on the page, and stores the information in the Google index, which is a large database.
After a page is crawled, Google tries to understand what the page is about. This stage is called indexing and it includes processing and analyzing the textual content and key content tags and attributes, such as <title> elements and alt attributes, images, videos, and more.
During the indexing process, Google determines if a page is a duplicate of another page on the internet or canonical. The canonical is the page that may be shown in search results. Google also collects signals about the canonical page and its contents, which may be used in the next stage, where we serve the page in search results.
Some signals include the language of the page, the country the content is local to, the usability of the page, and so on. Indexing also depends on the content of the page and its metadata.
Some common indexing issues can include:
(a) The quality of the content on page is low
(b) Robots meta rules disallow indexing
© The design of the website might make indexing difficult.
3. Serving search results: When a user searches on Google, Google returns information that's relevant to the user's query.
READ FULL GUIDE HERE!
In fact, the vast majority of pages listed in our results aren't manually submitted for inclusion, but are found and added automatically when our web crawlers explore the web.
This document explains the stages of how Search works in the context of your website. Having this base knowledge can help you fix crawling issues, get your pages indexed, and learn how to optimize how your site appears in Google Search.
Before we get into the details of how Search works, it's important to note that Google doesn't accept payment to crawl a site more frequently, or rank it higher. If anyone tells you otherwise, they're wrong.
Google doesn't guarantee that it will crawl, index, or serve your page, even if your page follows the Google Search Essentials.
Introducing the three stages of Google Search:
Google Search works in three stages, and not all pages make it through each stage:
1. Crawling: Google downloads text, images, and videos from pages it found on the internet with automated programs called crawlers.
The first stage is finding out what pages exist on the web. There isn't a central registry of all web pages, so Google must constantly look for new and updated pages and add them to its list of known pages.
This process is called "URL discovery". Some pages are known because Google has already visited them. Other pages are discovered when Google follows a link from a known page to a new page: for example, a hub page, such as a category page, links to a new blog post.
Once Google discovers a page's URL, it may visit (or "crawl") the page to find out what's on it. We use a huge set of computers to crawl billions of pages on the web.
During the crawl, Google renders the page and runs any JavaScript it finds using a recent version of Chrome, similar to how your browser renders pages you visit. Rendering is important because websites often rely on JavaScript to bring content to the page, and without rendering Google might not see that content.
Crawling depends on whether Google's crawlers can access the site. Some common issues with Googlebot accessing sites include
a) Problems with the server handling the site(b) Network issues© robots.txt rules preventing Googlebot's access to the page.2. Indexing: Google analyzes the text, images, and video files on the page, and stores the information in the Google index, which is a large database.
After a page is crawled, Google tries to understand what the page is about. This stage is called indexing and it includes processing and analyzing the textual content and key content tags and attributes, such as <title> elements and alt attributes, images, videos, and more.
During the indexing process, Google determines if a page is a duplicate of another page on the internet or canonical. The canonical is the page that may be shown in search results. Google also collects signals about the canonical page and its contents, which may be used in the next stage, where we serve the page in search results.
Some signals include the language of the page, the country the content is local to, the usability of the page, and so on. Indexing also depends on the content of the page and its metadata.
Some common indexing issues can include:
(a) The quality of the content on page is low
(b) Robots meta rules disallow indexing
© The design of the website might make indexing difficult.
3. Serving search results: When a user searches on Google, Google returns information that's relevant to the user's query.
READ FULL GUIDE HERE!
What do you think about this post?
We want to hear from you all.
