Everything about site indexing in search engines

Site indexing in search engines is something without which SEO promotion is impossible. If the pages of your site are not in the index, they are not displayed in the search results, which means that you are not yet participating in the competition for the top positions, and users from the search cannot go to you.

We figure out how Yandex and Google index sites, whether it is possible to speed up and control the process and which pages do not need indexing.

What is indexing and how does it work

Search engines on request give users suitable pages in a couple of seconds. Naturally, at this time the robots do not go through real sites - such a search would drag on for hours. To speed up the search engines turn to their own pre-assembled database. This base is the index.

Search engines index the site automatically, using search robots - they are called spiders or crawlers. Simplified it looks like this:

• Spiders are constantly monitoring new sites and pages on the Internet.
• Once on the site, the robot scans it. If the root folder contains a correctly filled robots.txt file (and we strongly advise you to register and place it there), the crawler will quickly understand which pages need to be indexed and which ones should not.
• On each page, the robot automatically evaluates the most important elements on which the ranking position depends (keywords, text, structure, meta tags, etc.).
• The indexed pages are uploaded to the database. During each visit, the robot processes not so many pages, up to 30 pages, so as not to overload your server.
• After the site is indexed, the pages begin to appear in the search results, because now the search engine knows about them.
• In the future, the robot continues to visit your site in order to crawl and add new pages to the database. Also, search engines can remove pages from the index, having found technical errors on them. If you publish content on a daily basis, the robot can visit you several times a day, and if the site remains unchanged for a long time, then search engines will pay less attention to it (up to 3 times a month).

How to get into the index faster

For successful promotion in Yandex and Google, it is important that new pages get into the index as soon as possible and start ranking. Therefore, we advise you to use the opportunities for accelerated indexing of the site.

How to attract the attention of search engines and take a place in the search results faster:

1. It is imperative to create a service robots.txt file and a Sitemap to help search bots.
2. Log in to the Google Search Console and Yandex.Webmaster control panels and leave a request for indexing specific pages.
3. Ensure the reliability and speed of the site - the robot, when it detects problems on the site, reduces the crawl frequency.
4. Take care of high-quality external links, which will add "weight" to you and increase the chances of a quick indexing of the site by keywords.
5. Regularly add new pages and publish fresh materials - then search engines will more often pay attention to your resource. Be sure to add new materials to the "Original Texts" in Yandex.Webmaster.
6. Optimize the site in terms of user friendliness - usability . First of all, an understandable directory and navigation system, a network of cross-internal links (interlinking) are important - everything that speeds up the movement of the bot on your site and helps it to index more pages at a time.

Also Read: How to Fast Index URL

How to get into the index faster

For successful promotion in Yandex and Google, it is important that new pages get into the index as soon as possible and start ranking. Therefore, we advise you to use the opportunities for accelerated indexing of the site.

How to attract the attention of search engines and take a place in the search results faster:

1. It is imperative to create a service robots.txt file and a Sitemap to help search bots.
2. Log in to the Google Search Console and Yandex.Webmaster control panels and leave a request for indexing specific pages.
3. Ensure the reliability and speed of the site - the robot, when it detects problems on the site, reduces the crawl frequency.
4. Take care of high-quality external links, which will add "weight" to you and increase the chances of a quick indexing of the site by keywords.
5. Regularly add new pages and publish fresh materials - then search engines will more often pay attention to your resource. Be sure to add new materials to the "Original Texts" in Yandex.Webmaster.
6. Optimize the site in terms of user friendliness - usability . First of all, an understandable directory and navigation system, a network of cross-internal links (interlinking) are important - everything that speeds up the movement of the bot on your site and helps it to index more pages at a time.

By the way, not only new sites need to quickly index pages. If you've updated prices or catalog descriptions, added attractive Description meta tags, or uploaded hot news to your blog, it's also important for you that indexing doesn't have to wait for months.

How to check the indexing of a site

In the first weeks after the launch of a new site, it is important to check whether the necessary pages have got into the index of search engines and how many pages are indexed in general. But in the future, we advise you to keep your finger on the pulse.

Here are the main ways to check the indexing of a site in search engines:

1. Manually via branded queries

We enter the name of the company in the search bar and look at the first couple of search results pages. The method is the simplest, without technical difficulties, but not fast and not very convenient.

2. Through the site operator in the search bar

A little more complicated, but more accurate: we drive in the operator site in the search for Yandex or Google, and after it - the address of the site.



For example, let's type the URL zamkitut.ru to check the indexing of the site. The search results will contain only the indexed pages of the given site, and you will not have to select anything manually.

3. In Yandex.Webmaster or Search Console in Google

A method for those who have access to the webmaster's panel. In Google, you need to open the "Google Index" tab and select "Indexing status", and in Yandex.Webmaster, expand the "Site indexing" menu, and then "Pages in search". Here you can see not only the current number of pages in the index, but also the dynamics.

4. Automatically using plugins, scripts or special services

Services and programs are a fast and reliable way to check the indexing of a site online. Don't manually type in requests: small verification programs can usually be saved in the browser as bookmarks and run when needed.

We advise you to try the RDS bar plugin for different browsers, as well as programs like Netpeak Spider or Netpeak Checker.

This is how Netpeak Spider's working window looks like: the program gives out a lot of additional information besides the number of pages in the index.

Methods 3 and 4 are the most informative and convenient, we advise you to monitor indexing in the course of website promotion in this way. In a special program or on the webmaster's panel, you will see the last time Yandex indexed the site, which pages were included in the index, what problems were identified by search engines, etc.

Why the site is not indexed and how to fix it

Sometimes there are problems with indexing: the entire site or individual pages are no longer displayed in the search results. Why Yandex does not index the site and how to fix it:

1. Not enough time has passed. If a site or a group of pages appeared recently, the search robot might simply not have time to index them - use our acceleration techniques from the list above.
2. Incorrectly filled robots.txt file. It is better to entrust the work with this service file to programmers, because an accidental error in it can prohibit indexing the site without your knowledge.
3. Errors that are critical for search engines on the site - it is important to find them and correct them in time.
4. No sitemap (SiteMap). Be sure to write it down and add it to the root folder - the map will help speed up the indexing of the site.
5. The site contains duplicate pages or content completely copied from other resources - such pages are fundamentally not indexed.
6. The server is working intermittently. If the site is intermittently unavailable or is loading too slowly, there may be problems with indexing, so it makes sense to move to a more reliable hosting.

How to close a site from indexing and when it is needed

Sometimes it is necessary to prohibit the indexing of a site or individual pages.

Why is this done:

1. So that the service pages of the site do not get into the search results: the user's personal account, shopping cart, control panel, etc.
2. So that the search robot gets to the landing pages faster and indexes them, and does not waste time on insignificant content that does not need to be promoted.
3. So that duplicate pages do not get into the index (search engines do not like this).

An example of a page that needs to be closed from indexing is a basket in an online store of door accessories.

The main ways to "hide" unnecessary pages or sections from search engines:

1. In the robots.txt file, the Disallow directive allows you to close the site from indexing - it is written next to the addresses of pages that should not be included in the search results.
2. In the page code, you can use the "noindex" tag - it does not block links from indexing, but only a part of the text enclosed within the tag.

Important: "noindex" only reads and understands Yandex. Use other methods from this list to manage indexing on Google.

3. The rel = "nofollow" attribute is applied to a specific link and tells search robots not to follow it. This attribute is understood by all search engines.
4. Meta robots tag - it gives search engines guidance on indexing. If you write "noindex, nofollow" in the content of this meta tag, the page will not be indexed.