How to Stop Search Engines from Crawling a WordPress Site

How can you stop search engines from crawling and indexing your WordPress site?

There are numerous reasons why you would want to prevent search engines from indexing your website and including it in search results.

You may prevent search engines from indexing your WordPress site by following the instructions in this article.

Why preventing a Search Engine from Crawling a WordPress Site?

The majority of visitors to WordPress sites come from search engines. You might be wondering why someone would wish to prevent search engines from indexing WordPress.

Search engines shouldn’t index your website in the following situations:

• When starting out, you may not know how to create a local development environment or a staging site, and instead develop your website while it’s live. You won’t want Google to index your site or blog page when it’s under construction or in maintenance mode.
• There are also many users that use WordPress to create private blogs. They don’t want search engines to list them because they’re private.
• When WordPress is used as an intranet or for project management, you really don’t want your internal documents to be publicly accessible.

A common misunderstanding is that search engines won’t find your website if you don’t have links pointing to it. 

For example, if someone else owned the domain name before you, there may be links to your site. Also, there are millions of web pages that just list domain names. One of these could list your site.

Block search engines on WordPress

Now that we’ve covered that, let’s look at how to stop search engines from crawling your site. Four methods to do so:

Method I: Asking Search Engines not to Crawl Your WordPress Site

This is the simplest method of preventing web crawlers from accessing your site, but it is not foolproof.

WordPress allows you to prevent search engines from indexing your site. This function is standard. Simply select “Search Engine Visibility” from the Settings » Reading menu.

After enabling this option WordPress will modify your header and add this line :

<meta name=’robots’ content=’noindex,follow’ />

WordPress will put these lined to your site’s robots.txt file:

User-agent: *

Disallow: /

With these lines, you tell web crawlers (also called “robots” or “search engine spiders”) not to index your pages. It is, however, ultimately up to individual search engines to comply or not. Although though the majority of search engines will respect this, there is still a possibility that some of your site’s pages or photos will be indexed.

If you don’t want search engines to be able to index or crawl your site, you’ll need to use Methods 3 or 4 to password-protect your WordPress site.

Method II: Forbid Search Engines to crawl individual Pages

In some cases, you may want search engines to scan and index your website, but you would prefer that some articles or pages not appear in the resulting search results.

Using an SEO plugin like Yoast SEO or RankMath SEO is the easiest way to do that. Here are the steps you need to take for RankMath, but they are mostly the same for other plugins. RankMath is one of several good SEO tools for WordPress and is trusted by over a million users.

For this tutorial, we’ll be using the free version as it includes everything we need. There is also a premium version of RankMath that offers some more features but usually the free version is powerful enough.

After activating RankMath you can use it to ask search engines not to index certain posts and pages. 

Open the post or page that you want to exclude from the Google Index. Scroll down to the RankMath Settings at the bottom of the WordPress editor and then click the ‘Advanced’ tab.

You will see the default robots txt settings. If you want to change them you need to switch the ‘Use Default Settings’ toggle to the off position.

Select ‘No Index’. Once the post is published or updated, the search engine will be notified the next time it tries to crawl the URL. 

Method III: Password Protecting an Entire Site Using cPanel

If your WordPress hosting company gives you access to cPanel to manage your hosting account, you can use it to protect your whole site. Simply navigate to the ‘Files’ section of your cPanel dashboard and select the ‘Directory Privacy’ icon.

Next, select your WordPress root folder which is usuallt the public_html folder. Click the ‘Edit’ button next to that folder.

Note: If you have more than one WordPress site in the public HTML directory, you have to click on the public html link to see all of them and then edit the folder for the site you want to protect with a password.

You’ll be taken to a screen where you may create a password.

Just click the “Save” button and tick the box that says “Password guard this directory.” If you choose, you may even modify the name of the encrypted folder.

A confirmation notice stating that the directory’s access rights have been modified will appear.

The next step is to select “Back” from the menu.

You’ll be directed to a screen where you can establish a username and password that will need to be used while accessing this directory.

After clicking the “Save” button you have successfully protected your WordPress site with a password.

The credentials you set up before will now be required anytime a user or search engine accesses your website.

Method IV: Password Protecting WordPress using a Plugin

You might not have access to cPanel if you’re using a managed WordPress hosting service. If that’s the case, you can install a plugin for WordPress that requires login credentials to access the site.

Password Protected

Password Protected is a plugin for WordPress that allows you to easily create password-protected pages and posts on your website. This is useful if you have content on your site that you only want certain users to be able to access.

With this plugin, you can create a password for any page or post on your site, and then share that password with the users who you with the users who you want to be able to access that content. This can be a useful way to easily manage access to private or sensitive information on your site.

This plugin does not protect images or uploaded files so if you enter the exact URL to an image file it will still be accessible.

Features include:

• • Password protects your WordPress site with a single password.
  • Option to allow access to feeds.
  • Option to allow administrators access without entering password.

Passster

One of the most widely used security plugins for WordPress is Passster. It lets you select to safeguard the full site or just the pages and posts you wish.

By appending a short area code, Passster can establish encrypted, password-protected sections of your website. It also integrates with WordPress page builders like Elementor, making your life much easier.

You can restrict access to specific pages or direct users to specific pages on the website using the Global Protection feature. A password or a captcha can also be used to secure yourself. Both options are available to you thanks to this plugin.

Also, this plugin makes it simpler to access passwords by allowing you to store them in cookies. The professional edition also includes an automated password expiration option.

We sincerely hope that this post was helpful in preventing search engines from indexing or crawling your WordPress site. You may also want to learn how to speed up your website performance or get an insight who really owns WordPress.