Home Life Trends How to Exclude WordPress Content from Google Search

How to Exclude WordPress Content from Google Search

Sometimes you want to exclude particular WordPress content material or information from being listed in Google search outcomes. Index, or “indexing” earlier than the emergence of Google and different search engine was a phrase principally related to books. It often resides in the back of most books, and this is the reason the Cambridge dictionary defines it on this context as:

Index: an alphabetical listing, similar to one printed in the back of a e book displaying which web page a topic, identify, and so on. is on.

Fast ahead to 1995, throughout the web growth, we now have companies like Yahoo search engine, and are available 1997, Google search has dramatically modified how we search and entry data on the web.

According to a survey executed in January 2018, there are 1,805,260,010 (over 1.eight billion) web sites on the web, and plenty of of these web sites get no guests in any respect.

What is Google Indexing?

There are completely different search engines like google and yahoo with a unique format of indexing, however the fashionable search engines like google and yahoo consists of, Google, Bing and for privacy-minded people, duckduckgo.

Google indexing usually refers to the method of including new net pages, together with digital content material similar to paperwork, videos and pictures, and storing them in its database. In different phrases, to ensure that your web site’s content material to seem on Google search outcomes, they first want to be saved in Google index.

What is Google Indexing?

What is Google Indexing?
Google is in a position to index all these digital pages and content material utilizing its spiders, crawlers or bots that repeatedly crawl completely different web sites within the Internet. These bots and crawlers do comply with the web site house owners’ directions on what to crawl and what needs to be ignored throughout crawling.

Why Websites Need to be Indexed?

In this period of the digital age, it’s virtually not possible to navigate by billions of internet sites discovering a specific matter and content material. It can be a lot simpler if there's a instrument to present us which websites are reliable, which content material is helpful and related to us. That’s why Google exists and ranks web sites of their search outcomes.

Indexing turns into an indispensable a part of how search engines like google and yahoo normally and Google particularly works. It helps establish phrases and expressions that finest describe a web page, and general contributes to web page and web site rating. To seem on the primary web page of Google your web site, together with webpages and digital information similar to videos, photographs and paperwork, first wants to be listed.

Indexing is a prerequisite step for web sites to rank properly on search engines like google and yahoo normally and Google particularly. Using key phrases, websites may be higher seen and found after being listed and ranked by search engines like google and yahoo. This then opens doorways for extra guests, subscribers and potential clients to your web site and business.

The finest place to cover a lifeless physique is web page two of Google.

While having lots of listed pages doesn’t robotically make your websites rank increased, if the content material of these pages is high-quality as properly you may get a lift by way of search engine optimization.

Why & How to Block Search Engine from Indexing Content

While indexing is nice for web site and business house owners, there are pages you might not need displaying up in search outcomes. you may danger exposing delicate information and content material on over the Internet as properly. Without passwords or authentication, non-public content material is liable to publicity and unauthorized entry if bots are given free rein over you web site’s folders and information.

In the early 2000s, hackers used Google search to display credit card information from web sites with easy search queries. This safety flaw was utilized by many hackers to steal card data from e-commerce web sites.

Another latest security flaw happened last year to box.com, a well-liked cloud storage system. The safety gap was uncovered by Markus Neis, menace intelligence supervisor for Swisscom. He reported that easy exploits of search engines like google and yahoo together with Google and Bing might expose confidential information and data of many business and particular person clients.

Cases like these do occur on-line and might trigger a loss in gross sales and income for business house owners. For company, e-commerce and membership web sites, it’s critically essential to first block search indexing of delicate content material and personal information after which most likely put them behind an honest consumer authentication system.

Let’s check out how one can management which content material and information that may be crawled and listed by Google and different search engines like google and yahoo.

1. Using Robots.txt For Images

Robots.txt is a file placed on the root of your web site offering Google, Bing and different search engines like google and yahoo bots with directions on what to crawl and what not. While robots.txt is often used to management crawling visitors and net (mobile vs desktop) crawlers, it is also used to stop photographs from showing in Google search outcomes.

A robots.txt file of regular WordPress web sites would appear to be this:

User-agent: *
Disallow: /wp-admin/
Disallow: /wp-includes/

The normal robots.txt file begins with an instruction for user-agent, and an asterisk image. The asterisk is an instruction for all bots that arrive on the web site to comply with all directions supplied beneath it.

Keep Bots Away From Specific Digital Files Using Robot.txt

Robots.txt will also be used to cease search engine crawling of digital information similar to PDFs, JPEG or MP4. To block search crawling of PDF and JPEG file, this needs to be added to the robots.txt file:

PDF Files

User-agent: *
Disallow: /pdfs/ # Block the /pdfs/listing.
Disallow: *.pdf$ # Block pdf information from all bots. Albeit non-standard, it really works for main search engines like google and yahoo.


User-agent: Googlebot-Image
Disallow: /photographs/cats.jpg #Block cats.jpg picture for Googlebot particularly.

In case you need to block all .GIF photographs from getting listed and displaying on google picture search whereas permitting different picture codecs similar to JPEG and PNG, you need to use the next guidelines:

User-agent: Googlebot-Image
Disallow: /*.gif$

Important: The above snippets will merely exclude your content material from being listed by third get together websites similar to Google. They are nonetheless accessible if somebody is aware of the place to look. To make information non-public so nobody can entry them you would want to use one other methodology, similar to these content restriction plugins.

The Googlebot-Image can be utilized to block photographs and a specific picture extension from showing on Google picture search. In case you need to exclude them from all Google searches, e.g. net search and pictures, it's advisable to use a Googlebot consumer agent as an alternative.

Other Google consumer brokers for various parts on an internet site consists of Googlebot-Video for videos from making use of within the Google video part on the net. Similarly, utilizing Googlebot user-agent will block all videos from displaying in google videos, net search, or mobile net search.

Robots txt No-Index

Robots txt No-Index
Please remember the fact that utilizing Robots.txt will not be an acceptable methodology of blocking delicate or confidential information and content material owing to the next limitations:

  • Robots.txt can solely instruct well-behaved crawlers; different non-compliant search engines like google and yahoo and bots might merely ignore its directions.
  • Robots.txt doesn't cease your server from sending these pages and information to unauthorized customers upon request.
  • Search engines might nonetheless discover and index the web page and content material you block in case they’re linked from different web sites and sources.
  • Robots.txt is accessible to anybody who might then learn all of your supplied directions and entry these content material and information straight

To block search indexing and defend your non-public data extra successfully, please use the next strategies as an alternative.

2. Using no-index Meta Tag For Pages

Using no-index meta tag is a correct and more practical methodology to block search indexing of delicate content material in your web site. Unlike the robots.txt, the no-index meta tag is placed within the <head> part of a webpage with a quite simple HTML tag:

<meta identify="robots" content material="noindex">

Any web page with this instruction on the header is not going to seem on Google search consequence. Other directives similar to dofollow and notranslate will also be used inform net crawlers not to crawl the hyperlinks and gives translation of that web page respectively.

You can instruct a number of crawlers through the use of a number of meta tags on a web page as follows:

<meta identify="googlebot" content material="dofollow">
<meta identify="googlebot-news" content material="nosnippet">

There are two methods to add this code to your web site. Your first choice is to create a WordPress child theme, then in your features.php you may make use of the WordPress wp_head motion hook to insert a noindex or another meta tags. Below is an instance of how you'd noindex to your login web page.

add_action( 'wp_head', operate() {
if ( is_page( 'login' ) ) {
echo '<meta identify="robots" content material="noindex">';
} );

Your second choice is to use your search engine optimization plugin to management a web page’s visibility. For instance, with Yoast search engine optimization you'll be able to go to the superior settings part on a web page and easily select “No” for the choices to enable search engine to present the web page:

Yoast SEO Search Results Setting

Yoast SEO Search Results Setting

3. Using X-Robots-Tag HTTP header for different information

The X-Robots-Tag offers you extra flexibility to block search indexing of your content material and information. In explicit, compared to the no-index meta tag it may be used because the HTTP header response for any given URLs. For instance, you should use the X-Robots-Tag for picture, video and doc information the place it’s not attainable to use the robots meta tags.

You can learn Google’s full robots meta tag guide, however right here’s how one can instruct crawlers not to comply with and index a JPEG picture utilizing the X-Robots-Tag on its HTTP response:

HTTP/1.1 200 OK
Content-type: picture/jpeg
Date: Sat, 27 Nov 2018 01:02:09 GMT
X-Robots-Tag: noindex, dofollow

Any directives that could possibly be used with an robots meta tag are relevant to an X-Robots-Tag as properly. Similarly, you'll be able to instruct a number of search engine bots as properly:

HTTP/1.1 200 OK
Date: Tue, 21 Sep 2018 21:09:19 GMT
X-Robots-Tag: googlebot: dofollow
X-Robots-Tag: bingbot: noindex
X-Robots-Tag: otherbot: noindex, dofollow

It’s essential to word that search engines like google and yahoo bots uncover the Robots meta tags and X-Robots-Tag HTTP headers throughout the crawling course of. So if you would like these bots to comply with your instruction not to comply with or index any confidential content material and paperwork, you need to not cease these web page and file URLs from crawling.

If they’re blocked from crawling utilizing the robots.txt file, your directions on indexing is not going to be learn, and so, ignored. As a consequence, in case different web sites hyperlink to your content material and paperwork, they are going to nonetheless be listed by Google and different search engines like google and yahoo.

4. Using .htaccess Rules for Apache Servers

You may add X-Robots-Tag HTTP header to your .htaccess file to block crawlers from indexing pages and digital contents of your web site hosted on a Apache server. Unlike no-index meta tags, .htaccess guidelines may be utilized a complete web site or a specific folder. Its help of standard expressions gives even increased flexibility for you to goal a number of file sorts without delay.

To block Googlebot, Bing and Baidu from crawling an internet site or particular listing, use the next guidelines:

RewriteEngine On
RewriteCond %{HTTP_USER_AGENT} (googlebot-bingbot-Baiduspider) [NC]
RewriteRule .* - [R=403,L]

To block search indexing of all .txt, .jpg, .jpeg, .pdf information across your complete web site, add the next snippet:

<Files ~ ".(txt-jpg-jpeg-pdf)$">
Header set X-Robots-Tag "noindex, dofollow"

5. Using Page Authentication with Username & Password

The above strategies will stop your non-public content material and paperwork from showing in Google search outcomes. However, any customers with the hyperlink can attain your content material and entry your information straight. For safety, it’s extremely advisable you arrange correct authentication with username and password in addition to function entry permission.

Using Page Authentication

Using Page Authentication
For occasion, pages that features private profiles of employees and delicate paperwork which should not be accessed by nameless customers needs to be pushed behind an authentication gate. So even when customers in some way handle to discover the pages, they are going to be requested for credentials earlier than they will try the content material.

WordPress Password Protect

WordPress Password Protect
To do that with WordPress merely set the visibility of a submit to password protected. This manner you'll be able to choose a password required to view the content material on that web page. This is pretty simple to do on a per-post/web page foundation. For extra complete web site privateness, strive including a membership plugin to your web site.

Please keep in mind that password-protected or hidden pages from search engines like google and yahoo and guests don't essentially defend the paperwork, videos and pictures connected to its content material. For actual safety of your WordPress file uploads, a premium service similar to Prevent Direct Access Gold comes extremely advisable.


In the race to be on web page considered one of Google, business house owners might not take into consideration what search engines like google and yahoo can see. Whether you’re blocking bots from a selected file, hiding a customized login web page, or password defending non-public consumer directories… there are many pages to contemplate no-indexing when it comes to search engines like google and yahoo.

Thats it, all about excluding content material from search outcomes or stopping sure information from bing listed