Google Webmaster Tools - Crawler Access


Click to +1 this page

See below for screenshot and explanation of Crawler Access.

Categories within Google's suite of Webmaster Tools

Here are the Google webmaster tools categories as they look when you are logged in (left).
Below is the same structure with clickable links which go to a Hard Web webpage with a screenshot of what is on that webmaster tools category page (in my own webmaster tools account).

Webmaster tools - Site Configuration - Crawler Access

The Crawler Access tool tells you whether you have a robots.txt file submitted or not.
This category should probably have been called Robots.txt files in my opinion.
The page also has a tool to generate a robots.txt file and to ask for removal of a specific URL from the Google index.
I used these tools in tandem recently (Robots.txt and URL removal) and began to feel there was a conflict whereby if I had a 301 permanent redirect on a URL (or folder) AND I had asked for it to be removed AND I had excluded the page (or "directory" aka "folder") via the robots.txt file...
Was I sending the robots on a mission only for them to hit a redirect? I thought that might hurt their feelings (and my rankings).

Help for the Site Configuration master category within Webmaster Tools (new window).
See the Webmaster Tools help page on "How to Block or remove pages using a robots.txt file" (new window).

Here is a screenshot of the category " Crawler Access" from within Webmaster tools (below).

From within a webmaster tools account this graphic is a  screenshot of the Crawler Access category

Screenshots on this page are copyright © 2018 Google Inc.
If this page is not helping then you must go to the fount...Google Webmaster Tools Help (new window).
Also click to signup to get your own Webmaster tools account (new window).

 

Contact Hardweb - making websites work!



Web Design SEO SEM

07 5526 2939
0401 352 269

Home/Office Address
25 Savoy Drive
Broadbeach Waters
Gold Coast Queensland 4218

07 5526 2939
0401 352 269