Google Webmaster Tools - Crawler Access
See below for screenshot and explanation of Crawler Access.
Here are the Google webmaster tools categories as they look when you are logged in (left).
Below is the same structure with clickable links which go to a Hard Web webpage with a screenshot of what is on that webmaster tools category page (in my own webmaster tools account).
- Site Configuration
- Your site on the web
- +1 Metrics
Webmaster tools - Site Configuration - Crawler Access
The Crawler Access tool tells you whether you have a robots.txt file submitted or not.
This category should probably have been called Robots.txt files in my opinion.
The page also has a tool to generate a robots.txt file and to ask for removal of a specific URL from the Google index.
I used these tools in tandem recently (Robots.txt and URL removal) and began to feel there was a conflict whereby if I had a 301 permanent redirect on a URL (or folder) AND I had asked for it to be removed AND I had excluded the page (or "directory" aka "folder") via the robots.txt file...
Was I sending the robots on a mission only for them to hit a redirect? I thought that might hurt their feelings (and my rankings).
Help for the Site Configuration master category within Webmaster Tools (new window).
See the Webmaster Tools help page on "How to Block or remove pages using a robots.txt file" (new window).
Here is a screenshot of the category " Crawler Access" from within Webmaster tools (below).
Screenshots on this page are copyright © 2018 Google Inc.
If this page is not helping then you must go to the fount...Google Webmaster Tools Help (new window).
Also click to signup to get your own Webmaster tools account (new window).
Contact Hardweb - making websites work!
Web Design SEO SEM
07 5526 2939
0401 352 269
25 Savoy Drive
Gold Coast Queensland 4218