Do you know about Fetch as Googlebot? It is a feature of Webmaster Tools that allows you to input a URL, evaluate it, and (should you choose) submit it to the Google index for a fresh crawl. In this post, I am going to show you how to use the Fetch as Googlebot feature like an SEO samurai.
Fetch as Googlebot has two modes.
Fetch – Quick Check
In this mode, Googlebot will crawl a URL based on a path you specify, and you can then view the response your site sent to Googlebot. This can be used to check connectivity, basic errors, redirects or security issues with your website.
Fetch & Render – Deeper View
In this mode, Google crawls the URL and displays the page the way a browser would.
Here are the basic steps:
There are four different Googlebot requests you can make in Fetch as Google.
If you would like to see how a desktop browser sees your site, select Desktop. To see how smartphone mobile crawlers see your site, choose Mobile Smartphone.
If you want to see how the mobile crawler for a feature phone sees the site, select Mobile cHTML or Mobile XHTML/WML. (Google notes that the Mobile cHTML is mainly for Japanese websites.)
If you do a basic Fetch, you can see the HTTP response that Google got from your page. When you do Fetch and Render, you get the response and get to see how your page looks in the specified browser.
(Note: The “Fetch And Render” option is not available for the WML and cHTML at this time, so unfortunately you won’t be able to see how your page renders for feature phones.)
Once you’ve entered your URL and fetched it, you will have the option of submitting it to Google for indexation. You’ll see a button allowing you to do so. (This button will also appear on the upper right-hand corner of the Fetch details page.)
After you submit a URL to be fetched, you can click on the path once it’s completed. This will bring you to a page with further details. At the top, you’ll see the URL that was fetched, the Googlebot type, and when it was completed. If you chose to Fetch and Render the page, you’ll see a “Fetching” tab and a “Rendering” tab.
The Fetching tab shows you the downloaded HTTP response, which can be very helpful if you are ensuring a page status, and it also shows you the code of the page. Finally, it shows you the download time of the page at the bottom.
The Rendering tab will give you an idea of how your page renders in the browser selected (desktop or smartphone). It will also list any page resources that Googlebot was not able to access.
So, there actually is a quota on fetches and submissions, but most of us will never exceed it. According to Google, you are limited to 500 fetches per week. (You’ll get a notification that you’re approaching your limit once you have hit 400.)
When it comes to actual submissions to the index, they give you 500 individual URL submissions and 10 submissions where Google will crawl that URL and all of the direct links as well.
With this tool, you can submit a site a to get it indexed quicker, see if a page is blocked by robots.txt, see if a page is being redirected, make sure a page works on mobile, see if your server is taking too long to respond and so much more. But you can’t do any of that if you don’t know what the fetch statuses mean. So here they are.
Fetch And Render Statuses
Now that we have covered the basics, I want to give you a few ideas on how you can use this tool.
As you can see, there are many applications for this tool. One thing that got me really excited about it and made me want to write this article was the indexing. I made some significant changes to a page, used this tool to submit it to the index, and within a few hours, I was visible for a competitive keyword. So, as you can see, this tool can help in a variety of areas.
This tool is a useful one to have in your arsenal. For me, it is right up there with the mobile testing tool, page speed tool, robots.txt tester and disavow tool in its ability to deliver useful technical SEO insights.
Do you use Fetch as Googlebot in a way I didn’t mention? Have a good story? Comment below.