GoogleBot Now Indexing JavaScript, AJAX & CSS

Google Bot

Improving the way that GoogleBot parses and interprets content on the web has always been integral to the Google mandate. It now seems that GoogleBot has reverently been bestowed the ability to parse JavaScript, AJAX and Cascading Style Sheets.

In the past developers avoided the use of JavaScript to deliver content or links to content due to the inherent difficulty by the GoogleBot to correctly index this dynamic content. Over the years it has become so good at the task that Google is now asking us to allow the GoogleBot to scan JavaScript used in our websites.

Google did not release specific details of how or what the GoogleBot does with the JavaScript code it finds due to fears that the knowledge would quickly be incorporated into BlackHat tactics designed to game Search Engine Results Pages (SERPs). A recent blog post on Tumblr is responsible for the media attention. The post has shown server logs where the bot was shown to be accessing JavaScript files.

The ability for the GoogleBot to successfully download and parse dynamic content is a huge leap forward in the indexing of the web and stands to cause many fluctuations in rankings as sites are re-crawled and re-indexed with this dynamic content now factored in to the page’s content.

Previously Google attempted to get developers to standardize the way dynamic content was handled so that it could crawl but the proposal (https://developers.google.com/webmasters/ajax-crawling/) has been more or less ignored.

The GoogleBot has to download the JavaScript and execute it on the Google Servers running the GoogleBot leading some to the conclusion that it may be possible to use the Google Cloud to compute data at a large scale.