This week Google announced an update to their technical Webmaster Guidelines that addresses the long time discussion regarding bots and the crawling of JavaScript, CSS and image files for optimal rendering and indexing.

Historically, to try and handle things such as the page layout algorithm (for better ranking purposes) many webmasters have blocked Googlebot from crawling both CSS and JavaScript. The new guideline now specify advises that webmasters should allow Googlebot access to the JavaScript, CSS, and image files that a page uses and that doing so will provide a site with optimal indexing.

Pierre Far, Google’s Webmaster Trends Analyst, stated that “Disallowing crawling of Javascript or CSS files in your site’s robots.txt directly harms how well our algorithms render and index your content and can result in suboptimal rankings”. Pierre Far also suggested that webmasters should double check their mobile sites to verify they aren’t blocking CSS and JavaScript.

From the technical guidelines, the following section has been added:
To help Google fully understand your site’s contents, allow all of your site’s assets, such as CSS and JavaScript files, to be crawled. The Google indexing system renders webpages using the HTML of a page as well as its assets such as images, CSS, and Javascript files.

So how does one know if their site is blocking this important information? Using the fetch and render tool that was created several months back will allow site owners the ability to see if GoogleBot can render their web pages properly. This feature visually shows a representation of what a Googlebot sees. If the indicator shows a partial rendering, web masters can review the errors to see what resources Googlebot is unable to access, what on the page is preventing the fetching and rendering.

Although the announcement did not provide a timeline for when the change would go into effect, in order to avoid negative impact on a site’s search results, it is recommended that webmasters provide access to these files immediately.