Make a Lasting Impression
ImpressCMS Blog > Dear Webmaster, Google says "We can't access all your JS and CSS files" - Really!

Dear Webmaster, Google says "We can't access all your JS and CSS files" - Really!

Google robots messageYesterday, webmasters around the globe were getting emails that looked like they were from Google, which many overlooked or treated with suspicion. I did at first - then, I signed into my Google Webmaster Tools (GWT) account and found they were, indeed, from Google.

The message read-

"Google systems have recently detected an issue with your homepage that affects how well our algorithms render and index your content. Specifically, Googlebot cannot access your JavaScript and/or CSS files because of restrictions in your robots.txt file. These files help Google understand that your website works properly so blocking access to these assets can result in suboptimal rankings."

A little bit of searching around the internet and I found other webmasters wondering what to do - especially those that use a content management system (CMS) to power their sites. The use of a robots.txt file has been part of ImpressCMS from the beginning, so now was a good time to look it over.

Google robots message

The idea behind a robots.txt file is to tell the search bots which directories on your server should not be included in their indexes - test folders, temporary folders, cache folders, code, alternate (duplicate) versions of content. For ImpressCMS, and other CMSs, folders like /class/, /cache/, /include/, /themes/, and /templates_c/ are all folders that could get indexed if not excluded.

Well, that all changes now, at least for Google. They want access to all the files that get rendered or sent to the browser, which includes javascript, stylesheets, and images. If they cannot access those files, they will warn you, if you are using GWT. If they continue to have problems, your site and pages could find themselves dropping in the search results.

What do we do? There is a simple little addition to the bottom of the robots.txt file that will address Google's complaint -

User-agent: Googlebot
Allow: /*.js*
Allow: /*.css*

You can test your site, or any page on it, using Fetch and Render as Googlebot - ... ebot-fetch

You can test your robots.txt file - ... sting-tool

Once we get past this little update mandate from Google, we can look at what the best use of robots.txt, if indeed there is one anymore.

Just a little something you can do for your sites and clients to make them a little bit better than the masses.

All posts by skenow
Subscribe to latest posts
The comments are owned by the poster. We aren't responsible for their content.
Poster Thread
Posted: 2015/7/30 7:47  Updated: 2015/7/30 7:47
Home away from home
Joined: 2009/3/3
From: Belgium
Posts: 1952
 Re: Dear Webmaster, Google says "We can't acces...
Previously, Google was only interested in the textual output of your site, but that has changed recently. Now, they also want to see how fast your site loads, and if your webpages are easy to view on mobile devices. For that, they need access to the javascript and the css.

I suspect that's the reason for this campaign now, as reports are starting to come out that the "mobilegeddon" as it was called, when google would downgrade your site in search results if your site wasn't mobile friendly, wasn't having a big effect.

I can imagine Google came to the conclusion that most sites were blocking access to the layout, so their detection was not working.
Posted: 2016/4/13 7:26  Updated: 2016/4/13 7:26
Just popping in
Joined: 2016/1/7
Posts: 1
 Re: Dear Webmaster, Google says "We can't acces...
This is unbelievable that Google can not crawl JS files.