Chapter 7. Robots.txt

One of the most important factors in SEO is robots.txt. If you have the wrong entries it could end up telling search engines not to index your site, which means it will not appear in search results.

The first thing that BF SEO does is check to see if you have a robots.txt file in your website root. If not, you have the option of creating one. A Joomla site normally comes with a robots.txt.dist file, so the software will attempt to create your robots.txt file based on that. If your site does not have a robots.txt.dist file, then a basic robot.txt file will still be created.

Once you have a robots.txt file, the software will check to make sure that it does not contain rules for blocking images, templates and media. Since Joomla 3.3 in April 2014, images, templates and media folders have not been blocked by default robots.txt file. If you do have any of these lines, you can delete them or comment out with a #, then press save to update your robots.txt file. Google must be able to access the templates folder so that it can see your CSS and JavaScript files, and determine if your site is mobile responsive.

Go to top