About robots.txt

Owned by Todd McGill
Last updated: Mar 11, 2016
1 min read

You will find robots.txt at the root of your Site. The purpose of this file is to suggest to search engines which areas of your site should be crawled and indexed and which areas should be ignored. As delivered, your robots.txt file tells search engines that all areas of your site should be crawled and indexed. If this is appropriate for your site, then you only need to change one line to identify your sitemap location.

Identifying your sitemap location

  1. Find that line that reads
    Sitemap: SITE_PRODUCTION_DOMAIN/sitemap.php
  2. Replace SITE_PRODUCTION_DOMAIN with your site URL.

Further configurations

If you would like to disallow search engines from crawling and indexing your site, please refer to the robots.txt tutorial on robotstxt.org.