You will find robots.txt at the root of your Site. The purpose of this file is to suggest to search engines which areas of your site should be crawled and indexed and which areas should be ignored. As delivered, your robots.txt file tells search engines that all areas of your site should be crawled and indexed. If this is appropriate for your site, then you only need to change one line.
Identifying your sitemap location
- Find that line that reads
Sitemap: SITE_PRODUCTION_DOMAIN/sitemap.php - Replace
SITE_PRODUCTION_DOMAINwith your site URL.
Further configurations
If you would like to disallow search engines from crawling and indexing your site, please refer to the robots.txt tutorial on robotstxt.org.