Robots.txt file is set of instructions for search engine spiders to index the website.