Robot.txt file is a .txt file that tell search engine spiders, it should crawl and which it should not crawl.