Robots.txt file is giving instruction to search engine what are pages need to crawl or Not Crawl.