A robots.txt file tells search engine crawlers which URLs the crawler can access on your site.