Robots.txt is a simple file on your website that tells search engine crawlers which pages they can or cannot access. It helps manage crawling, protect sensitive sections and improve overall site efficiency.