robots.txt is a notepad file in which we write commands to instruct the web crawler to not visit and index which we do not want it to. It is important for SEO and used when the site is newly built and developed.