The robots.Txt file is a part of the REP, a group of web necessities that control how robots crawl the web, access and index content material, and serve that content material up to users. These crawlers advised to disallow or allow the user agents.
It is not necessary in your website.
Bookmarks