Do I need robots.txt?
Oryon Networks | Singapore Web Hosting | Best web hosting provider | Best web hosting in SG | Oryon SG
Do I need robots.txt?
Oryon Networks | Singapore Web Hosting | Best web hosting provider | Best web hosting in SG | Oryon SG
No, a robots. txt file is not required for a website. If a bot comes to your website and it doesn't have one, it will just crawl your website and index pages as it normally would.
A robots.txt file is a plain text file that specifies whether or not a crawler should or shouldn't access specific folders
|
Bookmarks