Hay Users,

I start working on a Website and bit confuse about the robots. What is the right extension for search engines allow or not to crawl the website ?