What is Robot.txt?

[JanyPunk]

Hello guys,
I have a new website. Actually I know a little bit about SEO but I am confused which pages URLs I should put in robot.txt file because plenty of pages have no content? Please guide me if anybody has answer.....

[StuartSpindlow9]

if your site have no content simply disallow it in search engine
user-agents:*
Disallow: /

[Spindlow12]

robot.txt is a simple text file in which we instruct search engine to crawl or nor crawl a particular web page or any image . it block search engine to crawl a particular web page

[anky]

The robots.txt file is a simple text file placed on your web server which tells webcrawlers like Googlebot if they should access a file or not.

Allow full access of your website using robots.txt file:
User-agent: *
Disallow:

[infoseair]

Nice Information...

[nancy07]

The robots exclusion protocol (REP), or robots.txt is a text file webmasters create to instruct robots (typically search engine robots) how to crawl and index pages on their website.

[marisalewis24]

The Robots Exclusion Protocol (REP) is a group of web standards that regulate web robot behavior and search engine indexing.

[tieb]

Web site owners use the /robots.txt file to give instructions about their site to web robots; this is called The Robots Exclusion Protocol.

It works likes this: a robot wants to vists a Web site URL, say http://www.example.com/welcome.html. Before it does so, it firsts checks for http://www.example.com/robots.txt, and finds:

The "User-agent: *" means this section applies to all robots. The "Disallow: /" tells the robot that it should not visit any pages on the site.

[marksteve741]

robots.txt is file which is use for control the search engine bot from indexing and cache.

[mani ge3e]

Web Robots (also known as Web Wanderers, Crawlers, or Spiders), are programs that traverse the Web automatically. Search engines such as Google use them to index the web content, spammers use them to scan for email addresses, and they have many other uses.

On this site you can learn more about web robots.

About /robots.txt explains what /robots.txt is, and how to use it.
The FAQ answers many frequently asked questions, such as How do I stop robots visiting my site? and How can I get the best listing in search engines?"
The Other Sites page links to external resources for robot writers and webmasters.
The Robots Database has a list of robots.
The /robots.txt checker can check your site's /robots.txt file and meta tags.
The IP Lookup can help find out more about what robots are visiting you.

[sadianisar]

robots.txt is a notepad file in which we write commands to instruct crawlers of search engine to no visit and index the webpage which we do not want it to.. It is important in SEO.

[janpaul7890]

The robots.txt is a simple text file placed on your web server which tells webcrawlers like Googlebot if they should access a file or not.

[onlineuae]

Web Robots (otherwise called Web Wanderers, Crawlers, or Spiders), are projects that cross the Web naturally. Internet searchers, for example, Google use them to record the web content, spammers use them to filter for email locations, and they have numerous different employments.
on this website you can take in more about web robots.
About/robots.txt clarifies what/robots.txt is, and how to utilize it.
The FAQ answers numerous as often as possible made inquiries, for example, How would I stop robots going to my site? furthermore, How would I be able to get the best posting in web crawlers?"
The Other Sites page connections to outer assets for robot authors and website admins.
The Robots Database has a rundown of robots.
The/robots.txt checker can check your site's/robots.txt record and meta labels.
The IP Lookup can discover more about what robots are going by you.

[ShreyaKoushik]

Use of Robots.txt - The most common usage of Robots.txt is to ban crawlers from visiting private folders or content that gives them no additional information.

Robots.txt Allowing Access to Specific Crawlers.
Allow everything apart from certain patterns of URLs.

[ram.sh]

Robots.txt file helps us to allow and disallow our website page to google.

[fayeseom]

The robots exclusion protocol (REP), or robots.txt is a text file webmasters create to instruct robots (typically search engine robots) how to crawl and index pages on their website.

[SD Groupkol]

It helps to instructs search engine to crawl or no crawl website page.

[Richard1234]

robot.txt is a basic content record in which we train web crawler to creep or nor slither a specific page or any picture . it square web search tool to creep a specific website page

[Deepak5]

Robots.txt is a text (not html) file you put on your site to tell search robots which pages you would like them not to visit. Robots.txt is by no means mandatory for search engines but generally search engines obey what they are asked not to do. It is important to clarify that robots.txt is not a way from preventing search engines from crawling your site (i.e. it is not a firewall, or a kind of password protection) and the fact that you put a robots.txt file is something like putting a note “Please, do not enter” on an unlocked door – e.g. you cannot prevent thieves from coming in but the good guys will not open to door and enter. That is why we say that if you have really sen sitive data, it is too naïve to rely on robots.txt to protect it from being indexed and displayed in search results.

[24x7servermanag]

Robot.txt is used to crawl the pages website. It tells that which part of the area should not be accessed. We can define the pages which should not be accessed by putting the disallow tag in robot.txt. Those disallow pages are restricted to visit. It also help to index the web content.

You can ask your web hosting provider to upload it under your control panel (root directory of the website) and webmaster will pick it automatically.

If you have access then you can upload it from your end.

[sindhusmile]

The robots exclusion protocol (REP), or robots.txt is a text file webmasters create to instruct robots (typically search engine robots) how to crawl and index pages on their website.


I agree with your answer.