PDA

View Full Version : robots.txt



genie.liam
10-27-2009, 01:41 AM
What's the importance of robots.txt? How does it work?

~ServerPoint~
10-27-2009, 05:56 AM
That is information for the robots which visit your web site. Like which page to index and which not etc
I believe I have seen some articles regarding that somewhere in the net

Bestmeta Search
10-27-2009, 06:30 AM
Hi,

Robots.txt file is mainly for search engines and robots.txt file can be used to prevent access to certain parts of a web site, if not correctly implemented, and it can also prevent access to the whole site.This may be possible that sometimes the robots.txt is the only reason for not indexing of website in many search engines.
This file works mostly for all the search engines but there is one search engine searchspry where i didn't check this file.
You can check from here http://www.searchspry.com

Thanks.

Stenna11
10-28-2009, 09:44 PM
you can stop search engines to crawl certain pages of your website through Robots.txt

Fireman
10-30-2009, 02:45 AM
is robots.txt and RSS is same? whats the difference between on both?

helplinelaw
10-30-2009, 08:36 AM
robots.txt file in add sitemap link also becoz spider come on your site search a robots.txt firl and lookup a files instruction follow as a http://helplinelaw.com (http://www.helplinelaw.com)/ robots.txt .

There are two important considerations when using /robots.txt:

robots can ignore your /robots.txt. Especially malware robots that scan the web for security vulnerabilities, and email address harvesters used by spammers will pay no attention.

the /robots.txt file is a publicly available file. Anyone can see what sections of your server you don't want robots to use.

spinxwebdesign
09-24-2010, 01:27 AM
Most sites contain pages that should not be indexed by the search engines. Administrative pages, for example, ABC's "contact" page: "contact.html". There's no need to have it indexed, so we use the robots.txt file to tell the search engine spider (robot) to ignore it.

dombowkett
07-02-2018, 02:04 AM
Robots.txt file is set of instructions for search engine bots for indexing.

DianaBaity
07-05-2018, 07:58 AM
It give command to the search engine which page user want to crawl and index from the website.

ram.sh
07-05-2018, 08:02 AM
Robots.txt is imprtortant for SEO. By using this, you can allow or disallow pages your website. Allow means Google bot can crawl the webpages and disallow means pages are not crawl by spiders.

Rajdeep Bose
07-05-2018, 08:29 AM
Robots.txt is a file through which you can guide search engines to crawl or not to crawl certain sections of your website.

annagreenang
07-05-2018, 08:32 AM
Thanks for sharing helpful information

fisherjames731
07-05-2018, 11:35 PM
efore a search engine crawls your site, it will look at your robots.txt file as instructions on where they are allowed to crawl (visit) and index (save) on the search engine results.

Jessicad0505
07-19-2018, 09:11 AM
Google use rotots.txt to crawl or read you web page. We can make allow and disallow the search engine to crawl our web page through Robots.txt

Anamika
07-21-2018, 01:38 AM
I think before asking such question you should Google these question first or do some R&D for the same.

John-Smith
07-21-2018, 01:48 AM
What's the importance of robots.txt? How does it work?

Your site doesn't have the robots.txt file? or you don't have a site?