View Full Version : Which pages should be put in robot.txt file?
JanyPunk
07-20-2016, 05:05 AM
Hello guys,
I am new on this forum website. Actually, I know a little bit about SEO but not everything. I have a new website in most of the web pages have no content so I am confused how many web pages URL should I put in robot.txt file?
spindlow2016
07-20-2016, 06:33 AM
you can disallow the search engine to not crawl a web page. by using Disallow:/page1. or yau can simply use robots no follow tag in particular web page meta tags
daviddakarai
07-20-2016, 08:18 AM
You can keep all type of pages in robots. If you want to crawl the webpage you keep it in allow and if don't want to crawl a web page by search engine, use disallow. Many are using only disallow option to avoid pages by search engine crawlers which are most secure.
sadianisar
07-20-2016, 08:25 AM
In SEO, robots.txt plays an important role. If you want crawler/search engine to not allow to index your webpage, you can add the webpage url in Disallow code. It depends on you that which webpages you do not want to show to search engine.
onlineuae
07-28-2016, 09:49 AM
In SEO, robots.txt assumes a vital part. On the off chance that you need crawler/internet searcher to not permit to file your site page, you can include the website page url in Disallow code. It relies on upon you that which site pages you would prefer not to appear to web index.
onlineuae
07-28-2016, 09:55 AM
In SEO, robots.txt assumes a critical part. In the event that you need crawler/web index to not permit to list your website page, you can include the site page url in Disallow code. It relies on upon you that which site pages you would prefer not to appear to internet searcher.
janpaul7890
07-29-2016, 02:40 AM
Robots.txt instructs crawler to not permit to file your site page, you can add your page url in Disallow code. Its depend on you that which pages you would prefer not to appear to web index.
ajay49560
07-29-2016, 07:45 AM
Read this link: http://www.robotstxt.org/robotstxt.html
Hope you helps..
Powered by vBulletin® Version 4.2.4 Copyright © 2026 vBulletin Solutions, Inc. All rights reserved.