How do I use robots txt?

hi

How do I use robots txt?

User-agent
Disallow
Allow
Blocking sensitive information
Blocking low quality pages
Blocking duplicate content

Robots.txt is a text file webmasters create to instruct web robots (typically search engine robots) how to crawl pages on their website. The robots.txt file is part of the the robots exclusion protocol (REP), a group of web standards that regulate how robots crawl the web, access and index content, and serve that content up to users. The REP also includes directives like meta robots, as well as page-, subdirectory-, or site-wide instructions for how search engines should treat links (such as “follow” or “nofollow”).
Basic format:
User-agent: [user-agent name]
Disallow: [URL string not to be crawled]

A robots.txt shows which pages or files the Googlebot can or can't request from a website. Webmasters usually use this method to avoid overloading the website with requests.

Robots are applications to crawl through websites.
Syntax:
1. define the user agent: this section applies to all robots.
2. disallow: state the URL here, to block access to pages or a section of your website
3. allow: If you want to unblock a URL path within a blocked parent directly, enter the URL subdirectory path

How to use Robots.txt file?

Define the User-agent. State the name of the robot you are referring to (i.e.
Disallow. If you want to block access to pages or a section of your website, state the URL path here.
Allow.
Blocking sensitive information.
Blocking low quality pages.
Blocking duplicate content.

Check out this link.

Robots.txt file is at the root of the website that involves sectors of your website you don’t want to be attained by search engine crawlers. Webmasters use a robot.txt file to instruct the search engine robots on how to crawl & index the web pages.

Web site owners use the /robots.txt file to give instructions about their site to web robots; this is called The Robots Exclusion.