The robots.txt file is a text file which is uploaded to website’s root folder and this will control the pages to be crawled for search engines and in turn will help the search engines to index the website content. Robots.txt is also helpful for webmasters as it indicates the search engine spiders to skip or avoid some pages that are not important or contains some duplicate content.
No comments:
Post a Comment