Why to use Robot.txt?
·
It is nice once search engines often visit your web site and index your content however typically there square measure cases once compartmentalization elements of your on-line content isn't what you would like. for example, if you've got 2 versions of a page (one for viewing within the browser and one for printing), you'd rather have the printing version excluded from crawl, otherwise you risk being obligatory a replica content penalty. Also, if you happen to own sensitive knowledge on your web site that you just don't need the globe to examine, you may conjointly like that search engines don't index these pages (although during this case the sole positive approach for not compartmentalization sensitive knowledge is to stay it offline on a separate machine). in addition, if you would like to avoid wasting some information measure by excluding pictures, stylesheets and javascript from compartmentalization, you furthermore mght would like how to inform spiders to stay far from these things. Robot.txt
is a message to search engine bots to index the page or not. In case if your
don`t want search engines to index particular page then place “noindex”.
One way to inform search engines that files and folders on your computing machine to avoid is with the utilization of the Robots metatag. however since not all search engines browse metatags, the Robots matatag will merely go ignored. a far better thanks to inform search engines concerning your can is to use a robots.txt file.
One way to inform search engines that files and folders on your computing machine to avoid is with the utilization of the Robots metatag. however since not all search engines browse metatags, the Robots matatag will merely go ignored. a far better thanks to inform search engines concerning your can is to use a robots.txt file.
Subscribe to:
Post Comments
(
Atom
)
No comments :
Post a Comment