slot Can Be Fun For Anyone
the worth with the identify attribute (robots) specifies the rule relates to all crawlers. Both the identify and the content material characteristics are situation-insensitive. to handle a selected crawler, exchange the robots value of the title attribute With all the user agent token of your crawle