Here, it is useful to describe what meta tags are. Meta tags are an unseen portion of a web page that basically allow the creator of a web page to clearly specify what key words he/she wants the page to be indexed by. Evidently, the concept of meta tags can be extremely useful. There are cases when the words on a page might have double meaning. The meta tags can give the search engine some guidance in choosing the correct meaning of these words.
There is also a danger that the creator of web pages might misuse the usage of meta tags simply to get many hits on his page. The creator might add meta tags that fit very popular and searched topics but have nothing to do with the actual contents of the page. So spiders then got smarter and began to verify meta tags with the page contents, rejecting meta tags which they saw uncorrelated to page content.
There is also a completely different scenario that search engine spiders need to deal with. Sometimes web page creators and owners might not want the contents of a web page to show up on a search engine or might not even want a search engine spider visiting his page at all. For situations like this, to avoid situations like this, the robot exclusion protocol was developed. This protocol, implemented in the meta-tag section at the beginning of a Web page, tells a spider to leave the page alone -- to neither index the words on the page nor try to follow its links.
