txt file, However, tells search engines like google and yahoo what articles to not index (such as, certain plan web pages you don’t want to look in lookup.) produce both of those to hurry up the crawling and indexing https://medium.com/@linkdaddy