.Google's John Mueller responded to a concern regarding why Google indexes web pages that are prohibited from creeping through robots.txt as well as why the it's safe to disregard the relevant Look Console files regarding those crawls.Bot Web Traffic To Inquiry Criterion URLs.The individual asking the concern recorded that robots were generating hyperlinks to non-existent question specification Links (? q= xyz) to pages with noindex meta tags that are actually likewise blocked out in robots.txt. What cued the inquiry is that Google.com is actually creeping the links to those web pages, receiving shut out by robots.txt (without noticing a noindex robotics meta tag) after that getting shown up in Google.com Look Console as "Indexed, though blocked through robots.txt.".The individual inquired the adhering to inquiry:." However listed below is actually the huge question: why will Google.com mark pages when they can't even find the information? What's the benefit during that?".Google.com's John Mueller validated that if they can't crawl the webpage they can not observe the noindex meta tag. He additionally makes an intriguing acknowledgment of the internet site: search driver, suggesting to dismiss the results given that the "common" consumers will not observe those end results.He created:." Yes, you are actually correct: if our team can't crawl the webpage, our experts can't find the noindex. That said, if our experts can not creep the webpages, at that point there's certainly not a whole lot for our company to index. Thus while you might find some of those web pages along with a targeted internet site:- question, the typical individual will not find all of them, so I would not bother it. Noindex is also fine (without robots.txt disallow), it simply means the Links will certainly wind up being crept (as well as find yourself in the Explore Console document for crawled/not recorded-- neither of these standings induce problems to the rest of the internet site). The important part is actually that you do not make all of them crawlable + indexable.".Takeaways:.1. Mueller's answer validates the constraints being used the Web site: search advanced search driver for diagnostic reasons. One of those reasons is actually due to the fact that it's not attached to the routine hunt mark, it's a distinct factor altogether.Google.com's John Mueller commented on the website hunt operator in 2021:." The short solution is actually that a site: question is actually not suggested to become comprehensive, neither used for diagnostics reasons.A site concern is a specific type of search that limits the outcomes to a particular site. It's generally simply words internet site, a bowel, and after that the internet site's domain name.This question restricts the end results to a specific website. It is actually certainly not indicated to become an extensive compilation of all the pages coming from that web site.".2. Noindex tag without using a robots.txt is great for these sort of circumstances where a robot is actually linking to non-existent web pages that are actually getting found by Googlebot.3. Links along with the noindex tag will certainly create a "crawled/not indexed" item in Search Console and that those won't have an adverse impact on the rest of the web site.Check out the question and also answer on LinkedIn:.Why will Google mark webpages when they can't also observe the information?Featured Photo by Shutterstock/Krakenimages. com.