.Google.com's John Mueller addressed a concern regarding why Google.com indexes webpages that are actually forbidden coming from creeping by robots.txt and also why the it is actually risk-free to overlook the similar Explore Console records concerning those creeps.Crawler Website Traffic To Concern Guideline URLs.The individual talking to the inquiry chronicled that robots were producing hyperlinks to non-existent query criterion Links (? q= xyz) to pages with noindex meta tags that are also blocked out in robots.txt. What triggered the inquiry is actually that Google.com is creeping the web links to those pages, obtaining obstructed through robots.txt (without watching a noindex robotics meta tag) at that point receiving shown up in Google Search Console as "Indexed, though shut out by robots.txt.".The individual inquired the following concern:." But below's the significant inquiry: why would Google index pages when they can not even observe the information? What is actually the benefit during that?".Google.com's John Mueller verified that if they can't creep the page they can not view the noindex meta tag. He also produces an intriguing reference of the internet site: search operator, suggesting to neglect the outcomes given that the "typical" customers won't observe those results.He created:." Yes, you are actually correct: if our company can not crawl the webpage, our company can't observe the noindex. That claimed, if we can't creep the webpages, then there's certainly not a whole lot for our company to index. So while you might observe a few of those webpages with a targeted web site:- question, the typical individual will not see all of them, so I wouldn't bother it. Noindex is actually likewise fine (without robots.txt disallow), it merely implies the URLs will find yourself being actually crept (and wind up in the Browse Console file for crawled/not listed-- neither of these standings result in problems to the remainder of the website). The important part is that you don't produce them crawlable + indexable.".Takeaways:.1. Mueller's solution confirms the constraints in using the Internet site: hunt progressed search driver for diagnostic factors. One of those reasons is since it is actually not attached to the routine search mark, it's a distinct thing entirely.Google's John Mueller discussed the internet site search operator in 2021:." The quick answer is that a site: inquiry is not indicated to become comprehensive, neither made use of for diagnostics functions.A site inquiry is a certain kind of search that limits the outcomes to a specific web site. It's essentially simply the word website, a digestive tract, and afterwards the website's domain.This question confines the results to a specific internet site. It is actually certainly not suggested to become a detailed compilation of all the pages coming from that site.".2. Noindex tag without utilizing a robots.txt is great for these sort of scenarios where a bot is linking to non-existent web pages that are getting uncovered by Googlebot.3. Links along with the noindex tag will certainly create a "crawled/not catalogued" item in Explore Console and also those won't possess a bad effect on the rest of the website.Read the question and also address on LinkedIn:.Why would Google mark web pages when they can't also view the information?Included Photo by Shutterstock/Krakenimages. com.