The Robots Exclusion Protocol, commonly known by its filename robots.txt, is a standard used by websites to indicate to visiting web crawlers and other web robots which portions of the website they are allowed to visit. This relies on voluntary compliance. Not all robots comply with the standard; email harvesters, spambots, malware and robots that scan for security vulnerabilities may even start with the portions of the website where they have been told to stay out. The "robots.txt" file can be used in conjunction with sitemaps, another robot inclusion standard for websites. More information...
According to PR-model, robotstxt.org is ranked 137,044th in multilingual Wikipedia, in particular this website is ranked 155,100th in English Wikipedia.
The website is placed before wailer.de and after waterlooregioneats.com in the BestRef global ranking of the most important sources of Wikipedia.