robotstxt.org

The robots exclusion standard, also known as the robots exclusion protocol or simply robots.txt, is a standard used by websites to communicate with web crawlers and other web robots. The standard specifies how to inform the web robot about which areas of the website should not be processed or scanned. Robots are often used by search engines to categorize websites. Not all robots cooperate with the standard; email harvesters, spambots, malware and robots that scan for security vulnerabilities may even start with the portions of the website where they have been told to stay out. The standard can be used in conjunction with Sitemaps, a robot inclusion standard for websites. More information...

Multilingual Wikipedia

In June 2020 the website robotstxt.org was on the 84,128th place in the ranking of the most reliable and popular sources in multilingual Wikipedia from readers' point of view (PR-score). If we consider only frequency of appearance of this source in references of Wikipedia articles (F-score), this website was on the 170,467th place in June 2020. From Wikipedians' point of view, "robotstxt.org" is the 112,911th most reliable source in different language versions of Wikipedia (AR-score).

The website is placed before dnevnik.mk and after brics.dk in multilingual PR ranking of the most reliable sources in Wikipedia.

PR-score:
84,128th place
239,923
-28,790
AR-score:
112,911th place
23,519
+81
F-score:
170,467th place
37
0

English Wikipedia (en)

PR-score:
71,970th place
140,777
-15,941
AR-score:
107,996th place
10,444
+55
F-score:
205,896th place
9
0

Spanish Wikipedia (es)

PR-score:
45,034th place
28,533
-2,700
AR-score:
50,314th place
3,200
0
F-score:
130,579th place
2
0

Chinese Wikipedia (zh)

PR-score:
23,401st place
26,669
+1,768
AR-score:
97,469th place
254
+4
F-score:
76,607th place
2
0

German Wikipedia (de)

PR-score:
69,189th place
15,540
+1,780
AR-score:
100,598th place
2,145
0
F-score:
166,153rd place
2
0

Portuguese Wikipedia (pt)

PR-score:
46,034th place
8,396
-1,266
AR-score:
37,550th place
1,863
0
F-score:
78,396th place
2
0

Indonesian Wikipedia (id)

PR-score:
16,715th place
3,400
-3,100
AR-score:
9,780th place
1,800
0
F-score:
65,941st place
1
0

Persian Wikipedia (fa)

PR-score:
35,126th place
853
+300
AR-score:
32,184th place
453
0
F-score:
22,456th place
3
0

Hindi Wikipedia (hi)

PR-score:
15,605th place
133
0
AR-score:
16,960th place
111
+23
F-score:
12,862nd place
2
0
Show all Wikipedia languages...

BestRef shows popularity and reliability scores for sources in references of Wikipedia articles in different languages. Data extraction based on complex method using Wikimedia dumps. To find the most popular and reliable sources we used information about over 200 million references of Wikipedia articles. More details...