Chart: Number of sites that explicitly allow at least one AI crawler in their robots.txt over time, and number of sites that removed restrictions on AI crawlers (IMAGE)

University of California - San Diego

Chart: Number of sites that explicitly allow at least one AI crawler in their robots.txt over time, and number of sites that removed restrictions on AI crawlers

Caption

Number of sites that explicitly allow at least one AI crawler in their robots.txt over time, and number of sites that removed restrictions on AI crawlers. The vertical lines indicate public data deals between major publishers (who control 40+ domains) and OpenAI.

Credit

University of California San Diego

Usage Restrictions

Please credit

License

Original content

Disclaimer: AAAS and EurekAlert! are not responsible for the accuracy of news releases posted to EurekAlert! by contributing institutions or for the use of any information through the EurekAlert system.