26% of the top 100 websites are now blocking GPTBot
The number of popular websites blocking ChatGPT has increased 250% in the past month. Pinterest, Indeed and more big brands now block GPTBot.
At least 26 of the top 100 most popular websites – and 242 of the top 1,000 – are now blocking GPTBot, the web crawler OpenAI introduced Aug. 7, according to an updated analysis.
- That’s a 250% increase since last month, when just 69 of the top 1,000 websites had blocked GPTBot, according to an updated analysis from AI content and plagiarism service Originality.ai.
Why we care. To block or not to block ChatGPT? That has been a big question for many SEOs because ChatGPT does not cite or link to its sources. We have let search engines crawl our content because there is a clear potential benefit – we get traffic through direct links/citations. Clearly, even more of the most popular websites have decided to block GPTBot, presumably because they don’t want OpenAI scraping their data to help train its models – at least not without some form of compensation.
12 popular websites now blocking GPTBot. Among the new additions from the top 100 most popular sites in the past month, the majority of which publish news and information:
- pinterest.com
- indeed.com
- theguardian.com
- sciencedirect.com
- usatoday.com
- stackexchange.com
- alamy.com
- webmd.com
- dictionary.com
- washingtonpost.com
- npr.org
- cbsnews.com
One big reversal. Interestingly, Foursquare, which was blocking GPTBot last month, no longer is.
What about CCbot? Common Crawl’s web crawler is still blocked less – by just 130 websites. As a reminder, Common Crawl provides part of the training data used by OpenAI, Google and others.
- 109 of the top 1,000 websites block both GPTBot and CCbot.
Limitations. 67 robots.txt files out of the 1,000 websites were not identified/inspected as part of this analysis. (That’s why I wrote “at least” in the opening sentence.)
Originality.ai’s updated analysis. Websites That Have Blocked OpenAI’s GPTBot – 1000 Website Study
Dig deeper. Should you block ChatGPT’s web browser plugin from accessing your website?
Related stories
New on Search Engine Land
About the author
Goodwin has been editing and writing about the latest developments and trends in search and digital marketing since 2007. He previously was Executive Editor of Search Engine Journal (from 2017 to 2022), managing editor of Momentology (from 2014-2016) and editor of Search Engine Watch (from 2007 to 2014). He has spoken at many major search conferences and virtual events, and has been sourced for his expertise by a wide range of publications and podcasts.