Robots.txt is not the answer: Proposing a new meta tag for LLM/AI
Search Engine Land
JULY 18, 2023
Not all LLMs use crawlers and identify themselves The burden is on the website operator to identify and block individual crawlers, which may use and/or sell their data for generative AI products. The meta tag is the way Once a publisher has identified an appropriate license, this license still needs to be communicated.
Let's personalize your content