Consider https://arstechnica.com/robots.txt or https://www.nytimes.com/robots.txt and how they block all the stupid AI models from being able to scrape for free.
You must log in or register to comment.
The robots.txt construct is completely voluntary and some bots use it to specially target content.
In my opinion, anyone relying on this to protect their content has no business publishing anything online.
We will sue their unauthorized use in the marketplace of ideas.
