X updates its terms to ban crawling and scraping
X updates its terms to ban crawling and scraping
Elon Musk-owned X, has updated its terms of service to prohibit scraping and crawling — likely to fend off any AI models training on its data.
The new terms, which are effective from September 29, ban any kind of scraping or crawling without “prior written consent.”
NOTE: crawling or scraping the Services in any form, for any purpose without our prior written consent is expressly prohibited.
The previous version of the terms allowed crawling in accordance with robots.txt.
“NOTE: crawling the Services is permissible if done in accordance with the provisions of the robots.txt file, however, scraping the Services without our prior consent is expressly prohibited,” it read.
In the last few months, Twitter has also altered its robots.txt file — a file that gives instructions to robot crawlers about what parts of the site they are permitted to visit — to remove instructions for all crawler bots apart from Google.
In 2015, Twitter confirmed that it had a firehose deal in place with Google to surface tweets in search results. It is not clear if the nature or terms of that deal have changed under the new management.
You're viewing a single thread.
I gather, policy as written, that apart from bulk data collection, this also inadvertently prohibits usage of any alternative front-ends, such as Nitter? Does it also stop any archival (akin to Wayback Machine) from happening against their service?
It sounds like it might, but whether it stops them from working or just introduces liability such that they can sue is unclear. Likely the latter, but unclear.