Perplexity accused of scraping websites that explicitly blocked AI scraping

According to the Cloudflare, the Internet infrastructure, AI Startup confusion is crawling and scraping materials from websites that clearly indicate that they do not want to be scraped.

Monday, Cloudflair Research It observes that AI startup ignores blocks and hides its crawling and scraping activities. Researchers in Cloudflare wrote that the network infrastructure giant web pages were confused while trying to scrape the identity of “Cloudflare researchers.”

AI products provided by confusion depend on the Internet to fold a lot of data and AI startups scrap the text, images and videos from the Internet many times without their permission to work on their products. In recent times, websites have tried to fight using Web Standard Robots.Text files, which are the search engines and AI companies say which pages can be indicated and which one should not try, effort That has yet seen mixed results so farThe

These blocks seem to voluntarily interrupts the “user agents”, which means a signal that identifies a website visitors by their device and version type; According to the CloudFlair, changing their autonomous system networks or ASN, as well as a number that marks large networks on the Internet.

Read the CloudFlair’s post, “This activity was observed every day throughout several thousand domains and several million requests.

Wonderful spokesperson Jesse Duer dismissed Claudflair’s blog post as “sales pitch”, added an email to Techcunch that screenshots of the post “Show” no material was accessed. ” In a follow-up email, the Duer Claudflair blog claimed the boat “not even us.”

Claudflare says that after consumers complain that it first noticed the behavior that they were cracking and scraping their sites to block the familiar bottles of confusion, especially after adding rules to their robot file. Cloudflair says it has performed the tests to test and confirmed that confusion is blocked these blocks.

According to Cloudflair, “We noticed that the confusion not only uses their declared user-agent, but also used a generic browser to create a Google Chrome in Macos when their announced crawler was blocked.”

The company also says that it listed the D-Paix’s bot from its verified list and added new strategies to block them.

Cloudflair recently took a public position against the AI crewers. Cloudflair last month A Marketplace has announced to launch The website owners and publishers allow AI scraper to charge their sites to charge them. Mathew Prince, CEO of Claudflair Alarm playing At this time, AI says the internet business model, especially the publishers, are broken. Last year, Cloudflair too Has launched a free tool To prevent botches from scraping websites to train AI.

This is not the first time the confusion has been accused of scraping without approval.

Last year, news outlet, As wiredComplaint Confusion was stealing their contentsThe Weekly, CEO of confusion is Arrind Srinivas Was unable to answer immediately The disruption 2024 conference was asked to provide the company’s definition of the company during an interview with Devin Coldway at TechCrunch.