Inaccurate headline. The bill doesn't ban web scraping, it just requires that bots accurately identify themselves through the user agent string, and maybe some additional requirements to disclose the purpose of scraping the data.
this post was submitted on 09 Jun 2026
116 points (96.8% liked)
Technology
85274 readers
4389 users here now
This is a most excellent place for technology news and articles.
Our Rules
- Follow the lemmy.world rules.
- Only tech related news or articles.
- Be excellent to each other!
- Mod approved content bots can post up to 10 articles per day.
- Threads asking for personal tech support may be deleted.
- Politics threads may be removed.
- No memes allowed as posts, OK to post as comments.
- Only approved bots from the list below, this includes using AI responses and summaries. To ask if your bot can be added please contact a mod.
- Check for duplicates before posting, duplicates may be removed
- Accounts 7 days and younger will have their posts automatically removed.
Approved Bots
founded 3 years ago
MODERATORS
And there's a fine if the company doesn't comply which is basically now gonna be considered the cost of doing business
Realistically what good would it do once you already had scraped the pattern of news sites it's already over. All this is doing in actuality is preventing new start ups from competing in the ai space. so really this is the fastest enshitification world record of a medium. Whether you like or hate ai this is actually an enshitification of it. ( I hate ai.)
I'm guessing it's to eliminate the issue of a site not getting clicks because the article you were about to read is already summarized for you. It also opens the door for revenue negotiations for allowing their content to be scraped for that purpose, as the scraper bots would now be identified.
The only bots I hope gets exempt are Internet Archive bots. Only bots I support.
The article is about "stealth bots" that don't identify themselves as such. The Internet Archive bots have always been clearly identifiable.
How will they enforce it? Pretty please?
The New York measure defines a stealth crawler as any software that retrieves, scrapes or otherwise accesses a website, including AI agents. Under the bill, the attorney general’s office would be able to sue companies that fail to disclose such activity. Violations could net civil penalties of up to $15,000 per day.
🤔
"This website or search engine is not designed for the state of new york" I assume is going to be a disclaimer we will be seeing soon.
Soooo, that means archive.is too. This fucking sucks.
I hope those tools are exempt because, just like a browser, they respond only to specific commands issued by a human user. They don't "crawl" pages in the way we describe bots that jump from page to page.
Isn’t there something like 340 news sites that actively block them already?
Thanks, that becomes the cost of doing business. Harumph.
Honestly, I would love if forced ident was required. but archival services need a hard exemption from being blocked as well.
Why only news sites?
Any bot that ignores robots.txt should be banned.