this post was submitted on 27 Sep 2024
787 points (98.4% liked)
Technology
59589 readers
2891 users here now
This is a most excellent place for technology news and articles.
Our Rules
- Follow the lemmy.world rules.
- Only tech related content.
- Be excellent to each another!
- Mod approved content bots can post up to 10 articles per day.
- Threads asking for personal tech support may be deleted.
- Politics threads may be removed.
- No memes allowed as posts, OK to post as comments.
- Only approved bots from the list below, to ask if your bot can be added please contact us.
- Check for duplicates before posting, duplicates may be removed
Approved Bots
founded 1 year ago
MODERATORS
you are viewing a single comment's thread
view the rest of the comments
view the rest of the comments
Pro-tip for webscrapers: using AI to solve captchas is a massive waste of effort and resources. Aim to not be presented with a captcha in the first place.
I think thats much more difficult than it seems, because usually only residential IPs are the ones that don't get those. And if you start to use a residential proxy too much then that IP can also get flagged.
That's when you rotate the proxy. By default most residential proxies will give you a new proxy for each request unless you specify.