584
Based on this graph, and this graph alone, guess at what time I completely blocked OpenAI crawlers
(jlai.lu)
I really hope they die soon, this is unbearable…
this post was submitted on 13 Feb 2026
584 points (98.8% liked)
Selfhosted
59939 readers
308 users here now
A place to share alternatives to popular online services that can be self-hosted without giving up privacy or locking you into a service you don't control.
Rules:
-
Be civil: we're here to support and learn from one another. Insults won't be tolerated. Flame wars are frowned upon.
-
No spam.
-
Posts here are to be centered around self-hosting. Please ensure it is clear in your post how it relates to self-hosting.
-
Don't duplicate the full text of your blog or git here. Just post the link for folks to click.
-
Submission headline should match the article title.
-
No trolling.
Resources:
- selfh.st Newsletter and index of selfhosted software and apps
- awesome-selfhosted software
- awesome-sysadmin resources
- Self-Hosted Podcast from Jupiter Broadcasting
Any issues on the community? Report it using the report flag.
Questions? DM the mods!
founded 3 years ago
MODERATORS
you are viewing a single comment's thread
view the rest of the comments
view the rest of the comments
My best guess is that they don't just index things, but rather download straight from the internet when they need fresh training data. They can't really cache the whole internet after all...
Bingo, modern datasets are a list of URL's with metadata rather than the files themselves. Every new team/individual wanting to work with the dataset becomes another DDoS participant.
The sad thing is that they could cache the whole internet if there was a checksum protocol.
Now that I'm thinking about it, I actually hate the idea that there are several companies out there with graph databases of the entire internet.