Selfhosted

59939 readers

304 users here now

A place to share alternatives to popular online services that can be self-hosted without giving up privacy or locking you into a service you don't control.

Rules:

Be civil: we're here to support and learn from one another. Insults won't be tolerated. Flame wars are frowned upon.
No spam.
Posts here are to be centered around self-hosting. Please ensure it is clear in your post how it relates to self-hosting.
Don't duplicate the full text of your blog or git here. Just post the link for folks to click.
Submission headline should match the article title.
No trolling.

Resources:

selfh.st Newsletter and index of selfhosted software and apps
awesome-selfhosted software
awesome-sysadmin resources
Self-Hosted Podcast from Jupiter Broadcasting

Any issues on the community? Report it using the report flag.

Questions? DM the mods!

founded 3 years ago

MODERATORS

curbstickle@anarchist.nexus

curbstickle_lw@lemmy.world

584

Based on this graph, and this graph alone, guess at what time I completely blocked OpenAI crawlers (jlai.lu)

submitted 4 months ago by hylobates@jlai.lu to c/selfhosted@lemmy.world

80 comments fedilink hide all child comments

I really hope they die soon, this is unbearable…

you are viewing a single comment's thread
view the rest of the comments

[–] punrca@piefed.world 27 points 4 months ago (2 children)

It's best to use either Cloudflare (best IMO) or Anubis.

If you don't want any AI bots, then you can setup Anubis (open source; requires JavaScript to be enabled by the end user): https://github.com/TecharoHQ/anubis
Cloudflare automatically setups robots.txt file to block "AI crawlers" (but you can setup to allow "AI search" for better SEO). Eg: https://blog.cloudflare.com/control-content-use-for-ai-training/#putting-up-a-guardrail-with-cloudflares-managed-robots-txt

Cloudflare also has an option of "AI labyrinth" to serve maze of fake data to AI bots who don't respect robots.txt file.

[–] AHemlocksLie@lemmy.zip 17 points 4 months ago (2 children)

Pretty sure I've repeatedly heard about the crawlers completely ignoring robots.txt, so does Cloudflare really do that much?

[–] Sv443@sh.itjust.works 9 points 4 months ago (1 children)

Like a lock on a door, it stops the vast majority but can't do shit about the actual professional bad guys

[–] FreedomAdvocate@lemmy.net.au 2 points 4 months ago

Cloudflare definitely can and does stop the vast majority of actual professional bad guys.

[–] tomjuggler@lemmy.world 6 points 4 months ago

Yes, CloudFlare blocks agents completely if they ignore it's restrictions. The key is scale - CloudFlare has a birds eye view of traffic patterns across millions of sites and can do statistical analysis to determine who is a bot.

I hate the necessity but it works

[–] shane@feddit.nl 17 points 4 months ago (2 children)

If you're relying on Cloudflare are you even self-hosting?

[–] CyberSeeker@discuss.tchncs.de 15 points 4 months ago* (last edited 4 months ago) (1 children)

If you build a house, but hire a guard for the front gate, do you even own the house?!

[–] Impassionata@lemmy.world 10 points 4 months ago

If you use DNS at all, do you even own your street address!?!?

[–] sudoer777@lemmy.ml 9 points 4 months ago

Yes if it's tunneled to your self-hosting setup. With CGNAT you have to use similar services if you want to self-host.