This is a most excellent place for technology news and articles.
cross-posted from: https://programming.dev/post/35852706
Just provide a full dump.zip plus incremental daily dumps and they won't have to scrape ?
Isn't that an obvious solution ? I mean, it's public data, it's out there, do you want it public or not ?
Do you want it only on openai and google but nowhere else ? If so then good luck with the piranhas
I think the issue is that the scrapers are fully automatically collecting text, jumping from link to link like a search engine indexer.
I knew that was the worse option. Use the one that traps them in an infinite maze.
