lautan

I’m reaching out to see their thoughts. But there are limitations to what they can index.

Good point. They should know they are making public comments. If you want it private then send a private message.

I heard it will send out requests as fast as possible essentially creating a DOS attack but I could be wrong. Also the UI can use some improvement.

It will be very little if not downloading full html pages.

It's limited to only Peertube and it's not the most intuitive. I want to work with them on expanding this.

Yep, the idea is to simulate the type of results you get from Google. People trust Lemmy answers more than spam sites now a days.

I mean they are posting on the public internet, they should know that it can be read by anyone. I like the idea of users opting out.

Yep

The fediverse is a few thousand servers, from Mastodon, Lemmy, etc. Can't say the amount of posts but there are a lot.

So on the more technical side, I plan on using a light weight fast search engine called Sonic (It's written in rust). I have already used it in other projects and it can handle billions of messages / posts. But it has a cost it doesn't have faceted search, like for example if you want to exclude certain texts from the results. I think this is a fair trade off. The other solution would be to use something more mature like ElasticSearch but it'll be expensive (I'm assuming not much money will be made from this and I'm talking about donations)

For scanning sites there are premade lists to start with and it'll be possible to scan new sites from other instances if found. So a bit of both.

I heard it’s not optimized well but I’ll take a look at it.

Well that’s why I’m asking for input. And I won’t launch this on every instance without letting them know. Baby steps.

Yeah that would be the case.