Fediverse

With all due respect, I think you're not getting what the role of fact-checking is in journalism and how sites like MBFC fit that role.

Then how do they determine if the "fact-checker" is trustworthy?

There's a large degree of coincidence in their independent evaluations. As I said, some things cannot change no matter the bias.

At the risk of citing Wikipedia, I'll use it to illustrate my point:

"Scientific studies[19] using its ratings note that ratings from Media Bias/Fact Check show high agreement with an independent fact checking dataset from 2017,[15] with NewsGuard[20] and with BuzzFeed journalists.[21] When MBFC factualness ratings of ‘mostly factual’ or higher were compared to an independent fact checking dataset's ‘verified’ and ‘suspicious’ news sources, the two datasets showed “almost perfect” inter-rater reliability.[15][16][22] A 2022 study that evaluated sharing of URLs on Twitter and Facebook in March and April 2020 and 2019, to compare the prevalence of misinformation, reports that scores from Media Bias/Fact Check correlate strongly with those from NewsGuard (r = 0.81).[20]"
https://en.wikipedia.org/wiki/Media_Bias/Fact_Chec#Reception

As you can see, an 80% of overlap in its independent evaluations are not due to chance. And 20% discrepancy says that they're not copying each other, either.

If I start a "fact-checking" site today, would you just instantly trust me to report only facts and be unbiased?

Why would I do that if I'm telling you otherwise? I'm not sure how you got that. For your fact-checker, you'd need to build a good reputation first by providing highly accurate data that can be compared and we'll go from there.

You seem to be suffering from the idea that "fact checkers" are somehow inherently more trustworthy than the publications

I'm not "suffering" from any ideas, but I'm not sure you're getting what I mean. As I said, fact-checkers are subject to a large degree of scrutiny, probably more than the publications they check.