Yeah it's a mild curiosity for me. It's fairly rare that my posts get more than one downvote, but it is very common for my posts to get exactly one. I'm sure it is just my brain doing its thing and constantly looking for patterns that has me wondering if it's largely the same person in the first place. You're probably right that it is not coordinated or malicious, and I don't really suspect that it is.

I would definitely not be messaging anybody to ask why I'm being downvoted; personally it doesn't bother me because of course not everyone is going to agree with what I have to say. But I do acknowledge that there is greater potential for that to happen at scale if votes are totally public for everyone.

Edit: And I am not claiming to be perfect myself either. There were certainly a couple users who I have consistently downvoted for what I have seen as bad faith participation. One of those users has mellowed out a lot over time and only rarely gets a downvote from me these days; the other slowly ramped it up until I decided they were worth blocking. I try not to reflexively downvote comments I simply disagree with, but I'm definitely guilty of leaving a downvote in lieu of having the energy to respond with push back to something disingenuous.