Oh, yeah, at that point it'll be a scalability clusterfuck. No idea what the solution is. Maybe something with persistent caches run by third parties or something? That actually would be fine, since all the actions are signed with the private key of the actor, I think.
ActivityPub is not to me a real great designed protocol but it's whatever. Usually the key part for social networks is the "social" part of it; the protocol or the web site can be pure shite and if people like interacting with the other people there then it's fine. But yes, you are correct that beyond a certain point of scalability there are some dragons lurking that don't have obvious weak spots.
With the current way that ActivityPub works, this isn’t really possible. Every vote needs to be signed by some real user; if that changed such that anonymous votes were accepted then there’s nothing to stop any random person from adding 5 or 5,000 anonymous votes.