There is no such thing as a “vote” in ActivityPub/ActivityStreams. This idea of “up/down votes” is just an abstraction of a message saying “Actor A liked B”, where B is an Post/Comment (and a post/comment itself just being an abstraction of ActivityStreams objects).
That is to say: there is no way to selectively hide the content a message. If you want federation to work and you want people outside your own server to see your posts, then the server needs to broadcast the messages to anyone listening.
Tools like lemvotes are just exposing this information. There is no point in trying to censor the tool, because this information is available publicly, and any motivated person will be able to track this information.
If you are concerned about what people think of your “likes” and “dislikes”, then do not use a public social media service and only communicate with provably secure communication tools.
It’s not a Federated platform like LinkedIn, but I am working on CareerCupid , a “OkCupid for jobs” website where people can answer questions about their values and goals, and then find out people that are the most compatible to work with. The website does work with ActivityPub though, you can follow @thecupid@cupid.careers and you can see the new polls and job listings published on the website.