It could be implemented on both the server and the client, with the client trusting the server most of the time and spot checking occasionally to keep the server honest.
The origins of upvotes and downvotes are already revealed on objects on Lemmy and most other fediverse platforms. However, this is not an absolute requirement; there are cryptographic solutions that allow verifying vote aggregation without identifying vote origins, but they are mathematically expensive.
Who runs their email servers? You can outsource fediverse server hosting too…