I've had some luck establishing the bottleneck using strace on both the sender side and receiver side. This will show if the sending rsync is waiting on local reads or remote writes and if the receiving rsync is waiting on network reads or local writes.
This helps find the specific resources to check.
There is a lesson about unnoticed data damage in someone's future.
What happens when a software bug, ransomware, or hardware problem destroys or corrupts that data over days or months?