I don't see an issue as signal is designed not to trust the server. Signal also uses sealed sender and Perfect Forward Secrecy, which is something almost all e2ee messengers lack. What it means in practice is signal leaks very little if any metadata, if you leak metadata you give away details about who your talking to and for how long, etc. Examples might include talking with a suicide hotline, or a doctor, maybe a customer service agent at a company and for how long. Those details will give a lot away about you, even if the messages or calls themselves are encrypted. Matrix is not recommended for communication because it fails to properly hide metadata and actively trusts the servers. When you make a call on signal, as long as both users have "Always Relay Calls" set to disabled, your calls will be peer to peer instead of trusting a central server to facilitate the connection and trusting a middle man. What this means is since the connection is peer to peer you can leak your IP address to the user you're talking to, however a VPN fixes this issue.