Safety advice
Without proper means of prevention, the weak link lies in your own network and the public internet. Tools such as wireshark can trivially capture your network traffic and re-assemble the packets to readable call metadata and audio files.
Securing your own network
The closer the attacker is to your phone, the smaller the net he needs to cast. This makes your own network the most likely starting point for any attacker. The attacker could use tools like wireshark to capture all network traffic...