True, but what you should really be afraid of these days are spam filters and hackers. As much as I'd love to selfhost my own mail server, IMHO it's just not worth the risk of important emails getting flagged as spam (both outgoing and incoming), or losing control of accounts due to a zero day attack or other means. The latter might sound far-fetched but I just saw it happen firsthand to a friend. The modern internet is a battleground!
I was on a trip for a few weeks to visit family, and just a few days after I got there my server went down and I lost access to all my stuff (I'm about to fix it when I return tomorrow). Stuff like that is why I will never self-host something as critical as email