Take Signal to Matrix for example. They use different encryption protocols, which means a message sent from one end has to be decrypted, and then re-encrypted with the protocol of the recipient before they can actually receive it.
So basically, your encryption is not very e2e anymore, and the fact that someone can set this up, effectively giving encryption keys to a third party without their contacts being able to do anything about it is pretty fucked.
Oh, and different TOS between different services also come into play.
So if you do this, at least tell your contacts about it, so they can make an informed decision about whether or not that's okay for them.
I bet none of my contacts made an informed decision about which chat app they are using. I don't think that this really bothers one of them. Most of them do not know, what the difference between Insta-pms and Whatsapp even is, as far as security and privacy are concerned. And from my point of view I don't know it detailed enough too. Making an informed decision about a closed source software and as a non technical person is not as easy as you may think. At least from my point of view.