I don't like confrontation either but it is important to set boundaries. I've done this. It's okay for someone not to like me but I will not tolerate disrespect. I had somebody at work call me stupid because I could not solve their problem right then and there. When I get very angry, I speak quietly and enunciate my words. I said to this person, "I'm going to leave now. You let me know when you're ready to work with me." To which she responded that she would call me and I retorted, "No, you will not call me. You will send me a Teams message or an email." And I walked away. Naturally I got accused of all kinds of things but I documented what happened and signed and dated it. HR found in my favor.
This hits close to home. When I was candid about the fact that it took some courage to speak up about something that needed pushing back on, I got called a “scaredy cat” 🙄