You could try using a thin sharp object, like a sewing needle, and heating it up with a lighter, and then stabbing it into the broken off pieces. Once it's had a few minutes to cool off, hopefully the melted plastic sticks to the needle and you can pull it out.
To improve this method, stick the needle in at an angle, and then lift straight up and out (not along the angle of insertion, or lever it out. This way you can get it to come out even if it is poorly or not adhered to the needle.
That seems risky! It's so small, unless you have an extremely steady hand (and preferably experience) it would be really easy to accidentally get glue on the sides or top
Is it a hot swappable switch?
If it is, pop it out and take it apart, you'll need to push the stems from the holes but once it's apart it's fairly trivial to do.
If it is not hot swappable you can probably pry it apart but it will be considerably harder.