You could try using a thin sharp object, like a sewing needle, and heating it up with a lighter, and then stabbing it into the broken off pieces. Once it's had a few minutes to cool off, hopefully the melted plastic sticks to the needle and you can pull it out.
To improve this method, stick the needle in at an angle, and then lift straight up and out (not along the angle of insertion, or lever it out. This way you can get it to come out even if it is poorly or not adhered to the needle.
To my mind, that could work, the barb might not be helpful since the material might not flow quite as readily back over it as soft tissue does, but you could get an angled insertion and pretty easily get direct upwards lift with it maybe.
You might be able to fit a fish hook in down the side between the broken stem and the switch, and use the barb to pull out the stem.
That seems risky! It's so small, unless you have an extremely steady hand (and preferably experience) it would be really easy to accidentally get glue on the sides or top