Zigbee is for this usecase instant and definitely faster than any ESP wifi based solution. Your slowness here isn't for reaching HA, it's HA itself and primarily pushover and Alexa. What is it you are trying to do? For anything that should feel instant, getting rid of pushover and Alexa will be crucial.