Probably falls into the same or similar category as calling them dumb, ones open a time an acceptable word to call them, but not really today. You know unless you are looking to insult them for being mute.
Right. The term that would probably fit the context would be "infantile", which again has negative connotations.
English has a long history of descriptors of intellectual deficiencies becoming contemporary insults then terms to be avoided because of that insensitive use, then the use continuing until everyone's kinda desensitized to it but now it can't be used in the original context.
Also, many languages have a link between deafness and lacking intelligence, e.g. dumb meaning "not able to speak" and "not intelligent".
In general, being sensitive to people with disabilities (both physical and mental) is a rather young concept, hence anything that would make someone not be able to be part of society is often also an insult.
That's also why e.g. terms linked deafness/muteness are often an insult to someone's intelligence, while e.g. terms linked to blindness are not. Blind people might be unable to perform some things seeing people are able to, but blindness doesn't necessarily limit someone's ability to be part of a society unaccomodating to people with disabilities.
Might change your thoughts on blindness as an insult by attending a sporting event where a ref or ump makes a questionable call...but broadly speaking, I think your comment is definitely a worthwhile contribution.