That's always my first answer indeed.
"Well, I'm going to answer your question but first, just think about the abolition of legal slavery a while ago. What would you have thought about someone who would object : "yeah, but what will we replace slavery with ?".
I support the abolishment of prisons (at least as we know them), but I don't think this is a fair or convincing comparison. Prisons are a failed attempt to solve a real problem (what to do with people who break the rules), which is what people are concerned about. In an ideal society it will be replaced by something (rehabilitation?). Slavery solved no problem, and so needed nothing to replace it.
Ah, that's a convincing point, but I've always felt my answer actually opens up the door for important discussions like :
-Historical // between slavery and prisons
-Partial rebuttal of the question itself, or rather, its framing of the issue, my underlining it's completely legitimate to advocate for the abolition of something without having a plug'n'play replacement for it
-Usually, people will follow up by highlighting cases of pedophiles and rapists, which could further be used to frame the discussion with them : who are we talking about?
-If I'm in a bad mood, I'll also ask people : "My brother in Christ, weren't there folks around who thought slavery was indeed created to solve an actual problem?"
But yeah, that's only the first part of my answer, then we'll move to the more grounded post-carceral society discussion.