It does literally refer to penis size. Hiding it behind the word "energy" is just obfuscation. It reminds me of racist people defending the N word by saying, "there's black people, then there's Ns. Black people are fine, Ns aren't."
You're just kicking the can down the road. It's the exact same idea, phrased differently. You just wanna say "small dick" and get away with it, that's all.
Except when a lady friend wants a big dick, then all the big dick energy in the world isn't going to do a lick of good, that's when you have to bring out the lick of good.
Big dicks hurt women, most seem to prefer average sizes. Plus, penetration is often less effective at bringing women to orgasm than oral or fingering, which nearly anyone can learn to do well.
No reason not having a big dick should stop anyone from having a good time.