Miserable weather for either. I would honestly stay home instead of a trip tomorrow, but if forced out I would choose Ballarat. It will be a touch colder in Ballarat, but the road in Walhalla can be a bit dangerous and I wouldn't want to be driving through there in a storm.
I'm not sure when my wife and I will be able to go on a road trip again next time, so was like, fuck it let's just go lol. So will probably just go Ballarat lol.