And perhaps at one point they ate clay, so they would have been more reddish in color, or perhaps the dirt they were consuming was more reddish in color.
Googled it. It wasn't because of worms in general. It was from Vermiculus which is the diminutive of Vermis but also was how they called a very specific worm, at some point in time the only way they knew where to get red pigments from was by crushing this worm.