Anyone else feel like shit if they don't retain what they've learned with their special interest?

I get the same feeling like I failed a math exam, like I should've known better but didn't. I know I shouldn't feel like this, I know nobody cares, but I can't help it. It makes it hard to just relax and enjoy things.