It's definitely not brakes, usually happens when I've been sitting waiting to pick up some kids. I've kind of narrowed it down to the clutch as the culprit as well, since you rarely fully engage the clutch in reverse. It's not a long hill, or very steep, but it's one of those places you have to back out of slowly.
The only thing with the clutch, is that it doesn't smell like any clutch I've ever smelled. In addition, I've owned a lot of mt cars and have never had the problem of smoking the clutch when backing up.
Maybe it's just a Subaru quirk. That's what I'm trying to decide before I head off to the dealership just to be told I'm an idiot and they all do that.