Sounds like a starter solenoid going bad. If it is, it's a pretty easy fix. Starter comes out fairly easy, take solenoid apart and replace the 2 contacts inside, and clean the plunger up so it gets good contact.
Some will say to just replace the starter, as it will come with the solenoid, but in my experience with reman starters for Subarus you're more likely to get a junk one than a good one. Replacing the contacts is MUCH cheaper, easy, and most likely all that's wrong with yours.