There are a few things that could cause this and I've felt them all with cars that have everything adjustable, which your subaru would not.
Bump steer is possible, which would be caused by something bent in the tie rod(s), but even that would have to be pretty severe and I'd think it would be seen.
Other things like caster isn't going to make this happen unless it's different from one side to the other. Camber wouldn't do it at all.
Something that could do this is if something is bent in the spring mount or maybe the shock. It would have to have been enough to unweight (or weight) one front wheel more than the other. The car can teeter totter to some extent.
If you go to a race shop that sets up and/or builds racecars, tell them what's going on. Between f/r wheel distances, corner weighing and "real" measurement including caster, camber and bump steer (that a generic shop won't do), they can at least figure out what is out of whack. From there, they can track down what's wrong with the car.