I've owned both a WRX wagon and the Forester XT, and I kept the XT.
Everything everyone has said previously is correct. "backpack on wheels"--gotta be a Forester. You'll have more room, better ground clearance, and a really nice ride.
Nothing wrong with the WRX. It is just smaller. It does handle better, and is more of a sports car.
My only real complaint with the XT is gas mileage.
Substantive differences of the '03 WRX to 04 was electronic brake distribution, new headlight design, more efficient engine cooling.