I just got off the phone with Carter Subaru. They have apparently figured out the source of my under dash rattle. It's coming from the front right Rally Armor mud flap. They took the fronts off and immediately the noise stopped.
I guess it is possible that one or more of the fasteners worked it's way loose and allowed for some sort of vibration to get transmitted to the interior of the car. However the noise didn't start until about 500 miles after I installed them.
I guess I'll see what happens when I pick the car up later today, they are supposed to be putting them back on for me.