The only difference between an 04 and 06 rack (externally) is the mounting points. (Well, that is simplifying it, but you get the point)
The 04 uses 4 bolts, whereas the 06 uses 6 to mount to the crossmember. Look and see if there are places in the crossmember for those extra 2 bolts on the driver's side. They should (iirc) be a few inches towards the inside of the car from the current 2 bolts. I would think they're there... but maybe not.
Also, I would imagine you'd have to support/lift the engine in some manner when swapping crossmember since the engine mounts to the crossmember.