The various engine swaps can make the height appear different. The intake manifolds were different on some, correct? So that could adjust where the intercooler sits, in relation to the firewall. It would also affect height, to a degree...possibly making one swap appear to be a different height than another.
To determine it was the crossmember, we'd need to know the car, the crossmember, the specific engine swap(total, block, heads, intake manifold, etc...) and the intercooler used(and if it was modified).