Let's see, so you're saying that:
The intercooler is designed to soak up the heat from inside the engine bay and then does what with it? Tell us more...
Anyway, here's some more "misinformation" being spread.
An intercooler is a heat exchanger. Our intercoolers are air-to-air heat exchangers.
They are traversed by the air coming out hot from the compressor outlet. A heat exchange takes place, the air cooling down and the intercooler heating up.
They are traversed by the air stream from the environment, via hood scoop and car forward motion and a second heat exchange takes place, the stream heating up and the intercooler cooling down.
If after running your car at high loads, you immediately stop, open the hood and feel the intercooler hot, this means that most likely, it's doing it's job.
Heat transfer can take place in 3 ways:
The first and second heat transfer take place through convection although to an extent radiation could be present.
An intercooler that is cold to the touch after the first exchange, does not do... any exchange...
It is possible for an intercooler to cool down, in the interval between you quickly coming to a stop, opening the hood and checking the touch.
What would cool it off though?
There is no air passing through, to form the second convection/transfer...
So, to the extent to which there is radiation, yes, it will radiate some heat off, any maybe cool off a bit.
Although I don't have calculations and formulae to back this up, I don't see an Intercooler radiating its way down to environment temperature in 2-3 minutes.
I'm not saying it's impossible, though.