It is the Cruise control cable. It came from the factory too tight on many of the 03-04s. When looking at the throttle body from the front, it is the cable that loops to the left and attached to a small black box on the wheel well next to the turbo.
This is the issue. As the turbo heats up the area the box/cable deforms and tightens. Loosen the cable to have at least 1/2 of slack while cold and the issue will go away. Do NOT loosen the cable on the right as that is the actual throttle cable.