i ran around with mine this way for a while last summer (in FL -- UGH) since the bearing inside my a/c idler pulley decided to implode and make the worst metal on metal scraping noise known to man. so until i could get a new pulley, i just took the belt off.
even on a 1.8L engine, where that 2hp would have been nearly a 2% gain in power all the way around
, i was hard pressed to notice a difference, except for the added sweating.