Mine's just the opposite. It does sound like a diesel like somebody else said. The sound is strongest at around 1800-2000 RPM. And it goes away at about 2700 RPM.
I was thinking it was an oiling problem as well. But I don't know enough about oil circulation to draw any real conclusions.
Speaking of oil circulation, did you hear that Toyota has recalled a bunch of cars (including Lexus) because of oil gunking? It was only on two of the engines, the 2.4 and 3.0 I think. They claimed it was caused by owners not following the recommended maintenance schedule, but they've redesigned the system.