Odd smell

The 2004 Grand Prix GTP has been in the family about 3 years now, and recently it has started belching out an awful, dieselly odor from the HVAC every so often. Its been too intermittent for me to put my finger on the problem, so I'm curious if any other GM owners have come across such a problem.

The problem is that the car will just be driving along normally when it happens... not hitting any bumps, turning or braking. We've observed it with the AC on and off, with the Recirc on and off. The smell lasts about 5 seconds and smells remarkably like one drove into a big cloud of diesel smoke. Of course, there is no smoke... or any diesels for miles, usually. I would imagine a lot of people would not even notice this was going on.

I was suspecting the cabin filter, but with the Recirc on, no air should be being drawn through that. I know the recirc is working, as the AC is noticeably cooler with it on. The cabin filter, IIRC, is not that old... but will be the first thing I check, eventually. Its such a PITA to check it once you install the improved water deflector.

Any thoughts?