Agreed, it's not a big deal. I'm just surprised that it loads so slowly. I guess the HU has a slow CPU, and therefore takes a while to build up the directory structure from the flash drive. It's my guess that the directory structure gets recreated from scratch each time the car is turned on or USB input is selected.
It would be nice if even when I'm in another mode (for example, XM mode), the HU was smart enough to access the USB drive in the background and load up the directory structure so that when I switch to XM mode it's already ready and waiting to play. But I guess it's just not powerful/clever enough to do that.