I have the exact same problem if I leave my camera card, an SD Flash Card in the card reader while booting up. The computer will hang forever.
That seems to be a very common thing. I know of NO way around it, except to use your USB drive and then unplug it.
Does it have a power switch on it, that you could just switch OFF?
My external USB drive does!
I do suspect that it has something to do with the USB drives being in the boot order somewhere. I know my own boot order includes "USB Devices".
I leave it set that way because I use my 64meg Flash Drive to boot my system and run Ghost 8.3, to back up my system. I do that every few days.
Good Luck,
The Shadow
__________________
A person with experience is never at the mercy of a person with an argument.