I always pull my card straight out, BUT do NOT pull the card while it is in the process of reading\writing or it may well damage the card, but there is no risk just pulling the card when it has finished what it's doing IMHO.
Unlike other drives where they may be spinning while plugged in, these only work when instructed to read or write to the card, thats why it's safe IMO, using the safely remove feature will effectively uninstall the hub (card reader) itself, not the card, so feel free to pull the cards without worry so long as it isn't working at the time.