You don't have a popup blocker installed do you? If so, it may be trying to popup the compose window, and your popup blocker is...well blocking it. IE comes with a built in popup blocker, so that may be your problem.
Try holding down the ctrl key when you click the compose button and see if there is a difference.