Hey, i'm not exactly sure that your problem is in cross-browser compatibility but it may be so sorry, i think it could be in your settings (for firefox and javascript's privelages)... But knowing what kind of errors occur will help us in understanding the problem...
Okay, firstly i am going to recommend a few plugins to try and debug your problem(if you're not already using them)...
1) Firebug, this will show a big red message saying "* Errors" which you can then click on to show the errors
2) A Javascript shell (to ensure you can run the window.open(); method without using the flash)....
Okay for the shell i reccomend using The Javascript Shell Bookmarklet found here:
https://www.squarefree.com/bookmarklets/webdevel.html
right click the link (the little bookmark button) and add it as a bookmark, now click the bookmark and try and execute the window.open part of your script and see what happens (post back with a description of the result).... if it does the same thing its probably firefox's settings...
What version of firefox are you using?
Also i have a small idea of what the problem Could be... Window.open is designed to open a html... page rather than an image, this works fine in IE but may not work in FF, try placing the image on a page and opening that.... it may fix your problem, it may not...
Usually when i'm having problems with cross browser compatibility i use this resource:
http://www.reloco.com.ar/mozilla/compat.html
Again, check your javascript settings but i dont know what the problem is just yet...
Hope you get this sorted,
Jamey