Upgrading to Vista would probably make things worse. Stick with XP for now while it's still the better OS for gaming.
When you upgraded your hardware, did you check to see if your power supply unit (PSU) would be strong enough for the new components? When games that previously worked start to lag and freeze after a graphics card upgrade, this can be due to overheating or a weak +12V line on the PSU (underpowered). The 8800GT is very power-hungry and needs a PSU with at least 26A/+12V, probably a 650W or higher, depending on the rest of your components.
Of course, this doesn't explain why the game won't install, which is more likely due to remaining registry entries from your previous installation that are difficult to find, but it could explain the lag in games. Enter your full system specs into this
power calculator and set Capacitor Aging to 25%. Click the Calculate button, then add 30% to the total to allow for PSU inefficiency. Post back with this figure and the PSU label details (make/model, total watts, amps for +12V).
Are you up to date with your MS updates, chipset drivers, device drivers and DirectX? Is your XP SP2 or the newly released SP3?
Are you using 32bit or 64bit XP? 32bit can only use 3 to 3.5gb of your 4gb RAM. You would need a 64bit version of Windows or Linux to use the full 4gb.