They both have their ups and downs. Speed wise, I find them pretty much the same. I use safari on a Mac so I don't have to worry about the security aspect all that much
. Both are stable and expendable. Safari, supposedly, handles text better though I don't think I saw a noticeable difference.
It's all down to preference I guess. I don't think you'll notice any major differences between FF and Safari. I'd recommend leaving it on the system just in case your default browser develops a problem, but there shouldn't be a issue if you choose to remove it.