If you have a remote access program on your computer after replacing an entire drive (and re-installing the OS obviously) because you had a remote access program, chances are good that somebody using your computer is getting it infected.
The only way the same person would be able to find you again, other than by pure wild coincidence, would be if you had a static public IP, which I seriously doubt unless it's something like a web server.
You say nobody has physical access to the PC, but if you found a RAT and replaced the HDD and OS and then got the RATs back then somebody - you or another user - is almost certainly using the PC in a way that's giving it a malware infection. In theory, a virus can be stored in the PC's firmware (such that a replaced HDD and OS wouldn't clean it), but this is beyond unlikely. Even if it were a rootkit, an entirely new drive and OS would clean it out.
Tracing a malware infection to a specific source is both very difficult and generally pointless, as the vast, vast majority of infections are done by people who have no idea who you are. If you do trace it down, the trail will probably end somewhere short of the actual person responsible.
You need to get your computer cleaned up and try to determine how you got reinfected then take steps to prevent a third infection. Most common sources of infection include opening infected emails, visiting shady websites, downloading freeware, and drive-by downloads you get off otherwise legitimate sites. Proper internet usage habits, a regular cleaning regimine, disabling Java as much as possible, and keeping up-to-date antivirus protection are your best defenses on an individual machine. For networks you should be running a firewall and disabling ports you don't use in addition to that and you can also use a NIDS system to monitor access to the network.
CompTIA A+, Network+ certified