1stgRAIDer

I posted this in another tech forum but I am hoping for a quick response so I can begin the recovery process as soon as possible.

Before I begin, here is a quick list of my hardware:
Intel DG33TL mobo w/onboard ICH8R/ICH9R/ICH10R SATA RAID controller. There are six SATA ports total available for RAID on the board (one is marked for eSATA). All six ports are being used. 2 ports are single drives 250gb and 320gb, 3 ports are 500gb drives in a RAID 5 configuration, last port is connected to external 1tb eSATA drive. All drives are Western Digital including the external. The 3 RAID drives are Western Digital RAID edition hard drives. Intel Matrix Storage Manager v8.2.0.1001 is installed and the OS is Vista Ultimate 32bit.

Ok...here is the horrible thing that happened to my internal RAID5 setup. I lost two of the 3 drives in my RAID 5 array recently. I lost the second before I replaced the first one that failed. I figured after one went that I had some time before another would go. Well...the second drive went during the night a few nights ago and to my wonder in the morning I had no storage on my PC....just my OS drive and drive that I use as a file graveyard and testing area. I rebooted and the Intel controller information during bootup stated that the array was in a degraded state and that one of the RAID drives had an error.

So now that the second is gone, I am pretty upset since the array stored a lot of irreplaceable data. The array consisted of 3 WD RAID Edition 500gb SATA2 hard drives. After the first drive errorred itself out of the array, the array still ran with the two drives as storage ever since (I am not sure if this is still considered RAID 5 mode when while it is running without the full amount of drives supposed to be in the array since the size of the array remained at 1tb and there were only two drives. It almost seems as though it is just a spanned disk this way. Can somebody clarify this for recovery purposes?) I recently got a 1 TB eSata drive ready to backup the array but its too late now as the failure happened before I could do so.

What is strange is that I removed the first drive from the array after its failure and left it connected as a single drive to the pc (non RAID). I tested it using DFT and it passed the advanced tests....freaking weird!!! This drive is still being used as normal storage and there are no further issues with it. The other night the second drive did the same thing....same error. I have no idea why they are getting errors. It seems to me like it might be a controller issue since the first failed drive tested good outside of the array. I have not tested the second drive yet. I did recently reload my OS before this second RAID issue. I updated the Intel inf driver and sound driver for the motherboard.

I proceeded to try repairing the array. I re-added the first failed drive back to the array using the Intel Utility during boot up with CTRL+I. Once added I booted back into windows and ran the Intel Storage Utility which now sees all three drives. The utility added the third drive as a 'spare' which was the only option. Now in the utility, two drives are listed normally under 'RAID hard drives' and the third is listed as 'missing'. There are no actions available to any of the three drives. The only action available is to the array as a whole and is to 'Enable volume write back cache'.

A couple Google searches say that there is ways to rebuild or recover the array using software. Has anybody had success doing this? If so please tell me what method or software you used for this. I am looking for any input on how to recover this data. It is much more important for me to recover the data than to repair the array. I am assuming that the recovery would be easier if the array can be rebuilt.

Thanks for any input and help on this subject that you can offer.

wiseleo

Ah, welcome to my world!

R-studio or similar tool is what you want. And if your disks are OK physically, it shouldn't be too much of a problem. You will need at least twice as much free storage space as there is currently on the array.

I have a hunch that it may be your power supply that's killing the drives.

You are working with live drives. That's suicidal to your data. Image them, throw them into Raid Reconstructor - http://runtime.org/raid.htm or UFS explorer http://www.ufsexplorer.com or R-studio - http://www.r-studio.com - whichever one makes sense

You will find this useful: http://www.data-recovery-software.ne...ery_Manual.pdf

1stgRAIDer

Man, thank you for the great information.

Ok, before I start, is it possible to recover data from just two of the three hard drives from the RAID array since I have already formatted and used the first one that dropped out?

Thats interesting about the power supply being the possible culprit. I am running a DVD-RW, 5 hard drives and an nVidia 8600GT video card on a Q6700 CPU with 2gb of ram. The power supply is 500watts. I use the PC for business mostly. No gaming or really intensive tasks. Do you think that the power supply is dropping the drives out of the array? It appears that they are still good by testing in DFT...they are just no longer part of the array.

I will check out R-studio and use drive images to work with in Raid Reconstructor as you suggested. I have tried Raid Reconstructor without success but maybe since I wasnt using images and using the array directly.

How should I go about imaging the drives? Should I image the drive with a program like Ghost or Imgburn? What is the best way to image the drives and the steps to take so as not to make them any worse off? Can I image the drives from windows and with them in thier RAID array? Should I remove the drives from the array and reconnect them seperately?

Sorry for all the questions, It sounds like you have gone through this before and are a good resource.

Oh, and I just ordered a Cavalry 3tb eSATA external RAID device that I will use to transfer the data to and for future backups of the array in the PC.

Thanks!

wiseleo

My signature has instructions on how to get good images. If possible, do it on a system that is known good. Working on suspect systems produces questionable results.

I do this for a living, so rest assured those steps are about as good as it gets unless you spend thousands of dollars on the really fun stuff. Yes, it's possible to rebuild a RAID5 using 2 out of 3 disks. You must use software images for reconstruction because this can get tricky.

It's more about the quality of power. The 8600GT + 5 HDDs? Sounds like a power-hungry box. I am going to assume your drives are not in a staggered start configuration because they are not SCSI, where that's common. At startup, those drives spike to about 150W total.

Try a better PSU. That 500W might not be 500W... Either Anandtech or Tom's did a test on this. Your controller card may also be at fault.

1stgRAIDer

Wiseleo, I have received my external device (a Cavalry 3tb, 4x 750gb SATA RAID device running on a Silicon Image SATA PCI-E controller) and I am ready to start the recovery process. I read and printed your tutorial from your signature and before I go further, I have a couple questions.

I should start by mentioning I am using my main PC (with the failed array) as the recovery machine since I am limited to it. I believe that there should not be any worry since my main bootable hard drive is NOT included in the RAID 5 array (also located in the PC) and is fully functional.

Before starting SysRescueCD, I logged the state of the drives in the Intel Manager at bootup (CTRL + I). The 3 500gb drives are the RAID 5 array that is failed:
Strip size 64kb Status FAILED
Port# DISK STATUS
0 250gb Non RAID Disk
1 500gb Member Disk (0)
2 500gb Non RAID Disk
5 500gb Member Disk (0)

I then booted to the SysRescueCD and was faced with a lot of information at the main first screen where a command was to be entered. After several seconds of indecision, the next section automatically loaded and at the next command prompt I entered 'testdisk' as you noted in your guide. This brought up another place to enter a command and here I stand as I do not know how to proceed and I do not want to damage the media on the disks if there is any chance of recovery.

In testdisk it says this:
Select a media then press enter:
Disk /dev/sda 250gb
Disk /dev/sdb 500gb
Disk /dev/sdc 500gb
Disk /dev/sdd 500gb
Disk /dev/sde 750gb
Disk /dev/sdf 750gb
Disk /dev/sdg 750gb
Disk /dev/sdh 750gb
[proceed] [quit]

The 750gb drives listed above are located externally in the new Cavalry eSATA device and are setup in a RAID 5 configuration with just over 2tb available.

I also have access to a 1tb eSATA drive that I can use if needed. It is mostly full but not critical data so I can dump it if I need to.

Now the questions....
1) do I want to image all three of the original RAID 5 500gb drives to the external Cavalry 3tb array?
2) is this even possible since the drives are listed individually in SysRescueCD and not the array? Which drive do I use for the destination drive in the image command?
3)You wrote in your guide: "note the drive names....Quit testdisk.....run this command: ntfs3g/dev/sdb1/mnt/windows" Your final command is: ddrescue-n/dev/sda/mnt/windows/image.dd/mnt/windows/image.log."
How do I quit testdisk and at which prompts do I run these commands? Do I run this command for each of the RAID 5 drives?
4)Is there anything I am missing that I may need or will help with this process (beside lots of alcohol :) ?
5) Lastly, does it make a difference if the drives are still member disks in the array while I attempt to image them? If yes, how do I go about this?

Thanks for all of you expertise in this area. You are the only person that I have come across that is knowledgeable in manually recovering RAID data so far! I am looking forward to your comments.

wiseleo

OK, let's figure this out...

It looks like Linux doesn't see your Cavalry device as a single unit, but rather as a JBOD (just a bunch of disks) device. I suspect you setup a software RAID under Windows and you expected Linux to see it. It actually is good that Linux can't see it. RAID5 is renowned for pretty terrible write performance and in this case it won't do you any good.

I am going to simply try a few other things instead. If you thought single drive recovery was fun, you are in for a treat.

We will image each of the 500GB disks to a corresponding 750GB unit. Should be able to do it in parallel.

However, before we do any of this. What happens if you remove that power-hungry 8600GT card and replace the power supply with a better unit? Your array may simply come online if it gets enough quality power. Power supplies love to kill drives.

The procedure is relatively simple. Boot to Windows and attach the Cavalry device. Remove the RAID5 on the Cavalry under windows. Format each 750GB drive a basic simple volume with its own huge 750GB NTFS partition. Quick format is OK.

Boot from the CD.

mkdir /mnt/disk1 /mnt/disk2 /mnt/disk3 /mnt/disk4
ntfs-3g /dev/sde1 /mnt/disk1
ntfs-3g /dev/sdf1 /mnt/disk2
ntfs-3g /dev/sdg1 /mnt/disk3
ntfs-3g /dev/sdh1 /mnt/disk4

That should mount your external array in 4 NTFS mount points. The 4th disk is not really necessary, but I didn't want to mess with rearranging disk order as 8 disks is already confusing enough ;-)

Run df -k to be certain that everything is mounted correctly.

Under linux, you have ability to run commands in parallel and on multiple virtual terminals.

I want to make sure my other disks are not mounted

umount /dev/sda
umount /dev/sdb
umount /dev/sdc
umount /dev/sde

So...
ddrescue -n /dev/sdb /mnt/disk1/disk1.dd /mnt/disk1/disk1.log
Hit enter to start it.
Now hit Alt-F2, which will bring you to a new screen
ddrescue -n /dev/sdc /mnt/disk2/disk2.dd /mnt/disk2/disk2.log
Hit enter to start it.
Now hit Alt-F3, which will bring you to a new screen
ddrescue -n /dev/sdd /mnt/disk3/disk3.dd /mnt/disk3/disk3.log

Those log files are vital.

I am skipping testdisk commands in-between because I know what I am doing and I am making the big assumption that you won't reconfigure the drives from what you posted.

If all goes well, the imaging process is now underway and should complete with no errors. The rate is about 40GB/hr, in my experience with slower controllers.

You can monitor the progress by using Alt-F1, Alt-F2, Alt-F3 key combinations.

Once this is completed, you can use Windows-based software to attempt to reconstruct the array. I am still experimenting with it, so I don't have a definite recommendation at this time as to which RAID reconstruction software is best.

Theoretically, it should be possible to reconstruct arrays entirely under Linux with lvm and similar tools, but I have not had the chance to work with them yet.

1stgRAIDer

Good question. I have removed all drives except for the raid array in the past and it did not come online. I believe the drives originally did start staggered and I know that they shutdown staggered in Vista. All of the drives show in the Intel Utility at boot and in the Linux environment. Doesnt this show that they are powered up? With all devices connected I am showing .150kw (150watts?) on the LCD on my battery backup device. I really dont think I there is not enough power from the PSU but maybe on this particular power rail. Its definitely worth the try.

Before I try this, in my last post, I showed that in the Intel Manager at bootup (CTRL + I), disk 2 is shown as 'Non RAID Disk' as follows:
Port# DISK STATUS
0 250gb Non RAID Disk
1 500gb Member Disk (0)
2 500gb Non RAID Disk
5 500gb Member Disk (0)

How do I re-add the disk to the array without damaging the array? It did not do this automatically and I do not see an option to add the single drive to the array from within the manager.

Ok, I deleted the RAID volume in the SATARaid5Manager software that came with the Silicon Image card to manage the Cavalry unit. I then selected 'Make Pass-Through' in the same software. Now it shows all 4 750gb drives as full capacity with 698.63gb available. They did not show in My Computer at this point.
Here I went into the Windows Disk Management Utility to format the drives. I found there were 4 drives shown but they are not listed as I though they should be.

In Disk Management:
Disk 3 698.64gb Unallocated
Disk 4 2095.78gb Raw Healthy (Primary Partition)
Disk 5 698.64gb Unallocated
Disk 6 698.64gb Unallocated

It looks like it still thinks the drives are in RAID so I selected Disk 4 and deleted the volume. It is still 'working' after almost 30 minutes and the status has not changed. The Disk Management utility is currently not responding while 'deleting' the volume.

1stgRAIDer

--a quick update--

The Disk Management Utility ended its task from what I understand but it brought back the Computer Management area. I attempted to enter the Disk Management utility again but it does not load it. Strange. I will likely reboot but I am hoping to hear your input first.

wiseleo

Gotta disable RAID.

A reboot should clear all this up.

1stgRAIDer

Ok, it did not take the first time when deleting the Cavalry array in Windows Disk Management. I rebooted and the deleted it again, then rebooted again. During boot I entered the Raid controller management for the Cavalry to confirm it shows no RAID.

Back in Windows Disk Management it now shows correctly all 4 750gb drives inside the Cavalry but the one that was listed as the array last time is showing a slightly different size. The difference seems insignificant but I wanted to mention it.
Disk 1 698.63
Disk 2 698.51 **
Disk 3 698.63
Disk 4 698.63

Aside from the question I asked in my previous post(below), I think I am ready to start the image creation:

"Originally Posted by wiseleo
What happens if you remove that power-hungry 8600GT card and replace the power supply with a better unit? Your array may simply come online if it gets enough quality power.

Good question. I have removed all drives except for the raid array in the past and it did not come online. I believe the drives originally did start staggered and I know that they shutdown staggered in Vista. All of the drives show in the Intel Utility at boot and in the Linux environment. Doesnt this show that they are powered up? With all devices connected I am showing .150kw (150watts?) on the LCD on my battery backup device. I really dont think I there is not enough power from the PSU but maybe on this particular power rail. Its definitely worth the try.

Before I try this, in my last post, I showed that in the Intel Manager at bootup (CTRL + I), disk 2 is shown as 'Non RAID Disk' as follows:
Port# DISK STATUS
0 250gb Non RAID Disk
1 500gb Member Disk (0)
2 500gb Non RAID Disk
5 500gb Member Disk (0)

How do I re-add the disk to the array without damaging the array? It did not do this automatically and I do not see an option to add the single drive to the array from within the manager. "

wiseleo

The partition differences should be insignificant for the purposes of this discussion, but I'd probably RMA that drive. Don't like surprises.

You'll reconstruct the RAID in software. I don't think I'd trust that Intel controller.

You think too much. I'd have replaced the PSU long ago. Swapping parts is cheaper than diagnostics time. :)

1stgRAIDer

Well...the drive in question was not stable in rescuecd to work with (mounted but was not recognised as a proper ntfs partition) so I adjusted your direction and used disks 1,3, and 4. Luckily there were 4 drives to work with.

The process completed tonight and I now have a diskX.dd file on each hard drive. I am going to attempt to rebuild the array using software now. I will likely start with raid reconstructor unless you have any other recommendations.

I dont recognize the image format 'xxx.dd'. Will most recovery software recognize it or does it need to be converted into a manageable image format?

wiseleo

.dd is just a raw disk image.

Good luck with the software reconstruction. I don't have a firm opinion on which software to use.

1stgRAIDer

Thanks for all your help up to this point, Wiseleo.

I will post back when I have updates using the recovery software.

cb_oz

Thanks Wiseleo, your advice above to 1stgRAIDer was invaluable for my RAID5 restoration effort.
In particular ddrescue overcame limitations in nearly every other imaging application out there (I'm a new fan).
I personally used Raid Reconstructer and GetDataBack for NTFS once I had the images, whole process took 2 days but was worth it!

Thanks again!

rjmarshall

Another thorny RAID 5 problem...

Hi Wiseleo and everyone-

I have an Adaptec 6 port SATA RAID card with 6 WD 500GB SATA drives. My boot drive is separate and is still fine. A while ago, I set up a 5 drive array with the 6th drive as a hot spare. Yesterday I checked the server and saw the following message:

RAID Storage Manager Agent : Periodic scan found one or more degraded logical drives: controller 1. Repair as soon as possible to avoid data loss.

At that point, I determined which drive was offline, and wiggled the cable corresponding to that drive on the controller card. Thus begins my tale of woe...

As you probably would have told me, had I asked before doing this, this caused my array to fail. After reviewing this thread and the R-Drive Manual and looking at whatever else I could find on the web, I deleted the logical array and established each of the 6 drives as a simple volume in the RAID Storage Manager to be able to see them in R-Drive. I then ran RAID Reconstructor on a number of permutations of 5 drive arrays, but always have come up with not significant results. I also tried using R-Studio, but could not get the volume to appear. I am really bummed about this and decided to check with you before I cause any more problems.... Any ideas or guidance would be HUGELY appreciated.

Your humble correspondent,

Rick M.

Sertelegger

Hi everybody,

I have another RAID5 horror story to add to the mix. First of all I want to say thanks to wiseleo and everybody else for the useful information, however I might need some extra help with this one:

I have an older MSI motherboard with SIL 3114 SATA controller on it, which I used to run a RAID5 using 3x 500GB WD drives. So far so good. Then just before Christmas my PSU died, so I got a new one, which I installed and everything was fine again.

When the computer came up with the new PSU, I noticed that one of the drives was showing up as "Invalid RAID drive" in the SIL BIOS manager. So I thought no biggie, deleted and re-added the drive, thinking that this is exactly what RAID5 is for and that I shouldn't have a problem reconstructing the third drive.

This is where the pain begins: I used the SIL Windows GUI to rebuild the RAID group, and when I did that, the whole system just hung itself up. As it turns out the drive that was shown as bad really was bad and only occasionally comes up fine, but most of the time just disappears from the system. Well the Silicon Image controller did not like that and just crashed the system.

After the necessary reboot !!BOOM!! my whole RAID5 is gone! In the SIL BIOS manager now all three drives show up as invalid RAID disks. I still thought this should be easy to recover from, I deleted the RAID group and redefined it. However, when asked for the drive order I wasn't sure anymore what my original order was, especially since the description of the drives is exactly the same and I am not 100% sure that I re-cabled everything exactly the same way after replacing the PSU (which required me to remove the motherboard).

To come to the end: Windows only recognized the new RAID configuration as a new drive and wanted me to initialize and format it etc. which I did not do of course. I tried out different drive order combinations in the SIL BIOS manager hoping that it would show up in Windows again, but to no avail.

So I have two out of three drives of the RAID5 system, but I don't know what order they should be in. Sounds like something I should be able to recover from, right?

Well, I tried RAID Reconstructor, but just like rjmarshall it cannot come up with any significant results, even though I know the drive parameters except the order.

I also tried iRecover, which supposedly handles that exact case of determining the drive order and being able to reconstruct a RAID5 from two drives only, but just like with GetDataBackFromNTFS the results are garbage: I can see most of the directory and file names, but when I copy something over it only produces garbage. With JPG images it is easiest to see that something got messed up with the striping, because the photo will be put together with lots of stripes from different photos. So apparently the auto-detection wasn't so good after all.

I don't have the disk space lying around right now to try the same on disk images, and frankly I am not sure using disk images instead of the actual disks would show any difference.

Is all hope lost? I am trying to recover thousands of photos I had taken thinking they would be safer on a RAID...

UPDATE: I ran Raid Reconstructor again, but this time increased the probe size to some ridiculous amount (things sleep deprivation and frustration do to you), and to my very surprise this time it came up with a driver order recommendation, and the block size also matched what I already knew it was. So I'm running GetDataBackForNTFS on the Virtual image now to see what it comes up with and if it will be able to sort the stripes out correctly this time. I am even tempted to let the 6 hours run through instead of aborting to get a sneak peek at the results. I will update this post or post an update as I find out more. I am hopeful again!

Sertelegger

I did it!

Well, RAID Reconstructor + GetDataBackForNTFS did :-)

Looks like increasing the probe size for RAID Reconstructor did the trick, since it got the drive order right, and GetDataBackForNTFS was able to recover 100% of my data from the 2 drives that were still working fine.

So I guess all things considered even though I lost the information about the RAID5 for a while and thought I had lost everything in the end RAID5 saved my butt, because I wouldn't have been able to recover everything with one failed drive if it had been a RAID0 or just a bunch of drives.

@rjmarshall: Try to increase the probe size (just add a couple of zeros, but processing time will take MUCH longer then!) to see if RAID Reconstructor finds a significant result then. Good luck!