Storage Server 2008 BSODs

muhaynes2 · Jan 30, 2012

Hello!

· OS: Storage Server 2008 x64 SP2
· Original OEM installation
· Age of system: 8 Months old
· Age of OS installation: 8 Months old
· CPU: 2 x Intel Xeon E5620
· Video Card: Matrox G200e
· MotherBoard: Dell 0DPRKF (according to CPU-Z)
· Power Supply - Dell 620W redundant PSUs
· System Manufacturer: Dell
· Exact model number: NX3100

This thing has been crashing for months. It does a lot (B2D&T/vSphere box) so it's been quite difficult to narrow down. AFAIK the latest drivers are installed, Windows is up-to-date, and memtest came back clean on the memory. Any help would be much appreciated! Please see attachment for additional info & crashdumps.

usasma · Jan 31, 2012

I'm unfamiliar with the updates in Server 2008 - but most Vista SP2 systems have more than 250 updates after SP2. Check Windows Update to ensure that you have all available updated.

I suspect a hardware issue, but am going to ask someone else to have a look at this topic before I recommend anything specific.

One of these is a DRIVER_VERIFIER_DETECTED_VIOLATION (c4)

Here's a list of the drivers that showed in the last 2 memory dumps (both are from Jan 2012). Please note those that are highlighted in RED:
3RD PARTY DRIVERS PRESENT IN THE DUMP FILES
- Create a System Restore Point prior to doing any of this. DO NOT mess with the drivers themselves - leave the Windows\System32\drivers directory alone unless we specifically direct you to it!
- Please either update the older drivers from the device manufacturer's website - or uninstall them from your system. Reference links are included below.
- DO NOT use Windows Update or the Update Drivers function of Device Manager.
- Please feel free to post back about any drivers that you are having difficulty locating.
- Windows Update exceptions may be noted below for Windows drivers.

Code:

[font=lucida console]
NDIS.SYS                   Sat Apr 11 01:43:15 2009 (49E02DF3)
bxvbda.sys                 Thu Jan 06 13:55:14 2011 (4D261012)
lsi_sas2.sys               Tue Dec 14 10:15:14 2010 (4D078A02)
percsas2.sys               Mon Aug 09 14:20:48 2010 (4C604700)
quota.sys                  Sat Apr 11 00:59:38 2009 (49E023BA)
sisss.sys                  Thu Sep 17 05:33:52 2009 (4AB20280)
datascrn.sys               Sat Apr 11 00:59:35 2009 (49E023B7)
bxnd60a.sys                Fri Feb 04 19:58:43 2011 (4D4CA0C3)
tpfilter.sys               Thu May 13 15:52:28 2010 (4BEC587C)
[Color=Red]TAPE.SYS                   Sat Jan 19 01:29:22 2008 (479198C2)[/Color]
[Color=Red]halfinch.sys               Wed Jan 23 13:55:03 2008 (47978D87)[/Color]
[Color=Red]storflt.sys                Sat Nov 17 22:02:02 2007 (473FAB2A)[/Color]
ltotape.sys                Wed Jul 08 11:23:03 2009 (4A54B9D7)
G200eWm.sys                Mon Jul 27 23:01:02 2009 (4A6E69EE)
IPMIDrv.sys                Sat Apr 11 01:15:16 2009 (49E02764)
basp.sys                   Tue Dec 21 13:31:41 2010 (4D10F28D)
EraserUtilRebootDrv.sys    Fri Oct 21 21:18:29 2011 (4EA219E5)
StarPort.sys               Wed Sep 08 12:47:16 2010 (4C87BE14)
dcdbas64.sys               Thu Jun 11 13:44:04 2009 (4A314264)
vmnetadapter.sys           Mon Aug 10 08:04:53 2009 (4A800CE5)
VMNET.SYS                  Mon Aug 10 08:04:50 2009 (4A800CE2)
dump_percsas2.sys          Mon Aug 09 14:20:48 2010 (4C604700)
vmnetbridge.sys            Mon Aug 10 08:05:58 2009 (4A800D26)
SRTSP64.SYS                Fri Mar 04 15:39:16 2011 (4D714DF4)
SYMEVENT64x86.SYS          Thu Mar 24 19:02:36 2011 (4D8BCD8C)
VMkbd.sys                  Sat Mar 26 01:31:22 2011 (4D8D7A2A)
SRTSPX64.SYS               Fri Mar 04 15:39:38 2011 (4D714E0A)
EX64.SYS                   Fri Jul 29 09:15:11 2011 (4E32B25F)
[Color=Red]scsichng.sys               Thu Aug 23 23:01:40 2007 (46CE4A14)[/Color]
eeCtrl64.sys               Fri Oct 21 21:18:28 2011 (4EA219E4)
hcmon.sys                  Sat Mar 26 00:40:38 2011 (4D8D6E46)
vmci.sys                   Sat Mar 26 00:08:49 2011 (4D8D66D1)
vmx86.sys                  Sat Mar 26 02:18:58 2011 (4D8D8552)
[Color=Red]BNCHMRK2.vsd               Tue Feb 12 11:47:30 2008 (47B1CDA2)[/Color]
VeeamFSR.sys               Fri Apr 08 12:30:24 2011 (4D9F3820)
vmnetuserif.sys            Sat Mar 26 00:55:31 2011 (4D8D71C3)
vstor2-mntapi10-shared.sys Thu Nov 05 15:18:36 2009 (4AF3331C)
vstor2-vci10.sys           Wed Dec 22 19:08:29 2010 (4D1292FD)
vstor2-ws60.sys            Thu Aug 19 16:28:39 2010 (4C6D93F7)
vstor2.sys                 Tue Jan 04 22:12:53 2011 (4D23E1B5)
ENG64.SYS                  Fri Jul 29 09:17:51 2011 (4E32B2FF)
vdk.sys                    Fri Apr 08 12:31:07 2011 (4D9F384B)
[/font]

Code:

[font=lucida console]
Loading Dump File [C:\Users\FUBAR\_jcgriff2_\dbug\__Kernel__\Mini013012-01.dmp]
Built by: 6002.18484.amd64fre.vistasp2_gdr.110617-0336
Debug session time: Mon Jan 30 15:00:04.055 2012 (UTC - 5:00)
System Uptime: 10 days 10:56:07.795
*** WARNING: Unable to verify timestamp for Ntfs.sys
*** ERROR: Module load completed but symbols could not be loaded for Ntfs.sys
Probably caused by : ntkrnlmp.exe ( nt!RtlVirtualUnwind+16d )
BUGCHECK_STR:  0x1E_c0000005
DEFAULT_BUCKET_ID:  DRIVER_FAULT_SERVER_MINIDUMP
PROCESS_NAME:  ccSvcHst.exe
FAILURE_BUCKET_ID:  X64_0x1E_c0000005_nt!RtlVirtualUnwind+16d
Bugcheck code 0000001E
Arguments ffffffff`c0000005 fffff800`01ccc05d 00000000`00000000 00000000`000000d8
¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨``
Loading Dump File [C:\Users\FUBAR\_jcgriff2_\dbug\__Kernel__\Mini012012-01.dmp]
Built by: 6002.18484.amd64fre.vistasp2_gdr.110617-0336
Debug session time: Fri Jan 20 03:59:28.861 2012 (UTC - 5:00)
System Uptime: 4 days 22:39:37.451
Probably caused by : ntkrnlmp.exe ( nt! ?? ::FNODOBFM::`string'+2774 )
[B][U]DRIVER_VERIFIER_DETECTED_VIOLATION (c4)[/U][/B]
BUGCHECK_STR:  0xc4_91
DEFAULT_BUCKET_ID:  DRIVER_FAULT_SERVER_MINIDUMP
PROCESS_NAME:  tomcat6.exe
FAILURE_BUCKET_ID:  X64_0xc4_91_nt!_??_::FNODOBFM::_string_+2774
Bugcheck code 000000C4
Arguments 00000000`00000091 00000000`00000001 fffffa80`2c34d6c0 00000000`00000000
BiosVersion = 1.6.3
BiosReleaseDate = 02/01/2011
CPUID:        "Intel(R) Xeon(R) CPU           E5620  @ 2.40GHz"
MaxSpeed:     2400
CurrentSpeed: 2400
¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨``
  
  [/font]

Will post back when I've finished running the rest of the memory dumps (15 of them).....

usasma · Jan 31, 2012

15 memory dumps over a 5 month period
6 different BSOD error codes cited
3 different causes blamed
The only unfamiliar cause is VeeamFSR.sys which was blamed 3 times
It's a backup software from here: Veeam: #1 for VMware vSphere, ESX & ESXi Backup and Virtualization Management

I recommend uninstalling the Veeam product, then downloading and installing a fresh copy of the latest Server 2008 compatible version.

Beyond that, let's wait for the others to make some input here.
While we're waiting, I'll be looking up the drivers that don't have links listed below.

Further info on BSOD error messages available at: http://www.carrona.org/bsodindx.html

The following info is just FYI, I've already addressed the issues that I saw in the above paragraphs:
3RD PARTY DRIVERS PRESENT IN THE DUMP FILES
- Create a System Restore Point prior to doing any of this. DO NOT mess with the drivers themselves - leave the Windows\System32\drivers directory alone unless we specifically direct you to it!
- Please either update the older drivers from the device manufacturer's website - or uninstall them from your system. Reference links are included below.
- DO NOT use Windows Update or the Update Drivers function of Device Manager.
- Please feel free to post back about any drivers that you are having difficulty locating.
- Windows Update exceptions may be noted below for Windows drivers.

Code:

[font=lucida console]
NDIS.SYS                   Sat Apr 11 01:43:15 2009 (49E02DF3)
bxvbda.sys                 Thu Jan 06 13:55:14 2011 (4D261012)
lsi_sas2.sys               Tue Dec 14 10:15:14 2010 (4D078A02)
percsas2.sys               Mon Aug 09 14:20:48 2010 (4C604700)
quota.sys                  Sat Apr 11 00:59:38 2009 (49E023BA)
sisss.sys                  Thu Sep 17 05:33:52 2009 (4AB20280)
datascrn.sys               Sat Apr 11 00:59:35 2009 (49E023B7)
bxnd60a.sys                Fri Feb 04 19:58:43 2011 (4D4CA0C3)
tpfilter.sys               Thu May 13 15:52:28 2010 (4BEC587C)
TAPE.SYS                   Sat Jan 19 01:29:22 2008 (479198C2)
halfinch.sys               Wed Jan 23 13:55:03 2008 (47978D87)
[Color=Red]storflt.sys                Sat Nov 17 22:02:02 2007 (473FAB2A)[/Color]
ltotape.sys                Wed Jul 08 11:23:03 2009 (4A54B9D7)
G200eWm.sys                Mon Jul 27 23:01:02 2009 (4A6E69EE)
IPMIDrv.sys                Sat Apr 11 01:15:16 2009 (49E02764)
basp.sys                   Tue Dec 21 13:31:41 2010 (4D10F28D)
EraserUtilRebootDrv.sys    Fri Oct 21 21:18:29 2011 (4EA219E5)
StarPort.sys               Wed Sep 08 12:47:16 2010 (4C87BE14)
dcdbas64.sys               Thu Jun 11 13:44:04 2009 (4A314264)
vmnetadapter.sys           Mon Aug 10 08:04:53 2009 (4A800CE5)
VMNET.SYS                  Mon Aug 10 08:04:50 2009 (4A800CE2)
dump_percsas2.sys          Mon Aug 09 14:20:48 2010 (4C604700)
vmnetbridge.sys            Mon Aug 10 08:05:58 2009 (4A800D26)
SRTSP64.SYS                Fri Mar 04 15:39:16 2011 (4D714DF4)
SYMEVENT64x86.SYS          Thu Mar 24 19:02:36 2011 (4D8BCD8C)
VMkbd.sys                  Sat Mar 26 01:31:22 2011 (4D8D7A2A)
SRTSPX64.SYS               Fri Mar 04 15:39:38 2011 (4D714E0A)
EX64.SYS                   Fri Jul 29 09:15:11 2011 (4E32B25F)
[Color=Red]scsichng.sys               Thu Aug 23 23:01:40 2007 (46CE4A14)[/Color]
eeCtrl64.sys               Fri Oct 21 21:18:28 2011 (4EA219E4)
hcmon.sys                  Sat Mar 26 00:40:38 2011 (4D8D6E46)
vmci.sys                   Sat Mar 26 00:08:49 2011 (4D8D66D1)
vmx86.sys                  Sat Mar 26 02:18:58 2011 (4D8D8552)
BNCHMRK2.vsd               Tue Feb 12 11:47:30 2008 (47B1CDA2)
VeeamFSR.sys               Fri Apr 08 12:30:24 2011 (4D9F3820)
vmnetuserif.sys            Sat Mar 26 00:55:31 2011 (4D8D71C3)
vstor2-mntapi10-shared.sys Thu Nov 05 15:18:36 2009 (4AF3331C)
vstor2-vci10.sys           Wed Dec 22 19:08:29 2010 (4D1292FD)
vstor2-ws60.sys            Thu Aug 19 16:28:39 2010 (4C6D93F7)
vstor2.sys                 Tue Jan 04 22:12:53 2011 (4D23E1B5)
ENG64.SYS                  Fri Jul 29 09:17:51 2011 (4E32B2FF)
vdk.sys                    Fri Apr 08 12:31:07 2011 (4D9F384B)
G200eWm.sys                Tue Feb 17 12:19:22 2009 (499AF19A)
EraserUtilDrvI13.sys       Fri Oct 21 21:18:29 2011 (4EA219E5)
SYMEVENT64x86.SYS          Thu Apr 15 21:32:19 2010 (4BC7BE23)
EraserUtilRebootDrv.sys    Thu Jul 07 21:45:01 2011 (4E16611D)
EraserUtilDrv11120.sys     Fri Oct 21 21:18:29 2011 (4EA219E5)
eeCtrl64.sys               Thu Jul 07 21:45:01 2011 (4E16611D)
eqldsm.sys                 Fri Jun 11 16:21:19 2010 (4C129ABF)
dump_storport.sys          Sat Apr 11 01:34:30 2009 (49E02BE6)
wtlmdrv.sys                Thu Apr 16 17:12:58 2009 (49E79F5A)
[/font]

Code:

[font=lucida console]
Loading Dump File [C:\Users\FUBAR\_jcgriff2_\dbug\__Kernel__\Mini013012-01.dmp]
Built by: 6002.18484.amd64fre.vistasp2_gdr.110617-0336
Debug session time: Mon Jan 30 15:00:04.055 2012 (UTC - 5:00)
System Uptime: 10 days 10:56:07.795
*** WARNING: Unable to verify timestamp for Ntfs.sys
*** ERROR: Module load completed but symbols could not be loaded for Ntfs.sys
Probably caused by : ntkrnlmp.exe ( nt!RtlVirtualUnwind+16d )
BUGCHECK_STR:  0x1E_c0000005
DEFAULT_BUCKET_ID:  DRIVER_FAULT_SERVER_MINIDUMP
PROCESS_NAME:  ccSvcHst.exe
FAILURE_BUCKET_ID:  X64_0x1E_c0000005_nt!RtlVirtualUnwind+16d
Bugcheck code 0000001E
Arguments ffffffff`c0000005 fffff800`01ccc05d 00000000`00000000 00000000`000000d8
¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨``
Loading Dump File [C:\Users\FUBAR\_jcgriff2_\dbug\__Kernel__\Mini012012-01.dmp]
Built by: 6002.18484.amd64fre.vistasp2_gdr.110617-0336
Debug session time: Fri Jan 20 03:59:28.861 2012 (UTC - 5:00)
System Uptime: 4 days 22:39:37.451
Probably caused by : ntkrnlmp.exe ( nt! ?? ::FNODOBFM::`string'+2774 )
DRIVER_VERIFIER_DETECTED_VIOLATION (c4)
BUGCHECK_STR:  0xc4_91
DEFAULT_BUCKET_ID:  DRIVER_FAULT_SERVER_MINIDUMP
PROCESS_NAME:  tomcat6.exe
FAILURE_BUCKET_ID:  X64_0xc4_91_nt!_??_::FNODOBFM::_string_+2774
Bugcheck code 000000C4
Arguments 00000000`00000091 00000000`00000001 fffffa80`2c34d6c0 00000000`00000000
BiosVersion = 1.6.3
BiosReleaseDate = 02/01/2011
CPUID:        "Intel(R) Xeon(R) CPU           E5620  @ 2.40GHz"
MaxSpeed:     2400
CurrentSpeed: 2400
¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨``
Loading Dump File [C:\Users\FUBAR\_jcgriff2_\dbug\__Kernel__\Mini122811-01.dmp]
Built by: 6002.18484.amd64fre.vistasp2_gdr.110617-0336
Debug session time: Wed Dec 28 23:59:06.430 2011 (UTC - 5:00)
System Uptime: 0 days 11:24:32.243
Probably caused by : ntkrnlmp.exe ( nt!KiSystemServiceExit+209 )
PROCESS_NAME:  svchost.exe
BUGCHECK_STR:  RAISED_IRQL_FAULT
DEFAULT_BUCKET_ID:  DRIVER_FAULT_SERVER_MINIDUMP
FAILURE_BUCKET_ID:  X64_RAISED_IRQL_FAULT_svchost.exe_nt!KiSystemServiceExit+209
Bugcheck code 0000004A
Arguments 00000000`77126bfa 00000000`00000001 00000000`00000000 fffffa60`09082ca0
BiosVersion = 1.6.3
BiosReleaseDate = 02/01/2011
CPUID:        "Intel(R) Xeon(R) CPU           E5620  @ 2.40GHz"
MaxSpeed:     2400
CurrentSpeed: 2400
¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨``
Loading Dump File [C:\Users\FUBAR\_jcgriff2_\dbug\__Kernel__\Mini122611-01.dmp]
Built by: 6002.18484.amd64fre.vistasp2_gdr.110617-0336
Debug session time: Tue Dec 27 00:00:05.518 2011 (UTC - 5:00)
System Uptime: 3 days 8:47:28.829
*** WARNING: Unable to verify timestamp for VeeamFSR.sys
*** ERROR: Module load completed but symbols could not be loaded for VeeamFSR.sys
Probably caused by : VeeamFSR.sys ( VeeamFSR+1cc5e )
BUGCHECK_STR:  0x1E_c0000005
DEFAULT_BUCKET_ID:  DRIVER_FAULT_SERVER_MINIDUMP
PROCESS_NAME:  tomcat6.exe
FAILURE_BUCKET_ID:  X64_0x1E_c0000005_VeeamFSR+1cc5e
Bugcheck code 0000001E
Arguments ffffffff`c0000005 fffff800`01cc805d 00000000`00000000 00000000`0034b5e0
BiosVersion = 1.6.3
BiosReleaseDate = 02/01/2011
CPUID:        "Intel(R) Xeon(R) CPU           E5620  @ 2.40GHz"
MaxSpeed:     2400
CurrentSpeed: 2400
¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨``
Loading Dump File [C:\Users\FUBAR\_jcgriff2_\dbug\__Kernel__\Mini122311-01.dmp]
Built by: 6002.18484.amd64fre.vistasp2_gdr.110617-0336
Debug session time: Fri Dec 23 10:59:26.248 2011 (UTC - 5:00)
System Uptime: 5 days 5:37:18.033
*** WARNING: Unable to verify timestamp for Ntfs.sys
*** ERROR: Module load completed but symbols could not be loaded for Ntfs.sys
Probably caused by : ntkrnlmp.exe ( nt!RtlVirtualUnwind+63 )
DEFAULT_BUCKET_ID:  DRIVER_FAULT_SERVER_MINIDUMP
BUGCHECK_STR:  0x50
PROCESS_NAME:  SearchIndexer.e
FAILURE_BUCKET_ID:  X64_0x50_nt!RtlVirtualUnwind+63
Bugcheck code 00000050
Arguments fffffa60`052c0008 00000000`00000000 fffff800`01c8ff53 00000000`00000000
BiosVersion = 1.6.3
BiosReleaseDate = 02/01/2011
CPUID:        "Intel(R) Xeon(R) CPU           E5620  @ 2.40GHz"
MaxSpeed:     2400
CurrentSpeed: 2400
¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨``
Loading Dump File [C:\Users\FUBAR\_jcgriff2_\dbug\__Kernel__\Mini111911-01.dmp]
Built by: 6002.18484.amd64fre.vistasp2_gdr.110617-0336
Debug session time: Fri Nov 18 21:59:18.536 2011 (UTC - 5:00)
System Uptime: 4 days 8:40:16.202
*** WARNING: Unable to verify timestamp for Ntfs.sys
*** ERROR: Module load completed but symbols could not be loaded for Ntfs.sys
Probably caused by : Ntfs.sys ( Ntfs+12054 )
DEFAULT_BUCKET_ID:  DRIVER_FAULT_SERVER_MINIDUMP
PROCESS_NAME:  SearchIndexer.e
BUGCHECK_STR:  0x24
FAILURE_BUCKET_ID:  X64_0x24_Ntfs+12054
Bugcheck code 00000024
Arguments 00000000`001904aa fffffa60`0c92ac08 fffffa60`0c92a5e0 fffff800`01c8205d
BiosVersion = 1.6.3
BiosReleaseDate = 02/01/2011
CPUID:        "Intel(R) Xeon(R) CPU           E5620  @ 2.40GHz"
MaxSpeed:     2400
CurrentSpeed: 2400
¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨``
Loading Dump File [C:\Users\FUBAR\_jcgriff2_\dbug\__Kernel__\Mini111111-01.dmp]
Built by: 6002.18484.amd64fre.vistasp2_gdr.110617-0336
Debug session time: Fri Nov 11 18:59:13.730 2011 (UTC - 5:00)
System Uptime: 11 days 8:21:49.637
*** WARNING: Unable to verify timestamp for VeeamFSR.sys
*** ERROR: Module load completed but symbols could not be loaded for VeeamFSR.sys
Probably caused by : VeeamFSR.sys ( VeeamFSR+1cc5e )
DEFAULT_BUCKET_ID:  DRIVER_FAULT_SERVER_MINIDUMP
BUGCHECK_STR:  0x50
PROCESS_NAME:  dsm_sa_datamgr3
FAILURE_BUCKET_ID:  X64_0x50_VeeamFSR+1cc5e
Bugcheck code 00000050
Arguments fffff960`00a7c6a8 00000000`00000000 fffff800`01c98b3e 00000000`00000002
¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨``
Loading Dump File [C:\Users\FUBAR\_jcgriff2_\dbug\__Kernel__\Mini103111-01.dmp]
Built by: 6002.18484.amd64fre.vistasp2_gdr.110617-0336
Debug session time: Sun Oct 30 19:59:29.062 2011 (UTC - 5:00)
System Uptime: 6 days 9:42:13.042
*** WARNING: Unable to verify timestamp for Ntfs.sys
*** ERROR: Module load completed but symbols could not be loaded for Ntfs.sys
Probably caused by : ntkrnlmp.exe ( nt!RtlLookupFunctionEntry+13e )
DEFAULT_BUCKET_ID:  DRIVER_FAULT_SERVER_MINIDUMP
BUGCHECK_STR:  0x50
PROCESS_NAME:  Rtvscan.exe
FAILURE_BUCKET_ID:  X64_0x50_nt!RtlLookupFunctionEntry+13e
Bugcheck code 00000050
Arguments fffff960`00921c3c 00000000`00000000 fffff800`01ce5b3e 00000000`00000002
BiosVersion = 1.6.3
BiosReleaseDate = 02/01/2011
CPUID:        "Intel(R) Xeon(R) CPU           E5620  @ 2.40GHz"
MaxSpeed:     2400
CurrentSpeed: 2400
¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨``
Loading Dump File [C:\Users\FUBAR\_jcgriff2_\dbug\__Kernel__\Mini102411-01.dmp]
Built by: 6002.18484.amd64fre.vistasp2_gdr.110617-0336
Debug session time: Mon Oct 24 09:59:51.996 2011 (UTC - 5:00)
System Uptime: 8 days 5:35:13.532
Probably caused by : ntkrnlmp.exe ( nt!KiDoubleFaultAbort+b8 )
BUGCHECK_STR:  0x7f_8
DEFAULT_BUCKET_ID:  DRIVER_FAULT_SERVER_MINIDUMP
PROCESS_NAME:  Rtvscan.exe
FAILURE_BUCKET_ID:  X64_0x7f_8_nt!KiDoubleFaultAbort+b8
Bugcheck code 0000007F
Arguments 00000000`00000008 00000000`80050033 00000000`000006f8 fffff800`01cd46ec
BiosVersion = 1.6.3
BiosReleaseDate = 02/01/2011
CPUID:        "Intel(R) Xeon(R) CPU           E5620  @ 2.40GHz"
MaxSpeed:     2400
CurrentSpeed: 2400
¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨``
Loading Dump File [C:\Users\FUBAR\_jcgriff2_\dbug\__Kernel__\Mini090611-01.dmp]
Built by: 6002.18484.amd64fre.vistasp2_gdr.110617-0336
Debug session time: Tue Sep  6 11:58:53.341 2011 (UTC - 5:00)
System Uptime: 1 days 15:55:35.526
*** WARNING: Unable to verify timestamp for Ntfs.sys
*** ERROR: Module load completed but symbols could not be loaded for Ntfs.sys
Probably caused by : ntkrnlmp.exe ( nt!PspGetSetContextInternal+396 )
DEFAULT_BUCKET_ID:  DRIVER_FAULT_SERVER_MINIDUMP
BUGCHECK_STR:  0x50
PROCESS_NAME:  Rtvscan.exe
FAILURE_BUCKET_ID:  X64_0x50_nt!PspGetSetContextInternal+396
Bugcheck code 00000050
Arguments fffffa80`33e60000 00000000`00000000 fffff800`01f0054e 00000000`00000000
BiosVersion = 1.6.3
BiosReleaseDate = 02/01/2011
CPUID:        "Intel(R) Xeon(R) CPU           E5620  @ 2.40GHz"
MaxSpeed:     2400
CurrentSpeed: 2400
¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨``
Loading Dump File [C:\Users\FUBAR\_jcgriff2_\dbug\__Kernel__\Mini090411-01.dmp]
Built by: 6002.18484.amd64fre.vistasp2_gdr.110617-0336
Debug session time: Sun Sep  4 19:59:18.136 2011 (UTC - 5:00)
System Uptime: 3 days 5:10:42.352
*** WARNING: Unable to verify timestamp for Ntfs.sys
*** ERROR: Module load completed but symbols could not be loaded for Ntfs.sys
Probably caused by : ntkrnlmp.exe ( nt!RtlLookupFunctionEntry+13e )
DEFAULT_BUCKET_ID:  DRIVER_FAULT_SERVER_MINIDUMP
BUGCHECK_STR:  0x50
PROCESS_NAME:  Rtvscan.exe
FAILURE_BUCKET_ID:  X64_0x50_nt!RtlLookupFunctionEntry+13e
Bugcheck code 00000050
Arguments fffff960`00b1c6a8 00000000`00000000 fffff800`01ca0b3e 00000000`00000002
BiosVersion = 1.6.3
BiosReleaseDate = 02/01/2011
CPUID:        "Intel(R) Xeon(R) CPU           E5620  @ 2.40GHz"
MaxSpeed:     2400
CurrentSpeed: 2400
¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨``
Loading Dump File [C:\Users\FUBAR\_jcgriff2_\dbug\__Kernel__\Mini082811-01.dmp]
Built by: 6002.18484.amd64fre.vistasp2_gdr.110617-0336
Debug session time: Sun Aug 28 19:59:08.031 2011 (UTC - 5:00)
System Uptime: 8 days 11:56:38.242
*** WARNING: Unable to verify timestamp for Ntfs.sys
*** ERROR: Module load completed but symbols could not be loaded for Ntfs.sys
Probably caused by : ntkrnlmp.exe ( nt!RtlVirtualUnwind+63 )
DEFAULT_BUCKET_ID:  DRIVER_FAULT_SERVER_MINIDUMP
BUGCHECK_STR:  0x50
PROCESS_NAME:  Rtvscan.exe
FAILURE_BUCKET_ID:  X64_0x50_nt!RtlVirtualUnwind+63
Bugcheck code 00000050
Arguments fffffa60`00e46b3f 00000000`00000000 fffff800`01ca0f53 00000000`00000000
BiosVersion = 1.6.3
BiosReleaseDate = 02/01/2011
CPUID:        "Intel(R) Xeon(R) CPU           E5620  @ 2.40GHz"
MaxSpeed:     2400
CurrentSpeed: 2400
¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨``
Loading Dump File [C:\Users\FUBAR\_jcgriff2_\dbug\__Kernel__\Mini082011-01.dmp]
Built by: 6002.18484.amd64fre.vistasp2_gdr.110617-0336
Debug session time: Sat Aug 20 07:58:55.943 2011 (UTC - 5:00)
System Uptime: 0 days 18:41:19.732
Probably caused by : ntkrnlmp.exe ( nt!KiDoubleFaultAbort+b8 )
BUGCHECK_STR:  0x7f_8
DEFAULT_BUCKET_ID:  DRIVER_FAULT_SERVER_MINIDUMP
PROCESS_NAME:  StarWindService
FAILURE_BUCKET_ID:  X64_0x7f_8_nt!KiDoubleFaultAbort+b8
Bugcheck code 0000007F
Arguments 00000000`00000008 00000000`80050033 00000000`000006f8 fffff800`01c750a5
BiosVersion = 1.6.3
BiosReleaseDate = 02/01/2011
CPUID:        "Intel(R) Xeon(R) CPU           E5620  @ 2.40GHz"
MaxSpeed:     2400
CurrentSpeed: 2400
¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨``
Loading Dump File [C:\Users\FUBAR\_jcgriff2_\dbug\__Kernel__\Mini081911-01.dmp]
Built by: 6002.18484.amd64fre.vistasp2_gdr.110617-0336
Debug session time: Fri Aug 19 11:59:13.618 2011 (UTC - 5:00)
System Uptime: 1 days 11:09:49.691
*** WARNING: Unable to verify timestamp for VeeamFSR.sys
*** ERROR: Module load completed but symbols could not be loaded for VeeamFSR.sys
Probably caused by : VeeamFSR.sys ( VeeamFSR+1cc5e )
DEFAULT_BUCKET_ID:  DRIVER_FAULT_SERVER_MINIDUMP
BUGCHECK_STR:  0x50
PROCESS_NAME:  StarWindService
FAILURE_BUCKET_ID:  X64_0x50_VeeamFSR+1cc5e
Bugcheck code 00000050
Arguments fffffa80`33800000 00000000`00000000 fffff800`01eec54e 00000000`00000002
BiosVersion = 1.6.3
BiosReleaseDate = 02/01/2011
CPUID:        "Intel(R) Xeon(R) CPU           E5620  @ 2.40GHz"
MaxSpeed:     2400
CurrentSpeed: 2400
¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨``
Loading Dump File [C:\Users\FUBAR\_jcgriff2_\dbug\__Kernel__\Mini081411-01.dmp]
Built by: 6002.18327.amd64fre.vistasp2_gdr.101014-0432
Debug session time: Sun Aug 14 19:59:21.559 2011 (UTC - 5:00)
System Uptime: 17 days 6:44:37.191
*** WARNING: Unable to verify timestamp for Ntfs.sys
*** ERROR: Module load completed but symbols could not be loaded for Ntfs.sys
Probably caused by : ntkrnlmp.exe ( nt!RtlVirtualUnwind+250 )
DEFAULT_BUCKET_ID:  DRIVER_FAULT_SERVER_MINIDUMP
BUGCHECK_STR:  0x50
PROCESS_NAME:  Rtvscan.exe
FAILURE_BUCKET_ID:  X64_0x50_nt!RtlVirtualUnwind+250
Bugcheck code 00000050
Arguments fffffa60`101b5d70 00000000`00000000 fffff800`01c83140 00000000`00000000
BiosVersion = 1.6.3
BiosReleaseDate = 02/01/2011
CPUID:        "Intel(R) Xeon(R) CPU           E5620  @ 2.40GHz"
MaxSpeed:     2400
CurrentSpeed: 2400
¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨``
  
  [/font]

VirGnarus · Jan 31, 2012

To add to all of this, I noticed something very suspicious that I see popping up often in your crashdumps:

Code:

Unable to load image \SystemRoot\System32\Drivers\Ntfs.sys, Win32 error 0n2
*** WARNING: Unable to verify timestamp for Ntfs.sys
*** ERROR: Module load completed but symbols could not be loaded for Ntfs.sys

...

14: kd> lmvm ntfs
start             end                 module name
fffffa60`0120a000 fffffa60`01387000   Ntfs     T (no symbols)           
    Loaded symbol image file: Ntfs.sys
    Image path: \SystemRoot\System32\Drivers\Ntfs.sys
    Image name: Ntfs.sys
    Timestamp:        Fri Mar 19 12:28:07 2010 (4BA3A617)
    CheckSum:         0017BE07
    ImageSize:        0017D000
    Translations:     0000.04b0 0000.04e4 0409.04b0 0409.04e4

There's no symbols information that seems to be present for this driver, and that seems to be the case in all your crashdumps. This is a crucial driver for Windows that is a private driver, meaning it is made by MS and no other driver should touch it or attempt to overwrite it with their own version. Either this is being caused because the image of the file for the driver has been paged out onto disk (placed onto paging file and removed from memory), which is bad, or it has been replaced/redirected by some software (which will be from rootkits), which is doubly bad.

Can any other analyst do an lmvm on it to confirm I'm not dealing with a symbols problem personally? I doubt it, because I saw the file image doesn't have a header nor any dbg info. Though there's always the chance.

Anyways, I suspect foul play involved with the Ntfs driver. It's been popping up several times in your crashdumps and the fact that it has no symbol information worries me.

You might be able to check if it's paged out or a rootkit by going to the image path mentioned above for the ntfs file (which is, I would assume, C:\Windows\System32\Drivers\ntfs.sys) and check the Properties of the ntfs.sys file, namely that in the Details section. If it's blank, or the timestamp does not match what's present here, we are most certainly dealing with a rootkit. This is not an absolute positive, there's always the chance the rootkit - should it exist - be redirecting your kernel to point to another file impersonating as the genuine Ntfs.sys. So if you see that the properties and all look clean, that doesn't entirely rule out infection, but it does make it more evident we're dealing with a paged out image (which shouldn't happen here).

muhaynes2 · Jan 31, 2012

Thanks for the replies guys.

VirGnarus: I've checked the ntfs.sys in the \windows\system32\drivers directory, the timestamp on the file itself is 06/01/2011, however if I go to the Digital Signatures tab, it says it's been validated by Microsoft Windows and the signing date matches the date in your post... so I presume that means the file is legit?

usasma: Veeam might be the most difficult app on the server to reinstall. It currently backs up our entire virtual infrastructure and is maintaining 30 days of reversed incrementals - I will check to see if there are any updates and also check with their support to see if they have anything on file that speaks to the issue.

I forgot to mention one important piece of information in my original post. When the problem first began I noticed a trend, it always happened referencing rtvscan.exe (Symantec Endpoint Protection's scanner) and on Sundays. This is the day our servers run their full disk scan, so I thought it was a problem with Symantec and reinstalled it. That didn't resolve the problem, so eventually I just omitted the full scan on this box. I should mention that the data partition is, uh, very large. 14TB large. I didn't think that was a problem at the time as I thought all the newer versions of Windows could deal with large disks, but perhaps that's a little too large?

usasma · Jan 31, 2012

EDIT: Probably doesn't mean that it's clean - from this:

So if you see that the properties and all look clean, that doesn't entirely rule out infection, but it does make it more evident we're dealing with a paged out image (which shouldn't happen here).

Veeam might be the most difficult app on the server to reinstall.

I'm known for being difficult! :0)

You're going to have to eventually scan that 14 tB partition - just to be sure.

But, I'd first try to remove Symantec Endpoint Protection (I've seen problems with it recently on client systems), run the removal tool, the install a freshly downloaded copy of the latest compatible version. Maybe that'll help with the problem that cites rtvscan.exe.

Any disk quota violations? Run this from a command prompt: fsutil quota violations

usasma · Jan 31, 2012

My output of lmvm ntfs from the Mini013012-01.dmp file:

Code:

12: kd> lmvm ntfs
start             end                 module name
fffffa60`01202000 fffffa60`0137f000   Ntfs     T (no symbols)           
    Loaded symbol image file: Ntfs.sys
    Image path: \SystemRoot\System32\Drivers\Ntfs.sys
    Image name: Ntfs.sys
    Timestamp:        Fri Mar 19 12:28:07 2010 (4BA3A617)
    CheckSum:         0017BE07
    ImageSize:        0017D000
    Translations:     0000.04b0 0000.04e4 0409.04b0 0409.04e4

VirGnarus · Jan 31, 2012

@usasma:

Thanks, usasma. That coincides with what I have.

@muhaynes2:

That sounds about right. It could've just simply been paged out to disk, but considering we're dealing with the Ntfs.sys driver, I can't see why that would ever happen (on a storage system, no doubt). Again, this is still irrational behavior, and I still suspect something amiss (though probably not infection).

The Dell NX3100 is supposed to handle well over 14TB. I don't believe that is a concern. Though always check Dell for any necessary driver updates and especially BIOS/UEFI and firmware updates should they exist.

Another thing I find interesting is that the one Driver Verifier-detected crashdump that usasma mentioned, doesn't appear right. Driver Verifier doesn't appear that it's on using !verifier, and the bucket ID also shows it was not involved. I also don't appear to see Driver Verifier mentioned on the callstack either. This may just be a basic Windows check that mentions DV as being involved.

Do you recall turning on Driver Verifier at any time? I'm not sure DV would be the best thing for a production unit such as yours as it's designed to crash the system, as well as might cause performance degradation.

The big issue with diagnosing production systems like this is that they're not in a position to be tampered with for diagnostic purposes. The only way escalation engineers are able to debug production equipment like this without impeding on the production server is to actually purchase and build an identical setup and test inhouse, which they can then live kernel debug and have their way with it at their leisure. Evidently, that cannot be the case here at TSF.

You could give us the kernel dump for this system if you'd like. Try zipping up \MEMORY.DMP located in your Windows directory and see how large it is. Obviously it's going to be way too big still for this forum, but you can use a 3rd-party site to upload it too. However I can't guarantee my skills at this time will be adequate enough to analyze it thoroughly to give an exact cause, nor can I be assured that I will be working with a crashdump from a crash that will actually reveal a cause even if I had the skills to dive deep enough into it. Still, you may choose this option if you desire.

muhaynes2 · Jan 31, 2012

Yeah sadly this machine is in production basically Mon-Sat 24/7, Sundays are really the only day I can do anything drastic... and some of these scans (*cough 14TB chkdsk) will exceed that timeframe. Such is life though!

I've removed SEPM, because, well I know it's caused problems in the past and years of experience has skewed me towards blaming AV software. I checked driver verifier and it's not enabled. I realize this is a difficult diagnosis, the machine is quite big and has many roles, many components, and a lot of software - so I really appreciate you guys taking the time. That said, I have uploaded the zipped copy of the bigboy memory.dmp to my Dropbox, and would love it if someone has the time to take a look.

Linky: http://dl.dropbox.com/u/55105981/MEMORY.zip

Thanks again!

Storage Server 2008 BSODs

muhaynes2

Attachments

usasma

usasma

VirGnarus

muhaynes2

usasma

usasma

VirGnarus

muhaynes2

Top Contributors this Month