lemuppet

We have a remote server which is blue screening. It can boot OK into safe mode and safe moe with networking. It can (most of the time) boot into normal mode, but sometimes bluescreens stop code 9c. If you login you get an almost immediate BSOD.

It is running Exchange 2003 and SQL 2000 and is a DC. It also runs DNS, DFS and was a Symantec AV server (not any more)

I have managed to RPC and look at teh evnt logs and we get a few WMIxWDM errors (106) after a crash.

I have included one of the dump files outputs below...



Microsoft (R) Windows Debugger Version 6.11.0001.404 X86
Copyright (c) Microsoft Corporation. All rights reserved.


Loading Dump File [C:\Documents and Settings\SMorris\Desktop\HK Minidumps\Mini031709-07.dmp]
Mini Kernel Dump File: Only registers and stack trace are available

Symbol search path is: C:\WINDOWS\Symbols2003R2
Executable search path is:

"nt" was not found in the image list.
Debugger will attempt to load "nt" at given base 00000000.

Please provide the full image name, including the extension (i.e. kernel32.dll)
for more reliable results.Base address and size overrides can be given as
.reload <image.ext>=<base>,<size>.
Unable to load image nt, Win32 error 0n2
Unable to add module at 00000000
Debugger can not determine kernel base address
Windows Server 2003 Kernel Version 3790 (Service Pack 2) MP (4 procs) Free x86 compatible
Product: LanManNt, suite: TerminalServer SingleUserTS
Machine Name:
Kernel base = 0xe0800000 PsLoadedModuleList = 0xe08a6ea8
Debug session time: Mon Mar 16 23:45:36.988 2009 (GMT+0)
System Uptime: 0 days 0:02:36.859

"nt" was not found in the image list.
Debugger will attempt to load "nt" at given base 00000000.

Please provide the full image name, including the extension (i.e. kernel32.dll)
for more reliable results.Base address and size overrides can be given as
.reload <image.ext>=<base>,<size>.
Unable to load image nt, Win32 error 0n2
Unable to add module at 00000000
Debugger can not determine kernel base address
Loading Kernel Symbols

Loading User Symbols
*******************************************************************************
* *
* Bugcheck Analysis *
* *
*******************************************************************************

Use !analyze -v to get detailed debugging information.

BugCheck 9C, {4, e08977a0, f421a000, b2080a13}

***** Debugger could not find nt in module list, module list might be corrupt, error 0x80070057.

Probably caused by : Unknown_Image ( ANALYSIS_INCONCLUSIVE )

Followup: MachineOwner
---------

0: kd> !analyze -v
*******************************************************************************
* *
* Bugcheck Analysis *
* *
*******************************************************************************

MACHINE_CHECK_EXCEPTION (9c)
A fatal Machine Check Exception has occurred.
KeBugCheckEx parameters;
x86 Processors
If the processor has ONLY MCE feature available (For example Intel
Pentium), the parameters are:
1 - Low 32 bits of P5_MC_TYPE MSR
2 - Address of MCA_EXCEPTION structure
3 - High 32 bits of P5_MC_ADDR MSR
4 - Low 32 bits of P5_MC_ADDR MSR
If the processor also has MCA feature available (For example Intel
Pentium Pro), the parameters are:
1 - Bank number
2 - Address of MCA_EXCEPTION structure
3 - High 32 bits of MCi_STATUS MSR for the MCA bank that had the error
4 - Low 32 bits of MCi_STATUS MSR for the MCA bank that had the error
IA64 Processors
1 - Bugcheck Type
1 - MCA_ASSERT
2 - MCA_GET_STATEINFO
SAL returned an error for SAL_GET_STATEINFO while processing MCA.
3 - MCA_CLEAR_STATEINFO
SAL returned an error for SAL_CLEAR_STATEINFO while processing MCA.
4 - MCA_FATAL
FW reported a fatal MCA.
5 - MCA_NONFATAL
SAL reported a recoverable MCA and we don't support currently
support recovery or SAL generated an MCA and then couldn't
produce an error record.
0xB - INIT_ASSERT
0xC - INIT_GET_STATEINFO
SAL returned an error for SAL_GET_STATEINFO while processing INIT event.
0xD - INIT_CLEAR_STATEINFO
SAL returned an error for SAL_CLEAR_STATEINFO while processing INIT event.
0xE - INIT_FATAL
Not used.
2 - Address of log
3 - Size of log
4 - Error code in the case of x_GET_STATEINFO or x_CLEAR_STATEINFO
AMD64 Processors
1 - Bank number
2 - Address of MCA_EXCEPTION structure
3 - High 32 bits of MCi_STATUS MSR for the MCA bank that had the error
4 - Low 32 bits of MCi_STATUS MSR for the MCA bank that had the error
Arguments:
Arg1: 00000004
Arg2: e08977a0
Arg3: f421a000
Arg4: b2080a13

Debugging Details:
------------------

***** Debugger could not find nt in module list, module list might be corrupt, error 0x80070057.

NOTE: This is a hardware error. This error was reported by the CPU
via Interrupt 18. This analysis will provide more information about
the specific error. Please contact the manufacturer for additional
information about this error and troubleshooting assistance.

This error is documented in the following publication:

- Bios and Kernel Developers Guid for AMD Athlon(r) 64 and AMD Opteron(r) Processors
Bit Mask:

MA Model Specific MCA
O ID Other Information Error Code Error Code
VV SDP ___________|____________ _______|_______ _______|______
AEUECRC| | | |
LRCNVVC| | | |
^^^^^^^| | | |
6 5 4 3 2 1
3210987654321098765432109876543210987654321098765432109876543210
----------------------------------------------------------------
1111010000100001100111111111111110110010000010000000101000010011


VAL - MCi_STATUS register is valid
Indicates that the information contained within the IA32_MCi_STATUS
register is valid. When this flag is set, the processor follows the
rules given for the OVER flag in the IA32_MCi_STATUS register when
overwriting previously valid entries. The processor sets the VAL
flag and software is responsible for clearing it.

OVER - Error Overflow
Indicates that a machine check error occurred while the results of a
previous error were still in the error-reporting register bank (that
is, the VAL bit was already set in the IA32_MCi_STATUS register).
the processor sets the OVER flag and software is responsible for
clearing it. Enabled errors are written over disabled errors, and
uncorrected errors are written over corrected errors. Uncorrected
errors are not written over previous valid uncorrected errors.

UC - Error Uncorrected
Indicates that the processor did not or was not able to correct the
error condition. When clear, this flag indicates that the processor
was able to correct the error condition.

EN - Error Enabled
Indicates that the error was enabled by the associated EEj bit of the
IA32_MCi_CTL register.

ADDRV - IA32_MCi_ADDR register valid
Indicates that the IA32_MCi_ADDR register contains the address where
the error occurred.

BUSCONNERR - Bus and Interconnect Error BUS{LL}_{PP}_{RRRR}_{II}_{T}_err
These errors match the format 0000 1PPT RRRR IILL



Concatenated Error Code:
--------------------------
_VAL_OVER_UC_EN_ADDRV_BUSCONNERR_213

This error code can be reported back to the manufacturer.
They may be able to provide additional information based upon
this error. All questions regarding STOP 0x9C should be
directed to the hardware manufacturer.

BUGCHECK_STR: 0x9C_AuthenticAMD

CUSTOMER_CRASH_COUNT: 7

DEFAULT_BUCKET_ID: DRIVER_FAULT_SERVER_MINIDUMP

CURRENT_IRQL: 0

LAST_CONTROL_TRANSFER: from e0a64154 to e0827c83

STACK_TEXT:
WARNING: Frame IP not in any known module. Following frames may be wrong.
e0897770 e0a64154 0000009c 00000004 e08977a0 0xe0827c83
e08978a4 e0a5b86f e0042000 00000000 00000000 0xe0a64154
00000000 00000000 00000000 00000000 00000000 0xe0a5b86f


STACK_COMMAND: kb

SYMBOL_NAME: ANALYSIS_INCONCLUSIVE

FOLLOWUP_NAME: MachineOwner

MODULE_NAME: Unknown_Module

IMAGE_NAME: Unknown_Image

DEBUG_FLR_IMAGE_TIMESTAMP: 0

BUCKET_ID: CORRUPT_MODULELIST

Followup: MachineOwner
---------

The server 2003 is running 2 x AMD Opterons.

We have tried various things, but do not really know where to go next?

Steve

djaburg

Sounds like the issue:

If this hasn't been going on long, I check to see what updates have been run recently and determine if the blue screen issue corresponds to the timing of an update.

lemuppet

Thanks for your post.

Ther have been no updates applied since early January whule the server only started blue screening this saturday (two months later).

I have managed to semi-stabalise it using msconfig and disabling non-essential services and startup programs. We can now log in using Terminal Server without it crashing straight-away. The main console is running appalling slow and is probably due to the unistallation of drivers during the attempted fix - it is so slow that using our KVM over IP I cannot login as each key is taking an eternity to display. The machine did eventually still blue screen after running for about 18 hours last night and produced a stop code 9c.

Any further ideas? I am sort of resigned to it being a hardware issue. If it is processor related I am wondering if we can disable one somehow - I think I saw a way in the boot.ini spec...

Steve