S
Simone Dutt
Guest
To cut a long story as short as possible, our Dell PE SC1420 running Server
x64 standard SP2 keeps intermittently restarting from a stop 0x00000050 error
and logs only a "the previous system shutdown at x on x was unexpected" event
in the log. The WinDbg dump analysis (included below) always shows the error
to be a nonpaged pool expansion caused by our DAT tape driver (pvdatw2k),
which is current and supplied by Dell (there is no native x64 driver). Dell
have replaced the drive and cable and most of the major components in the
system, but the error keeps occuring. I sent them the crash dump analysis and
I have run my own diagnostics and swapped the RAM module positions and a
litany of other things (which the system passed) to try and get to the bottom
of this issue, but to no avail, and Dell have now closed this case. Whilst I
am able to run the dump analysis, I am not really able to make much useful
sense of it. I have been keeping records of the memory locations that show up
in the crash dump and have noticed that both memory referenced and the
instruction address referencing are very similar each time (see below also
for previous memory references). The shutdowns always occur when the server
is not doing anything e.g. at 3.00am on a weekend. We use ntbackup (and
therefore Volume Shadow Copy) to back the server up and I was thinking that
may be at fault, but the backup is usually finished before any crash occurs.
Has anyone else had VSC cause a nonpaged pool expansion? Is that what the
crash dump analysis is showing, or is it the driver? It's clear what memory
is at fault, but where is that freed memory exactly?
PAGE_FAULT_IN_NONPAGED_AREA (50)
Invalid system memory was referenced. This cannot be protected by try-except,
it must be protected by a Probe. Typically the address is just plain bad or
it
is pointing at freed memory.
Arguments:
Arg1: fffffadfd2986003, memory referenced.
Arg2: 0000000000000000, value 0 = read operation, 1 = write operation.
Arg3: fffffadfc8aec075, If non-zero, the instruction address which
referenced the bad memory
address.
Arg4: 0000000000000000, (reserved)
Debugging Details:
------------------
READ_ADDRESS: fffffadfd2986003 Nonpaged pool expansion
FAULTING_IP:
pvdatw2k+1075
fffffadf`c8aec075 410fb65103 movzx edx,byte ptr [r9+3]
MM_INTERNAL_CODE: 0
IMAGE_NAME: pvdatw2k.sys
DEBUG_FLR_IMAGE_TIMESTAMP: 412f435b
MODULE_NAME: pvdatw2k
FAULTING_MODULE: fffffadfc8aeb000 pvdatw2k
DEFAULT_BUCKET_ID: DRIVER_FAULT
BUGCHECK_STR: 0x50
PROCESS_NAME: System
CURRENT_IRQL: 1
TRAP_FRAME: fffffadfc8e8d550 -- (.trap fffffadfc8e8d550)
NOTE: The trap frame does not contain all registers.
Some register values may be zeroed.
rax=0000000000000000 rbx=0000000000000000 rcx=0000000000000000
rdx=0000000000000000 rsi=0000000000000003 rdi=fffff800011a98fd
rip=fffffadfc8aec075 rsp=fffffadfc8e8d6e0 rbp=0000000000000003
r8=0000000000000000 r9=fffffadfd2986000 r10=fffffadfd2986004
r11=0000000000000010 r12=0000000000000000 r13=0000000000000000
r14=0000000000000000 r15=0000000000000000
iopl=0 nv up ei pl nz na pe nc
pvdatw2k+0x1075:
fffffadf`c8aec075 410fb65103 movzx edx,byte ptr [r9+3]
ds:0001:fffffadf`d2986003=??
Resetting default scope
LAST_CONTROL_TRANSFER: from fffff800010a6072 to fffff8000102e890
STACK_TEXT:
fffffadf`c8e8d478 fffff800`010a6072 : 00000000`00000050 fffffadf`d2986003
00000000`00000000 fffffadf`c8e8d550 : nt!KeBugCheckEx
fffffadf`c8e8d480 fffff800`0102d459 : 00000000`00000000 00000000`00000000
00000000`00000000 00000000`00000000 : nt!MmAccessFault+0xa1f
fffffadf`c8e8d550 fffffadf`c8aec075 : fffffadf`d16e6b90 fffffadf`c8aec54a
fffffadf`cead30d8 00000000`00000030 : nt!KiPageFault+0x119
fffffadf`c8e8d6e0 fffffadf`c8aec54a : fffffadf`cead30d8 00000000`00000030
00000005`00000000 fffffadf`cd254e00 : pvdatw2k+0x1075
fffffadf`c8e8d6f0 fffffadf`c8aec6f1 : 00000000`00000000 00000000`00000000
fffffadf`c8e8d8e0 00000000`00000000 : pvdatw2k+0x154a
fffffadf`c8e8d760 fffffadf`c8aec7ed : 00000001`00060000 fffffadf`c8e8d7d8
fffffadf`c8e8d7d8 00000000`00000002 : pvdatw2k+0x16f1
fffffadf`c8e8d7d0 fffffadf`c8b82bb4 : fffffadf`cdd2b830 fffffadf`c8e8d8e0
00000005`00000000 00000000`00000144 : pvdatw2k+0x17ed
fffffadf`c8e8d830 fffffadf`c8390acd : 00000000`00000000 fffffadf`cbe93980
fffffadf`cdd2b830 fffffadf`cdd2b830 : TAPE!TapeDeviceControl+0x484
fffffadf`c8e8da00 fffff800`0128e1f8 : 00000000`00000001 fffffadf`d47890f0
fffffadf`cdd2b830 fffff800`0128e1e0 : CLASSPNP!ClasspFailurePredict+0xfd
fffffadf`c8e8dcd0 fffff800`010375ca : 00000000`00000000 fffffadf`d4789001
00000000`00000000 fffffadf`d47890f0 : nt!IopProcessWorkItem+0x17
fffffadf`c8e8dd00 fffff800`0124a972 : fffffadf`cead3040 00000000`00000080
fffffadf`cead3040 fffffadf`c8aa3680 : nt!ExpWorkerThread+0x13b
fffffadf`c8e8dd70 fffff800`01020226 : fffffadf`c8a9b180 fffffadf`cead3040
fffffadf`c8aa3680 00000000`00000000 : nt!PspSystemThreadStartup+0x3e
fffffadf`c8e8ddd0 00000000`00000000 : 00000000`00000000 00000000`00000000
00000000`00000000 00000000`00000000 : nt!KxStartSystemThread+0x16
STACK_COMMAND: kb
FOLLOWUP_IP:
pvdatw2k+1075
fffffadf`c8aec075 410fb65103 movzx edx,byte ptr [r9+3]
SYMBOL_STACK_INDEX: 3
SYMBOL_NAME: pvdatw2k+1075
FOLLOWUP_NAME: MachineOwner
FAILURE_BUCKET_ID: X64_0x50_pvdatw2k+1075
BUCKET_ID: X64_0x50_pvdatw2k+1075
Followup: MachineOwner
Previous memory references -
Thursday 15th November 2007
Arg1: fffffadffe7f7003, memory referenced.
Arg2: 0000000000000000, value 0 = read operation, 1 = write operation.
Arg3: fffffadfc8b6c075, If non-zero, the instruction address which
referenced the bad memory
address.
Friday 14th September 2007
Arg1: fffffadffd6cc003, memory referenced.
Arg2: 0000000000000000, value 0 = read operation, 1 = write operation.
Arg3: fffffadfc8aec075, If non-zero, the instruction address which
referenced the bad memory
address.
Thursday 6th September 2007
Arg1: fffffadffbee6003, memory referenced.
Arg2: 0000000000000000, value 0 = read operation, 1 = write operation.
Arg3: fffffadfc8b3c075, If non-zero, the instruction address which
referenced the bad memory
address.
Monday 20th August 2007
Arg1: fffffadffc344003, memory referenced.
Arg2: 0000000000000000, value 0 = read operation, 1 = write operation.
Arg3: fffffadfc8b5c075, If non-zero, the instruction address which
referenced the bad memory
address.
Sunday 29th July 2007
Arg1: fffffadfff8a3003, memory referenced.
Arg2: 0000000000000000, value 0 = read operation, 1 = write operation.
Arg3: fffffadfc8b8c075, If non-zero, the instruction address which
referenced the bad memory
address.
Thursday 19th July 2007
Arg1: fffffadffb58a003, memory referenced.
Arg2: 0000000000000000, value 0 = read operation, 1 = write operation.
Arg3: fffffadfc8aac075, If non-zero, the instruction address which
referenced the bad memory
address.
Thanks in advance to anyone who can help me make sense of this.
x64 standard SP2 keeps intermittently restarting from a stop 0x00000050 error
and logs only a "the previous system shutdown at x on x was unexpected" event
in the log. The WinDbg dump analysis (included below) always shows the error
to be a nonpaged pool expansion caused by our DAT tape driver (pvdatw2k),
which is current and supplied by Dell (there is no native x64 driver). Dell
have replaced the drive and cable and most of the major components in the
system, but the error keeps occuring. I sent them the crash dump analysis and
I have run my own diagnostics and swapped the RAM module positions and a
litany of other things (which the system passed) to try and get to the bottom
of this issue, but to no avail, and Dell have now closed this case. Whilst I
am able to run the dump analysis, I am not really able to make much useful
sense of it. I have been keeping records of the memory locations that show up
in the crash dump and have noticed that both memory referenced and the
instruction address referencing are very similar each time (see below also
for previous memory references). The shutdowns always occur when the server
is not doing anything e.g. at 3.00am on a weekend. We use ntbackup (and
therefore Volume Shadow Copy) to back the server up and I was thinking that
may be at fault, but the backup is usually finished before any crash occurs.
Has anyone else had VSC cause a nonpaged pool expansion? Is that what the
crash dump analysis is showing, or is it the driver? It's clear what memory
is at fault, but where is that freed memory exactly?
PAGE_FAULT_IN_NONPAGED_AREA (50)
Invalid system memory was referenced. This cannot be protected by try-except,
it must be protected by a Probe. Typically the address is just plain bad or
it
is pointing at freed memory.
Arguments:
Arg1: fffffadfd2986003, memory referenced.
Arg2: 0000000000000000, value 0 = read operation, 1 = write operation.
Arg3: fffffadfc8aec075, If non-zero, the instruction address which
referenced the bad memory
address.
Arg4: 0000000000000000, (reserved)
Debugging Details:
------------------
READ_ADDRESS: fffffadfd2986003 Nonpaged pool expansion
FAULTING_IP:
pvdatw2k+1075
fffffadf`c8aec075 410fb65103 movzx edx,byte ptr [r9+3]
MM_INTERNAL_CODE: 0
IMAGE_NAME: pvdatw2k.sys
DEBUG_FLR_IMAGE_TIMESTAMP: 412f435b
MODULE_NAME: pvdatw2k
FAULTING_MODULE: fffffadfc8aeb000 pvdatw2k
DEFAULT_BUCKET_ID: DRIVER_FAULT
BUGCHECK_STR: 0x50
PROCESS_NAME: System
CURRENT_IRQL: 1
TRAP_FRAME: fffffadfc8e8d550 -- (.trap fffffadfc8e8d550)
NOTE: The trap frame does not contain all registers.
Some register values may be zeroed.
rax=0000000000000000 rbx=0000000000000000 rcx=0000000000000000
rdx=0000000000000000 rsi=0000000000000003 rdi=fffff800011a98fd
rip=fffffadfc8aec075 rsp=fffffadfc8e8d6e0 rbp=0000000000000003
r8=0000000000000000 r9=fffffadfd2986000 r10=fffffadfd2986004
r11=0000000000000010 r12=0000000000000000 r13=0000000000000000
r14=0000000000000000 r15=0000000000000000
iopl=0 nv up ei pl nz na pe nc
pvdatw2k+0x1075:
fffffadf`c8aec075 410fb65103 movzx edx,byte ptr [r9+3]
ds:0001:fffffadf`d2986003=??
Resetting default scope
LAST_CONTROL_TRANSFER: from fffff800010a6072 to fffff8000102e890
STACK_TEXT:
fffffadf`c8e8d478 fffff800`010a6072 : 00000000`00000050 fffffadf`d2986003
00000000`00000000 fffffadf`c8e8d550 : nt!KeBugCheckEx
fffffadf`c8e8d480 fffff800`0102d459 : 00000000`00000000 00000000`00000000
00000000`00000000 00000000`00000000 : nt!MmAccessFault+0xa1f
fffffadf`c8e8d550 fffffadf`c8aec075 : fffffadf`d16e6b90 fffffadf`c8aec54a
fffffadf`cead30d8 00000000`00000030 : nt!KiPageFault+0x119
fffffadf`c8e8d6e0 fffffadf`c8aec54a : fffffadf`cead30d8 00000000`00000030
00000005`00000000 fffffadf`cd254e00 : pvdatw2k+0x1075
fffffadf`c8e8d6f0 fffffadf`c8aec6f1 : 00000000`00000000 00000000`00000000
fffffadf`c8e8d8e0 00000000`00000000 : pvdatw2k+0x154a
fffffadf`c8e8d760 fffffadf`c8aec7ed : 00000001`00060000 fffffadf`c8e8d7d8
fffffadf`c8e8d7d8 00000000`00000002 : pvdatw2k+0x16f1
fffffadf`c8e8d7d0 fffffadf`c8b82bb4 : fffffadf`cdd2b830 fffffadf`c8e8d8e0
00000005`00000000 00000000`00000144 : pvdatw2k+0x17ed
fffffadf`c8e8d830 fffffadf`c8390acd : 00000000`00000000 fffffadf`cbe93980
fffffadf`cdd2b830 fffffadf`cdd2b830 : TAPE!TapeDeviceControl+0x484
fffffadf`c8e8da00 fffff800`0128e1f8 : 00000000`00000001 fffffadf`d47890f0
fffffadf`cdd2b830 fffff800`0128e1e0 : CLASSPNP!ClasspFailurePredict+0xfd
fffffadf`c8e8dcd0 fffff800`010375ca : 00000000`00000000 fffffadf`d4789001
00000000`00000000 fffffadf`d47890f0 : nt!IopProcessWorkItem+0x17
fffffadf`c8e8dd00 fffff800`0124a972 : fffffadf`cead3040 00000000`00000080
fffffadf`cead3040 fffffadf`c8aa3680 : nt!ExpWorkerThread+0x13b
fffffadf`c8e8dd70 fffff800`01020226 : fffffadf`c8a9b180 fffffadf`cead3040
fffffadf`c8aa3680 00000000`00000000 : nt!PspSystemThreadStartup+0x3e
fffffadf`c8e8ddd0 00000000`00000000 : 00000000`00000000 00000000`00000000
00000000`00000000 00000000`00000000 : nt!KxStartSystemThread+0x16
STACK_COMMAND: kb
FOLLOWUP_IP:
pvdatw2k+1075
fffffadf`c8aec075 410fb65103 movzx edx,byte ptr [r9+3]
SYMBOL_STACK_INDEX: 3
SYMBOL_NAME: pvdatw2k+1075
FOLLOWUP_NAME: MachineOwner
FAILURE_BUCKET_ID: X64_0x50_pvdatw2k+1075
BUCKET_ID: X64_0x50_pvdatw2k+1075
Followup: MachineOwner
Previous memory references -
Thursday 15th November 2007
Arg1: fffffadffe7f7003, memory referenced.
Arg2: 0000000000000000, value 0 = read operation, 1 = write operation.
Arg3: fffffadfc8b6c075, If non-zero, the instruction address which
referenced the bad memory
address.
Friday 14th September 2007
Arg1: fffffadffd6cc003, memory referenced.
Arg2: 0000000000000000, value 0 = read operation, 1 = write operation.
Arg3: fffffadfc8aec075, If non-zero, the instruction address which
referenced the bad memory
address.
Thursday 6th September 2007
Arg1: fffffadffbee6003, memory referenced.
Arg2: 0000000000000000, value 0 = read operation, 1 = write operation.
Arg3: fffffadfc8b3c075, If non-zero, the instruction address which
referenced the bad memory
address.
Monday 20th August 2007
Arg1: fffffadffc344003, memory referenced.
Arg2: 0000000000000000, value 0 = read operation, 1 = write operation.
Arg3: fffffadfc8b5c075, If non-zero, the instruction address which
referenced the bad memory
address.
Sunday 29th July 2007
Arg1: fffffadfff8a3003, memory referenced.
Arg2: 0000000000000000, value 0 = read operation, 1 = write operation.
Arg3: fffffadfc8b8c075, If non-zero, the instruction address which
referenced the bad memory
address.
Thursday 19th July 2007
Arg1: fffffadffb58a003, memory referenced.
Arg2: 0000000000000000, value 0 = read operation, 1 = write operation.
Arg3: fffffadfc8aac075, If non-zero, the instruction address which
referenced the bad memory
address.
Thanks in advance to anyone who can help me make sense of this.