LTCG incorrectly optimizes accesses to static pointers > 2GB

[EDN Admin]

This appears to be an encoding error, it only appears with LTCG.
The bytes for the encoding are below. The value of the constant was 0x00000000ffe8b00e
StrPtr = (CHAR8 *)(UINT64)(0xffe8b000UL + 0x08UL);
while (!*StrPtr) {StrPtr++;}
Here is what is generated
0000000000063B24: 44 38 B0 0E B0 E8 FF cmp byte ptr [rax-174FF2h],r14b
44 = extend the R field in the mod R/M byte.
38 = compare
B0 = modrm = [rax+disp32], and r = r14
Disp32 = ffe8b0038

Notice that the constant value is encoded directly as a displacement. But, in 64-bit modes, displacements are sign-extended (see 2.2.1.3 in Vol 2 of the Intel Ref) to 64-bits, which means it acts, -effectively, like a subtraction. For an unsigned integer, it must be encoded with a register. It would work for addresses < 2GB. In fact, a few instructions before, you can see:
0000000000063B14: BA 0E B0 E8 FF mov edx,0FFE8B00Eh
0000000000063B19: 41 8B C6 mov eax,r14d
0000000000063B1C: 44 38 32 cmp byte ptr [rdx],r14b
0000000000063B1F: 74 0C je 0000000000063B2D
0000000000063B24: 44 38 B0 0E B0 E8 cmp byte ptr [rax-174FF2h],r14b
0000000000063B24: 44 38 B0 0E B0 E8 cmp byte ptr [rax-174FF2h],r14b
FF
0000000000063B2B: 75 F4 jne 0000000000063B21
0000000000063B2D: 48 FF C0 inc rax



As if it was preparing for exactly that. So this would have been better encoded as cmp byte ptr [rax + rdx], r14b
I have verified this issue exists in (at least) vs2008 sp1, vs2010 AND vs2012sP2
Tim

View the full article