How to prevent VC++ compiler from re-using local variable stack space

[LonC]

I have encountered a bug in which the VC++2017 compiled the code wrong. What I am now struggling with is how to work around it to prevent the compiler from making this mistake. It appears that the compiler "deduces" that a local early in the function is no longer used, so that it can reuse that stack space for local that is used later. The deduction is flawed, because the code stored an address to that data. So my question is whether there is any way to tell the optimizing compiler that it can't re-use the stack space used by that variable?

To illustrate the issue, I'll show the relevant lines of the code here -- all in the same function:

vector<DataSorter*> sorters;
...
DataSorter heightSorter(...);
...
sorters[1] = &heightSorter;
...
Pen pen(Color::Black);
...
sorters[1]->Sort(...);




The disassembly for the 2nd line shows that the compiler positioned heightSorter at address [rbp+0E8h]:

DataSorter heightSorter(gr, absHeightKeys, 2);
000000014040D111 mov dword ptr [rbp+0E8h],2 // size_ is first member in DataSorter
000000014040D11B lea rdi,[rbp+210h]
000000014040D122 mov qword ptr [rbp+0F0h],rdi
000000014040D129 mov qword ptr [rbp+0F8h],rsi


The key info is the 1st disassembly line which shows us where the compiler located heighSorter on the stack.

Later the Pen constructor is in-lined. The inline'd code is

Pen(IN const Color& color,
IN REAL width = 1.0f)
{
Unit unit = UnitWorld;
nativePen = NULL;
lastResult = DllExports::GdipCreatePen1(color.GetValue(),
width, unit, &nativePen);
}


which the following disassembly (my annotations):

000000014040DCB3 xor ebx,ebx // nativePen=NULL
000000014040DCB5 mov qword ptr [rbp+0E8h],rbx
000000014040DCBC lea r9,[rbp+0E8h] // &nativePen
000000014040DCC3 xor r8d,r8d // unit = UnitWorld
000000014040DCC6 movss xmm1,dword ptr [__real@3f800000 (0140B7CBE8h)]
000000014040DCCE mov ecx,0FF000000h // color = Color::Black
000000014040DCD3 call qword ptr [__imp_GdipCreatePen1 (0140A2CC30h)]
000000014040DCD9 mov dword ptr [rbp+0F0h],eax



Here the 3rd line reveals that the compiler has located Pen at location [rbp+0E8h] -- the same location used for heightSorter earlier. This clobbers the contents of heightSorter, so that when the 5th line is executed, it gets and access violation.

After the 3rd line (sorters[1] = &heightSorter), the name heightSorter is not used again, so my conjecture is that the compiler concluded that the stack space used by that variable can now be reused. But because the code used its address, that assumption was flawed. This address is only used in this same function, this isn't a case of returning an address from the stack, which is obviously flawed.

I've left out lots of logic -- the function in question is very long (yes, too long in IMHO). There is logic in there that selects which sorters get used in the specific situation, so although I extracted the relevant part, there is a higher logic to what it is doing.

I know it would be great to isolate this into a bug report. Easier said that done, but perhaps something I'll eventually be able to do. Right now, I am just looking for a work around.

So to emphasize, the question is: Is there a way to tell the compiler that it cannot reuse the space for a local variable on the stack for another variable after it thinks the first variable is no longer used? VC++2017 compiler with /std:c++17

Thanks!

Continue reading...