Build fails in a Windows Server Core Hyper-V Container if source is on a named volume

  • Thread starter Thread starter Rick Altherr
  • Start date Start date
R

Rick Altherr

Guest
I'm setting up a CI process using Windows containers. I've created a Docker image based on microsoft/windowsservercore:1803 and installed Visual Studo 2017 BuildTools with the needed C# and C++ components. The CI system uses a separate container to checkout the source so the build process is roughly:

  1. docker volume create mysource
  2. docker run --rm -it -v mysource:C:\build registry.gitlab.com/andrewleech/gitlab-runner/gitlab-runner-helper:win-x86_64-latest
  3. Inside that container, use git to clone source into C:\build
  4. docker run --rm -it --isolation=hyperv --cpu-count=2 --memory=8gb <visual-studio-docker-image>
  5. msbuild C:\build\<proj> /t:all /m /nr:false

The build goes through a few steps before hitting a C++ compile and fails with the following:


ClCompile:
C:\Program Files (x86)\Microsoft Visual Studio 14.0\VC\bin\x86_amd64\CL.exe /c /Zi /nologo /W3 /WX- /MP /O2 /D _WINDOWS /D AMD64 /D NDEBUG /D _USRDLL /D _WINDLL /D _UNICODE /D UNICODE /Gm- /EHsc /MD /GS /fp:precise /Zc:wchar_t
/Zc:forScope /Zc:inline /Yc"stdafx.h" /Fp"x64\Release\PrevotyHookingLib_x64.pch" /Fo"x64\Release\\" /Fd"x64\Release\vc140.pdb" /Gd /TP /errorReport:queue stdafx.cpp
stdafx.cpp
C:\Program Files (x86)\Microsoft Visual Studio 14.0\VC\bin\x86_amd64\CL.exe /c /Zi /nologo /W3 /WX- /MP /O2 /D _WINDOWS /D AMD64 /D NDEBUG /D _USRDLL /D _WINDLL /D _UNICODE /D UNICODE /Gm- /EHsc /MD /GS /fp:precise /Zc:wchar_t
/Zc:forScope /Zc:inline /Fo"x64\Release\\" /Fd"x64\Release\vc140.pdb" /Gd /TP /wd4091 /wd4099 /errorReport:queue corhlpr.cpp
corhlpr.cpp
C:\Program Files (x86)\Microsoft Visual Studio 14.0\VC\bin\x86_amd64\CL.exe /c /Zi /nologo /W3 /WX- /MP /O2 /D _WINDOWS /D AMD64 /D NDEBUG /D _USRDLL /D _WINDLL /D _UNICODE /D UNICODE /Gm- /EHsc /MD /GS /fp:precise /Zc:wchar_t
/Zc:forScope /Zc:inline /Yu"stdafx.h" /Fp"x64\Release\PrevotyHookingLib_x64.pch" /Fo"x64\Release\\" /Fd"x64\Release\vc140.pdb" /Gd /TP /errorReport:queue PrevotyHookingLib.cpp ILMethod.cpp ILParser.cpp ManagedAssembly.cpp Manag
edClass.cpp ManagedMethod.cpp ManagedModule.cpp Profiler.cpp ProfilerImpl.cpp
PrevotyHookingLib.cpp
ILMethod.cpp
11>ILMethod.cpp(1): fatal error C1041: cannot open program database 'C:\build\gozer\csharp\hookinglib\x64\Release\vc140.pdb'; if multiple CL.EXE write to the same .PDB file, please use /FS [C:\build\gozer\csharp\hookinglib\hookingli
b.vcxproj]
ILParser.cpp
11>ILParser.cpp(1): fatal error C1041: cannot open program database 'C:\build\gozer\csharp\hookinglib\x64\Release\vc140.pdb'; if multiple CL.EXE write to the same .PDB file, please use /FS [C:\build\gozer\csharp\hookinglib\hookingli
b.vcxproj]
ManagedAssembly.cpp
11>ManagedAssembly.cpp(1): fatal error C1041: cannot open program database 'C:\build\gozer\csharp\hookinglib\x64\Release\vc140.pdb'; if multiple CL.EXE write to the same .PDB file, please use /FS [C:\build\gozer\csharp\hookinglib\ho
okinglib.vcxproj]
ManagedClass.cpp
11>ManagedClass.cpp(1): fatal error C1041: cannot open program database 'C:\build\gozer\csharp\hookinglib\x64\Release\vc140.pdb'; if multiple CL.EXE write to the same .PDB file, please use /FS [C:\build\gozer\csharp\hookinglib\hooki
nglib.vcxproj]
ManagedMethod.cpp
11>ManagedMethod.cpp(1): fatal error C1041: cannot open program database 'C:\build\gozer\csharp\hookinglib\x64\Release\vc140.pdb'; if multiple CL.EXE write to the same .PDB file, please use /FS [C:\build\gozer\csharp\hookinglib\hook
inglib.vcxproj]
ManagedModule.cpp
11>ManagedModule.cpp(1): fatal error C1041: cannot open program database 'C:\build\gozer\csharp\hookinglib\x64\Release\vc140.pdb'; if multiple CL.EXE write to the same .PDB file, please use /FS [C:\build\gozer\csharp\hookinglib\hook
inglib.vcxproj]
Profiler.cpp
11>Profiler.cpp(3): fatal error C1041: cannot open program database 'C:\build\gozer\csharp\hookinglib\x64\Release\vc140.pdb'; if multiple CL.EXE write to the same .PDB file, please use /FS [C:\build\gozer\csharp\hookinglib\hookingli
b.vcxproj]
ProfilerImpl.cpp
11>ProfilerImpl.cpp(1): fatal error C1041: cannot open program database 'C:\build\gozer\csharp\hookinglib\x64\Release\vc140.pdb'; if multiple CL.EXE write to the same .PDB file, please use /FS [C:\build\gozer\csharp\hookinglib\hooki
nglib.vcxproj]
11>Done Building Project "C:\build\gozer\csharp\hookinglib\hookinglib.vcxproj" (build target(s)) -- FAILED.


Note that /MP is being passed which, according to -FS (Force Synchronous PDB Writes), enables /FS by default.

If I omit the named volume and clone the source directly into the VS container, the build works perfectly fine:

PS C:\> cp -Recurse .\build\gozer\ gozer
PS C:\> cd .\gozer\
PS C:\gozer> msbuild .\csharp\csharp.proj /t:all /m /nr:false
Microsoft (R) Build Engine version 15.8.169+g1ccb72aefa for .NET Framework
Copyright (C) Microsoft Corporation. All rights reserved.

Build started 9/27/2018 12:18:12 PM.
<...snip...>
ClCompile:
C:\Program Files (x86)\Microsoft Visual Studio 14.0\VC\bin\CL.exe /c /Zi /nologo /W3 /WX- /MP /O2 /Oy- /D WIN32 /D _WINDOWS /D NDEBUG /D _USRDLL /D _WINDLL /D _UNICODE /D UNICODE /Gm- /EHsc /MD /GS /fp:precise /Zc:wchar_t /Zc:f
orScope /Zc:inline /Yc"stdafx.h" /Fp"Release\PrevotyHookingLib_x86.pch" /Fo"Release\\" /Fd"Release\vc140.pdb" /Gd /TP /analyze- /errorReport:queue stdafx.cpp
stdafx.cpp
C:\Program Files (x86)\Microsoft Visual Studio 14.0\VC\bin\CL.exe /c /Zi /nologo /W3 /WX- /MP /O2 /Oy- /D WIN32 /D _WINDOWS /D NDEBUG /D _USRDLL /D _WINDLL /D _UNICODE /D UNICODE /Gm- /EHsc /MD /GS /fp:precise /Zc:wchar_t /Zc:f
orScope /Zc:inline /Fo"Release\\" /Fd"Release\vc140.pdb" /Gd /TP /wd4091 /wd4099 /analyze- /errorReport:queue corhlpr.cpp
corhlpr.cpp
C:\Program Files (x86)\Microsoft Visual Studio 14.0\VC\bin\CL.exe /c /Zi /nologo /W3 /WX- /MP /O2 /Oy- /D WIN32 /D _WINDOWS /D NDEBUG /D _USRDLL /D _WINDLL /D _UNICODE /D UNICODE /Gm- /EHsc /MD /GS /fp:precise /Zc:wchar_t /Zc:f
orScope /Zc:inline /Yu"stdafx.h" /Fp"Release\PrevotyHookingLib_x86.pch" /Fo"Release\\" /Fd"Release\vc140.pdb" /Gd /TP /analyze- /errorReport:queue PrevotyHookingLib.cpp ILMethod.cpp ILParser.cpp ManagedAssembly.cpp ManagedClass
.cpp ManagedMethod.cpp ManagedModule.cpp Profiler.cpp ProfilerImpl.cpp
PrevotyHookingLib.cpp
ILMethod.cpp
ILParser.cpp
ManagedAssembly.cpp
ManagedClass.cpp
ManagedMethod.cpp
ManagedModule.cpp
Profiler.cpp
ProfilerImpl.cpp

Additionally, keeping the source on a named volume while using --isolation=process on Windows Server 1803 works fine. So it seems something about using Hyper-V containers with named volumes triggers this problem.

As a last experiment, I tried restricting cl.exe to a single core while allowing msbuild to run parallel tasks:

PS C:\build\gozer> msbuild .\csharp\csharp.proj /t:all /m /p:CL_MPCount=1 /nr:false
Microsoft (R) Build Engine version 15.8.169+g1ccb72aefa for .NET Framework
Copyright (C) Microsoft Corporation. All rights reserved.

Build started 9/27/2018 12:42:15 PM.

ClCompile:
C:\Program Files (x86)\Microsoft Visual Studio 14.0\VC\bin\CL.exe /c /Zi /nologo /W3 /WX- /MP1 /O2 /Oy- /D WIN32 /D _WINDOWS /D NDEBUG /D _USRDLL /D _WINDLL /D _UNICODE /D UNICODE /Gm- /EHsc /MD /GS /fp:precise /Zc:wchar_t /Zc:
forScope /Zc:inline /Yc"stdafx.h" /Fp"Release\PrevotyHookingLib_x86.pch" /Fo"Release\\" /Fd"Release\vc140.pdb" /Gd /TP /analyze- /errorReport:queue stdafx.cpp
stdafx.cpp
C:\Program Files (x86)\Microsoft Visual Studio 14.0\VC\bin\CL.exe /c /Zi /nologo /W3 /WX- /MP1 /O2 /Oy- /D WIN32 /D _WINDOWS /D NDEBUG /D _USRDLL /D _WINDLL /D _UNICODE /D UNICODE /Gm- /EHsc /MD /GS /fp:precise /Zc:wchar_t /Zc:
forScope /Zc:inline /Fo"Release\\" /Fd"Release\vc140.pdb" /Gd /TP /wd4091 /wd4099 /analyze- /errorReport:queue corhlpr.cpp
corhlpr.cpp
C:\Program Files (x86)\Microsoft Visual Studio 14.0\VC\bin\CL.exe /c /Zi /nologo /W3 /WX- /MP1 /O2 /Oy- /D WIN32 /D _WINDOWS /D NDEBUG /D _USRDLL /D _WINDLL /D _UNICODE /D UNICODE /Gm- /EHsc /MD /GS /fp:precise /Zc:wchar_t /Zc:
forScope /Zc:inline /Yu"stdafx.h" /Fp"Release\PrevotyHookingLib_x86.pch" /Fo"Release\\" /Fd"Release\vc140.pdb" /Gd /TP /analyze- /errorReport:queue PrevotyHookingLib.cpp ILMethod.cpp ILParser.cpp ManagedAssembly.cpp ManagedClas
s.cpp ManagedMethod.cpp ManagedModule.cpp Profiler.cpp ProfilerImpl.cpp
PrevotyHookingLib.cpp
ILMethod.cpp
ILParser.cpp
ManagedAssembly.cpp
ManagedClass.cpp
ManagedMethod.cpp
ManagedModule.cpp
Profiler.cpp
ProfilerImpl.cpp
11>ProfilerImpl.cpp(173): warning C4018: '<': signed/unsigned mismatch [C:\build\gozer\csharp\hookinglib\hookinglib.vcxproj]
Generating Code...
C:\Program Files (x86)\Microsoft Visual Studio 14.0\VC\bin\CL.exe /c /Zi /nologo /W3 /WX- /MP1 /O2 /Oy- /D WIN32 /D _WINDOWS /D NDEBUG /D _USRDLL /D _WINDLL /D _UNICODE /D UNICODE /Gm- /EHsc /MD /GS /fp:precise /Zc:wchar_t /Zc:
forScope /Zc:inline /Fo"Release\\" /Fd"Release\vc140.pdb" /Gd /TC /analyze- /errorReport:queue PrevotyHookingLib_i.c
PrevotyHookingLib_i.c
ResourceCompile:
C:\Program Files (x86)\Windows Kits\8.1\bin\x86\rc.exe /D NDEBUG /D _UNICODE /D UNICODE /l"0x0409" /IRelease\ /nologo /fo"Release\PrevotyHookingLib.res" PrevotyHookingLib.rc
Link:
C:\Program Files (x86)\Microsoft Visual Studio 14.0\VC\bin\link.exe /ERRORREPORT:QUEUE /OUT:"..\bin\PrevotyHookingLib\Release\Win32\PrevotyHookingLib_x86.dll" /INCREMENTAL:NO /NOLOGO corguids.lib kernel32.lib user32.lib gdi32.l
ib winspool.lib comdlg32.lib advapi32.lib shell32.lib ole32.lib oleaut32.lib uuid.lib odbc32.lib odbccp32.lib /DEF:".\PrevotyHookingLib.def" /MANIFEST /MANIFESTUAC:"level='asInvoker' uiAccess='false'" /manifest:embed /DEBUG /PD
B:"..\bin\PrevotyHookingLib\Release\Win32\PrevotyHookingLib_x86.pdb" /SUBSYSTEM:WINDOWS /OPT:REF /OPT:ICF /TLBID:1 /DYNAMICBASE /NXCOMPAT /MACHINE:X86 /SAFESEH /DLL Release\PrevotyHookingLib.res
Release\corhlpr.obj
Release\PrevotyHookingLib.obj
Release\ILMethod.obj
Release\ILParser.obj
Release\ManagedAssembly.obj
Release\ManagedClass.obj
Release\ManagedMethod.obj
Release\ManagedModule.obj
Release\PrevotyHookingLib_i.obj
Release\Profiler.obj
Release\ProfilerImpl.obj
Release\stdafx.obj
Creating library ..\bin\PrevotyHookingLib\Release\Win32\PrevotyHookingLib_x86.lib and object ..\bin\PrevotyHookingLib\Release\Win32\PrevotyHookingLib_x86.exp
11>corguids.lib(corprof_i.obj) : warning LNK4099: PDB 'corguids.pdb' was not found with 'corguids.lib(corprof_i.obj)' or at 'C:\build\gozer\csharp\bin\PrevotyHookingLib\Release\Win32\corguids.pdb'; linking object as if no debug info
[C:\build\gozer\csharp\hookinglib\hookinglib.vcxproj]
11>LINK : fatal error LNK1318: Unexpected PDB error; RPC (23) '(0x000006E7)' [C:\build\gozer\csharp\hookinglib\hookinglib.vcxproj]
11>Done Building Project "C:\build\gozer\csharp\hookinglib\hookinglib.vcxproj" (build target(s)) -- FAILED.


So it seems that cl.exe and link.exe are having trouble with mspdbsrv.exe under Hyper-V containers when building on a named volume. Note that mspdbsrv.exe _is_ running after the build fails:

PS C:\build\gozer> Get-Process mspdbsrv

Handles NPM(K) PM(K) WS(K) CPU(s) Id SI ProcessName
------- ------ ----- ----- ------ -- -- -----------
68 7 1776 5244 0.23 1408 1 mspdbsrv

Continue reading...
 
Back
Top