A call to a method in an out of process COM (ATL) server blocks when called from a C# method

[Sagar R. Kapadia]

I have a 64 bit [and also a separate 32 bit] out of process COM server written in ATL, named ProcessMonitor, the purpose of which is to inject a DLL into other processes for monitoring keystrokes and mouse movements. I need to be able to shut down this server upon certain events. I send a WM_QUIT message to its window and handle the shut down in the message loop of the window. The method which does that is as follows

public void ShutdownMonitor()
{
try
{
if (processMonitorX64 != null)
{
logger.Info("Shutting down ProcessMonitorX64");
// Thread thread = new Thread(()=> processMonitorX64.Shutdown());
// thread.Start();
processMonitorX64.Shutdown();
logger.Info("Shut down ProcessMonitorX64");
}
else
{
logger.Info("ProcessMonitorX64 was null");
}
}
catch (Exception e)
{
logger.Error("Exception shutting down ProcessMonitorX64");
logger.Error(JsonConvert.SerializeObject(e));
}
try
{
if (processMonitorX86 != null)
{
logger.Info("Shutting down ProcessMonitorX86");
processMonitorX86.Shutdown();
logger.Info("Shut down ProcessMonitorX86");
}
else
{
logger.Info("ProcessMonitorX86 was null");
}
}
catch (Exception e)
{
logger.Error("Exception shutting down ProcessMonitorX86");
logger.Error(JsonConvert.SerializeObject(e));
}
}


It basically checks if the object has been instantiated and if so, attempts to shut it down. The Shutdown method of ProcessMonitor is

STDMETHODIMP CProcessMonitorX64::Shutdown()
{
SendMessage(hWndMonitor, WM_COMMAND, MAKEWPARAM(ID_MENU_EXIT, 0), NULL);
return S_OK;
}




This works well enough in a console C# app I use for testing. But the actual project is larger, and uses a plugin style design. [Plugins are loaded at runtime based on a configuration file] .This code is called from a plugin [in a separate app domain]. When I stepped through the code in the debugger, I found that the execution stops when this method is called. I have enabled native code debugging, and I also attach the process [the COM server] before calling the shutdown method on it. I have also placed a break point on the Shutdown method, but it is never reached.

However, this process also has a system tray icon, with an exit menu. When I click on the exit menu in the system tray icon, the break point is hit and the process shuts down properly

Also, if I shut down Process Monitor by clicking on the exit menu after the execution has blocked, there is a COM exception in the C# code, which says "An RPC called failed".

I cannot figure out why the execution blocks when I call the method. Other methods called on ProcessMonitor do not block,and work properly [I call a method to inject a hook into another process when the plugin is loaded, and that gets done properly].

Please help.

Thanks

Continue reading...