Unhooking a hook set with SetWindowsHookEx more than once crashes the hooked process

[Sagar R. Kapadia]

I am trying to hook the keyboard and mouse events of a target application.
I followed the SO question How to hook external process with SetWindowsHookEx and WH_KEYBOARD, and the hooks are installed and uninstalled correctly the first time. However, after I uninstall the hook once and then install it again, trying to uninstall the hook the second time crashes the target application. The purpose of the hook is to monitor application idle time, so that I can do some tasks when the application is idle.
I apologize for the length of the question, but I tried to put in all the details that might help

I need to be able to install and uninstall the hooks based on menu commands from a system tray icon. I have a console application [HookApp] which calls the install and uninstall methods in a DLL [HookDLL]. The console application also creates a window to handle menu events. I use the thread of the window to actually install and uninstall the hooks, because the same thread that installed the hook must uninstall it. The Console and the window both must be invisible. Only the system tray icon and its associated menu must be visible.
I start the hooking application from the command line with the parameters
i. ProcessId of the target process
ii. Process Name
iii. Idle time [in seconds] -Idle time before starting the action in the DLL
I am wondering if the timer I start in the HookProcs is responsible for the crash.

Event Viewer Logs
Faulting application name: notepad.exe, version: 10.0.17134.1, time stamp: 0x9d4727c2
Faulting module name: HookDLL_x64.dll_unloaded, version: 0.0.0.0, time stamp: 0x5c31aabd
Exception code: 0xc0000005
Fault offset: 0x00000000000ba505
Faulting process id: 0x2bac

The exception code seems to suggest a memory access violation.


I have tried the following
I.
a. Call the SendMessageTimeout API with HWND_BROADCAST from HookApp
b. Call the SendMessageTimeout API with HWND_BROADCAST from HookDLL
based on Unloading DLL from all processes after unhooking global CBT hook

II.
a. call FreeLibraryAndExitThread API in the DLLMain method on DLL_PROCESS_DETACH AND DLL_THREAD_DETACH -both together and singly
based on Unloading DLL crashes exe - Programming


III. Made the HookApp console and the window visible and hidden to see if it makes any difference

IV. Was unable to try the following because of 64 bit architecture
Unload an injected dll or thread from a process

Other Links I referred to
a. Several programs crash when unhooking with UnhookWindowsHookEx()
b. How to correctly use SetWindowsHookEx & CallNextHookEx
c. Unloading an Injected DLL
d. FreeLibraryAndExitThread crashes program when unloading injected DLL

//Code in the DLL
extern "C" __declspec(dllexport) void install(unsigned long threadID,int _nIdleTime) {
nIdleTime = _nIdleTime;
Log(L"install proc called from app: _nIdleTime", _nIdleTime);
hhKeyboard = SetWindowsHookEx(WH_KEYBOARD, KeyboardProc, hinst, threadID);
hhMouse = SetWindowsHookEx(WH_MOUSE, MouseProc, hinst, threadID);
logger->Log("Install");
logger->Log("Keyboard", (LONG)hhKeyboard);
logger->Log("Mouse", (LONG)hhMouse);

}

//Uninstall the dll from the Target Process
DWORD WINAPI UnInjectDLLFromTarget() {
uninstall();
//DWORD_PTR dwResult = 0;
//SendMessageTimeout(HWND_BROADCAST, WM_NULL, 0, 0, SMTO_ABORTIFHUNG | SMTO_NOTIMEOUTIFNOTHUNG, 1000, &dwResult);
return TRUE;
}

LRESULT CALLBACK KeyboardProc(int code, WPARAM wParam, LPARAM lParam) {
if (code < 0) {

return CallNextHookEx(0, code, wParam, lParam);
}
StartTimer();
Beep(1000, 20);


return CallNextHookEx(hhKeyboard, code, wParam, lParam);
}

LRESULT CALLBACK MouseProc(int code, WPARAM wParam, LPARAM lParam) {
if (code < 0) {

return CallNextHookEx(0, code, wParam, lParam);
}
// StartTimer();
//Beep(1000, 20);

return CallNextHookEx(hhMouse, code, wParam, lParam);
}
BOOL WINAPI DllMain(__in HINSTANCE hinstDLL, __in DWORD fdwReason, __in LPVOID lpvReserved) {


if (fdwReason == DLL_PROCESS_DETACH || fdwReason==DLL_THREAD_DETACH) {
FreeLibraryAndExitThread(hinstDLL, 0);
return 0;
}
}

//Code in the Application [HookApp]
Similar to [But not exactly the same - It is a bit more involved because of the need to create the message pump and the menu event handling window]
HINSTANCE hinst = LoadLibrary(_T("MyDLL.dll"));

if (hinst) {
typedef void (*Install)(unsigned long);
typedef void (*Uninstall)();

Install install = (Install) GetProcAddress(hinst, "install");
Uninstall uninstall = (Uninstall) GetProcAddress(hinst, "uninstall");

install(threadID);

Sleep(20000);

uninstall();
}

1. SendMessageTimeout in the DLL [HookDLL] method Uninstall immediately crashes the application
2. SendMessageTimeout in the application [HookApp] doesnt appear to do anything useful.
3. FreeLibraryAndExitThread does not appear to do anything useful.
However, with 2 and 3, I think I am able to install the hook for the second time. Without them, the first unhook crashes the application.
4. I was unable to try the suggestion using assembly language inline because of 64 bit architecture.

Please help.

Continue reading...