Some thought on the making of enbFreePlugins, a enbseries dll plugin that hooks into LoadLibrary and unload all other .dllplugins when enbseries d3d11.dll is unloaded.
The reason for the mod, is because my .dllplugin are having problems with GTA5. And after further digging, it appears that GTA5 unloads then reloads d3d11.dll during startup.
[2023-03-22 17:26:35.753+08:00][24924][info] enbFreePlugins createad by kingeric1992 (Mar.22.2023)
[2023-03-22 17:26:35.754+08:00][24924][info] proc attach E:\Grand Theft Auto V\enbseries\enbFreePlugins.log
[2023-03-22 17:26:35.843+08:00][24924][info] ...Callback Linked....ENB:0x0000000180000000, callback:0x00007FFB79C576D8
[2023-03-22 17:26:35.843+08:00][24924][info] MH: hooked
; ENBseries loading system d3d11
[2023-03-22 17:26:35.844+08:00][24924][info] LoadLibrary [0x0000000000000000] -> [0x00007FFC60BF0000] C:\WINDOWS\system32\d3d11.dll
; ENBseries loading enbhelper (failed)
[2023-03-22 17:26:35.942+08:00][24924][info] LoadLibrary [0x0000000000000000] -> [0x0000000000000000] enbseries\enbhelper.dll
[2023-03-22 17:26:35.944+08:00][24924][info] LoadLibrary [0x0000000000000000] -> [0x0000000000000000] enbseries\enbhelper.dll
[2023-03-22 17:26:35.946+08:00][24924][info] LoadLibrary [0x0000000000000000] -> [0x0000000000000000] enbhelper.dll
; GTA5 free directx
[2023-03-22 17:26:35.946+08:00][24924][info] FreeLibrary 0x00007FFC63570000 C:\WINDOWS\SYSTEM32\dxgi.dll
[2023-03-22 17:26:35.946+08:00][24924][info] FreeLibrary 0x0000000180000000 E:\Grand Theft Auto V\D3D11.DLL
; ENBseries free libraries
[2023-03-22 17:26:35.946+08:00][24924][info] FreeLibrary 0x00007FFB79F70000 E:\Grand Theft Auto V\d3dcompiler_46e.dll
[2023-03-22 17:26:35.947+08:00][24924][info] FreeLibrary 0x00007FFC60BF0000 C:\WINDOWS\system32\d3d11.dll
; GTA5 reload directx
[2023-03-22 17:26:36.329+08:00][24924][info] LoadLibrary [0x0000000000000000] -> [0x0000000180000000] D3D11.DLL
[2023-03-22 17:26:36.329+08:00][24924][info] LoadLibrary [0x0000000000000000] -> [0x00007FFC63570000] dxgi.dll
; GTA5 prepare to create device
[2023-03-22 17:26:36.555+08:00][24924][info] LoadLibrary [0x0000000180000000] -> [0x0000000180000000] D3D11.DLL
[2023-03-22 17:26:36.555+08:00][24924][info] LoadLibrary [0x0000000000000000] -> [0x00007FFC444B0000] D3D10_1.DLL
; ENBseries loading compiler
[2023-03-22 17:26:36.557+08:00][24924][info] LoadLibrary [0x0000000000000000] -> [0x0000000000000000] d3dcompiler_46f.dll
[2023-03-22 17:26:36.559+08:00][24924][info] LoadLibrary [0x0000000000000000] -> [0x0000000000000000] d3dcompiler_46f
[2023-03-22 17:26:36.560+08:00][24924][info] LoadLibrary [0x0000000000000000] -> [0x00007FFB79460000] d3dcompiler_46e.dll
; ENBseries loading plugin dlls
[2023-03-22 17:26:36.560+08:00][24924][info] Ignore module: E:\Grand Theft Auto V\\enbseries\enbFreePlugins.dllplugin
[2023-03-22 17:26:36.561+08:00][24924][info] ...Callback Linked....ENB:0x0000000180000000, callback:0x00007FFB79C576D8
; ENBseries loading system d3d11
[2023-03-22 17:26:36.562+08:00][24924][info] LoadLibrary [0x0000000000000000] -> [0x00007FFC60BF0000] C:\WINDOWS\system32\d3d11.dll
As shown in above trace log, the enbseries d3d11.dll would get unmapped from game through FreeLibrary calls, while dllplugins still remains in game memory.
Which means two things: the plugin’s callback that was registered to enbseries when it’s first loaded is invalidated, and secondly, plugin’s DllMain PROC_ATTACH won’t get invoked again when enbseries loads dllplugins the second time. So, the result is that the dllplugin ends up not receving any enb events if it relies on PROC_ATTACH to initialize and register for enb callbacks.
There’s obviously multiple ways to mitigate this, but the solution that first came to my mind is that, well, how about I just hook to LoadLibrary and unmap .dllplugins there so that it would get PROC_ATTACH later when loaded. hence:
HMODULE LoadLibraryA_hook(LPCSTR cstr)
{
if (auto l = strnlen_s(cstr, MAX_PATH); l > 10 && !strcmp(cstr + l - 10, ".dllplugin"))
{
auto h = GetModuleHandleA(cstr);
for (int i = 0; i < 10 && FreeLibrary(h);)
_INFO("FreeLibrary({})", ++i);
}
return LoadLibraryA_orig(cstr);
}It loops FreeLibrary() a coupe of times to ensure the plugin is really unmapped from memory. If all goes well, .dllplugins should now reinit properly for my mods.
However, things did not go well. My other .dllplugin mod crashes the game at first attempt, then it refuse to unload after that.
Fine, the crash is easy fix. The .dllplugin in questioned uses MinHook to hook DirectInput but I didn’t configure it to remove itself and restore original code during exit, meaning the hooked function would access invalid address once the dllplugin is unmapped from memory. A simple MinHook uninitialization at dll PROC_DETACH fixed the problem.
The question now comes to why FreeLibrary wouldn’t unload the dllpugin. Eventually I came across this discussion why-windows-generates-an-ignorefreelibrary and the name “IgnoreFreeLibrary” does ring some bell to what’s happening to me right now. And voila! The reg key does actually present for GTA5.exe and my plugin.
Removed the entry and all is good to go!