Mapping DLL that uses OpenSSL

[risky]

I'm currently building software using ASIO as a network library and its SSL counterpart. It has worked perfectly, but I've recently tried to initialize the context in a manually mapped DLL and SSL_CTX_new is failing when ASIO initializes it's OpenSSL components. The odd thing is that no error can be resolved from SSL_CTX_new's failure, it simply returns error code 0.

So, I dug into the code and I believe I've dumbed down the failure to OPENSSL_init_crypto. The only problem I can think of is TLS (Thread Local Storage). I've handled TLS manually, and TLS seems to be fixed, but the problem persists.

Note the code works perfectly when the DLL is loaded via LoadLibrary, but LoadLibrary can not be used in distributed builds.

For further context, the OpenSSL version used is 1.1.1c. There must be major discrepancies if LoadLibrary can map the DLL fine and my mapping code can't, I just can't spot what it would be based off what LdrpLoadDll runs.

Any help is greatly appreciated.

Thank you.

[kao]

Manually mapping DLL in a production environment is a bad bad idea. Debugging such issues will be pain-in-the-butt right now and impossible when a customer runs in some configuration-specific issue. So don't do it.
Any particular reason for not linking OpenSSL statically?

There are so many things that could go wrong. For example, how are you implementing callbacks to DLLMain with reason DLL_THREAD_ATTACH? 

Single-stepping through OPENSSL_init_crypto and figuring out which particular code fails will be probably the fastest way to locate the issue.

Cheers,
kao.

[risky]

9 hours ago, kao said:

Manually mapping DLL in a production environment is a bad bad idea. Debugging such issues will be pain-in-the-butt right now and impossible when a customer runs in some configuration-specific issue. So don't do it.
Any particular reason for not linking OpenSSL statically?

 

Single-stepping through OPENSSL_init_crypto and figuring out which particular code fails will be probably the fastest way to locate the issue.

Cheers,
kao.

Hey, thanks for your reply. You're right it is a bad idea, but LoadLibrary here is not viable. I'd rather reverse-engineer the entire process of LdrpLoadDll than use LoadLibrary. Debugging this is extremely difficult just due to the fact that this issue isn't present in a LoadLibrary scenario, just when the DLL is mmapped. I also forgot to mention, I am linking OpenSSL statically.

9 hours ago, kao said:

There are so many things that could go wrong. For example, how are you implementing callbacks to DLLMain with reason DLL_THREAD_ATTACH? 

Well, I don't think I'm properly handling that. Are you talking about how LdrpInitializeThread calls the init routine with DLL_THREAD_ATTACH? I'm not too sure how I'd that.

9 hours ago, kao said:

Single-stepping through OPENSSL_init_crypto and figuring out which particular code fails will be probably the fastest way to locate the issue.

That's true but it is proving to be extremely difficult to debug in a manual mapped state. The only thing's that come to mind in OPENSSL_init_crypto failing are the check 'if ( stopped )' where the function will simply just return 0. Another reason would be CRYPTO_THREAD_run_once failing, which I don't know how it could. One reason I thought of is if the OpenSSL stop handlers are for some reason being executed before I can even initialize myself.. I will continue debugging, but I have been stumped for the past 2 days .

Also, this is a completely different scenario, but it seems like this guy had the exact same issue: https://github.com/openssl/openssl/issues/10500

Weird.
 

EDIT:

Further debugging and I found that OPENSSL_init_crypto fails here.
 

v13 = CRYPTO_THREAD_run_once((int *)&load_crypto_nodelete, ossl_init_load_crypto_nodelete_ossl_);
v14 = 0;
if ( v13 )
	v14 = ossl_init_load_crypto_nodelete_ossl_ret_;
if ( !v14 )
	return 0i64;

https://github.com/openssl/openssl/blob/master/crypto/init.c#L115

Holy shit.... I guess I can fix this with a band-aid hook..

Thanks again,
risky

Edited October 11, 2020 by risky

[risky]

Sorry for the double post, but update - The hook allowed the init function to proceed as it should. If anyone goes through the nightmare of adding SSL on a mmapped DLL, make sure to hook GetModuleHandleExW if adjusting OpenSSL source-code is not feasible.

Thank you for your help, @kao.