Is this possible to set hook without calling SetWindowsHookEx?

(What I will say is general for any other Win API and does not provide alternative)

Anything is possible...

Professional malware authors use different techniques to make FUDs. (aside from using cryptors):

 

1- Some go deep and call the Windows NT native API (undocumented) instead of the regular subsystem Win API (documented).

https://en.wikipedia.org/wiki/Native_API vs https://en.wikipedia.org/wiki/Windows_API

That being said, you can disassemble SetWindowsHookEx and see what functions are being called internally.

You can write your own function to get something like: MySetWindowsHookEx.

 

2- Others go much deeper (actually crazier) and disassemble the functions and recreate them in C or assembly.

Sometimes, the ultimate goal of this approach is to create a concise pure shellcode to do the whole job

 

IDA HexRay tools and x64dbg (Snowman Decompiler) are ideal tools for both jobs

Good luck with all of that headache!

Edited June 5, 20178 yr by Alzri2

If I understand your question correctly, you can always create a detour.

Here is some code that can get that done:

void HookFunction(uint32_t AddressToHook, void *JumpTo) {
  DWORD protection;
  VirtualProtect(AddressToHook, 5, PAGE_EXECUTE_READWRITE, &protection); //5 should pretty much always be enough.
  *(char*)AddressToHook = 0xE9;
  *(char*)(AddressToHook + 1) = (uint32_t)(JumpTo - (uint32_t)AddressToHook) - 5; 
  VirtualProtect(AddressToHook, 5, protection, &protection);
}

 

I know SetWindowsHookEx will allow you to do some flags to easily locate some of the functions you can hook. You can replace this for WINAPI functions by using GetProcAddress to locate their addresses. Example:

int WINAPI MessageBoxB(HWND hWnd,LPCTSTR lpText, LPCTSTR lpCaption, UINT uType) {
  printf("I was called instead!");
}


DWORD address = (uint32_t)GetProcAddress(GetModuleHandle("user32.dll"), "MessageBoxA");
HookFunction(address, MessageBoxB);