API Hooking Methods 

www.madshi.net

Sometimes you need to hook a specific function or API. Unfortunately there's no official way to do such things, instead you'll find a lot of different hacks that were invented by a lot of different people. All those API hooking hacks have advantages and disadvantages. There's no perfect solution.

When analysing API hooking methods you should ask the following questions. Each question does have a sense, of course. The questions are meant to make existing differences between all the available API hooking methods more evident:

1. Does the method need to manipulate the to-be-hooked binary files on the harddisk or is it capable to do it's work by only patching memory? The latter solution should be prefered in most cases.
2. Can the method install a hook on functions, which are not exported? Sometimes it is useful to hook some Delphi/C++ RTL functions or to hook non-virtual methods or similar stuff.
3. Can the method safely install a hook on each and every API, regardless of how the binary code of the API is structured? Some API hooking methods get unstable, if the binary code of the to-be-hooked API is shorter than 5 or 6 bytes or if it is structured strangely.
4. If hooking a specific API would end up in instability, does the hooking method automatically detect this and does it consequently react to the situation by either refusing to install the hook or even by automatically switching to an appropriate alternative hooking method?
5. Can "shared APIs" be hooked process wide? "Shared APIs" are functions which are exported by shared DLLs. A shared DLL is a DLL which is located in the shared area. The shared area is in win9x the memory area from $80000000 - $FFFFFFFF. The most important win9x system DLLs (e.g. kernel32.dll and user32.dll) are shared DLLs. In the winNT family there is no shared area, thus no shared DLLs and no shared APIs, either.
6. Can shared APIs be hooked system wide?
7. Once installed, does the hook catch those API calls, which were statically linked before the hook was installed?
8. Once installed, does the hook catch those API calls, which were statically linked after the hook was installed?
9. Once installed, does the hook catch those API calls, which were dynamically linked before the hook was installed?
10. Once installed, does the hook catch those API calls, which were dynamically linked after the hook was installed?
11. Once installed, does the hook catch those APIs calls, where the caller of the API is a win9x shared DLL?
12. Can the method properly undo an API hook?
13. Can the original API be called directly without having to temporarily unhook the API?
14. Is a real hook chain built automatically? In other words: What happens if you hook an API twice and then unhook in the wrong order? If the hook method doesn't build a real hook queue, you'll leave the hooked API in a wild state, which will result in a crash as soon as the API is called the next time.
15. Is it possible to safely unload a hook DLL? If a hook DLL uninstalls its API hooks when being unloaded from a process, some hook callback functions might still be in use. In that case the process will later crash.

Generally I prefer solutions which doesn't touch the standard/system DLLs on harddisk, so I'll only talk about methods here, which work purely in memory. Here comes a description of the most important hooking methods:

Import Table Patching

The best known and most often used method is "Import Table Patching". Each win32 module (application/DLL) has a so-called "import table", which is basically a list of all APIs, which this module calls. Patching this import table is a quite easy job and works very nicely. Unfortunately only the statically linked API calls go through the import table. Dynamically linked API calls (GetProcAddress) are not caught by this method at all. That means: It's absolutely possible, even quite probable that you'll miss some API calls when using this method.

To reduce the danger of missing API calls, you should not only patch the import table of the application's module, but you should also patch the import tables of each and every DLL that is loaded by the application. But please note, that you must not patch the import tables of win9x shared DLLs, because this could eventually make other processes crash. That means: When using "Import Table Patching" the danger of missing API calls is even higher in win9x than in winNT.

There's a slight danger when using import table patching. Some DLLs have a shared import table, even though they are not located in the shared area. One example is the "winspool.drv". When patching the import table of this module in win9x, you risk system stability.

Both Matt Pietrek's and Jeffrey Richter's API hooking stuff, and also Oleg Kagan's "Syringe" use the "Import Table Patching" method.

Extended Import Table Patching

There are some ways to make "Import Table Patching" catch more APIs. You can hook the API "LoadLibrary" to be notified about when new DLLs are loaded. Then you can right away patch those DLLs, too. If you don't do that, you won't catch API calls that are made from DLLs, which are loaded after you did install the hook.

Another extension is to hook the API "GetProcAddress" and then when called not return the real API address, but the address of your callback function instead. This way you would also get those API calls, that are dynamically linked after you've installed your hook. Unfortunately if you do this, you can't unhook such calls, anymore.

Export Table Patching

Each win32 module has an "export table", which lists all functions that this module exports. Patching this export table is not very difficult and brings some interesting results: All new linkings (both static and dynamic) to the patched API will automatically be bound to your callback function instead of to the original function. That's very nice! Well, but unfortunately patching an export table has absolutely NO effect on already existing API linkings (e.g. static bindings of already loaded DLLs). So using export table patching alone is only of quite restricted use. Besides, undoing the patch has again no effect on past bindings. So you can't really undo an API hook which was established by export table patching.

You should not patch the export table of DLLs which are loaded in the shared area of win9x. Also you should not patch the export table of DLLs, which are not loaded there, but have a shared export table nevertheless. If you do that, you risk system stability again.

Simple Code Overwriting

Instead of changing any tables, you can also directly manipulate the API's binary code in memory. The most often used method is to overwrite the first 5 bytes of the API code with a JMP instruction, which then jumps to your callback function.

This method is quite effective. It catches really 100% of all API calls. But it has also some big disadvantages. The biggest is that you can't call the original API in your callback function anymore, because when you would do that, the JMP instruction would again jump to your callback function, which would result in an endless loop.

Of course you could temporarily undo the hook in order to first restore and then call the original API, but that's a bad idea because of two reasons: (1) Steadily hooking and unhooking consumes precious time. (2) While you've temporarily unhooked the API, you're in big danger to miss a lot of API calls.

Another disadvantage of this method is that you normally can't overwrite the API binary code of win9x shared APIs, because it's protected. Well, and even if you could do that, it would not help too much, because when you change the binary code of a shared API, this change has effect system wide. That means when hooking non-shared APIs we have process wide API hooking, when hooking shared APIs we have system wide hooking. That's not very senseful. System wide API hooking is very difficult, mainly because your callback function has to be accessible system wide while your code is normally only accessible inside of your own process.

Finally the code of the to-be-hooked API has to be at least 5 bytes long. Otherwise you would overwrite more than the original API code, which would most probably result in a crash sooner or later. Furthermore there must not be any jumps later in the API code, which jump into the bytes 2-5 of the API code. In such a case overwriting the first 5 bytes would also probably result in a crash sooner or later.

The hooking package available at both programsalon.com and hookapi.com (actually it's the same hooking package) still uses "Simple Code Overwriting".

Extended Code Overwriting

The biggest disadvantage of the "Simple Code Overwriting" method was that you can't call the original API, at least as long as the API is hooked. Now clever people thought about it and found a way to erase this problem completely.

The solution sounds simple: We need to overwrite 5 bytes of the API's code. So we simply copy these bytes to another location and call it there, whenever we want to call the original API. Unfortunately it's not as easy as it sounds. You need to know that every binary code consists of single instructions. Each instruction can have a different length. So if we copy 5 bytes from the API, that can be exactly one instruction, or one and a half, or two and a half or ... We don't know that for sure. But we know for sure that when executing half copied instructions we will make our program crash. That means we need a real little disassembler, which can tell us how many bytes we really have to copy, so that we copy complete instructions instead of half instructions.

The "Extended Code Overwriting" method is quite difficult to implement, but once you have it ready, it is a very good method. The only 2 disadvantages are that (1) you can hook shared APIs only system wide and (2) the to-be-hooked API's code may be structured in a way, that doesn't allow you to use the "Code Overwriting" method without risking crashes.

All three packages Detours, ApiSpy32 and EliCZ are using the "Extended Code Overwriting" method. As far as I know neither Detours nor ApiSpy32 care how the binary code of the to-be-hooked API looks like. I think, EliCZ's solution does check the binary code somehow, but I'm not sure how extensive/reliable the check is. If an API is hooked by using code overwriting, although the binary code is not suitable for that, you're almost guaranteed to get crashes sooner or later.

Madshi's Code Overwriting

The method used by my "madCodeHook" package is a slight variation of the "Extended Code Overwriting" method. Instead of overwriting the API code with a 5 byte (relative) JMP instruction I overwrite it with a 6 byte absolute JMP call. That makes it easy to build up a real hook queue. That again makes it possible to realize stable process wide API hooking even for shared APIs. When using my method you can choose whether you want to hook a shared API process wide or system wide. The only remaining problem now is that the code of some APIs is too short (or strangely structured) to be hookable by any code overwriting method.

madCodeHook's disassembler examines the whole to-be-hooked API (in win9x some DLLs are even completely disassembled) before hooking it and then decides whether code overwriting can be safely used. If not, madCodeHook automatically switches to:

Madshi's Mixture Mode

Now what can we do with APIs, whose code is too short to be hooked by code overwriting? Well, we can simply enlarge the code! That is quite easily done in 2 steps: (1) We build an API header, which does nothing but jump to the original API. The header itself is long enough to be hooked by code overwriting. (2) Now we patch the export table and all import tables so that everything points to our newly allocated API header. From now on it looks as if the original API has always begun at our API header. And it will remain this way until the process exits (or for shared APIs in win9x: until the next reboot). The result of the whole operation is simply, that now such a manipulated API can be hooked by code overwriting.

Because this solution combines import table patching, export table patching and code overwriting, I've called it the "Mixture Mode". It has one disadvantage: API calls that were linked dynamically before the API was hooked the first time, will not be caught, because they still jump directly to the original API instead of to our enlarged API header.

 

Now let's compare the best known API hooking methods/packages in a table. Each column is one API hooking method/package. Each row answers one of the 14 questions asked on the beginning of this page:

Import
Table
Patching
Ext.Imp.
Table
Patching
Export
Table
Patching
Simple
Code
Overwr.
Ext. Code
Overwr.
Detours
ApiSpy32
Version
3.0
EliCZ
Code
Overwr.
Madshi's
Code
Overwr.
Madshi's
Mixture
Mode
1. only patches memory
2. hook non exported funcs
3. hook every API
4. detect non hookable APIs *
5. shared APIs process wide **
6. shared APIs system wide
7. catch old static links
8. catch new static links
9. catch old dynamic links
10. catch new dynamic links
11. catch shared DLLs calls
12. can call original API
13. proper unhook possible
14. automatic hook queue
15. safe DLL uninjecting

* I'm not sure how reliable the detection is. Also if the detection fires alarm, the hook just does not install. There's no alternative hooking method being used automatically in that case.

** EliCZ might not agree with me here. His code overwriting method actually is able to hook shared APIs process wide. But there are two different sources of instability in his approach, because of which I consider this part of his offered functionality as not being really usable.