|
|
When doing API hooking you need to follow some basic rules. If you don't follow them, your API hooks might make things unstable. Here's a list of the most important rules:
| 1. | Always use *exactly* the same parameters, the same result type and the same calling convention for the original API, your callback function and for the "NextHook" function variable. |
| 2. | Only do what is absolutely necessary. Don't build monster callback functions, try to keep things short and easy. |
| 3. | Be careful what you do in your hook DLL, if you plan to inject it into NT system/service processes. You're not allowed to do any GUI stuff (e.g. FindWindow, PostMessage, MessageBox, etc) inside of a non-interactive system process. If you do nevertheless, things like sudden OS reboots can happen. Check out madCodeHook's Tool Functions. They can tell you in what kind of process your DLL is currently running. If you need to do inter process communication, please use madCodeHook's IPC functions instead of FindWindow/PostMessage. |
| 4. | In your hook DLL link to as few DLLs as possible. Avoid linking to RasApi32.dll, because NT 16-bit processes will otherwise tend to crash when being started the first time. Linking your DLL to oleaut32.dll will make "Paint" crash in win95. However, this problem can be fixed by applying this patch: http://support.microsoft.com/support/kb/articles/Q162/5/00.asp |
| 5. | Make sure that your hook callback function does not accidently alter the Get/SetLastError value. You may alter it, if you want to signal a failure. But if you are calling the next hook and don't want to impact the flow of things, please make sure that neither you nor any of the functions/APIs you're calling messes around with the last error value. Some APIs like CreateMutex do change this value. So if you call CreateMutex inside of your hook callback function, please save the last error value and restore it afterwards. The madCodeHook Tool Functions are last error neutral. |
| 6. | Each API hook needs its own callback function and its own "NextHook" variable. Don't try to save code by using callback functions and/or NextHook variables twice. |
| 7. | In winNT in hook callback function of wide APIs (like e.g. GetModuleFileNameWCallback or CreateFileWCallback) don't do anything which results in the OS trying to convert ansi <-> wide strings. Don't let Delphi automatically convert ansi <-> wide strings for you. Don't ask Windows to convert strings for you. Don't call any ansi APIs (this internally results in ansi <-> wide conversion again). Instead use the madCodeHook Tool Functions to convert strings. This rule doesn't apply to win9x. Also it doesn't apply to ansi hook callback functions in winNT. It only applies to wide hook callbacks in winNT. |
| 8. | In the initialization of your DLL avoid to directly or indirectly call VirtualAlloc, because otherwise in winNT 16-bit processes will tend to crash. |
This all might sound complicated and difficult. But hey, after all API hooking *is* a difficult thing. madCodeHook tries to free you from all problems as far as is possible. But the rules mentioned above can not be controlled by madCodeHook. Those are general hooking rules which are valid regardless of which API hooking method or package you're using and I can't change that.
You have to understand what happens when you hook an API: You are violently breaking into the flow of a program, which someone else has written. This other programmer has not intended that you do that, nor has he even calculated with that. That means you have to be very careful that you don't break his code. You have to try to not change anything which could bring the original program into problems. Try to change only what is necessary, not one bit more.
You can violate some of the rules without big impact, if you're only hooking one specific process. You should then test things deeply, but if no problems arise, it's okay. However, when doing program independent system wide API hooking, things are different. It's totally impossible to check your hooking stuff against all existing programs in the world, there are just too many to check them all. So you can just follow the rules and hope that your hooks don't break anything. But hey, if you are careful, thing are looking quite positive. So don't worry too much about these things. Just make sure that you follow the rules.