Since Windows XP SP2, Microsoft has implemented a range of protection mechanisms to Windows to guard against malware. These include Data Execution Prevention, which is a hardware functionality in a CPU that is also implemented at a software level. Another lesser known technology is a pair of function calls in Kernel32.DLL of the Windows API, these functions are “EncodePointer” and “DecodePointer”. These function calls are used to protect pointers by obfuscating them using a “secret” value for an encoding or decoding process.
For reasons which I don’t really know about or have time to go into, we encountered a problem with a legacy application (RM Integris Classic) at our site which turned out to be much slower in execution and eventually hung. I decided to use an API tracing tool (Microsoft Dependency Walker, in this case, a tool which shows DLL dependencies) to see if there were problems with missing DLLs or function call exports being experienced in the application.
In this case, a major difference noted between two systems running Windows Vista, one of which could run Integris and one which could not, was the execution on the latter system of numerous and repetitive calls to Kernel32.DLL’s EncodePointer and DecodePointer functions. When I looked up these functions to see what they are used for, they turn out to be part of this new functionality that has been put into later releases of Windows. Specifically, EncodePointer and DecodePointer were introduced in Windows XP SP2. However it is interesting to note that I have not seen any compatibility problems before now with Windows XP running RM Integris.
I haven’t gone further into why the application might be incompatible with these calls, instead I have just looked to use the compatibility settings in Windows Vista and Windows 7 (on another computer that is running the Release Candidate). In the Vista (x86) computer, setting Omnis compatibility mode to Windows 2000 fixed the problem, but trying the same on the Windows 7 (x64) computer did not. However this gives me a solution for our planned migration to Vista; I’m currently building a master image of Vista for our HP laptops and intend to start deploying it early next week, and Integris is important to our site. Having that fix is useful when we will need to implement a transition to Windows Vista over the next year or so.