It is widely accepted that immediate detection of security rules violations can be achieved by monitoring the system calls made by processes. This in turn makes possible to prevent malicious invocations of system calls from breaking system security.
We have developed the WHIPS (Windows Host Intrusion Prevention System) prototype for monitoring those critical system calls which may be used to subvert the execution of privileged applications.
WHIPS employs a simple mechanism for system calls interception at the OS kernel level and requires minimal additions to the kernel code and no change to the syntax and semantics of existing system calls. Basically, the system call execution is allowed just in case the invoking process and the value of the arguments comply with the rules kept in an Access Control Database (ACD) within the kernel. Common penetration techniques that involve tricking the system into running the intruder's own program in privileged mode are blocked by this approach.
Specifically, WHIPS blocks buffer overflow attacks before they can complete. Note that these are just examples of possible attacks, since our approach intends to protect against any technique that allows an attacker to hijack the control of a privileged process.