An AV (Anti Virus) is a piece of software that analyses files and monitors the OS (Operative System) status to detect malware. AVs can also analyze the machine’s traffic to determine suspicious behaviour. AVs only work in the machine they are installed in. An AV cannot protect a whole network. To do so, IDS (Intrusion Detection System) and IPS (Intrusion Prevention System) are used. Both come in hardware and software, unlike AVs which only come in software. There are different types of AVs, from user ones like Avast to whole EDR (Endpoint Detection and Response) systems such as SentinelOne. Endpoint protection systems are AVs used to monitor large amounts of machines. Businesses, especially large ones, tend to have endpoint protection systems implemented in all workstations. These systems help them protect their employees, but most importantly also help them to create automatic efficient contingency responses, to remove a threat when detected, or isolate a workstation to prevent the malware from spreading. This project will focus on hiding malware from AVs.

AV Analysis

AVs use many different techniques to detect malware. Generally, techniques used by AVs can be separated into two main categories, static analysis and dynamic analysis.

Static Analysis

Static analysis groups all techniques that do not execute the software to analyse it. The first step an AV goes through when analysing a piece of software is usually a static analysis.

Signature Analysis

Signature analysis is a static analysis that gathers pieces of data and hashes them, which then compares the hash with a database of known malware. The AV will use checksums and hashes to detect known malware. This technique works great to discard a huge percentage of known malware, it is a good security measure but it is not fault-proof. Any malware that has not been previously detected, and therefore its signature is not present in the AVs database, will not get detected by this technique. Some malware uses polymorphic and metamorphic engines to mutate themselves and change their signature with each infection. As an important note, when developing malware it is important to disable the AVs option of saving any signatures of the malware when it gets flagged when testing, because once it gets flagged its signature will be added to the database and will get constantly detected.

Heuristics Static Analysis

The heuristics static analysis consists of decompiling the code and checking for instruction patterns that are typical of malware. This technique can be tricky and unreliable because legitimate software might use some instruction patterns malware also uses. Therefore to avoid false positives the AV needs to be accurately tuned.

Dynamic Analysis

Dynamic analysis includes all analysis techniques that rely on executing the code to analyse it. Dynamic analysis is great for detecting unknown malware since it does not rely on previously known signatures. There are different types of dynamic analysis.

Sandbox Behavioural Analysis

Since dynamic analysis needs to execute the code to analyse it, it can be very dangerous, because it does not know what that piece of software can do to the machine. That is why the AV copies the software into a secure and monitored environment where it executes it and analyses its behaviour. This technique works great to detect new malware since it does not rely on knowing previous malware as heavily as the Signature analysis. However, as great as this technique can be, it is not perfect. The AV needs to keep the analysis quick to avoid the user having to wait too long. Therefore large programs might not be analysed completely. Usually, the environments created by the AV try to simulate a real OS as accurately as possible to make it harder for the malware to recognize that it is in a controlled environment.

Real-Time Behavioural Analysis

Real-time behavioural analysis is very similar to sandbox but with the difference that it will not execute the software in a safe environment, but it will watch the operative system as a whole, looking for strange behaviour such as: trying to modify a file, communicating through the web... These events will make the AV focus on the software's behaviour to determine if it is a piece of malware. This technique is risky because it involves the user executing the malware. If the AV doesn’t detect it early enough it can be too late to avoid the infection. On the other hand, this technique is incredibly useful, because it can detect malware that has bypassed previous techniques. This is going to be the hardest analysis to bypass. This type of analysis has two main components that are going to be triggered when something suspicious happens: userland hooks and kernel land hooks. Hooks are the process of intercepting a function call. Intercepting Windows API calls allows the AVs to inspect the process and search for malicious activity. To place these hooks, an AV has its own DLL loaded into all processes using its signed kernel driver when the process starts. When the DLL is loaded it will place the hooks on the tracked APIs. AVs will often try to hook the lowest level call possible to make it harder for malware to bypass these hooks. As explained before there are different API level calls, firstly in user mode there is kernel.dll, then in a lower level but still in user mode there is ntdll.dll and finally there are the syscalls in kernel mode.

Userland Hooks

As explained in Part 1, Windows has two modes: user mode and kernel mode. Userland hooks are hooks that are placed in the user mode area of Windows. These hooks affect, among others, the kernel32.dll and the ntdll.dll. To place a hook, AVs need to stick to the following process: Imagine an API function that the AV wants to hook. First, it gets the first few bytes of the instructions of the API function and stores them in memory:

Now it will substitute the bytes copied from the API function for a jmp instruction that will redirect to the AVs custom function with the desired code, jmp myCustomFunc fig 3.2. In an AV this code analyses and monitors the process.

In the final step, it will add 2 redirects to complete the trampoline. The first one will be from the custom function to the memory containing the first few original bytes of instructions copied from the API function in the first step, return bytesStored(). And second, it will add a jmp instruction at the end of the copied API function’s first bytes that are stored in the memory, this jmp will redirect to the original API function just after the jmp myCustomFunc, jmp apiFunc, fig 3.3.

This diagram is a theoretical way of how implementing a very common type of hook called trampoline hook. The implementation of this is far more complex than it looks, especially in x64 systems.

Kernel Land Hooks

Before the introduction of the KPP (Kernel Patch Protection), AVs were able to patch the kernel to achieve the same as in user land hooks. Right now, patching the kernel is no longer possible thanks to the KPP, which means that AV vendors had to find a new way to watch malware from the kernel. Lucky for them Microsoft introduced kernel callbacks. It is a Windows feature that allows drivers to register callback routines, which are called when a specific event occurs. This allows AVs to monitor from the kernel without having to modify the kernel. These callbacks have already been explained in Part 1.