Internet Protocols are known to be complicated and allow multiple interpretations during implementation. Attackers use unusual combinations and several evasion tools such as fragroute, whisker, and nikto to evade network security systems. If the network security device used to secure a network uses traditional methods to detect attacks, the attack slips into the network undetected. The attacker can now start advanced persistent threats (APTs) from a remote location.

How different AETs operate:
Each AET uses inherent features in a protocol to pass through the network security system undetected. In this document, we compare the effects of a traditional IPS versus those of Trellix IPS. To exhibit these effects, we consider the following AETs:

• Server message block (SMB) or Microsoft Remote Procedure Call (MSRPC) fragmentation
• SMB padding
• Application layer evasion through document obfuscation
• JavaScript obfuscation
• HTTP UTF-8 encoded content
• HTTP evasion (URL encoding)
• TCP Urgent Pointer
• TCP split handshake
• IP fragmentation
• IP fragmentation with chaff

SMB and MSRPC:
RPC is an inter-process communication framework. It allows a computer program to cause a subroutine or procedure to execute in another address space (commonly on another computer on a shared network). In effect, it takes on much of the low-level network functions and allows the programmer to focus on methods and procedures that they want to implement.

Microsoft Windows (Windows) ships in both server and desktop variants. All variants of Windows ship with implementations of numerous network services. The Server service is one such service that supports file, print, and named-pipe sharing over the network for Windows-based computers. Named-pipe communication is used for RPC because it supports communication between applications running on distributed systems.

There's a buffer overflow vulnerability in the Server service when processing RPC requests. Remote unauthenticated attackers can exploit this vulnerability by sending specially crafted messages to the affected interface. Successful exploitation can lead to arbitrary code execution with system-level rights.

SMB is a protocol used by Windows for locating and accessing resources like shares, printers, or computers. SMB has characteristics that an attacker can use to hide malicious traffic to exploit SMB- and MSRPC-based vulnerabilities. In addition, SMB acts as a transport for MSRPC.

The RPC mechanism allows an application to seamlessly invoke remote procedures, as if these procedures were executed locally.

There are two main implementations of the RPC mechanism:

• Open Network Computing (ONC) RPC
• Distributed Computing Environment (DCE) RPC

MSRPC is the Microsoft implementation of the DCE RPC mechanism. Microsoft added new transport protocols for DCE RPC, such as ncacn_np transport. This protocol uses named pipes carried into the SMB protocol.

MSRPC uses one or more of the following protocols for transport:

• TCP
• UDP
• HTTP
• SMB

For any MSRPC request to be made, the client first contacts the destination and establishes an SMB handshake. After the SMB handshake is established, an MSRPC request and response can follow. An illustration of the elements involved in the prelude to an MSRPC request is provided below.


SMB or MSRPC fragmentation:
After an SMB handshake is successfully completed, an MSRPC request is made. An MSRPC request is frequently used at the back-end in Windows. Consider an example when you try to access email using Microsoft Outlook. After you provide credentials, Outlook communicates with the Exchange Server to download these mails. During this procedure, several MSRPC requests are made to the Exchange Server. Or, consider another example in which you want to use a network printer to print a document. When you issue a print command, the client makes MSRPC requests to the printer server.

An attack can take place either during an SMB handshake or during an MSRPC request. Just as the payload can be fragmented in TCP and IP, it can be fragmented in SMB and MSRPC. The payload can then be spread across multiple fragments. But, unlike TCP and IP, SMB and MSRPC fragments can't be sent out of order. A traditional IPS, which doesn't support such detection, analyzes the behavior of completing the handshake and request but doesn't look further into these processes. The IPS validates the SMB handshake and MSRPC request. The IPS then detects the attack by matching the string with the attack signature available in its signature set. But, a traditional IPS is unable to reassemble the SMB and MSRPC fragments. The IPS fails to recognize the attack because parts of the attack are spread over multiple fragments.

The illustration below depicts SMB fragmentation.

For instance, if an attacker wants to send some malware that contains the characters B, A, and D in it, they might consider sending it as BAD. But, if the IPS contains this attack signature, it detects and blocks the attack.

But, assume that the MTU size on the path permits 3 bytes at a time. They could simply fragment the traffic and pass it on 1 byte or 2 bytes at a time to the destination. They could consider sending it as B and then AD, or BA and then D, or B, A, and D. In any case, traffic is sent over multiple fragments, as represented in the illustration above. A network security device that isn't capable of reassembling that data before analysis allows the malicious traffic to pass through. This restriction compromises the network.

MSRPC fragmentation works in much the same way with the exception that fragmentation is carried out in the MSRPC datagrams. The illustration below depicts the MSRPC fragmentation.

Trellix IPS detection for SMB or MSRPC fragmentation:


Trellix IPS reassembles the SMB or MSRPC fragments before the detection process begins and can detect an attack.

SMB padding:
To keep the SMB payload aligned to a specific memory size, SMB supports the addition of pads within its segments. The pad is inserted between the SMB header and the start of the actual payload.

The SMB header has a data pointer field that shows the start of the actual payload when a pad is present. All data after the SMB header until the pointed byte is discarded.

Windows discards any pads present and continues directly to the payload. But, a traditional IPS that's unable to discard such pads extracts them with the payload during detection and ultimately allows that traffic to pass through.

An attacker who wants to send malware containing characters B, A, and D in its string can pad these characters to evade detection. When this string passes through the IPS, it's extracted for detection as padBpadApadD. As a result, a traditional IPS misses the malware between each of the pads.

Trellix IPS detection for SMB padding:

Trellix IPS can detect the presence of pads and skip those bytes before the payload is extracted for detection.

Application layer evasion through document obfuscation:
Document obfuscation has become a common method for malware authors to distribute malware and infect networks. Documents such as Word, Excel, and PDF files are the most commonly used files. But, given the flexibility of PDFs to be rendered easily, PDF obfuscation has gained considerable popularity in recent years.

Malicious PDF files are some of the most common threats used to infect users with malware. Trellix IPS focuses much of its attention in the proper detection of these files. In contrast, creators of malicious files focus heavily on evading such detection techniques.

Detecting malicious documents is made complicated for the following reasons:

• Basic building blocks of the document such as objects in PDF files are complex and have numerous forms.
• Streams within documents that show text or images can be compressed and stored in numerous ways.
• Most document formats support scripting to some degree. For example, JavaScript in PDF can be heavily obfuscated, concealing the actual code.
• Documents support embedding other documents. For example, a malicious executable file can be embedded inside a non-malicious PDF file. This file can then be embedded in another document.

So, detection of malicious documents requires your IPS to have several engines that dynamically profile document behavior. In this article, we see one of the most prominent document obfuscation techniques, PDF obfuscation using JavaScript, among other methods.

PDF obfuscation can be carried out using several approaches, such as the following:

• Name fields, which are the basic building blocks of PDFs, can contain objects inside them that are hex-encoded. A traditional IPS looks for ASCII characters when analyzing the Name fields.
• String fields within objects can be split across multiple lines. When string fields are split, they become unrecognizable to a traditional IPS.
• JavaScript can be encoded using several filters such as ascii85, asciihex, lzw, run length encoding, and deflate.
  These filters can be applied on the JavaScript multiple times. Using any of these filters compresses data. The compressed data is unrecognizable to a traditional IPS, unless the IPS can decompress it.
• Malicious JavaScript can be hidden inside XFA or Annotation fields.
• Filter names in objects can be abbreviated. For example, AsciiHexDecode becomes Ahx.

If an IPS isn't able to detect such anomalies, a malicious PDF slips into the network. Among the most common PDF exploits, attackers prefer to use some form of embedded JavaScript.
For example, string replacements, try-catch exceptions, and fromCharCodeloops all work in PDF files as well.

Detection for PDF obfuscation:
Trellix IPS has been designed to detect malicious PDFs even when the above-mentioned obfuscation methods are used. Decoders for frequently used JavaScript encoding/compression techniques are used. Trellix IPS can locate JavaScript hidden in locations such as XFA or Annotations.

Detection for JavaScript in PDF obfuscation:
If the vulnerability being exploited isn't present in the Acrobat JavaScript engine, JavaScript still plays a role in creating a malicious PDF. The extraction and analysis of embedded JavaScript is important to detect PDF-based threats.

JavaScript obfuscation:
Consider an example where a simple Hello World program in JavaScript looks as follows.

NOTE: Traditional IPS sees the string Hello World as an alert (Hello World).
The equivalent obfuscated version in JavaScript looks as follows.
 

var _1204;var
_2288='120B105E146A1044F1099C1064F1129A1139C759C729A919D1064F1099D1099E1114C719E994B1114F1129F1099D1059A729D764A854A';var
_1804=/[\x41\x42\x43\x44\x45\x46]/;var _8663=2;var _5586=_2288.charAt(_2288.length-1);var _5657;var _2069=_2288.split(_1804);var
_9062=[String.fromCharCode,isNaN,parseInt,String];_2069[1]=_9062[_8663+1](_9062[_8663](_2069[1])/21);var
_6574=(_8663==8)?String:eval;_5657='';_11=_9062[_8663](_2069[0])/_9062[_8663](_2069[1]);for(_1204=3;_1204<_11;_1204++)
_5657+=(_9062[_8663-2]((_9062[_8663](_2069[_1204])+_9062[_8663](_2069[2])+_9062[_8663](_2069[1]))/_9062[_8663](_2069[1])-_9062[_8663](_2069[2])+_9062[_8663](_2069[1])-1));var
_2799='_5900';var _5430='_2799=_5657';function _1039(_6946){_6574(_7060);_1039(_7196);_7196(_5430);_1039(_2799);}var
_7060='_1039=_6574';var _7196='_7196=_1039';_1039(_5586);

Traditional IPS fails to detect Hello World in this obfuscated program.

Detection for JavaScript obfuscation:
Trellix IPS doesn't perform a string match here. Instead, it executes the JavaScript within a JavaScript VM and profiles runtime behavior to determine whether it's malicious or not.
Trellix IPS can detect malware despite JavaScript obfuscation, and ignore the JavaScript if it's not malicious. Further, it doesn't automatically trigger an alert for JavaScript obfuscation if traffic isn't malicious.

HTTP UTF-8 encoded content:
Any browser needs to know what character set to use to decode a page and render it correctly. To support multiple languages, several formats of character encoding are used.
In a typical attack sequence when exploiting such evasion, the attacker encodes the contents of the HTML page using UTF-8. In this scheme, the character A for example can be represented either as 41 or as c1a1. A traditional IPS searches for the character A and misses it if it has been encoded.

Detection for HTTP UTF-8 evasion:
Trellix IPS is designed to look at the content type field of the HTTP header. This field contains information about the nature of data in the body of the webpage. The charset subfield in the header indicates what character set is used to encode the webpage. This subfield provides Trellix IPS the needed insight to decode the page. Trellix IPS parses these fields and invokes the appropriate character decoder based on the charset field.

HTTP evasion (URL encoding):
Every URL, when sent to a webserver, displays webpages that are fetched from that webserver using an HTTP GET request. When you type in a URL in the address bar, the browser requests appropriate webpages to display.

An attacker tries to execute actions in the webserver. These actions include displaying information from specific directories or executing certain commands. To evade network security devices, an attacker can encode or obfuscate URLs. Default behavior of a recipient HTTP server is to normalize such URLs. A traditional IPS is unable to emulate the behavior of a webserver. The IPS can't normalize URLs before detection, and allows these attacks to penetrate network defenses. HTTP evasion is shown in the following examples.
 

HTTP evasion	Original String	Encoded String
HTTP URL hex encoding	GET /index.html	GET /%69%6e%64%65%78%2e%68%74%6d%6c
HTTP URL UTF encoding	GET /index.html	GET /%u0069%u006e%u0064%u0065%u0078%u002e %u0068%u0074%u006d%u006c
HTTP URL with self-referential paths	GET /index.html	GET /././index.html
HTTP URL with fake directories in the path	GET /index.html	GET /FAKE../index.html
HTTP URL with multiple spaces and tabs	GET /index.htm HTTP/1.0l	GET /index.html HTTP/1.0
HTTP absolute URL	GET /index.html	GET http://192.168.1.10/index.html HTTP/1.0

In the examples above, a webserver normalizes an encoded or obfuscated URL and parses it as the original URL. The exception is the last example. A traditional IPS isn't able to emulate this behavior and is, as a result, not able to match the string with any of the attack signatures.

Alternate method HTTP evasion:
HTTP supports different verbs like GET, POST, PUT, and HEAD for actions such as fetching a page, sending data, or requesting an HTTP server header. Certain URL-based vulnerabilities might have a vulnerability trigger in the URL. Any of these verbs can be used with the malicious URL to send a request to the webserver to trigger the vulnerability.

For example, if a URL that's greater than 128 bytes triggers a buffer overflow in a webserver, an attacker can send one of the following requests to set off that vulnerability into an attack.

• GET /home/aaaaaaaaaaaaaa[more than 256 times] HTTP/1.0
• POST /home/aaaaaaaaaaaaaa[more than 256 times] HTTP/1.0
• PUT /home/aaaaaaaaaaaaaa[more than 256 times] HTTP/1.0

Most commonly, the IPS is programmed to only look for GET /home/aaaaaaaaaaaaaa [more than 256 times] HTTP/1.0.

Detection for HTTP evasion using URL encoding:
Trellix IPS can normalize encoded or obfuscated URLs before detection. During detection, these URLs match malicious URL strings present in attack signatures and block the attack.
As an administrator, you can program your IPS to detect requests such as POST and PUT, and focus only on the URL.

TCP Urgent Pointer:
Urgent data denotes data that must be processed as soon as possible. This data is processed even before "normal" data that might be waiting. The presence of urgent data is signified by setting the urgent flag. If the urgent flag is set, it indicates that the urgent pointer is valid and points to that data.


Urgent data might also be seen as out-of-band data and the urgent pointer refers to the last byte of such data. Protocols such as SMB and HTTP ignore such data. A traditional IPS doesn't ignore this data. This data then becomes part of the reassembled payload used for detection. This decision can cause malware to evade detection.

In an example similar to BAD (see the "SMB or MSRPC fragmentation" section), the string BAD could be split across three segments, each of which begins with a byte of urgent data, X. If this urgent data isn't ignored, the reassembled payload appears as XB XA XD. The string when extracted by a traditional IPS appears as XBXAXD. IPS matches this extracted string with its attack signature, but doesn't find anything malicious, and allows it to pass through.

Detection for TCP urgent pointer evasion:



At the time of TCP stream reassembly, Trellix IPS ignores urgent data for SMB and other protocols known to ignore it.

TCP split handshake:
The TCP three-way handshake in Transmission Control Protocol is used by TCP to set up a TCP/IP connection over an Internet Protocol-based network. TCP's three-way handshaking technique is often referred to as SYN, SYN-ACK, ACK because there are three messages transmitted by TCP to negotiate and start a TCP session between two computers.

NOTE: In the context of this document, we consider a client and an HTTP server.

The TCP handshaking mechanism is designed so that two computers trying to communicate can negotiate the parameters of the network TCP socket connection. The computers can negotiate this connection before transmitting data such as SSH and HTTP web browser requests.

In addition to this formally accepted handshake, there's also another method of establishing a TCP/IP connection between a client and server. This method is legitimate, but rarely adopted and is known as a simultaneous-open handshake.


With this connection, both a client and server send a SYN packet to each other at about the same time. Then, both sides send ACK packets to each other in response. This unconventional variant of the TCP handshake isn't practiced much in the real world. The RFC accepts this variation as a legitimate way to start a TCP connection. When performing a TCP split handshake evasion, the attacker uses the concept of a simultaneous-open handshake. The attacker is the server that's seeking to gain access to a network.

Image Properties
Essentially, a client sends a SYN packet to a server, intending to complete a normal three-way handshake. But, rather than completing the client’s three-way handshake, a malicious server starts by replying as though it were performing a simultaneous-open connection. The server then starts its own three-way handshake in the other direction (from server to client). A traditional IPS is confused into analyzing this behavior as the initiation of a fresh three-way handshake. So, the attacker can then gain control over the client.

Trellix IPS detection for TCP split handshake evasion:

Apart from the regular TCP handshake, Trellix IPS can detect a split handshake and any attack that occurs across such a connection.

IP fragmentation:
The MTU size of a link determines whether there's a need to fragment a datagram into smaller units or not. If the MTU allowed by the link is lesser than the original datagram size, IP uses fragmentation so that packets might be formed to pass through the link. An attacker can deliberately fragment an IP packet so that the IP packet gets spread across multiple fragments.


An attacker can exploit this feature to evade a network security device. They can use an evasion tool to fragment the original attack packets. They can then use one of the following techniques to send out fragments:

• All fragments are sent in the following manner:
  • In proper order (increasing value of frag offset)
  • In reverse order (decreasing value of frag offset)
  • In proper order, except for the first fragment, which is sent last
  • In the proper order, except for the last fragment, which is sent first
• Fragments are shuffled and sent out of order

Assume that malware contains ATTACK in its IP datagram; it's fragmented as shown above. A traditional IPS can't reassemble fragments to detect the attack. It examines traffic as separate frames and allows an attack to slip through.

Detection for IP fragmentation evasion:


Trellix IPS can reassemble the malicious string and detect the attack.

IP fragmentation with chaff:
As in IP fragmentation, an attacker uses an evasion tool to fragment the original IP datagram that contains attack packets. After fragmenting the packet, the attacker interleaves chaff packets with the fragments.


A chaff packet is an invalid packet that the target stack ignores, based on certain checks mentioned below. The chaff can contain any one of the following anomalies because of which it's ignored:

• Bad checksum, because of which it gets dropped by the target stack
• A small TTL, because of which it expires before it reaches the target
• An invalid options field, because of which it's dropped by the target stack

A traditional IPS doesn't perform the same validation on chaff packets. So, an invalid payload in the chaff becomes part of the reassembled stream at the IPS and evades the detection mechanism.

Detection for IP fragmentation with chaff evasion:


At the time of reassembly, all IP header fields are examined to contain valid content before the payload is used for detection. If a packet is suspected to be chaff, it's dropped and the invalid payload carried by it is ignored.

Testing the effectiveness of Trellix IPS against advanced evasion techniques:
You can use an advanced evasion technique testing tool to test the effectiveness of Trellix IPS against advanced evasion techniques. For this article, we show this test using Trellix Evader.

IMPORTANT: The test environment used for this test is similar to the one used during NSS testing. Trellix IPS achieved 100% detection during lab testing.

You must use the following versions of software and signature sets:

1. Sensor:
   • M-Series: 8.1.3.52 or later
   • NS-Series: 8.1.5.59 or later
2. Manager: 8.1.7.5 or later
3. Signature set: 8.7.46.3 or later
4. Evader: 2013_4_594 or later

Configure Sensor policy:
You must configure your Sensor to use the All-Inclusive-with-Audit policy or a clone of this policy. To optimize the number of alerts, you receive and minimize the number of low-severity alerts:

1. Log on to your Manager.
   Click Policy, <Admin Domain Name>, IntrusionPrevention, IPS Policies.
2. Select the All-Inclusive-With-Audit policy or an unmodified clone and click View/Edit.
3. In the policy editing window that appears, click the Severity column header to sort by severity.
4. Beginning at the top of the list, use the shift key and the down arrow key to select all attacks categorized as 0 (Info) and 1 (Low).
5. Right-click in the list and click Disable. All selected attacks must disable the X symbol.
6. Now search for the attacks listed below and enable only these attacks:
   • 0x4070d200 – NETBIOS-SS: Microsoft Server Service Remote Code ExecutionVulnerability (Info)
   • 0x4070d000 – NETBIOS-SS: Potential DCERPC Big Endian Evasion
   • 0x4070d100 – NETBIOS-SS: Potential TCP Segmentation Evasion
   • 0x4070cf00 – NETBIOS-SS: Potential TCP URG Pointer Evasion
   • 0x4070ca00 – NETBIOS-SS: Suspicious SRVSVC Call
   • 0x00010300 – TCP: Small Window Packets Detected
   • 0x40011400 – TCP: Illegal Packet Missing SYN, ACK, or RST fields
   • 0x40011500 – TCP: Anomalous Urgent Flag Set without ACK
   • 0x00009e00 – TCP: 3-Way Handshake PAWS Fail DoS
      
7. Click Save to complete your configuration.
   For details about activating this policy, see the IPS Administration Guide for your release.
   
   NOTE: You might see a minor difference in results when run for an extended period. This difference is because the Evader tool can generate attacks using randomized sets of evasion techniques.
   We suggest that you use this tool to compare the relative performance of several products and run the test multiple times for each product. This comparison provides an understanding of the ability of each product tested, to detect and block attacks masked with these techniques

After you've made sure of the above prerequisites, follow the steps below.

1. Use the (Stonesoft) User Guide to install and set up the Evader Test environment.
2. After your deployment of Trellix IPS, verify that you've made the following settings in the Manager before using Evader:
   1. Click Policy, Advanced, TCP Settings. Set TCP flow Violation to DENY.
      Configuring this setting makes sure that TCP flow violations are dropped.
       
   2. Click Policy, Advanced, TCP Settings.
      Set Cold Start Drop Action to DROP.
       
   3. Click IPS Interfaces, <Interface or sub-interface Name>, Protection Profile, Advanced Traffic Inspection,
      Enable Inspect MS RPC/SMB fragments for malicious payload.
       
   4. While running the conficker test, go to the Manager and configure the response action as blocked. Set a TCP RST response of the source and the destination IP addresses, for the following attacks:
      • 0x47602f00 – DCERPC: Suspicious DCERPC Call
      • 0x4021c500 – HTTP: phpBB Viewtopic.php Remote Command Execution
      • 0x4023a400 – HTTP: Possible IPS evasive with Apache HTTP Server
      • 0x40018e00 – IP: Options Validation Vulnerability
      • 0x00010a00 – IPv4: Malformed Options Evasion Attempt Detected
      • 0x40709d00 – NETBIOS-SS: Microsoft Server Service Remote Code ExecutionVulnerability (High)
      • 0x4070d200 – NETBIOS-SS: Microsoft Server Service Remote Code ExecutionVulnerability (Info)
      • 0x4070d000 – NETBIOS-SS: Potential DCERPC Big Endian Evasion
      • 0x4070d100 – NETBIOS-SS: Potential TCP Segmentation Evasion
      • 0x4070cf00 – NETBIOS-SS: Potential TCP URG Pointer Evasion
      • 0x4070ca00 – NETBIOS-SS: Suspicious SRVSVC Call
      • 0x00012f00 – SMB: NB Chaff Detected
      • 0x00013000 – SMB: SMB Chaff Packet Detected
      • 0x00012e00 – SMB: SMB Payload Suspicious pad bytes
      • 0x40011400 – TCP: Illegal Packet Missing SYN, ACK or RST fields
      • 0x40011500 – TCP: Anomalous Urgent Flag Set without ACK
      • 0x00010300 – TCP: Small window packets detected
      • 0x00009e00 – TCP: 3-Way Handshake PAWS Fail DoS
        
        To learn more about IPS policies in Trellix IPS, see the IPS Administration Guide.

Summary:
Attackers have become more adept at scripting attacks. Attackers now start complex evasions that cut across multiple layers of the protocol stack. They incorporate evasions at more than one layer, in more than one protocol. In a situation where a client sends an HTTP GET request to a rogue server, the server responds with a malicious page. At layer 7, the attacker might have chunked webpages before serving them to the client. Further, at the TCP level, the attacker can segment TCP data. At the IP level, the attacker can fragment traffic before serving it to the client. The attacker can also add chaff between IP data. A traditional IPS misses some or all these techniques of evasion and allows malicious traffic to pass through. But, as shown in the examples above, Trellix IPS is equipped to protect your network from some of the most insidious attack campaigns. This application note discusses only a few examples of evasions that Trellix IPS supports. Trellix IPS supports several other combinations of advanced evasion techniques that occur at each layer of the protocol stack, where multiple exploits can be applied.