APT - The Plain Hard Truth

The survivors from the front line have reported in. We stand on the ridge, a tangled mess of bodies behind us. We are the ones who have chased the demon, descending into the binary pit the users call the “enterprise”, and climbed up the other side. What we have seen is not pretty. The collective corporate filesystem is a parking lot for castaway software barely able to run on modern operating systems, squeezing the last bit of life out of burned out win32 DLL’s. There are big piles of unwashed garbage downloaded by employees that were passing by, never deleted, never clean. The strangest mutated crap has been swept tightly into temporary directory corners that have since calcified and become permanent.

More than a single digit percentage of these software programs are a biohazard. Some are just plain broken, wheezing out juice from a hooked windows message chain just long enough to cough up and die, only to be resurrected by the swift kick of a boot-time registry key the next time the machine reboots. Some have pretty little labels of well-known companies – clearly so you won’t look twice at them and notice how they are exfiltrating personal browsing statistics and other data to some cloud server – really like malware but allowed by the EULA that you didn’t read. Some of these things don’t seem to have any purpose but to act as a low-fidelity binary listening device.

Everything looks bad. So, it’s no wonder that hackers can just plug something new in and nobody notices. As long as it doesn’t infect five million residential banking customers then nobody is going have a description of the suspect. That is the reality of hacking today, and it has nothing to do with advanced persistent threat. It has to do with the enterprise and the complete LACK of control you have over the endpoint. When security is limited to the network perimeter, you are not in control. Oh, and what a breath of fresh air the mobile device is. A new pile of software, mostly social media, that is directly connected to thousands of strangers that are not your employees, communicating in real-time with processes running within your defensive wall. In effect, you now have thousands of potential multi-homed routers to 3G-space* from your network that don’t belong to you.

*4G if your lucky

Here are some basic security facts:

• Today, malware is a tool for persistent adversaries
  
• Adversaries are financially or politically motivated
  
• Intrusions involve a real human being or hacking group that targets your organization directly (*)
  
• Attackers are motivated to steal something from your network

*Somehow in the mid-2000’s it seems like the security industry lost its way and forget about the basic tenants of Hacking Exposed – unfortunately you cannot condense a set of MD5 checksums out of the hacker problem.

Recently during presentations I have outlined three primary threat groups we face today. I have illustrated the evolution of these in the following diagram.


A. Criminal Enterprise – these are the guys who make more money than drug cartels and the reason a malware economy emerged over the last few years. This is what mere mortals mean when they talk about malware, and the reason people get malware and hackers mixed up all the time.

B. Rogues – these are the hacking groups that you can enumerate on any given day. There are hundreds, if not thousands worldwide. These guys are all capable. The graph expands much slower than criminal enterprise because they aren’t fueled by cash. As early as 2000 these guys were already defacing, DDOSing, and partaking in ‘mostly harmless’ hackery. Yet, a small subset have always been deeply malicious and get pleasure out of destroying things. Others pick up a cause and act like cyber terrorists. And still others really are cyber terrorists.

C. Rogues meet cash - these hired mercenaries are the ones who write malware, sell zero day, and get sucked into the vortex of organized crime. These guys are very, very dangerous.

D. The problem today - all the membranes have been breached - the threat is blended. We live in a time where a state interest can simply buy access to adversary networks from criminals who are selling their botnets. Where state sponsored attacks can be vectored through private hacking groups. Where private hacking groups can fund their operations from cybercrime, while targeting corporations and governments with methodology indistinguishable from APT. There is no tidy bucket to place the threat, all the wires are now crossed. The only thing that is consistent here is that hacking is hacking, and it always looks and smells the same when you see it. This is why the term ‘APT’ is so tired.

E. Private hackers working for the man - when you catch a Chinese malware in a DoD contractor network, it almost always looks like it was written by a “kid”. This “kids” malware is then used to steal the plans for a weapons program that can only have value to the PLA. All the security vendors looking at APT come up with corny little codenames for all the hacking groups (HBGary included), but at the end of the day it’s all the same thing.

F. Thank God for APT - a board room level term that we can all use to cover our you-know-what when we tell the man our millions of dollars in security spending has done nothing for us.

If you want a no-holds-barred, no excuses, and no-snakeoil analysis of APT and the reality of countering it, you should check out HBGary’s new whitepaper The New Battlefield.

-Greg

Inside an APT Covert Communications Channel

Note: I shortened the title of the post from "Inside an APT “Comment Crew” Covert Communications Channel" to "Inside an APT Covert Communications Channel". To be clear, multiple threat groups are using HTML comments as a means of COVCOM. Thus, this should be considered a general technique as opposed to attribution on a specific group. Both Shady RAT and "Comment Crew", as well as others with additional codenames, have been associated with the use of HTML comments as a means of COVCOM.

For many years, hackers operating out of China have been attacking a myriad of commercial and government systems here in the US and abroad. The term “APT” or Advanced Persistent Threat has often been used to describe these attackers. While HBGary is primarily a product company selling an enterprise incident response product, the team has been deep into APT analysis for over five years. Most of the analysis work is in direct support of Digital DNA – an automated system for detection of unknown malware and APT intrusions. I presented a technical description of how this attribution works, what is solves and what it doesn’t, at the BlackHat Conference last year. The work is about tracking threat groups – that is, tracking the humans and the human factors behind the digital artifacts we see. There are many hacking groups involved in these intrusions. One such group has often been called “Comment Crew” for their use of HTML comments as a means of command and control. This group has been associated with the recent “Shady RAT” intrusion revealed by McAfee. For this article I am going to give you a technical in-depth tour of how such a group operates.

For starters, the attackers will gain access to the network via spear-phishing. In almost all cases we have investigated, spear-phishing was the initial point of infection. These phishing emails are full of very specific project names, names of associates, official sounding documents, etc. It is very clear that the hacking group is using stolen email to learn about their targets before crafting a very convincing email. This underscores why the recent spate of SQLi attacks over the last few months pose a far greater threat than most people realize.


Exploit and Dropper

Once access is gained into the network, the hacking group places remote access tools into the environment. These are backdoor programs that are downloaded automatically by the exploit email – we called these “droppers”. In the diagram, point A shows the exploit email ‘detonating’ after being viewed by the victim, point ‘B’ is a server where a ‘dropper’ is stored, and point ‘C’ is the dropper backdoor being placed onto the compromised computer.

Once the dropper has established a beachhead into the network, a hacker will access the host and uninstall the original backdoor, replacing it with a new and more powerful backdoor. These backdoors, especially the secondary and more powerful one, are called “RAT”s – for Remote Access Tool. Many of these RATs are custom written and that can be the basis for a great deal of attribution, allowing us to detect the malware in physical memory.


Interaction with the Host

Remember that most networks are firewalled. This means the attacker can’t just make a TCP connection into the RAT program. The RAT program is within the internal network so it must first make an outbound connection to the attacker. The RAT is designed to connect outbound over port 80 or 443, a port that is allowed outbound by almost all firewall policies. Once the outbound connection is made, the attacker can use the established TCP session to interact with the host, download tools, run command line programs, and laterally move about the network. In the diagram, point A is where the RAT makes an outbound connection to a server on the Internet, point B is a server under the hacker’s control, and point C is where the hacker uses the established TCP connection to interact with the RAT program and subsequently the host environment, potentially exploiting additional machines nearby in the network.

One of the greatest challenges for an incident response team is discerning the difference between ‘normal’ malware and an APT attack. As we can see in this example, an APT attack involves a real human at the other end of the keyboard performing actions on the host. We call this ‘interaction with the host’ and we recommend that an IR team pull a timeline of last-access times from the MFT (master file table), browsing history from index.DAT, event log, and other sources to determine if such interaction is occurring. This is a fast and easy way to discern the difference between a non-targeted external threat (which over 80% of all adverse events will fall into this category) and external targeted attacks (of which APT is included, probably less than 2% of all adverse events).

The RAT program doesn’t contain any fancy stealth or anti-forensics measures. In fact, we rarely even see packers in use (a packer is a method of obfuscating a program after compilation and is a low-cost way for a hacker to add anti-forensics to his malware). It seems the most of the covert methods are applied to the way to RAT communicates with the hacker. This makes sense. Consider that most of the intrusion detection capability lies at the perimeter of the network, and this is what the hacker is trying to defeat. Thus, the HTML comment method of configuring and controlling the RAT programs.


Hidden Comments for Covert Communication (COVCOM)

Instead of letting the RAT connect directly to his personal server, the hacker will first exploit a webserver somewhere on the Internet. This exploited webserver will then be used as the ‘middleman’ to communicate with the RAT. The hacker will place a hidden comment on an otherwise normal webpage and have the RAT connect outbound to this page. Using the hidden comment, the hacker will be able to give commands to the RAT. The RAT will make periodic outbound connections, sometimes waiting days before checking the page. The hidden comment will contain an encoded message that the RAT knows how to decipher. In this case example, the hidden data is base64 encoded. In this diagram, point A is the RAT program making a periodic outbound connection, point B is a compromised webserver somewhere on the Internet, point C is the hidden comment on the webpage, and point D is where said comment is decoded into actual instructions for the RAT. An example of such a comment is shown in the next image. It is interesting to note that the hacker has attempted to make the page look like a 404 HTML error page if viewed in a normal web browser.


Example of BASE64 Encoded Hidden Comment

Once the RAT decodes the message, the data becomes a configuration file for the malware. The file has many features, such as the ability to specify which server addresses to use on the Internet, including backup servers, configuration of the check-in times, and even has the ability to completely update the RAT binary in the field (shown in the diagram as a .bmp file – this is actually a normal PE header executable).


The Decoded Configuration File

All of the above technical information can be detected on a host after intrusion. The RAT program itself is near trivial to detect once you know what you are looking for. But beyond that, because the RAT program has certain outbound connection characteristics, sleep timers, and built-in “host interaction” capabilities, HBGary’s Digital DNA lights it up like a Christmas Tree (example shown in image).


Digital DNA Detects Unknown Malware

Even if you had no prior knowledge about this specific RAT, you would have detected it with HBGary. Beyond that, the decoded configuration file can also be found in physical memory – the primary search method used by Active Defense. Regardless of the configuration values, the option headers shown in the example above have a specific pattern that can be detected quite easily, even if fragmented over multiple buffers. This is exactly the kind of information I am referring to when I talk about “actionable threat intelligence”. Once you know about the attackers TTP’s (tactics, techniques, and procedures) you can encode this into an enterprise-wide scan. We call it ‘continuous protection’ when you adopt continual scanning while also updating the threat intelligence as you learn more about the attacker. In essence, you are applying attrition against the attacker’s presence in your network. For example, if you know how to detect the above configuration file, then the attacker has to change the way that configuration file looks to defeat you – something that also requires them to recode their parser in the malware. Hence, you cost the attacker time and money. That is a Good Thing.

I hope this gave you a somewhat concrete tour of how a real APT covert communication (COVCOM) channel works. Also, I hope it has illustrated some of the threat intelligence that you access on the host. Using enterprise-wide scans, your IR or security team can put a severe dent in the APT presence in your network. As far as product solutions to enable you, obviously we build HBGary’s Active Defense. If you are interested in continuous protection and threat intelligence, we offer 50-node evaluations of Active Defense that can be installed on a laptop. We also offer a deploy-on-demand license for incident response teams (our 500-node pack has been quite popular), as well as the perpetual node model for full enterprise proactive deployments.

-Greg

Scripting with Responder™ Community Edition

One of the most powerful features of Responder (all three versions, including the free Community Edition) is the ability to write custom plugins. The entire application is basically a GUI over an API. You have the ability to access this same API and extend the application in any way. HBGary hasn’t produced an official SDK document yet, so it’s best to learn by example. For this exercise, I am going to illustrate a plugin that ties information from Responder into Google maps.

First, you should become familiar with the object tree. The object tree (shown in the graphic below, point A) illustrates how the data is organized within Responder after a physical memory snapshot has been reconstructed. You can query any of this data directly using the Responder API’s. For example, you could query low-level details about running processes (point B).



For this example, we are going to query the open network sockets. These are reconstructed from internal undocumented structures within the kernel (the same ones used by tcpip.sys and afd.sys). Even if a rootkit is hooking netstat, the data would still be revealed in Responder. In our example, we have some outbound connections to China. Using our plug-in, we are going to read the connection data and plot the location of the registering entity using Google Maps.



To load the script, first go to the script TAB and select OPEN. Once open, the script will be visible in a code-editing window. Press the PLAY button to load the script.



As you can see, the script is written in C#. Almost all of the GUI components in Responder are written using C# and, for those who haven’t tried it, you will find it to be very similar to Java. The language is very easy to learn and use.
After we load the plugin, the list of network connections are obtained along with registration data. The address of the registration is then plotted on Google Maps.



When a plugin is loaded, the OnLoad function will be called with a list of all open “Documents”. In Responder, a “Document” is a container for data. The architecture requires that the user-interface be decoupled from the data. For those of you with programming experience, you may recognize the “Document/View” pattern here. At any rate, the list of open documents is passed into the OnLoad function and we need to locate the “NetworkBrowserDocument”. The network browser document has the list of all open sockets.


public bool OnLoad(ArrayList OpenDocuments)
{
try
{
// get the frame document, this allows us to add menu items and menu bars
_frame = FindMainWindow(OpenDocuments);

// see the Launch() subroutine to learn how to launch your own popup window
Launch();

// init the whois class for later use
_whois.ResponderForm = (Form)_frame.MainWindowInstance;
_whois.Inspector = FindInspector(OpenDocuments);
// the network browser document gives access to open sockets
_whois.Net = FindNetworkBrowserDocument(OpenDocuments);


For those who want to explore other documents, there are several example plugins that ship with Responder. For example, "StringsBrowserDocument" is responsible for showing lists of strings associated with a livebin. "SymbolsBrowserDocument" is responsible for symbols when a livebin has been disassembled (Responder PRO only). The "DriversBrowserDocument" has the list of detected device drivers.

In this plugin example, we have a helper function defined to locate the network browser document. Notice we use GetType() to locate the actual type of each document in the list. As stated, there are many different document types in Responder, usually one type for every visible window or panel in the application.


Logic.NetworkBrowserDocument FindNetworkBrowserDocument(ArrayList documents)
{
// note the use of IDocument interface class here,
// use GetType() to compare instanced type against Logic.XXXX where
// XXXX is the document type you are after. Use reflection to see the
// whole list...
foreach (IDocument doc in documents)
if (doc.GetType() == typeof(Logic.NetworkBrowserDocument))
return (Logic.NetworkBrowserDocument)doc;

return null;
}


After finding the network document we can use it to query the list of sockets. Documents will have custom methods and utility functions for dealing with specific data (these are all different depending on document type). You can also access the raw data directly, usually in the form of name/value pairs (my preferred way to do it). This is shown below. Each attribute has a specific name and type as shown.


ArrayList socks = _net.Sockets();

// all objects are referenced by GUID
foreach (Guid socketEntryID in socks)
{
// src and dest ip are stored as string
string source = _net.ObjectName(socketEntryID, "sSource") as string;
string target = _net.ObjectName(socketEntryID, "sDestination") as string;

// remember that 'i' is UNSIGNED
UInt32 sourcePort = (UInt32)_net.ObjectName(socketEntryID, "iSourcePort");
UInt32 targetPort = (UInt32)_net.ObjectName(socketEntryID, "iDestinationPort");

// the src and dest DNS names, obviously string as well
string sourcename = _net.ObjectName(socketEntryID, "sSourceName") as string;
string destname = _net.ObjectName(socketEntryID, "sDestinationName") as string;

// a bool stores whether the session is TCP or UDP
bool bTcp = (bool)_net.ObjectName(socketEntryID, "bIsTCP");

string sockType = ((bool)(_net.ObjectName(socketEntryID, "bIsTCP"))) ? "TCP" : "UDP";


The socket list is stored as a list of object ID’s. Responder uses a GUID to identify every object in the project database. Every object that is found in the physical memory snapshot is assigned a GUID and can subsequently be looked up. In this example, we have a list of objects which represent sockets. The object ID can then be used to query additional attributes. In this example we query “sSource” “sDestination” “iSourcePort” etc. This is the generic attribute naming system used by Responder. The prefix is a type. ‘s’ means string, ‘i’ means integer, 'b' means bool. There are hundreds of these named attributes across the application - something I hope HBGary writes an SDK document for soon.

After obtaining the source and destination IP’s, our example plugin has a Whois class that is used to lookup the name and address of the registrar. This data is then passed to a browser control along with the URL for Google Maps so the location will be mapped on the right.

This plugin could be extended in many ways. For example, a geoip database or service like ip2location could be used to locate the missile-coordinates for a specific IP address, as opposed to the registration data. The plugin could also be extended to extract IP addresses from artifacts in memory, as opposed to active connections in the socket list. For example, IP address fragments stored in tagged page pool memory.

The plugin is open source and can be downloaded from HBGary’s support site.

Cheers,
-Greg

Ps. Thanks to Dean, the HBGary engineer who wrote this plugin