Inside an APT Covert Communications Channel

Note: I shortened the title of the post from "Inside an APT “Comment Crew” Covert Communications Channel" to "Inside an APT Covert Communications Channel". To be clear, multiple threat groups are using HTML comments as a means of COVCOM. Thus, this should be considered a general technique as opposed to attribution on a specific group. Both Shady RAT and "Comment Crew", as well as others with additional codenames, have been associated with the use of HTML comments as a means of COVCOM.

For many years, hackers operating out of China have been attacking a myriad of commercial and government systems here in the US and abroad. The term “APT” or Advanced Persistent Threat has often been used to describe these attackers. While HBGary is primarily a product company selling an enterprise incident response product, the team has been deep into APT analysis for over five years. Most of the analysis work is in direct support of Digital DNA – an automated system for detection of unknown malware and APT intrusions. I presented a technical description of how this attribution works, what is solves and what it doesn’t, at the BlackHat Conference last year. The work is about tracking threat groups – that is, tracking the humans and the human factors behind the digital artifacts we see. There are many hacking groups involved in these intrusions. One such group has often been called “Comment Crew” for their use of HTML comments as a means of command and control. This group has been associated with the recent “Shady RAT” intrusion revealed by McAfee. For this article I am going to give you a technical in-depth tour of how such a group operates.

For starters, the attackers will gain access to the network via spear-phishing. In almost all cases we have investigated, spear-phishing was the initial point of infection. These phishing emails are full of very specific project names, names of associates, official sounding documents, etc. It is very clear that the hacking group is using stolen email to learn about their targets before crafting a very convincing email. This underscores why the recent spate of SQLi attacks over the last few months pose a far greater threat than most people realize.


Exploit and Dropper

Once access is gained into the network, the hacking group places remote access tools into the environment. These are backdoor programs that are downloaded automatically by the exploit email – we called these “droppers”. In the diagram, point A shows the exploit email ‘detonating’ after being viewed by the victim, point ‘B’ is a server where a ‘dropper’ is stored, and point ‘C’ is the dropper backdoor being placed onto the compromised computer.

Once the dropper has established a beachhead into the network, a hacker will access the host and uninstall the original backdoor, replacing it with a new and more powerful backdoor. These backdoors, especially the secondary and more powerful one, are called “RAT”s – for Remote Access Tool. Many of these RATs are custom written and that can be the basis for a great deal of attribution, allowing us to detect the malware in physical memory.


Interaction with the Host

Remember that most networks are firewalled. This means the attacker can’t just make a TCP connection into the RAT program. The RAT program is within the internal network so it must first make an outbound connection to the attacker. The RAT is designed to connect outbound over port 80 or 443, a port that is allowed outbound by almost all firewall policies. Once the outbound connection is made, the attacker can use the established TCP session to interact with the host, download tools, run command line programs, and laterally move about the network. In the diagram, point A is where the RAT makes an outbound connection to a server on the Internet, point B is a server under the hacker’s control, and point C is where the hacker uses the established TCP connection to interact with the RAT program and subsequently the host environment, potentially exploiting additional machines nearby in the network.

One of the greatest challenges for an incident response team is discerning the difference between ‘normal’ malware and an APT attack. As we can see in this example, an APT attack involves a real human at the other end of the keyboard performing actions on the host. We call this ‘interaction with the host’ and we recommend that an IR team pull a timeline of last-access times from the MFT (master file table), browsing history from index.DAT, event log, and other sources to determine if such interaction is occurring. This is a fast and easy way to discern the difference between a non-targeted external threat (which over 80% of all adverse events will fall into this category) and external targeted attacks (of which APT is included, probably less than 2% of all adverse events).

The RAT program doesn’t contain any fancy stealth or anti-forensics measures. In fact, we rarely even see packers in use (a packer is a method of obfuscating a program after compilation and is a low-cost way for a hacker to add anti-forensics to his malware). It seems the most of the covert methods are applied to the way to RAT communicates with the hacker. This makes sense. Consider that most of the intrusion detection capability lies at the perimeter of the network, and this is what the hacker is trying to defeat. Thus, the HTML comment method of configuring and controlling the RAT programs.


Hidden Comments for Covert Communication (COVCOM)

Instead of letting the RAT connect directly to his personal server, the hacker will first exploit a webserver somewhere on the Internet. This exploited webserver will then be used as the ‘middleman’ to communicate with the RAT. The hacker will place a hidden comment on an otherwise normal webpage and have the RAT connect outbound to this page. Using the hidden comment, the hacker will be able to give commands to the RAT. The RAT will make periodic outbound connections, sometimes waiting days before checking the page. The hidden comment will contain an encoded message that the RAT knows how to decipher. In this case example, the hidden data is base64 encoded. In this diagram, point A is the RAT program making a periodic outbound connection, point B is a compromised webserver somewhere on the Internet, point C is the hidden comment on the webpage, and point D is where said comment is decoded into actual instructions for the RAT. An example of such a comment is shown in the next image. It is interesting to note that the hacker has attempted to make the page look like a 404 HTML error page if viewed in a normal web browser.


Example of BASE64 Encoded Hidden Comment

Once the RAT decodes the message, the data becomes a configuration file for the malware. The file has many features, such as the ability to specify which server addresses to use on the Internet, including backup servers, configuration of the check-in times, and even has the ability to completely update the RAT binary in the field (shown in the diagram as a .bmp file – this is actually a normal PE header executable).


The Decoded Configuration File

All of the above technical information can be detected on a host after intrusion. The RAT program itself is near trivial to detect once you know what you are looking for. But beyond that, because the RAT program has certain outbound connection characteristics, sleep timers, and built-in “host interaction” capabilities, HBGary’s Digital DNA lights it up like a Christmas Tree (example shown in image).


Digital DNA Detects Unknown Malware

Even if you had no prior knowledge about this specific RAT, you would have detected it with HBGary. Beyond that, the decoded configuration file can also be found in physical memory – the primary search method used by Active Defense. Regardless of the configuration values, the option headers shown in the example above have a specific pattern that can be detected quite easily, even if fragmented over multiple buffers. This is exactly the kind of information I am referring to when I talk about “actionable threat intelligence”. Once you know about the attackers TTP’s (tactics, techniques, and procedures) you can encode this into an enterprise-wide scan. We call it ‘continuous protection’ when you adopt continual scanning while also updating the threat intelligence as you learn more about the attacker. In essence, you are applying attrition against the attacker’s presence in your network. For example, if you know how to detect the above configuration file, then the attacker has to change the way that configuration file looks to defeat you – something that also requires them to recode their parser in the malware. Hence, you cost the attacker time and money. That is a Good Thing.

I hope this gave you a somewhat concrete tour of how a real APT covert communication (COVCOM) channel works. Also, I hope it has illustrated some of the threat intelligence that you access on the host. Using enterprise-wide scans, your IR or security team can put a severe dent in the APT presence in your network. As far as product solutions to enable you, obviously we build HBGary’s Active Defense. If you are interested in continuous protection and threat intelligence, we offer 50-node evaluations of Active Defense that can be installed on a laptop. We also offer a deploy-on-demand license for incident response teams (our 500-node pack has been quite popular), as well as the perpetual node model for full enterprise proactive deployments.

-Greg

Shady RAT is Serious Business

Ira Winkler makes some interesting points in his CIO article on Shady RAT. I tend to agree with his observation that security vendors spend too much energy infighting when we all should be facing a common enemy. It is true that Shady RAT is just one of many other, similar attacks. There is no harm in trying to draw attention to the elephant in the room - APT is a grave and serious threat to U.S. companies as well as national security. Shady RAT may appear to be 'sloppy' but it can still be APT. Within infosec the term APT has been debated - but we at HBGary have a very simple definition: if there is interaction with the host, we call it APT. Now, most of the attacks we deal with are targeting intellectual property and appear to have state sponsored underpinnings. The attackers usually leave tools behind, additional backdoors, etc., but none of these are very complex. The malware and techniques are mostly unsophisticated and sloppy, but yet they succeed and remain persistent. Our assumption on this - APT does the minimum necessary to get the job done. If they don't need hard core boot sector viruses and kernel rootkits, they aren't going to use them. We as an industry have a responsibility to protect our customers from a very serious and evolving threat. Downplaying the seriousness of this threat undermines the reason we are here.

-Greg

Command Line Programming with Responder PRO

One little known feature of HBGary’s Responder product is that it ships with the full source code to a command-line version. This command-line version of the product can be customized for automated tools, batch processing, and statistical utilities. HBGary is still working to produce an 'official' documentation on the SDK, but in the meantime I figured I would walk the more adventurous of you through some code.

First you need Microsoft Visual Studio. I use VS2008 Pro Edition with version 3.5 SP1 of .NET. In the SDK subdirectory of your Responder installation, you should find the ITHC directory. Just a backstory, but ITHC means Inspector Test Harness Client – it was originally a test harness used by our QA team that eventually proved so useful for batch processing that we included it for customers. The code is written in C#.

When I first opened the .sln file on my Responder install, I found that the project file needed some tweaking. Your mileage may vary, but here are some steps I had to take. First, the references to all the Responder DLL’s were broken. By editing the .csproj file I was able to fix this. The trick is to use a HintPath variable with a relative path to the main install directory, which is two folders above the ITHC directory (see image). I’m not sure why it shipped this way, but alas I was able to fix it.


Fixing the references

Now, in most cases, I like programming in Debug mode so I can single step, use breakpoints, inspect variables, etc. I ran into a snag with my debug build and had to get one of the HBGary engineers to take a look. Again, it was a configuration thing. When you make build settings, the platform will probably be set to AnyCPU. You will need to set the platform target to x86 (see image). This has something to do with mixed mode code and if you don’t set this to x86 you will get a binding error when you attempt to run the ITHC exe. Lastly, I set my output path so the ITHC.exe ended up in the main Responder install directory (see image).


Setting the platform target


Setting the output path

Running the tool requires some precise command line arguments (see image). The project path needs to be as shown path/projectname/projectname.proj and the path to the memory image needs to be fully qualified. If you want to change any of that, you can edit the code in NewProject() and OpenProject() to parse the path differently. At this point I had a fully functional ITHC.exe that would analyze Windows physical memory snapshots.


Command line parameters to the tool

Most of the analysis magic happens in THCAnalyzeFile(). The project file ends with the .proj extension and this will be created or opened if it already exists. There is also a .tmp file that contains cached lookup data for Responder which only exists after an analysis. THCAnalyzeFile() will handle all of this.

At this point I need to explain packages and classes. In Responder, a package is any binary object. For example, the physical memory snapshot is a package. Every extracted livebin is also a package. If you import a file for static analysis, that file is considered a package.

Both packages and classes can have parent/child relationships. The difference is that a class is simply a container without any associated binary data. Think of it as just a folder. In fact, in the Responder GUI, classes are shown as folder icons. Just remember that packages can have child classes, classes can contain other classes, classes can contain packages – there is no restriction on the way you nest these objects.

Around line 249 in the ITHC example you will see the creation of the root package (see image). Every project has a single root package that everything else will reside under. Usually this package has no associated binary object and is simply a placeholder. We usually set this to the name of the forensic case – such as “Case 04321”. In Responder’s GUI, the root package is always shown with a safe icon. Depending on the project type, a class will be created directly under this root package. The name of this class is very important and affects the kinds of things Responder will let you do. So, for a physical memory analysis you need to name this first class "Physical Memory Snapshot". You will see this created around line 266.


root package, bulk update, named attributes

Now just a word on event management. Responder has a robust event alerting system that will post an event to your code whenever an object is modified. You could subscribe to these events and be notified if the user changed a property of an object anywhere in the GUI, for example. But, there is a flipside – if you make a large number of changes all at once you will flood the system with these messages. Most of the time if you are going to change a bunch of objects all at once, you want to disable events for a short time. To do this, you use the BeginBulkUpdate() and EndBulkUpdate() methods. You will see these in use around line 249 (see image).

Around this same section of code you will also see named attributes being set on the case. These attributes are being applied to the root package, the one that shows up as a safe icon when you view it in Responder’s GUI. Any object, including packages and classes, can have named attributes set. The attribute system is typed and the first letter of the name indicates the type. See my previous post on plugin development for a description of these.

Around line 293 you will see the creation of a second package. This package is the one associated with the physical memory snapshot. It is placed under the root node and folder. You will also see the creation of something called a snapshot that is then linked with the package. This is how you link a binary to the package – via the snapshot object. The snapshot is just a small header of metadata that is associated with the binary file – including the path to the file – and this is set as the “.InitialSnapshot” property of the package. After this step, the package and the binary are linked.


package and snapshot for the physical memory image

The most important function is then called – the AnalyzeMemory function (around line 329). This function performs the bulk of the memory analysis. It returns true or false depending on whether it understood the memory snapshot. Just a note; it will return false if you don’t have a valid license. If you have the free version of Responder CE, you still have a license file that must be present or this call will bail out on you.

After analysis is complete, the analysis history is updated to include “WPMA”. This tells Responder that “WPMA” analysis has already completed, so it won’t attempt a second analysis later. Note: WPMA means Windows Physical Memory Analysis. Responder has other analysis types that can be added to this history. You can also add your own for reference later.

Now that analysis is complete you can parse the datastore, query all the found windows objects, processes, modules, etc. You can also query the DDNA results if you are using the Pro version. Some object types, such as control flow, disassembly, dataflow, graph objects, and recon traces are only available in the Pro version. However, the results of the windows memory analysis are fully available in all versions, including the free CE version. See the THCDumpProject() function for more information on parsing the project’s object tree.


Package: ws2_32.dll
Parent Package: svchost.exe
Length: 0 bytes.
Class: Symbols
Class: Strings
Class: Report Items
Class: Global
Strings:
Package: vmwaretray.exe
Parent Package: VMwareTray.exe
Length: 0 bytes.
Class: Strings
Class: Global
Class: Report Items
Class: Symbols
Strings:
Package: msctf.dll
Parent Package: IEXPLORE.EXE
Length: 0 bytes.
Class: Strings
Class: Symbols
Class: Global
Class: Report Items


a short snippit of output from the THCDumpProject() function

For those of you using the Pro version, ITHC includes examples of not just physical memory analysis, but also extraction of livebins and code-level analysis of extracted livebins. If you made it this far, then take a look at AnalyzePackage(), AnalyzeExtractedPackage(), and ExtractPEImageFromMemory() to get more familier with the code level analysis features. I hope that I can write some more specific posts about these features in the near future.


ITHC.exe analyzing a memory snapshot

Because the ITHC utility is written in C# it’s very easy to interface to other systems. Microsoft has done a good job building a robust set of API’s that can be used for SQL database access, serializing files, communicating over the web or TCP/IP, regular expressions, etc. All of this is at your fingertips and can be interfaced with the results of physical memory assessments. I am partial to building bulk analysis tools for large directories of memory snapshots. You are only limited by your imagination.

The SDK directory should be in your Responder install directory. If you are using the free Community Edition you may not have the SDK directory. In this case you can download the SDK as a small but separate download from the free tools section on HBGary's support site. Visit www.hbgary.com for more information.

Asymmetric Warfare and Cyber Terrorism

In the newly released document, “DoD Strategy for Operating in Cyberspace", the Pentagon states that “while the threat to intellectual property is often less visible than the threat to critical infrastructure, it may be the most pervasive cyber threat today.” Pervasive, yes – but not necessarily the most dangerous.

In 2003, I founded my company, with the help of the federal government’s Small Business Initiative Research (SBIR) program, to develop products to counter these advanced unknown, stealth cyberthreats today often referred to today within the security community as Advanced Persistent Threats (APT).

While the APT threat is significant, the attacker can take months or even sometimes years to steal the information. However, the recent attacks made by small hacking groups illustrate a highly more tangible, immediate, and potentially more severe form of economic damage. It is appropriate to classify these acts as asymmetric warfare, and possibly as a type of cyberterrorism.

In contrast to APT threat actors and other traditional cyber criminals, cyberterrorists are not motivated by monetary gain. Instead, the cyberterrorist wants to cause grave harm or economic damage as quickly as possible, and to get attention for it. Attacks may be economic, political, or even shutting down the power in the dead of winter. The technical aspects of the attack may be similar to APT, but the intent and goal is wholly different.

Cyberterrorism first was a buzzword in the late 90’s associated with power outages and explosions orchestrated over computer networks. These types of attacks seemed like the digital equivalent of IED’s. While traditional terrorists clearly use the Internet to recruit and communicate, we operate under the assumption that the ‘ground of action’ is still the physical world – think suicide bombers. But, recent events have shown that attacks don’t have to be kinetic to cause damage. The ground of action can be entirely in cyberspace and damages can be measured in billions of dollars of stock value and the threats to persons are very real.

Edit: There are different views on the definition of cyberterrorism. In 'Computer Attack and Cyberterrorism: Vulnerabilities and Policy Issues for Congress', Clay Wilson defines two forms of cyberterrorism:

Effects-based: Cyberterrorism exists when computer attacks result in effects that are disruptive enough to generate fear comparable to a traditional act of terrorism, even if done by criminals.

Intent-based: Cyberterrorism exists when unlawful or politically motivated computer attacks are done to intimidate or coerce a government or people to further a political objective, or to cause grave harm or severe economic damage.

Since the early 2000’s, ‘electronic jihadists’ (i.e., Younes Tsouli, Mohammad Peerbhoy, etc) and other hacking groups (many can be researched on www.zone-h.org) have been content with web defacement and the occasional DDOS. But, these actions never gained the media attention like the recent spree of hacks in 2011. This is, in part, due to the advent of social networking. Former British Prime Minister Margaret Thatcher once stated “Publicity is the oxygen of terrorism”. Anyone studied in matters of terrorism knows that the primary goal of terrorism is media attention. The act is secondary to the message.


Younes Tsouli and Mohammad Peerbhoy, both criminal hackers working with Islamic extremist groups (photos via Associated Press)


A small sampling of criminal hacking groups operating in the Middle East. All of these groups are at least as-skilled as the current Lulzsec/Anonymous hackers, as evidenced by similar techniques, use of SQL injection, etc. The myth that traditional terrorist groups don't have access to hacking skill is simply outdated. (groups via zone-h.org)

In the words of William Gibson, “Terrorism is ultimately about branding”. Every press release, tweet, and claim is part of that brand to raise awareness for their cause or message. And, the media can function as an extension of the group’s propaganda machine. As TechCrunch columnist Paul Carr recently pointed out in his piece on the media coverage of the now defunct LulzSec group, most journalists were all too happy to hop aboard the ‘Lulz Boat’ and parrot propaganda verbatim without a hint of criticism and provide ‘celebrity fluff’ reporting. Paul especially calls out online journalists and bloggers as “downright shameful” for showing support for these criminal hackers. Gene Spafford, the professor and director at Purdue University and a leading security expert, has also objected to how reporters romanticize criminal hackers, drawing a parallel to computer virus authors in the early 90’s portrayed as “swashbuckling, electronic pirates” (pointing out that their legacy is now costing billions in damages).

Even in recent days, reporters have used lofty, inconsistent terms such as “masked crusaders,” a “loose hacker movement” and an “online activist group” to describe Anonymous. The fear of retribution by the criminal hackers within this group is real. No one wants to become a target. News organizations need to take a step back and take a close look at how they are covering these incidents and make sure they aren't enabling these groups’ propaganda machine.

Edit: as a case in point, notice the significant lack of the word 'criminal' when media reports on Anonymous/Lulzsec. To illustrate, here is how reporters/bloggers described Anonymous in the 24 hours following the Monsanto/Booz Allen Hamilton attacks:

"Online activist collective" - CNET
"hacker group" -- IT Business Edge
"Hactivist collective" -- The Inquirer
"Hacking Group" -- MSNBC
"Hacktivist Group" -- SC Magazine
"Hacker Group" -- WSJ
"Hacker Group" -- Network World/IDG
"Notorius Hactivist Collective" -- The Register
"Group of hactivist computer-savvy hackers" -- Economist
"Loose-hacker movement" -- Forbes
"Masked crusaders" -- Time
"Cyber-activist group" -- Financial Times
"Hacker Group" -- Dark Reading
"Online Activist Group" -- Associated Press
"Hacker Group" -- BBC News
"Hacking collective" -- NY Times
"Hacker Group" -- Washington Post

While the threat landscape is always changing, we must continue to highlight that a real criminal is at the other end of the keyboard, and that he is persistent and will keep coming back. While the DoD outlines some important initiatives for a more secure cyberspace, we, as citizens, also have a role. Just as we all participate in our local neighborhood watch to keep our physical community safe, we, as Internet users, need to be vigilant and work together to ensure our cyberspace remains safe.

-Greg Hoglund

Scripting with Responder™ Community Edition

One of the most powerful features of Responder (all three versions, including the free Community Edition) is the ability to write custom plugins. The entire application is basically a GUI over an API. You have the ability to access this same API and extend the application in any way. HBGary hasn’t produced an official SDK document yet, so it’s best to learn by example. For this exercise, I am going to illustrate a plugin that ties information from Responder into Google maps.

First, you should become familiar with the object tree. The object tree (shown in the graphic below, point A) illustrates how the data is organized within Responder after a physical memory snapshot has been reconstructed. You can query any of this data directly using the Responder API’s. For example, you could query low-level details about running processes (point B).



For this example, we are going to query the open network sockets. These are reconstructed from internal undocumented structures within the kernel (the same ones used by tcpip.sys and afd.sys). Even if a rootkit is hooking netstat, the data would still be revealed in Responder. In our example, we have some outbound connections to China. Using our plug-in, we are going to read the connection data and plot the location of the registering entity using Google Maps.



To load the script, first go to the script TAB and select OPEN. Once open, the script will be visible in a code-editing window. Press the PLAY button to load the script.



As you can see, the script is written in C#. Almost all of the GUI components in Responder are written using C# and, for those who haven’t tried it, you will find it to be very similar to Java. The language is very easy to learn and use.
After we load the plugin, the list of network connections are obtained along with registration data. The address of the registration is then plotted on Google Maps.



When a plugin is loaded, the OnLoad function will be called with a list of all open “Documents”. In Responder, a “Document” is a container for data. The architecture requires that the user-interface be decoupled from the data. For those of you with programming experience, you may recognize the “Document/View” pattern here. At any rate, the list of open documents is passed into the OnLoad function and we need to locate the “NetworkBrowserDocument”. The network browser document has the list of all open sockets.


public bool OnLoad(ArrayList OpenDocuments)
{
try
{
// get the frame document, this allows us to add menu items and menu bars
_frame = FindMainWindow(OpenDocuments);

// see the Launch() subroutine to learn how to launch your own popup window
Launch();

// init the whois class for later use
_whois.ResponderForm = (Form)_frame.MainWindowInstance;
_whois.Inspector = FindInspector(OpenDocuments);
// the network browser document gives access to open sockets
_whois.Net = FindNetworkBrowserDocument(OpenDocuments);


For those who want to explore other documents, there are several example plugins that ship with Responder. For example, "StringsBrowserDocument" is responsible for showing lists of strings associated with a livebin. "SymbolsBrowserDocument" is responsible for symbols when a livebin has been disassembled (Responder PRO only). The "DriversBrowserDocument" has the list of detected device drivers.

In this plugin example, we have a helper function defined to locate the network browser document. Notice we use GetType() to locate the actual type of each document in the list. As stated, there are many different document types in Responder, usually one type for every visible window or panel in the application.


Logic.NetworkBrowserDocument FindNetworkBrowserDocument(ArrayList documents)
{
// note the use of IDocument interface class here,
// use GetType() to compare instanced type against Logic.XXXX where
// XXXX is the document type you are after. Use reflection to see the
// whole list...
foreach (IDocument doc in documents)
if (doc.GetType() == typeof(Logic.NetworkBrowserDocument))
return (Logic.NetworkBrowserDocument)doc;

return null;
}


After finding the network document we can use it to query the list of sockets. Documents will have custom methods and utility functions for dealing with specific data (these are all different depending on document type). You can also access the raw data directly, usually in the form of name/value pairs (my preferred way to do it). This is shown below. Each attribute has a specific name and type as shown.


ArrayList socks = _net.Sockets();

// all objects are referenced by GUID
foreach (Guid socketEntryID in socks)
{
// src and dest ip are stored as string
string source = _net.ObjectName(socketEntryID, "sSource") as string;
string target = _net.ObjectName(socketEntryID, "sDestination") as string;

// remember that 'i' is UNSIGNED
UInt32 sourcePort = (UInt32)_net.ObjectName(socketEntryID, "iSourcePort");
UInt32 targetPort = (UInt32)_net.ObjectName(socketEntryID, "iDestinationPort");

// the src and dest DNS names, obviously string as well
string sourcename = _net.ObjectName(socketEntryID, "sSourceName") as string;
string destname = _net.ObjectName(socketEntryID, "sDestinationName") as string;

// a bool stores whether the session is TCP or UDP
bool bTcp = (bool)_net.ObjectName(socketEntryID, "bIsTCP");

string sockType = ((bool)(_net.ObjectName(socketEntryID, "bIsTCP"))) ? "TCP" : "UDP";


The socket list is stored as a list of object ID’s. Responder uses a GUID to identify every object in the project database. Every object that is found in the physical memory snapshot is assigned a GUID and can subsequently be looked up. In this example, we have a list of objects which represent sockets. The object ID can then be used to query additional attributes. In this example we query “sSource” “sDestination” “iSourcePort” etc. This is the generic attribute naming system used by Responder. The prefix is a type. ‘s’ means string, ‘i’ means integer, 'b' means bool. There are hundreds of these named attributes across the application - something I hope HBGary writes an SDK document for soon.

After obtaining the source and destination IP’s, our example plugin has a Whois class that is used to lookup the name and address of the registrar. This data is then passed to a browser control along with the URL for Google Maps so the location will be mapped on the right.

This plugin could be extended in many ways. For example, a geoip database or service like ip2location could be used to locate the missile-coordinates for a specific IP address, as opposed to the registration data. The plugin could also be extended to extract IP addresses from artifacts in memory, as opposed to active connections in the socket list. For example, IP address fragments stored in tagged page pool memory.

The plugin is open source and can be downloaded from HBGary’s support site.

Cheers,
-Greg

Ps. Thanks to Dean, the HBGary engineer who wrote this plugin

Changing APT Tactics: Remote-Access Tools vs. Stolen Credentials

Advanced Persistent Threats (APT) are adaptive, their tactics will cycle after an intrusion takes place. For example, an APT group may start to lean away from RATs (remote-access tools) and rely more on stolen credentials. Let me explain.

An APT initially will enter the network via malware, typically through spear-phishing. Once on the compromised host, the threat actor will place one or more RATs into the environment. If we pick up RATs with our Digital DNA solution or another indicator, we start hunting them down.  After targeting and removing these RATs in the customer environment, we have found that specific malware will last about a week, maybe two, before the APT drops it altogether and switches tactics to remain in the network. We commonly see APT shift to using stolen credentials and no malware at all.

Stolen credentials are the very currency of APT. As it turns out, it’s much harder to detect malicious users than to detect RATs. In fact, the APT will use these accounts the same way a legitimate admin would – making it very hard to tell the difference. They create file shares,  use the ADMIN$ share, and defrag the hard drive. APTs will even update the AV and patch the machine. Of course, the defrag is actually a way to cover up forensic evidence on the drive, and the ADMIN$ is a way to laterally move malware and tools between machines. One would think that upgrading the AV would be counter to an APT’s self preservation. Actually, the APT updates it purely for self preservation – to appear “normal” as a legitimate admin.

At this point in the investigation, in terms of malware, we are still picking up a great deal of material – but not RATs. When the APT shifts to credentials, we start to pick up password sniffers and keyloggers that have no outside network capability. The malware in this case is entirely focused on obtaining more credentials. Finally, once the customer updates all the passwords, one or more RATs pop out of the woodwork and the cycle repeats itself. 

 

A Brief History of Physical Memory Forensics

Lately, we have been doing a lot of work around physical memory forensics. Recently, we released the free, community edition of our Responder™ product and plan to release the fourth generation of our memory analysis engine later this year. During this work, I have been reflecting on the origins and advancements in the field of physical memory forensics over the last 10 years.

In the early 2000’s, two headline-making malware infections, Code Red and SQL Slammer, demonstrated the possibility that malware could reside only in memory and never leave a file on disk. In the world of incident response, the evidence challenged the traditional notion of dead-box forensics. It meant that critical data would not be obtained by the traditional forensic methodology. It also set the stage for future malware that would subvert API calls, forcing live response scripts to rely on the OS as little as possible.

Physical memory analysis started as crash dump analysis for debugging, but it soon became apparent that volatile data in memory could contain encryption keys, passwords, and other critical information about recent user activity. From a tools perspective, the well-known dd utility has been able to acquire memory from the start, simply by reading /dev/mem or /device/physicalmemory. Other memory tools also emerged. In 2002, Eoghan Casey documented how Arne Vidstrom’s PMDump tool could be used to dump virtual memory and defeat PGPTray.

Rootkits helped drive development of memory forensics –more for malware detection than evidence collection. In 2003, Jamie Butler demonstrated the DKOM (Direct Kernel Object Manipulation) method for hiding processes by removing items from a linked list directly in memory. This was a data-only attack and didn’t involve any kernel hooking. It would be a few years before researchers like Andreas Schuster and Chris Betz developed memory-forensics methods for finding hidden processes that countered Butler’s DKOM . Things took another significant step forward in 2005 when Sherri Sparks released Shadow Walker, a rootkit that was able to hide sections of virtual memory from scanning tools. This lead to the notion of physical memory acquisition – using a raw dump of RAM instead of using OS- supplied virtual memory reads – as a means for rootkit detection.

Attempts at OS reconstructions didn’t really start until the DFRWS memory analysis challenge in 2005, where George Garner [kntlist] and Chris Betz [memparser] developed process and thread reconstruction for Windows®. Everything changed after this – instead of searching for binary patterns and strings, the memory image was seen as a complex snapshot of interrelated structures and data arrays. A keystone development was the ability to discover the page tables in physical RAM and thus translate virtual addresses to their physical offset. In February 2006, I wrote the first version of this technology for HBGary using the self-referencing physical address pointer trick (AFAIK first publically documented by Joe Stewart w/ the TRUMAN project), and we soon added PAE support. Physical memory forensics had become a hot new area of research. Later that year Mariusz Burdach presented on physical memory forensics at the Blackhat conference. Jamie continued his research as well and presented numerous advances in physical memory analysis to detect rootkits at the Blackhat 2007 conference. Shortly after Jamie’s talk, AAron Walters released Volatility. It were these initial advances with page table translation and OS reconstruction that lead to ”modern” physical memory analysis.

By this time, Brian Carrier and Joe Grand had already released Tribble, a PCI card that could monitor and analyze physical memory. It was later that several commercial attempts were made to build a rootkit protection solution in the form of a PCI card. Via a DHS grant, HBGary was subcontracted to work on a similar project and this lead to a prototype PCI card that could analyze Windows XP and detect kernel hooks. Jamie Butler joined Komoku, which had already built a similar device, around that time. Joanna Rutkowska was quick to respond to all of this and developed an extremely low level software-only rootkit for Windows that could defeat even a PCI-based physical memory read – by reprogramming microchips that are part of the bus controller and I/O chipset. In the end, a hardware solution for rootkit detection was not economically feasible and these projects were never successfully commercialized.

HBGary’s work on the hardware PCI card was the genesis for more R&D memory forensics work to come. We abandoned the hardware approach and developed a software library called WPMA (Windows Physical Memory Assessment) - written in C++ and core to Responder’s memory parser. We later developed a second-generation parser and started reverse engineering all the different memory footprints left by every conceivable version of Windows and service pack (we didn’t analyze NT 4.0 – only Win2K and newer). It took about two years to get the Windows platform complete. This work led to the development of our flagship product, Responder™, and the library that performs the physical memory parsing is integrated into our enterprise product’s Active Defense™ agent as well.

I’ve highlighted only a few of the researchers in this important field of physical memory forensics – there are many others who have also made significant contributions. At HBGary, as I mentioned, we will soon release a completely rewritten version of our physical memory analysis engine marking the fourth generation of the technology. Recently, I was watching the performance testing in the lab and I have yet to see it cap 150 MB memory usage while analyzing a 10-gig snapshot, and it is about 30% faster than our current generation. I will post more details on this work as we progress, as the new engine has many additional features that extend our Digital DNA™ technology.

-Greg Hoglund