Post Execution - A New Paradigm for Debugging Malware

With the upcoming 2.0 Release of Responder, REcon plays a much more integrated role in the analysis of malware. The report automatically details all the important runtime behavior from a malware sample, including network activity, file activity, registry activity, and suspicious runtime behavior such as process and DLL injection activity. All activity is logged down to the individual disassembled instructions behind the behavior, nothing is omitted. Code coverage is illustrated in the disassembly view and data samples are shown at every location. This is like having a post-execution debugger, with registers, stack, and sampled data for every time that location was visited.

Post-execution debugging is a paradigm shift from traditional interactive live debugging. Traditional debugging is cumbersome and requires micromanagement to collect data. The traditional debugging environment is designed for CONTROL of the execution, as opposed to OBSERVATION ONLY. Typically, the malware analyst does not need to control the execution of a binary at this level, and instead only needs observe the behavior. HBGary's REcon approach to debugging is far superior because the analyst can see and query so much more relevant data at one time without having to get into the bits and bytes of single-stepping instructions and using breakpoints. It's like having a breakpoint on every basic block 100% of the time, without having to micromanage breakpoints.

HBGary and Palantir

I am very excited about our new partnership with Palantir. The Palantir link analysis capability is outstanding. Our team here in Sacramento is currently processing over 1.5 gigs of malware drops per day for Digital DNA, the new link analysis capabilities will allow us to move from the malware developer to the actual individuals who are operating the malware. We are already able to use forensic toolmarks to identify the individual malware developers, but this next step of analysis is to actually track those that have purchased or funded the development of the malware weaponry, and subsequently operate the malware in live operations. While tracking the developers themselves is easy due to the nearly impossible-to-avoid toolmarking caused by code and compiler, tracking the operators is much more difficult. But, HBGary is going to make a dent in this problem. We are approaching malware the same way an intelligence agency would.

Using Handle Tables in Physical Memory

One of the challenges we face at HBGary while developing Responder is the sheer volume of information available in physical memory. We have to reverse engineer a large volume of underlying data structures within the Windows operating system, not just for one operating system but for every single version of Windows (that includes service packs). In other words, a lot of time spent in windbg. One of the more interesting information sources available from the kernel is the handle table. Handle tables are extremely powerful and allow you to determine, for example, what files and registry keys are open by a given application.

Consider the way malware drops and installs itself. Many times, malware will inject itself into other processes as part of its "installation and deployment". At HBGary, we divide all the various behaviors of malware into categories we call "malware development factors". One of those malware development factors is called "installation and deployment" and it's all the stuff that malware does to survive reboot. In our Digital DNA(tm) we have well over 400 rules to detect just this kind of behavior. For example, malware may install itself as an extension to Internet Explorer and sniff online banking credentials. There are many ways to detect the presence of these capabilities. File handles can be used to detect when malware has been injected into secondary processes.

Malware injected into Internet Explorer opens a suspicious log file. This data is obtained from the kernel handle table.

Another anomaly that you can look for is a strange file path or executable name. Simply compare all the paths that are available in all the modules. In Responder, you can double click the drivers folder,m or the 'All Modules' folder, to view all drivers or modules along with their paths, respectively. Most modules are in consistent locations. When you examine all the module paths together in a single view, the anomalies will stand out. You don't really notice this until you see it in reference to all the other paths on the system. Seeing everything at once helps you detect an outlier quite quickly. This is just one of the types of things you can do when you have all the information at your fingertips.

Executables that are located in a suspicious temp directory. Also, one of executables has a non-standard file extension (.tmp).

A very suspicious kernel-mode device driver. This driver has no path, just an odd name with no file extension. This is a rootkit that was later identified as a variant of the "Black Energy" rootkit family. Notice that Digital DNA has automatically identified this driver as highly suspicious.

Executables that are running in a process that don't have a corresponding module name or path. These are very suspicious and we determined these were injected DLL's that were unlinked from the module list. Notice that Digital DNA has automatically identified these as highly suspicious.

One thing I really like about Responder is that you can sort the information in columns alphabetically and bring the outliers right to the top. Another thing that I like about Responder is that you can also write plug-ins that extend the functionality of the user interface at any point. For example, I could write a regular expression that would search all the file handles for certain patterns and that could include executables used in multi-stage installation, such as CMD.EXE or RUNDLL32.EXE, or files that appear to be in suspicious paths.

Regex r = new Regex(".*\\\\temp\\\\.*\\.exe$", RegexOptions.IgnoreCase);
Match m = r.Match(stringData.Name);
if (true == m.Success)
{
         // add an item to the report, associated with the DLL
 IReportObject wo = theDLLPackage.CreateReportObject(
             "suspicious exe in temp path",
             "This path looks suspicious, examine further");
} 

Responder scripts are written in C#.

I could even automatically add those to my report. The upcoming 2.0 release of Responder has an interactive report where you basically just drag and drop any specific data item to your report and drop it where you want the report item to appear. The dropped data actually just appears right there in the report at that location, including a description of what it is. Of course, this is editable by you, and you can expand upon it, but it makes it very easy for you to assemble a collection of those things that you find important in the memory snapshot. In summary, the report is a way to export and print the data I care about. I guess it also gives me a way to come back and reference those report items later (if you find yourself re-exploring old memory images). This is a short summary of some things you can do using the handles and the paths that are available to you in a physical memory snapshot.

Puffer Machines, El Al, and Defense in Depth

Airline security is a great case study in large systems security, and specifically the challenges of defense in depth implementation. While the U.S. will rapidly get back to business-as-usual, forgetting the near disaster of a Christmas day airline bombing, other countries have the threat of attack put in their face daily. This isn't something a person wishes the U.S. to experience, but you cannot ignore the ongoing annealing effect this has on the security posture of these foreign communities. It translates into large expenditures of money being applied to security because the threat is real. Contrast the TSA here in the States with El Al in Israel. El Al catches terrorists using multiple layers of security. The first defense is a knowledge of the world they live, the status of their enemies and who is likely to target them at any given time. This means putting intelligence to work and informing different organizations so they can work together. The second layer is a skilled person interrogating passengers. El Al realizes it is not a right to fly, it is a privilege. The screener focuses on the person. El Al respects the power of human threat detection by incorporating the interview into the screening process. The traveler may be asked to produce receipts for the places he reports to have stayed. If the person makes the screener nervous, that person gets set aside for more in depth screening. Simple. This increases the time it takes to check in, but this process has been proven effective. Humans are the best threat detectors in our known universe (seconded by our trusted animal companions). But, here in the States, we are so afraid of being accused of racial profiling and discrimination that TSA is forced to ignore human solutions, and instead relies on mechanical procedures and a compulsive focus on carry-on luggage. The next layer of security is technology based - if a traveler fails to pass the human screener, he or she may be asked to submit to a full body scan, a technology that raises hackles here in the States because of 'privacy' - never mind that it will actually detect plastic explosives taped to the body. The extra cost of sending would-be passengers through a puffer machine is easily shouldered by El Al, because they know it works at detecting explosives. The defense-in-depth goes even further: El Al has special reinforcements in the aircraft fuselage to protect the weakest point against an explosive blast. This is the kind of security that would make me feel safe to fly. All of this is expensive, time consuming and necessary, because it saves lives. The cost of security can't always be measured by money. Some things are more valuable like reputation, goodwill, and peace of mind. The U.S. should take some lessons from this, and start spending smart money on a few key defense-in-depth strategies that work, not only in our airline screening process but in our networks and infrastructure as well. Risk Intelligence is a lot cheaper to implement than we think if we consider the consequences.

Not Kind, Not Gentle. The turn of the decade in security.

The decade in review: The most painful thing we learned is that computer security hasn’t worked. We are, at this very moment, MORE insecure than we were in the year 2000. Billions of dollars were wasted on security technology that isn't working. In the last ten years, true cybercrime was born. Maybe we were just naïve about the coming storm. At the turn of the century, it was hard to get past the romantic idea of a university student hacker who prowled systems harmlessly for fun. Blocking ports and preventing network based buffer overflow attacks seemed so important. None of this technology prevented true criminals from pulling off the biggest heist in computer history – the massive theft of identity and subsequent banking fraud of the last few years. The traditional hacker is dead. Hackers are now called terrorists. The Russian mafia pays developers six figure salaries to write rootkits and malware. Independent researchers can and will sell a reliable working exploit of Internet Explorer for more than $50,000 USD. It began to hurt so bad that even Microsoft had to jump on the secure coding bandwagon, declaring a massive effort to make their code more secure. But this isn’t working either. You see, we are adopting technology at a rate far faster than we can secure it. By the time we have secured something, the landscape has changed and the attackers have moved on. In fact, that is why desktop exploitation has become the dominant attack vector. Over the last few years, malicious documents and media, especially “rich content” that contains embedded logic, parse-able metacode or script, and other logical constructs that can be malformed, emerged as the dominant method of exploitation. The API’s, COM objects, and other hoo-hah piled sky high on your windows workstation is a garden of carnal delights to a skilled attacker. Exploits of this nature have been mostly delivered via Internet Explorer and email. In fact, Internet Explorer is quite possibly the largest software disaster ever. As a software program, it has probably caused over a hundred billion dollars in damages since its release. This isn't about blame - if IE wasn't there, someone else's browser would have been the target. The browser is the portal into the Enterprise, so it's going to be where the bad guys focus. Finally, even before all this was going on, every nation state on the planet was standing in the shadows scared out of their britches. Smart people in high (low?) places could see the writing on the wall. It is TRULY AMAZING that a terrorist hasn’t hacked into the SCADA systems of a municipal power utility, started a cascade failure, and shut down half a state in the dead of winter. It’s because of this that I think [most of] those so-called terrorists aren’t very bright. As we close out the first decade, we must realize we have just entered one of the biggest arms races in the history of warfare. In fact, one can easily say that true cyber warfare was birthed in the last ten years.

So, now my predictions for the next ten years: Very early in the next decade, online identity theft and banking fraud will replace drug trafficking as the dominant criminal problem worldwide. Cyber cartels will make more money annually than drug cartels. Exploitation will continue to be focused on content-based delivery – that is, malicious documents & media. This will be coupled with a massive growth in online social networking. Trust, as a human concept, will be exploited as a means to spread malware throughout social networks via your online digital identity. Again, we will adopt new technology at a rate faster than we can secure it. The largest domain of attack will be software running on cellular phones. The phone will truly evolve into a network terminal – a slightly thicker thin client, loaded with more software in the palm of your hand than you could cram into a Windows 95 box in the year 2000. Yep, you guessed it, another garden of carnal delights – these new platforms will arrive unsecured – the development tools to make software will be insecure, and the people writing the code aren’t going to give a bug’s butt about secure coding practices. So, cyber crime is going to get a lot worse. Meanwhile, we are going to see at least one major SCADA based terrorist attack. We may have no idea that a terrorist did it, because the authorities will never admit it if they can plausibly lie, but it will happen. In fact, it may have already happened. Security spending will shift as well. Starting now, and reaching a heyday in about 6 years, security spending will shift towards host based security solutions. First the government, and then commercial enterprises, will realize that netflows and gateway solutions are not going to stop malware – it’s just too hard to predict what software will do without actually running it. And, online social relationships will be an extension of our professional identity - in other words, when an employee sits down at his workstation, his entire social network sits down with him. Network based security cannot hope to analyze complex documents and media, much less who to trust and when. Because everything will be hosted online, blocking content will effectively break the Internet, and looking inside the content will never happen at the network gateway (don’t invest in companies that think they can solve that problem). Concepts like malware-tolerance will become a hard reality, people will realize you can't keep the bad guys out. While the majority of online crime will continue to be in banking fraud, we are going to see industrial espionage and state-sponsored attacks in the press more than once. And, while banking fraud hurts the individual, the scope and damage of espionage is far far greater. Whether its classified state secrets or the recipe for Coke makes no difference, when the criminals out there figure out the value of information, they WILL steal it. The next ten years are not going to be kind or gentle to the security space. The hardest hit are going to be the biggest in the space – AV vendors are going to take the hardest fall. Their signature based solutions don’t work today, but not everyone knows that yet. But over time, that truth will seep farther into the IT space. So, perhaps my biggest prediction is this – AV will lose their place as the #1 security expenditure in the Enterprise. I’m not sure what will replace it exactly, but I do know that people are going to stop throwing good money after bad.

Blackhat Training is almost here!

I am gearing up for the Blackhat Training session on Monday-Tuesday of next week. We have made room for 30 students. We spent almost four weeks working on materials, remastering the demo and recap videos, and collecting malware samples that illustrated each of the subjects we are presenting. The task was alot harder than I originally expected, especially the collection of malware. I discovered a great trick using our feed processor, which is the clever use of search terms against strings to locate malware that were using specific techniques, keylogging methods, hooking styles, even specific languages. We have a solid methodology we teach behind our Responder product, so I had to find malware that illustrated specific concepts, as opposed to tailoring a training around whatever malware happened to be available. I will try to keep the training as high level as I can, and stay out of disassembly code as much as possible, but as expected there are some key reversing skills that can never be avoided, such as the reconstruction of parameters passed to a call. But, as for arithmetic and hard logic reconstruction, the only exercise where we get into that level of detail will be the one on crypto and stego. We have one coding exercise using the new built-in scripting interface, so thats a short bit of hardcore fun as well. But most of the material is about getting reverse engineering done rapidly, getting what you need, and not bogging down - which is the name of the game.

Reverse engineering process-injecting malware

I posted a video demonstrating some RE work with Responder:

Process Injecting Malware

The RE process starts by searching a livebin's symbols for "remote". A livebin is the in-memory version of an EXE as extracted by Responder. It’s not an executable format, but instead represents the exact layout of the PE formatted file once it loads into virtual memory. Section information does not need to be interpreted to remap the binary in this case, as the OS loader has already done that, including remapping and any other modifications that are made to the layout of code and data at runtime. Many malware programs will be packed on disk, but the livebin will contain large unpacked sections that can be analyzed without the RE having to know anything about the packing methods used, as in effect they are already unpacked for you.

The symbol "CreateRemoteThread" is of interest. For process injection malware you will find this API call almost 100% of the time. There are a few other API calls that are used in conjunction. We drag the symbol to the canvas and examine the region around it. Specifically, we see WriteProcessMemory and VirtualAllocEx - this is a dead giveaway that process injection is in use. Usually a malware will inject a thread that points to the function "LoadLibrary" with the first argument being a path to a DLL that was decompressed to disk - typically in a temporary directory. This is part of the malware's installation system.

In the example, we find that a PID is used to locate a process to inject into. We follow this PID and find the argument is used with LEA. In this case, the LEA is like using the address-of operator in 'c' code.

Imagine the following code:
void *myFunctionPointer;
some_function_call( &myFunctionPointer );


The "some_function_call" is getting a reference to myFunctionPointer, and this means it has the ability to initialize or assign a value into myFunctionPointer. The LEA instruction you see in the video is the assembly version of this same operation. We see this, and follow the function to find a loop where ToolHelpSnapshot32 is used. The toolhelp API set is another very suspicious behavior - if you see this in a potential malware you are very likely dealing with something that enumerates other processes on the system. This is usually a step prior to injection (or an attempt to find a virus scanner or firewall exe and kill it).

There is a string comparison in the process hunting loop - so the malware author is attempting to find a process by name. We follow the arguments back up and see that it's searching for "explorer.exe". The steps shown in the video require moderate-level RE skills, but are not daunting. With a little practice you can follow arguments in and out of function calls without losing your place. The trick is simply to remember that arguments are usually a positive base off of EBP, and local variables are a negative offset. "Parameters are Positive" - use that rule to remember.

The is one exception that is likely to drive you crazy - malware written in Delphi (and there is ALOT of that) usually passes parameters in registers. This can be harder to follow, but again if you label the arguments going into the function you can see these labels at the function boundaries so you don't lose your place.