Malware Persistence in the Cloud

The cloud is certainly going to change some things about malware infection. When a desktop is reset to clean state every time an employee logs in, you now have to wonder how malicious attackers are going to maintain persistent access to the Enterprise. This is similar to what happens when an infected computer is re-imaged only to end-up infected all over again.

There are several ways to maintain persistent access without having an executable-in-waiting on the filesystem. Memory-only based injection is an old concept. It has the advantage of defeating disk-based security. One common observation is that such malware doesn't survive reboot. That is true in the sense that the malware is not a service or a driver - but this doesn't mean the malware will go away. Stated differently, the malware can still be persistent even without a registry key to survive reboot. This applies to the problem of re-infection after re-imaging (a serious and expensive problem today in the Enterprise) and it also applies to the future of cloud computing (where desktop reset is considered a way to combat malware persistence).

The most common method for persistence without reboot is re-infecting the system from a neighboring, already infected system. It has sometimes been called the "Hack Finn" model - two or more malware programs that know about each other. Unless you kill both of them simultaneously the one will re-create the other. In today's world, the neighbor doesn't need to be physically nearby - it can be anything that has some access path to the other machine. This neighbor could be a social networking peer, a shared desktop (think exploited .ini), or a machine with lateral domain credentials.

Another way to maintain access is to store crafted (exploit) data in a commonly used document - think PDF exploit but for google docs. User's in a cloud based environment are going to have persistent data storage, whether this is up in the cloud or down on a USB stick. When the execution environment is constantly reset, as it might in a desktop cloud, the attacker can move method of persistence to the data itself. The malicious code must obtain execution cycles - think of the cloud based desktop simply as an execution space. The user opens said boobytrapped document every day as part of their work, and the malicious code activates. Or it can be delivered via a system used on a daily basis, such as an exploited image on an ad-banner, or the little calendar program in the corner of your timecard system.

For the window of time the user is interacting with the desktop, the code has execution cycles. This is when data is most at risk - this is when other documents are open, other social network contacts are online, and the user's access token is live and can be used to access other resources.

Remember, the attackers always adapt to new environments. The cloud just provides new ways for our adversaries to attack us.

Intel Validates the End Node

In the security industry, there is an obsession with the perimeter, which is why this Intel-McAfee announcement, at first glance, seems so surprising. McAfee represents visibility and control at the end node - the environment where the bad guys actually live. Intel's investment into end-node security is significant because it represents a fundamental shift in security . Adoption of mobile technology is faster than ever before and the chip-war cannot continue to be based solely on horsepower or power consumption - there needs to be a platform and security has to play a big part. An integration between hardware and software is natural and can create a platform for the future mobile user. To understand this acquisition, you have to think ahead. Intel knows the future is in small, always-on access-to-the-cloud -- and they can obviously deliver the horsepower required for immersive environments such as High-Def, 3D that will become the norm. But the future is not without risk. To create a secure online experience for the user, Intel understands that the security features in the chip need to be united with application layer surfaces. While the benefits for Intel are clear, this deal also might be the best thing that ever happened to McAfee. They have the domain knowledge about hooking into software flows and scanning for patterns, but the end-node for McAfee is the PC -- not the future devices. Intel's business has always been about massive scale and this might push McAfee into the future. This event may signal an emergence period of security integration into the end node. The perimeter is vanishing daily - this event just validates that the end-point is where the action is.

Blackhat is almost here again! This has always been one of the coolest security shows with great research presentation. The show is a great place to catch up with friends and colleagues (most of whom I only see once a year). In many ways, Blackhat is the reason I decided to get into the security industry back in the late 90's - I lived very near Jeff (the founder of Blackhat and Defcon) and he inspired me to reach for something beyond just a network admin job. I am glad to see that both Jeff and Blackhat have survived. Hope to see you all at the show!

A dose of clarity for the "APT"

Finally a dose of clarity for the "APT". It is an overused word, one used to sell security products, even if these are the 'same' security products you have been using for the past 10 years. In his recent Spotlight report, Josh Corman of The 451 Group really laid out the term, where it came from, what it means, and more importantly WHAT IT DOES NOT mean. I posted a similar blog on the topic a while ago and got comments like “it’s the person, not the malware”. I know that. I’ve been saying that for years, but how the term APT is used today by most people make it sound like it's ONLY malware. In fact, it’s not only malware - it’s the actors and their intent. Josh gets this -- more importantly he felt the need to speak up about it. I agree, it’s about the ADVERSARY. Malware is just a tool, one of MANY that these adversaries use. Focusing on one aspect of security is not going to make you secure; rather, it’s understanding what they are trying to get. I would argue a 'slightly' different take in that I don’t necessarily believe it’s only scarce resources these adversaries are after. They want actually anything that gets them 'closer' to the info they are seeking. This could be money, IP, marketing plans, hiring plans, IT resources, or personally identifying information. Because while APT were at one time ONLY focused on military, they’ve expanded.

I also applaud Josh’s note that APT uses existing tools. Other experts seems to think this is not the case, or that they don’t use packed malware, or that APT don't use botnets. Why wouldn’t they? It seems the more that someone tells me what APT isn't, the more it becomes clear they have no idea what APT really is. If APT use existing malware, which I’ve always maintained, then packing is par for the course, because it's a cheap way to defeat signature based detection definitions at the gateway and host alike. Perhaps the APT did some recon into the network and learned that using XYZ packer would defeat the AV solution at the desktop. The US government created the term APT to characterize a class of threats originating from Chinese and Russian state sponsored and criminal elements, not to classify a type of malware.

Since the government coined the term "APT" it has always been about Russian and Chinese attackers, BOTH criminal and state sponsored. For the government, it's very difficult to draw a line between the two. If you understand information operations, then you know that APT will use any and all means at their disposal to achieve the mission objective. If this means use of packers, so be it. The same applies to _any_ rule or definition someone puts in my face telling me what APT is and is not. An IO campaign will include a full spectrum of capabilities. In the context of cyber, each attack on a government facility, contractor, or commercial entity could be a single operation that is part of a larger campaign. Operations could be designed to assume false personas such as impersonating college students in a dorm room, or even a false-flag - impersonating the intelligence service of another foreign country. If you truly know what APT is about, you know that you can't start boxing it up and packaging it.

As Josh pointed out, the adversaries are constantly evolving and adapting. The old models are being defeated. The government has known about APT for a while and that new approaches to enterprise security had to emerge. This is one of the reasons that the USAF and DHS both funded HBGary to address this evolving threat. Our company and our technology are pushing the envelope forward.

Cyber Threat

I imagine cyberspace like in the movies, as points of light. It's like looking down from an airplane at night - mostly vast darkness but interspersed with brightly lit areas of activity. There is a lot going on in these brightly lit areas: data being moved, software executing. In real life, there are people down there doing people stuff. But in cyberspace, the people are represented by software entities. Software is created by people, and thus is an extension of human intent.

When we talk about cyber threats, we need to conceptualize the relationship between the person and the software entity. Cyber threats are not just about malware, nor just about a human actor. You can't separate the two. It doesn't make sense to talk about human threats that don't manifest in cyberspace. And, malware that isn't backed by human intent would cease to be a threat. Our job is to explore this relationship between a human and his manifestation in cyberspace. The very word 'cyber threat' implies this relationship. It's direct and basic.

Active Reversing

I first presented active reversing in 2007 at the Blackhat show in Vegas. At that time, the concepts were mostly theory, backed by a few prototypes I demo'd during the talk. It's taken us three years but I am very proud to say we have stayed the course and delivered something that is a true game changer for reverse engineers. It works so well that it's cheating.

Active reversing is when you obtain program understanding via runtime instrumentation, data collection, and statistics. Active reversing puts the focus on volatile runtime behavior as opposed to static disassembly. This transforms the tradecraft of reversing in many ways. For one thing, it promotes reversing to a larger professional audience. People who are already comfortable reading packet sniffer logs can now reverse engineer software. The data collected from a point in memory resembles the kind of data collected by a packet sniffer, the only difference being the data structure of the packet is actually an internal structure within a software program. Observing just a string can reveal what a function is responsible for.

Active reversing empowers you to reverse engineer by exercising a software program's capabilities and features. Consider that runtime code coverage reveals which functions are executing and when. This allows you to simply observe which functions execute in response to an action you have taken with the software. For example, if you want to find the password handling function, just review which functions executed after you typed in the login information. Filtering can be used to remove code that has executed more than once, or code that has already executed in response to another action. This so-called background noise may represent utility functions and general purpose packet handling. Once filtered, you are left with only the newly executed password handler functions. This approach can identify functions when searching for data might be difficult - for example if the data is numeric and not easily predicted. More than anything, this approach is fast. In just a few minutes you can have most of the major features of a program mapped to code.

HBGary just published a very technical whitepaper on software exploitation using REcon, with lots of screenshots and step-by-step how-to. Check it out: Software Exploitation with HBGary's REcon.

The Nature of Funded Threats

Most incidents of espionage are never reported - even though tens of investigations are currently underway as you read this. Without consolidated statistics it will be hard to really understand the damages caused, even direct measurable damages. Espionage is not new, and the primary threat to your intellectual property has always been your front door. The only thing new about e-espionage is the 'e'. A few years back I gave a keynote at DFRWS titled "Funded Threats", in preparation for which I had to dig up some statistics. Although this was back in 2007, I would posit that not much has changed between then and now. I found some FBI statistics earlier that year stating that industrial espionage and IP theft was costing US companies more than $100 Billion USD per year. That was only a national figure. Measured worldwide the damages would probably be in the trillions. Those include indirect costs. Also of interest is that over 70% of a corporations intellectual property is stored online, digitally. It's easy to visualize the changing risk landscape - malware and information living on the same systems. ZDNet had just reported that in 2006, 80% of all malware samples were being missed by the top three AV vendors. Meanwhile, HBGary had just completed an extensive DARPA study into the nature of current rootkit threats (back then everyone liked to use the term 'rootkit' to describe 'advanced' attack tools). The report went dark, but I can sum up some salient points: as early as 2006 there was a global "cyber-arms bazaar" where cyber weapons could be bought and sold. We had samples that would bypass everything (by everything, I mean all the commercial and freeware anti-malware solutions). Most of the participants in this growing black market were foreign to the US. The stuff was well tested and well written, backed by a real software development lifecycle. One of the most specific things we learned was that every major AV and IDS product was installed and tested against during the QA process. When we presented this material to the DoD we joked around saying "the bad guys malware is higher quality than the commercial stuff they are exploiting". It was also becoming clear that information monetization was occurring - mostly identity theft. However, certain activities such as source code theft established themselves well before the mob figured out how to bank online. If you rewind back five years or more, you are going to find lots of interesting indicators that foretell our current situation. The people who operate malware today are the same people who operated malware back then, it's a career choice.

Responder 2.0 Released!

We have been slaving away long hours since early December last year, but we made it. The 2.0 release of Responder is released. It's amazing. Hard to believe, but it's been two years since we announced Responder at the CEIC show in Vegas. After so many years of coding, I have come to understand there is a big difference between building commercial products, and just building hacker tools. There is so much work you wouldn't expect going in. I would like to publicaly thank my engineering team for all their hard work.

For fun, here are some sneak peeks of Responder 2.0

Above is the REcon recorded execution timeline from a particularly nasty APT malware that plagues the DoD by constantly hopping from NIPRNet to SIPRNet and vice versa, primarily through the dreaded USB stick.

The above diagram is a second malware. This malware puts the capital 'P' in APT. We were first introduced to this one back in 2005 during a DoD infection. The above sample we obtained just a few weeks ago, and clearly it's a variant of the original - that's almost 5 years and still going strong! The diagram shows the code and dataflow for a function that enumerates machine-specific data after an infection drops, this is then sent via the C&C channel when the malware reports in. This kind of behavior is very common in persistent malware, as the bad guys need to keep track of their machine-infection inventory. A little story goes like this: location 1) this queries the uptime of the machine.. 2) checks whether it's a laptop or desktop machine... 3) enumerates all the drives attached to the system, including USB and network... 4) gets the windows username and computername... 5) gets the CPU info... and finally, 6) the version and build number of windows. Quite thorough eh?

In a nutshell, Responder 2.0 cuts through APT like butter. More Info.

See You in Sac-Town at ISSA

I will be giving a talk on APT at the Sacramento Valley Chapter of the ISSA, Feb 19th. This should be fun, since Sac-Town is my home turf. When I first thought of moving to Sacramento, I had horrid fears, but these were unfounded. Sac is a really cool place in California. Not just because it's the Capital, but it has great and diverse ethnic restaurants, more trees than you can count, and EPIC FISHING within an hour drive in any direction. It's also a great place for a tech company right now. Way better than the 'valley. Housing is much more affordable, and there is a great hiring pool with HP, Intel, and two major universities in the area. There are numerous initiatives to support small business growth, including tax incentives to build out on either Mather or McClellan AFB (I drive past Mather on the way into HBGary in the morning). Sac also has hacker history too, callbacks to the old days. HBGary already reached out to the locals last year when Penny outfitted the Sac PD with free copies of Responder Field Edition. In the local area we have given memory forensics training at International High Technology Crime Investigation Association (HTCIA) and Internet Crimes Against Children Task Force (ICAC). There is a lot going on around here and I glad to be a part of it.

Is the term ‘malware’ eclipsed by ‘APT’?

I am wondering why we need to change the term 'malware' to 'APT'. APT stands for 'Advanced Persistent Threat' and the term was cooked up by the Department of Defense to describe malware that worked, as opposed to malware that got caught. The term APT has been in the press since the very public Google hacking incident. I like the term APT, but I still wonder why we need it. To its credit, APT is a great term because it accurately describes the problem. On the other hand, it also confuses people. It makes APT sound different and new, when in fact there nothing is new.

Many of us have been analyzing malware for years and functionally the malware today is just the same as it was three years ago, but back then we didn't call it APT. In general, this got me wondering why people change the terms used to describe something (for anything in life)? For this malware and APT thing the best I could come up with is that many existing security companies adopted the term 'malware' many years ago, and thus their product offerings become tightly associated with the word 'malware'. Over the last few years or so, as people within the DoD and elsewhere realized that existing security investment wasn't adequate to protect their network, instead of calling the investment a failure for missing the malware, they instead decided to call the missed malware by a new name. Think: "This isn't malware; it's something new, so of course the millions we've spent doesn't address it!". A stretch?

The fact is this - malware has always had the ability to be updated in the field, it has always been able to be remote controlled, and it has always had the ability to spawn a remote shell to a live attacker. And, it has always had the ability to scan the file-system for files like source-code and CAD drawings, and it has always had the ability to exfiltrate those files. At all times and without exception, these malware programs have been operated by real and persistent humans at the other end. The malware doesn't operate itself, it's not an automaton. For the last 365 days, I just called that malware.

Post Execution - A New Paradigm for Debugging Malware

With the upcoming 2.0 Release of Responder, REcon plays a much more integrated role in the analysis of malware. The report automatically details all the important runtime behavior from a malware sample, including network activity, file activity, registry activity, and suspicious runtime behavior such as process and DLL injection activity. All activity is logged down to the individual disassembled instructions behind the behavior, nothing is omitted. Code coverage is illustrated in the disassembly view and data samples are shown at every location. This is like having a post-execution debugger, with registers, stack, and sampled data for every time that location was visited.

Post-execution debugging is a paradigm shift from traditional interactive live debugging. Traditional debugging is cumbersome and requires micromanagement to collect data. The traditional debugging environment is designed for CONTROL of the execution, as opposed to OBSERVATION ONLY. Typically, the malware analyst does not need to control the execution of a binary at this level, and instead only needs observe the behavior. HBGary's REcon approach to debugging is far superior because the analyst can see and query so much more relevant data at one time without having to get into the bits and bytes of single-stepping instructions and using breakpoints. It's like having a breakpoint on every basic block 100% of the time, without having to micromanage breakpoints.

HBGary and Palantir

I am very excited about our new partnership with Palantir. The Palantir link analysis capability is outstanding. Our team here in Sacramento is currently processing over 1.5 gigs of malware drops per day for Digital DNA, the new link analysis capabilities will allow us to move from the malware developer to the actual individuals who are operating the malware. We are already able to use forensic toolmarks to identify the individual malware developers, but this next step of analysis is to actually track those that have purchased or funded the development of the malware weaponry, and subsequently operate the malware in live operations. While tracking the developers themselves is easy due to the nearly impossible-to-avoid toolmarking caused by code and compiler, tracking the operators is much more difficult. But, HBGary is going to make a dent in this problem. We are approaching malware the same way an intelligence agency would.

Using Handle Tables in Physical Memory

One of the challenges we face at HBGary while developing Responder is the sheer volume of information available in physical memory. We have to reverse engineer a large volume of underlying data structures within the Windows operating system, not just for one operating system but for every single version of Windows (that includes service packs). In other words, a lot of time spent in windbg. One of the more interesting information sources available from the kernel is the handle table. Handle tables are extremely powerful and allow you to determine, for example, what files and registry keys are open by a given application.

Consider the way malware drops and installs itself. Many times, malware will inject itself into other processes as part of its "installation and deployment". At HBGary, we divide all the various behaviors of malware into categories we call "malware development factors". One of those malware development factors is called "installation and deployment" and it's all the stuff that malware does to survive reboot. In our Digital DNA(tm) we have well over 400 rules to detect just this kind of behavior. For example, malware may install itself as an extension to Internet Explorer and sniff online banking credentials. There are many ways to detect the presence of these capabilities. File handles can be used to detect when malware has been injected into secondary processes.

Malware injected into Internet Explorer opens a suspicious log file. This data is obtained from the kernel handle table.

Another anomaly that you can look for is a strange file path or executable name. Simply compare all the paths that are available in all the modules. In Responder, you can double click the drivers folder,m or the 'All Modules' folder, to view all drivers or modules along with their paths, respectively. Most modules are in consistent locations. When you examine all the module paths together in a single view, the anomalies will stand out. You don't really notice this until you see it in reference to all the other paths on the system. Seeing everything at once helps you detect an outlier quite quickly. This is just one of the types of things you can do when you have all the information at your fingertips.

Executables that are located in a suspicious temp directory. Also, one of executables has a non-standard file extension (.tmp).

A very suspicious kernel-mode device driver. This driver has no path, just an odd name with no file extension. This is a rootkit that was later identified as a variant of the "Black Energy" rootkit family. Notice that Digital DNA has automatically identified this driver as highly suspicious.

Executables that are running in a process that don't have a corresponding module name or path. These are very suspicious and we determined these were injected DLL's that were unlinked from the module list. Notice that Digital DNA has automatically identified these as highly suspicious.

One thing I really like about Responder is that you can sort the information in columns alphabetically and bring the outliers right to the top. Another thing that I like about Responder is that you can also write plug-ins that extend the functionality of the user interface at any point. For example, I could write a regular expression that would search all the file handles for certain patterns and that could include executables used in multi-stage installation, such as CMD.EXE or RUNDLL32.EXE, or files that appear to be in suspicious paths.

Regex r = new Regex(".*\\\\temp\\\\.*\\.exe$", RegexOptions.IgnoreCase);
Match m = r.Match(stringData.Name);
if (true == m.Success)
{
         // add an item to the report, associated with the DLL
 IReportObject wo = theDLLPackage.CreateReportObject(
             "suspicious exe in temp path",
             "This path looks suspicious, examine further");
} 

Responder scripts are written in C#.

I could even automatically add those to my report. The upcoming 2.0 release of Responder has an interactive report where you basically just drag and drop any specific data item to your report and drop it where you want the report item to appear. The dropped data actually just appears right there in the report at that location, including a description of what it is. Of course, this is editable by you, and you can expand upon it, but it makes it very easy for you to assemble a collection of those things that you find important in the memory snapshot. In summary, the report is a way to export and print the data I care about. I guess it also gives me a way to come back and reference those report items later (if you find yourself re-exploring old memory images). This is a short summary of some things you can do using the handles and the paths that are available to you in a physical memory snapshot.