Puffer Machines, El Al, and Defense in Depth

Airline security is a great case study in large systems security, and specifically the challenges of defense in depth implementation. While the U.S. will rapidly get back to business-as-usual, forgetting the near disaster of a Christmas day airline bombing, other countries have the threat of attack put in their face daily. This isn't something a person wishes the U.S. to experience, but you cannot ignore the ongoing annealing effect this has on the security posture of these foreign communities. It translates into large expenditures of money being applied to security because the threat is real. Contrast the TSA here in the States with El Al in Israel. El Al catches terrorists using multiple layers of security. The first defense is a knowledge of the world they live, the status of their enemies and who is likely to target them at any given time. This means putting intelligence to work and informing different organizations so they can work together. The second layer is a skilled person interrogating passengers. El Al realizes it is not a right to fly, it is a privilege. The screener focuses on the person. El Al respects the power of human threat detection by incorporating the interview into the screening process. The traveler may be asked to produce receipts for the places he reports to have stayed. If the person makes the screener nervous, that person gets set aside for more in depth screening. Simple. This increases the time it takes to check in, but this process has been proven effective. Humans are the best threat detectors in our known universe (seconded by our trusted animal companions). But, here in the States, we are so afraid of being accused of racial profiling and discrimination that TSA is forced to ignore human solutions, and instead relies on mechanical procedures and a compulsive focus on carry-on luggage. The next layer of security is technology based - if a traveler fails to pass the human screener, he or she may be asked to submit to a full body scan, a technology that raises hackles here in the States because of 'privacy' - never mind that it will actually detect plastic explosives taped to the body. The extra cost of sending would-be passengers through a puffer machine is easily shouldered by El Al, because they know it works at detecting explosives. The defense-in-depth goes even further: El Al has special reinforcements in the aircraft fuselage to protect the weakest point against an explosive blast. This is the kind of security that would make me feel safe to fly. All of this is expensive, time consuming and necessary, because it saves lives. The cost of security can't always be measured by money. Some things are more valuable like reputation, goodwill, and peace of mind. The U.S. should take some lessons from this, and start spending smart money on a few key defense-in-depth strategies that work, not only in our airline screening process but in our networks and infrastructure as well. Risk Intelligence is a lot cheaper to implement than we think if we consider the consequences.

Not Kind, Not Gentle. The turn of the decade in security.

The decade in review: The most painful thing we learned is that computer security hasn’t worked. We are, at this very moment, MORE insecure than we were in the year 2000. Billions of dollars were wasted on security technology that isn't working. In the last ten years, true cybercrime was born. Maybe we were just naïve about the coming storm. At the turn of the century, it was hard to get past the romantic idea of a university student hacker who prowled systems harmlessly for fun. Blocking ports and preventing network based buffer overflow attacks seemed so important. None of this technology prevented true criminals from pulling off the biggest heist in computer history – the massive theft of identity and subsequent banking fraud of the last few years. The traditional hacker is dead. Hackers are now called terrorists. The Russian mafia pays developers six figure salaries to write rootkits and malware. Independent researchers can and will sell a reliable working exploit of Internet Explorer for more than $50,000 USD. It began to hurt so bad that even Microsoft had to jump on the secure coding bandwagon, declaring a massive effort to make their code more secure. But this isn’t working either. You see, we are adopting technology at a rate far faster than we can secure it. By the time we have secured something, the landscape has changed and the attackers have moved on. In fact, that is why desktop exploitation has become the dominant attack vector. Over the last few years, malicious documents and media, especially “rich content” that contains embedded logic, parse-able metacode or script, and other logical constructs that can be malformed, emerged as the dominant method of exploitation. The API’s, COM objects, and other hoo-hah piled sky high on your windows workstation is a garden of carnal delights to a skilled attacker. Exploits of this nature have been mostly delivered via Internet Explorer and email. In fact, Internet Explorer is quite possibly the largest software disaster ever. As a software program, it has probably caused over a hundred billion dollars in damages since its release. This isn't about blame - if IE wasn't there, someone else's browser would have been the target. The browser is the portal into the Enterprise, so it's going to be where the bad guys focus. Finally, even before all this was going on, every nation state on the planet was standing in the shadows scared out of their britches. Smart people in high (low?) places could see the writing on the wall. It is TRULY AMAZING that a terrorist hasn’t hacked into the SCADA systems of a municipal power utility, started a cascade failure, and shut down half a state in the dead of winter. It’s because of this that I think [most of] those so-called terrorists aren’t very bright. As we close out the first decade, we must realize we have just entered one of the biggest arms races in the history of warfare. In fact, one can easily say that true cyber warfare was birthed in the last ten years.

So, now my predictions for the next ten years: Very early in the next decade, online identity theft and banking fraud will replace drug trafficking as the dominant criminal problem worldwide. Cyber cartels will make more money annually than drug cartels. Exploitation will continue to be focused on content-based delivery – that is, malicious documents & media. This will be coupled with a massive growth in online social networking. Trust, as a human concept, will be exploited as a means to spread malware throughout social networks via your online digital identity. Again, we will adopt new technology at a rate faster than we can secure it. The largest domain of attack will be software running on cellular phones. The phone will truly evolve into a network terminal – a slightly thicker thin client, loaded with more software in the palm of your hand than you could cram into a Windows 95 box in the year 2000. Yep, you guessed it, another garden of carnal delights – these new platforms will arrive unsecured – the development tools to make software will be insecure, and the people writing the code aren’t going to give a bug’s butt about secure coding practices. So, cyber crime is going to get a lot worse. Meanwhile, we are going to see at least one major SCADA based terrorist attack. We may have no idea that a terrorist did it, because the authorities will never admit it if they can plausibly lie, but it will happen. In fact, it may have already happened. Security spending will shift as well. Starting now, and reaching a heyday in about 6 years, security spending will shift towards host based security solutions. First the government, and then commercial enterprises, will realize that netflows and gateway solutions are not going to stop malware – it’s just too hard to predict what software will do without actually running it. And, online social relationships will be an extension of our professional identity - in other words, when an employee sits down at his workstation, his entire social network sits down with him. Network based security cannot hope to analyze complex documents and media, much less who to trust and when. Because everything will be hosted online, blocking content will effectively break the Internet, and looking inside the content will never happen at the network gateway (don’t invest in companies that think they can solve that problem). Concepts like malware-tolerance will become a hard reality, people will realize you can't keep the bad guys out. While the majority of online crime will continue to be in banking fraud, we are going to see industrial espionage and state-sponsored attacks in the press more than once. And, while banking fraud hurts the individual, the scope and damage of espionage is far far greater. Whether its classified state secrets or the recipe for Coke makes no difference, when the criminals out there figure out the value of information, they WILL steal it. The next ten years are not going to be kind or gentle to the security space. The hardest hit are going to be the biggest in the space – AV vendors are going to take the hardest fall. Their signature based solutions don’t work today, but not everyone knows that yet. But over time, that truth will seep farther into the IT space. So, perhaps my biggest prediction is this – AV will lose their place as the #1 security expenditure in the Enterprise. I’m not sure what will replace it exactly, but I do know that people are going to stop throwing good money after bad.

Blackhat Training is almost here!

I am gearing up for the Blackhat Training session on Monday-Tuesday of next week. We have made room for 30 students. We spent almost four weeks working on materials, remastering the demo and recap videos, and collecting malware samples that illustrated each of the subjects we are presenting. The task was alot harder than I originally expected, especially the collection of malware. I discovered a great trick using our feed processor, which is the clever use of search terms against strings to locate malware that were using specific techniques, keylogging methods, hooking styles, even specific languages. We have a solid methodology we teach behind our Responder product, so I had to find malware that illustrated specific concepts, as opposed to tailoring a training around whatever malware happened to be available. I will try to keep the training as high level as I can, and stay out of disassembly code as much as possible, but as expected there are some key reversing skills that can never be avoided, such as the reconstruction of parameters passed to a call. But, as for arithmetic and hard logic reconstruction, the only exercise where we get into that level of detail will be the one on crypto and stego. We have one coding exercise using the new built-in scripting interface, so thats a short bit of hardcore fun as well. But most of the material is about getting reverse engineering done rapidly, getting what you need, and not bogging down - which is the name of the game.

Reverse engineering process-injecting malware

I posted a video demonstrating some RE work with Responder:

Process Injecting Malware

The RE process starts by searching a livebin's symbols for "remote". A livebin is the in-memory version of an EXE as extracted by Responder. It’s not an executable format, but instead represents the exact layout of the PE formatted file once it loads into virtual memory. Section information does not need to be interpreted to remap the binary in this case, as the OS loader has already done that, including remapping and any other modifications that are made to the layout of code and data at runtime. Many malware programs will be packed on disk, but the livebin will contain large unpacked sections that can be analyzed without the RE having to know anything about the packing methods used, as in effect they are already unpacked for you.

The symbol "CreateRemoteThread" is of interest. For process injection malware you will find this API call almost 100% of the time. There are a few other API calls that are used in conjunction. We drag the symbol to the canvas and examine the region around it. Specifically, we see WriteProcessMemory and VirtualAllocEx - this is a dead giveaway that process injection is in use. Usually a malware will inject a thread that points to the function "LoadLibrary" with the first argument being a path to a DLL that was decompressed to disk - typically in a temporary directory. This is part of the malware's installation system.

In the example, we find that a PID is used to locate a process to inject into. We follow this PID and find the argument is used with LEA. In this case, the LEA is like using the address-of operator in 'c' code.

Imagine the following code:
void *myFunctionPointer;
some_function_call( &myFunctionPointer );


The "some_function_call" is getting a reference to myFunctionPointer, and this means it has the ability to initialize or assign a value into myFunctionPointer. The LEA instruction you see in the video is the assembly version of this same operation. We see this, and follow the function to find a loop where ToolHelpSnapshot32 is used. The toolhelp API set is another very suspicious behavior - if you see this in a potential malware you are very likely dealing with something that enumerates other processes on the system. This is usually a step prior to injection (or an attempt to find a virus scanner or firewall exe and kill it).

There is a string comparison in the process hunting loop - so the malware author is attempting to find a process by name. We follow the arguments back up and see that it's searching for "explorer.exe". The steps shown in the video require moderate-level RE skills, but are not daunting. With a little practice you can follow arguments in and out of function calls without losing your place. The trick is simply to remember that arguments are usually a positive base off of EBP, and local variables are a negative offset. "Parameters are Positive" - use that rule to remember.

The is one exception that is likely to drive you crazy - malware written in Delphi (and there is ALOT of that) usually passes parameters in registers. This can be harder to follow, but again if you label the arguments going into the function you can see these labels at the function boundaries so you don't lose your place.

There are no isolated networks anymore

Highly specialized networks, such as those that control power grids, or esoteric equipment, such as MRI scanners, are not typically considered at risk from Internet attacks. Yet, the recent conficker worm was able to infect these things. It is important to understand that just because hardware seems specialized and distant, it can still be connected to a TCP/IP network. Even if the equipment doesn't offer a convenient web-addressable interface to hack, it can still have a protocol and perform I/O.

Almost all modern but specialized equipment has embedded TCP/IP capabilities and the associated ethernet jack. Web and TCP/IP based technology is a good choice for machine interfacing and configuration. Browsers eliminate the need for specialized client software. Non-specialized programmers can write code that works with a HTTP or HTTPS interface to provide remote configuration capability - this equals lower software development costs.

Specialized equipment often contains a remote data terminal (RDT) which is like an embedded board that contains a mini-OS, likely based on a linux variant or even something like VXWorks. Newly emerging technology, like System on a Chip (SoC) is both inexpensive, and easy to interface to. Even when an RDT type function is not available, these devices may stream large volumes of data outbound over TCP/IP, with the port intended to be used in a specialized LAN configuration for image capturing or other functions (think medical equipment like MRI scanners or X-Ray machines that are interfacing to the PACS network).

The overall point is that these machines are connected to a network that talks TCP/IP. And, following the very nature of TCP/IP, it's easy to make connections that are unintended. So, even though the MRI scanner is not supposed to be connected to the Internet, the imaging workstation will need to talk with the database in Radiology which is then connected to the Hospital Information System (HIS), which is connected to the Internet. You now have an MRI scanner that is attached to machines that can browse the Internet. This is how Conficker got into Heart Monitors running an old unpatched Win2K systems.

Even old equipment falls prey to these unintended exploit paths. Especially for older SCADA equipment, there are tons of devices that will interface good old serial ports to ethernet and TCP/IP pathways. To lower costs, SCADA networks have been refitted with remote access that is routable over ethernet and TCP/IP. The protocols are old and weird, but anyone who does their research can attack them. Even when not directly connected to the Internet (and yes, sometimes they are), devices like power relays are just a few hops away from the Internet-facing gateway. These devices really do control power for small northeastern towns in the dead of winter.

A large amount of the risk here is simply that specialized networks are connected to the Internet via unintended means. These unintended connections between the so-called “protected” networks, and the totally unpatched open equipment is something like a void. It’s not well audited. In some cases, the IT staff may even be discouraged from auditing. In one factory a few years back, the IT staff were forbidden from even running port scans to inventory the network. Apparently doing so once crashed a SCADA controlled machine on the factory floor, so management had forbidden the practice hence. To make things worse, it's incredibly easy to bridge networks without thinking about the security implications. An end user can co-fuse two networks just by plugging in a cable incorrectly. A network admin may not have an extra switch so they use the existing one out of convenience. There are countless scenarios where it's easier to think of specialized systems as non-internet devices, thus not a problem for security.

When dealing with network security, you should always think of every networked device as containing an operating system. It would not harm your security to even think of them as embedded windows operating systems that are vulnerable to conficker worms. You should never think of them as non-internet devices.

Ongoing SCADA Attacks and Network Probes

Consistent and ongoing recon-probes continue to be launched into the US Infrastructure, including government and municipal systems. Boldly stated, all large Enterprises (government and corporate alike) are compromised by some form of malware that is CURRENTLY under C&C from a remote attacker. Malware infections are the tip of the spear - at the other end of an active malware C&C network is a human being or organization with intent and funding.

Recon-probes are malware implants that only scan ports and inventory the resources in the network, then phone home with the data. In many cases, probes are not targeted (not aimed only at your network, but rather like a shotgun approach) - there is an ongoing effort to simply map everything that sits behind the public gateways. In particular, probes have been launched into the US Infrastructure SCADA networks - think power grids and water plants.

Probes will be less complex than a full-blown botnet agent. One component to be on the lookout for is a TCP/IP-only capability - not something that injects into IE or Explorer, but rather a cleaner implant with a port scanning and sniffing capability. These probes will have a C&C backchannel of some kind, but are likely to store their information on disk for a while, as they don't phone home very often. These are forward probes designed to map your network. They may even query the hard-drive serial number via IOCTL's to the NTFS driver, this is for node identification decoupled from the IP address of the host. There will also be a query to see if the box has multiple interfaces.

If you find a probe operation, immediately assume that secondary attack tools have been brought into the network, perhaps in select subnets or on critical gateway machines. Be especially attentive to any sniffing capability on a collision domain near a gateway, or even on the gateway itself. In some cases, secondary capabilities have been dropped that have the ability to shutdown and destroy the computer. If you have captured a probe, immediately check all embedded registry keys and file paths for potential storage locations for secondary equipment.

Rich and Greg in Va. – Ghillie Suits, AR-15’s, Russian Ammunition and Chinese Malware

The morning was spent discussing how lame Conficker.C turned out to be and how it was most likely just barrage jam… meaning a smoke screen diversion to throw off the scent for the “real” slow and low pdf attacks that were slipping into financial institutions in droves. Then on Friday morning HBGary was lucky enough to receive a nice excel spear phishing attack. Unlike most companies we love this stuff. This gives us something to do over the weekend. Greg and I also discussed our new global services offerings which will soon appear on our web site.

After breakfast Greg and I went to an undisclosed location in the Northern Virginia area, got suited up with Ghillie-Suits, AR-15’s, and a 1000 rounds of Wolf Performance Ammunition from Russia. Our mission was to get from point A to point B without getting caught by numerous “individuals”. If we weren’t caught and we made it to point B, we were then to shoot the 500 rounds each at targets from 25 yards up to 100 yards. We had up to 4 hours to cover the terrain, get to Point B, shoot our rounds and get back to point A. without getting caught; again the main point was not getting caught. It was a great! We covered a crazy amount of territory in a short period of time climbing through all kinds of thick brush and most of it was straight up hill to reach point B. At one point we we’re within 50 yards of some of the “individuals” but remained completely still and since we were in our Ghillie-Suits we remained completely still and remained undetected, just like a good rootkit. ;) We ultimately made it to Point B., where we celebrated by drinking some water and dropping our packs to load our rifles. We target practice with Russian ammunition because it’s cheap, pretty reliable and readily available.

As the sun was setting, we had already infected a VM with one of the recent boobytrapped PDF documents. Using a snapshot and Flypaper, we extracted several binaries with Responder and discovered a running botnet out of Russia. The PDF document immediately grabs a malware loader executable from a hacked chinese website, including a flash module. Once the loader executes, the main loader contacts a bot controller located in the ukraine, and the subsequent payload that is downloaded loads a kernel mode rootkit and a usermode module that communicates with a single drop point - a single commercial hacked website to store a drop point, and from this scripted location, data being emailed to a completely different and single hacked email account. The bot control software is something called "JRoger BManager v1.5" and in this case, was operated from a Russian language asset. We made heavy use of NetWitness Informer to capture C&C traffic and compressed downloads of infection modules. We are now tracking this threat to learn more.

Here are some pics:

The Bot Controller

Responder graph of the usermode portion of the malware

NetWitness really boils off the fat. You can slice and dice the data from a packet capture in so many ways. Here are shots:

Overall a good day.

The Sky is Falling, When it Rains

We have come to distrust any doomsaying in the security industry. We can't identify an authoritative and impartial entity that can stand back and really make an assessment of risk. Claims about the cyber threat level resemble the Orange Threat Level at the airport - a distant flag of color, washed out behind the gate call and the long line at Starbucks. To an outsider, the latest threat reports published by security companies seem to be coat tailing on Conficker - a recycling furnace of self-fulfilling prophecy, the press thermometer following along, ticking up to the final doomsday hour when conficker went... fizzle pop. Conficker a bust. Move on, this is not the threat you're looking for.

From y2k to Al Qaeda threats on the Capital, the lack of materialization can lead us beyond healthy skepticism to a place where we conceptually disenfranchise threat intelligence as a whole. This is where we have to be careful and step softly in those dark woods beyond the campfire. Just because conficker didn't blow up the Internet does not mean it couldn't. If anything, conficker brought a lot of press attention to the problem of malware, and that is a Good Thing. When tens of millions of computers remained infected with a variant of conficker on April 1st and still today, we all need to understand that someone somewhere could have lit the flash powder. Conficker is old news. New variants of malware are released daily. In one discussion I heard upwards of fifty thousand new variants per 24 hour period (think autopacking on deployment). If conficker is truly controlled by the Russian Mafia, then blowing up the Internet serves no purpose for the their bottom line. Silent ongoing presence is what steals intellectual property and banking credentials; not DDOS, not software vulnerabilities that amount to sexed up access violations. Real attacks are about reliable access to money and information. The security industry can sometimes get caught up in stuff that really doesn't matter that much, while ignoring the threat that is right there, in front of your face, in your computer right now.

Malware commonly hunts down and kills anti-virus programs

Much of the malware we are processing has the ability to locate and kill anti-virus programs and desktop firewalls. The following malware example illustrates the behavior clearly. There are long strands of code that query through a list of known security software process-names and subsequently sabotage them.

Click for larger image

The means by which the malware detects the security software is by process name. There are long lists of process names that appear in sequence, these nodes are shown on the graphic as label 'A'. Almost all variants of this behavior are similar in structure, even though they are employed across many different and unrelated malware strains.

update:
I took the time to zoom in on one single operation (marked as B. in the following image) from the strand of control flow shown above. The strand shown above contains hundreds of these.



And, here is the disassembly for one operation:

10001A98 BB 5C 65 00 10 mov ebx,0x1000655C // webtrap.exe
10001A9D 53 push ebx
10001A9E E8 C3 25 00 00 call 0x10004066▼ // __imp_MSVCRT.dll!strlen
10001AA0 ASCII: %
10001AA0 : 25 00 %.
10001AA0 : 25 00 00 %..
10001AA3 loc_10001AA3:
10001AA3 59 pop ecx
10001AA4 50 push eax
10001AA5 53 push ebx
10001AA6 8D 4D F0 lea ecx,[ebp-0x10]
//__imp_MSVCP60.dll!?assign@?$basic_string@DU?$char_traits@D@std@@V?$allocator@D@2@@std@@QAEAAV12
10001AA9 FF 15 8C 50 00 10 call dword ptr [0x1000508C]
10001AAF loc_10001AAF:
10001AAF 8D 45 F0 lea eax,[ebp-0x10]
10001AB2 8D 4D DC lea ecx,[ebp-0x24]
10001AB5 50 push eax
10001AB6 FF 75 E4 push dword ptr [ebp-0x1C]
10001AB9 E8 BB 22 00 00 call 0x10003D79▼ // sub_10003D79
10001ABE loc_10001ABE:
10001ABE 57 push edi
10001ABF 8D 4D F0 lea ecx,[ebp-0x10]
// __imp_MSVCP60.dll!?_Tidy@?$basic_string@DU?$char_traits@D@std@@V?$allocator@D@2@@std@@AAEX_N@Z
10001AC2 FF 15 90 50 00 10 call dword ptr [0x10005090]
10001AC8 loc_10001AC8:
10001AC8 8A 45 0B mov al,byte ptr [ebp+0xB]
10001ACB 56 push esi
10001ACC 8D 4D F0 lea ecx,[ebp-0x10]
10001ACF 88 45 F0 mov byte ptr [ebp-0x10],al
// __imp_MSVCP60.dll!?_Tidy@?$basic_string@DU?$char_traits@D@std@@V?$allocator@D@2@@std@@AAEX_N@Z
10001AD2 FF 15 90 50 00 10 call dword ptr [0x10005090]

What is interesting is that the list of malware to kill is commonly cut and paste around, but the methods used to parse it can vary based on which malware author is developing it. In this instance, the malware author uses the standard template library. This is a good fingerprint that can be used to find other variants of this particular author's work. Data like this can be used as an attribution factor.

Nabbing Conficker with Digital DNA

What follows is a step by step analysis of Conficker using HBGary Responder. The conficker worm represents a significant and current threat. The following analysis was performed against a sample at HBGary's lab. The first step was to obtain a dropper for one of the conficker variants and subsequently infect a 'sacrificial lamb' machine. The sacrifice machines have no network card and the USB ports are blocked as a precaution. There is a secure one-way method to take a physical memory snapshot and pull it from the machine. We resort to such measures when the malware won't execute in a VM. Once the infection was deployed and the snapshot obtained, we simply import the memory snapshot into Responder. The Digital DNA (available in Pro edition and also for the Enterprise via the McAfee ePO integration) detects and weights digital objects based upon a numerical string that is generated for every identified object. In this case, the Digital DNA calculated for one of the VAD tree memory ranges indicates high suspicion.

Click for a larger image

This memory range is subsequently extracted and disassembled / decompiled. Code, data, symbols, and strings are all recovered from the dynamic snapshot. This is an interesting fusion between static and dynamic analysis, given that its a snapshot in time of an actual running instance of the worm. Buffers contain fixed up address data, decoded data, full call stacks, etc. Many arguments can be reconstructed that would not be available in a traditional static file-based analysis. By using memory, for example, we didn't even have to worry about the packer. In this case, the packer has already ran and the malware is sitting in memory fully unpacked. We start our analysis by dragging all the strings to the graphing canvas and sorting them into colored layers.

Click for a larger image

Further analysis is largely graph-driven. Each layer represents a different property or 'factor' of the malware. For example, all of the installation capabilities are put on their own set of layers, while the communications are isolated to a different layer. The sorting to layers takes about 10-15 minutes by hand.

Click for a larger image

Once sorted, I pick off an individual layer - in this case the 'installation and deployment' layer. I toggle off the visibility of all the other layers and just focus on this single layer.

Click for a larger image

I pick through the layer sorting each small island of nodes into a more refined set of layers - moving some to a layer regarding the DLL injection capability, another describing the patch conficker makes to the tcpip.sys driver, etc. This process continues for about an hour as I drill down on connect many nodes on the graph. As I go, I bring up the code view and label functions with bookmarks that will end up in my report. The bookmarking lets me make annotations to the disassembly and the graph that are preserved for reporting purposes.

Click for a larger image

Whenever I dive into a function, I use the built-in dataflow analysis and graph-based decompilation. You can see in the screenshot how all the graph nodes are annotated with the logical conditions required to follow the branch. For example, in the screenshot a loop is detected and the loop control conditions are shown. This is a low level feature.

Click for a larger image

Finally, after about an hour or so, I have built graphs into over a dozen layers describing portions of the conficker worm's code and capability set.

Click for a larger image

When I am finished I auto-generate a report in word format, clean up the edges a bit, and print it to a PDF file. The following link shows a partial report detailing some select areas of the conficker worm.

Responder Report for Conficker (PDF)

Responder is better than IDA Pro for analyzing malware.

Greg's Note: This blog post was made by Martin Pillion, the most senior reverse engineer at HBGary and one of the most skilled RE's I've met. I have crossposted it to Fast Horizon so it would be picked up on the RSS feeds. Martin's assessment of IDA vs Responder is timely and highlights the amount of experience required to look at assembly vs. graphs. Features like proximity browsing lower the bar significantly so that more practioners can help solve hard RE problems.

Responder is better than IDA Pro for analyzing malware. I do not make this statement lightly. I have been reverse engineering code (on and off) since the late 1980s, I am a long time user of IDA Pro (since ~2000), and I have written a fair number of IDA Pro scripts. IDA Pro has been the definitive disassembly tool for nearly 10 years.

Let me also point out that I am an HBGary employee and I certainly have a bias toward the Responder product. I have helped design, architect, and develop the Responder product for many years. During that same time, I often worked with the HBGary Services division to assist customers with reverse engineering malware. I used Responder as much as possible, but often found myself loading up IDA Pro and running both products at the same time. However, in the past several months, Responder has improved to the point that I no longer use IDA Pro at all.

Responder utilizes a different approach to reverse engineering than IDA Pro. Where IDA Pro relies on inspecting large amounts of assembly, Responder relies on a more visual, graph based approach.

Responder displays information visually and is navigated based on relationships. Responder has a feature called 'Proximity Browsing' that allows you to expand a graph based on the cross references to or from the currently selected node. This makes it easy to quickly locate related code and visually examine those relationships. IDA has a popup dialog to list xrefs, but you must click each xref individually to examine it (time consuming).

Responder uses the color of xref lines to indicate what kind of xref it is. Grey for data xrefs, black for block xrefs, and red for call xrefs. Node shapes can indicate function starts, ends, regular blocks, data, comments, or bookmarks. Node color is used to organize graphs and is determined by layer. Layers are similar to the layer concept in Adobe Photoshop.

IDA Pro relies on WinGraph32 to perform graphing and it is clearly a secondary feature in the tool. There is support for a limited set of graph based analysis features, mainly built around flow chart and call flow graphing. The UI also leaves a lot to be desired... it is not possible to modify an existing graph, instead you must go back the text based UI and regenerate an entirely new graph.

Responder lets you "Collapse" a graph node. A Collapsed node is an entire function, instead of a single block of disassembly. You can Proximity Browse from a collapsed node and you will expand only call xrefs, with additional nodes also being collapsed functions. Using this feature you can quickly identify the purpose of main functions and label them appropriately.

Responder can also perform "Data Flow" tracing. Data Flow tracing allows Responder to follow the movement of data through a function, even if it is moved to a memory address (like the stack) and later moved into a register. This means that Responder can follow stack variables on functions with frame pointer omission.

Data Flow tracing is powerful and Responder utilizes it every time you rename an instruction operand. This means that your custom labels may show up later in the function and be used in a way that you did not realize. Data Flow tracing will track multiple levels of dereferences and indirections, memory addresses, registers, and even logical manipulations.

These are just a few of the features make Responder better than IDA Pro for malware analysis. I discuss others and also expand on the technologies behind each feature in future blogs. Ultimately, the primary work flow of a reverse engineering is one of organizing and understanding data. Responder enables me to do this with malware faster than I would be able to using IDA Pro.

- Martin

Server-class Analysis Now Possible with Responder

HBGary has been lifting some heavy iron, testing a variety of large memory configurations over the last few weeks. The latest version of HBGary Responder now sets the milestone: 64 gigabytes physical memory analysis - a sizeable snapshot indeed. This makes Responder a server-class product. This is an important step forward for HBGary, as the Digital DNA and malware analysis capabilities can now be applied against critical servers in the Enterprise. Large memory footprints can be found on server class machines running Windows Vista, 2003, and 2008. Ensuring servers remain free of rootkits and malware is crucial for regulatory compliance. A case in point, Visa recently announced that PCI compliance was being revoked for both RBS WorldPay and Heartland, due to malware intrusions and subsequent breach of security. Early detection of an intrusion can prevent data theft, as malware typically infects a system and remains there for quite some time. A recent data-breach study by Verizon (spanning over 4 years and 500 intrusions) reports that over 70% of victim companies had been compromised for over a year before the intrusion was detected. FISMA, PCI-DSS, and HIPPA all mandate various forms of intrusion detection to help limit the scope of damage caused by an intrusion. Sound defense in depth strategy advocates that Enterprises monitor server memory for zero-day malware and rootkits.

SMM Rootkit: Old, Obscure, and Unnecessary

Be mindful that you don't overreact to the 'new' SMM exploit (properly, reported by Loic Duflot, a very accomplished low level hardware researcher, at the recent cansecwest conference). The exploit itself is really a documented 'feature' of the Intel 5100 Memory Controller chipset, and has been a known issue with SMM for quite some time. See the 5100 data sheet:

In order to make cacheable SMM possible, the chipset must accept EWB’s and must absorb IWB data regardless of the condition of the SMMEM# pin. The Intel® 5100 MCH Chipset will not set the error bit EXSMRAMC.E_SMERR in this case. Because of this, care must be used when attempting to cache SMM space. The chipset/platform cannot protect against processors who attempt to illegally access SMM space that is modified in another processor’s cache. Any software that creates such a condition (for example, by corrupting the page table) will jeopardize the protective properties of SMM.

You might ask why it hasn't received more attention until now? Stated simply, such low level tactics are simply unnecessary for a real rootkit to be effective. Remember that you need to be in ring-0 (kernel) before you can even attempt installing into 'ring -1', and being at ring 0 is plenty of privilege for even the most stealthy of rootkits.

There are other reasons that an SMM rootkit is best left to the science fair: to make one that is effective across more than a select hardware platform, you would need to invest boat loads of development dollars in testing. At worst, someone might build an SMM rootkit that works on a well-known and distributed model of laptop and post that for publicity - but the real criminals don't build stuff like this, nor do they have to.

The majority of malware threats today are, in fact, usermode. The bad guys simply don't need to go any lower to get their work done. Remember, the lower you go, the less re-usable code you can leverage. That is, lower means no libraries, no API's. Lower means you write all the device, memory, and hardware logic yourself. It approaches the complexity of device driver development and operating system design. This all translates into expensive and non-ubiquitous. Malware avoids this development cost by simply installing itself like any other program, re-using the existing API's and libraries under windows that already provide network access, memory management, file access, and the like.

The modus operandi of real malware authors is: Write once, use many times. An SMM rootkit is a really neat science experiment and excites technical curiosity, but such an approach is not useful in practice. Let's stay focused on our Day Job, tackling real threats.

Digital DNA - Numerical Expressions to Describe Malware Behaviors

HBGary unveiled Digital DNA today at the Infosec Conference in Orlando. (I wasn't able to make it down to the show, although I had planned to be there. Last minute stuff and I had to jet back to the West Coast.) The engineering team has been working on Digital DNA for months. In a nutshell, we have automated the reverse engineering of loaded modules in the physical memory snapshot and generate Digital DNA (DDNA) based on the collected data (millions of data points). All of these data points are codified in way that allows them to be matched against rules. The Digital DNA system will "sequence" a software program or document and generate trait-codes based on the behaviors and schematic artifacts found in the software or document. Each trait has a complex rule (think regular expression with boolean logic) associated with it, and if the rule matches the trait is considered "expressed". Expressed traits are concatenated together to make a "sequence". We chose to do it this way because the final DDNA sequence looks and smells like a hash, even though it's not actually a hash at all. But, customers are used to managing hashes, thinking about hashes, and cut-n-pasting hashes - so a hash it would be.

Digital DNA is based on the reverse engineered behaviors, not the specific compilation or packer used with the malware. You can pack the same malware with three different packers and it will still produce the same Digital DNA. Two similar programs will produce similar DDNA. Here is an example of two versions of Rustock.B.



Interestingly, the technology can identify digital objects. Here is an example of tracking Intellectual Property with it.



Digital DNA is a Big Idea. For now, HBGary is going to focus it on detection of zero-day malware threats. We have over 2,000 traits in the DDNA genome currently, and will probably have many more soon. We sort all the traits into Factors, Groups, and Subgroups, defining a "genome" of behaviors that are common to malware. This part plays into a weighting system. I will blog more about this over the coming weeks - dinner is calling.

Your online payments are being sniffed; accept it, live with it

PCI compliance is clearly not enough to protect credit card numbers or account information. It’s about time everyone who uses an account for online payment simply accept the facts: your credit card numbers have been stolen. Check your statements monthly. Why? This isn’t about Heartland or the breach-of-the-week; this is about a constant effort well funded by a criminal underground. The primary tool in the cyber criminal hand, the malware program, keeps getting better. Malware authors are intelligent and focused developers who are well paid for their work. They have developed toolkits so they can generate new malware with little development overhead. They can generate new attack bits in a matter of hours that, to a virus scanner, may as well be a zero day – no signature means no detection, and no protection. Most of this malware decrypts live to memory and never touches the disk. The computing infrastructure is easy prey. It has never been secure, and won’t be secure anytime in the next ten years. Computer security is a constant effort that will never fully work. It’s partial risk reduction, not resolution. The billions of dollars spent since the turn of this century on IDS, firewalls, and virus scanning hasn’t made a more secure Internet. The growth of online technology has far outpaced our ability to secure it. Millions of credit card numbers are being stolen THIS MORNING. They were being stolen yesterday. They are going to continue to be stolen tomorrow.

Melissa Hathaway, on track to make a difference?

Unlike previous cybersecurity czars, Ms. Hathaway has experience. She understands how hard national security can be. Notably, Ms. Hathaway has been working on the Dark Side (think classified) of the government, which means she knows the reality of cyber threats - how effective cyber espionage really is, what is being stolen, and who is stealing it. It also means she knows the definition of a "Funded Threat." And, to combat these funded threats, she understands that it's not just defense, but also offense (think geolocation, trace back to the human, and the money). During his campaign, President Obama stated that he would take cyber attacks as seriously as nuclear or biological. A strong statement like this ultimately translates to budget.

Obama seems to want to dip his toe in the water first. Ms. Hathaway will not have the White House power position, at least not yet - there will be some bureaucracy between her and the president. We will have to see what happens in the next 60 days. But, bureaucracy will be one of Ms. Hathaway's greatest challenges. To her credit, she comes from the right community. She has the relationships in place that can help her succeed.

One of the things I like about Ms. Hathaway is her understanding that cooperation between agencies is required for success. The government is a big place, and the computer networks within it are like little fiefdoms. Coordination is difficult -- not because people lack the will to work together (although that adds difficulty), but because searching through ALL the information is required to find out what's important or critical. Most people want security to be someone else's problem. Those responsible for security want it to be easy. But that is core of the problem. Security is NOT easy. There is no shiny button.

Real security takes work. Ms. Hathaway supports building new technology to address new types of threats that go beyond what yesteryear had. We need to realize that people are out to get us, we are being attacked, and if smart people in the Enterprise say it's an "arms race" you better believe the government knows it is. She needs to be frank with everyone that there is no magic pill. She must require people to step up and do more and not rely on outdated security technology but to supplement with newer technologies.

The 60-day security review may bring back bad news - that things are terrible out there and the Nation's security is worse than it has ever been. We are in tough times, and some tough decisions will likely be made. Ms. Hathaway appears to have the big picture -- finally someone who might actually be able to change security for the better. Hopefully Obama will give her the authority to do so.