Its going to get a lot worse before it gets better

There is an interesting mix of problems going on right now that, when combined, create a sort of "cybercrime" perfect storm. Historically, there is an obvious correlation between economic downturns and the rise in crime. What makes the modern downturn interesting is the ease with which cybercrime can be perpetrated. First, there is the growing and fluid blackmarket for financial data. One doesn't have to browse far to find reports of a rise in phising, drive-by web infections, and advances in bot-net technology. Insiders with access to financial information will find easy money. Large financial institutions are already experiencing a rise of internal investigations. Layoffs in the high technology sector are closely related to intellectual property theft - employees are very likely to download intellectual property that may help them secure a new job - its a simple backup plan that is easy for the human mind to justify. This isn't even that high-tech - it's as simple as USB thumbdrive and an unprotected port. Internationally, high tech workers are losing their jobs, and programmers out of work are willing to take malware development jobs for low pay. IT professionals out of work in Eastern Europe and Asia are already getting roped into the identity theft blackmarket, using toolkits to develop and deploy phising attacks. The endpoint systems within enterprises are frail and easy to attack with malware, they are already infected to a large degree. The virus scanning technology that is the leaning post of enterprise security just doesn't work. The massive investment in security solutions over the last decade hasn't helped at all - enterprises are just as vulnerable and exploited today as they were in the late 90s. I think its an obvious conclusion to be drawn, the malware problem is going to experience a surge over the next 24 to 36 months. Investigators are just now starting to understand that there IS a problem, much less combat it.

64 Bit Analysis - the future is here

64 bit systems are more than 10 years old, but only until recently did they emerge in the market. In the past, 64 bit was found doing the “hard stuff” - complex fluid mechanics, computational modeling, and ultra large databases. Today they are mass market - it’s hard to find a 32 bit machine, and you pretty much have to custom order to get XP. More complex applications are going to demand upgrade - virtual environment collaboration for example - but even more important is just the fact we are dealing with large data - think > 4GB files. If you need an example, think of a DVD movie. Multimedia alone is driving upgrades in the consumer market.

64 bit architecture is just plain faster - it's not so much about the CPU speed - it's more about how much data can be moved. Data movement is more important, IMHO, than Data calculation for most users. With the possible exception of codec's and compression, people just don't USE their CPU's (Video cards with cooling manifolds, GPU's, and "Left 4 Dead" addicts aside :-).

This means “big” things for memory forensics – and also malware analysis in terms of the 64 bit operating system. While I think malware will continue to be coded in 32 bit (for maximum compatibility if nothing else) the stations they infect are migrating to 64 bit windows.

Simply stated, workstations are the point of penetration into the Enterprise. It’s the place we care most about analyzing. To understand malware from our (that is, HBGary’s) point of view, we need to understand the operating system – and by this I mean Windows Vista 64bit, Windows XP 64bit, Windows 2003 64bit, and Windows 2008 64bit. We follow the OS, we find the malware. Pretty much any pre-2008 rootkit technology just raises its hand “here I AM” in an offline memory analysis. From a rootkit developer’s point of view, things are happening live-action, hooking a function pointer makes sense because the HIDS is going to be totally subverted. Offline, everything changes. Said hook is a big red flag waving “over here – find me over here!” But, of course, this means being able to analyze the OS data structures in the first place. Hence, the importance of 64 bit.

HBGary released the 1.3 version of Responder a few days ago. This is the 64 bit platform upgrade. It includes analysis of 64 bit Windows platforms, and the FDPro dumping utility that can dump physical memory images from 64 bit systems (including those that require signed drivers). This was the longest development iteration for our team so far this year. The 64 bit upgrade was a lot harder and more work that I originally expected – there were upgrades and point-fixes in every part of the product from the GUI controls down to the memory-acquisition routines. Over 1,000 points of code had to be fixed just for 64 bit address and offset support. Yeah, big job. And the testing, wow… So here it is. I think HBGary might be the first to market with 64 bit support (that means analysis AND acquisition, and full shipping non-alpha). The new version of FDPro is pretty nice too, supporting compression and probing, speed upgrades, and nearly 100% reliable memory-page queries even for systems with more than 4GB of RAM.

I am glad this release went out the door and hopefully I won’t spend the rest of the year in a troll-cave. I’m going to be in New York City next week and also a few days in D.C. showing off the new bits. If you’re a current customer you should click the upgrade button in the about box :-) Contact sales@hbgary.com for an eval and the website is www.hbgary.com.

p.s. Thanks to everyone who uploaded me 64 bit memory dumps for QA – you’re the best!

Been working on 64-bit

I'm sorry it has been so long since I have posted. Stated bluntly, I've been scrambling with product releases. Although I hope to rise into a management position deserving of my title at HBGary, -- I am in fact a developer. I code every day, new feature or bug fixes. We made the decision a while back to adopt agile development - a buzz word I know, but one actual hard side effect is the short development iteration. We try to patch our Responder product about once every two weeks. We develop on very short cycles. Even a few bug fixes are good enough to warrant a patch. Even more interesting, we adopted a commerical patching solution used by MMO's (massive multiplayer online games) - we figured the patch-every-2-weeks system was very close to the game industry requirement. I think that model is pretty advanced for a software company. Our upcoming releases are very exciting, least of which is the 64-bit upgrade to our platform. That means analysis of 64 bit windows, running on 64 bit windows, and also imaging 64 bit systems. The acquisition of 64-bit data is much harder than it sounds, it turns out. Not only do we have to analyze physical memory snapshots from vista and 2003 64 bit images, we also have to be able to acquire them. To do this requires a device driver. And, since we are adding pagefile support, we also have to build a parser for raw NTFS filesystems - THAT is a major effort and is non-trivial. Our next patch will add pagefile support for 32 bit systems, and the follow on will add 64 bit. There is a huge amount of engineering going on internally right now. I haven't had time to write about world events.

Crossing the Streams – Blizzard vs MDY

In the case against Michael Donnelly, Blizzard has once again conjured interpretations of the DMCA that border on magical. Without dissecting the whole thing, the claim is basically this:

1) If you copy an executable (.exe) from disk into RAM, this is copyright infringement

If the argument stopped there, this means that anyone that executes the software is, by claim, infringing upon copyright and violating the DMCA. Obviously there has to be more. So, it goes like this:

2) If you agree to and abide by the EULA, the EULA grants you a limited license to copy the program into RAM

This means that legitimate users who agree to ALL the terms of the EULA have a license to copy the software into RAM (aka, execute it). For most regular users, this is where it stops. For anyone else who may have violated the EULA, this doesn’t stop. Here it what it means:

3) If you violate the EULA, you have violated your license to copy the program into RAM

4) Since the copy of the program in RAM is no longer licensed, this is copyright infringement

Stop there. If you connect the dots to this point, we have just drawn the following:

5) If you violate the EULA, you violate copyright

It is exactly here where the streams are crossed and the dark gate opens.

What started as a breach of contract has been polymorphed into a copyright violation. In the real world, copyright law and contract law are two entirely different sets of laws. Damages are even paid in different ways. They don’t belong in the same lawsuit.

In this case, there is a well established body of contract law that protects Blizzard against what Donnelley is doing that has nothing to do with copyright law. Yes, wowglider breaks the EULA, but NO Wowglider does not make any illegal copies of the video game, never attempted to in the first place, and has no intention of doing so. It’s a program for botting, not making copies of the game. The DMCA has no place in this case.

So, what I am trying to understand is - why Blizzard is bothering to use copyright claims at all? By selling a bot that knowingly causes the user to break the EULA, Donnelley may held accountable for damages that result from this widespread breach of contract under something known as “tortious interference with contracts”. Why are they trying to grossly over-extend the DMCA to make it apply to EULA breaches via the “RAM connection”?

I am left with the feeling that something sneaky is at play. This isn’t just because of bnetd case history or even that the attorneys at Sonnenschein have had prior success with DMCA arguments. It may only be that DMCA is new, so it’s easier to “get away with things”. It may simply be that Hollywood-types like to make everything about copyrights. But, even these straightforward answers don’t add up for me. I think there is something much greater at stake here. The future of persistent virtual worlds is huge – billions of dollars. I think that by polymorphing EULA’s into copyright infringement, Blizzard is setting the stage not for WoW, but for their next big game. This case law will be used to protect their future games from competition.

As we all know, EULA’s typically contain clauses about the use of 3rd party programs, publication of performance statistics, and forbid reverse engineering. Taken to the extreme, I could become a copyright infringer because I publish in my blog the percentage of CPU usage used by WoW as reported by Microsoft’s taskman utility. BTW, the usage averages about 49-51% on my Vista machine. So, I have violated the EULA, therefore my copy of WoW in RAM is no longer valid, and I am now a copyright infringer that may have to pay six figures in willful damages.

Blizzard has pulled out the DMCA billy-club again and again - and they may have reached too far this time. The argument that RAM is the magic connection between a EULA agreement and a copyright infringement is creative – yes they get lawyer credits for coming up with that. However, the claim that making a copy of a copyrighted executable into RAM is copyright infringement is simply wrong. It’s wrong because it’s against the law. Unfortunately, in this ruling, the Judge didn’t see it that way.

Copyright law specifically addresses the RAM issue clearly in 17 U.S.C. §117. In order to execute a copyrighted software program, a copy must be made into RAM. That a copy has to be made into RAM is a simple requirement to get the software to execute. This section of copyright law makes it clear:

(..) it is not an infringement for the owner of a copy of a computer program to make or authorize the making of another copy or adaptation of that computer program provided:

(1) that such a new copy or adaptation is created as an essential step in the utilization of the computer program (...)

Could it be more plainly stated? The copy into RAM is not something that infringes copyright, per the law. However, the court’s opinion was that “users of WoW and Glider are not entitled to a section 117 defense, and Glider users therefore infringe Blizzard's copyright”. This is a travesty.

Nobody here is saying that cheating should go “unpunished” – it’s clearly a violation of the EULA. It may be a stretch to prove damages that result from wowgliding, but Blizzard is welcome to try. It is reasonable for Blizzard to want to stop botters. There are established laws and methods by which Blizzard can do this (again, that have nothing to do with the DMCA). But stop there. Bringing the DMCA into play is reckless and short-sighted. If handled improperly, it stands to set legal precedents that take away fundamental rights granted to software users. EULA’s can contain any restriction imaginable - they could turn the most absurd actions into DMCA violations. Let me just ask you this - does a company that pulls in $1.5 Billion in annual revenue need to employ such brazen tactics to be and stay successful? One has to ask, how much is enough? Frankly, I think it borders on rude.

Whitelisting is the next snake oil

Most people experienced in computer security know that ‘signatures’ are the dominant technology used to combat malware. Signatures – short descriptions of otherwise large binaries, are extremely effective at detecting specific, known programs and documents. They are perfect for scanning the enterprise for known malware, known insecure software, known intellectual property. They are the cash cow of the anti-virus companies.

There are two approaches to signatures – blacklisting and whitelisting. The idea is simple – signatures of known bad stuff is a blacklist, signatures of known good stuff is a whitelist. Blacklisting has been the preferred method for AV over the last decade. Blacklisting has the benefit of near-zero false positives – something customers expect. Blacklisting also keeps the customers coming back – new malware means new signatures – perfect for recurring revenue models for vendor’s balance sheet.

Blacklisting sounds ideal, but it doesn’t work. New malware emerges daily that has no corresponding blacklist signature. The malware must first be detected, and then processed. There is always a time window where Enterprises have no defense. Recent figures suggest that the AV vendors are falling so far behind this curve that they will never catch up with the deluge of new malware arriving daily. It can take weeks for a signature to become available.
This deluge of new malware is due to several factors. First, there is more money behind malware development than ever before. Second, we weren’t really that good at capturing malware in the past. Today, new malware can be automatically collected, without human intervention. The slow trickle of malware turned into a flood as honeypot technology emerged. Sensor grids can obtain new malware samples with efficiency - they automatically ‘drive by’ (aka spidering) malicious websites to get infections and leave open ports on the ‘Net so automated scanners will exploit them. In parallel to the automated collection efforts, cybercrime has risen to epic levels. Finally, the barrier to entry has dropped for the cyber criminal. Cyber weapon toolkits have become commonly available. Anti-detection technology is standard fare. New variants of a malware program can be auto-generated. A safe bet is to expect thousands of new malware to hit the Internet per day.

The flaw in blacklisting has been exposed – it cannot address new and unknown malware threats. Figures range, but a safe claim is that 80% of all new malware goes undetected. This isn’t just a minor flaw; it’s a gross misstep in technology. Blacklisting is, and always has been, snake oil.

Enter the whitelist. The whitelist seems like a natural response to the “new and unknown malware” problem. Anything that is not known to be good will be considered suspicious, or possibly bad. Sound familiar? Whitelisting is not new, of course. Programs like “Tripwire” were in the market in the 90’s – and proven not to work. I founded rootkit.com originally to disprove the entire concept of OS-based whitelisting.

I agree with the idea that “suspicious” is good enough to warrant a look. This is smart thinking. Whitelisting is the solution.

There is a lot more “not-known-good” in the Enterprise than actual malware. Obviously the Enterprise cannot afford the additional workload caused by “false positives”. So, racing to catch up are the whitelist vendors – to remove all the “noise” so the staff can focus on the signal. Millions of dollars are already being invested into whitelisting files – and there are solid technical reasons this doesn’t work.

Whitelists are based upon files on disk. A whitelist, in current industry terms, means a list of the MD5 sums for files ON DISK. Please understand that files on disk are not the same as files in memory. And all that matters is memory. When a file is LOADED into memory, it CHANGES. This means on-disk MD5 sums do not map to memory. There are several reasons memory is different:

1) Memory contains much more data than the on disk file
2) Memory contains thread stacks
3) Memory contains allocated heaps
4) Memory contains data downloaded from the Internet
5) Memory contains secondary or tertiary files that were opened and read
6) Memory contains data that is calculated at runtime
7) Memory contains data that is entered by a user

All of the above are not represented by the file on disk. So, none of the above are represented by the whitelist MD5 sum. Yet, when the file hash on disk passes for white-listed, the running in-memory file is considered whitelisted by proxy. This is where the whole model breaks down. In memory, there are millions of bytes of information that are calculated at runtime – they are different every time the program is executed, the DLL is loaded, or the EXE is launched. These bytes are part of the process, but unlike the file on disk they change every time the program is executed. Therefore, they cannot be whitelisted or checksummed. This data can change every minute, every second of the program’s lifetime. None of this dynamic data can be hashed with MD5. None of this dynamic data is represented by the bytes on disk. So, none of it can be whitelisted.

While an executable file on disk can be whitelisted, well over 75% of that program cannot be whitelisted once it’s actually running in memory. This missing 75% can easily contain malicious code or data. It can contain injected code. It can contain booby-traps in the form of malicious data. It can represent an injected thread. The assumption that an on-disk whitelist match means that this dynamic data is ‘trusted by proxy’ is absurd. Yet, this is what the whitelisters want us to believe.

For malware authors, the whitelist is a boon. It means that a malware author only needs to inject subversive code into another process that is whitelisted. Since the whitelist doesn’t and cannot account for dynamic runtime data, the malware author knows his injected code is invisible to the whitelist. And, since the process is whitelisted on disk, he can be assured his malware code will also be whitelisted by proxy. So, in effect, whitelisting is actually WORSE than blacklisting. In the extreme, the malware may actually inject into the desktop firewall or resident virus scanner directly as a means of obtaining this blanket of trust.

The mindset that “suspicious is good enough to warrant a look” is a step in the right direction. But, whitelisting is not the correct approach. The only way to combat modern malware is to analyze the physical running memory of a system. In memory will be found the indicators of suspicion, and this is much more like a blacklist than a whitelist – except that it’s generic and based on the traits and behaviors of software, not hard signatures. For example, there are only so many ways you can build a keylogger. Once you can detect these traits in runtime memory, you are going to detect the keylogger regardless of who wrote it, what it was compiled with, what attack toolkit was used, or what it was packed with. As a security industry we need to stop climbing uphill with traditional approaches proven not to work. We need to change the fundamental way we do things if we are going to win.

Flypaper 1.0 Released

I'm happy to announce the release of a free tool from HBGary. It's something I put together to save me time when doing malware analysis for customers.

Most malware is designed into two or three stage deployment. First, a dropper program will launch a second program, and then delete itself. The second program may take additional steps, such as injecting DLL's into other processes, loading a rootkit, etc. These steps are taken quickly, and it can be difficult for an analyst to capture all of the binaries used in the deployment. HBGary Flypaper solves this problem for the analyst.

HBGary Flypaper loads as a device driver and blocks all attempts to exit a process, end a thread, or delete memory. All components used by the malware will remain resident in the process list, and will remain present in physical memory. The entire execution chain is reported so you can follow each step. Then, once you dump physical memory for analysis, you have all the components 'frozen' in memory - nothing gets unloaded. All of the evidence is there for you.

HBGary Flypaper is designed to be used with a virtual machine. Once activated, Flypaper will also block network traffic to and from the machine. If you are using HBGary Responder with the virtual machine, only the traffic to and from Responder is allowed, effectively quarantining the malware for analysis. (Note, this blocking operation would not block NDIS level rootkit material, only malware that uses the existing TCP/IP stack.)

You can get it from the HBGary website. (www.hbgary.com)

Microsoft wipes out 700,000 - too late to the game

A very interesting post came out on the MMPC blog today – Microsoft added some sigs to capture Taterf and Frethog malware variants and captured waaaay more than they expected (http://blogs.technet.com/mmpc/). On the first day alone they detected 700,000 Taterf variants, millions in the first week. What is interesting is the sheer volume of malware designed to steal online gaming credentials. This is equivalent to the threat faced by financial institutions every day in the form of keyloggers that steal financial credentials. Except, in this case, the money is stored in game servers. But, like all money – money is just a digit in a computer somewhere. This is not different. The target smells the same if you step back. Just like stolen banking accounts, these accounts are stored in a bad-guy SQL server somewhere and sold for cash based on whatever inventory the character happens to have. The Asia-Pac region is already full of companies that farm gold (aka ‘real cash economy’) – they already have existing relationships with real purchasers in the real-cash economy with set quotas. So, it’s not a stretch to imagine they can clear out and launder 50 million wow gold in 90 days. At the scale of the malware infection described in Microsoft’s blog, this was a huge operation (with the sheer volume of flash and quicktime exploits over Q1 this doesn’t surprise me either). And, by the time these infections were cleaned by Microsoft, it was too late. The game was already over.

Welcome to Greg Hoglund's new Blog

Welcome to my new blog, Fast Horizon. I have retired my old blog on rootkit.com and opened up shop here at blogger. I am the CEO of HBGary, Inc. (http://www.hbgary.com/) – a new company in the computer security industry. We released our first product this year (Responder, www.hbgary.com/responder_pro.html). HBGary is actually about five years old, but until now we have been a services company working primarily for the U.S. Dept. of Defense and Intelligence Community. I am excited to be part of the shift toward product development. This is my third startup. I am the author of three books and have been educating people about security threats – especially rootkits – for almost 10 years. I have a great foresight for trends – thinking of ideas about 5 years too soon for the market - and an almost cynical edge to my observations. Most people know me as a hacker, but in truth I probably know more about business and product development than hacking at this point. All of my startups have been in software development. I have probably experienced every management nightmare that can be listed, and dealt with it. I like to take big bites - so HBGary is tackling the biggest threat in computer security today – malware. Unlike most companies however, we aren’t selling snake-oil. Instead, our philosophy is that it’s IMPOSSIBLE to keep the bad guys out. The billions of dollars spent on security since the millennium has been a complete waste. Instead, we assume the bad guys will succeed – and it’s our job to catch them once they get in.

I could describe our solution as a platform for analyzing physical memory. You see, if there truly is a cyberspace in the Enterprise, it’s represented by the ones and zeroes in physical RAM.

There are only three kinds of data in the enterprise:
- Data at rest, on hard drives
- Data in motion, over the network
- Data in execution, in RAM

For any data to be used, it has to exist in RAM. Everything that matters must exist in RAM. By being in RAM, you are the center of the universe. Yet for all its power, until now nobody has a platform to analyze RAM. There are host-based IDS products, and AV, but all of these depend on the OS to query things about the OS – age old rootkit problem. The system is subverted and it’s game over. Our solution steps aside the OS and analyzes the physical RAM snapshot –offline-, thus avoiding any malware trickery.

There is a high barrier to entry to this work. We open the RAM, look inside, and extract objects. We reverse engineered every version and service pack of Microsoft Windows to be able to do this. We can find every process, every driver, and every line of assembly code of every software component. And, we do it without using the operating system – we do it without executing the environment we analyze.

In my grand vision we will build a picture of the true enterprise cyberspace. We have radical new technologies, like Digital DNA, that can be used to identify fragments of documents, strains of malware, intellectual property, fingerprints of email attachments, etc. Although we are tackling malware, our platform is generic and could be used for many other markets (IP asset tracking, E-Discovery, etc). As a company, we couldn’t ask to be in a better place in a market. We are set to explode.