Virtual Infrastructure Security Facts

• The number of virtual servers will rise to more than 1.7 million physical servers by 2010, resulting in 7.9 million logical servers.  Virtualized servers will represent 14.6% of all physical servers in 2010 compared to just 4.5% in 2005 (IDC)
• 60% of production virtual machines will be less secure than their physical counterparts through to 2009 (Gartner)
• More than 75% of respondents cited reducing infrastructure hardware and software costs as the critical driver in data center planning (Ziff Davis)
• Overall virtualization market has grown from approximately $560 million in 2005 to a forecasted $2.7 billion in 2009 (IDC)
• 10% of servers will be virtual by 2009, 60% by 2013 (Gartner)
• Fewer than 10% of organizations are doing anything special for virtualization security (Ziff Davis)

Virtual Appliances (VAs) have several advantages over Live CD distributions.  They are easier to enable persistence and customize (especially for real performance in a VM, instead of via a bootable ISO).  It’s easier to take snapshots that represent a “point-in-time” to rollback configurations — or prevent security scanners from running into loop or crash conditions.  Cloning and templating can have significant advantages in terms of agility for testing and scaling architectures, in addition to aiding changes and repair processes.

Microsoft (including the free Hyper-V Server) and VMware (including the free ESXi) are the major players for hardware-VMM server virtualization, with the FOSS project, Xen, being prominent in some other product implementations.

Both VMware and Microsoft have their own disk formats for importing VMs (aka “Guests”) on to their Hosts (aka Hypervisor or Virtual Machine Monitor — VMM).  There is also a third, open format called OVF (or Open Virtualization Format).

1. Microsoft: VHD (Virtual Hard Disk)
2. VMware: vmdk (virtual machine disk)
3. Open Virtualization Format: ovf

Sometimes, one-off scenarios will utilize tar, zip, or rar files to distribute VMs or encapsulated VMs, but this is becoming more and more rare.

Virtual Appliances

A Virtual Appliances is a pre-packaged VM.  Normally, a VM is just like a new machine — no OS, no nothing.  Virtual Appliances come with stuff, and usually only require booting into a DHCP-enabled network, where they self-configure themselves and become available via a web interface for further interaction.

You can find VAs at the following sources:

• VMware — http://www.vmware.com/appliances/ http://www.vmware.com/appliances/partner/
• Microsoft — VHD Test Drive Program, VHD Partners, VHDs by Product

For those of you still using the outdated OSI model (i.e. you stupid network security geeks, j/k ;> ), here is a general layout of what is available for you:

• Layer 7 – Stonegate Virtual IPS
• Layers 5&6 — Checkpoint VPN-1 Virtual Edition
• Layer 4 — X-m0n0wall
• Layer 3 — Vyatta Community Edition 5
• Layer 2 — HoneyMole

Certainly, if you haven’t read or seen Chris Hoff’s various recent presentations, then you’re going to screw this up.  However, anyone with even a few weeks of virtual infrastructure experience will understand the application of the above VAs in a virtual infrastructure environment.

VMware is very useful for fuzz testing (as seen with Sulley and other frameworks which include interfaces to VMware monitors), and full-state or kernel debugging (as seen with Syser, the replacement to the classic SoftICE), but this is more often for the VMware Server/Workstation products, not their Virtual Infrastructure products (i.e. ESX, ESXi, Virtual Center, vCenter Server, and vSphere).

Many ISOs are moving to VAs.

Many demo-ware and software evaluations are moving from standalone installs directly to VAs (i.e. demo the new app on the new OS at the same time!).

Take these examples outlined in the next sections for a test drive.

Pen-test VAs

Would it be nice if you could setup a perfect pen-test environment, save it, and then clone it a bunch of times in order to tweak one specific thing and then run all your tests in parallel (say, with different credentials).  Well this is exactly what Pen-test VAs are going to allow you to do.  One machine: 4 web application security scanners.

Or better — run DRS (VMware’s Distributed Resource Schedule), which will automatically move VMs around contended Host resources.  Say you have four physical machines, all with a dual-core 2.2GHz proc and 3GB of memory.  Now say that you’re scanning some client machines in far away places (with constant ISP bandwidth churn on both ends — and in between).  Let’s pretend you have this setup:

• IBM AppScan running default-mode with regular user credentials
• Acunetix WVS with AcuSensor tweaked specifically to the app using (at the very least) the web configuration files and structural layout.  One of your co-workers is changing the configuration as he/she learns more about the app from the client and working with the Acunetix support team
• WebInspect running in four more VMs, two with admin rights — two others with user rights.  They’re setup to do parameter tampering and see if they can pollute access controls from admin to admin, user to user, or any combination

If any of you know what CloudAV is… think what CloudWASS would look like.  I call it “WhiteRockSec”, which is… “like WhiteHatSec, but on Crack”.

Of course nobody has built these VAs yet.  In the meantime, you can use these two VAs to accomplish something similar:

1. OWASP Live CD VA
2. InGuardians Samurai Web Testing Framework

WAF VAs or as I like to call them: VA+WAF

VA+WAF is a Virtual Appliance that includes a WAF.  To those of you who don’t love my humor, you’re bound to definitely hate me for flipping the script on this marketing terminology.

Because network vendors (F5, Citrix, Breach, Cisco, Barracuda, Imperva, et al) really like to sell expensive appliances, it’s likely that they aren’t too keen on the idea of selling a software-based VA that is equivalent to their mind like an ISO (anyone remember the presentation on how to reverse-ISO a Netscreen IDP onto cheap PC hardware?). So you don’t see too many of these around yet.

I did happen to find these two though:

1. Microsoft IAG 2007 Virtual Machine Trial 
2. Security Enhanced Web Application Server with mod-security

AppDev/AppSec VAs

Again, there really isn’t much here yet.

Microsoft has:

1. Visual Studio Team System 2008 VSTS Hyper-V Image (Trial)
2. Visual Studio Team System 2008 TFS Hyper-V Image (Trial)
3. Microsoft Pre-release Software Visual Studio 2010 and .NET Framework 4.0 Community Technology Preview (CTP)

Note well that the last link above, for the VSTS 2010 pre-release, has the VA in “vmc” format.  “vmc format” was from Microsoft’s older product.  Searching the Microsoft Download Center for vmc or vhd both have great results, but hopefully Microsoft will standardize on VHD or OVF.  For now, you can convert in many ways — including the latest tool from Microsoft, the VMC to Hyper-V Import Tool.

Integrating AppSec with the above VSTS and TFS tools is relatively easy.  For those not familiar with FxCop, StyleCop, and CAT.NET — you certainly should be.  TFS has some great built-ins for Governance that apply equally well between quality and security.  The TFS Team Blog has some decent postings on topic, not directly to security yet (but probably in the future).  I’m working on additional ideas, heavily borrowed from the Microsoft Process Templates and Tools development center — and from watching how Microsoft uses TFS with their new MPT toolkit.

Security folk such as myself might want to just load Source Insight (or the Microsoft Express Editions) along with using the command-line CAT.NET or possibly SharpDevelop until Ounce O2 is widely available.

For Java, you can search the VMware Appliance Directory, but I found nothing useful.  Currently, the easiest and cheapest way to get JEE AppDev/AppSec going is to use EasyEclipse.   There is a commercial equivalent called Yoxos that also sounds very promising.  I think most of us would be flying blind without a few Eclipse plugins such as Classlocator, Jupiter, Flow4J, IvyDE, FindBugs, and PMD.  Build server ISOs such as Buildix would be wonderful to turn into a VA.

Again, us security folk would probably stick to Source Insight and/or SciTE along with the command-line versions of FindBugs and PMD.  Static analysis tools are slowly turning to be out of vogue these days… so YMMV.

Summary

Learning Virtual Infrastructure is going to take some time, but the payoff is worth it.  In no time, you’ll be turning your minimally-equipped Security Operations Center or appsec group into a real infrastructure to fear.

Download the hardware-VMMs to “whitebox supported” hardware (note: this doesn’t always have to be on an “official list” from the vendor).   Try both the evaluation versions (Microsoft Windows Server 2008 R2 Beta with Hyper-V Role enabled ; VMware ESX and vCenter Server VA) and the free ones (Microsoft Hyper-V Server 2008 R2 Beta ; VMware ESXi).  Download a few VAs in various formats and learn how to import and start them.  You’re on your way!

I thought I’d take a moment to post about some web security tools I use pretty often, which help as a security consultant when responding to various web hacking related incidents. These tools have helped me write my own scripts whenever I’m in a jam and need something good and quick to do the job.

Application Log File Forensics: The Hard Way

The first thing a security professional or administrator usually think of when handling an application security incident is to check the logs for the applications, databases, and other application-tiers involved.  Often, these logs are either on the servers that run the applications themselves, or possibly in a central logging location.  If a certain attacker tool can be identified from the log files (or other sources such as full packet-capture), then it may be of interest to run that exact same tool against your own application-under-target (preferably in a mocked-up lab or test environment, if it mirrors production well enough).

The most popular web servers, Apache httpd and Microsoft IIS, do create local log files by default.  According to most compliance regulations and standards (e.g. COBIT, HIPAA, GLB, PCI-DSS, FISMA, EU Directive on Privacy and Electronic Communications, ISO 17799/27002, CA SB1386 and similar), logging must be centrally located, or may have other required provisions.  This may include application-layer information, such as the log information from Apache and IIS.  It may be very likely that your organization already has centralized logging where this information is available.

If centralized logging does not exist, it may be a good time to start up a project to enable it.  The Apache Cookbook, 2E, is the best place to go in order to configure httpd to start sending syslog information.   It’s about as simple to add “ErrorLog syslog:user” into the httpd.conf file, but this only logs error messages, not authentication/access_log messages.  The book gives two prescriptions, one using “AccessLog “|/usr/bin/logger” combined” if your OS supports the logger command properly.  The other is to run a custom log message through a Perl script, as seen below:

CustomLog |/usr/local/apache/bin/apache_syslog combined
cat > apache_syslog
#!/usr/bin/perl
use Sys::Syslog qw( :DEFAULT setlogsock );
setlogsock('unix');
openlog('apache', 'cons', 'pid', 'user');
while ($log = <STDIN>) {
syslog('notice', $log);
}
closelog;

Microsoft IIS will need to go through the Event Log, which can be converted to syslog messages using a third-party software package such as Snare or MonitorWare Agent.  If IIS logs can also be converted to w3c standard log format, then Apache log analyzer tools such as AWStats could also be used.  W3C also has their own log analysis tool that also does HTML validation, called the Log Validator.  These may be useful to run following your own scan of the application using the same or similar attacker tool, as they will not only point out where in your application the scan/tool covered, but also where you may have the most errors or lack of quality/security controls.

The book Practical Information Security Monitoring also makes some suggestions for log collections, including the use of Sawmill or Splunk to sort/search log messages and gain further information and detail.  There may also be further adjustments you will want to do at the application (or other tier) layer, such as logging POST data.  We discussed logging HTTP referrers on our old post: Using Google Analytics to Subvert Privacy.  Practical Information Security Monitoring talks about Oracle audit logging, but there is also a detailed article on Pete Finnigan’s blog on Oracle forensics and UKOUG.  At the recent BlackHat DC conference, David Litchfield gave a talk on The Forensic Investigation of a Compromised Oracle Database Server, which may also be of interest (once the slides are available).  There are also some new books coming out on the topic of Oracle Forensics in the next few months / year.

Web Application Incident Handling: The Easy Way

Most of the logfile “digging” takes time, even when consolidated and using expert tools and analysis.  There are some very easy approaches that we’ve come up with, or seen others using and talking about.  These tools integrate well at the HTML and Script layers.

Over a year ago, Mario Heiderich started the PHP-IDS project, as a way to build protection and monitoring capabilities into PHP applications. Several side projects spurred up as a direct result of the incredible work that was put into PHP-IDS, mainly its default_filter.xml regular expressions. This XML file of regular expressions provides capabilities to detect a vast range of attacks, including XSS, CSRF, SQL Injection, Directory Traversal, Local/Remote File Execution, DoS, and Information Disclosure. Part of the success behind the PHP-IDS project, was the constant testing and attacking of PHP-IDS regex filters, which can be reviewed extensively in this sla.ckers.org thread. More info on PHP-IDS can be found in the PHP-IDS FAQ.

Romain Gaucher, wrote Scalp, an Apache log analyzer in Python, which leverages PHP-IDS’ default_filter.xml to detect attack strings in logs. I’ve used Scalp on numerous occasions, including a recent attack attempt on tssci-security.com. By nature, Scalp cannot examine POST content because Apache logs do not contain POST data. (See PHP-IDS or mod_security for those purposes)

Simply use Scalp by running it as follows (keep in mind there may be false positives with regards to the attack type, though it is very good at pulling attack queries from the log):

./scalp.py --log access_log --filters ./default_filter.xml --html --tough --exhaustive

Arshan Dabirsiaghi recently released OWASP Scrubbr. Scrubbr works by detecting input data in a specified database that does not match up with a specified AntiSamy policy file. Because Scrubbr uses an AntiSamy policy to validate data, does not mean it necessarily detects XSS in your database. Note, one does not require AntiSamy to be implemented in an application to use Scrubbr. Using Scrubbr, you have the capability of validating each and every column capable of holding strings of every row of every table in a database.

Together, Scalp and Scrubbr make for excellent web application security forensic tools. Scalp can help detect attacks in Apache logs, and Scrubbr can help you clean your database of content that does not match your site’s policy.

There is no doubt in my mind that some very strong experts out there have put WAF or WAF-like technology to good use.  However, WAF is dead and dying regardless.

I think that very large-installation, Internet-facing web applications require Anti-DDoS technology in the form of an appliance, preferably one that does rate-based behavior detection.  I often feel that those same organizations also require SLB appliances, although I prefer to see these integrated with a switch fabric in a chassis-based, large backplane network switch.  In a year’s time, SLB Layer-2 technology could be replaced by VMware DRS clustering and/or an equivalent like Microsoft PRO.  I was always a fan of Anycast to replace SLB at Layer-3.  I continue to suggest these models/architectures today.

Can whitelist WAF technology be used by those same devices in the short-term (Anti-DDoS or SLB appliance)?  Absolutely, as long as it’s done by an expert and tuned to the applications.  Should these devices sometimes be separated out of a traditional operational role, due to auditability and for compliance scoping purposes?  Probably not.  Should they perform monitoring, debugging capability, or solving hard production problems?  Probably not.

The reason that the first question is a yes, and the others are a no is because Anti-DDoS and SLB devices are already performance-ready-capable of providing WAF whitelisting functionality (note: not in all cases, but this works especially well for devices that provide rate-based behavior detection before mitigation).  Monitoring does NOT require an inline device.  All it requires is network taps (or potentially port-mirroring, but most professionals recommend taps over SPAN ports).  Also, infrastructure is changing rapidly, so it’s not wise to invest in a dying model.

Additionally, I know that companies like Sourcefire and Reflex Systems are integrating at the VMsafe API layer, which is a hypervisor introspection layer much like XenAccess.  This is really where much of the AV/IDS/IPS/HIMS/DLP/WAF/blacklisting-whitelisting technology belongs.  VNET will also change the introspection layer (in addition to almost completely eliminating the physical network layer and SIM/SEM/SIEM/NMS/EMS moves & changes), as it simply adds to introspection functionality.  I have already alluded to Cisco AXG becoming a VNET “module”, but what if Reflex Systems or StoneSoft start integrating WAF not only as a VNSS (Virtual Network Security System), but also at the hypervisor introspection layer?

Fortunately, for application security, server virtualization and the evolutions it’s bringing with it e.g VNET and VMsafe, are going to dominate traditional networks and cut their existing budgets.  Unfortunately for application security, the new virtualization evolution also brings with it tons of object reuse (there are at least two new controls channels available to adversaries), and new ways of establishing covert channels.

This means a few things.  First of all, the word “firewall” is dead, and therefore, the word “web application firewall” and the associated acronym, WAF, are also dead.  Imagine today if there existed a control channel that, when taken over by adversaries, it became a covert channel that had unlimited object reuse control of every physical RAM on every computer in existence all at once.  This is cloud computing, but virtualized.

Not only that, but we are saying that adversaries have already bypassed traditional firewalls by using the application layer i.e.  Hacking Intranets from Jeremiah Grossman.  Thus, this master, covert control channel is already on its way to being built (at least as man-in-the-browser).  Imagine for a second that you don’t use NoScript with Firefox and additionally implement the features of Chrome by using multiple Firefox profiles.  Imagine for a second that you are a regular user, with all of those Clickjacking and modern application attacks available to anyone who wants to get to you.

Like many of us used the words “brick-and-mortar” to describe backwards-companies during the dotCom bubble, I think “fire-and-wall” well-describes organizations that continue to cling to traditional networks and network security as answers to Internet, Enterprise IT, and any operational risk.

Do I intend to sell you on the idea that we should all instead jump to Fortify RTA or Microsoft SRE?  No.  There are potential consequences to any of this.  This is only the functionality required to reduce risk to applications, not the assurance that risks have been removed.

TCSEC says that we need to balance functionality and assurance.  But nobody ever bothered to do any assurance.  Assurance is the Microsoft SDL, SDL Pro, and SDL-IT.  @Stake and Foundstone are gone and have split into tons of fractured security evaluation and risk assessment boutiques that have 1-300 developer-security-tester guru’s that mix SAST and DAST with expert review.

But the SAST+DAST market is less than $100M, while WAF is at least 20% more than that (although probably inflated).

I hate to be the bearer of bad news, but you don’t just say “DO BOTH” because nobody will do the SAST+DAST work.  We tried that last time, when tcp_wrappers and the DEC firewall came about  The underground that wanted to keep their covert control channels alive started dumping rootkits on pre-pwned Unix machines.  Then Dildog and others made it possible to easily access Windows machines, and after that - botnets and the like have reigned.  There are already backdoors in our web applications.  OWASP Scrubbr is not going to save us all by itself.

Who did the work back then?  OpenBSD?  Certainly not Microsoft, and even today their SDL appears to be failing by some, but imagine if it did not exist at all.  We obviously have to do better with assurance practices.

Can functionality-based controls work easier, better, and faster than assurance ones?  Are they that less complex and easier to train?  Or is there just more written about them because it’s easier to SELL them by baking them into products rather than customizing them to an ISV organization or an Enterprise development team?

If you are part of the group that is spending $120M on WAF technology, then you are hurting the SAST+DAST market because you’re taking away that spending.  Clearly, risk analysis is not taking place and people are spending based on familiarity in addition to PCI-DSS requirement 6.6, which all but forces the inequality to happen.

Look at the best in exploitation-countermeasure functionality-based controls that work on object reuse problems e.g. DEP, ASLR, SafeSEH, SafeInt, et al.  Are adversaries still bypassing these?  Security researchers in the offensive-research space are.  These countermeasures are closer to the code (even HIPS is closer than network-based IPS), like many WAF suggestions.  Is is true that we still require assurance even after 15 years of exploitation-countermeasure optimization?  I remember when stack-guard protections were first coming out - they were seen as a huge joke (i.e. toy/researcher technology), much like Fortify RTA, CORE GRASP, Microsoft AntiXSS-SRE/AntiCSRF, GDS Security GPF, and HDIV are seen now.

I know to many of you out there, this looks like a rant, and I really could go on forever about this topic.  So, go to the datacenter, give your WAFs a hug, and continue to buy into the “functionality is better than assurance” argument.  You’ll feel better in the morning, right after you forget that you just opened up your database to any talented people who want to make money from the data in it.

Also, pen-testing is dead.  We no longer need to prove that applications are insecure.  We know they’re insecure - no matter how many functionality controls you layer on top of them.  Unless YOU prove that the applications that YOU are responsible for ARE secure, you are working against the rights of users, consumers, cardholder data, personally identifiable information tied to healthcare and financial records, trade secrets, and the ability to control our critical infrastructure.  Enjoy.

Jim Manico invited Dre and I to join him with Brian Holyfield on this week’s OWASP Podcast. Topics of discussion included our thoughts on web application security, WAFs, training, among others. Give it a listen, and tell us what you think.

OWASP Podcast Series #6 (direct download link)

Brian introduced a tool he has been working on, SPF - Secure Parameter Filter, which has the features we would like to see in WAFs, and would recommend people checking out as an alternative to implementing a commercial WAF as a short-term fix.

Hey all, I’d like to introduce all of you to a new site Tyler Reguly and I, along with Romain Gaucher and Jay Graver set up last week, SSLFail.com. The site’s purpose is to point out the failures in various sites’ SSL implementations. We’ll be publishing tutorials, and informative articles on SSL in addition to pasting screenshots of high profile sites’ failures.

We came up with the idea for the site when Romain came upon an SSL failure with Gmail. Tyler then blogged about it, and then I was getting errors with Facebook.

The interesting things about Gmail, when you go to https://gmail.com, Firefox was the only browser we tested to follow the 301 Redirect to another domain (www.google.com) with a proper SSL certificate. IE7 and Google Chrome on the other hand, asked the user for confirmation before the redirect. Is this a Firefox SSL failure? I don’t know, and several others I’ve spoken with aren’t sure how a browser should handle it either.

Anyways, just wanted to point out this new site, which has already gotten some attention from lonervamp at terminal23 and hype-free.