tssci security

2007 Security Testing tools in review

In my last post, I explored some ways of using formal method tools to perform security testing in the most advanced scenarios. It may have been over the heads of many people, so I wanted to offset that by talking to some basic tools which I think anyone can utilize effectively assuming they bring the most important tool when security testing: their brain.

Of course, you need to think like an attacker, and saying that, I can't recommend a better source than CAPEC and their outreach and enhancement documents. Books that come highly recommended (read at least one of these) include:

I really think that having some background in development practices is also necessary, so I recommend other books such as:

If you want to write or develop exploits in addition to finding security bugs there are a few must have books such as:

If you have little operations background and want to learn how modern-day systems and networks are protected, I highly recommend:

These books will introduce you to hundreds of tools - many of which are useless, difficult to configure, difficult to understand, and/or about as robust as the software you're trying to break. While learning some background and technique is useful, by and large I think you'll get the hang of a few tools that do one thing well and stick with them.

The most important and widely used security testing tool starts with program understanding / comprehension. If you have source code for the application, SciTE is an excellent source code editor that can be scripted using a language that will appear throughout this blog entry: Python. Probably the best way to learn Python is to code Python to help you code Python using SciTE. This will also introduce you to recursion and meta-level concepts. However, you don't need to become an expert in every programming language to use SciTE for secure code review -- all you need to do is learn how to perform a manual DFA (or CFA) for security weaknesses on each language one-time, and then automate that DFA/CFA through Python scripts in SciTE.

Before manual code review, developers may have created program specifications. In the last post I mentioned formal top-level specifications, which is rare. More common may be UML models or DSD's. Using open-source tools such as Fujaba, which can move between UML models and Java code in Eclipse, may prove useful. Ideally, this would be done using a formal UML tool such as the commercial Rational Rose, or better -- one specifically for security design inspection, such as Klocwork K7. There aren't any free or open-source tools that are specifically geared towards secure design inspection, but expect this to change in the near future. The best work I've seen lately has been done by the Octotrike team, also recently spoke to at Toorcon 9 as Privilege-Centric Security Analysis.

As most security testers will come to understand, not all program source is easily available. For .NET assemblies (CLR), try using the .NET Reflector; Java can use Jode; and Flash can use Flasm or Flare. There are some C and other compiled-language decompilers, but consider using a debugger for your language if it becomes too complex to easily decompile (or too inaccurate). In other words, use a debugger such as the basicDebugView under Windows (for basic stuff), JSwat for Java on any platform, ImmDbg for advanced Windows debugging, and IDA Pro (for as many platforms as you own and have access to) if you really want to rock.

Sometimes it might be better to work with some languages in their native IDE instead of SciTE, such as using Visual Studio for .NET work, Eclipse for Java, Flash for Flash, or Aptana for Javascript. Again, it is often better to work from one solid tool than to install and use several different tools, although there may be a few one-off scenarios where you would want to use them. SciTE might not fail you as often as you would think, especially if you use it as a code reader (instead of relying on it to build code/classes like an IDE would, which isn't really what you should be worried about - let the developers do that hard work for you!).

IDA Pro, while being both an excellent debugger and disassembler, is a cut above the rest, although I could name plenty of other tools in its class. IDAPython is the language of choice for such a tool, bridging over to Immunity Debugger and several other tools. There is a free version of IDA Pro, but any serious person will shell out the money for the non-free version if they value the primary tool in their toolchain.

Debugging web application client-side code such as Javascript can't get any easier than by using FireBug, although I have found myself wanting better representations of code before using it. I can't recommend the Web Developer Toolbar functionality for viewing Javascript, source, and "generated source" highly enough -- but also View Source Chart, DOM Inspector (along with InspectThis), and UrlParams in the sidebar. For Flash apps, there is the FlashTracer add-on that can also be used in the sidebar, which responds to trace operations (you may need to put these in yourself), and also a local proxy called Charles, which supports Flash Remoting.

Viewing code and debugging code will help with program flow and understanding. Some Python tools such as HeapDraw and Python Call Graph make great visualization tools to aid in this type of work. You'll want to learn how to implement automated DFA (data flow analysis), view control flow graphs, basic block graphs, and how to view these under a scenario called hit-tracing. Hit-tracing will allow you to watch the flow of the application only under certain conditions that you want to see, while filtering out everything else. The PaiMei tool will provide this, and by doing so - you're really discovering an important aspect of code known as coverage; how much the testing affects the code it runs through.

There are right ways and wrong ways for both security testers and developers to utilize code coverage in their testing approaches. Besides the PaiMei tool, there are code coverage tools that are specific to certain languages, and that often require flags at compile/build time. gcov and lcov are examples of code coverage tools that require being built in to the software. Other approaches, such as NCover for .NET, and EMMA for Java (using EclEmma while in Eclipse) do not have this requirement. There are even coverage validators available for Javascript.

Normally developers utilize code coverage during a recent invention of quality testing made specifically for developers by developers called "unit testing". Unit tests are quick 1-3 second checks in their IDE that they can use to assert behavior and provide an instant fix. More advanced unit tests called "component tests" can be done during a build, along with mocks (fake database or other system necessary to assert behavior of a fully implemented application). Typically these are all for functional tests, and possibly also for performance. Rarely are applications tested for security in this manner, and unit testing was really built for functional tests (although they theoretically could be adapted for security testing).

It's sometimes ok to pretend you're a developer just to see what idiotic things they skip over when they write and build their code. In this case, you will want to load the code into its native IDE (say, Eclipse for Java). Then, promote all the warnings you can to errors, build, and see what the environment spits out at you. Load automated static bytecode analyzers, which often have a large amount of security checks (e.g. FindBugs, FxCop, CAT.NET, and AspectCheck), as well as source code checkers -- and use this time to write some of your own basic checks (especially to find low-hanging fruit). For Java, PMD is extensible through XPath, for an example see the PMD SQLi Rules. Also check what demos are available in terms of CWE-Compatible tools - I've used Fortify SCA to great effect here. There are also plenty of open-source static analysis tools, but many aren't as complete as their CWE-Compatible commercial cousins. If you can find some basic unit tests to run (e.g. Crap4J), these may also provide better program understanding, especially when combined with inspection tools such as Armorize CodeSecure, Fortify SCA, OunceLabs, Klocwork K7, GrammaTech CodeSonar, and the bytecode checkers (i.e. FindBugs, FxCop, CAT.NET, and AspectCheck).

Security testers have created their own sort of unit tests which involve injecting faults into an application, or even better -- sending random or specially-crafted data to the application inputs -- often called fuzzing or `fuzz testing'. Fuzz testing tools are numerous, although one of the best tools is EFS, which stands for Evolutionary Fuzzing System. EFS provides random data to an application along with code coverage reports from PaiMei. It enhances the tests using this code coverage data by implementing genetic algorithms. Some of these tests cannot be solved, but a satisfiability solver can provide extensive coverage beyond GA's in this situation. STP, zChaff, and Saturn are examples of satisfiability solvers, while catchconv integrates these concepts as a variant to Valgrind (a popular dynamic analysis tool, similar to IDA Pro, but focused mostly on memory debugging). Valgrind's Memcheck, DynamoRIO, Purify (commercial), DynInst, and Pin are actually dynamic binary instrumention (DBI) tools which would work well with PaiMei's crash binning routines when attempting to find off-by-one's and incrementing counter crashes (and other issues that may be better found through formal method security testing).

Random fuzzing along with code coverage is probably one of the most advanced tools in a security tester's toolbox. However, there are some cases where binary protocols hit protocol parsers on both sides of a connection. Tools such as ProxyFuzz and Universal Hooker can make it easy to determine whether one side or both sides of a connection require more than just random fuzz testing. If this is the case, a lot of manual work may be required to determine which parts of the data are encrypted (if any), compressed (if at all), or separated by TLV's (type, length, or values). Using a specification may be the easiest way to implement a specially-crafted fuzz testing tool for the "application under test" (AUT). Using a fuzzing framework such as Sulley or Peach can make building this tool much easier than doing so by hand. In some cases, files (not protocols) are loaded or unloaded by applications. Certain fuzz testing tools work specifically well for writing random or specially-crafted files, and others are provided by frameworks. Depending on the platform or type of application you're targeting, you'll want to look at a few different file fuzzing tools/techniques, but FileFuzz (by Mike Sutton) seems to stand out as a good starting point (similar to ProxyFuzz for protocols). For other file fuzzing tools, see FileP (my favorite, written by Tim Newsham), SPIKEfile and notSPIKEfile, Ufuz3, FuzzyFiles, and Mangle. I'll leave targeting browsers, web servers, and web applications with fuzz testing techniques for later, but these should give you a general idea of what is available. There's also fuzzing environmental variables, in-memory, etc. Note that Sulley, Peach, and FileP are all Python frameworks for coding specially-crafted fuzz testing tools.

Often there are situations where rolling your own fuzz testing framework or dissecting a protocol or file format may seem like the last thing you want to do. Prototyping this work should be a first step to determine if the effort is worthwhile. I gave examples of using both ProxyFuzz and uhooker as starting points. Combine this with code coverage, and check to see if the results show how well you've tested. Before starting a fuzz testing project using a framework or writing your own tool (or a full-blown client), you may want to do a few more checks. This is where knowledge of bash and C can come in as extremely helpful. Small C programs are fast, easy to write if you're just trying to do one thing well (run a looped test), and can be easily piped on the Unix command line to "try different things out". The bash shell provides ease of use when writing loops or conditionals for a bunch of small programs piped together, to combine speed with agility and flexibility in prototyping. Using the Matasano Blackbag tools (or by writing your own tools similar in nature/effect), it becomes easier (and more fun) to reverse binary protocols/formats than by reading specs and building run-once Python scripts. Of course, finding the Matasano Blackbag tools may be difficult as they now recommend writing protocol dissectors using Ruby along with BitStruct. A commenter posts using Construct with Python, as well as the popular scapy Python library found quite often in the literature.

Almost a lost art, but related to file fuzzing and binary protocol dissection is binary analysis. Many continue to use IDA Pro to perform these sorts of bug-finding techniques, using IDC scripts such as the one by Halvar Flake, BugScam. Tools such as bugreport, BEAST and OBOE can also be used, but it may be better to re-write this functionality in IDAPython, possibly by using x86 emulators, or PEiD to remove potential code obfuscations. Halvar Flake's company, SABRE Security, also maintains the best tools for program analysis, debugging, and testing for security. The primary reference tool, BinNavi, works on embedded processors as well as most platforms you can think of. I've heard he can make it available to people who cannot afford it depending on your project scope. BinNavi/BinDiff have simply the best reputation in the business, and like PaiMei, require IDA Pro.

After a security-related bug is found (using automated inspection, automated fault-injection, automated random fuzzing, or by partially automating specially-crafted fuzz testing, binary analysis, or code review), some will choose to report it to vendors, some will write exploits, and some will do none of the above. If an exploit is intended to be written, you'll find lots of support for doing so using the Immunity Debugger and the Metasploit package. There are some automated exploit writers including the MSF-eXploit Builder, Byakugan/noxdbg, and Prototype-8. Python appears to be dominant for exploit writing were it not for the Metasploit project, which has moved to Ruby. If the focus of a security testing team is dominant towards writing exploits, then it is suggested that Ruby equivalents replace SciTE, ImmDbg, IDAPython, PaiMei, and the various fuzzing frameworks and tools already mentioned, or that security testers learn both Python and Ruby equally well.

Writing exploits for web applications is a quite different matter. There is a reference platform for fault-injection and fuzz testing of web applications, which provides source for every vulnerability check. Unfortunately, it's the commercial Cenzic Hailstorm. Also unfortunately, this means learning another language: Javascript, but web application security testers should possibly make learning Javascript (and Flash) a priority over Python, C, or Ruby. CORE Impact (also commercial) has begun to add the low-hanging fruit exploits (although they claim no XSS yet), and it is imagined that ImmunitySec will do so as well (and these could be open-sourced like the SQL Hooker tool). For now, the Metasploit, GNUCITIZEN AttackAPI, and the BeEF framework appear to be the dominant exploit tools for web applications. The W3AF framework uses BeEF, but also includes many modules (some pinned for future release) that allow for pivoting including an RFI proxy, as well as using log or source code information to continue a more advanced attack. There are concepts such as XSS Tunneling (to run Nessus or similar through a man-in-the-browser exploit) and W3AF's Virtual Daemon (integrating web application exploits with Metasploit payloads).

In fact, finding web application vulnerabilities is often more than just program understanding along with standard fault-injection and fuzz testing. There are many intricacies to finding XSS and SQLi -- even other critical vulnerabilities such as HTTP Splitting/Smuggling can take many forms (see: Amit Klein on HRS). Fortunately, many of these are covered in The Web Application Hacker's Handbook, in fairly clever detail. The book even covers triple-encoded XSS attacks, second-order SQLi, OOB SQLi, and inference attacks. Almost all examples use the Burp Suite, as the primary author (Portswigger) also wrote that tool. The book makes special mention of some other point-tools, including the aforementioned inference attacks using Absinthe. SQLi can also benefit from non-standard methods such as using PQL.

Earlier in this entry, I mentioned some Firefox add-ons such as UrlParams. There exists a similar add-on that can be used as an HTTP editor (but only to modify requests, not responses): TamperData. While mentioned in Portswigger's book, he doesn't cover it as well as he should. TamperData can be opened in the sidebar and looks surprisingly similar to UrlParams -- this has great benefit, especially when testing for low-hanging fruit SQLi, XSS, and HRS vulnerabilities; as well as path traversal, logical flaws, URL redirection, and similar vulnerabilities which require direct tampering of the HTTP request headers. If you want to build advanced attacks, besides Hailstorm or Burp Scanner/Suite I can also recommend Wfuzz (written in Python) and CAL9000 (a browser driver, ideal for testing XSS in several different browsers using the same tool). Of course, there is the original HTTP/CGI attack tool, which has recently made a comeback as Nikto 2.

One tool is mentioned that I had not heard of before, and interestingly, the approaches given to attack authentication, session management, and access control are very different than the approaches that I propose (and the tools that I use). The CookieWatcher add-on is proposed as a way of looking for session ID's in applications under test. This is an excellent idea, and I've already integrated it into my testing strategies, as it is easy to change which session ID type you are looking for (it's the only tool option/preference, which has a drop-down menu that shows a list of cookies seen), delete, or copy (as well as view it in the status bar). While the lack of options is somewhat aggravating, the add-on works well along with CookieSwap (to have switchable cookie profiles), CookieSafe (to turn first and third party cookies off on a case-by-case basis), CookieCuller (to permanently protect cookies), and Add N Edit Cookies (to use the Cookie Editor to modify content, path, security, or expire information, as well as to add completely new cookies). WebScarab, Stompy, and the forthcoming Burp Sequencer are probably excellent tools for session ID analysis, and the book covers how to do this in detail, while pointing to the excellent NIST FIPS 140-2 standard, which has four levels of validation criteria. Interesting characteristics that may affect authentication or session ID randomization include IP address, BGP prefix, DNS forward or reverse entries, time, and HTTP Referer or User-Agent. Tools such as RefControl, User Agent Switcher, Tor, and pbounce should be used when testing for authentication (IP, DNS, and Referer authentication checks should also be configuration or source-code assisted if possible).

A lot of the functionality of the Burp Suite is utilized rather effectively in the book as well. However, the lack of inclusion of certain web application security aspects, such as Ajax security, automated DOM-based XSS scanning, and Web services security also happen to be problems with the Burp Suite itself. For Ajax crawling, I suggest checking out all the work done by Shreeraj Shah (especially Crawling Ajax-driven Web 2.0 Applications and Scanning Ajax for XSS entry points), including his new BlueInfy tools. For DOM-based XSS, the W3AF tool has a module to support this that is equivalent to Shreeraj's work. Web services security is also best covered by Shreeraj Shah, although there will be a few books coming out in the next month that should re-address the issues of Ajax and Web services rather well. From my experience, the OWASP Interceptor tool, as well as the commercial (free limited version) SOAPSonar tool are good starting points for Web services security testing outside of Shreeraj's tools already mentioned. There are fuzzers, including the OWASP WSFuzzer, SIFT, and iSecPartner's WSBang, as well as generic XML fuzzers such as untidy and Schemer.

It is strange that without mentioning much about Ajax, XML, or Web services that the authors included a very detailed section on JSON Hijacking and CSRF. While I was familiar with both CSRF dorks and CSRF Redirectors, the book contains excellent material on how to test (although does not provide a tool reference). Most recently, OWASP released a new project called CSRFTester, which looks promising. OWASP has a lot of great projects, but their integration (I agree with Ory Segal on this) could be vastly improved. There are very unique tools such as Pantera (Python extensible) and ProxMon (also Python extensible) that do passive analysis for web application security testing, which can save countless hours of manual penetration-testing using TamperData or Burp Suite. I often wish that other tools such as the THC releases and Foundstone free tools could also be combined, as I often reference these tools for the SSL checking support as well as a variety of other reasons.

Other tools such as DirBuster, JBroFuzz, and DFF Scanner can be used for predictable resource locations (PRL) -- especially while using ProxMon or Pantera passive analysis techniques (or by using these with FileMon/RegMon/Process Monitor on the Windows IIS web server, or strace, lsof, fstat, ktrace, or truss on the Apache/Unix web server). I expect the w3af and Burp Scanner will both integrate full Javascript crawlers, Flash walkers, and other advanced RIA features in the future -- which could even surpass the CWE-Compatible web application security scanners out there. In addition to this, they should add passive analysis tools and work with "agents" available in "white-box assisted" tests, such as the case with ImmunitySec's SQL Hooker as well as the PRL techniques described at the beginning of this paragraph. There will probably be four different types of agents: 1) the standard local proxy that sits in between the browser and the web server, 2) an agent that monitors the code/files/services/queries on the web server, 3) a proxy agent that sits in between the web server and database server (or other external services) and monitors like a testing spy (e.g. JDBC spy), and 4) an agent that monitors the code/queries on a database server or other external web service. Nobody has built this yet, but since we're on the topic, this is what I'd like to see in modern web application scanning tools. Maybe these agents or passive tools will also be able to measure code coverage and web surface coverage in a similar way that the commercial tools, Fortify Tracer and Chorizo Scanner (and the open-source Grabber/Crystal) accomplish this through bytecode instrumentation or other hooks.

For the year 2008, I'd like to see replacement of standbys like RSnake's XSS Cheat Sheet. While incredibly useful to me in 2006, it no longer holds its weight compared to the WASC Script Mapping project and tools such as .mario's PHP Charset Encoder, or Gareth Heyes' Hackvertor. One of the biggest lessons I learned about software weaknesses (SQLi and XSS in particular) is that there is a concept of low-hanging fruit (LHF), but at some point after the root-cause is found, more complex attacks often work. However, this is largely true only when a security tester has full source-code and framework knowledge. In this instance, a spreadsheet such as the one found in the Microsoft Press' Hunting Security Bugs companion content, "Reference -- ASP.NET Control Encoding", can be used to figure out which classes encode on a per HTML, Script, or URI basis. What I've been describing for awhile now is what will be known as hybrid analysis, with hybrid (static + dynamic) tools becoming more rapidly available to security testers.

For web applications, there are some great open-source static analysis tools that can get a penetration-tester started in using a source-code assisted black-box testing method. For PHP there is Inspekt, Pixy, RATS, SWAAT, PHP-SAT, PHPSecAudit, PSA3, and FIS (File Inclusion Scanner, with the extended tool, WebSpidah). For Java, there is Milk (based on Orizon), LAPSE, and SWAAT (only JSP). ASP classic and ASP.NET have ASP Auditor, SWAAT, and DN_BOFinder. Javascript has JSLint. There are also static code analysis tools specifically built for browsers, such as Oink and DeHydra.

Speaking to browsers, I promised some information on fuzzing browsers (to include ActiveX as well). First of all, we must mention Michal Zalewski, who not only recently brought us Bunny The Fuzzer, but also started a modern version of the Crusades to create a battle between browsers and web applications when he wrote MangleMe. hdm followed to create a bunch of tools: Hamachi, CSSDIE, DOM-Hanoi, and AxMan (for ActiveX similar to iDefense's COMRaider). Before these guys, PROTOS had c05-http-reply, and eEye was still thinking about TagBruteForcer. Even the Mozilla team released a tool, jsfuzzfun this year, including collaboratively with Opera. The latest comes from GNUCITIZEN as WEB CLIENT FUZZER.PY and Ronald as BrowserFry.

Robert Hansen (RSnake) recently spoke at OWASP AppSec 2007 about Web Browser (In)-Security (slides not available yet, so I don't know what he covered). It's true that 89% of security vulnerabilities in browser plug-ins from this year were in ActiveX applications. However, that doesn't mean that you shouldn't keep your browser, Adobe Flash Player, Java installation, Adobe Reader, QuickTime, Windows Media Player, and Mozilla extensions up-to-date at all times. It also doesn't necessarily mean that IE is more insecure than Firefox, Opera, or Safari. They're all insecure and it's your job to find out where and why. RSnake did release some very cool code this year with the Master Recon Tool (aka Mr. T)` <http://browserfry.0x000000.com>`_. Expect to see more available at Jay Beale's ClientVA website (and read his presentation from Toorcon 9 on that same page).

Outside of regular security testing at home or in the lab, I'd like to address the two commercial Software-as-a-Service (SaaS) solutions available as outside security testing augmentation. WhiteHat Security makes a product called Sentinel, which embodies the WASC Threat Classification (i.e. a way of understanding attack-paths against web applications). WhiteHat has a mature understanding of web application vulnerabilities from an attacker perspective, which is ideal for people that are learning how to think like an attacker. Veracode also provides a service, SecurityReview, which also happens to be CWE-Compatible (and the only commercial solution that has to-date formally passed the criteria besides SofCheck, an Ada source code analyzer). CWE is invaluable information that can be given back to developers in order to fix security vulnerabilities and avoid software weaknesses in coding efforts.

For embedded systems such as the iPhone, or routers such as those made by Cisco Systems, there is plenty of research out there to get you started. If the device has a web browser, try using JS Commander to proxy a low-featured (but working) Javascript debugger. FX has plenty of material in The Shellcoder's Handbook (2nd Edition) to get someone started on writing vulnerabilities or exploits for Cisco IOS. The BinNavi tool has support for at least Cisco IOS and Juniper ScreenOS, as well as a few other key architectures. If any area is in need of formal methods for security testing - it's software that is burned into hardware - so consider taking embedded hardware security to the next level by referencing my previous blog entry on Formal Methods and Security.

Posted by Dre on Saturday, November 24, 2007 in Hacking and Security.

blog comments powered by Disqus
blog comments powered by Disqus