Dynamic testing tools - feedback from a beSTORM® buyer
Software applications are by nature prone to vulnerabilities. Of course, software developers identify and fix functional or logical errors during the development phase itself. But as so many Zero Day exploits continue to remind us, many unknown software vulnerabilities do still manage to slip past the testing phase without being found or fixed, sometimes with disastrous results.
Conducting manual tests on any software with large quantities of programming codes is impractical. Tool-based, automated techniques are needed to manage this effort proactively. One of the most effective ways to identify software vulnerabilities by automated testing is the use of Fuzzing.
Fuzzing is a tool-based technique used to identify software bugs during the verification phase; this can contribute to identifying undisclosed security relevant bugs. To this end, the input interfaces of the target software to be tested are identified, to which targeted data are directed in an automated fashion while the software is being monitored for potential bugs. This makes it possible to prevent third parties from identifying vulnerabilities and thus from developing zero-day-exploits.
But there is no single Fuzzing method. Efficiency of each method depends on its creator. While one Fuzzing method works without having any knowledge about the data it is mutating while another method seeks to understand the application better so as to mutate its individual elements better. Select of fuzzers is, therefore, always based on needs of each application or operational environment.
For our heart rate monitoring device software, we wanted to test, analyze and locate vulnerabilities in its software. The idea was to fuzz our system by systematically sending invalid or unexpected inputs and reporting consequences of such actions, exposing software defects and vulnerabilities. Further, as our device connects with other systems on the network over Bluetooth, Ethernet or Wi-Fi we wanted a multi-protocol, environment-variable 'Smart fuzzer' that could understand overall application better and test individual elements dynamically.
Our fuzzer selection requirements were:
- Exhaustive testing beginning with likely weaknesses and then expanding to every possible input variation dynamically
- Extended Test range needed to include every field in entire protocols with every possible input variation
- Certified ISA Secure with EDSA 1.0 & EDSA 2.0 qualifications
- Re-create the attack with proof of concept code exported during test phase as PERL script
- Existing software testing staff should be able to run tests on all protocols
- Easy to understand - UI for the Fuzzer should be easy to understand by our test team
Although, there are more than 250 Fuzzers in the market, most are used to test web applications (25%), network protocols (45%), file formats (15%), Web browsers (10%) and APIs (7%). Only two multi-protocol, environment variable fuzzers are available in the market today; Codenomicon Defensics® and Beyond Security beSTORM®.
We set out to see for ourselves how Beyond Security beSTORM® compares to Codenomicon Defensics®. And this is what we found:
|Fuzzing Method||Smart Fuzzing
Model based fuzzing
|Model Based fuzzing|
|Test Target||Server, Client, Applications, API, DLL||Server, Client, Applications|
|Proprietary Protocol Support||Supported via customer or vendor development
Auto Learn capabilities
|supported via vendor development only|
|Protocols Support||250 (extendable)||220 protocols|
|Test Cases||Thousands to billions per protocol
Users can also add test cases and Attack Vector type
|Attacks are pre-defined|
|Recreating the Vulnerability||Possible to recreate using Perl Script without beSTORM® (Proof-of-concept code export)||Possible to recreate using only Defensics®|
|Load Test||Supported (maximum: 50 parallel threads and 250,000 attacks per second)||not supported|
|Monitoring||network monitering, process moniter and an API for custom moniter support||network monitering only|
|Certification||Certified ISA Secure (Embedded device and Systems Software qualification) EDSA 1.0 & EDSA 2.0||Certified ISA Secure (Embedded device and Systems Software qualification) EDSA 1.0|
|Testing design||Exhaustive testing that begins with likely weaknesses but then expands out to every possible input variation||Selected, highrisk, known protocol weaknesses|
|Test range||Every field of entire protocol is tested with every possible input variation||Selected fields are tested for selected, known vulnerabilities|
|Test numbers||Millions or tens of millions of tests||thousands or tens of thousands|
|Monitoring capability||Process monitering, application ping, ICMP / ARP and custom monitoring||Process monitor|
|Testing of proprietary protocols||Yes. 'Learn' function converts the BNF description used in RFC documents into attack language||No|
|Supports native DLL calls||Yes||No|
|Full source code provided||Yes||No|
|Complete reporting on attacks||Exports proof of concept code in Perl script that re-creates the attack for easily repeating the failure||No|
|User interface||Moderate complexity, with typical training done in 1-2 days||High complexity, requiring extensive training|
|Cost of ownership||Existing software testing staff can run tests on all protocols of all products||Highly trained, expensive staff required for operation|
|Cost of purchase||Median priced||Highest of all commercial, multi-protocol fuzzers|
|Custom protocols||No additional cost option||Added cost option|
|Scalable||Use multiple processors or machines to reduce test time||No|
During our selection trials, Beyond Security beSTORM® showed higher standards of user-friendly handling and operation, fuzzing techniques supported and analytics and proved to be a superior Codenomicon Defensics® competitor. While both Beyond Security beSTORM® and Codenomicon Defensics® enabled us to reset the target application after system failure and implement the reproduction of bugs identified, only beSTORM® was able to generate a PERL script that recreates the vulnerability. Codenomicon Defensics® was definitely costlier. Although it supported a larger number of fuzzing techniques, feature utilization was difficult due its complex UI. We finally chose Beyond Security beSTORM® for our requirement as it scored higher on the following parameters:
- Supported Fuzzing techniques and protocols
- Analytical abilities
- Software ergonomics
- Costs and license
Overall we chose Beyond Security beSTORM® as it was found to be the ideal Codenomicon Defensics® alternative for customers like us who are keen on a more flexible, cost effective, scalable and easy to use 'smart' Fuzzer.