Open Source vs Commercial Load Testing Tools
One of the most important decisions when embarking on a load and performance testing project is which tool to use. Unfortunately, it isn’t an easy decision or one that can be made quickly as each tool has its own pros and cons and the wrong choice for your application can lead to untold headaches.
Within the tool discussion is the open source vs commercial tool debate. There are biased zealots in both camps that will vilify the other. However, the truth is that there are plenty of instances where either type of tool can be successful on a project.
Regardless of which tool you’re critiquing, you should compare the tools based on factors like cost, accuracy, ease of scripting, extensibility, monitoring and reporting.
Below are some of the advantages and disadvantages you should consider before taking the leap.
Open Source Tools
Some of the most popular open source load testing tools include JMeter, The Grinder, Gatling, Webload, Tsung, and more. Advantages- No initial cost involved other than learning curve and training
- Easily extensible plug-in architecture to add additional functionality
- Easy to use in a development environment to test at small loads
- Little to no support or training
- Free tools but your time is expensive and may also affect project deadlines
- Limited UI support for script development and feedback while running tests
- Overly focused on web based protocol
- Memory and CPU issues when running large loads
- Large scale tests may be difficult
Commercial Tools
There are lots of commercial load testing tools on the market including HP Loadrunner, Silkperformer, Neoload, and SOASTA CloudTest. Some of our favorite tools are cloud based and offer pay-as-you-use pricing models. Advantages- Constantly updated as technologies evolve helping you concentrate on your application and not your tool
- Relatively easy to use scripting interface
- Readily available support teams, which can help you, fix unforeseen problems
- Supports a wide range of protocols with support for proprietary software
- Supports strong real-time monitoring helping identify server side bottlenecks
- Excellent integrated analysis tools that can dynamically generate graphs
- A large number of customized graphs can be generated
- Advanced reporting with the ability to generate different formats of report
- Easy availability of experienced tool users
- High cost associated with these tools for both licensing and support
- Can only generate as much load as your license allows
- Dependent on the software vendor to implement desired features.
Objective Evaluation
At Total Performance Consulting our clients depend on us to pick the right tool for the task. Over the years we have developed a comprehensive evaluation process that starts with our customers’ requirements. The tool, whether open source or commercial, needs to meet the requirements including protocols that need to be supported, ease of scripting, need for dedicated IP address, test from the cloud, how many virtual users, prefer SaaS or host themselves. The list is quite long and they need to be ranked in order of importance. We then take multiple tools through a scoring process by actually exercising the tools and assigning scores. We take a close look at 9 distinct areas including platform, scripting, test scenario creation, test execution, analysis, reporting, support, and cost/license type. Then, we score on nearly 50 different attributes! While it may seem like a lot, when you start to dive into your load and performance testing demands, it quickly becomes apparent that you need to look at everything. If you base your decision on license cost only you are overlooking 98% evaluation process! Below is a simple example of how to evaluate a load testing tool :
