One of the most important decisions when embarking on a load and performance testing project is which tool to use. Unfortunately, it isn’t an easy decision or one that can be made quickly as each tool has its own pros and cons and the wrong choice for your application can lead to untold headaches.
Within the tool discussion is the open source vs commercial tool debate. There are biased zealots in both camps that will vilify the other. However, the truth is that there are plenty of instances where either type of tool can be successful on a project.
Regardless of which tool you’re critiquing, you should compare the tools based on factors like cost, accuracy, ease of scripting, extensibility, monitoring and reporting.
Below are some of the advantages and disadvantages you should consider before taking the leap.
Open Source Tools
Some of the most popular open source load testing tools include JMeter, The Grinder, Gatling, Webload, Tsung, and more.
- No initial cost involved other than learning curve and training
- Easily extensible plug-in architecture to add additional functionality
- Easy to use in a development environment to test at small loads
- Little to no support or training
- Free tools but your time is expensive and may also affect project deadlines
- Limited UI support for script development and feedback while running tests
- Overly focused on web based protocol
- Memory and CPU issues when running large loads
- Large scale tests may be difficult
There are lots of commercial load testing tools on the market including HP Loadrunner, Silkperformer, Neoload, and SOASTA CloudTest. Some of our favorite tools are cloud based and offer pay-as-you-use pricing models.
- Constantly updated as technologies evolve helping you concentrate on your application and not your tool
- Relatively easy to use scripting interface
- Readily available support teams, which can help you, fix unforeseen problems
- Supports a wide range of protocols with support for proprietary software
- Supports strong real-time monitoring helping identify server side bottlenecks
- Excellent integrated analysis tools that can dynamically generate graphs
- A large number of customized graphs can be generated
- Advanced reporting with the ability to generate different formats of report
- Easy availability of experienced tool users
- High cost associated with these tools for both licensing and support
- Can only generate as much load as your license allows
- Dependent on the software vendor to implement desired features.
At Total Performance Consulting our clients depend on us to pick the right tool for the task. Over the years we have developed a comprehensive evaluation process that starts with our customers’ requirements. The tool, whether open source or commercial, needs to meet the requirements including protocols that need to be supported, ease of scripting, need for dedicated IP address, test from the cloud, how many virtual users, prefer SaaS or host themselves. The list is quite long and they need to be ranked in order of importance.
We then take multiple tools through a scoring process by actually exercising the tools and assigning scores.
We take a close look at 9 distinct areas including platform, scripting, test scenario creation, test execution, analysis, reporting, support, and cost/license type. Then, we score on nearly 50 different attributes! While it may seem like a lot, when you start to dive into your load and performance testing demands, it quickly becomes apparent that you need to look at everything. If you base your decision on license cost only you are overlooking 98% evaluation process!
Below is a simple example of how to evaluate a load testing tool :
When you get right down to it, making the decision between open source or commercial load testing tools mainly depends on the type of application you are testing and the level of complexity involved. Do not take shortcuts during the evaluation process as your ultimate choice has to work with YOUR application and provide you with reporting that meets YOUR needs.
Don’t fall in love with the price of open source tools. Nothing is free. We all know that. Understand the true costs and limitations of open source tools – do not believe the marketing hype of a sales deck.
Finally, put the tools through their paces or hire experts who know how to do this for you to reinforce your decision making process.
We would love to know what you look for when you evaluate a tool. Let us know your thoughts or contact us and we can help you with your tool evaluation needs.