
Open Source Tools
Some of the most popular open source load testing tools include JMeter, The Grinder, Gatling, Webload, Tsung, and more. Advantages- No initial cost involved other than learning curve and training
- Easily extensible plug-in architecture to add additional functionality
- Easy to use in a development environment to test at small loads
- Little to no support or training
- Free tools but your time is expensive and may also affect project deadlines
- Limited UI support for script development and feedback while running tests
- Overly focused on web based protocol
- Memory and CPU issues when running large loads
- Large scale tests may be difficult

Commercial Tools
There are lots of commercial load testing tools on the market including HP Loadrunner, Silkperformer, Neoload, and SOASTA CloudTest. Some of our favorite tools are cloud based and offer pay-as-you-use pricing models. Advantages- Constantly updated as technologies evolve helping you concentrate on your application and not your tool
- Relatively easy to use scripting interface
- Readily available support teams, which can help you, fix unforeseen problems
- Supports a wide range of protocols with support for proprietary software
- Supports strong real-time monitoring helping identify server side bottlenecks
- Excellent integrated analysis tools that can dynamically generate graphs
- A large number of customized graphs can be generated
- Advanced reporting with the ability to generate different formats of report
- Easy availability of experienced tool users
- High cost associated with these tools for both licensing and support
- Can only generate as much load as your license allows
- Dependent on the software vendor to implement desired features.
Objective Evaluation
At Total Performance Consulting our clients depend on us to pick the right tool for the task. Over the years we have developed a comprehensive evaluation process that starts with our customers’ requirements. The tool, whether open source or commercial, needs to meet the requirements including protocols that need to be supported, ease of scripting, need for dedicated IP address, test from the cloud, how many virtual users, prefer SaaS or host themselves. The list is quite long and they need to be ranked in order of importance. We then take multiple tools through a scoring process by actually exercising the tools and assigning scores. We take a close look at 9 distinct areas including platform, scripting, test scenario creation, test execution, analysis, reporting, support, and cost/license type. Then, we score on nearly 50 different attributes! While it may seem like a lot, when you start to dive into your load and performance testing demands, it quickly becomes apparent that you need to look at everything. If you base your decision on license cost only you are overlooking 98% evaluation process! Below is a simple example of how to evaluate a load testing tool :