Open Source vs Commercial Load Testing Tools
Published by cyang on
One of the most important decisions when embarking on a load and performance testing project is which tool to use. Unfortunately, it isn’t an easy decision or one that can be made quickly as each tool has its own pros and cons and the wrong choice for your application can lead to untold headaches.
Within the tool discussion is the open source vs commercial tool debate. There are biased zealots in both camps that will vilify the other. However, the truth is that there are plenty of instances where either type of tool can be successful on a project.
Regardless of which tool you’re critiquing, you should compare the tools based on factors like cost, accuracy, ease of scripting, extensibility, monitoring and reporting.
Below are some of the advantages and disadvantages you should consider before taking the leap.
Open Source Tools
Some of the most popular open source load testing tools include JMeter, The Grinder, Gatling, Webload, Tsung, and more. Advantages- No initial cost involved other than learning curve and training
- Easily extensible plug-in architecture to add additional functionality
- Easy to use in a development environment to test at small loads
- Little to no support or training
- Free tools but your time is expensive and may also affect project deadlines
- Limited UI support for script development and feedback while running tests
- Overly focused on web based protocol
- Memory and CPU issues when running large loads
- Large scale tests may be difficult
Commercial Tools
There are lots of commercial load testing tools on the market including HP Loadrunner, Silkperformer, Neoload, and SOASTA CloudTest. Some of our favorite tools are cloud based and offer pay-as-you-use pricing models. Advantages- Constantly updated as technologies evolve helping you concentrate on your application and not your tool
- Relatively easy to use scripting interface
- Readily available support teams, which can help you, fix unforeseen problems
- Supports a wide range of protocols with support for proprietary software
- Supports strong real-time monitoring helping identify server side bottlenecks
- Excellent integrated analysis tools that can dynamically generate graphs
- A large number of customized graphs can be generated
- Advanced reporting with the ability to generate different formats of report
- Easy availability of experienced tool users
- High cost associated with these tools for both licensing and support
- Can only generate as much load as your license allows
- Dependent on the software vendor to implement desired features.