The Hybrid Performance Test Methodology

  Does your application require a Hybrid performance approach? A hybrid load test consists of emulating virtual users via a network protocol and real browsers. At TPC, we can build this load test approach, which delivers actionable insights on both the application’s overall scalability as well as the real end user experience. Traditional load testing of web applications proves invaluable in determining the scalability and capacity of applications. This type of traditional load testing uses virtual users driven by network protocols such as Http/s, Websocket, Flex, RTMP, HLS, GWT, etc. Protocol based performance testing emulates a realistic production load via active sessions and transaction throughput rates.  Monitoring the deployment during protocol load tests identifies infrastructure resource bottlenecks which limit the overall scalability of an application. However, more clients are requiring the “end user” experience integrated into their performance test approach.  Enter the Hybrid load test: Performance and load testing using protocols to drive virtual user traffic while launching real browsers to capture page load and elements rendering response times. The hybrid load test is typically configured with over 90% protocol and under 10% real browsers.

Protocol and Browser load test go hand-in-hand.

The integration of both of these protocol and browser results, into a single viewing pane, skyrockets the value of a hybrid load test. For example, analysis of the protocol load test results shows a specific page as having an average of .8 seconds response time. That’s the time to execute the backend business logic and download all the embedded resources which comprise that page. However, the browser load test result shows an average response time of 3.2 seconds for that same page. This means that the backend is able the service that page request in .8 seconds but the browser time came in as 3.2. Below is an example waterfall chart that demonstrates the backend vs frontend split. Below you see it takes approximately .25 seconds for the backend server to get the first bite of data to the client (frontend).  Frontend is everything else including network time for downloading all the resources referenced in the page. The reason this is included in the frontend time is because frontend developers can do a lot to reduce frontend time including using a CDN, asynchronous script loading and concatenating stylesheets and scripts. Developers have made pages “heavier” by adding code logic which executes solely on the browser side. This reduces the overall network traffic. There are often contingencies and “waiting” for certain conditions. Other browser behaviors are the number of parallel requests that can be executed and the real caching policies as related to the specific user session.  To further identify where the time is being spent, you can add timers around each element of the page. For example, a timer can be placed around a javascript which executes purely on the browser without any backend communication. Isolating the timing of this javascript vs the entire loading of a page will show you how much time is being spent on this code execution. Another example is a banner which is a call to a third party site, timers will reveal how much time is being spent rendering this banner compared to the rest of the page. The challenge is that browser load testing requires significantly more load generation resources (CPU, RAM) than the protocol tests. Typically, browser level load testing can’t scale more than a few users before the load generator machine becomes saturated. The solution is hybrid tests where the majority of the load is protocol and the minority is browser based. There are many tools which address this challenge: LoadRunner’s TruClient, NeoLoad/Selenium integration, Blazemeter, OctoPerf, etc. A performance engineer will profile the page to identify the components which require timers. Many companies already use Selenium for QA functional test cases and some slick commercial load tools are now taking Selenium scripts and converting them to protocol scripts. In summary, protocol based load testing is used to understand the scalability and capacity of the infrastructure (load balancers, web servers, app servers, messaging systems, database servers, etc). Without the deployment being able to scale to the required capacity, there is absolutely no chance that the browser response times will be acceptable. Browser based load testing gives a more accurate insight into the end user experience and will also lead performance engineers to understand the needed changes on the browser side of rendering to meet the business SLA’s. Contact TPC to build your Hybrid Performance Test Today!  

Together