Load Testing:
HTTP vs Headless vs Real Browser
Slow loading or non-responsive web pages have an impact on financial revenue because frustrated users will most likely not return once the issue has been solved. Therefore, performance testing has become a fundamental part of the development chain and the demand is still growing.
Performance testing platforms provide a broad range of load simulation methods such as HTTP, headless, and real browser-based. In this paper, we will outline the main aspects of those followed by a comparison matrix, which you can use for choosing an appropriate simulation approach.
In the early days of the digital age, HTTP-based testing was very popular. With the rise of rich web client technology the HTTP-based simulation approaches have become increasingly outdated. A typical HTTP-based test driver executes service requests and parses responses. Modern web 2.0 applications consist of many client-side scripts, which are totally ignored, and not measured in this type of test execution. In worst-case scenarios, complex use cases cannot be simulated on protocol level due to a shortage of client side generated ids.
Due to their request and response-based nature, the overhead on the load injection machine is very low. A typical load test server can simulate up to 800 simultaneous sessions. Complex protocol based use cases can be difficult to implement. A performance engineer needs to deal with cookies, session IDs, and other dynamic parameters. Depending on the type of system being tested, some web form names often change once a new version has been deployed which will cause the HTTP-based script to fail.
// sample protocol level script
transaction TMain
var
hContext: number;
begin
WebPageUrl(“http://lab3/st/”, “Greetings”);
WebPageStoreContext(hContext);
WebPageLink(“Join the experience!”, ” – New Visitor”);
WebPageSubmit(“Continue”, CONTINUE001, “Main menu”);
WebPageLink(“Products”, “ShopIt – Products”);
WebPageLink(NULL, “ShopIt – Product”, 3);
WebPageSubmit(“Search”, SEARCH001, ” – Search”, 0, NULL, hContext);
end TMain;
dclform
CONTINUE001:
“name” := “jack”,
“New-Name-Button” := “Continue”;
SEARCH001:
“search” := “boot”;
After all, protocol level scripts are good for component level tests in continuous integration environments and the perfect setting for stress testing due to their low footprint on load injection machines.
With the rise of web 2.0 technologies, the testing business was faced with serious challenges. Rich browser applications could no longer be tested or simulated on the protocol level due to the missing client side logic during script replay. Therefore, several headless browsers have been introduced such as HtmlUnit, PhantomJS or SlimerJS. They are often built on top of WebKit, the engine behind Chrome and Safari.
Headless Browsers are similar to real browsers without the GUI. Many test-automation and performance testing platforms are using headless browsers to simulate traffic.
Headless browsers have their own pitfalls; as new browsers enter markets headless browser kits need to catch up to all the performance and feature enhancements of the real browsers.
The simulation of client side rendering of is not for free. A typical load injection server can simulate up to 10-12 simultaneous headless browser sessions, versus 500 of HTTP based sessions.
Test script implementation and customization is not too difficult. If you have basic coding skills you will be able to create simple scripts. Not all headless-browsers provide visual replay features and without visual replay script debugging or error analysis could become very tricky.
// sample phantomjs script
“use strict”;
var page = require(‘webpage’).create(),
server = ‘http://posttestserver.com/post.php?dump’,
data = ‘universe=expanding&answer=42’;
page.open(server, ‘post’, data, function (status) {
if (status !== ‘success’) {
console.log(‘Unable to post!’);
} else {
console.log(page.content);
}
phantom.exit();
});
In the sample script seen here a simple post request is executed. You need Java programming skills to customize such test scripts.
Web2.0 applications are full of JavaScript, Flash, Ajax and CSS. Without a full browser, it’s not possible to measure the actual end-to-end response times of the whole web page. Real browser-based performance testing allows you to verify the site’s functionality and speed as perceived by the end-user.
A typical real browser performance test solution collects loading times of images, JavaScript, CSS and more. Often they provide waterfall charts, which visualize the load time of those components.
The footprint of a real browser based browser is slightly higher. Considering the fact that headless browser simulation does not deliver 100% realistic response times it’s fair to say that real browser based simulation should be preferred. In real life scenarios, headless browsers perform only 20% better than real browsers. So, if average-sized single load injector can run 10-12 headless browser sessions, the same load injector can run 8-10 real browser sessions.
Implementation and maintenance of test scripts is easy because user actions are directly reflected and the visual replay makes debugging easy. In the sample script below the browser opens a URL, inserts user and password and clicks the login button.
// sample real browser-based script
transaction TMain
begin
BrowserStart(BROWSER_MODE_DEFAULT, 800, 600);
// navigate to the login site
BrowserNavigate(“http://demo.com/TestSite/LoginForm.html”);
// set the authentication for the secure site
BrowserSetText(“//INPUT[@name=’user’]”, “User1”);
BrowserSetPassword(“//INPUT[@name=’pwd’]”, “Password1”);
// Login
BrowserClick(“//INPUT[@value=’Login’]”, BUTTON_Left);
end TMain;
After all, real browser simulation is useful for realistic end-to-end load tests, but it may get expensive for stress testing high user volumes, because the footprint on the load injection server is too high.
Performance Test Types
Component Speed Tests
In recent years software development methods have moved in the agile direction. Short release sprints are essential. Developers and test engineers automate their quality assurance and performance checks. Typically, they implement service based performance tests on protocol level or they simulate real browser based performance checks to compare end-to-end response times with agreed performance boundaries.
Objectives
+ Repeatability
+ Automated interface and end-to-end performance checks
+ Compare response times with agreed thresholds
Load Tests
For several reasons load tests are the ideal testing method when it comes to verification of non-functional requirements. One being that the response times can be verified under reproducible conditions. Another being that those tests allows verification of response time thresholds. Realistic response time measurement is essential in load test scenarios. Therefore, test engineers use headless or real browser based user simulation for their load test settings.
Objectives
+ Reproducible load simulation
+ Verification of response time thresholds
+ Identify bottlenecks under production like load conditions
+ Realistic end-to-end test scenarios
Stress Test
If you have to test the reliability of your application under peak load conditions, run a stress test. In this type of test you specify mainly the max number of users and the time over which the ramp up and the steady state load should be on your application. The goal is to identify the breaking points of your application under test.
Objectives
+ Proof scalability and stability
+ Simulate peak load conditions
+ Exact reproducibility is not important
Comparison
Obviously, there are good reasons for protocol, headless or real browser based user simulation. The matrix below provides some guidance to choose the appropriate approach.
Criteria
HTTP
Headless Browser
Real Browser
User simulation
No client side rendering
Some client-side elements are simulated
Real user simulation
Script implementation and customization
Difficult when web sites are complex
Developer skills required to build robust scripts
Simple scripts, easy to customize
Script replay
Low level analysis required
Depending upon the engine used, visual replay is possible
You see what you get
Script maintainability
Programming skills required
Errors in un-rendered sections are tricky to solve
Easy because you see failures during replay
Multi Browser Support
Some tools emulate web browsers, but this is not comparable
Yes, but some elements are often missing
Some support other versions and different browsers
Footprint on load injection machine
Low, up to 800 sessions per server
Medium, up to 10-12 sessions per server
High, up to 6 sessions per server
Recommended for DevOps
Depends on actual test scenario
No, often complex scripts
Yes, easy to use and realistic figures
Recommended for Load Tests
No, client-side processing is skipped
Yes, better than HTTP simulation
Yes, realistic user simulation
Recommended for Stress Tests
Yes, because there is a low overhead on the load generator machine
No, overhead on the load generator machine is too high
No, highest overhead on load generator machine
Costs
Low
High
High
Recommended for high volume, low cost tests webserver stress tests (where possible)
Not Recommended
Recommended for real-life complex application simulations.
Stress Test
If you have to test the reliability of your application under peak load conditions, run a stress test. In this type of test you specify mainly the max number of users and the time over which the ramp up and the steady state load should be on your application. The goal is to identify the breaking points of your application under test.
Objectives
+ Proof scalability and stability
+ Simulate peak load conditions
+ Exact reproducibility is not important
Comparison
Obviously, there are good reasons for protocol, headless or real browser based user simulation. The matrix below provides some guidance to choose the appropriate approach.
| Criteria | HTTP | Headless Browser | Real Browser |
| User simulation | No client side rendering | Some client side elements are simulated | Real user simulation |
| Script implementation and customization | Difficult when web sites are complex | Developer skills required to build robust scripts | Simple scripts, easy to customize |
| Script replay | Low level analysis required | Depending on used engine is visual replay possible | You see what you get |
| Script maintainability | Programming skills required | Errors in not rendered sections are tricky to solve | Easy because you see failures during replay |
| Multi Browser Support | Some tools emulate web browser but this is not comparable | Yes, but some elements are often missing | Some support other versions and different browsers |
| Footprint on load injection machine | Low, up to 800 sessions per server | Medium, up to 10-12 sessions per server | High, up to 6 sessions per server |
| Recommended for DevOps | Depends on actual test scenario | No, often complex scripts | Yes, easy to use and realistic figures |
| Recommended for Load Tests | No, client side processing is skipped out | Yes, better than HTTP simulation | Yes, realistic user simulation |
| Recommended for Stress Tests | Yes, because there is a low overhead on load generator machine | No, overhead on load generator machine is too high | No, highest overhead on load generator machine |
| Costs | Low | High | High |
| Recommended for high volume, low cost tests webserver stress tests (where possible) | Not Recommended | Recommended for real-life complex application simulations.
|
- https://developers.google.com/web/tools/chrome-devtools/device-mode/testing-other-browsers
- https://watirmelon.blog/2015/12/08/real-vs-headless-browsers-for-automated-acceptance-tests/
- http://news.softpedia.com/news/what-is-a-headless-browser-and-what-s-it-good-for-485162.shtml
- https://github.com/dhamaniasad/HeadlessBrowsers
- https://circleci.com/
