blog logo

High-Value Load Testing: Fast and Functional Time-to-Market

Stephen Chinn • February 7, 2017

Brilliant new features are what keep digital storefronts relevant and popular, but it doesn’t matter how brilliant the feature is if it collapses under load as excited visitors attempt to interact with it.

Infrastructure with rapid scaling capability can help avoid high-load disasters but their massive CPU power requirements can be very expensive. Furthermore, not every problem can be quickly solved by increasing infrastructure resources.

How then can we ensure new features will function under load without spending valuable resources on rapid scaling? Frequent high-value load testing.

Why Load Testing?

The value of load testing depends on how load tests and testing mechanisms are integrated into the feature delivery process. Load testing should do more than simply validate a system’s operation status at a level of user activity: It should help engineers understand how user behavior, feature code, and infrastructure resources interact in order to drive efficient and resilient features.

Load Testing
Figure 1: Infrastructure designed to support frequent load testing as part of a continuous delivery system


High-Value Load Testing

High-value load testing is a critical piece of the “Continuous Delivery” software engineering approach now widely used by eCommerce companies to drive the fastest possible time-to-market for new features.

These high-value load testing systems have most, if not all, of the following characteristics:

  1. White Box Tests Results: Produce detailed data collected from inside the system
  2. Repeatable Tests: Tests can be easily reproduced, either perfectly or with minor controlled changes
  3. Easy to Frequently Run: They are not too expensive – in either time or money – to be run often
  4. Go Beyond User Simulation: Include tests designed to stress specific components of the application architecture
  5. Inductive and Deductive Experimentation: Both help to avoid production problems and to quickly resolve any problems that do emerge in production
  6. Measure Trends Over Time: Comparing new results to historic data has added value

White Box Tests

Many services can be used to generate simulated traffic load, but most of these services can’t see inside the applications they are testing. On their own, these are called black box tests. This leaves the most of the value of the test unrealized. The results of a white box test include detailed performance data collected from all of the individual servers being stressed by the test. The lack of internal application insight leaves the most of the value of these tests unrealized. The results of a white box test include detailed performance data collected from all of the individual servers being stressed by the test.

The results of a white box test include information such as detailed performance data collected from all of the individual servers being tested. This added layer of analysis provides critical value to your feature testing.

Load Testing
Figure 2: Black Box Test vs White Box Test

Repeatable Tests

To generate truly meaningful insight, it is often necessary to repeat tests multiple times. Test result confidence requires a base understanding of whether results are typical or outliers.

Repeating tests and aggregating their results increases the reliability of the data and improves the accuracy of data-driven insights. Moreover, our ability to exactly reproduce a load test provides a foundation for further experimentation. Only when we understand all the variables that affect test can we begin to attribute causation to these variables.

Easy To Run Frequently

Far too often, load testing is only used performed sporadically – usually when management is nervous about a system’s to handle a high-traffic event. For example, a load test will be thrown together in late October to make sure an eCommerce platform can handle the anticipated Black Friday orders.

Though infrequent load testing is certainly better than none at all, the additional value of load testing is lost when it is not repeated frequently. In today’s constantly shifting digital landscape, data points need to be gathered consistently and frequently.

Hands down, the most common reason load testing is not performed frequently is cost. Sometimes financial costs will interrupt testing, but most frequently it is actually the time and effort required to perform a test that holds people back.

This obstacle is why we strongly recommend load testing be integrated into every development cycle. When load testing is a part of every sprint – an essential step in the march towards production deployment – process automation and frequent incremental tweaks mitigate the time costs associated with load testing, driving more frequent data collection and more valuable insights.

Load Testing
Figure 3: Driving frequent load testing is critical. LYONSCG often integrates the open source tool iMeter (above) into client’s continuos delivery systems

Go Beyond User Simulation

The aggregated behavior of many users invariably creates a very complicated pattern of interactions. This complexity obfuscates the cause and effect relationships within a system, making decisions in regards to infrastructure requirements difficult to make.

In order to understand the system, tests must be purposely constructed to be unlike any real user traffic we may experience. These load tests discover relationships which can be then monitored over time. They help gauge the available capacity of a system and provide insight into how best to scale the system when load rises.

Don’t get me wrong: attempts to simulate actual user behavior are important and useful for validating the capacity of the system, but these tests alone lack the degree of insight we need to successfully manage a system over time.

Load Testing
Figure 4: Targeted Process Testing vs Actual User Traffic Testing

Inductive and Deductive Experimentation

If you haven’t guessed by now, I am a strong advocate a scientific approach to load testing. Load testing results should help our teams accurately understand the systems they are supporting, and not just how to react when traffic spikes.

Architects must carefully consider performance under load when they design new features. Designs should be treated as hypotheses: requiring ample testing before being used in a production environment. This mindset injects critical deductive reasoning into the design process.

Load Testing

On the other side, it is also critical to engage in inductive reasoning. When performance stress appears in a production environment, our first move should be reproducing said stress pattern in a controlled testing environment.

For this, the load testing environment must reasonably reflect production. A strong testing environment follows the pattern of the production environment it supports. Server roles need to be organized facsimile to production in order for performance data points to align.

Measure Trends Over Time

When used as part of a strong continuous delivery process, load testing delivers a historic performance impact record against change. These trends greatly improve a DevOps team’s ability to easily provide accurate predictions when more changes are proposed. These predictive advantages enhance the accuracy of cost/benefit analysis, facilitates rapid development of productive new features, and enhances operational efficiency. 

High-Value Load Testing –  A High-Value Strategy

Getting new features quickly in-market is critical for companies competing in the rapidly changing eCommerce marketplace; however, the added value of speedy development is wasted if features perform poorly under traffic.

Successful DevOps teams rely on a continuous delivery model: integrating frequent load testing into normal sprint cycles and implementing systems designed to be readily available tools able to leverage scientific methods to quickly solve problems and accurately drive successful feature deployment.

Stephen Chinn

About the author

Stephen Chinn

Subscribe to our blog

Let's discuss the next step in your commerce journey.

XSchedule a meeting