High-Value Load Testing: Accelerating Time-to-Market
Brilliant new features are what keep digital storefronts relevant and popular, but it doesn’t matter how brilliant the feature is if it collapses under load as excited visitors attempt to interact with it.
Infrastructure with rapid scaling capability can help avoid high-load disasters but their massive CPU power requirements can be very expensive. Furthermore, not every problem can be quickly solved by increasing infrastructure resources.
How then can we ensure new features will function under load without spending valuable resources on rapid scaling? Frequent high-value load testing.
Why Load Testing?
The value of load testing depends on how load tests and testing mechanisms are integrated into the feature delivery process. Load testing should do more than simply validate a system’s operation status at a level of user activity: It should help engineers understand how user behavior, feature code, and infrastructure resources interact in order to drive efficient and resilient features.
High-Value Load Testing
High-value load testing is a critical piece of the “Continuous Delivery” software engineering approach now widely used by eCommerce companies to drive the fastest possible time-to-market for new features.
These high-value load testing systems have most, if not all, of the following characteristics:
- White Box Tests Results: Produce detailed data collected from inside the system
- Repeatable Tests: Tests can be easily reproduced, either perfectly or with minor controlled changes
- Easy to Frequently Run: They are not too expensive – in either time or money – to be run often
- Go Beyond User Simulation: Include tests designed to stress specific components of the application architecture
- Inductive and Deductive Experimentation: Both help to avoid production problems and to quickly resolve any problems that do emerge in production
- Measure Trends Over Time: Comparing new results to historical data has added value
White Box Tests
Many services can be used to generate simulated traffic load, but most of these services can’t see inside the applications they are testing. On their own, these are called black box tests. This leaves most of the value of the test unrealized. The results of a white box test include detailed performance data collected from all of the individual servers being stressed by the test. The lack of internal application insight leaves most of the value of these tests unrealized. The results of a white box test include detailed performance data collected from all of the individual servers being stressed by the test.
The results of a white box test include information such as detailed performance data collected from all of the individual servers being tested. This added layer of analysis provides critical value to your feature testing.
To generate truly meaningful insight, it is often necessary to repeat tests multiple times. Test result confidence requires a base understanding of whether results are typical or outliers.
Repeating tests and aggregating their results increases the reliability of the data and improves the accuracy of data-driven insights. Moreover, our ability to exactly reproduce a load test provides a foundation for further experimentation. Only when we understand all the variables that affect test can we begin to attribute causation to these variables.
Easy To Run Frequently
Far too often, load testing is only used performed sporadically – usually when management is nervous about a system’s to handle a high-traffic event. For example, a load test will be thrown together in late October to make sure an eCommerce platform can handle the anticipated Black Friday orders.
Though infrequent load testing is certainly better than none at all, the additional value of load testing is lost when it is not repeated frequently. In today’s constantly shifting digital landscape, data points need to be gathered consistently and frequently.
Hands down, the most common reason load testing is not performed frequently is cost. Sometimes financial costs will interrupt testing, but most frequently it is actually the time and effort required to perform a test that holds people back.
This obstacle is why we strongly recommend load testing be integrated into every development cycle. When load testing is a part of every sprint – an essential step in the march towards production deployment – process automation and frequent incremental tweaks mitigate the time costs associated with load testing, driving more frequent data collection and more valuable insights.
Go Beyond User Simulation
The aggregated behavior of many users invariably creates a very complicated pattern of interactions. This complexity obfuscates the cause and effect relationships within a system, making decisions in regards to infrastructure requirements difficult to make.
In order to understand the system, tests must be purposely constructed to be unlike any real user traffic we may experience. These load tests discover relationships which can be then monitored over time. They help gauge the available capacity of a system and provide insight into how best to scale the system when load rises.
Don’t get me wrong: attempts to simulate actual user behavior are important and useful for validating the capacity of the system, but these tests alone lack the degree of insight we need to successfully manage a system over time.
Inductive and Deductive Experimentation
If you haven’t guessed by now, I am a strong advocate for a scientific approach to load testing. Load testing results should help our teams accurately understand the systems they are supporting, and not just how to react when traffic spikes.
Architects must carefully consider performance under load when they design new features. Designs should be treated as hypotheses: requiring ample testing before being used in a production environment. This mindset injects critical deductive reasoning into the design process.
On the other side, it is also critical to engage in inductive reasoning. When performance stress appears in a production environment, our first move should be reproducing said stress pattern in a controlled testing environment.
For this, the load testing environment must reasonably reflect production. A strong testing environment follows the pattern of the production environment it supports. Server roles need to be organized facsimile to production in order for performance data points to align.
Measure Trends Over Time
When used as part of a strong continuous delivery process, load testing delivers a historic performance impact record against change. These trends greatly improve a DevOps team’s ability to easily provide accurate predictions when more changes are proposed. These predictive advantages enhance the accuracy of cost/benefit analysis, facilitates rapid development of productive new features, and enhances operational efficiency.
High-Value Load Testing – A High-Value Strategy
Getting new features quickly in-market is critical for companies competing in the rapidly changing eCommerce marketplace; however, the added value of speedy development is wasted if features perform poorly under traffic.
Successful DevOps teams rely on a continuous delivery model: integrating frequent load testing into normal sprint cycles and implementing systems designed to be readily available tools able to leverage scientific methods to quickly solve problems and accurately drive successful feature deployment.