When looking at a load test, many people choose to focus on the specific elements involved in that specific test, including the number of threads that should be used, response times and hits. This post, however, looks at the process itself - hopefully giving you a better chance for an effective and successful load test event.
If you often run load tests, you probably have a mental checklist of questions that run through your mind, including:
- How many threads per JMeter engine should I use?
- Can my engine handle 10% more threads?
- Should I start with the maximum load, or add as I go?
All of these questions are important and should be carefully considered - but what about the load test itself? Have you considered how the actual procedure will be managed?
It’s easy to get so wrapped up in the technical details that you lose sight of the overall picture. Even with BlazeMeter’s self-service load testing solution - which is designed to simplify performance and load testing for developers and QA testers - it’s still vital to take control of the process management to ensure each test runs smoothly.
Load tests require a great deal of cooperation and teamwork. Everyone needs to be in sync in order to quickly analyze the data, identify the bottlenecks and even solve them (if possible) during the test.
For example: the developers/QA testers should create and share the script with all departments involved in the test, to make sure that everyone understands the scenario and how it can affect their particular system. An n-tier application is made up of several disciplines, each one requiring the same level of attention - be it database, front-end, back-end, or monitoring services. It is imperative that the staff understand the upcoming load test scenario, regardless of their knowledge of JMeter and/or load testing, so as to create the most agile environment for the duration of the test. If the staff clearly understand what is happening when a bottleneck is hit, it’s much easier for them to analyze the data and quickly tend to the issue. If they aren’t on par, there’s a much greater chance that they won’t be able to overcome the bottlenecks and delays will occur.
Once the ins and outs of the infrastructure are understood, further managerial tasks are required. For example: sending out updates and involving your operations team and third parties like Akamai (News - Alert) or Amazon to avoid complications such as blocking, shaping, or even inadvertently breaking a legal agreement during the test. While they may seem excessive, it’s really worth taking these supplementary steps before starting the load test to ensure the best results.
The Technical Aspects of a Load Test
When it comes to the technical aspects of load testing, as detailed in How to Run a Load Test of 50k+ Concurrent Users, it’s important to supply all involved parties with a test scenario outlining the exact processes of the test. By taking this step, even staff members who are not JMeter experts will understand that during the test, they may experience issues like an excessive number of log-ins, 'add to cart' requests, image usages, database queries, and so on. The scenario should be simplified and easy to read for everyone involved. It should include a defined scaling process and explain that the ultimate goal for the number of users may only be met within a number of load tests.
The technical approach usually includes building the script (and its data) to address the specific scenario, testing it locally and verifying that it actually works in JMeter. Once the script performs as expected using the SandBox feature, you should evaluate how many threads can be applied to a single load engine without crashing (ensuring the load engines themselves won’t be the bottleneck). Then, the script will be scaled to a cluster of engines, and more clusters can be added as you go.
Important points to consider:
Does your data need to be unique per engine?
If so:
Does your scenario have data manipulation in real time?
If so:
Do you use timers?
If not, you should (even if it’s just 300ms) because every user has some think time between pages. You can use the Throughput Shaping Timer or the Constant Throughput Timer to control your test hits/s.
The Managerial Aspects of a Load Test
From a management perspective, of course, every company is unique. But most share the same key criteria, and should take these similar steps to ensure a load test goes smoothly:
Schedule a meeting before the actual test
Two weeks is generally enough time for this. It allows everyone involved to see the expected script scenario and everyone can discuss issues like who to alert about the test and possible interferences.
Include a representative from each department during the initial load tests
This will ensure smoother sailing once the test is up and running. In later tests, you might want to schedule the actual environment you’re going to load test. For example: in a staging environment, the VP of R&D needs to know that the environment is going to be used, which may affect updates to production. Whereas, if load tests are run on production environments, a landing page should be created to notify users that maintenance procedures will take place during that particular time.
Create a drawing or flow chart of the actual script
This is an additional measure, but a useful one - after all, a single picture says a thousand words. It helps people understand what’s expected to happen, even those who aren’t familiar with JMeter or aware of what’s happening with particular clients or the front-end.
Depending on the test’s complexity, you can make a flowchart of your script flow, cache warmup (after doing a reset cache), a test scenario, the flow of the load (for example: 2-hour test, ramping up to 50K within 30 minutes, then continually adding), or the event success criteria (for example: reaching 50K users/80K RPS).
This method effectively divides the load test into two parts: the pre-load test, which creates all of the data, and the actual load test itself. Appointing a load test supervisor to announce the various stages of the test will help everyone involved to stay on the same page.
Find a place to share information in real time
Utilizing an organized platform, such as Campfire or HipChat will allow everyone to communicate quickly during the load test, as well as enable the supervisor to control and deliver critical assignments when needed. Additionally, these platforms provide a space for each department to present their conclusions from the test. You can also record the test - which will be a great help when it comes to running future tests and reports.
Schedule the right people for your load test
Make sure you have every discipline you need during the load test including your 3rd parties! You don’t want to start the load test and discover that suddenly your 3rd CDN provider is throttling your requests (or even blocks them due to DDoS policies). Some key people you should consider to have with you:
Taking all of the above into consideration while you plan your main event will allow you to get everyone on board with the process and understand its importance. You’ll probably discover some critical steps you missed to make a load event more efficient, and you’ll be able to react quickly you saturated your application and perform real time analysis to pinpoint that bottleneck. And eventually, you’ll have a better chance for an effective and successful load test event
About the Author
With over ten years as project manager and systems developer in the Israeli Defense Forces, Refael's invaluable experience brings a deep proficiency to BlazeMeter as the devop team leader. As a specialist in Enterprise Infrastructure Systems, Refael managed the establishment of the decentralized operating systems team, a department which managed all Op System solutions in the IDF both in relation to server technologies and the end user.