infoTECH Feature

October 09, 2015

Preparing for a Load Test: The Vital Points You May Be Overlooking

By Special Guest
Refael Botbol, Director of Professional Services

When looking at a load test, many people choose to focus on the specific elements involved in that specific test, including the number of threads that should be used, response times and hits. This post, however, looks at the process itself - hopefully giving you a better chance for an effective and successful load test event.

If you often run load tests, you probably have a mental checklist of questions that run through your mind, including:

- How many threads per JMeter engine should I use?
- Can my engine handle 10% more threads?
- Should I start with the maximum load, or add as I go?

All of these questions are important and should be carefully considered - but what about the load test itself? Have you considered how the actual procedure will be managed?

It’s easy to get so wrapped up in the technical details that you lose sight of the overall picture. Even with BlazeMeter’s self-service load testing solution - which is designed to simplify performance and load testing for developers and QA testers - it’s still vital to take control of the process management to ensure each test runs smoothly. 

Load tests require a great deal of cooperation and teamwork. Everyone needs to be in sync in order to quickly analyze the data, identify the bottlenecks and even solve them (if possible) during the test.

For example: the developers/QA testers should create and share the script with all departments involved in the test, to make sure that everyone understands the scenario and how it can affect their particular system. An n-tier application is made up of several disciplines, each one requiring the same level of attention - be it database, front-end, back-end, or monitoring services. It is imperative that the staff understand the upcoming load test scenario, regardless of their knowledge of JMeter and/or load testing, so as to create the most agile environment for the duration of the test. If the staff clearly understand what is happening when a bottleneck is hit, it’s much easier for them to analyze the data and quickly tend to the issue. If they aren’t on par, there’s a much greater chance that they won’t be able to overcome the bottlenecks and delays will occur.

Once the ins and outs of the infrastructure are understood, further managerial tasks are required. For example: sending out updates and involving your operations team and third parties like Akamai (News - Alert) or Amazon to avoid complications such as blocking, shaping, or even inadvertently breaking a legal agreement during the test. While they may seem excessive, it’s really worth taking these supplementary steps before starting the load test to ensure the best results. 

The Technical Aspects of a Load Test
 

When it comes to the technical aspects of load testing, as detailed in How to Run a Load Test of 50k+ Concurrent Users, it’s important to supply all involved parties with a test scenario outlining the exact processes of the test. By taking this step, even staff members who are not JMeter experts will understand that during the test, they may experience issues like an excessive number of log-ins, 'add to cart' requests, image usages, database queries, and so on. The scenario should be simplified and easy to read for everyone involved. It should include a defined scaling process and explain that the ultimate goal for the number of users may only be met within a number of load tests.

The technical approach usually includes building the script (and its data) to address the specific scenario, testing it locally and verifying that it actually works in JMeter. Once the script performs as expected using the SandBox feature, you should evaluate how many threads can be applied to a single load engine without crashing (ensuring the load engines themselves won’t be the bottleneck). Then, the script will be scaled to a cluster of engines, and more clusters can be added as you go.


Important points to consider:

Does your data need to be unique per engine?

If so:

  • Set up a different CSV for each engine
  • Use the “Split CSV file” feature to unique files and distribute among the load servers”
  • Use JMeter functions like __InstanceID, __RandomString(), __threadNum() and __UUID() to add unique data to your test (e.g: email registration)

Does your scenario have data manipulation in real time?

If so:

Do you use timers?

If not, you should (even if it’s just 300ms) because every user has some think time between pages. You can use the Throughput Shaping Timer or the Constant Throughput Timer to control your test hits/s.

The Managerial Aspects of a Load Test 

From a management perspective, of course, every company is unique. But most share the same key criteria, and should take these similar steps to ensure a load test goes smoothly:

Schedule a meeting before the actual test

Two weeks is generally enough time for this. It allows everyone involved to see the expected script scenario and everyone can discuss issues like who to alert about the test and possible interferences.

Include a representative from each department during the initial load tests

This will ensure smoother sailing once the test is up and running. In later tests, you might want to schedule the actual environment you’re going to load test. For example: in a staging environment, the VP of R&D needs to know that the environment is going to be used, which may affect updates to production. Whereas, if load tests are run on production environments, a landing page should be created to notify users that maintenance procedures will take place during that particular time.

Create a drawing or flow chart of the actual script

This is an additional measure, but a useful one - after all, a single picture says a thousand words. It helps people understand what’s expected to happen, even those who aren’t familiar with JMeter or aware of what’s happening with particular clients or the front-end.

Depending on the test’s complexity, you can make a flowchart of your script flow, cache warmup (after doing a reset cache), a test scenario, the flow of the load (for example: 2-hour test, ramping up to 50K within 30 minutes, then continually adding), or the event success criteria (for example: reaching 50K users/80K RPS). 

This method effectively divides the load test into two parts: the pre-load test, which creates all of the data, and the actual load test itself. Appointing a load test supervisor to announce the various stages of the test will help everyone involved to stay on the same page.

Find a place to share information in real time

Utilizing an organized platform, such as Campfire or HipChat will allow everyone to communicate quickly during the load test, as well as enable the supervisor to control and deliver critical assignments when needed. Additionally, these platforms provide a space for each department to present their conclusions from the test. You can also record the test - which will be a great help when it comes to running future tests and reports. 

Schedule the right people for your load test

Make sure you have every discipline you need during the load test including your 3rd parties! You don’t want to start the load test and discover that suddenly your 3rd CDN provider is throttling your requests (or even blocks them due to DDoS policies).  Some key people you should consider to have with you:

  • DevOp engineer (who will be responsible for the APM (News - Alert) data that will help you analyze your application during the load)
  • Security engineer (to make sure the load is not being blocked)
  • A product manager / senior manager - to take management actions if needed
  • Any of your 3rd parties which will be affected by the load test


Taking all of the above into consideration while you plan your main event will allow you to get everyone on board with the process and understand its importance. You’ll probably discover some critical steps you missed to make a load event more efficient, and you’ll be able to react quickly you saturated your application and perform real time analysis to pinpoint that bottleneck. And eventually, you’ll have a better chance for an effective and successful load test event

About the Author

With over ten years as project manager and systems developer in the Israeli Defense Forces, Refael's invaluable experience brings a deep proficiency to BlazeMeter as the devop team leader. As a specialist in Enterprise Infrastructure Systems, Refael managed the establishment of the decentralized operating systems team, a department which managed all Op System solutions in the IDF both in relation to server technologies and the end user.




Edited by Kyle Piscioniere
FOLLOW US

Subscribe to InfoTECH Spotlight eNews

InfoTECH Spotlight eNews delivers the latest news impacting technology in the IT industry each week. Sign up to receive FREE breaking news today!
FREE eNewsletter

infoTECH Whitepapers