Every SAP system is business-critical when it runs a company’s core business processes. Slow response times are frustrating, but the risk with load-related problems is that a system that is functionally perfect can unexpectedly grind to a halt on heavy-usage days. While every SAP system has its own volume, throughput, and performance-related requirements, there are common types of load and performance tests that should be run to minimise this risk for most projects.
Front-end analysis
Manually stepping through each in-scope business process and inspecting the HTTP traffic is a source of many performance optimization “quick wins”.
- Files that could be compressed to save download time
- Cache settings that ensure that each user downloads static files only once
- Transaction steps that are slow even with a single user
Mobile users
With the popularity of Fiori, SAP has become a lot more mobile-friendly, but mobile devices typically have slower CPUs and are used on slower mobile data networks. Testing performance for mobile users can be done by emulating different network conditions (latency, bandwidth). A poorly configured SAP system is much more noticeable when mobile users have to download megabytes of uncompressed JavaScript every time they log in.
Single-user baseline test
The single-user baseline represents the best-case response times (as measured by your load testing tool), and it can be run in any environment. If an SAP system is slow for a single user it does not generally get faster, no matter how much bigger the Production environment is (exceptions to this rule-of-thumb are batch jobs and complex queries against HANA). The single-user baseline is a good test to run early (as soon as functionality is delivered), and provides a point of comparison to response times measured in the Peak Load test.
Peak Load test
The Peak Load test measures response times under “busiest hour of the busiest day” load levels. The mix of business processes and their peak hour volumes is defined in a Workload Model. Sometimes the Workload Model can be base on historical usage data, but often it is based on estimates, so it is good to document all the assumptions that are built into the model. There may actually be several “peak hours” with different mixes of transactions – an end-of-month (or end-of-financial year) peak is likely to have a different profile to a peak related to Thanksgiving sales period. As response times might vary for different user profiles (e.g. users may load more tiles during Fiori login), it is relevant to include the mix of user roles that will perform each business process. This testis generally run for at least an hour at peak.
Stress/Break test
The aim of the Stress/Break test is to find the point at which the SAP system fails or becomes unusably slow. Every system fails under some level of load, and the Stress/Break test shows how much safety margin there is between the expected Peak Load transaction volumes and transaction volumes that cause the system to fail. If there is only a small amount of headroom, then the project may choose to tune configuration settings and increase hardware capacity just in case the business has a busy day that is higher than the predicted maximum.
Failover testing under load
While not all SAP systems are deployed with an architecture that fully supports high-availability, there are usually some attempts to remove single points of failure (e.g. multiple web servers with a load balancer). After failover behaviour has been verified with a single user, the failover test is re-run under peak load conditions.
Soak Test
The aim of a soak test (or volume test, or endurance test) is to uncover memory leaks and other problems that cause system performance or stability to degrade over time. The test is usually run for 12 or 24 hours and metrics are trended to highlight any increase in response times or system resource utilisation over the course of the test. Running a soak test is also a great way to create bulk data to test batch jobs with realistic volumes.
Batch and Sociability testing
SAP system that have a user-heavy workload during the day and a batch-heavy workload at night might be configured with “daytime” and “overnight” operation modes. Operation modes reconfigure the number of processes available to process each type of work. Typical concerns with daytime batch and overnight are:
- When the daytime jobs run, do they impact response times for end-users?
- Can the overnight jobs finish within their allocated execution window when they have to process a realistic amount of data?
Other system activities (such as backup jobs) may also impact end-user response times or batch execution times, so these can also be included in test scope. Usually data conversion/data migration jobs are not included in the scope of performance testing, as these are once-off activities.
Interface testing
SAP system will often have integrations with other systems both SAP (SuccessFactors, Concur, Hybris, Ariba, etc.) and non-SAP (via EDI, etc.). Traffic from interfacing systems is either low-volume/low-impact, in which case response times should be tested with a single user, or high-volume/high-impact, which means that it should be included in the Workload Model for the Peak Load test.
MaxAttention/VTO
Many projects pay for the SAP MaxAttention VTO service. As the project gets close to their deployment date, the MaxAttention team will conduct a 3-day exercise to reviews the configuration of the SAP system and observe system metrics while a Peak Load test is run. They do not run any tests themselves, so the Peak Load test must be set up and ready to run when they arrive. To help the project get the best value from this expensive service, it is wise to find and fix as many load and performance problems as possible, as there is a limited amount that they can achieve in such a short time.
Dress Rehearsal
On an SAP project, the dress-rehearsal is a chance to practice all the cutover and day-1 activities that will be performed at Go Live. This generally includes running data conversion (or the final deltas if the database is huge), technical production verification tests, and executing end-to-end business processes. Dress rehearsal may be executed multiple times.
Sometimes projects will include a “day 1” load test as part of dress rehearsal. In the pre-HANA days, this test would sometimes uncover problems that were not present in the Performance environment – like poor Oracle query performance because the database had no stats to use for optimise execution plans. If the environment is freshly built, this test sometimes finds that manual configuration changes that were made in the Performance environment have not been applied to Pre-Production.
Victor Kouznetsov
There are as many load and performance test cases for SAP as there are variations of SAP products and configurations. Unfortunately it is not possible to just cut and paste a list of test cases into a Performance Test Plan – it takes deep specialist SAP experience to understand what should be in-scope for each project. I enjoy helping each unique project get better performance outcomes with the MyLoadTest SAP performance testing service.
