TROUBLESHOOTING PERFORMANCE ISSUES IN TEST ENVIRONMENTS : A REAL-WORLD SCENARIO

Functional Testing is an essential part of the software development process, ensuring that applications work as intended. However, what happens when your functional testing encounters performance problems in the test environment ? Surprisingly, performance problems don't always wait until production to rear their ugly heads.

Increasingly, we witness performance issues causing delays and frustrations in test environments, creating headaches for program managers and testers alike. These problems can lead to significant delays in various forms of testing, jeopardizing project timelines and budgets.

THE TEST ENVIRONMENT DILEMMA

Performance or stability problems in test environments can be particularly vexing because they are often poorly understood. Vague references to issues like slow databases or network problems leave testing teams in a quandary. Developers are busy tackling functional issues, while system administrators are preparing the production environment, whilst testers and program managers find themselves stuck in the middle. Diverting resources from bug fixing or production platform preparations can initially exacerbate the delays caused by performance problems. Quick fixes, like moving a test environment to a larger platform, seldom provide the desired resolution.

Performance testing typically occurs alongside other types of testing, often on a separate platform. The process can be time-consuming, involving the preparation of scripts, user accounts, and input data. The primary aim of performance testing is to deliver performance improvements in time for a smooth production rollout. However, waiting for the formal performance testing process to complete is not always feasible when issues arise.

A SOLUTION FOR PERFORMANCE PROBLEMS IN TEST ENVIRONMENTS

We recommend a flexible approach to dealing with performance problems in the test environment. While no two applications are identical, the following approach can be adapted to address different scenarios:

FORM AN INFORMAL GROUP

Gather a small, informal group with representatives from various areas involved in the application, including a DBA, middleware administrator, web or Citrix administrator, performance tester, application designer, functional tester, business user, and any other relevant individuals. These team members don't need to be fully dedicated to performance issues, but they should take ownership of performance problems in their respective areas.

RESPONSIBILITIES

‍Assign responsibilities to the group members, including liaising with their teams about performance issues, investigating and defining performance problems in their areas, dispelling myths and rumors with concrete information, and, most importantly, fostering communication about the application's characteristics and behavior.

COLLABORATION IS KEY

Performance problems in test environments are rarely isolated. They often have multiple causes, with some issues causing a domino effect. For instance, queuing in a middleware component may result from slow response times in the database rather than a lack of threads. Adding more threads may exacerbate the issue. The middleware and database administrators need to collaborate to resolve the problem.

COMMON PERFORMANCE PROBLEMS IN TEST ENVIRONMENTS

Several typical performance problems are experienced in test environments, including:

•  Applying production configurations to components in the test environment that exceed the platform's memory capacity, leading to resource overutilisation.

•  Using default values for all parameters in an unconfigured test environment, underutilising resources.

•  Running functional tests against a database loaded with a large volume of data, revealing inefficient SQL. This is common for new applications, as developers may not fully tune SQL in a development platform.

•  Inefficient configuration of connections between architectural components' layers.

•  Incorrectly applied load balancing against components, resulting in both overutilised and underutilised servers.

CHANGE CONTROL AND COMMUNICATION

Change control for performance-related changes is crucial. While rigorous control and management of functional issues are standard during the testing phase, the same level of attention is often lacking for performance-related changes. Effective communication and management of performance problems are essential to ensure they are addressed and migrated correctly to the production platform.

In the world of software development, performance issues can strike when you least expect them, even in the test environment. But with a collaborative and proactive approach, you can mitigate delays, address problems, and ensure your application meets its performance goals, on time and within budget.