Testing in the Cloud

 

Situation: Testing your large-scale applications would take more infrastructure than you can afford.  

Some of the most impressive global applications are customer facing retail systems that are underutilized most of the time, then must perform well during extreme bursts of activity that are nearly impossible to replicate in any testing environment.  Often the flaws in these applications only show up in production when the price of failure is high. Recently we spoke with Steve Dykstra, Product Management Director for Micro Focus and asked him a few questions about testing in the cloud.  His answers offer a pretty good overview of the subject and why it might be useful to look into.

Q.  Most large scale applications go through boom and bust periods of performance, yet testing peak performance would require huge investments in infrastructure.  How do organizations avoid purchasing large volumes of hardware to support the scale of tests that they need?

A.  Increasingly, companies are looking to the “Cloud” as a way to improve IT performance.  Imagine being able to focus the power of the world’s largest data centers onto your computing needs – just when you need it most.  That’s what the Cloud provides.  It is a virtual set of computing resources that can be securely accessed via the Internet.

And now through Cloud-based computing you can apply the power of the Cloud to peak load testing.  Cloud-based performance testing lets software quality teams rapidly launch any size peak-load performance test without the burden of managing complex infrastructures.  Now, you can test and diagnose Internet-facing applications under immense global peak loads.

In principle, your Cloud-based peak load testing solution should allow users to define the numbers of virtual users required for a given test, and schedule the computing resources necessary.  They should be able to match the geography of these computing instances with plans.  And they should be able to define the scripts that will be used to simulate user behavior.

These scripts are then accessed by Cloud-based test agents and run on a performance testing schedule.  Managers would then be able to monitor tests and efficiently locate and diagnose where performance issues arise in their applications -- a critical step for any test manager.

Q.  We have seen that some industries are particularly susceptible to problems with peak performance.  Retail is a good example.  Which retail-specific events lead to performance related issues?

A.  Thousands or even millions of customers, partners and employees engage with your company every day via the applications that you develop and test. So, these systems must perform as expected and be accessible when needed.

But your applications are under massive stresses.  Growing volumes of customers access your systems at peak times of the day, season, or at discrete events like following a major marketing promotion.

And it’s not just the volume of users that leads to higher demand on your applications; there is also increased application complexity.  Today’s Web 2.0 applications, which are designed to be more responsive to users, can be highly resource intensive.  This compounds the effects of increased demand.

When your application is subject to sudden spikes or sustained usage volumes, it can behave in unexpected ways.  The system may crash or become too slow to use.  It could also become inaccessible as customers compete for access.  Regardless, it leads to frustrated users and potentially lost business.  Every moment that the application is inaccessible is potentially millions of dollars in lost revenue.  Sudden spikes could be due to a holiday season or a sale.  The business of retail is built on these kinds of spikes, so your applications should be too.

Q.  Web 2.0 and social applications are often used to enhance shopping experiences.  Will that affect how retail organizations should handle performance testing?

A.  Today’s applications are often developed as dynamic and highly interactive Web 2.0 applications.  These Rich Internet Applications require special testing in order to access functionality developed in AJAX, Silverlight, and Flex, among other technologies.  Simple application testing is no longer sufficient because the background interactions and richness of the client-side are so great.  This is well-suited to Cloud-based testing as it requires more computing power to perform these more complex tests.  However, very few solutions are sufficiently mature to handle the dynamism of this style of application.  When looking at a solution, make sure to investigate whether Web 2.0 support is needed and the degree of coverage you have.

While many applications today are entirely browser-based, that is not often the case for large enterprise applications.  For instance, a bank’s core banking application will likely have a hybrid model.  Part of the application is browser-based Internet banking and another part may be only accessible via bank tellers’ terminals.

This means that you need to test both routes to your system for completeness.  An approach that relies solely on Cloud-based testing will be lacking since it needs to access the application via Internet protocols.  As a result, it is important to determine upfront whether you need a mixed model that combines Internet protocols with support for .NET, Java, Oracle, SAP, Siebel, COM, and other enterprise application protocols.

Q.  Most retail applications on the web are global.  Do testing needs change with a distributed user base?

A.  First off, not all Clouds are equal.  Some can scale more readily than others.  So, you should consider the levels that you need.  Whether you need computing power for tests with 50,000 to 100,000 to 200,000 virtual users and beyond.  Of course not all Cloud based load testing providers can scale to this degree and not all testing solutions can effectively harness these kinds of resources, so care should be taken when selecting vendors.

Further, you should be able to simply schedule time for a test and resources are automatically provisioned.  This avoids testing bottlenecks and prevents long delays as internally managed hardware is acquired and set-up.

Also, the global nature of some Cloud-based solutions lets you “place” Virtual Users in a variety of locations to test international performance.  No longer do you need to maintain hardware in a variety of countries in order to test.  Not all Cloud-providers or test solutions can provide this capability, so it is key to evaluate if global-readiness is a requirement for you.

Discovering that your application failed under peak loads is clearly not enough.  You want to discover why it failed and how to correct it.  While this may seem obvious, diagnostic tools are often excluded from testing solutions.  This is often the case when Cloud-based peak load tests are operating against an application you manage in-house.

Cloud-based tooling on its own cannot analyze the internal behavior of the application under test.  This can mean that applications may be incompletely repaired following tests, increasing the risk of a real-world failure.  It is more effective to combine the power of Cloud-based testing with “on the ground” diagnostics of your application performance.