Whether you’re developing native apps for mobile devices, hybrid apps, or browser-based applications which will run on the mobile web, you need to do thorough testing on all of the key mobile platforms. That’s a given.
And you already know that automated testing saves money, reduces test time by one or more orders of magnitude, and allows you to extend the depth and range of your testing far beyond what is practical or even possible with manual testing.
So making the move to automated mobile testing should also be a given. But for far too many organizations, the pathway to automated mobile testing looks more like an obstacle course than a well-traveled highway.
But it doesn’t need to be that way. In this post, we’ll take a look at the most common hurdles (potential, real, or merely apparent) to automated mobile testing, and how you can identify and get past the barriers which your organization may face in setting up a comprehensive automated mobile testing regime.
FINDING THE RIGHT TOOLS FOR AUTOMATED MOBILE TESTING
Which tools are best for automated mobile testing? This is really a two-part question—first, you need to determine which tools are best suited to your mobile testing needs. Then, you need to find out which ones provide the best automation features for your testing regime.
One of the most widely-used mobile testing frameworks is Appium, and it can be used with both iOS and Android. Calabash, another well-known framework, includes separate versions for iOS and Android.
Other test frameworks are designed to work with only one of the two major mobile operating systems. iOS-driver, for example, works with iOS, while Robotium and Espresso are designed for testing with Android.
Each of these frameworks offers some advantages and has some drawbacks with regard to automated testing.
THE APPIUM WAY
Open-source Appium, for example, minimizes much of the work required for automation, eliminating the need to recompile or modify code. It does this in part by using automation frameworks provided by OS vendors, then using a standard API (WebDriver) and client-server protocol (JSON Wire Protocol) to manage communication between the automation frameworks and the test clients (which can be written in any language).
Needless to say, Appium also has some potential limitations (configuration for iOS and Android can be complicated, for example), but you can weigh those limitations against the specific requirements of your app and test regime when deciding whether Appium is suitable for your purposes.
SIMPLE AND FOCUSED
This holds true for other test systems. Both Robotium and Espresso, for example, are open source frameworks for testing Android apps. They both have fairly simple test APIs and are easy to set up. Robotium is good for black-box and gray-box tests, while Espresso is designed for white-box tests, and includes detailed debug output. Both frameworks require Java for testing (although Espresso simplifies the process with Hamcrest).
SCRIPTING LANGUAGES
Robotium and Espresso’s use of Java brings up another question: What language(s) do you and your test staff want to use for test scripts? There’s no overall standard among test frameworks. Some, such as Appium, give you broad latitude in your choice of scripting languages, while others require familiarity with a single language.
Calabash, for example, typically uses Cucumber, although it can work with other Ruby-based scripting systems. If your testing team is comfortable with Cucumber or Ruby, Calabash may be a good choice; if not, then the potential learning curve may be a consideration.
In the iOS world, Apple’s UI Automation tool uses JavaScript and allows you to record manual tests for automation, while KeepItFunctional requires Objective-C, and iOS-driver supports a broad range of scripting languages.
WHAT CAN YOU AUTOMATE?
What parts of your mobile test regime should you automate? What parts can you automate? The ideal, of course, would be to automate everything; in practice, some elements of your application will lend themselves to automated testing more easily than others.
It’s often easier to automate testing for the parts of a program which do things in a reasonably standardized way—for example, options that are presented to the user by means of commonly used GUI components. A truly non-standard interaction or internal process may present more challenges when it comes to test automation, but in general, those challenges will be of the “what to look for” variety (i.e., where and how to look for the expected result of an action), rather than the “How can I even test it?” sort.
For the most part, you should be able to automate tests of user interactions (and pretty much anything with an input and output), as well as stress and load tests, and tests for internal handling of resources, saved states, interaction with the OS, and many elements of the GUI itself.
BATTERIES MAY BE REQUIRED
Depending on the test framework, you may need to take extra steps to automate tests for specific features or types of interaction. Some of the underlying resources which Appium uses, for example, may require extra scripting or specialized APIs to handle gestures. While Appium is a broad-spectrum mobile testing automation framework, some other frameworks are much more focused, and may be suitable for in-depth testing of specific program elements or types of functionality.
CHOOSING AN AUTOMATION FRAMEWORK
Before you choose a mobile test automation framework, it’s important to have a clear overall picture of what you need to test, and of any specialized testing requirements. Here’s a quick (although by no means complete) checklist which can serve as a starting point for determining your automated mobile testing requirements:
- Which mobile platforms do you need to test on? iOS? Android? Both? Which releases of each OS are the most important?
- Is your app native, hybrid, or mobile web? If it’s mobile web, which browsers (and versions) are the most important for testing?
- Which elements of your app are you most concerned with testing? GUI? Memory and resource use? Load and stress? Basic functionality? All of these?
- Are there any specialized/unique features that you need to test? Do you have a clear strategy for testing them?
- Do you need code-level testing? Black-box, white-box, or gray-box?
- Approximately how many different tests (within an order of magnitude) do you anticipate running? How many of those can be run in parallel?
- Which test scripting languages do your test engineers have the most experience with? Do they have a preference (e.g., are they happy with Python and JavaScript, but don’t want to touch Ruby or C#—or vice versa)?
Some of these questions are probably not relevant to your application or your testing needs, and there are very likely to be others which you need to ask as well. But if any of these questions help you get past any of the hurdles to mobile test automation which you face, then they (along with this article) will have done their job.
