Today, mobile-first and even mobile-only applications and services have become a common practice. It can be safely said that organizations should have a strong mobile presence to beat the competition and gain a competitive advantage. Having a cutting-edge mobile strategy means more emphasis on development, and having robust testing to provide a flawless, effortless experience to users.
While working on a mobile testing strategy, organizations often face a variety of challenges like layout issues, device incompatibility, OS issues, network issues, authentication issues, etc. But with the growing array of mobile devices available on the market, testing apps for all the possible problems on a heterogeneous mix of devices may create a dent in the pockets of an organization. A recent survey suggests that 36% of organizations are spending more than $100,000 on mobile app testing. Having a planned approach can help optimize output while keeping expenses under control. One such approach, as described by Kwo Ding, is the mobile test pyramid approach.
THE MOBILE TEST PYRAMID
The mobile test pyramid can be broken into three layers or levels:
- Real devices
- Mobile simulators and emulators
- Desktop browsers
Each layer comprises specific ways to perform mobile testing and has associated cost brackets. For example, testing desktop browsers could be a cost-effective alternative with quick turnaround time, but it may miss a few essential capabilities, like testing for native mobile APIs. Mobile simulators/emulators are easy to download and set up, and they provide a better simulation of the actual mobile environment — but they also lack a few essential aspects and testing scenarios like connectivity with Bluetooth, NFC, network connections, etc. Real devices can be even faster than emulators and provide the actual environment, but procurement of the devices is usually costly in terms of both time and money. For an optimized outcome, organizations should focus on their requirements and decide on performing the right things at the right layer. An in-depth look at each of these layers can help build understanding of the tradeoffs better, and help testers in setting up the right expectations with each of these layers.
MOBILE TESTING PYRAMID — DESKTOP BROWSERS
Desktop browsers are the most readily available environment for testing mobile apps. Almost all major web browsers support mobile app testing, making it a smooth and cost-effective method. Desktop browsers’ responsive designs allow for playing with user agents and checking for compatibility with different screen sizes at the click of a button, or by dragging to resize. Desktop browsers are ideal for functional system testing (e.g., for having isolated browser tests for carrying out a full functional test).
ADVANTAGES
Desktop browsers provide quick setup, as they can be launched within milliseconds. They are scalable and can be extended easily to 10+ instances per machine. Their compatibility across all major operating systems makes them suitable for cross-platform testing scenarios as well.
CHALLENGES
The use of desktop browsers is restricted due to several limitations. With desktop browsers, extensive visual checks are not possible because the rendering is very different from the actual mobile device. A lack of native mobile integrations like keyboards and incoming calls restricts testing of several functionalities. You cannot perform native integration like radio connections, switching between apps, etc. Debugging a mobile app can also be a tough task with desktop browsers due to the lack of appropriate error logs.
MOBILE TESTING PYRAMID — SIMULATORS AND EMULATORS
Mobile simulators and emulators provide a more realistic environment than desktop browsers. Simulators/emulators for all major mobile device platforms are readily available and can be set up very quickly. In addition to all the features of desktop browsers, simulators provide several additional capabilities like native API integrations, and touch interactions such as swipe and tap — hence, providing a realistic user experience. It is also easy to debug simulators/emulators, as they are already hooked up to the machine via access logs. These are ideal for testing functional end-user flows (like click paths throughout the application).
ADVANTAGES
Like browsers, mobile simulators/emulators are also easily scalable, and users can run several instances in parallel in the virtualized environment of a single machine. Several native API integration capabilities are provided, like incoming calls, GPS navigation, file attachments, etc. Additionally, the simulators and emulators built on Intel-based architecture usually perform quickly.
CHALLENGES
Simulators/emulators are available only in vanilla versions. Although manufacturer skins are often available, device behavior is still based on stock only. Due to a lack of support for an actual display environment (like a capacitive touchscreen), simulators may lead to inaccurate visuals like the color display in light/dark. Also, the lack of actual resource usage (CPU/memory), and lack of real connectivity options (Bluetooth, NFC, network connection) can make it challenging to test several real-world scenarios.
MOBILE TESTING PYRAMID — REAL DEVICES
Mobile testing on real devices is the closest possible realistic end-user testing environment. It allows for the actual validation of usability, like click areas, touch actions and voice over. It also provides realistic performance scenarios with actual CPU/memory usage, battery, and network strengths. All the native API integrations are also available with actual resource sharing (camera, GPS, NFC, Bluetooth) and interruptions (like incoming calls, push notifications). This makes it ideal for testing real-world scenarios, giving testers a realistic view of performance.
ADVANTAGES
All native API functionalities can be tested in real conditions, not just with injections for automation (for example an actual NFC touch, or battery/CPU/memory usage scenarios). This also provides testing with the specific manufacturer’s OS and built-in browsers, which are often different from the standard and well-known browsers such as Chrome. Real devices are often faster than emulators which are slowed down by the simulation of hardware. This is especially true in the case of ARM-based emulators.
CHALLENGES
Procuring a real device may take time, and is a costly affair, as devices are available on a per device/cradle basis. New devices are usually not available on-the-fly. Also, there is often a need for additional formalities, such as iOS apps needing to be signed with a Development Distribution Certificate (DDC), and a Provisioning Profile for automation.
CONCLUSION
The mobile test pyramid provides a comprehensive approach towards mobile app testing, whereby developers can decide upon the suitable options based on available budget and resources. This can help developers to cover all areas to be tested before publishing the mobile app to the store in an optimized and economical way.
