Effective Approach to Mobile App Testing
Mobile application testing tools have greatly matured in the last few years and I believe that it is a good time to start thinking about covering your apps with automated tests if you haven’t already. I have been investigating this topic for a while and maintaining tests in a handful of projects, and want to share my findings with you. Let me entertain you on the topic of testing by answering the following questions:
1. Which parts of the app to test?
2. What kind of tests to write for maximum efficiency?
3. How to motivate your team to contribute?
Which parts of the app to test
The short answer is this: the most important ones. What exactly is important depends on the type and size of the application, as well as the size of your development team. Below is a list that I would go through to determine the most critical parts of the app that must be covered, ordered from most to least important:
The main user flows
What is the worst possible thing that can happen to your app? It stops working. By that, I mean your users are not able to do things they are using your app for. For the majority of the apps, these are typically registration, login, some sort of search and booking functionalities. The critical features differ a lot per app type: if you have a car-sharing app, you want location tracking and reservations to work like charm, for food ordering that would be food selection and payments, while for banking apps you have to ensure that the user is always able to transfer and receive money without issues. The point here is the following: figure out the most common sequence(-s) of actions that the users perform with your app and reproduce these exact actions in your end-to-end tests. To sum up, the goal of these tests would be to ensure that the user can perform the most important operations with the app.
Payments and Paid features
While the app is free, even though you as an author are obliged to provide good user experience, there isn’t much that your users can do in case something doesn’t work as expected. Most of the time, they would “just” delete the app and try to find a competitor that works, without demanding anything from you as the author. However, the moment you add premium features to your application for which you are getting paid, things become a little more complicated. For example, unhappy users would start asking their money back and this could kill your entire business in no time. Therefore, it is crucial to cover your payments-related features with automated tests. Luckily, most payment providers have support for testing these features, and if you happen to come across one without any documentation about testing, you better think seven times about using it in your business.
Parts that everyone is afraid to touch
I think most of us came across a feature in some app that was perhaps written 5 years ago, had no documentation and would break every time someone would try to “fix” it. If you didn’t, you will, because this is the reality. When this happens, most developers would become ignorant to this piece of the app and never dare to even open the package where it resides. The problem is, such kind of “pearls” can cost companies a lot of trouble, because the management is never aware of these technical issues and just thinks that it “works” as expected, and any change to such functionality would take 10x the time of an engineer to implement. Here is what you could do to save the situation: isolate the feature as much as you can and hide the implementation complexity behind interfaces; write a test around this “black box” feature, where you would be passing certain inputs and expecting the right outputs. A few tests like this would not only ensure that you or your fellow developers are not going to break this in the next release, but also give you a possibility to refactor the implementation in the future when you would have some spare time from “building new features”.
Parts that are planned to be refactored
Continuing from the previous tip, hiding implementation details behind an interface and covering it with tests before a refactor is one of the best things you can do. Even though you would have to do additional work to make your current code testable and actually writing the tests, the benefits would very often outweigh the added work and save you time in the end. First of all, you are going to be able to rethink certain aspects of the feature while hiding implementation details behind an interface and maybe come up with a better solution. Secondly, you could turn the whole process into Test-Driven-Development since you could simply be tweaking your new implementation until the tests pass instead of manually testing after each change and hoping for the best. And last but not least, you would have your refactored code covered by tests for the future, which would again make it harder for everyone to break the functionality.
Fragile components, such as encryption, offline sync or complicated calculations
This matters if you have certain security requirements and are, for example, using AndroidKeyStore for secure key storage, Ciphers for encryption and decryption, or have some complicated calculations performed in the app, e.g. plotting some difficult graphs. Typically, only a few people in the organization would have decent cryptography knowledge, making it hard for others to maintain such functionality. Therefore, it is important to cover security-related functionality, e.g. encrypted SharedPreferences, with tests. Other apps might have critical offline functionality requiring synchronization with the server, which definitely needs to be tested as the systems are decoupled from each other. Regarding calculations, the story is similar. Typically these would be related to some complicated business logic that someone implemented at some point and only that one person really understands what is actually happening. If these calculations are covered with tests however, anyone could at least attempt to make changes without risking to break things.
What kind of tests to write
Since this article is about a practical approach to testing, I would like to have another look at the traditional testing pyramid and explain why you should probably turn it around in most applications. I am going to compare three types of tests: unit, integration and UI tests. If you don’t know the differences between these three types of tests, you might want to read this short post and come back. Here is how the testing pyramid looks:
What you can read in most resources related to software testing is the following: most of your test suite should consist of unit tests, then you should have some integration tests and a very limited number of UI tests. While this is required and works well in enterprise organizations with massive backend systems and several teams working independently on the same product, for the majority of mobile applications this is not going to work out well. Let me relate it to the previous section and explain why.
Unit tests require more maintenance
Unit tests are supposed to be fast to write and that is one of the selling points of this type of testing. Sounds good, but what you also have to keep in mind when writing unit tests is mocking. In reality, you will end up spending more time writing mocks for your unit tests than the actual test scripts themselves and most likely end up with more code compared to an integration test or a UI test. But even this is not an issue. The real issue comes with maintaining these unit tests during updates and fixes to functionality, or, what is even more painful, refactors. Based on my experience, unless your application is well-architected and your “units” have their implementations abstracted properly behind interfaces or delegates, your team will end up either @Ignore -ing these tests, commenting them out or removing from the project completely whenever a major change is made to the functionality. In the end, what is going to happen is this: you spend time writing the mocks and the test; a few months later someone on your team pushes a change to this functionality and the test starts to break; they try to fix it for 15 minutes and eventually remove the test because they are sure “it works” and it “is not worth spending the time to fix”.
I believe that it is worth unit testing encryption, complex calculations, backend synchronization logic (in offline-first applications) and perhaps new components that are well abstracted. In other cases, integration or UI tests are more effective. Here is why.
Integration tests cover a broader spectrum of the application
It’s like a 3-in-1 coffee: you write one test which uses several “units” or “components”, effectively testing all of them at the same time. These are similar to unit tests, except that you are not testing your “units” in total isolation. As a result, you would benefit in two ways: less mocking and more functionality covered with a single test. From here, you would also be able to identify components that are the most fragile and perhaps cover them with isolated unit tests. On Android, integration tests can be executed on JVM using Robolectric to run faster than on an emulator or a real device.
Only UI tests are covering the main user flows
According to the previous section of this article, the most important parts of your app are the functions that the users can perform. Often user interface serves as both the input and the output of these core actions that can be performed with the help of the app. Therefore, without UI test scripts reproducing the exact user actions, we would not be able to ensure that these actions would produce the desired result. Similarly to integration tests, UI tests require little maintenance, except for situations when the user interface changes. This typically happens less often than other changes such as refactors, therefore one can keep improving the implementation as much as they want without a need to change UI tests.
There are two drawbacks to such tests, however: slow execution and a need for an emulator or a physical Android device in the CI setup.
The first problem is a really nice one to have since it means you have an extensive test suite and your application’s functionality is hard to break. In such situations, you can optimize your tests to reduce duplicates. For example, to test 3 scenarios of feature B, you first need to enter some information in feature A, and you end up filling exactly the same information inside feature A three times. What you could do is fill this information in one of the three tests in UI, while in the remaining two tests the data could be entered programmatically and passed to feature B directly, saving precious execution time. However, such optimizations have their limits. If the test suite still takes too long to run, it is time to split up the test suite and run several suites in parallel. This could be achieved by either connecting and configuring several emulators or real devices with your CI server, or by using the emulator in a Docker container and running several jobs in the cloud at the same time. The second approach is more scalable, although it can be a bit more costly and you are limited to an emulator, so keep that in mind.
To conclude this section, UI and integration tests would save you a lot of time since they typically require less maintenance and cover a bigger part of your application. Unit tests are suitable for certain specific components, but in most cases, the costs of maintaining them are too high compared to the benefits that you could get from them.
Now that we have answered the first two questions, I will share some tips on how to keep your tests relevant by motivating your team to contribute.
How to motivate your team to contribute
It all starts with one person eager to improve the quality of their product and introduce automated testing. Up to a certain point, it might be enough, but as the team grows, it is essential to keep everyone on the same page and contributing to maintaining these tests. While enforcing certain coverage in your development pipeline might do the job, I believe that motivation would work better than enforcing any boundaries in this case.
Showcase your achievements
If you want your team to see the value in automated tests and join the effort of maintaining them, you have to share your experience and passion with them, showing the value of tests. In your regular tech meetings within the company (you definitely have these, right?), showcase your latest achievements in testing the application. Tell them how you covered the feature with tests and were able to refactor the implementation in less time without risking of breaking the functionality. Show them how you were able to rethink the implementation and improve it because you wanted to write properly testable code. Tell a story about this new developer at company X who was able to understand how the app functions on his own by just looking at the test scripts and watching them run. Yet another thing you could tell is how faster the QA process goes when tests are in place, i.e. fewer broken features = less QA manual testing cycles = less frustration for developers and a faster release procedure. Even if all of this doesn’t spark interest in your colleagues, at least you would learn something about them and practice your presentation skills.
Make use of analysis tools
In addition to the above, you could integrate some nice tooling to motivate your colleagues to contribute to test coverage. For example, you can integrate a static analysis tool such as Sonarqube in your project, which also keeps track of code coverage and provides some useful metrics to track your team’s performance. You can read about our experience with Sonarqube at Pinch in this article, where I explain what the tool is about and how we integrated it in our development pipeline, as well as share some thoughts on the outcome of this integration. It is always nicer to contribute to this kind of improvements like application test coverage when you can actually track your team’s progress.
The story ends here. I hope you were able to get something out of it. If you agree, disagree or have any questions, let’s discuss it right here in the comments. Thank you for your attention and happy testing!
PINCH is a leading Dutch app agency. Being familiar with and working with the latest technologies is part of our work. That, together with our flexible, down-to-earth approach and startup mentality with corporate experience, allows us to add extra value for our customers. PINCH is creating apps for iOS, Android, Tizen, Apple Watch & Apple TV.
