A guide to test pyramid in Android — Instrumented tests — Part 3
A detailed guide about instrumented tests in the Android pyramid
In the first and second parts of the test pyramid posts, we had an overview of the basic concepts and how the base of the pyramid is structured. In this post, we will explore its center and talk specifically about the instrumented tests of your Android project.
By definition, instrumented tests on Android are those that run on physical devices or emulators. In these tests, the use of mocks is reduced and we gain the advantage of using the actual implementation of the Android framework, thus enabling the use of classes such as SharedPreferences, Context, Parcelable and Intent. A characteristic of this layer is that the tests run slower but also achieves a higher level of reliability and integration.
In the center of the pyramid, I usually separate the tests into 3 categories:
- Integration tests exclusively with AndroidJUnitRunner + AndroidJUnit4
- UI tests with Espresso
- End-to-end tests with UI Automator
Testing with AndroidJUnitRunner + AndroidJUnit4
AndroidJUnitRunner is a test runner that allows you to run JUnit3 or JUnit4 tests on Android devices. It serves as the basis for all instrumented tests and through it, we can have access to Android framework classes in our tests. Summarizing, AndroidJUnitRunner is responsible for:
- Loading the test package to a device
- Loading the app under test to a device
- Running tests in an instrumented environment
- Reporting test results
We also have an optional runner called AndroidJUnit4 that we can use with the @RunWith() annotation in case we want to ensure that our tests are going to use JUnit4 in our instrumented environment.
To gain access to the instrumentation API and build tests at this level, Android provides the InstrumentationRegistry class. Below is a very simple example of what this type of test looks like:
Note that as much as we are using an emulator or smartphone to perform this test, it is not yet considered a UI test. The above test only saves a value in the SharedPreferences and verifies that the value has been saved. This type of test is also commonly referred to as instrumented unit test.
Tests involving data persistence or accessing real Android resources (such as strings.xml) can be done at the instrumented layer of the pyramid.
With the release of the new AndroidX test libraries, we now have a new way to access Android resources in the instrumented layer. We can also use ApplicationProvider instead of InstrumentationRegistry as shown in the example below:
Espresso tests
Advancing to a higher level in the pyramid, we have UI testing with Espresso. Espresso is a tool designed for developers who are more familiar with the codebase. One advantage of Espresso is that you can initialize a screen directly without going through all the flow until you get to it. Check out below how an Espresso test looks like:
The way we write an Espresso test is different from the way we write Robolectric tests or JUnit instrumented tests. Here we write code in a way that resembles what a user would do in your App: Selecting a view (onView), doing some action (perform) and checking something on-screen (check). Just like the example below:
Espresso provides several ways to get views from your UI, perform actions and make assertions. Below we have a sheet with the main Espresso commands:
It’s also interesting to mention that with Espresso we can also test WebViews, Intents and accessibility in your app. I consider it to be a must-have tool in your Android testing pyramid.
End-to-End tests with UI Automator
Arriving at this layer of the pyramid, we come across end-to-end tests. This kind of test aims to test that a user flow is behaving properly from beginning to end.
UI Automator is an end-to-end testing tool provided by Google. The main difference between Espresso and UI Automator is that Espresso tests the UI at a much more isolated scope and UI Automator can test at a much broader scope that goes beyond the app that is being built.
With UI Automator we can write a test that:
- Accesses device settings through the launcher
- Enters another application
- Tests over an obfuscated APK
The possibilities are much greater.
Another advantage of UI Automator is the ability to write end-to-end tests using the same project in Android Studio. Some other end-to-end testing tools do not allow this so you have to have a separate repository to write your tests. To make our life easier when writing this type of test, some support tools have been created for us:
- UI Automator Viewer: Tool designed to inspect your screen layout in more detail and help extract information such as IDs and content descriptions. This tool is located at <android-sdk>/tools/bin path.
- UiDevice API: Class in code that provides helper functions such as changing the orientation of the device, pressing the back or home button and even opening the notification menu.
Below we can find an example of a UI Automator test using the UiDevice API:
Be aware that UI Automator tests are the heaviest and most sensitive to system changes. Any updates to Google Play Services or to the operating system itself may impact your testing. As the official documentation itself shows:
Caution: We recommend testing your app using UI Automator only when your app must interact with the system UI or another app to fulfill a critical use case. Because UI Automator interacts with a particular system UI, you must re-run and fix your UI Automator tests after each platform version upgrade and after each new release of Google Play services.
So we should choose very carefully the tests we want to build in this tool. If we can do the same thing with Espresso, try to do it with Espresso.
There are so many tools! How can we work with all of them?
Yes, having a big quantity of tools can be a problem as well. As mentioned in the previous post, it was announced that Robolectric will be part of Android’s official testing tools, and now that we know some more testing tools as Espresso and UI Automator, we may come across the following question:
Wouldn’t it be too messy to have it all together in the same pyramid? How do all these tools relate? Do I have to learn them all?
It is a very valid question. On the one hand, we have a huge range of options for writing our tests, on the other hand, we may lack standardization if we are not careful. That’s why Google wants to unify the way we write our tests in a single API. This is one of the goals of the Nitrogen project and the Jetpack Test Support Library.
For example, it seems that from Robolectric 4.0 onwards we will be able to standardize test writing to be very similar to what Espresso proposes. Below we have an example of a test with Robolectric 4.0 using androidx in a non-instrumented layer:
The example above has the same syntax as Espresso, but it is Robolectric! The objective with that is to lessen the cognitive load when building our tests. "Write it once, run it everywhere" they say.
This standardization is still advancing in the Android world and we will hear more about it in the near future.
Bonus Topic: Flaky Tests
Flaky tests are the same as non-deterministic tests. Tests that have inconsistent behavior that sometimes passes and sometimes fail. We often have problems with intermittent testing as our test suite grows, especially in the higher parts of the pyramid. The reasons can be several:
- State from the previous tests that are passed on to the next tests
- Infrastructure issues such as low memory of the machine running the test, especially when using emulators
- Animations that are not very well managed by the test tools
- Thread Concurrency Issues
- And others…
The big problem with intermittent testing is that they take away the reliability of your suite and greatly disorder the continuous delivery process. So I will mention some tips that can help mitigate this issue on Android:
- Using Android Test Orchestrator creates a sandbox for each instrumented test making it very difficult for them to share state. The downside is that Test Orchestrator makes your test suite much slower
- Remove all animations from the device being tested as stated in the setup guide
- Make use of IdlingResources, which are operators that we can implement to warn Espresso when their UI will be available for verification
- In the worst-case scenarios, we can add a re-run on failed tests. So we don’t need to run the entire test suite again. This practice is not recommended as it doesn’t solve the root problem and can greatly increase your suite execution time
And with that, we end the third part of the test pyramid series. In the next part, we will explore a little more about end-to-end tests that are built outside of Android codebase and also explore a little bit of manual testing.
Feedbacks are very welcome, thank you very much!
