A view on testing Android apps

Automated testing in Android has become a huge trend in the last years, with many recommendations about what to test, how to test and when to test. Also, there are many tools that can help you with this, so how do we choose between Espresso, Robolectric, JUnit etc ? This article will explain the different tools that can be used, will explain why we need both unit and integration tests and focus on the integration tests.

Tools of the Trade

• JUnit. A Popular unit-testing framework for Java, provides simple and convenient APIs to write tests and perform common testing operations such as setup, teardown, and assertions with ease.
• Mockito/Easymock. The core idea of unit tests is to test a unit of code in isolation from any of its dependencies. When you are unit testing a piece of code, you really don’t care if its dependencies work properly or not, since you will need to test them separately anyway. Mockito or Easymock help you mock these dependencies.
• Dependency injection Framework. If you find your codebase littered with the “new” keyword everywhere, be sure that you will have a hard time using Mockito or Easymock to mock your dependencies. Dependency injection framework such as Dagger and Toothpick can help you with this.
• Espresso is an excellent framework developed by Google to help you easily and quickly automate UI tests that simulate real user behavior. These tests will run on a real device or emulator.
• Robolectric makes it possible to run Android tests right on your local machine by providing a dummy implementation of the Android SDK using shadow classes.
• MockWebServer is part of OkHttp. With MockWebServer, you can easily mock server responses including body, headers, status code etc.

Let’s start with the different types of tests that we can write

The Testing Pyramid

From https://developer.android.com

If you want to get serious about automated tests for your software there is one key concept you should know about: the test pyramid. This pyramid will give you a better idea of the different types of tests you need to write and how they differ from each other. Google recommends you to write 3 kinds of tests, small (unit tests), medium (integration tests) and large (end to end test).

In my experience most projects focus only on unit tests (written by developers) and on the E2E tests (written by QA Automation Engineers or developers) and they don’t have the middle layer of tests.

I imagine that most of the people reading this are already familiar with unit testing so for the rest of this article I’ll focus more on the integration tests.

Why do we need Integration tests ?

• Integration tests verify how different units collaborate with one another.
• With only single-class tests, the test suite may pass but the feature may be broken, if a failure occurs in the interface between modules. Integration tests will verify end-to-end feature behaviour and catch these bugs.
• Attempting to refactor a system that only has single-class tests is often painful, because developers usually have to completely refactor the test suite at the same time, invalidating the safety net. The integration tests are the safety net for the refactoring.
• Although small tests are fast and focused, allowing you to address failures quickly, they’re also low-fidelity and self-contained, making it difficult to have confidence that a passing test allows your app to work.
• An integration test typically covers a larger volume of your system than a single-class test, so it’s more efficient.
• Writing both unit and integration tests increases the coverage faster and improves the reliability of the tests.

In the ideal case you would have all your classes unit tested but we all know that in real case scenarios you probably don’t have enough time to do that. So the integrations tests can help you increase your coverage in a more efficient way. Even without a full coverage by unit tests, integration tests are useful to get more confidence in high level user stories working well.

Writing Integration tests

Let’s see what Google recommends:

Medium tests are integration tests that sit in between small tests and large tests. They integrate several components, and they run on emulators or real devices.

The advantage of running these tests on a real device or emulator is that it will simulate the exact same environment as the user will have. But, there is a major downside here: SPEED.

Imagine you have a very big project and you have hundreds of integration tests. Your tests might quickly take more than 30 minutes to run. Nobody wants to wait 30 minutes or more for a build to finish. The purpose of a good test suit is to find as many problems as possible as early as possible in your development cycle. That means having a large and comprehensive set of tests and running them early and often. If you have a large project with a lot of tests and they take a long time to run, you won’t run them as often and that defeats the purpose of having a good test suit.

We already established that unit tests are not enough, but the integration tests that run on a real device or emulator are too slow, so what should we do?

What we can do is use Robolectric and run the tests on our local machine. Not having to run the Android tests on an emulator greatly increases our speed of running tests.

So this improved our test speed, but we still need to load the entire Android Framework for each test so robolectric based tests are way slower then JUnit tests. Also, Robolectric is provided by a third party company so you have to wait for their implementation every time a new Android version appears.

And the answer is no. Robolectric is useful when you want to do black box testing where you don’t mock the Android dependencies, but if you just want to test your code, you don’t really need it.

From their documentation:

You can still use a mocking framework along with Robolectric if you like.

So why slow down all of your integration tests with Robolectric when you only need to test the integration with the framework in some tests?

A new view of the Pyramid

What we did here is we separated the integration tests into two layers, the Integration tests that use Robolectric, and Component/Feature tests which only test our code, with no dependencies on the framework and that can run fast and work on every build.

The bottom two layers are tests that you can easily run at every push/build. The top 2 layers can be runned periodically without blocking the development time.

Robolectric integration tests

• Validate how your code interacts with other parts of the framework but without the added complexity of a UI framework.
• Run reasonably fast. Anywhere from a few seconds to a minute
• May or may not mock external dependencies
• May access a local database or filesystem or network
• May or may not run in parallel
• Can test the integration with the views
• Can be used as a black box testing technique. E.g. Click on a button and expect some result
• Useful when you can’t mock the Framework

Feature/Component integration tests

• Test the collaboration of multiple classes that are building blocks for a specific feature/component.
• These tests should mock the outside dependencies and also mock dependencies from other components/features. A dependency injection strategy in your app would make it easier to mock the classes that are not part of the feature or component you wanna test.
• These tests should also mock the network and the filesystem since these are JUnit tests.

If you are using OkHTTP, you could use the MockWebServer so that your integration tests check that the feature/component performed the right request with the right params. Also, this way you can mock the server response with your own Json.

Try to write code as decoupled from the Android framework as possible in order to be able to write this kind of tests.

So when one and when the other?

So here are a few guidelines about when to use one and when another.

Writing Robolectric integration tests:

• If your feature heavily relies on integrations with the Android SDK or the UI, you can consider writing a Robolectric test to test this integration
• When testing the interaction with the database or the filesystem.

Writing feature/component tests:

• Every feature/component should have at least two JUnit integration tests (one with a happy flow and one with maybe an error/corner case flow). Write more if possible.
• Classes that contain no business logic, just delegate responsibility to dependencies should have feature/component tests.
• Classes that are so interdependent and the code is in such shape that you can’t unit test it. This will help you once you refactor it.
• You can mock fully tested dependencies if you want to force a specific scenario.

Keep in mind that writing integration tests for all possible scenarios would be equivalent to covering all combinations of all unit test of the constituent units. This would be a huge number, so pick the most common scenarios, and write unit tests for the other cases.

Writing unit tests:

• Is the class pure business logic? If yes, cover it fully with unit tests. Helpers, Util classes are good candidates.
• Is the class used in more features or by many classes? If yes, write unit tests for it. Everyone that uses it should have the security that that class does what it’s supposed to do.
• Does the class have good coverage with the feature/component tests? If no, add unit tests for the cases that the feature/component test did not cover.
• Your unit tests should catch edge cases and confirm correct class behavior.

Let’s take an example

Let’s say that this is the class diagram of a feature:

How would we test “Move John Snow To Winterfell”?

Robolectric integration test

A robolectric integration test could use the mockWebServer to mock interaction with the server, perform a click on a button, and assert that the right location was saved in the database and that a certain view has become visible in the UI.

Feature integration test

A feature test would:

1. mock the database,
2. setup the mockWebServer
3. set currentLocation inside JohnCharacterManager to “KingsLanding”.
4. call moveJohn(“Winterfell”). Here we start from the ViewModel, not from the view as in a Robolectric test.

At the end it would assert that:

• MockServer was called with the right params and the right headers
• dao.saveNewLocation(JohnSnow, Winterfell) was called
• JohnCharacterManager has stored the new location (“Winterfell”)

In this feature test you could also mock the PathGenerator to force a certain path. If the logic inside JohnCharacterManager or MovementRepository is dependent on the clothes of the character and the path, mock PathGenerator and the CharacterDresser in your feature test to test more scenarios.

Next, write an error case feature test where you override the response with an error like 404 and you assert that the location did not change, and that the repository returned the correct error.

Unit tests

Right from the start we see that PathGenerator and CharacterDresser are pure logic classes that might be used by multiple characters in different features. Also, our feature test did not catch all the possible locations, genders and seasons to provide a good coverage so we fully unit test them.

Let’s take a look at JohnCharacterManager. We can run the Jacoco report at this time and see that we have a good coverage for the move method so we can leave it alone. If there is a corner case where something should happen based on the path, the clothes or the location, you can write a unit test for that.

We can apply the same thought process for the HeaderGenerator. Since in our feature test we already verified that the headers are send to the server, we only need to see if there are some scenarios that are not covered and write unit tests for them.

Same goes for the ErrorHandler. If the ErrorHandler is specific for this feature, apply the same recipe as above. If this is used by multiple features, unit test it.

The ApiClient and Dao already have enough coverage from the feature tests so there is no need to unit test them.

Final thoughts

All the stuff above are just some guidelines on writing more efficient tests and you should customize what tests you write and when based on your needs.

You also wanna make sure that you don’t test the Android SDK components or other libraries that you use. Those are already tested and you should test only your integration with those components. E.g. you don’t have to test that when you save a file on disk it is actually saved. Also, testing you code is supposed to help you refactor when needed so if you test implementation details, this doesn’t help.

Due to its simplicity the essence of the test pyramid serves as a good rule of thumb when it comes to establishing your own test suite. The important thing is to remember two things from the test pyramid:

1. Write tests with different granularity
2. The more high-level you get, the fewer tests you should have

Stick to the pyramid shape to come up with a healthy, fast and maintainable test suite. If your code is not yet designed to support unit tests, it’s ok to start with the integration tests. These tests will allow you to refactor your code with more confidence and as you refactor it, you can start adding unit tests.

Given the fact that naming is hard, it’s totally okay to come up with other names for your test layers, as long as you keep it consistent within your codebase and your team’s discussions.