Stress-testing Android apps

Before starting with the recently published Trade Me app for jobs hunters, we spent some time reading and thinking about how we can build a stable app and how to make sure that the quality of the app is maintained over time.

I’ll not be covering in this post stuff like having a CI (we use Jenkins), choosing to go with basic MVP, DI (Dependency Injection), mocking HTTP responses, etc… And the value of having presenters like simplifying writing unit tests and forming a habit of writing new test cases to cover new code. They also helped making sure that the correct methods on the view were triggered with the right parameters.

The above gave us enough confidence that we were not breaking old code by adding new one, and that the quality is hopefully maintained over time.

We had a debug drawer in the internal builds like all the other Trade Me apps. Check JakeWharton u2020 sample app if you want to build something similar in your app. It helps with emulating bad network connectivity, delays, network errors, etc…

I remember sitting with Gabriel our mobile tester and discussing how we can make sure that what we build is stable and ready for release and we came up with the following checklist:

1. Use the debug drawer to change the Network delay
2. Use the debug drawer to change the Error rate
3. How does the app behave in Offline mode
4. Orientation change, what happens if the app was in the middle of a web request, does it cache and restore its state correctly, etc…
5. App killed while in the background
6. App Update, any required migration, serialization issues, etc…
7. Key Bashing
8. Multi-Window mode (Android N)
9. TransactionTooLargeException (Android N)

It starts to get really interesting when we combine multiple items on this list together.

1. Network delay

Using the debug drawer we can change the network delay which is usually a good test when combined with orientation change or app killed in the background.

While a network request is in progress, how would the app handle screen orientation changes or leaving the app (and then coming back to it after a while)?

2. Error rate

Using the debug drawer we can change the error rate to check how the code handles unexpected errors.

Does the app handle HTTP errors and other errors as expected? Does it notify the user or log the error to Crashlytics as non-fatal crash?

3. Offline mode

Simply by turning OFF the WiFi or turning ON the airplane mode (for total connectivity loss).

We want to test things like:

• Will the app crash if a network request was already in progress?
• Notify the user that the connection was lost?
• Is it caching what should have been cached?

4. Orientation change

Activities and non-retained fragments get destroyed and recreated on orientation change (assuming configChanges orientation was not set in the manifest, check Handling Runtime Changes).

We want to check the impact of orientation change on the performance and behavior of the app:

• What happened to any ongoing requests?
• Was the app state restored correctly?
• Was the correct related landscape/portrait layout/resources loaded?

5. App killed while in the background

Sometimes Android will kill an app while it’s in the background to cleanup memory for other running apps. When we go back in the app all the dependencies required by the current screen should be re-initialized if needed (does the current view depend on something initialized in a previous screen).

Ways to reproduce:

1. Open the app
2. Press device Home button
3. Run the following command in terminal

adb shell am kill YOUR_PACKAGE_NAME

Another way would be to go to device Settings > Developer Options > look for “Do not keep activities” and check it

Optimizing for Doze and App Standby, is an interesting, somewhat related read.

6. App Update

Did the app sustain any previous data after the update? In other words, is this update backward compatible?

You need to worry about Serializable objects stored in the SharedPreferences, for example: will they get deserialized (changing the fields order can break deserialization), database scheme changes, etc…

7. Key Bashing

Vigorous swiping and tapping can produce some strange errors.

You can use Monkey for that, it is a command-line tool that you can run with the following command:

adb shell monkey -p YOUR_PACKAGE_NAME

8. Multi-Window mode (Android N)

If we enable multi-window support, we need to verify the following behavior under both multi-window and standard modes.

• Launch the app in standard mode, then switch to multi-window mode by long-pressing the Overview button. Verify that the app switches properly.
• Launch the app directly in multi-window mode, and verify that the app launches properly.
• Resize your app in multi-window mode by dragging the divider line. Verify that the app resizes without crashing, and that necessary UI elements are visible.
• If you have specified minimum dimensions for your app, attempt to resize the app below those dimensions. Verify that you cannot resize the app to be smaller than the specified minimum.
• Through all tests, verify that your app’s performance is acceptable. For example, verify that there is not too long a lag to update the UI after the app is resized.

Testing checklist

We tried these operations in both standard and multi-window mode to verify app’s performance.

• Enter and leave multi-window mode.
• Switch from your app to another app, and verify that the app behaves properly while it is visible but not active. For example, if your app is playing video, verify that the video continues to play while the user is interacting with another app.
• In multi-window mode, try moving the dividing bar to make your app both larger and smaller. Try these operations in both side-by-side and one-above-the-other configurations. Verify that the app does not crash, essential functionality is visible, and the resize operation doesn’t take too long.
• Perform several resize operations in rapid succession. Verify that your app doesn’t crash or leak memory.
• Use your app normally in a number of different window configurations, and verify that the app behaves properly. Verify that text is readable, and that UI elements aren’t too small to interact with.

More info about this here: https://developer.android.com/preview/features/multi-window.html

9. TransactionTooLargeException

If you are targeting Android Nougat, you need to be careful not to store more than 1MB (in theory, less in practice) in Binder transaction.

For example, saving data in onSaveInstance when moving between activities.

Technically the 1Mb limit is shared by all transactions in progress for the process so you might hit the limit way before that.

So to sum it up, make sure to stress test your app on a variety of devices before releasing it to the wild.

If you liked this article make sure to 👏 it, and follow me on Twitter!