Android Code Coverage on Firebase Test Lab (Part 1: The Basics)

Photo by Mikail McVerry on Unsplash

This series of articles will explain how to generate a code coverage report for Android instrumentation tests running on Firebase Test Lab.

Final goal

We’ll build up a step at a time, with a final goal of an implementation that will

• Produce XML & HTML reports for code coverage of on-device tests
• Run on API 28, API 29, and API 30 devices in Firebase Test Lab
• Support Android applications targeting API 28, API 29, and API 30
• Support Android Test Orchestrator
• Combine results with off-device unit tests
• Support multi-module applications
• Integrate with flank
• Use scripts to run in a CI/CD pipeline

Requirements for Part 1

Later articles will build up to satisfy all the requirements mentioned, but for now we’ll concentrate on these requirements:

• Produce XML & HTML reports for test coverage
• Run on API 28, API 29, and API 30 devices in Firebase Test Lab
• Support Android applications targeting API 28, API 29, and API 30

Getting started with API 28

We’ll start with an extremely simple API 28 application, created by starting a new Android project in Android Studio Arctic Fox without any activities, and then editing the project-level build.gradle to target API 28. We’ll add a Java class and a Kotlin class, each with two methods, and add an instrumentation test that will call one method on each class.

Building and running

You can easily build this with

./gradlew clean assembleDebug assembleDebugAndroidTest

This article assumes that you already have Firebase Test Lab set up for command line execution, if not then follow the directions at https://firebase.google.com/docs/test-lab/android/command-line

Once you’re ready to use the Google Cloud command line tool, it’s a simple matter of running the right command. We’ll base this on the example at https://firebase.google.com/docs/test-lab/android/command-line#running_your_instrumentation_tests but tweak the parameters to write the coverage file to /sdcard/Download/coverage.ec instead of /sdcard/coverage.ec, for reasons that will become clear later.

gcloud firebase test android run \
--type instrumentation \
--no-performance-metrics \
--no-record-video \
--app app/build/outputs/apk/debug/app-debug.apk \
--test app/build/outputs/apk/androidTest/debug/app-debug-androidTest.apk \
--device model=Pixel2,version=28,locale=en,orientation=portrait \
--environment-variables coverage=true,coverageFile=/sdcard/Download/coverage.ec \
--directories-to-pull /sdcard/Download

If you watch the output, you’ll see a line that looks like

Raw results will be stored in your GCS bucket at [https://console.developers.google.com/storage/browser/test-lab-<project-id>/<timestamp>/]

If you follow the link you’ll come to a page that looks like

And if you click the link for the device we used (Pixel2–28-en-portrait) you’ll find yourself on a page that looks like

And then if you click the link for instrumentation.results you’ll come to a page where you can download the contents of the file. It should look something like

And if you click the “download” link and scroll to the bottom, you’ll see the results of our attempt to generate a code coverage report…

Error: Failed to generate Emma/JaCoCo coverage. Is Emma/JaCoCo jar on classpath?

Oh.

Enabling code coverage

Looks like we haven’t enabled code coverage on our build. We’ll need to turn this on by adding testCoverageEnabled within the module-level build.gradle. However, there is some performance impact of building with code coverage enabled, so we’ll gate it behind an optional flag.

buildTypes {
    debug {
        testCoverageEnabled (project.hasProperty('coverage'))
    }
}

We can then do a clean build by passing the -Pcoverage parameter like this:

./gradlew -Pcoverage clean assembleDebug assembleDebugAndroidTest

After building and running the gcloud command again, once we open instrumentation.results again, we see

Error: Failed to generate Emma/JaCoCo coverage. 

We’ll need to open logcat to see exactly what’s going on, so we’ll click on the link for logcat in the same place we found the instrumentation.results link, click download, and search through the logcat spew for “jacoco”. There we find

E/CoverageListener(6620): Failed to generate Emma/JaCoCo coverage. 
E/CoverageListener(6620): java.lang.reflect.InvocationTargetException
E/CoverageListener(6620): 	at java.lang.reflect.Method.invoke(Native Method)
E/CoverageListener(6620): 	at androidx.test.internal.runner.listener.CoverageListener.generateCoverageReport(CoverageListener.java:101)
E/CoverageListener(6620): 	at androidx.test.internal.runner.listener.CoverageListener.instrumentationRunFinished(CoverageListener.java:70)
E/CoverageListener(6620): 	at androidx.test.internal.runner.TestExecutor.reportRunEnded(TestExecutor.java:92)
E/CoverageListener(6620): 	at androidx.test.internal.runner.TestExecutor.execute(TestExecutor.java:65)
E/CoverageListener(6620): 	at androidx.test.runner.AndroidJUnitRunner.onStart(AndroidJUnitRunner.java:395)
E/CoverageListener(6620): 	at android.app.Instrumentation$InstrumentationThread.run(Instrumentation.java:2145)
E/CoverageListener(6620): Caused by: java.io.FileNotFoundException: /sdcard/Download/coverage.ec (Permission denied)

The issue is that jacoco agent is trying to write the coverage data to /sdcard/Download/coverage.ec, but it doesn’t have permission to write external storage.

Granting permission

The solution is to grant the WRITE_EXTERNAL_STORAGE permission in AndroidManifest.xml. (we’ll create a new AndroidManifest.xml in app/src/debug/AndroidManifest.xml so that we’re not affecting the production version of our application)

One final time, we’ll run gcloud, and now instrumentation.results shows that we’ve succeeded!

Generated code coverage data to /sdcard/Download/coverage.ec

If we look back in Google Cloud Storage (in the same location we saw the instrumentation.results and logcat links) and open the artifacts folder, there is now an sdcard folder that contains a Downloadfolder that contains coverage.ec. Finally.

Supporting Android 10 (API 29)

To update our application to target API 29, we just need to change the project-level build.gradle’s compileSdk and targetSdk settings.

android {
    compileSdk 29
    defaultConfig {
        targetSdk 29
        ...
    }
    ...
}

Running the same gcloud command as before, we see that we work just fine on an API 28 device. coverage.ec is generated and copied to the artifacts/sdcard/Download folder, just like before.

But we also want to check that we can run on an API 29 device in Firebase Test Lab. So we’ll change the “version” portion of the --device parameter:

gcloud firebase test android run \
--type instrumentation \
--no-performance-metrics \
--no-record-video \
--app app/build/outputs/apk/debug/app-debug.apk \
--test app/build/outputs/apk/androidTest/debug/app-debug-androidTest.apk \
--device model=Pixel2,version=29,locale=en,orientation=portrait \
--environment-variables coverage=true,coverageFile=/sdcard/Download/coverage.ec \
--directories-to-pull /sdcard/Download

However, when we run the gcloud command now instrumentation.results reports

Error: Failed to generate Emma/JaCoCo coverage

and logcat contains

E/CoverageListener(10124): Failed to generate Emma/JaCoCo coverage. 
E/CoverageListener(10124): java.lang.reflect.InvocationTargetException
E/CoverageListener(10124): 	at java.lang.reflect.Method.invoke(Native Method)
E/CoverageListener(10124): 	at androidx.test.internal.runner.listener.CoverageListener.generateCoverageReport(CoverageListener.java:101)
E/CoverageListener(10124): 	at androidx.test.internal.runner.listener.CoverageListener.instrumentationRunFinished(CoverageListener.java:70)
E/CoverageListener(10124): 	at androidx.test.internal.runner.TestExecutor.reportRunEnded(TestExecutor.java:92)
E/CoverageListener(10124): 	at androidx.test.internal.runner.TestExecutor.execute(TestExecutor.java:65)
E/CoverageListener(10124): 	at androidx.test.runner.AndroidJUnitRunner.onStart(AndroidJUnitRunner.java:395)
E/CoverageListener(10124): 	at android.app.Instrumentation$InstrumentationThread.run(Instrumentation.java:2189)
E/CoverageListener(10124): Caused by: java.io.FileNotFoundException: /sdcard/Download/coverage.ec: open failed: EACCES (Permission denied)

The problem is that API 29 introduces new restrictions on where applications can write to. (“scoped storage”) Luckily, we can easily opt out by adding requestLegacyExternalStorage to app/src/debug/AndroidManifest.xml.

Running the gcloud command again, coverage.ec is generated and copied to the artifacts/sdcard/Download folder, as we saw before running on an API 28 device.

Supporting Android 11 (API 30)

To update our application to target API 30, we again change the project-level build.gradle’s compileSdk and targetSdk settings.

android {
    compileSdk 30

    defaultConfig {
        targetSdk 30
        ...
    }
    ...
}

Running against API 28 and API 29 devices in Firebase Test Lab (by modifying the version portion of the --device parameter of our gcloud command) works just fine.

And amazingly, even when we run against API 30 devices, everything just works, even though neither WRITE_EXTERNAL_STORAGE nor requestLegacyExternalStorage are supported on Android 11. What’s going on?

This is where the switch to /sdcard/Downloadpays off; on Android 11 devices /sdcard/Download is accessible to everyone. (see https://medium.com/androiddevelopers/scope-storage-myths-ca6a97d7ff37)

And while we don’t strictly need to make any more changes, for the purposes of good code hygiene we shouldn’t request WRITE_EXTERNAL_STORAGE permission on Android 11 devices, since that permission doesn’t do anything there. We can add maxSdkVersion to the uses-permission block, like this:

Excellent, so now we can collect code coverage numbers on test devices whether they’re running Android Pie (API 28), Android 10 (API 29) or Android 11. (API 30)

Now let’s move on to actually using the coverage data.

Downloading the .ec file

The gsutil utility allows us to easily copy files from gs:// links, and we can use pieces of the GCS results bucket’s URL to compute that link.

For example, if the URL for the GCS results bucket is

https://console.developers.google.com/storage/browser/<project-id>/<timestamp>

then the gs:// link for the coverage.ec file (for our chosen device) is

gs://<project-id>/<timestamp>/Pixel2-30-en-portrait/artifacts/sdcard/Download/coverage.ec

And we can download it by issuing the gsutil command. We’ll put it into app/build/outputs/code_coverage for now.

mkdir app/build/outputs/code_coverage
gsutil cp gs://<project-id>/<timestamp>/Pixel2-30-en-portrait/artifacts/sdcard/Download/coverage.ec app/build/outputs/code_coverage

Building reports from external .ec files

The .ec file is not fit for human consumption, however. Since our goal is to build XML and HTML reports to view the coverage information, we’ll need to use JaCoCo’s report generating activities. Unlike the support for instrumenting binaries (which is built into gradle as long as you pass testCodeCoverage), if we want to generate reports for arbitrary .ec files obtained externally, we’ll need to explicitly define a dependency on JaCoCo in our project-level build.gradle:

In our module-level build.gradle, we’ll also need to add a dependency on the JaCoCo plugin as well as add a custom task to build the report.

Once all this is in place, we can generate the report by calling

./gradlew jacocoReport

and then can open the report found at app/build/reports/jacoco/jacocoReport/html/index.html

TL;DR

And at this point we’ve achieved the requirements we targeted for this article. Looking back, these were the required steps to get here:

1. Set testCoverageEnabledto true for debug builds to enable code coverage (but only when -Pcoverage is passed)
2. Add the WRITE_EXTERNAL_STORAGE permission for debug builds (with the appropriate maxSdkVersion) to allow coverage.ec to be written on pre-Android 11 devices.
3. Add requestLegacyExternalStorage for debug builds to allow coverage.ec to be written on Android 10 devices.
4. Use gsutil to download coverage.ec from the results bucket
5. Add a dependency on jacoco to the project-level and app-level build.gradle files.
6. Add a new gradle task to generate reports from arbitrary .ec files.

You can see a solution that combines all of these changes by looking at https://github.com/Aidan128/FirebaseTestLabCoverageExample and checking out the git tag part_one.

Next time

This article achieved the basic requirements that we targeted for this article, but we still have a long way to go. In Part Two we’ll look at Android Test Orchestrator, multi-module projects, and combining Firebase Test Lab results with off-device unit tests, and then in Part Three we’ll look at using Flank and scripting to make this even easier.

Special thanks

Many thanks to the Firebase engineers in the #testlab Slack channel. (https://firebase.community/) Without them I never would have solved the permissions issues that came up in Android 11.

In addition, a number of people have written on this topic in the past, and their articles helped me a lot in my investigations. I thank them.

Generating code coverage from espresso tests running in Firebase test lab

Unified Code Coverage for Android: Revisited

The hidden cost of code coverage