Going Deep on JetPack Benchmark

Performance has been an important part to ensure smooth user experience, and Android Team had released the first Jetpack Benchmark library at I/O 2019 to help doing performance monitoring in Android. For introduction and usage, please refer to android docs. In this post, I will give a deep dive into what benchmark library has done to improve the result accuracy.

Minimizing Interference

The library will launch a simple opaque activity used to reduce interference from other windows.

For example, sources of potential interference:

• live wallpaper rendering
• homescreen widget updates
• hotword detection
• status bar repaints
• running in background (some cores may be foreground-app only)

Launching opaque activity is done by AndroidBenchmarkRunner

before every test start and after every test finish, an IsolationActivity will be launched if not launched before.

Clock Stability

CPU frequency of the mobile device will vary according to the temperature of the device which will causing an unstable benchmark result. So, the CPU frequency fluctuation should be minimized

Lock Clock

Lock clock require adb root permission, so it’s not applicable to every device. If you have a root device, you can run gradlew lockClocks. lockClocks.sh will be pushed into /data/local/tmp/lockClocks.sh and being executed.

the script will choose a propriate frequency and set it to sys/devices/system/cpu/cpux/cpufreq/scaling_maxfreq and sys/devices/system/cpu/cpux/cpufreq/scaling_minfreq so that CPU can run at a stable speed

# scaling_max_freq must be set before scaling_min_freq
echo ${chosenFreq} > ${freq}/scaling_max_freq
echo ${chosenFreq} > ${freq}/scaling_min_freq
echo ${chosenFreq} > ${freq}/scaling_setspeed

Sustained performance mode

For long-running applications (games, camera, RenderScript, audio processing), performance can vary dramatically as device temperature limits are reached and system on chip (SoC) engines are throttled. To address these limitations, Android 7.0 introduced support for sustained performance. In this mode, the CPU will run in a more moderate speed so that the device will not heat up and performance can keep in a stable status for a long time.

In IsolationActivity, benchmark library will try to turn on the sustained performance mode on every activity launched if CPU not locked and sustained performance supported by device.

besides, it will launch a continuously spinning Thread to get into multi-threaded sustained performance mode.

Together, these ensure that the app runs in a stable performance mode, which locks the maximum clock frequency to prevent thermal throttling.

Thread Priority

before test start, the benchmark thread will be set to the highest thread priority, and JIT thread priority will be set to lower than benchmark thread, to reduce the interference of the JIT.

Warmup, Iteration, Repeat

Usually, we write our benchmark code like below

BenchmarkRule.getState will return the BenchmarkState in BenchmarkRule

in BenchmarkState.keepRunning(), the BenchmarkState will go through Warmup → Running → Finished state. keepRunning() return true until the state reached Finished.

three states for BenchmarkState

In warmup state, the code under benchmark will be run multiple times, and when slow Exponential moving average approach the fast Exponential moving average, the warmup state will transfer to running state

In running state, there will be several measurement repeats, inside every repeat, there will be several iterations, only the average of the iterations will be recorded. So, the results will be [average of repeat1, average of repeat2…. average of repeatN].

In finished state, the keepRunning will return false to quit the loop. The statistics of all the measurement repeats will be sent to Android studio console and append to the json file result.

Note that the result shown in Android Studio console is the min of the several repeats, for more statistics, please refer to the output json file.

Dry run mode, Startup mode, Normal mode

There are three modes that benchmark library support to test code in different scenarios.

• Dry run mode: no warmup, running each benchmark loop only once, to check for errors/crashes without capturing measurements.

defaultConfig {
    testInstrumentationRunnerArgument 'androidx.benchmark.dryRunMode.enable', 'true'
}

• Startup mode: no warmup, 10 repeat measurements, 1 iteration per repeat

defaultConfig {
    testInstrumentationRunnerArgument 'androidx.benchmark.startupMode.enable', 'true'
}

• Normal mode: warmup, 50 repeat measurements, iterations of every repeat determined by warmup result

.--------------.--------.--------.----------------------.
|     Mode     | Warmup | Repeat | Iteration Per Repeat |
:--------------+--------+--------+----------------------:
| Dry run mode | no     |      1 | 1                    |
:--------------+--------+--------+----------------------:
| Startup mode | no     |     10 | 1                    |
:--------------+--------+--------+----------------------:
| Normal mode  | yes    |     50 | determined by warmup |
'--------------'--------'--------'----------------------'

Conclusion

For now, hope you can understand what Jetpack Benchmark has done to ensure that we have an accurate benchmark result.

• to prevent interference from other windows
• to lock the CPU performance into a consistent state during benchmark
• to prevent thread scheduling from other thread.
• using warmup and loop average to ensure a more stable result

you can explore the source code of the benchmark library to gain more detail, thanks for reading!