Android CameraX: Preview, Analyze, Capture.
Introduction
CameraX is an Android Jetpack library that was built with the intent to make camera development easier, which until now has been quite painful. In contrast with the fine grained control camera2's API offered, CameraX (which uses the Camera2 API under the hood) aims to strike a balance between abstracting away the difficult bits of managing the camera while allowing flexibility and customization.
Beside the API’s simplicity and ease of use, CameraX solves compatibility issues developers faced while accounting for device/manufacturer specific issues (Samsung and Motorola, I’m looking at you). And since it works on devices running Android Lollipop -API 21- or newer (90% of phones in the market), camera based apps should work more consistently across a wider range of devices, from entry-level phones to flagships, especially that Google has invested in an automated test lab with hundreds of devices in order to test things such as photo capture latency, startup/shutdown latency, orientation changes and image capturing against different camera hardware levels, Android API levels, etc.
CameraX structure
CameraX is a use case based API, it has abstracted 3 main handles which you can use to interact with the camera: Preview, Image analysis and Image capture. Choosing which use case(s) to use depends on what you intend to use the camera for.
- Preview: Provides a way to get the camera preview stream.
- Image analysis: Provides a way to process camera data (frames).
- Image capture: Provides a way to capture and save a photo.
As mentioned above, using the CameraX API is -more or less- simple, you’ll see how to use each of its use cases below.
Preview
Set up the Preview use case by configuring Preview.Builder, then building it using Preview.Builder.build().
Preview.Builder provides options to set either the target aspect ratio or resolution to be used (you can only set one of them at a time); depending on the camera’s capabilities, these options may or may not be respected, in the latter case the API will choose the closest available resolution possible. The rotation can also be configured, which should typically match the device’s orientation. When not configured, these options will fallback to their default values which depend on the camera’s capabilities.
The Preview use case needs a Surface to display the incoming preview frames it receives from the camera. You can provide a Surface by calling Preview.setSurfaceProvider(SurfaceProvider), the SurfaceProvider passes the preview Surface to be used by the camera. There are many challenges around handling the Surface, like making sure it’s valid while the camera’s using it, and providing a new one if it’s released prematurely, which is why it is recommended to use PreviewView, a custom View that manages the preview Surface, handles scaling, rotating and translating the preview frames to match the display, and can easily be attached to a Preview use case. Use PreviewView.createSurfaceProvider(CameraInfo), it creates a SurfaceProvider that you can pass to the Preview use case to start the preview stream.
SurfaceProvider to the Preview use case using a PreviewViewFor more advanced preview usages, like applying effects to the preview, consider using OpenGL and a SurfaceTexture to handle the preview Surface.
Image Analysis
Similar to the Preview use case, the ImageAnalysis use case is instantiated by configuring ImageAnalysis.Builder, then building it using ImageAnalysis.Builder.build().
In addition to the configuration parameters used in the Preview use case (resolution, aspect ratio and rotation), ImageAnalysis also accepts a back pressure strategy parameter, it specifies how the images passed in for analysis are selected. Two modes are possible, STRATEGY_KEEP_ONLY_LATEST and STRATEGY_BLOCK_PRODUCER. The former takes in the latest image from the image acquisition pipeline while disregarding any other older images, the latter takes in the next image in the pipeline.
Lastly, the depth queue specifies the number of images available in the analysis pipeline. Increasing it has an impact on the camera’s performance and memory usage.
- When
STRATEGY_KEEP_ONLY_LATESTis used, only the latest image in the pipeline is analyzed, so increasing the depth queue (making the queue bigger) will give the analyzer more time to analyze an image before stalling the pipeline. - When
STRATEGY_BLOCK_PRODUCERis used, an increase of the depth queue will help make the pipeline run smoother, especially when the device is under high load.
An Analyzer can be attached to an ImageAnalysis use case, it specifies what to do with the incoming images from the camera, it receives ImageProxy objects, which are wrappers around Android Image objects and contain information that can be processed for image analysis through its planes attribute which contain pixel data about the image. The analysis should be handled fast enough so as not to stall the image acquisition pipeline, or the image data should be copied elsewhere for longer processing. Make sure to close each received image, failing to do so will throttle the analysis pipeline.
Image capture
The end goal for most camera usages is to take a photo, this is the role of the ImageCapture use case. It follows the same pattern as the 2 use cases above.
ImageCapture.Builder allows to set the flash mode to be used when taking a photo, it can be one of the following: FLASH_MODE_ON, FLASH_MODE_OFF or FLASH_MODE_AUTO. It also takes a capture mode, which can be either CAPTURE_MODE_MINIMIZE_LATENCY to minimize image capture latency, or CAPTURE_MODE_MAXIMIZE_QUALITY to capture better quality images (which can be at the expense of latency).
ImageCapture provides 2 ways to capture an image, you can either receive an in-memory captured image, or you can choose to store the image in a file.
To get an in-memory captured image, use ImageCapture.takePicture(executor, OnImageCapturedCallback), if the image is successfully captured, onCaptureSuccess() is called with an ImageProxy that wraps the capture image. Make sure to close it before returning from the method. If the image capture fails, onError() is invoked with the type of the error. Both callbacks are run in the passed in Executor, if they need to run on the main thread (e.g. If you display the captured image in an ImageView, or display the error in a Toast), make sure to pass in the main thread Executor.
To save the captured image to a file, use ImageCapture.takePicture(outputFileOptions, executor, OnImageSavedCallback). The OutputFileOptions define where to store the captured image and what metadata to save with it. ImageCapture can write the captured image to a File, an OutputStream or MediaStore. OnImageSavedCallback contains 2 callbacks: onImageSaved() and onError(), they are both run in the passed inExecutor.
If you’re saving the captured image to the device’s external storage, make sure to specify the WRITE_EXTERNAL_STORAGE permission in your manifest, and to request it at runtime on Android APIs 23 and higher.
Lifecycle binding
While building instances of the use cases is the biggest part of what’s needed to get your app up and running, there’s one more step that’s required: Binding these use cases to a Lifecycle. This step takes the burden off the developer to start and shutdown the camera resources in the right order in the correct lifecycle callbacks. CameraX now takes care of that for you.
When the Lifecycle becomes active, the preview comes up, the ImageAnalysis Analyzer begins to receive images at the camera frame-rate and you can take pictures with ImageCapture.
Extra: Camera permission
Using a device’s camera requires requesting the camera permission from the user, CameraX does not handle this, so it’s up to you to request it at runtime before interacting with the API. Also, If you store captured images to the device’s external storage, make sure to request the permission to write to the external storage.
Add the following permissions in your manifest.
Then request the required permissions at runtime from your activity or fragment. Runtime permissions are only required on Android APIs 23 and above.
Once the user grants the required permissions, you’ll be able to begin using the CameraX API.
Conclusion
Having personally experienced the challenges of working with the camera2 API on Android, I definitely do appreciate the CameraX API. It provides the opportunity to focus more on what to build with and around the camera, which is great. In addition, assuming the automated test lab continues to test CameraX on a multitude of devices, the need for manual testing will become less of a necessity, and dealing with certain camera bugs that occur on only specific devices will be less of a problem.
It’ll be interesting to see how the CameraX API evolves and its adoption rate among popular camera based apps. It’s moved out of Alpha for some time, and video capture support isn’t public yet (a VideoCapture use case is available, but its usage is currently restricted).
