A Story about FFmpeg on Android. Part II: Integration.

The compilation of FFmpeg is only a half of a story about using it on Android. The next thing we need to do is to actually integrate those compiled libraries into an Android project and make some use of them.

The other parts are available here:

A Story about FFmpeg on Android. Part I: Compilation.

A Story about FFmpeg on Android. Part III: Extension

In this article I would like to cover shared libraries integration, actual data reading from FFmpeg and Android-specific issues that I faced during the development.

Shared libraries integration

In order to execute a function inside a shared library from the JVM part (or vice versa) we need a Java Native Interface (JNI) layer. That mechanism consists of 2 things:

• A JVM class with a method specially marked: with native keyword in Java or external in Kotlin.
• A C function with certain naming and parameters to match the aforementioned JVM method.

The JVM part for our project resides in VideoFileConfig class:

Several methods and properties are marked with external. The actual implementation resides in the C part. The nativePointer property is actually an address of a C struct, casted to the Long type. It is managed by the C part.

In the companion object we have a list of shared libraries to import with System.loadLibrary(). That call binds external methods to their implementations in C at runtime. Note that the order of libraries to load is important, at least for Android API 16 and 17. Rule of thumb: a dependant library has to be loaded after all its dependencies. See the link for details.

One more important thing here: you have to carefully set Proguard rules for such a JVM class. For JNI a name for a C function has to match the JMV method name, so obfuscation process should skip those JVM methods. The nativePointer property also has to be left as is, because it is accessed from the C part in a reflection-style API.

An example of the C counterpart for the nativeNewFD(fd: Int) looks like this:

JNIEXPORT void JNICALL Java_com_javernaut_whatthecodec_VideoFileConfig_nativeNewFD
(JNIEnv *env, jobject instance, jint jFileDescriptor) {    
    ...
}

The name of such a function has ‘Java’ prefix, full JVM class and the target method names separated by underscores. jobject instance is a JVM object whose external method was called and jint jFileDesсriptor is the parameter that was passed. So the C part gets all necessary data from the JVM part. For more details about syntax of C functions in JNI please refer to this link.

How do all those shared libraries come to an APK in the first place?

Let’s start with FFmpeg’s libraries. The first important thing to understand is when you use a particular version of the FFmpeg (or a particular commit) then the output shared libraries and header files remain the same (assuming your configuration arguments are also the same). That means there is no point in spending time reassembling the library multiple times, as the result will be the same. For that reason I compile the FFmpeg only once on a machine by executing the ffmpeg-android-maker’s script.

We can go even further and add compiled binaries to version control system. Thus we can free other developers and CI from NDK installing and FFmpeg compilation process completely. I skipped this step, because those arguments in ffmpeg-android-maker were changing with time.

build.gradle

After shared libraries are compiled we need to tell Android build system to package them with the app. In my case I just added ffmpeg-android-maker as a git submodule to the root of the Android project. And thus the app’s module build.gradle file looks like this:

In externalNativeBuild block we define a path to a CMake configuration file that describes how the JNI layer should be built. Note that compiling your JNI library against another *.so library doesn’t package that *.so library with the app. This is why we have to specify all FFmpeg’s shared libraries using sourceSets block. The ndk.abiFilters restricts ABIs to build for the JNI layer. This is needed because MIPS processors are not supported by Android NDK anymore and ffmpeg-android-maker doesn’t produce binaries for them.

CMakeLists.txt

Let’s have a look at the CMake configuration file:

Here we do several things:

• Specify a directory with FFmpeg’s header files with include_directories();
• Import FFmpeg’s shared libraries with add_library() and set_target_properties();
• Define a shared library for our JNI layer and specify all *.cpp files for it (also with add_library());
• Link the JNI layer library against FFmpeg’s shared libraries, Android logging library and Bitmap processing library with target_link_libraries().

So now we have the Android project configured to build the JNI layer and package all necessary shared libraries to the output APK.

Work with FFmpeg. Finally.

We have all things set up and we are ready to actually use the FFmpeg functionality.

The FFmpeg is a C library and its whole API is defined in header files. They also have valuable documentation.

Meta info

Remember libavformat library (a part of FFmpeg)? This library works with media files or other media sources in general. Its main data type is AVFormatContext struct defined in libavformat/avformat.h. The struct can be created in different ways, but the simplest one is to use avformat_open_input() function. This functions accepts a url to a media source:

#include <libavformat/avformat.h>
...
AVFormatContext *avFormatContext = nullptr;

if (avformat_open_input(&avFormatContext, url, nullptr, nullptr)) {
    // Error handling here
    return nullptr;
}
// Here we can work with opened avFormatContext.
// The video file format name can be accessed like this:
avFormatContext->iformat->long_name;

We have found the first piece of information that our app should display to a user. The rest is a video stream-specific. In order to find such a media stream inside the AVFormatContext we need to do such things:

#include <libavcodec/avcodec.h>
...
if (avformat_find_stream_info(avFormatContext, nullptr) < 0) {
    avformat_free_context(avFormatContext);
    return nullptr;
}

for (int pos = 0; pos < avFormatContext->nb_streams; pos++) {
    AVCodecParameters *parameters = 
        avFormatContext->streams[pos]->codecpar;
    // Getting the very first video stream    
    if (parameters->codec_type == AVMEDIA_TYPE_VIDEO) {
        AVCodec *avVideoCodec = 
            avcodec_find_decoder(parameters->codec_id);
        int videoStreamIndex = pos;
        // Saving parameters, avVideoCodec and videoStreamIndex
        break;
    }
}

First we call avformat_find_stream_info() function that initializes certain streams-specific fields of AVFormatContext. Then we can iterate through all media streams to find a video stream and its AVCodec. This functionality comes from libavcodec. And now we have access to the rest of the meta info that is shown to a user:

// Video codec name
avVideoCodec->long_name;
// Video frame metrics
parameters->width;
parameters->height;

I created a wrapper struct VideoConfig to hold all this FFmpeg’s objects as they will be used multiple times for the same video file. The pointer of this struct is saved to JVM’s VideoFileConfig.nativePointer.

Displaying a frame

We can read a frame from a video stream in a form of a bitmap. Android app has to provide a surface for drawing such a bitmap. In order to keep the C part simple the JVM part passes a preallocated Bitmap object of a desired size and the native code only fills that object with frame’s data. Such approach is ok if you want to display static images.

The Android Bitmap object has to have a specific pixel format: ARGB_8888. But a frame from a video file can have an arbitrary pixel format. Also the frame’s size can differ from the Bitmap’s size. The libswscale solves both problems by scaling the frame and changing the pixel format.

Let’s have a look at the actual implementation:

This listing is long but simple. It does several things:

• Makes an AndroidBitmapInfo object to access jBitmap’s byte buffer;
• Gets a VideoConfig struct saved in jVideoConfig;
• Creates a AVCodecContext object that is capable of transforming data stored in a video stream to actual frames;
• Sets up a SwsContext object with sizes and pixel formats both for a source frame and a target bitmap;
• The AVPacket object is used to keep a read chunk of data from a media stream. That data is decoded by AVCodecContext and is sent to a AVFrame;
• Frames may consist of multiple packets. We have to check the result of avcodec_receive_frame() for AVERROR(EAGAIN) value and read and process one more packet if so;
• Then another AVFrame is created and is set up with a byte buffer from the AndroidBitmapInfo with av_image_fill_arrays() function. It is a destination where we need to send data from the frame read from the video stream;
• The sws_scale() function scales, changes the pixel format and sends the read frame to the destination frame. And thus the jBitmap receives actual data to display;
• Don’t forget to clean up the allocated stuff, there is no GC behind the JNI;
• Errors handling is omitted for simplicity. For a more robust approach please refer to this link.

Now we can access all the necessary data and display it to a user:

A radio group in UI presumes switching between number of frames shown and each time a number is selected the app has to read all frames from the beginning of the file again. Such seeking back to the beginning causes a problem in certain case.

To understand that problem we have to understand the way FFmpeg reads data from a media source in a context of Android OS.

Supported protocols for reading data

The first one is well known file protocol. It represents an address of a file in a file system and looks like file:///path/to/your/file.mp4. FFmpeg works well with this protocol because it supports seeking to an arbitrary position in any direction (forward and backward).

But in recent years Android adds more and more restrictions to file protocol using. In Android 7 (API 24) apps no longer may share file:// URIs. So when your app gets a URI from another app most likely it will be content://, but FFmpeg doesn’t support it, as it is Android-specific. Android 10 brings Scoped Storage thing which restricts file access even more.

The replacement Android offers for developers is opening a file descriptor for a provided content:// URI. Is there a way to pass it to FFmpeg?

FFmpeg supports pipe protocol. A url argument for avformat_open_input() function in this case looks like pipe:${file_descriptor}. The protocol works in general, but has one major disadvantage: it doesn’t support seeking backward. So when a media resource is accessed this way we can’t return back to the beginning and read data from it. Seeking forward is supported however, as it just skips read bytes. Also sometimes with certain file formats this protocol fails to read frames completely.

The pipe protocol suits purposes when data has to be read just one time like frame extracting or playing a video without rewinding backward.

The app I wrote tries to reconstruct a file address for a content URI it receives in a manner like this. This isn’t a new approach, but it helps a lot enabling frame number switching for the most of Android devices.

Other things good to know

First and obvious thing to do in the app is to add Runtime Permission support for READ_EXTERNAL_STORAGE. If you’re going to publish an app to Google Play nowdays it is a must have thing. Apart from file reading allowing this permission facilitates reconstructing a file address for a content URI.

Second, the app has 4 sets of native libraries: one for each processor type it supports. There is no need to embed all of them into a single APK, as only one of those sets will be used on a device. You can either generate 4 different APKs and upload all of them to Google Play, or use the App Bundle format. I have experience with both of this approaches and prefer exactly the App Bundle.

Update

There are several good questions (and answers) regarding the FFmpeg integration in the responses of my previous article, so have a look there.

Epiloge

I hope this story will help to understand basics of how the FFmpeg can be embedded into an Android app in a way you control completely.

The source code for the app is available here:

Javernaut/WhatTheCodec

The ffmpeg-android-maker script is available as a separate repository and is easy to integrate into any app. You are free to use it for your own affairs.

Javernaut/ffmpeg-android-maker

Thanks for reading and have a good time!

Cheers!