A Story about FFmpeg on Android. Part I: Compilation

One way or another, many of us know FFmpeg. It’s a great tool for video/audio processing. However, it was never trivial to make it work on Android. Compilation, Android NDK and OS restrictions are a mad brew and I want to tell you a story of cooking FFmpeg for Android today. I dedicate this story to all man-hours spent by all people trying to accomplish the same goal.

The other parts are available here:

A Story about FFmpeg on Android. Part II: Integration.

A Story about FFmpeg on Android. Part III: Extension

Let’s define a task first

Imagine we have to make an app that shows basic info about a video file: container and video codec info and a frame size. Also we would like to actually display a frame from the video. We could show info about audio and subtitles too, but let’s keep focus on video only.

How can we solve it with FFmpeg? We have two options:

1. Use already existed Java library that wraps prebuilt FFmpeg binaries. This is a good option if you have time constraint, because everything is ready to be used by your Java/Kotlin codebase. And you don’t have to deal with all this compilation stuff.
   But this simplicity comes with a price: you don’t control the content of that library, you just accept whatever is or isn’t prebuilt. Moreover, such libraries may even miss 64-bit support which is mandatory nowadays.
2. You can build FFmpeg yourself. Thus you will be able to package exactly stuff you need and to be sure you support all necessary ABIs. You can also add external libraries according to your needs. That way also simplifies migration to newer versions of FFmpeg.
   And there is a price here too: this way is more time consuming, because you have to compile the FFmpeg on your own and bind its native calls to your JVM part.

I’d like to create a smallest possible Android app that fulfills my task now and is prepared to possible additional goals in the future. This means I need enough control over FFmpeg building process. With this in mind I choose the second option.

Time to start building

Now it’s worth mentioning what FFmpeg actually is. It is a set of C libraries:

• libavformat works with file containers, can read and write media streams;
• libavcodec is responsible for media streams decoding and encoding;
• libswscale efficiently scales images;
• libavutil contains utility functions for all other libraries.

There are other libraries but they are not needed for our purposes.

Also there are 3 command-line tools: ffmpeg, ffprobe and ffplay which work as a facade for all aforementioned libraries. You can actually compile them too and execute them on Android in Runtime.getRuntime().exec(...) manner. But that approach doesn’t suit us.

What else does FFmpeg have? The way of compiling itself. There is a configure shell script which is a part of a configure && make && make install approach of building software. It resides in the root of FFmpeg’s source code. It checks available build tools and prepares internal modules for compilation. Despite it is a shell script, it also can be executed on Windows. Stay tuned.

The FFmpeg itself is HIGHLY configurable. That configure script can accept a lot of different arguments that affect the output of the whole building process. And the most hard part of making FFmpeg work on Android is to pass proper arguments.

So… What do we need to pass to that script? That depends on the version of FFmpeg. Fortunately, with time FFmpeg has become more Android-friendly and the configuration becomes simpler. This article is based on FFmpeg 4.2. That version doesn’t require any source code changes in order to be compilable.

Let’s divide this into two parts:

1. Build configuration

FFmpeg consists of C and assembler code. In order to compile the sources we need a compiler. Here comes the first complication: we actually need to produce binaries for 4 types of processors: ARM and x86 both 32-bit and 64-bit versions. We don’t consider mips anymore, as its support was deleted from Android NDK completely. Assuming our workstation is based on x86 only, we need a cross-compiler - the one that can produce binaries for other types of processors. You also need a linker and some other tools (this subset of tools is called binutils) for building process. Final component - a sysroot directory. It represents the root directory of Android OS with essential libraries and header files. With certain simplification, we can say that compiler + binutils + sysroot = toolchain.

So where do we get a toolchain? It is inside the Android NDK - a development kit that allows native applications development for Android. The version of NDK is also important, because toolchains structures were not permanent previously. Recently GCC compiler was removed (as of r18), so now we have only Clang. Also parts of a single toolchain could be split across different directories inside the NDK. For example, headers and libraries of sysroot could reside in completely different locations. As of NDK r19 all parts of a toolchain are finally structured conveniently to be passed to FFmpeg’s configure script.

Let’s look through the specific example line by line:

--prefix=${BUILD_DIR}/${ABI} – the path to place the output header files and binaries. They will be placed in include and lib directories respectively. There also can be a bin directory if you wish to build command line tools like ffprobe.
--target-os=android – our target OS is Android. The configure script does certain configuration for compilation for you, for example it adds -fPIE compiler flag.
--arch=${TARGET_TRIPLE_MACHINE_BINUTILS} – There will be 4 values in this variable: arm, aarch64, i686 and x86_64. The are 2 important things here: 1) those values are a part of other variables creation and 2) configure script supports aliases for the same architecture. For example, aarch64 and arm64 are treated identically by the script. For the whole list of aliases please refer to the configure script.
--sysroot=${SYSROOT} – the needed sysroot directory now resides directly inside toolchain’s directory: $TOOLCHAIN_PATH/sysroot. By the way, the toolchain path itself is $NDK_ROOT/toolchains/llvm/prebuilt/$HOST_TAG. The HOST_TAG depends on the OS for which the NDK was built.
--cross-prefix=${CROSS_PREFIX} – this is a way of telling all binutils tools to FFmpeg’s configure script at once. The prefix will be appended by ld to get a linker, for example. Actual values of prefixes are here:

armeabi-v7a: $TOOLCHAIN_PATH/bin/arm-linux-androideabi-
arm64-v8a:   $TOOLCHAIN_PATH/bin/aarch64-linux-android-
x86:         $TOOLCHAIN_PATH/bin/i686-linux-android-
x86_64:      $TOOLCHAIN_PATH/bin/x86_64-linux-android-

Just note that the last word for prefix for armeabi-v7a ABI is androideabi and not just android. And also note that this prefixes don’t depend on Android API level.

--cc=${CC} – the C compiler has to be passed slightly differently. Let me show you actual values first:

armeabi-v7a: $TOOLCHAIN_PATH/bin/armv7a-linux-androideabi16-clang
arm64-v8a:   $TOOLCHAIN_PATH/bin/aarch64-linux-android21-clang
x86:         $TOOLCHAIN_PATH/bin/i686-linux-android16-clang
x86_64:      $TOOLCHAIN_PATH/bin/x86_64-linux-android21-clang

1. Those are not prefixes, but paths to actual files. They all end with clang;
2. Those are not binaries, but shell scripts. They just redirect all arguments to $TOOLCHAIN_PATH/bin/clang binary and add a couple more. Just have a look inside one of those files. One more thing here: sets of additional arguments for 32-bit and 64-bit versions differ;
3. There is an Android API level in each file name. We just need to choose it according to minSdkVersion of our app. Minimum such version in NDK r19 is 16. For 64-bit versions it starts with 21, because Lollipop is the first Android version that got 64-bit support in the first place;
4. Note that for armeabi-v7a ABI the first words of cross-prefix and cc name are different: arm and armv7a.

--extra-cflags=”-O3 -fPIC” – here is how we can pass additional arguments for C compiler. The -O3 is the optimizaiton level. The -fPIC is needed if we are building shared objects (*.so libraries) for Android.
--enable-shared and --disable-static – and here we are actually telling the configure script that we are interested in shared libraries only, not static ones.
How can we choose between static and shared libraries? Long story short: shared libraries are used as separate files. So they actually can be imported one by one and even only a subset that is actually needed right now. That can decrease memory pressure. This is just a tiny optimization. Static libraries have to be embedded into other shared libraries or executables. This way you will need to import only one large binary (if you merge everything together) and the aforementioned optimization isn’t just possible. Anyway, in our case we will import all of them at once, but I still prefer shared objects.
${EXTRA_BUILD_CONFIGURATION_FLAGS} – Additional flags that are ABI-specific:
- x86_64 adds --x86asmexe=${TOOLCHAIN_PATH}/bin/yasm to specify the assembler compiler that resides inside the toochain’s directory.
- x86 instead of specifying and assembler compiler it disables assembler optimizations completely with --disable-asm, because otherwise the result binaries will have Text Relocations which are forbidden in Android as of Marshmallow. That will cause certain performance hit. This issue will not be solved on FFmpeg’s side, see the link for details. So in the end we have to make this sacrifice.

And this is it. All mandatory things for compilation are passed and all components of ffmpeg will be compiled with make && make install command. Just give your machine some time.

2. Content configuration

As I mentioned before, FFmpeg is highly configurable and you can cut off certain parts that you don’t need for your purpose. Let me show your my version of additional flags to configure script to have binaries smallest possible and which still fulfill my task:

For the full list of possible flags just have a look inside the configure script itself to see what is available to disable and what is enabled by default.

To highlight certain parts of that listing:

• My app will only read data from video files. So I need only demuxers and decoders, not muxers and encoders. Check out this link to become more familiar with terminology;
• The app will work with video decoders only, so I prepared a specific list of video decoders that are available in FFmpeg out of the box, then disabled all decoders and enabled only video-specific ones. Thus audio and subtitles decoders aren’t packaged. The DECODERS_TO_ENABLE variable is actually a sequence of --enable-decoder= $video_decoder_name.
• There is an --enable-small flag that produces even smaller binaries, but I had to omit it because it deletes names of video codecs and containers and my app actually wants to display this info.
• The rest is just unnecessary for me parts of FFmpeg like networking, media stream filters or command line interfaces.

I want to emphasize that this content configuration is specific to my application. Just review your requirements and make your own configuration. This part is necessary if your want to have smallest possible binaries.

Let’s sum everything up

I’ve prepared a repository with a single script that does all aforementioned things and even more:

• It downloads FFmpeg’s source code for you;
• It checks result binaries for Text Relocations;
• It integrates Travis CI to execute the script in a cloud;
• It can be executed on macOS, Linux and Windows;

Check it out!

Javernaut/ffmpeg-android-maker

That was a story about how FFmpeg can be compiled for Android today. With its latest for now version 4.2 and NDK r19 it is much easier than it was before.

Next time I’ll tell you another story of how the output of this script can be integrated into an Android app and how to use it in order to achieve my goal.

Cheers!