A Story about FFmpeg on Android. Part III: Extension

Oleksandr Berezhnyi
ProAndroidDev
Published in
9 min readJan 3, 2020

--

FFmpeg is a great tool with many features available right out-of-the-box. On top of that it supports integration of many external libraries that provide even more features. And they are available through the single FFmpeg API. In this part, I’d like to show the concept of integration of an external library as well as several specific examples.

This post is a continuation of previous parts:

What is the goal for now?

Just to remind you, in previous articles I created an app that decodes video frames from a video file. So I’m interested in video decoders. AV1 codec gets more popularity and even is supported by Android 10. FFmpeg supports this codec through external libraries:

  • libaom – a reference implementation of AV1 specification that supports both encoding and decoding. It is supported starting with FFmpeg 4.0;
  • libdav1d – a separate library that implements only a decoder of AV1 and claims to be faster than libaom. It is supported starting with FFmpeg 4.2;
  • librav1e – also a separate library that implements only an encoder of AV1 with the same claim. Its support currently is in master branch of FFmpeg and will be released in FFmpeg 4.3. I’m not interested in this library because it can only encode AV1.

I’d like to integrate both libaom and libdav1d into FFmpeg and explain the principals of such an integration. Additionally I’ll integrate the libmp3lame (an MP3 encoding library). It isn’t actually needed for the app, but its integration has some special features.

General approach to an external library integration

The list of libraries that can be integrated into FFmpeg can be found in the documentation or directly in the configure script. The latter is really a source of valuable information, so don’t hesitate to look inside of it. This list grows with time.

To integrate an external library into FFmpeg you need:

1. Compile the library

First you need to prepare a static or a shared library and place it in a specific directory structure along with header files. The standard directory structure is usually made by a build system of a library for you and it looks like this:

Here the x86 directory is a place where both libaom and libdav1d are installed. It is convenient to mix several libraries in the same directory structure. The include directory has header files that usually contains subdirectories per each specific library. The lib directory contains static (*.a) and/or shared (*.so) libraries.

You should prepare such directories as x86 for each ABI you want to support by compiling the library for each such ABI.

The way you trigger the compilation process heavily depends on a build system chosen by library developers. Below I’ll give you several examples.

An important thing to consider here is the type of library you wish to build: static or shared. FFmpeg supports both, yet I used only static versions.

  • Static: such libraries will be embedded into the final FFmpeg binaries. And all their functionality will be available only via FFmpeg’s API. It is the simplest option.
  • Shared: it is possible to build an external library as a shared library. This may keep the result APK file as small as possible in case this exact external library is used by something other than FFmpeg too. The whole purpose of being shared is to serve multiple other libraries and not get embedded into each one of them. Just don’t forget to package this shared library in the APK and load it in runtime.

Each library has a lot of flags that can be used to tune the result binary or to strip off parts that are not necessary for you. For example, I disable the encoder in libaom, which gives me ~60% smaller static library and faster compilation process.

2. Make it available to be picked up by FFmpeg building process

The lib/pkgconfig directory contains *.pc files. Those files contain meta-info about a library such as its name, version, but more importantly compiler and linker flags that are necessary to properly set this library as a dependency (shared libraries) or to embed it (static libraries). All this information can be extracted from *.pc by pkg-config tool. And FFmpeg building process does exactly this for most of its dependencies.

The only thing needed for FFmpeg to get all the info from those files is to know the path to pkgconfig directory. This can be done by setting PKG_CONFIG_LIBDIR environment variable to an appropriate path.

The most important part about *.pc files: not all libraries generate them. The libmp3lame is such an example. And thus it has to be set as a dependency manually:

  • --extra-cflags (compiler flags) argument of FFmpeg’s configure script should have paths to the include directory of each external library, prefixed by -I. For example:
    --extra-cflags="-I/path/to/compiled/lame/include".
  • --extra-ldflags (linker flags) argument of FFmpeg’s configure script should have paths to the lib directory of each external library, prefixed by -L. Also here you need to add all external libraries names prefixed by -l. For example:
    --extra-ldflags="-L/path/to/compiled/lame/lib -lmp3lame".

There can be a lot of -I, -L and -l flags. It is convenient to install all the compiled external libraries into the same directory structure, like x86. Thus it is possible to pass only one -I and one -L flag. The number of -l flags are not changed in this case.

And once again, if a library generates a *.pc file, you don’t need to worry about this stuff. However, refer to the FFmpeg’s configure script itself to find out what libraries are expected to be found by pkg-config and what are expected to be referenced via -extra-cflags and --extra-ldflags.

3. Add an appropriate flag to FFmpeg’s configure script

That is the easiest part. The FFmpeg’s configure script contains a list of actual flags that are needed to enable a certain external library. As an example, the --enable-libdav1d flag explicitly says that FFmpeg has to consider the compiled libdav1d library.

Beware that certain external libraries upgrade the license of FFmpeg from LGPL to GPL. An example of such library is x264.

Examples of build scripts of external libraries

Here and after all scripts are executed at the root of source code of a library.

libmp3lame (shell, make)

Let’s start with the simplest one.

The script itself resembles the way to build FFmpeg itself: configure && make && make install. It just has a different set of arguments. Let’s have a look at them:

  • --prefix - contains a path where a compiled library will be installed during make install command;
  • --host - since we are doing a cross compilation, here we pass the triplet that describes the host platform (where the compiled binary will be used). It depends on the ABI we are compiling the library for;
  • --with-sysroot - the sysroot directory. The same that we pass to FFmpeg’s configure script;
  • --disable-shared and --enable-static - here we say that we want to build only a static library;
  • --with-pic - build the position independent code. This is a way to eliminate text relocations in the library;
  • A bunch of --disable-xxx flags for disabling certain parts of the library;
  • Additionally we need to specify the C compiler CC, the archiver tool AR and the ranlib tool RANLIB. Those tools are specific to the target ABI;
  • The libmp3lame doesn’t produce the *.pc file and the FFMPEG_EXTRA_LD_FLAGS variable is just a way to accumulate all -l arguments in a single variable. Later it will be appended to --extra-ldflags argument. This is a thing of ffmpeg-android-maker and not the FFmpeg’s.
  • Then the clean build is made and the library is installed in the appropriate place.

Notice, that this script doesn’t have any hardcoded paths. It uses predefined variables that are already set to the appropriate values. The idea is to execute this script per each ABI you are targeting (armeabi-v7a, arm64-v8a, x86 and x86_64). Before each such execution all those variables are setup properly. And all libraries that are compiled (including the FFmpeg itself) will use the same values. I encourage you to use the same approach. To see how these variables are setup, please refer to these 2 files: export-host-variables.sh and export-build-variables.sh.

By extracting the values generation out of a build script it is possible to be focussed on a particular library building rather on how to obtain such generic values as a path to a C compiler for x86 ABI, for example.

libaom (cmake, make)

First let’s look at the build script:

The libaom requires CMake tool. Here we just create a new subdirectory and invoke the cmake from it, which will prepare all make files. Then make && make install will compile and install the library.

We also pass some arguments to tune the libaom output:

  • -DCMAKE_INSTALL_PREFIX - path to install the library. Same as for libmp3lame;
  • -DCONFIG_PIC=1 - we want the position independent code. Again, just like for libmp3lame;
  • -DCONFIG_AV1_ENCODER=0 - disabling the encoder we don’t need;
  • A bunch of flags that disable certain parts of the libaom;

But where is the C compiler? The archiver? Since CMake is used, we are able to write a CMake script to wedge into the configuration step. And this is the way to fix certain problems in libaom’s configuration script, as it doesn’t fully support Android.

We pass the path to that CMake script in -DCMAKE_TOOLCHAIN_FILE argument. Additionally we have to specify Android API Level as -DANDROID_PLATFORM and ABI for compilation as -DANDROID_ABI. Here is the CMake script itself:

The whole point here is to utilise the android.toolchain.cmake file which resides in Android NDK. It will automatically setup all default CMake variables for compilers, linkers, etc. We could even pass it directly as -DCMAKE_TOOLCHAIN_FILE, but as I mentioned the libaom’s cmake script has certain problems. And we need to define or redefine certain libaom-specific variables manually. That is why we use our own CMake script.

libdav1d (meson, ninja)

The libdav1d is built with the Meson tool and ninja. Let’s see the actual script:

According to Meson’s documentation we need to create ‘cross build definition files’ to specify cross-compilation parameters. Such files can’t be created in advance, because they contain both build machine-specific and target ABI-specific values, so different build machines will have their own versions of these files. A better way is to generate them on the fly.

Then we create a new subdirectory for the actual building and execute meson for generating ninja build files in that subdirectory. We pass the ‘cross build definition file’ as the --cross-file argument and make sure the static library is generated by--default-library=static. The rest of the argument is just a tuning of the result binary. After that we execute ninja && ninja install from the build subdirectory to build and install the libdav1d.

One more interesting detail. When building x86 ABI with asm enabled you need exactly nasm to be installed.

In my personal experience, the libdav1d is ~30% faster than libaom. And it is smaller, so it is a better option for a mobile app.

Travis CI setup caveats

External libraries may require certain software to be installed on a build machine before building is possible. The libdav1d requires meson, ninja and nasm. It was a problem to get them installed on a standard Travis CI machine for Android builds, because it uses quite outdated distribution of Ubuntu. The solution was to switch to a generic machine for Java builds with newer distribution and install all of the necessary software manually, including the Android SDK and NDK:

Getting the source code

Along with a library-specific scripts for building I also made a bunch of library-specific scripts for downloading source code archives and extracting actual sources from them. I personally stick to the idea of having just shell scripts that will do all the work for you. The only thing needed is preinstalled software.

But this approach has its cost: without the Internet connection or without access to a library’s archives hosting the whole building process will fail. The solution here is to have these archives or actual source trees in advance. Consider this way for higher reliability.

Conclusion

Building such software isn’t so complex, actually. The clear separation of concerns helps structuring the whole process into relatively small and understandable scripts.

Check out the Git repo and its Wiki page for more details about how all this stuff works together.

I wish you all a fun 2020!

Cheers.

--

--