Realistic Realm - lessons learned after using it for 1.5 years

Realm has been known in the mobile (and not only) developers’ community for a long time. A year and a half ago the following line appeared in the build.gradle of our project:

classpath “io.realm:realm-gradle-plugin:0.87.5”

During this year, the Realm code grew to version 3.3, got a lot of features and resolved a lot of bugs, implemented new functionalities and got a cloud backend. Let’s talk more in detail about Realm in the domain of Android development and discuss the subtle moments arising from its use.

About us

We are developing an application for communication within the team, something between Telegram and Slack. Android application has been written in Kotlin from the very beginning, used offline-first approach, i.e. when all the data displayed on the screen is obtained from a cache. Having tried several different databases, we stopped at Realm and actively used it throughout a year and a half. This article grew out of the internal document on the use of Realm. The article is not a translation of documentation and does not pretend to be a complete description; it is rather a collection of recipes and analysis of subtle points. For a full understanding, we strongly recommend you to read the official documentation. We will tell you about our experiences and what failures we made this year. All the code for the article is written in Kotlin, you can find it on Github.

Realm as a startup

If we talk about Realm as a company, then it is a Danish startup founded in 2011. Earlier the project was called tight.db. During its existence, 29M $ of investments were attracted. The company plans to earn on the basis of Realm Mobile Platform. The database itself is free and open source. Realm for Android appeared in 2014 and has been constantly evolving since then. Some updates break backward compatibility; however fixes can be made quite easily and quickly.

Realm as a database

Realm is a database for several platforms. They write about themselves:

The Realm Mobile Platform is a next-generation data layer for applications. Realm is reactive, concurrent, and lightweight, allowing you to work with live, native objects.

In short, this is a native NoSQL database for Android (Java, Kotlin), iOS (Objective-C, Swift), Xamarin (C#), and JavaScript (React Native, Node.js).

Also there is a backend, which allows you to synchronize data from all sources.

From the key features it is worth noting zero copy, MVCC and ACID. There is no built-in mechanism for obsolescence and data cleaning. Realm also doesn’t have auto increment for primary keys.

Realm has very good documentation and lots of examples on GitHub.

Realm employees are periodically monitoring StackOverflow, it is also possible to create an issue on GitHub.

Hello world

Hello world under Android looks like this:

Add to build.gradle

In Application, configure Realm Configuration

After that you can start working with the database:

Main features

There are three main features that should be taken into account:

Live Objects — All objects received from Realm are, in fact, a proxy to the database. Due to this, zero copy is achieved

Transactions — All changes of the linked data objects need to be done inside the transaction.

Open / Close — Need to open / close the instance database.

Live Objects

All objects from Realm can be received synchronously or asynchronously.

Synchronous reading

Call the Realm method and lock the thread until we get the object or null. It’s not allowed to use objects received in other threads, so you must block UI or use asynchronous requests for use in the main thread. Fortunately, Realm provides us with a proxy, not the object itself, so everything happens quickly enough. You can work with the object immediately after receiving it.

Asynchronous reading

A very unobvious case. What do you think will happen in this code?

The correct answer is: we will get the error java.lang.IllegalStateException.

At asynchronous reading though we receive an object at once, but we cannot work with it, until it is loaded. You need to check this with the isLoaded() function or call the blocking function load().It looks rather inconvenient, so it’s better to use GC here. Convert to observable and get the loaded object in OnNext. Asynchronous operations are available only in threads with Looper.

The main features of Realm objects

• Obtaining objects from the database is very fast, there is no deserialization as such, reading from the disk occurs only when accessing a particular field.
• For this purpose that there is a requirement to make all fields private and access through getters.
• The copyFromRealm() method allows you to retrieve untethered, fully assembled objects, just like a normal ORM. The truth and all the advantages of Realm become unavailable. On the input accepts the default deserialization depth MAX_INT.
• In the debugger, all fields will be null. To get any value, we need to go through a getter.
• All objects are Live. Changes are propagated instantly within the same thread. More complex cases see below (multithreading).
• Filtration of objects is done by fields, and you specify the field names in the form of a line by hand. For example: . equalTo(“id”, 1). This complicates the refactoring and leads to errors in the name of fields for filtering. Unfortunately Realm does not generate variables with field names, so it’s better to hide all the samples inside the function (or use the field name generator from cmelchior):

fun findFirstDataObject(id: Long, realm: Realm) : DataObject? = realm.where(DataObject::class.java).equalTo(“id”, id).findFirst()

• This point changed right at the time of writing the article (a vivid example of how the project develops):

It was: It is impossible to use DiffUtil when changing an object, it is impossible to understand which fields of the object has changed. That is, if you are notified of a change in the object, it is impossible to understand what has changed. This is due to the fact that both objects (old and new) are live objects and refer to the same data, they will always be equal.

Became: You can use RealmObjectChangeListener to understand what has changed: RealmObjectChangeListener.

• Any object is accessible only while instance of realm is opened, from which we received it. You can check using isValid methods. When accessing a non-object, we get an exception.
• Objects are available only in the stream in which they are created. You cannot access from another thread, we will get an exception.

Similarly, the lists (RealmResult) of objects (query results) are proxy to Realm, this leads to the following:

• Getting lists is very fast; in fact we get only count. All queries are lazy; get a big list of complex objects we can very quickly.
• Lists are read-only, any change methods result in an exception.
• Since we can get all the elements quickly and cheaply, we can forget about the problem of pagination. We always give a complete list of elements, while scrolling to objects, and they are quickly obtained from the database. If we need to load data, we start the download, get the data, save it in Realm, get a full list with the loaded items and display it.
• Until recently (prior to version 3.0) there was a problem with redrawing all elements of the list. If we use the list for the adapter, then when you change one element, the whole list will be completely redrawn. Use DiffUtils and compare which objects have changed, it does not work, because it’s live objects. In Realm 3.0 appeared OrderedCollectionChangeSet, which tells us DeletionRanges, InsertionRange, ChangeRanges. It was finally possible to understand which objects and how they changed.

Transactions

You can change the objects attached to Realm only inside the transaction, while changing outside the transaction, an error occurs. On the one hand, it is not very convenient, on the other hand — it disciplines and does not allow changing objects in any part of the code, only in a certain layer (database). You also need to remember that transactions within another transaction are prohibited.

Transactions can be made synchronously and asynchronously. Let’s take a closer look at each of the options:

Synchronous transactions:

You can also perform transactions between beginTransaction and commitTransaction, however it is recommended to use executeTransaction.

Unfortunately, synchronous transactions do not support onError callback, so error handling remains on your conscience. There is an issue for adding an onError callback from June 2016.

Asynchronous transactions:

Asynchronous transactions are started by the asyncTransaction method. At the entrance we give the transaction itself and the callback onSuccess and onError, on the output we get the RealmAsyncTask object, with which we can check the status or cancel the transaction. Asynchronous transactions are only run in threads with Looper. Example of an asynchronous transaction:

We present to you a couple of important nuances:

You cannot assign an object that is not tied to Realm through the setter. You must first put the object in the database, and then attach the bound copy. Example:

It is better to combine many transactions into one. In Realm, there is an internal queue on the transaction (size 100) and if you exceed it, the exception will drop.

All asynchronous transactions run on one executor.

// Thread pool for all async operations (Query & transaction) 
static final RealmThreadPoolExecutor asyncTaskExecutor = RealmThreadPoolExecutor.newDefaultExecutor();

If you have many asynchronous operations in a short time, you get an error.

RejectedExecutionException. The way out of this situation is to use a separate thread and run synchronous transactions in it or merge multiple transactions into one.

Open/Close Realm

All objects from the database we get using a specific instance of Realm, and we can work with them while this instance is opened. Once we call realm.close (), any attempt to read the object will turn into an exception for us. If we do not close Realm in time, it will lead to memory leaks, since the Garbage Collector does not know how to work correctly with the resources used by Realm.

In the official documentation, it is recommended to Open/Close Realm:

• For Activity: onCreate/onDestroy
• For Fragment: onCreateView/onDestroyView

However, if you want to take the logic of working with Realm from Activity/ Fragments to presenters, you have to use the life cycle methods.

In case you need to somehow change the data or add new ones, it’s easiest to get a new instance, write down the data and then close it. In Kotlin, you can use the .use () method.

Realm.getDefaultInstance().use { // it = realm instance}

To read objects using Rx, you can use an “isolated” instance and close them in doOnUnsubscribe (or use Observable.using).

There is also a feature associated with closing Realm in onDestroy/onDestroyView. Sometimes, after Realm closes, FragmentManagerImpl.moveToState → ViewGroup.removeView → … → RecyclerViewAdapter.getItemCount() the list.size() method is called from the invalid collection. So here you need to check isValid () or unbind the adapter from recyclerView.

If you use Kotlin Android Extensions, then you can work with view (from kotlinx.android.synthetic.*) from Fragment only starting with the onViewCreated () method, it is better to configure all listeners in this method, so as not to get NPE.

After analyzing the three most important features, let’s take a look at the less important ones:

Notifications, RxJava

Realm supports notifications about data changes, both of the object itself and of the nested objects (all the linked objects). Implemented this with a RealmChangeListener (the object returned to us), RealmObjectChangeListener(returns the changed object and the ObjectChangeSet for it, you can understand what fields have changed), or using RxJava(in onNext we get the object, in the case of an asynchronous request, it is necessary to check isLoaded (), it works only in threads with Looper).

RxJava2 has not yet been implemented, the question is hanging from September 2016, it is unknown when it will be implemented, use Interop.

Similarly, you can listen to changes in collections or whole instance Realm. Listening to changes within transactions is prohibited.

Example of Rx:

Multithreading and Asynchronous

Realm is an MVCC database. Here what’s written in Wikipedia About MVCC :

“Multiversion concurrency control (MCC or MVCC), is a concurrency control method commonly used by database management systems to provide concurrent access to the database and in programming languages to implement transactional memory.
If someone is reading from a database at the same time as someone else is writing to it, it is possible that the reader will see a half-written or inconsistent piece of data. There are several ways of solving this problem, known as concurrency control methods. The simplest way is to make all readers wait until the writer is done, which is known as a lock. This can be very slow, so MVCC takes a different approach: each user connected to the database sees a snapshot of the database at a particular instant in time. Any changes made by a writer will not be seen by other users of the database until the changes have been completed (or, in database terms: until the transaction has been committed.).”

In practice, it looks like this: we can listen to object changes or use RxJava to get modified objects in onNext. In case the changes occur in thread A, and we work with the object in thread B, then thread B learns about the changes after Realm instance closes in thread A. Changes are transmitted by Looper. If there is no Looper in the thread B, then the changes will not reach (you can check with the isAutoRefresh() method). The way out of this situation is to use the waitForChange() method.

As for asynchronous calls and transactions, it is better not to use them at all. It is more convenient to translate actions to a separate thread and perform synchronous operations there. There are several reasons:

• It’s not allowed to mix asynchronous and synchronous transactions, if mix, all transactions will become synchronous.
• It’s not allowed to use asynchronous calls in threads without Looper.
• For long transactions, you need to open a separate instance of Realm, otherwise Realm can be closed during the transaction and you will get an exception.
• All actions in an asynchronous transaction occur on a separate internal executor, as a result you cannot use external Realm objects, and it is possible that the executor is overflowing; Realm object changes do not propagate between threads and other inconveniences.

Testing

Previously Realm.java was final and for testing you needed powerMock or other similar tools. At the moment Realm.java has ceased to be final and you can safely use the usual mockito. You can see examples of tests in the demo project or on the official repository.

One Realm is good, but three are better

Working with Realm, we always mean a standard realm, but there are still In-Memory Realm and Dynamic Realm.

• Standard Realm — can be obtained using Realm.getDefaultInstance() methods or using a specific Realm.getInstance(config) configuration, there can be unlimited configurations, these are essentially separate databases.
• In-Memory Realm is Realm, which stores all the recorded data in memory without writing them to disk. As soon as we close this instance, all the data will be lost. This is suitable for short-term storage of data.
• Dynamic Realm — used primarily for migration, allows you to work with realm objects without using the generated RealmObject classes. Access to them is carried out by the field names.

Inheritance and Polymorphism

Realm does not support inheritance. Any realm object must both be inherited from RealmObject or implement the RealmModel marker interface and be marked with an @RealmClass annotation. You cannot inherit from different Realm objects. It is recommended to use the composition instead of inheritance. A very serious problem, the issue is hanging since January 2015.

Kotlin

Realm initially works with Kotlin.

Data classes do not work; you need to use a regular open class.

It’s worth to note that Kotlin-Realm-Extensions are convenient extensions for working with RealmObject.

Realm mobile platform

First time Realm was presented only by databases for different platforms, now they rolled out the server for synchronization between all devices. Now the platform consists of:

• Realm Mobile Database — database for data storage
• Realm Object Server — the server responsible for automatic synchronization and event processing
• Realm Data Integration API — to connect and synchronize data with existing databases (Oracle, MongoDB, Hadoop, SAP HANA, Postgres and Redis)

Debugging

For debugging we have several tools:

• RealmLog — outputs the log, there are different levels of logging
• Realm browser — you need to browse the database from your computer. Works only under Mac. To view the database on Windows, you can use Stetho Realm.
• There are also several Android libraries for easy viewing of the data on the device.
• WriteCopyTo() — allows you to copy the database to a file and send it to the analysis.
• NDK Debugging — to analyze the errors in the native code you can use Crashlytics NDK Crash Reporting

Architecture

Realm is great for MV * architectures, when the entire implementation is hidden behind the database interface. All calls and fetches take place in the database module (repository), on top are given Observable with an automatically closed realm at unsubscribe. Or we take an instance of realm and enter all the actions with it. When recording objects, we open realm, recording the data and close it, only the object to save is fed into the input. Both examples you can see at GitHub.

Alas, using Realm (without copyFromRealm) imposes serious restrictions on the use of clean architecture. Use different data models for different layers will not work the entire meaning of live objects and list proxy disappears. Also, difficulties arise when creating independent layers and opening/closing Realm, since this operation is tied to the Activity/Fragment lifecycle. A good option would be an isolated layer for getting data, converting objects, and storing them in a database.

Realm is very useful for building offline-first applications, when all the data for display we get from the database.

Useful links

To continue acquaintance and analysis of subtle points, we recommend the following articles:

Three articles from @Zhuinden:

1. Basics of Realm: A guide to using Realm 1.2.0

2. How to use Realm for Android like a champ, and how to tell if you’re doing it wrong

3. Realm 1.2.0 + Android Data Binding

Two articles about Realm integration from @ Viraj.Tank:

1. Safe-Integration of Realm in Android production code, Part-1 with MVP

2. Deep integration of Realm in Android production code, Part-2, with MVP

Multithreading, detailed parsing:

1. Designing a Database: Realm Threading Deep Dive

2. Docs — Auto-Refresh

3. Docs — Threading

Conclusion

Realm is harder than it seems at first glance. However, all the shortcomings associated with the losses are covered by its power and convenience. Live objects, notifications and Rx, a handy API and many other things make it easy to create applications. From competitors, you can select requery, ObjectBox and GreenDao. Fully Realm reveals itself when building offline-first applications, when all the data we get from the cache and we need complex samples, as well as constant updating of the data.

All the code you can find on Github.

This article is a translation of my article from habrahabr