Migrating to Room in the real world — Part 3

Bringing everything together.

Photo by Dominic Hampton on Unsplash

In the first two parts (part 1 and part 2), we outlined the process we would be following for migrating to Room from our previous ORM, Cupboard, and we identified 4 main phases for doing so. We also explored in detail what Phase 1 and Phase 2 were all about.

In this final part, we will at the last two phases:

• Phase 3: make the existing migrations work with Room and handle the seed database
• Phase 4: flip the switch

Phase 3: make the existing migrations work with Room and handle the seed database

Since its creation, the Blinkist app has seen 12 explicit database migrations. Why explicit, you might be wondering? As it turns out, Cupboard allows you to do simple changes to the database schema, like adding or removing columns, without the need for an explicit SQL migration. However, Room doesn’t come with such a feature, and thankfully so! Our team feels strongly against implicit automatic aids, like this one, that might be cause of undesired regressions. For this reason, we had to manually track down where and when new fields were added/removed, which would in turn be reflected on the schema by means of adding/removing columns.

Having collected all the information, we had to choose between converting every single migration to Room or defining some sort of threshold, below which we would perform a “destructive migration” (i.e., wiping the database clean, so that the schema can be safely changed). After having looked at the usage data across our user base, we decided to only support migrations from a version that dates back to the beginning of 2018. A user who would update from an older version straight to the new one would experience a destructive migration; as bad as this might sound, it only means that we’ll trigger a sync procedure after the migration is over and fetch all the user’s data once more (same behaviour as if you were logging in the app again).

The way to do this, although not straightforward, is still very simple:

Note how we need to provide a vararg of Int!

Making the migrations work with Room consisted literally of replacing the SQLiteDatabase reference with the SupportSQLiteDatabase one that comes with the Room library. Since we were simply running SQL statements to alter the schema, no other change was required.

The first time we tried to simulate a migration, we ended up with a crash. Let’s all agree that the stacktrace could have been a little more helpful:

A somewhat opaque error message, given the underlying reason.

To the best of our knowledge, we weren’t explicitly closing the database at any point in time. Thankfully, after quite some searching, we ended up in this StackOverflow answer. Apparently, Room was detecting some kind of mismatch between the schema that it was expecting and the schema that was found in the SQLite database file, causing the migrations to fail. We opened one issue to make this error message more explicit, please go ahead and star it!

Remember the seed database, our initial pre-populated database used to avoid the “empty screen” problem? The way it used to be working is that the seed database file was simply copied from the assets/databases directory to the app’s private directory, and that very same database would be used as the main one. Once more, the community had already provided a solution for us, and all it took was pulling this library in and using it, as shown in the following one-line statement:

Having an open helper factory is such a welcome feature! Props to the Room team.

Now that migrations were in place, and we had a way to handle the seed database, it was finally time to glue everything together and flip the switch.

Phase 4: flip the switch and test

All it took to “flip the switch” was to provide the Room-based repositories instead of the Cupboard-based ones in our dependency graph, together with providing the Room database itself.

We did. And then we ran the app. And then…

It crashed on startup. We will spare you the countless trial-and-error tales: suffices to say that we encountered several errors, which required several fixes, and we simply present them here, hoping that they can be helpful to you as well.

Error 1: trying to access an already closed database

java.lang.IllegalStateException: attempt to re-open an already-closed object: SQLiteDatabase: (database path)

This error occurred because we were trying to access the same underlying SQLite database file with both Room and Cupboard (we forgot to change one provider from Cupboard to Room). It is important that, when switching to Room, you do it completely. 🤦‍♂️

Error 2: accessing the database on the main thread

Apparently, some bits of the existing code were accessing the database on the main thread. This can be temporarily resolved by allowing Room to operate on the main thread, but the proper fix is to track down those accesses and offloading them to a worker thread.

Error 3: mismatch between database schema and Room models

java.lang.IllegalStateException: attempt to re-open an already-closed object: SQLiteDatabase: (database path)

Even though this error is identical to Error 1 mentioned above, its cause in this case is completely different. Compared to Cupboard, Room is much more strict when it comes to schema validation: even the slightest difference, for example a mismatch in the type of a column, will make Room throw an exception.

Our database schema, as it has now been mentioned several times, is very simple, and assumed everything to be nullable. Since Room enforces consistency checks on the schema, it was expecting to see that nullability reflected in its models (and viceversa). We had the option to either run a migration, or to reflect the nullability in our models. We chose the latter option, sticking to our previous decision of not to change the schema.

Error 4: indices’ names mismatch

Some of the columns for our entities were used as SQL indexes, and Room expects that entity models correctly map to them, both in terms of name and respective column. Cupboard uses a naming convention that goes like _cb${tableName}_${columnName}”, while Room uses index_${tableName}_${columnName}. We decided to keep the existing names and to simply have Room map to them:

Mapping the existing indices thanks to the name property of the Index annotation.

After all these errors were taken care of, we managed to get the app up and running. Not only that, but our team performed a brief smoke test and couldn’t find any issue.

We then coordinated once more with our relentless QA team to define a testing strategy, in order to guarantee a high success chance when releasing to production. We identified three main scenarios that we wanted to run regression tests for:

1. Fresh install
2. Update from the current production version
3. Update from a version that required all the migrations to be run

All tests passed with flying colors.

It was time to release to production.

Lessons learned and conclusions

This engineering journey was invaluable under so many different aspects. We performed an initial analysis, outlined an execution plan and a time/cost estimation, and eventually coordinated with other departments. Not only it proved us right in our assumption that we could “simply migrate” to another ORM while not blocking any other development, and with minimal impact on the codebase: it also gave the entire team more confidence and knowledge about the codebase (as some of the classes were created a long time ago), it reaffirmed the effectiveness of our code review process, it showed a strong collaboration between the Android team and the QA team; hopefully, it will ultimately help you, if you’re reading this and considering to do the same.

An even more tangible upside of sticking to this process is the following: until today, we had 0 (zero!) regressions in production. Let that sink in for a moment.

We had 0 (zero!) regressions in production

Room is mature, and Google is putting a lot of efforts into making sure that it gets even better. This experience has proven this statement to be extremely true for us.

We won’t lie though. At some point, the ever present dilemma of “replacing” vs “incrementally refactoring” hit us too. It hit us twice, actually. Before even beginning this endeavour, we knew that we could simply start using Room as a separate database for one of the upcoming features of Blinkist. Many of us in the team had done it in the past, it was almost risk-free, painless and effective. However, we carefully evaluated our situation, calculated the pros and the cons, and decided to take a calculated risk: we estimated how many sprints this switch would take, and decided it was worth the investment. Phase 1, for example, would have been beneficial to us and to the codebase regardless of whether we ultimately switched to Room or not. We also agreed that if, at any point, we realized the migration was too risky or was lagging behind, we would regroup and reconsider whether to proceed or not.

In the end, we managed to be somewhat precise with our estimations, and we are extremely happy with the outcome.

So long, Cupboard, and thanks for all the fish!

Thanks for sticking to the very end! If you missed part 1 and part 2, be sure to check them out.

If you have any feedback, you can find Sebastiano and David on Twitter. Happy coding! 🤓