On-device Google Translate with Jetpack Compose & MLKit

In this article will do a code walkthrough of implementing Google Translate’s character recognition (along with text translation) in Jetpack Compose using MLKit’s DigitalInk and On-device Translation.

Here’s the source code of my implementation.

surajsau/ML-Android: Implementing various ML usecases in Android

Drawing on Canvas

Let’s setup a @Composable Canvas where users will draw the characters on screen.

This is a standard use-case where we intercept the touch events on the View and translate it as a Path on the canvas. Extending it to Jetpack Compose,

• Using Modifier.pointerInteropFilter we get the MotionEvent similar to that obtained in android.view.View OnTouchListener.
• For convenience I’ve created sealed class DrawEvent which will be mapped to the corresponding MotionEvent.
• remember the Path object, since it’ll not be changing over recompositions. Setting value to mutableStateOf<DrawEvent?> within the callback will trigger recompositions where either a new subpath is created or the current one is modified.
• path.lineTo() would also do instead of path.quadraticBezierTo(). I’ve gone with a quadratic subpath only to smoothen out sharp corners.
• Finally, at the end of each composition calldrawPath()

MLKit’s Digital Ink

MLKit by Google provides Digital Ink, a ready-to-use package for character recognition for over 300+ languages. It is also powers the handwriting recognition for services like Gboard, Google Translate.

The main components that we will be primarily dealing with are Ink.Stroke, DigitalInkRecognitionModel and DigitalInkRecognizer.

Let’s initialise them as we discuss their purpose.

• Ink.Stroke : this data object contains the list of coordinates (Ink.Point) information of the drawn shape which is the input to our RecognitionModel. MotionEvent coordinates obtained in the callback are then added to this.

• DigitalInkRecognitionModel : this is the ML model corresponding to the specific language whose characters are to be recognized. The builder takes in a ‘modelIndentifier’ to specify the language of character recognition. For this example we’ll be using the Japanese Language model, so we pass DigitalInkRecognitionModelIdentifier.JA
• DigitalInkRecognizer : this takes in the generated Ink.Stroke as input to the ML model and outputs a result of characters predicted by the model.

Checking Model’s availability

Before we use Digital Ink, for it to support language-specific character recognition the language specific ML model data (roughly, 20MB per language) needs to be downloaded and available locally.

You can checkout Digital Ink | Managing model downloads for more details. Here’s my code for reference.

Check if Model is available locally using RemoteModelManager

Download Model using RemoteModelManager

Building Ink.Stroke & Recognizing

With a drawable Canvas and MLKit model ready, let’s add callbacks to handle building Ink.Stroke and triggering Recognizer.recognize() when done.

For convenience I’ve added sealed class DrawEvent mapped to the corresponding MotionEvent.

Inside the onDrawEvent callback,

• MotionEvent.ACTION_DOWN, MotionEvent.ACTION_MOVE: This indicates that the user is touching the canvas to draw. We add the obtained event coordinates to Ink.Stroke.Builder
• MotionEvent.ACTION_UP: This indicates that user has stopped touching the canvas thus signalling end of drawing. We can now build the generated Ink.Stroke and pass it to Recognizer.recognize().
  In onSuccess we get back the prediction results (RecognitionResult.candidates) for display. In onComplete we reset the Ink.Stroke.Builder to start taking in coordinates of the next character to be drawn.

At this point the above code snippet runs fine. However, in the example below the predictions results aren’t correct at all.

without debounce

While some characters can be written in a single stroke, characters like i, t & complex characters like 終, even in cursive, cannot be written in one stroke. Every MotionEvent.ACTION_UP calls Recognizer.recognize() where we reset Ink.Stroke after receiving the results. So instead of passing coordinates of all subpaths combined, we’re calling .recognize() for each individual subpath.

This issue can be mitigated, by adding a ‘debounce’ like behavior which will prevent successive .recognize() calls for a specific time period after MotionEvent.ACTION_UP.

Adding debounce

To implement debounce, we have to cancel out all successive calls to Recognizer.recognize()for a specific time period (the debounce offset) after MotionEvent.ACTION_UP. There might be better ways to implement it, however this is the solution I came up with.

We add a delay(offset) within a coroutine before calling Recognizer.recognize() in MotionEvent.ACTION_UP. We can then cancel the launched coroutine in the next MotionEvent.ACTION_DOWN.

If user touches the Canvas within the specified delay, the launched coroutine is cancelled thus avoiding Recognizer.recognize(). This will allow users to add multiple strokes as long as they start drawing the next stroke (trigger the next MotionEvent.ACTION_DOWN within this specific offset.

MLKit’s On-device Translation

Implementing text translation is pretty much similar as above, instead we’ll be using MLKit’s On-device Translation. Like the name suggests the translation model runs on your device without requiring internet, which is why it might be slightly inaccurate at times.

Like DigitalInkRecognizer in Digital Ink, we’ve Translator here. This handles both downloading the translation as well as translation. While initialising Translator we’ve to specify the source and target languages. Rest of the implementation is pretty self explanatory.

Finally .close() everything

Even in the documentation it is mentioned clearly to .close() the Translator & Recognizer objects if no longer used. We can add a lifecycle observer and close those objects in LifeCycle.Event.ON_STOP.

Thank you for sticking along this pretty long read! 😅

This article is a part of Goodpatch’s Advent Calendar 2021🎄 We have a series of wonderful articles lined up for you this holiday season. While most of the articles will be in Japanese, if you are interested in going through the contents, you can use DeepL translator. It gives pretty accurate translations for technical articles in Japanese ♨️