Using TensorFlow on Android — step by step code explanation
Nowadays almost everyone heard about machine learning or deep learning. But do we know what it is exactly and how to use it? I didn’t know. In the matter of fact, I started to be more interested in this topic just couple months ago and it turned out to be indeed very interesting.
I wanted to create an app which uses something learned by machine learning. I found TensorFlow, Google’s open source library which was created for this purpose. With the use of TensorFlow we are able to create a deep neural network, train it, save it and use it in our app.
You could got confused by now, because I’ve mentioned terms like machine learning, deep learning and neural networks without any explanation. So what are they? Long story short:
- Machine learning is a branch of Artificial Intelligence that gives computer the ability to learn by themselves using large data sets.
- Deep learning is one of the machine learning methods that uses artificial neural networks.
- Artificial neural networks consists of artificial neurons and connections between them. There can be many layers of connected neurons. This neural network have ability to learn on some data set.
If you are interested in how the neural network works I encourage you to read article by Radu Raicea about how deep learning works. It’s a good start. But it’s worth to mention that we won’t need all the specific knowledge right from the beginning. As Android developers we should have the minimum knowledge which helps us to understand and use the TensorFlow library in our app.
In this article we use an example app called ‘Hot or Not’ and we’ll go through each aspect of the code and explain everything step by step. We’re going to use already created classifier and see how to use it.
Hot or Not?
The app itself is pretty simple. We’re taking a photo, the photo is classified if it’s hot or not and the results are shown on the screen.
Take a photo
Let’s start from taking a photo. The code looks like this (you can skip this snippet if you know how to take a photo):
We simply take a photo and from the onActivityResult() we’re gonna to classify it.
Classify the photo
The next step is to create a classifier interface which our TensorFlow classifier will implement.
As you can see, all we need to do is just one method with Bitmap parameter which is fact our photo. This method returns a Result.
The Result class has two properties — result string (‘hot’ or ‘not’) and confidence of the result. We’ll use the confidence after the classification.
Now it’s time to create an instance of the classifier inside the MainActivity.
We changed a bit the previous code of the MainActivity. Let’s look a the createClassifier() method. Inside it we use ImageClassifierFactory to create it. Before we jump the this class, let’s look at the parameters that we pass to the factory.
The assets parameter is an AssetManager instance. The other parameters are placed in Constants class.
Ok, so what are they?
- GRAPH_FILE_PATH — path to our classifier file in the assets folder. We’ll be using words ‘classifier’ or ‘graph’ alternately. Graph because our classifier is a saved neural network graph. You can also call it a ‘model’.
- LABELS_FILE_PATH — path to our labels file in assets folder. Simply put, labels are our possible results. Our classifier was trained to tell if a photo is hot or not. This is all it can do, so our labels are just ‘hot’ and ‘not’.
- GRAPH_INPUT_NAME — name of the classifier’s input. We pass the image to the classifier via input.
- GRAPH_OUTPUT_NAME — name of the classifier’s output. We get the result of the classification from the output.
- IMAGE_SIZE — the size of the image in pixels. Our classifier can understand images which are 224x224 pixels — that’s how it was trained.
Ok, now we have each parameter explained. To have better understanding of how we’ll be using our classifier, let’s look at the image below.
We’ve already know that our classifier is a neural network that was trained to understand hot or not photos but in fact all we need is just to think of it as some black box.
We take a photo and pass it via input. The photo has to be 224x224. Then the classifier classifies the photo. After the classification, we can take the results from the output — the results are always both of our labels with the confidence. For example ‘hot’ in 0.7 and ‘not’ in 0.3 — the confidence sum up to 1 so we can treat it as probability.
Cool, we created the classifier instance and we understand what it is. Now we’re going to use it in the Activity and then we’ll jump to the classifier factory implementation.
We take a photo and pass the file to the classifyPhoto() method. Inside this method, we crop the bitmap to fit 224x224 pixels. After this operation we call recognizeImage() method on our classifier instance and we get the results (it’s happening in the background thread because it can take some time).
To crop the image we use ImageUtils.getCroppedBitmap(). I took the algorithm from the ImageUtils class in TensorFlow example on GitHub and converted this class to Kotlin.
Let’s jump to the factory method.
Based on the parameters that are passed from the Activity, the factory creates an instance of the classifier, but the classifier itself needs a bit more specific information. Couple of the parameters are just passed but not all of them:
- labels — a list of strings. We use
FileUtilsclass to get the labels from the text file. - imageBitmapPixels —the array that we allocate based on image size. We’ll fill this array with bitmap pixels later on during classification.
- imageNormalizedPixels — the array that we allocate based on the image size and color channels (in our case 3). Our classifier uses normalized values so we have to convert integers to floats later on.
- results — the array that we allocate based on labels count. The classifier returns probability for each label.
- tensorFlowInference — the API from the TensorFlow library. It takes the asset manager and the graph file path to load the graph.
Now we have better understanding of the parameters — why and how they are created — so finally we can look inside the ImageClassifier.
Inside the recognizeImage() method we can see exactly what is happening. First of all we have to preprocess bitmap pixels into normalized values. In the preprocessImageToNormalizedFloats() is an algorithm responsible for that which I also took from the TensorFlow examples on GitHub. It’s placed inside the TensorFlowImageClassifier.
Then, we can classify the image. The process consists of three phases.
- Feed the classifier using
feed()method. To this method we have to pass three parameters — input name, array with normalized values and dimensions of the input. In our case the dimensions are: 1 (because we put there one image) * image size * image size * color channels. - Run the classifier with the
run()method. This method needs two parameters — array with possible outputs (in our case just one — ‘final_result’) and a flag to enable stats. - Get the results from the classifier using
fetch()method. The parameters here are — output name from which we want to take the results and the array which will be filled with these results. This is aFloatArraybecause the classifier returns probabilities for each of our two labels.
The last step is to map the results to our Result object. For that we use PriorityQueue that we fill with the results and poll the one with the highest confidence.
Show the result
The last part of our example is to show the result.
This is nothing fancy — just set a text and a background color based on the result.
Hot or Not example
The whole code you can find on my GitHub repo.
Wrap up
Using TensorFlow in Android is not that easy as we could expect. It’s just a library, right? What can go wrong? On one hand it’s true but on the other hand it’s a library with a lot of specific knowledge behind it — the machine learning. We should have a minimum knowledge about the topic to start working with it, but as you could see it’s not that much.
Hope that this example was helpful and thanks to it you’ll be more interested in the topic of machine learning.
If you’re interested how the model was created, you can about it here:
Stay save!
