Custom Image Labeling with Firebase AutoML

The newest and one of the most interesting features from the Firebase — AutoML.

For the last few months, Firebase provided 2 options for using image labeling on mobile clients:

1. Use their own pre-built model which can recognize around 400 categories which perform quite well until you want to be specific.
2. Build your own model by using TensorFlow Lite and use it within Firebase ML. More about TensorFlow Lite you can check here.

Deeper comparison can be found here.

Okay, assuming you are here because the pre-built model is not really what your business needs cover and the second option with building your own model is too much, that's when AutoML appears on the stage. Or AutoML Vision Edge to be specific for mobile usage.

Usage of AutoML Vision Edge consists of two steps:

1. Getting trained model from the images.
2. Use it on the iOS/Android device.

Step 1. Getting trained model from the images

To start with AutoML open your firebase console at ML Kit tab. Within it, you can find AutoML labeled as “new”. Based on your plan you can have one or more created datasets. Also, Spark and Flame datasets are stored in a different way inside the platform, so in case your project will change the plan to Blaze and you want to have all the benefits from it please consider using this official migration guide.

Once your dataset is created you can proceed with providing the images that you want to recognize. To do it, you can use zip archive where folders will be labels for the images inside them or upload images on your own and assigning labels to them. Small note — Spark plan can only have 1000 images in one dataset.

First pitfalls during importing the images — documentation states that each individual image should be less than 30Mb but the reality is that only 5Mb otherwise you will get “Internal server error while importing data to dataset.” (for archive approach) and more descriptive error if you upload the image.

Official Firebase AutoML documentation.

Also, I was not able to start training the model with 10 images for each label — got “Please upload more images to start training” error. Actually, until I’ve refreshed the page without cache.

So, you have all the needed images and labels in place and ready to train the model. A really interesting feature is estimated latency and size of the built model, especially when iPhone 8 is the fastest from the list beating others Android devices almost in 3 times.

Let the training begin — 1 dataset, 3 labels, 169 images, general purpose, training duration 30 minutes.

Important note — Spark plan gives you only 3 hours of training time in total for the project so it will be a good idea to try it on the test project before doing it on the one that specific to your app.

After training will be finished you will receive an email about it which is really nice. After that, you are able to download the model to bundle with the apps or publish it to use remotely and save a few megabytes of user traffic. Also, Firebase provides a nice dashboard to check general model metrics and test page to test it.

Step 2. Using AutoML on mobile (Android)

To start using the results of your work in your app first of all it should be Firebase ready. If it’s not, check the official guide to integrate Firebase.

Then we need to add two dependencies:

Based on your UI and goals you can use local, remote or local+remote approach.

For the local part, you should have downloaded an archive from the Firebase AutoML dashboard with these 3 files model.tflite, dict.txt, and manifest.jsonand put them into /assets/<model name>.

For the remote usage — use “Publish” option via the Firebase AutoMl dashboard.

Then you are ready to add some code:

1. Register local/remote models in FirebaseModelManager.getInstance()
2. Create an instance of FirebaseVisionImage from Bitmap, media.Image, ByteBuffer or a file path.
3. Setup image labelerFirebaseVision.getInstance().getOnDeviceAutoMLImageLabeler(labelerOptions)
4. (OPTIONAL but recommended for remote approach) Check if the remote model is downloadedFirebaseModelManager.getInstance().downloadRemoteModelIfNeeded(remoteModel)
5. Provide an image to labeler and wait for a few milliseconds to magic happen labeler.processImage(image)
  .addOnSuccessListener {labels}
  .addOnFailureListener {exception}

Detailed code example can be found on the official page.

Summary

AutoML is a nice tool with a promising future especially for newcomers to the machine learning field. On the scales of the complexity comparing other ML Kit features for image labeling, AutoML will be right in between using pre-built models and custom models.