TensorFlow Lite is the official framework for running TensorFlow models on mobile and edge devices. It is used in many of Google’s major mobile apps, as well as applications by third-party developers. When deploying TensorFlow Lite models in production, you may come across situations where you need some support features that are not provided out-of-the-box by the framework, such as:
In these cases, instead of building your own solutions, you can leverage Firebase to quickly implement these features in just a few lines of code.
Firebase is the comprehensive app development platform by Google, which provides you infrastructure and libraries to make app development easier for both Android and iOS. Firebase Machine Learning offers multiple solutions for using machine learning in mobile applications.
In this blog post, we show you how to leverage Firebase to enhance your deployment of TensorFlow Lite models in production. We also have codelabs for both Android and iOS to show you step-by-step of how to integrate the Firebase features into your TensorFlow Lite app.
You may want to deploy your machine learning model over-the-air to your users instead of bundling it into your app binary. For example, the machine learning team who builds the model has a different release cycle with the mobile app team and they want to release new models independently with the mobile app release. In another example, you may want to lazy-load machine learning models, to save device storage for users who don’t need the ML-powered feature and reduce your app size for faster download from Play Store and App Store.
With Firebase Machine Learning, you can deploy models instantly. You can upload your TensorFlow Lite model to Firebase from the Firebase Console.
You can also upload your model to Firebase using the Firebase ML Model Management API. This is especially useful when you have a machine learning pipeline that automatically retrains models with new data and uploads them directly to Firebase. Here is a code snippet in Python to upload a TensorFlow Lite model to Firebase ML.
# Load a tflite file and upload it to Cloud Storage. source = ml.TFLiteGCSModelSource.from_tflite_model_file('example.tflite') # Create the model object. tflite_format = ml.TFLiteFormat(tflite_source=source) model = ml.Model(display_name="example_model", model_format=tflite_format) # Add the model to your Firebase project and publish it. new_model = ml.create_model(model) ml.publish_model(new_model.model_id)
Once your TensorFlow Lite model has been uploaded to Firebase, you can download it in your mobile app at any time and initialize a TensorFlow Lite interpreter with the downloaded model. Here is how you do it on Android.
val remoteModel = FirebaseCustomRemoteModel.Builder("example_model").build() // Get the last/cached model file. FirebaseModelManager.getInstance().getLatestModelFile(remoteModel) .addOnCompleteListener { task -> val modelFile = task.result if (modelFile != null) { // Initialize a TF Lite interpreter with the downloaded model. interpreter = Interpreter(modelFile) } }
There is a diverse range of mobile devices available in the market nowadays, from flagship devices with powerful chips optimized to run machine learning models to cheap devices with low-end CPUs. Therefore, your model inference speed on your users’ devices may vary largely across your user base, leaving you wondering if your model is too slow or even unusable for some of your users with low-end devices.
You can use Performance Monitoring to measure how long your model inference takes across all of your user devices. As it is impractical to have all devices available in the market for testing in advance, the best way to find out about your model performance in production is to directly measure it on user devices. Firebase Performance Monitoring is a general purpose tool for measuring performance of mobile apps, so you also can measure any arbitrary process in your app, such as pre-processing or post-processing code. Here is how you do it on Android.
// Initialize a Firebase Performance Monitoring trace val modelInferenceTrace = firebasePerformance.newTrace("model_inference") // Run inference with TensorFlow Lite interpreter.run(...) // End the Firebase Performance Monitoring trace modelInferenceTrace.stop()
Performance data measured on each user device is uploaded to Firebase server and aggregated to provide a big picture of your model performance across your user base. From the Firebase console, you can easily identify devices that demonstrate slow inference, or see how inference speed differs between OS versions.
When you iterate on your machine learning model and come up with an improved model, you may feel very eager to release it to a production right away. However, it is not rare that a model may perform well on test data but fail badly in production. Therefore, the best practice is to roll out your model to a smaller set of users, A/B test it with the original model and closely monitor how it affects your important business metrics before releasing it to all of your users.
Firebase A/B Testing enables you to run this kind of A/B testing with minimal effort. The steps required are:
Here is an example of setting up an A/B test with TensorFlow Lite models. We deliver each of two versions of our model to 50% of our user base and with the goal of optimizing for multiple metrics.
Then we change our app to fetch the model name from Firebase and use it to download the TensorFlow Lite model assigned to each device.
val remoteConfig = Firebase.remoteConfig remoteConfig.fetchAndActivate() .addOnCompleteListener(this) { task -> // Get the model name from Firebase Remote Config val modelName = remoteConfig["model_name"].asString() // Download the model from Firebase ML val remoteModel = FirebaseCustomRemoteModel.Builder(modelName).build() val manager = FirebaseModelManager.getInstance() manager.download(remoteModel).addOnCompleteListener { // Initialize a TF Lite interpreter with the downloaded model interpreter = Interpreter(modelFile) } }
After you have started the A/B test, Firebase will automatically aggregate the metrics on how your users react to different versions of your model and show you which version performs better. Once you are confident with the A/B test result, you can roll out the better version to all of your users with just one click.
Check out this codelab (Android version or iOS version) to learn step by step how to integrate these Firebase features into your app. It starts with an app that uses a TensorFlow Lite model to recognize handwritten digits and show you:
Amy Jang, Ibrahim Ulukaya, Justin Hong, Morgan Chen, Sachin Kotwani
If you're using Cloud Firestore or Cloud Storage for Firebase, you're also using Security Rules. (If you're using the default rules instead of tailoring them to your app, this is where to start!) We're excited to announce that in the last few months we've released some substantial improvements to the tools for writing and debugging Rules, improvements to the Rules language itself, and increases to the size limits for Rules!. These are a few of the great new features. Check out the Security Rules Release Notes for a comprehensive list of everything we've released.
We've released several improvements to make the rules language more expressive and succinct. One particularly verbose pattern was comparing the new values of a document to existing values. The new Set type available in Rules is purpose-built for these comparisons, and also has methods for functionality you'd expect for a Set, like getting the intersection, union, or difference between Sets. For example:
Set type
Allow a user to create a document if the document has required and optional fields, but not others:
allow create: if (request.resource.data.keys().toSet() .hasOnly(["required","and","optional","keys"])
Sets come with == and in operators and hasAll, hasAny, hasOnly, difference, intersection, union, and size methods.
==
in
hasAll
hasAny
hasOnly
difference
intersection
union
size
Sets are most useful in conjunction with the Map class, and because the request and resource objects are both structured as maps, you're probably already familiar with it. Map recently got a few new methods, diff and get, that will hopefully open the door to more concise rules for everyone. Here's how they work:
Map
request
resource
diff
get
Map.diff() is called on one map, and takes the second map as an argument: map1.diff(map2). It returns a MapDiff object, and all of the MapDiff methods, like addedKeys, changedKeys, or affectedKeys return a Set object.
map1.diff(map2)
addedKeys
changedKeys
affectedKeys
Set
Map.diff() can solve some verbose patterns like checking which fields changed before and after a request. For example, this rule allows an update if the "maxLevel" field was the only field changed:
allow update: if request.resource.data.diff(resource.data).changedKeys().hasOnly(["maxLevel"]);
In the next example, posts have a field indicating the user role required to modify the post. We'll use Map.get() to get the "roleToEdit" field. If the document doesn't have the field, it will default to the "admin" role. Then we'll compare that to the role that's on the user's custom claims:
"roleToEdit"
"admin"
allow update, delete: if resource.data.get("roleToEdit", "admin") == request.auth.token.role;
Keep in mind that because Sets are not ordered but Lists are. You can convert a List to a Set, but you can't convert a Set to a List.
Local variables have been one of the most requested features in Rules, and they're now available within functions. You can declare a variable using the keyword let, and you can have up to 10 local variables per function.
let
Say you're commonly checking that a user meets the same three conditions before granting access: that they're an owner of the product or an admin user, that they successfully answered a challenge question, and that they meet the karma threshold.
rules_version = '2'; service cloud.firestore { match /databases/{database}/documents { match /products/{product} { allow read: if true; allow write: if (exists(/databases/$(database)/documents/admins/$(request.auth.uid)) || exists(/databases/$(database)/documents/product/owner/$(request.auth.uid))) && get(/databases/$(database)/documents/users/$(request.auth.uid)) .data.passChallenge == true && get(/databases/$(database)/documents/users/$(request.auth.uid)) .data.karma > 5; } match /categories/{category} { allow read: if true; allow write: if (exists(/databases/$(database)/documents/admins/$(request.auth.uid)) || exists(/databases/$(database)/documents/product/owner/$(request.auth.uid))) && get(/databases/$(database)/documents/users/$(request.auth.uid)) .data.passChallenge == true && get(/databases/$(database)/documents/users/$(request.auth.uid)) .data.karma > 5; } match /brands/{brand} { allow read, write: if (exists(/databases/$(database)/documents/admins/$(request.auth.uid)) || exists(/databases/$(database)/documents/product/owner/$(request.auth.uid))) && get(/databases/$(database)/documents/users/$(request.auth.uid)) .data.passChallenge == true && get(/databases/$(database)/documents/users/$(request.auth.uid)) .data.karma > 5; } } }
Those conditions, along with the paths I'm using for lookups can all now become variables in a function, which creates more readable rules:
rules_version = '2'; service cloud.firestore { match /databases/{database}/documents { function privilegedAccess(uid, product) { let adminDatabasePath = /databases/$(database)/documents/admins/$(uid); let userDatabasePath = /databases/$(database)/documents/users/$(uid); let ownerDatabasePath = /databases/$(database)/documents/$(product)/owner/$(uid); let isOwnerOrAdmin = exists(adminDatabasePath) || exists(ownerDatabasePath); let meetsChallenge = get(userDatabasePath).data.get("passChallenge", false) == true; let meetsKarmaThreshold = get(userDatabasePath).data.get("karma", 1) > 5; return isOwnerOrAdmin && meetsChallenge && meetsKarmaThreshold; } match /products/{product} { allow read: if true; allow write: if privilegedAccess(); } match /categories/{category} { allow read: if true; allow write: if privilegedAccess(); } match /brands/{brand} { allow read, write: if privilegedAccess(); } } }
You can see at a glance that the same conditions grant access to write to documents in the three different collections.
The updated version also uses map.get() to fetch the karma and passChallenge fields from the user data, which helps keep the new function concise. In this example, if there is no karma field for a user, then the get returns false. Keep in mind that Map.get() fetches a specific field, and is separate from the DocumentReference.get() that fetches a document.
map.get()
karma
passChallenge
Map.get()
DocumentReference.get()
This is the first time we've introduced an if/else control flow, and we hope it will make rules smoother and more powerful.
if/else
Here's an example of using a ternary operator to specify complex conditions for a write. A user can update a document in two cases: first, if they're an admin user, they need to either set the field overrideReason or approvedBy. Second, if they're not an admin user, then the update must include all the required fields:
overrideReason
approvedBy
allow update: if isAdminUser(request.auth.uid) ? request.resource.data.keys().toSet().hasAny(["overrideReason", "approvedBy"]) : request.resource.data.keys().toSet().hasAll(["all", "the", "required", "fields"])
It was possible to express this before the ternary, but this is a much more concise expression.
And finally, here's a feature for those of you with longer rules. Until now, rules files had to be smaller than 64 KB. (To be more specific, the compiled AST of the rules file had to be smaller than 64 KB, and you wouldn't know you were within the limit until you tried to deploy the rules.) This limit was holding some developers back, and once you reached the limit, you had to start making tradeoffs in your rules. We definitely wanted to fix this.
Since this is one of the limits that helps rules return a decision in nanoseconds, we wanted to find a way to increase the limit without sacrificing performance. We optimized how we compile and store the Rules file, and we were able to quadruple the limit to 256 KB!
The limits on rules are in place to keep rules fast enough to return a decision in nanoseconds, but we work hard to keep them workable. Let us know if you start to outgrow any of them
All of these features are informed by the feedback we hear from you about what's great, what's hard, and what's confusing about Firestore Security Rules, so keep letting us know what you think!
Over the last decade, mobile gaming has evolved from stacking colored blocks on a phone to encompass multidimensional and multiplayer experiences that engulf people in imaginative, high-fidelity worlds. As games have gotten more engaging, they’ve also become more complex to develop and maintain. Here at Firebase, we’re committed to providing game developers with the tools and resources you need to build, release and operate successful games that delight players.
We participated in the Google for Games Developer Summit in March, launched Unity and C++ SDKs for Cloud Firestore into open alpha, shipped improved workflows for Unity developers, debuted some new video tutorials, and wrote some new blogs too - so you can easily add Firebase to your game and get the most out of our platform.
We have more exciting updates to come, driven by your feedback and stories. Recently, we spoke to Gameloft to understand how they’ve been using Firebase.
Gameloft is one of the world’s largest gaming studios, employing 4,600 people across 19 global offices. Their vast portfolio consists of over 190 critically-acclaimed, award-winning games that are played by 80 million people each month. Gameloft develops original franchises, like Dungeon Hunter, and also partners with major entertainment studios like Disney.
With such a broad portfolio and audience, having the ability to monitor the stability of their games in a holistic way, and the agility to respond to issues quickly, is key to their success. However, they were lacking proper information about crashes, having a hard time reproducing crashes, and unforeseen technical issues were taking too long to track and fix, upsetting both their team and their players.
Gameloft realized they needed a robust crash reporting tool that could help them improve game quality in a more streamlined and efficient way, so they turned to Firebase Crashlytics.
Crashlytics aggregates crashes by root cause, highlights the impact on players, and provides contextual information on each issue, which helps Gameloft prioritize the crashes to fix first and speeds up their troubleshooting process. By using Crashlytics, Gameloft achieved a 10% reduction in users experiencing crashes overnight for one of their games (Overdrive City). According to Gameloft, this also led to a bump in the game’s Play Store rating and increased session duration by 16%! Find out what Gameloft did to drive these results with Crashlytics by reading the full case study. And learn more about how Firebase can help you build and grow your game, and see what other game studios are using Firebase.
With our announcement of the Firebase Crashlytics SDK Beta in February, we provided a path to help you remove all Fabric dependencies in your code, and complete your migration to Firebase. Since then, tens of thousands of apps have adopted the Beta SDK, and several hundreds of millions of end-users are using apps that include the SDK. Thank you to the adopters of our Beta SDKs; with your help, we have been able to polish and debug our SDK so we could ship it in its best possible launch state.
We are excited to announce that the Firebase Crashlytics SDK is now publicly available! Whether your app was migrated from Fabric or created in Firebase, we encourage you to upgrade to the official version of the Firebase Crashlytics SDK.
Now that the Firebase Crashlytics SDK is publicly available, we are deprecating the legacy Fabric SDK. The Fabric SDK will continue reporting your app's crashes until November 15, 2020. On this date, the Fabric SDK, and beta versions of the Firebase Crashlytics SDK, will no longer send crashes to your Firebase dashboard. To continue getting crash reports in the Firebase console, make sure your app has the following versions of the Firebase Crashlytics SDK, or newer: 17.0.0+ for Android, 4.0.0+ for iOS, and 6.15.0+ for Unity. Developers new to Crashlytics should begin with our getting started docs.
As always, we’d love to hear your feedback! Let us know what you think through our official support page!
Happy coding!
