This article originally appeared in the Firebase Developer Community blog.
We like saying lots of impressive things about Cloud Firestore's performance -- "performance scales with the size of the result set, not the underlying data set", and that "it's virtually impossible to create a slow query." And, for the most part, this is true. You can query a data set with billions upon billions of records in it, and get back results faster than your user can move their thumb away from the screen.
But with that said, we occasionally hear from developers that Cloud Firestore feels slow in certain situations, and it takes longer than expected to get results back from a query. So why is that? Let's take a look at some of the most common reasons that Cloud Firestore might seem slow, and what you can do to fix them.
Probably the most common explanation for a seemingly slow query is that your query is, in fact, running very fast. But after the query is complete, we still need to transfer all of that data to your device, and that's the part that's running slowly.
So, yes, you can go ahead and run a query of all sales people in your organization, and that query will run very fast. But if that result set consists of 2000 employee documents and each document includes 75k of data, you have to wait for your device to download 150MB of data before you can see any results.
The best way to fix this issue is to make sure you're not transferring down more data than you need. One simple option is to add limits to your queries. If you suspect that your user only needs the first handful of results from your employee list, add a limit(25) to the end of your query to download just the first batch of data, and then only download further records if your user requests them. And, hey, it just so happens I have an entire video all about this!
limit(25)
If you really think it's necessary to query and retrieve all 2000 sales employees at once, another option is to break those records up into the documents that contain only the data you'll need in the initial query, and then put any extra details into a separate collection or subcollection. Those other documents won't get transferred on that first fetch, but you can request them later as your user needs them.
Having smaller documents is also nice in that, if you have a realtime listener set up on a query and a document is updated, the changed document gets sent over to your device. So by keeping your documents smaller, you'll also have less data transferred every time a change happens in your listeners.
So Cloud Firestore's offline cache is pretty great. With persistence enabled, your application "just works", even if your user goes into a tunnel, or takes a 9-hour plane flight. Documents read while online will be available offline, and writes are queued up locally until the app is back online. Additionally, your client SDK can make use of this offline cache to avoid downloading too much data, and it can make actions like document writes feel faster. However Cloud Firestore was not designed as an "offline first" database, and as such, it's currently not optimized for handling large amounts of data locally.
So while Cloud Firestore in the cloud indexes every field in every document in every collection, it doesn’t (currently) build any of those indexes for your offline cache. This means that when you query documents in your offline cache, Cloud Firestore needs to unpack every document stored locally for the collection being queried and compare it against your query.
Or to put it another way, queries on the backend scale with the size of your result set, but locally, they kinda scale with the size of the data in the collection you're querying.
Now, how slow local querying ends up being in practice depends on your situation. I mean, we're still talking about local, non-network operations here, so this can (and often is) faster than making a network call. But if you have a lot of data in one single collection to sort through, or you're just running on a slow device, local operations on a large offline cache can be noticeably slower.
First, follow the best practices mentioned in the previous section: add limits to your queries so you're only retrieving the data that you think your users will need, and consider moving unneeded details into subcollections. Also, if you followed the "several subcollections vs a separate top level collection" discussion at the end of my earlier post, this would be a good argument for the "several subcollections" structure, because the cache only needs to search through the data in these smaller collections.
Second, don't stuff more data in the cache than you need. I've seen some cases where developers will do this intentionally by querying a massive number of documents when their application first starts up, then forcing all future database requests to go through the local cache, usually in a scheme to reduce database costs, or make future calls faster. But in practice, this tends to do more harm than good.
Third, consider reducing the size of your offline cache. The size of your cache is set to 100MB on mobile devices by default, but in some situations, this might be too much data for your device to handle, particularly if you end up having most of your data in one massive collection. You can change this size by modifying the cacheSizeBytes value in your Firebase settings, and that's something you might want to do for certain clients.
Fourth, try disabling persistence entirely and see what happens. I generally don't recommend this approach -- as I mentioned earlier, the offline cache is pretty great. But if a query seems slow and you don't know why, re-running your app with persistence turned off can give you a good idea if your cache is contributing to the problem.
So zig-zag merge joins, in addition to being my favorite algorithm name ever, are very convenient in that they allow you to coalesce results from different indexes together without having to rely on a composite index. They essentially do this by jumping back and forth between two (or more) indexes sorted by document ID and finding matches between them.
But one quirk about zig-zag merge joins is that you can run into performance issues where both sets of results are quite large, but the overlap between them is small. For example, imagine a query where you were looking for expensive restaurants that also offered counter service.
restaurants.where('price', '==', '$$$$').where('orderAtCounter', '==', 'true')
While both of these groups might be fairly large, there's probably very little overlap between them. Our merge join would have to do a lot of searching to give you the results you want.
So if you notice that most of your queries seem fast, but specific queries are slow when you're performing them against multiple fields at once, you might be running into this situation.
If you find that a query across multiple fields seems slow, you can make it performant by manually creating a composite index against the fields in these queries. The backend will then use this composite index in all future queries instead of relying on a zig zag merge join, meaning that once again this query will scale to the size of the result set.
While Cloud Firestore has more advanced querying capabilities, better reliability, and scales better than the Firebase Realtime Database, the Realtime Database generally has lower latency if you're in North America. It's usually not by much, and in something like a chat app, I doubt you would notice the difference. But if you have an app that's reliant upon very fast database responses (something like a real-time drawing app, or maybe a multiplayer game), you might notice that the Realtime Database feels… uhh… realtime-ier.
If your project is such that you need the lower latency that the Realtime Database provides (and you're anticipating that most of your customers are in North America), and you don't need some of the features that Cloud Firestore provides, feel free to use the Realtime Database for those parts of your project! Before you do, I would recommend reviewing this earlier blog post, or the official documentation, to make sure you understand the full set of tradeoffs between the two.
Remember that even in the most perfect situation, if your Cloud Firestore instance is hosted in Oklahoma, and your customer is in New Delhi, you're going to have at least 80 milliseconds of latency because of that whole "speed of light" thing. And, realistically, you're probably looking at something more along the lines of a 242 millisecond round trip time for any network call. So, no matter how fast Cloud Firestore is to respond, you still need time for that response to travel between Cloud Firestore and your device.
First, I'd recommend using realtime listeners instead of one-time fetches. This is because using realtime listeners within the client SDKs gives you a lot of really nice latency compensation features. For instance, Cloud Firestore will present your listener with cached data while it's waiting for the network call to return, giving you the ability to show results to your user faster. And database writes are applied to your local cache immediately, which means that you will see these changes reflected nearly instantly while your device is waiting for the server to confirm them.
Second, try to host your data where the majority of your customers are going to be. You have the option of selecting your Cloud Firestore location when you first initialize your database instance, so take a moment to consider what location makes the most sense for your app, not just from a cost perspective, but a performance perspective as well.
Third, consider implementing a reliable and cheap global communication network based on quantum entanglement, allowing you to circumvent the speed of light. Once you've done that, you probably can retire off of the licensing fees and forget about whatever app you were building in the first place.
So the next time you run into a Cloud Firestore query that seems slow, take a look through this list and see if you might be hitting one of these scenarios. While you're at it, don't forget that the best way to see how well your app is performing is to measure its performance out in the wild in real-life conditions, and Firebase Performance Monitoring is a great way of doing that. Consider adding Performance Monitoring to your app, and setting up a custom trace or two so you can see how your queries perform in the wild.
Analytics is the key to understanding your app's users: Where are they spending the most time in your app? When do they churn? What actions are they taking? To answer these questions, you need the right set of tools – and that’s why analytics has been a core part of Firebase since the beginning… and today, we’re excited to share with you that we’re adding more to our solution to help take app analytics to the next level!
Thanks to our continued partnership with Google Analytics, you can now upgrade your Firebase projects to the next generation of app analytics! This seamless upgrade requires no code changes and unlocks exciting new features while preserving data continuity. Now you can enjoy both the intuitive dashboards and free and unlimited event reporting you’re already enjoying in the Firebase console and all new capabilities to help you understand your user journeys in Google Analytics.
At a glance, here are some of the new capabilities and features you’ll be able to access in Google Analytics after making the upgrade:
The process for upgrading your existing Firebase project to the next generation Google Analytics experience is easy. Just follow the steps below to make the upgrade:
From that point, you’ll be able to access these new features in Google Analytics.
The upgrade will be available to all Firebase users over the coming weeks.
Yes! This upgrade enables a lot of new features and reports you will be able to access in Google Analytics, but will not affect your existing Firebase project analytics data in the Firebase console. You will be able to continue using the same dashboard and workflows you’re using today after the upgrade, but you’ll also have access to some advanced features in Google Analytics. Note that if you decide to enable cross-device reporting after making the upgrade, some user counts in your analytics data in the Firebase console may go down as the data will be de-duplicated with User-ID.
Cross-device reporting and unified analytics with User-ID will be available in both Google Analytics and the Analytics dashboard in the Firebase console.
To enable cross-device reporting for both, check out this article on implementing the User-ID in analytics. You can then update your Google Analytics settings in the Firebase console to view your reports using this property to enable cross-device reporting.
We are excited about the benefits this next generation app analytics experience will unlock for you, and look forward to hearing your feedback as you try out these new features!
The Firebase Test Lab team is pleased to announce that developers now have the ability to write Cloud Functions triggers that receive test results from Firebase Test Lab. Previously, developers had to manually check for test results, or write code to periodically poll the Test Lab API for test results. With Cloud Functions triggers, it becomes much easier to know immediately when a test finishes and get a summary of its results.
Firebase Test Lab enables you to run scripted and automated tests against your app on a wide variety of Android and iOS devices hosted in a Google data center. Cloud Functions now extends the capabilities of Test Lab by providing a fully managed backend to let developers write and deploy code that triggers when a test completes. Test Lab triggers written for deployment to Cloud Functions take the following form when using the Firebase SDK for JavaScript and deployed with the Firebase CLI:
exports.matrixOnComplete = functions.testLab.testMatrix().onComplete(testMatrix => { const matrixId = testMatrix.testMatrixId; switch (testMatrix.state) { case 'FINISHED': // Test finished with results // Check testMatrix.outcomeSummary for pass/fail break; case 'ERROR': // Test completed with an infrastructure error break; // check other possible status codes... } return null; });
You can use these triggers to programmatically notify your team of test results, for example, sending an email, posting a message to a Slack workspace, creating an issue in JIRA, as well as integrating with other team workflow tools.
Test Lab support appears in version 3.2.0 of the firebase-functions node module. Be sure to read the documentation to get more details for Test Lab triggers, and use the API reference to discover all the information available in the test matrix results. There is also a quickstart and sample code available in GitHub to help you get started. For discussion, join the Test Lab engineering team in the #test-lab channel on the Firebase Slack.
how to deal with asynchronous logic in Unity
The Firebase Unity SDK makes judicious use of asynchronous logic for many of its calls. Unity itself isn’t super resilient to threaded logic, with most of the classes and functions in the UnityEngine namespace just flat out throwing exceptions if invoked off of the main Unity thread. My goal with this post is to provide you the tools you need to not only safely use Firebase’s asynchronous function calls, but to do so in a way that best suits your own programming style and preferences. Ideally even giving you more confidence to thread other parts of your game to provide your players with the smooth and responsive gameplay they expect from a modern video game.
UnityEngine
Let’s get started with a very innocent looking demo script:
using Firebase; using Firebase.Auth; using UnityEngine; using UnityEngine.Assertions; public class FirebaseContinueWith : MonoBehaviour { // Start is called before the first frame update void Start() { Debug.Log("Checking Dependencies"); FirebaseApp.CheckAndFixDependenciesAsync().ContinueWith(fixTask => { Assert.IsNull(fixTask.Exception); Debug.Log("Authenticating"); var auth = FirebaseAuth.DefaultInstance; auth.SignInAnonymouslyAsync().ContinueWith(authTask => { Assert.IsNull(authTask.Exception); Debug.Log("Signed in!"); var successes = PlayerPrefs.GetInt("Successes", 0); PlayerPrefs.SetInt("Successes", ++successes); Debug.Log($"Successes: {successes}"); auth.SignOut(); Debug.Log("Signed Out"); }); }); } }
The first thing I do is ensure that Firebase’s dependencies are available on the player’s device with CheckAndFixDependenciesAsync. Note that I’m not really handling any failure cases in this example. This shouldn’t be an issue for this post, but you’ll want to do more than assert in your own games.
CheckAndFixDependenciesAsync
Next I use ContinueWith to create a continuation and I start signing in anonymously with SignInAnonymouslyAsync.
ContinueWith
SignInAnonymouslyAsync
When sign-in completes, I figure out how many times this script has run successfully before by reading PlayerPrefs. Then I increment this value, and write it back out before logging the new number of successes.
This is all super straightforward. I run it and… I just see the log “Signed In!” then nothing. What happened?
Firebase does a lot of work that’s dependent on I/O. This can either be out to disk, or even out to the network. Since you don’t want your game to lock up for potentially many seconds for network latency, Firebase uses Tasks to perform much of this I/O work in the background.
Whenever you continue from this work, you have to be careful to come back into your game in a graceful manner. I’ve done none of that here, and have just charged right into a shared resource managed by the UnityEngine in the form of a call to PlayerPrefs. This most likely raised an exception, but it even got lost in the background thread! What can you do to fix it?
PlayerPrefs
C# has the concept of a TaskScheduler. When you say ContinueWith, rather than just letting it continue on whatever thread the task completed on, you can use a TaskScheduler to force it onto a specific thread. So, I can modify the example to cache the TaskScheduler on which Start() was called. Then I pass that into the ContinueWith statement to be able to safely change the state of objects in my game:
TaskScheduler
Start()
Debug.Log("Checking Dependencies"); var taskScheduler = TaskScheduler.FromCurrentSynchronizationContext(); FirebaseApp.CheckAndFixDependenciesAsync().ContinueWith(fixTask => { Assert.IsNull(fixTask.Exception); Debug.Log("Authenticating"); var auth = FirebaseAuth.DefaultInstance; auth.SignInAnonymouslyAsync().ContinueWith(authTask => { Assert.IsNull(authTask.Exception); Debug.Log("Signed in!"); var successes = PlayerPrefs.GetInt("Successes", 0); PlayerPrefs.SetInt("Successes", ++successes); Debug.Log($"Successes: {successes}"); auth.SignOut(); Debug.Log("Signed Out"); }, taskScheduler); });
Since Start executes on the Unity main thread, I grab the scheduler with TaskScheduler.FromCurrentSynchronizationContext(). This way I can get back to the main thread later by passing the scheduler into my second ContinueWith statement. Now whatever work I do in that ContinueWith block will be done in sequence with the game rather than in parallel with it, preventing any threading issues.
TaskScheduler.FromCurrentSynchronizationContext()
When I run the script, I can see that I finally have one success (and that this script hasn’t succeeded before).
This pattern is really common so Firebase provides an extension method named ContinueWithOnMainThread that does all of that hard work for you. If you’re using a newer version of the Firebase Unity SDK, you can write the above as simply:
ContinueWithOnMainThread
Debug.Log("Checking Dependencies"); FirebaseApp.CheckAndFixDependenciesAsync().ContinueWith(fixTask => { Assert.IsNull(fixTask.Exception); Debug.Log("Authenticating"); var auth = FirebaseAuth.DefaultInstance; auth.SignInAnonymouslyAsync().ContinueWithOnMainThread(authTask => { Assert.IsNull(authTask.Exception); Debug.Log("Signed in!"); var successes = PlayerPrefs.GetInt("Successes", 0); PlayerPrefs.SetInt("Successes", ++successes); Debug.Log($"Successes: {successes}"); auth.SignOut(); Debug.Log("Signed Out"); }); });
A word of caution as well. In C#, you can safely assume that anything in your ContinueWith block hasn’t been garbage collected. The same doesn’t hold true with Unity’s design. If you were to access any fields of this MonoBehaviour or its encompassing GameObject after OnDestroy is invoked, you would want to check that this hasn’t become null. Due to the way Unity implemented this as well, you cannot do so with the ?? operator.
MonoBehaviour
GameObject
OnDestroy
this
??
Hopefully I’ve shed a little light on what’s happening in these tasks and continuations in Unity. You may also be a little frustrated now. What should be a simple block of code where we fix dependencies, sign on, then do work has become this ugly mess of nested statements that just becomes harder to read as we chain more steps into the logic. If only there were a better way!
Although the goal of tasks is to perform operations in parallel, so much logic in programming is sequential. Since continuations get hard to read, C# provides a mechanism in async/await syntax to represent this sequential logic. To use this mechanism, I’ll rewrite the Start method like this:
Start
async void Start() { Debug.Log("Checking Dependencies"); await FirebaseApp.CheckAndFixDependenciesAsync(); Debug.Log("Authenticating"); var auth = FirebaseAuth.DefaultInstance; await auth.SignInAnonymouslyAsync(); Debug.Log("Signed in!"); var successes = PlayerPrefs.GetInt("Successes", 0); PlayerPrefs.SetInt("Successes", ++successes); Debug.Log($"Successes: {successes}"); auth.SignOut(); Debug.Log("Signed Out"); }
The first thing you’ll notice is that I denote Start as async. This tells the C# compiler “this function will perform work in the background. Do something else whilst it finishes up.”
Then, I replace ContinueWith with the await keyword. If I were doing anything with the Task’s result, I could store the result in a variable.
await
Task
This reads much better, but why doesn’t my code break like the very first sample? It turns out that async functions will always return to the thread they’re awaited on. This way you don’t have to be as careful about thread safety in functions where you do this. In fact, by default, async functions that are awaited will typically execute on the thread that called them unless the developer explicitly did something else.
async
There is one downside compared to the ContinueWith sample though: the code following CheckAndFixDependenciesAsync will execute on the Unity main thread rather than potentially running on a background thread. In practice, this won’t be much of an issue. It could be a behaviour of note if you’re doing some significant amount of work between calls to await. Be aware as well that this code is very similar to the continuation example above. Just like how Unity may clean up your underlying MonoBehaviour before ContinueWith executes, Unity may clean it up when the call to await completes. If you access any member fields after a call to await, you should check to ensure this is not yet null.
Unity has the concept of coroutines, which used to be the preferred method of performing asynchronous work across multiple frames. The interesting bit about Coroutines is that they’re not really asynchronous, behind the scenes they simply generate IEnumerators which are evaluated on the main thread.
Unity has some special yield instructions such as WaitForEndOfFrame and WaitForSeconds, allowing you to jump around to different moments in your game’s time. I choose to implement a new CustomYieldInstruction to wait for a task to complete. I’ve even seen some developers convert something like this into an extension method on the Task class itself!
using System.Threading.Tasks; using UnityEngine; public class YieldTask : CustomYieldInstruction { public YieldTask(Task task) { Task = task; } public override bool keepWaiting => !Task.IsCompleted; public Task Task { get; } }
I can now use yield return on a Task, such as the Tasks typically used for Firebase, to make my asynchronous logic read sequentially. Inside it, I wait for a task to complete with a standard continuation. If I were actually doing something with the result of a Task, I’d have to build this class out a little more. For the time being, this will work to illustrate my basic point.
yield return
I can then reimplement my async/await logic using coroutines like this:
using System.Collections; using System.Collections.Generic; using Firebase; using Firebase.Auth; using UnityEngine; public class FirebaseCoroutine : MonoBehaviour { void Start() { StartCoroutine(DoWork()); } private IEnumerator DoWork() { Debug.Log("Checking Dependencies"); yield return new YieldTask(FirebaseApp.CheckAndFixDependenciesAsync()); Debug.Log("Authenticating"); var auth = FirebaseAuth.DefaultInstance; yield return new YieldTask(auth.SignInAnonymouslyAsync()); Debug.Log("Signed in!"); var successes = PlayerPrefs.GetInt("Successes", 0); PlayerPrefs.SetInt("Successes", ++successes); Debug.Log($"Successes: {successes}"); auth.SignOut(); Debug.Log("Signed Out"); } }
My Start function now just immediately calls a function called DoWork, which is my coroutine.
DoWork
Since I’m not doing anything with the return value of the async function calls, I just allocate temporary YieldTask objects and return them in the DoWork coroutine.
YieldTask
Now there are some important pros and cons to consider if you use this type of logic. This will have a performance hit as not only does the work between each yield return call execute on the main thread, but the property keepWaiting is queried every frame. On the other hand coroutines only exist for as long as a MonoBehaviour hasn’t been destroyed. This means that those caveats I mentioned above with having to check for null after an await or inside a ContinueWith don’t apply to coroutines!
keepWaiting
Sometimes the performance characteristics of coroutines don’t match up exactly to what you want. Remembering that a CustomYieldInstruction is queried every frame, you may end up in a state where Unity is performing many checks against against the keepWaiting property. In this case, it may be beneficial to queue these actions on Unity thread manually by adding work to a queue when it’s ready to be processed. Note that this is effectively how ContinueWithOnMainThread works and you should use that method when possible.
With that in mind, let’s look at an example of how I’ve implemented an action queue:
using System; using System.Collections.Generic; using System.Linq; using Firebase; using Firebase.Auth; using UnityEngine; using UnityEngine.Assertions; public class FirebaseQueue : MonoBehaviour { private Queue<Action> _actionQueue = new Queue<Action>(); void Start() { Debug.Log("Checking Dependencies"); FirebaseApp.CheckAndFixDependenciesAsync().ContinueWith(fixTask => { Assert.IsNull(fixTask.Exception); Debug.Log("Authenticating"); var auth = FirebaseAuth.DefaultInstance; auth.SignInAnonymouslyAsync().ContinueWith(authTask => { EnqueueAction(() => { Assert.IsNull(authTask.Exception); Debug.Log("Signed in!"); var successes = PlayerPrefs.GetInt("Successes", 0); PlayerPrefs.SetInt("Successes", ++successes); Debug.Log($"Successes: {successes}"); auth.SignOut(); Debug.Log("Signed Out"); }); }); }); } public void EnqueueAction(Action action) { lock (_actionQueue) { _actionQueue.Enqueue(action); } } void Update() { while (_actionQueue.Any()) { Action action; lock (_actionQueue) { action = _actionQueue.Dequeue(); } action(); } } }
This starts much like all of my continuation based examples. Unlike those, I call EnqueueAction to perform work on the main thread. I would highly recommend breaking this into two different MonBehaviours so you don’t forget to call EnqueueAction, but I’m compressing this for illustrative reasons.
EnqueueAction
MonBehaviour
The EnqueueAction function locks the Queue and adds some nugget of logic in the form of a C# Action into a list of logic to be executed later. If you get really clever, you may be able to just replace all of this with a lockless thread safe queue.
Finally, in Update, I execute every enqueued Action. It is very important to NOT execute the action with the _actionQueue locked. If the Action itself enqueues another Action, you’ll end up in a deadlock.
Update
Action
_actionQueue
Similar to the coroutine, this does involve checking whether or not the queue is empty every frame. Using what you’ve learned above about task schedulers and coroutines, I’m confident that you could reduce this burden with little effort if this becomes an issue.
Finally, there is a hip concept running around many programming circles known as reactive programming. Game developers I talk to tend to either love this or hate it, and there is enough public discourse that I won’t spend this post trying to turn you for or against this paradigm.
Reactive programming tends to favor logic that can come in streams -- that is logic that you would typically register for an event or query something every frame for -- and where you’ll perform functional operations on the streams as they flow through your game. For the purpose of staying consistent with the rest of this post, I’ll use it with the current example with the note that I’m not giving reactive programming its chance to shine.
So, with all that said, first I import UniRx from the Unity Asset Store. Then I have to make sure that there’s a MainThreadDispatcher in my scene:
MainThreadDispatcher
Now I can write my logic in UniRx form:
using Firebase; using Firebase.Auth; using UniRx; using UnityEngine; public class FirebaseRx : MonoBehaviour { // Start is called before the first frame update void Start() { Debug.Log("Checking Dependencies"); FirebaseApp.CheckAndFixDependenciesAsync().ToObservable().Subscribe(status => { Debug.Log("Authenticating"); var auth = FirebaseAuth.DefaultInstance; auth.SignInAnonymouslyAsync().ToObservable().ObserveOnMainThread().Subscribe(user => { Debug.Log("Signed in!"); var successes = PlayerPrefs.GetInt("Successes", 0); PlayerPrefs.SetInt("Successes", ++successes); Debug.Log($"Successes: {successes}"); auth.SignOut(); Debug.Log("Signed Out"); }); }); } }
What’s interesting about UniRx is that I can compose behaviour to form complex interactions. A brief example of this is the call ObserveOnMainThread, which guarantees that the following Subscribe executes on the main thread.
ObserveOnMainThread
Subscribe
For an example like this one, I would not pull in the complexities of UniRx but it’s useful to put it on your radar. If you were instead trying to build game logic around realtime database updates or periodically invoking cloud functions based on streams of events in game, you could do worse than combining UniRx and Zenject to quickly build a robust system around asynchronous logic.
I hope that I’ve not only given you some tools to help understand Firebase’s asynchronous API, but have empowered you to deal with them in a way that best suits your own game and coding style. I would strongly encourage you to create a small project using each of the techniques I’ve outlined here to really get a feel for the shape of each solution, and encourage you to think about ways you might improve your game’s performance using threads elsewhere. I’ve personally gotten some great mileage out of background tasks when dealing with peer to peer communication in games as well as processing and annotating screenshots without halting gameplay. I’ve even found that sometimes things like enemy AI doesn’t actually need to finish processing every frame, and it can sometimes be perfectly fine to let it run for a bit in the background over the course of a few frames.
Hey, there Firebase developers. Did you hear the exciting news? Last month at Google I/O, we announced support for collection group queries in Cloud Firestore! Let's dig into this new feature a little more, and see if we answer some of your burning questions…
Q: So, what are collection group queries and why should I care?
In Cloud Firestore, your data is divided up into documents and collections. Documents often point to subcollections that contain other documents, like in this example, where each restaurant document contains a subcollection with all the reviews of that restaurant.
In the past, you could query for documents within a single collection. But querying for documents across multiple collections wasn't possible. So, for instance, I could search for all reviews for Tony's Tacos, sorted by score, because those are in a single subcollection.
But if I wanted to find reviews for all restaurants where I was the author, that wasn't possible before because that query would span multiple reviews collections.
reviews
But with collection group queries, you're now able to query for documents across a collection group; that is, several collections that all have the same name. So I can now search for all the reviews I've written, even if they're in different collections.
Q: Great! So how do I use them?
The most important step in using a collection group query is enabling the index that allows you to run a query in the first place. Continuing our example, if we want to find all reviews that a particular person has written, we would tell Cloud Firestore, "Go index every author field in every single reviews collection as if it were one giant collection."
author
You can do this manually by going to the Firebase Console, selecting the "Index" tab for Cloud Firestore, going to the "Single Field" section, clicking the "Add exemption" button, specifying you want to create an exemption for the "reviews" collection with the "author" field and a "collection group" scope, and then enabling ascending and/or descending indexes.
But that's a lot of steps, and I tend to be pretty lazy. So, instead, I like enabling collection group indexes the same way I enable composite indexes. First, I'll write the code for the collection group query I want to use and attempt to run it. For example, here's some sample code I might write to search for all reviews where I'm the author.
var myUserId = firebase.auth().currentUser.uid; var myReviews = firebase.firestore().collectionGroup('reviews') .where('author', '==', myUserId); myReviews.get().then(function (querySnapshot) { // Do something with these reviews! })
Notice that I'm specifying a collectionGroup() for my query instead of a collection or document.
collectionGroup()
When I run this code, the client SDK will give me an error message, because the collection group index hasn't been created yet. But along with this error message is a URL I can follow to fix it.
Following that URL will take me directly to the console, with my collection group index ready to be created.
Once that index has been created, I can go ahead and re-run my query, and it will find all reviews where I'm the author.
If I wanted to search by another field (like rating), I would need to create a separate index with the rating field path instead of the author field.
rating
Q: Any gotchas I need to watch out for?
Why, yes! There are three things you should watch out for.
First, remember that collection group queries search across all collections with the same name (e.g., `reviews`), no matter where they appear in my database. If, for instance, I decided to expand into the food delivery service and let users write reviews for my couriers, then suddenly my collection group query would return reviews both for restaurants and for couriers in the same query.
This is (probably) not what I want, so the best thing to do would be to make sure that collections have different names if they contain different objects. For example, I would probably want to rename my courier review collections something like courier_reviews.
courier_reviews.
If it's too late to do that, the second best thing would be to add something like an isCourier Boolean field to each document and then limit your queries based on that.
isCourier
Second, you need to set up special security rules to support queries. You might think in my example that if I had a security rule like this:
I would be able to run this collection group query. After all, all of my review documents would fall under this rule, right? So why does this fail?
Well if you've seen our video on Cloud Firestore security rules, you would know that when it comes to querying multiple documents, Cloud Firestore needs to prove that a query would be allowed by the security rules without actually examining the underlying data in your database.
And the issue with my collection group query is that there's no guarantee it will only return documents in the restaurants → reviews collection. Remember, I could just as easily have a couriers → reviews collection, or a restaurant → dishes → reviews collection. Cloud Firestore has no way of knowing unless it examines the results of the data set.
So the better way to do this is to declare that any path that ends with "reviews" can be readable based on whatever security rules I want to implement. Something like this:
Note that this solution requires using version 2 of the security rules, which changes the way recursive wildcards work.
Third, keep in mind that these collection group indexes are counted against the 200 index exemptions limit per database. So before you start creating collection group indexes willy-nilly, take a moment and ask yourself what queries you really want to run, and just create indexes for those. You can always add more later.
Q: Can I do collection group queries for multiple fields?
Yes. If you're doing equality searches across multiple fields, just make sure you have an index created for each field with a collection group scope.
If you're combining an equality clause with a greater-than-or-less-than clause, you'll need to create a composite index with a collection group scope. Again, I find it's best to just try to run the query in the code and follow the link to generate the index. For instance, trying to run a collection group query for all reviews that I wrote with a rating of 4 or higher gave me a URL that opened this dialog box.
Q: It still seems like I could do all of this in a top-level collection. How are collection group queries better?
So this question is based on the idea that one alternative to creating collection group queries is to not store data hierarchically at all, and just store documents in a separate top level collection.
For instance, I could simply keep my restaurants and my reviews as two different top-level collections, instead of storing them hierarchically.
With this setup, I can still search for all reviews belonging to a particular restaurant…
As well as all reviews belonging to a particular author…
And you'll notice that with the separate top level collection, I no longer need to use one of my 200 custom indexes to create this query.
So, why go with the subcollection setup? Are collection group queries needed at all? Well, one big advantage to putting documents into subcollections is that if I expect that I'll want to order restaurant reviews by rating, or publish date, or most upvotes, I can do that within a reviews subcollection without needing a composite index. In the larger top level collection, I'd need to create a separate composite index for each one of those, and I also have a limit of 200 composite indexes.
Also, from a security rules standpoint, it's fairly common to restrict child documents based on some data that exists in their parent, and that's significantly easier to do when you have data set up in subcollections.
So when should you store things in a separate top level collection vs. using subcollections? If you think you have a situation where you're mostly going to be querying documents based on a common "parent" and only occasionally want to perform queries across all collections, go with a subcollection setup and enable collection group queries when appropriate. On the other hand, if it seems like no matter how you divide up your documents, the majority of your queries are going to require a collection group query, maybe keep them as a top level collection.
But if that's too hard to figure out, I would say that you should pick the solution that makes sense to you intuitively when you first think about your data. That tends to be the correct answer most of the time.
Hope that helps you get more comfortable with collection group queries! As always, if you have questions, feel free to check out our documentation, or post questions on Stack Overflow.
