If you're like most app developers, you know that small changes can often make a big difference in the long term success of your app. Whether it's the wording that goes into your "Purchase" button, the order in which dialogs appear in your sign-up flow, or how difficult you've made a particular level of a game, that attention to detail can often make the difference between an app that hits the top charts, or one that languishes.
But how do you know you've made the right changes? You can certainly make some educated guesses, ask friends, or run focus groups. But often, the best way to find out how your users will react to changes within your app is to simply try out those changes and see for yourself. And that's the idea behind A/B testing; it lets you release two (or more!) versions of your app simultaneously among randomly selected users to find out which version truly is more successful at getting the results you want.
And while Firebase Remote Config did allow you to perform some simple A/B testing through its "random percentile" condition, we've gone ahead and added an entirely new experiment layer in Firebase that works with Remote Config and notifications to make it quick and easy to set up and measure sophisticated A/B tests. Let's take a quick tour of how it works!
With the new A/B testing feature, you can create an A/B test that will allow you to play with any combination of values that you can control through Remote Config. Setting up an A/B test allows you to define how the experiment will behave in a number of different ways, including determining how many of your users are involved with the experiment at first…
…how many variants you want to run, and how your app might behave differently for each variant…
...and what the goal of the experiment is.
Different experiments might have different desired goals, and A/B testing supports a number of common outcomes, like increasing overall revenue or retention in your app, reducing the number of crashes, or increasing the occurrence of any event you're measuring in Google Analytics for Firebase, such as finishing your in-app tutorial.
Once you've defined your A/B test, Firebase takes over by delivering these different variations of your app to randomly-selected members of your audience. Firebase will then measure your users' behavior over time, and let you know when an experiment appears to be performing better, based on those goals you've defined earlier. Firebase A/B testing measures these results for you with the same Bayesian statistical models that power Google Optimize, Google's free testing and personalization product for websites.
Fabulous, a motivational app for building better habits, recently made improvements to their app's onboarding flow by using Firebase A/B testing. When the user first starts an app, Fabulous shows them how to complete a habit, presents them with a letter about forming better habits, and then asks them to commit to a simple routine. The team suspected that if they removed a few steps from this onboarding process, more people might complete it.
Some of the screens a typical user encounters when first using Fabulous
So they ran an A/B test where some users didn't see the letter, others didn't see the request to commit to a simple routine, and others skipped both of those steps. The Fabulous team found that by removing both of these steps from the onboarding process, there was a 7% improvement in the rate of users completing the onboarding flow. More importantly, they confirmed that this shorter onboarding experience didn't have any negative impact on their app's retention.
You also have the ability to A/B test your app notification messages through the Firebase Notifications console. You can try out different versions of your notification message and see which ones lead to more users opening up your app from that notification, or which messages lead to users performing some intended goal within your app, like making a purchase.
A/B testing is available in beta to all Firebase developers starting today. If you're excited to get started, you should make sure that your app is wired up to use Remote Config and/or Firebase Cloud Messaging, and that you've updated these libraries to the latest and greatest versions. You can always find out more about A/B testing in our documentation, or check out the A/B Test Like a Pro video series we've been building.
Then, head on over to the Firebase Console and start making your app better — one experiment at a time!
