Yesterday the OpenSSL Project released an update to address the "Heartbleed" vulnerability. This is a serious vulnerability that allows an attacker to see the contents of memory on a server. Given the widespread use of OpenSSL and the versions affected, this vulnerability affects a large percentage of services on the internet.
Although the Firebase realtime servers were unaffected by this exploit, our servers managing authentication to the Firebase website were vulnerable. This could have allowed an attacker to obtain authentication credentials while they were in-flight. Internally we protect passwords by hashing them using bcrypt, but that precaution did not help given the nature of this exploit.
Once the exploit was revealed our infrastructure team responded immediately and all Firebase services were secured the same day, by 11pm PDT on Monday, April 7. Firebase is no longer vulnerable to this exploit.
We do not have any evidence that passwords or any other private information has been compromised. However, given that this exploit existed in the wild for such a long time, it is possible that an attacker could have stolen passwords without our knowledge. As a result, we recommend that all Firebase users change the passwords on their accounts. We also recommend that you reset your Firebase Secrets, which you can do from the "Secrets" tab in your App Dashboard.
The safety and security of our customer data is our highest priority. We are continuing to monitor the situation and will be responding rapidly to any other potential threats that are discovered.
If you have any questions or concerns, please email us directly at security@firebase.com.
We’re excited to announce that the Firebase REST API now supports streaming! This means you can add realtime notifications to your Firebase app on currently unsupported platforms. We’ll outline how to get started with the new streaming protocol, and then we’ll dive into some Ruby and Python examples that make use of streaming.
Streaming changes to a location in your Firebase is the equivalent of attaching a single listener to a Firebase reference on any of our client libraries. The new streaming feature makes use of the EventSource / Server-Sent Events protocol, an API for creating an HTTP connection for receiving push notifications from a server.
To stream changes to a single location in your Firebase, you’ll need to set the client’s Accept header to “text/event-stream”.
You’ll also need to be sure your app respects HTTP Redirects, specifically HTTP status code 307. The Firebase servers will direct your client to the correct machine for the data that you want to stream via a URL specified in the Location header of an HTTP 307 response.
If the Firebase location you’re streaming changes to requires permission to read, you must include the auth parameter. The auth parameter is supported by all request types, and authenticates a request to allow access to data protected by your Security and Firebase Rules. The argument can either be your Firebase Secret or an authentication token. Here is an example:
curl \ https://SampleChat.firebaseIO-demo.com/users/jack/name.json?auth=CREDENTIAL
As the data at your Firebase location changes, the server will send named events in the following structure:
event: event name data: JSON encoded data payload
The structure of these messages conforms to the EventSource protocol. The server may send the following five event types: put, patch, keep-alive, cancel, or auth_revoked. Here is one example of a set of events returned from the server:
put
patch
keep-alive
cancel
auth_revoked
// Set your entire cache to {"a": 1, "b": 2} event: put data: {"path": "/", "data": {"a": 1, "b": 2}} // Put the new data in your cache under the key 'c', so that the complete cache now looks like: // {"a": 1, "b": 2, "c": {"foo": true, "bar": false}} event: put data: {"path": "/c", "data": {"foo": true, "bar": false}} // For each key in the data, update (or add) the corresponding key in your cache at path /c, // for a final cache of: {"a": 1, "b": 2, "c": {"foo": 3, "bar": false, "baz": 4}} event: patch data: {"path": "/c", "data": {"foo": 3, "baz": 4}}
Check out the Streaming documentation for a full explanation of each event type.
We’ve created a simple command line chat client in Python and Ruby to demonstrate the REST API’s new streaming feature. Both chat clients have the same architecture:
A thread to manage the UI
Simply clone the EventSource-Examples repo on GitHub to get started.
We encourage you to try out the new streaming feature and let us know what you think. If you have any questions or feedback, post them in our Firebase Google Group or email me at greg@firebase.com. Submit a pull request to our example repo if you have any streaming examples you’d like to share.
We’re excited to see what you create!
Just in time for EmberConf next week, we’re excited to announce a new version of our EmberFire bindings, along with an example blogging app to showcase the magic of using Firebase with Ember.
Our new Ember bindings now work directly with Ember Data. This means you can now use Ember Data’s relationships with Firebase to define hasMany and belongsTo relationships between your models. If you’re new to Ember Data, Ember’s guide to models is a great place to start.
With these updates, EmberFire now works only with Ember Data. For Ember developers who aren’t using Ember Data, Michael Jackson has written ember-firebase - an awesome, thoroughly tested set of Firebase bindings for Ember. If you’d still like to use the previous version of EmberFire, it is available in the v0.1.0 release on GitHub.
With our new bindings, you can start using EmberFire in just two steps. First, you’ll need to include the necessary libraries:
<head> <!-- Ember + Ember Data --> <script src="http://builds.emberjs.com/canary/ember.min.js"></script> <script src="http://builds.emberjs.com/canary/ember-data.min.js"></script> <!-- Firebase --> <script src="https://cdn.firebase.com/js/client/1.0.17/firebase.js"></script> <!-- EmberFire --> <script src="https://cdn.firebase.com/libs/emberfire/1.0.13/emberfire.min.js"></script> </head>
You can also download EmberFire and all its dependencies from Bower via bower install emberfire --save.
bower install emberfire --save
Then, simply create an instance of the DS.FirebaseAdapter in your app and add your Firebase URL:
DS.FirebaseAdapter
App.ApplicationAdapter = DS.FirebaseAdapter.extend({ firebase: new Firebase('https://<my-firebase>.firebaseio.com') });
With the adapter set up, you can now interact with the data store as you normally would with Ember. For example, calling find() with a specific ID will retrieve that record from Firebase. It will also start watching for updates and will update the data store automatically whenever anything is added or removed.
To help you understand our new EmberFire bindings we’ve created FireBlog, a realtime blogging app with posts, comments, and users. We’ll walk through the core concepts here, and the full code is available on GitHub.
Because both Ember Data and Firebase work best with denormalized data, FireBlog does not embed comment data directly in each post. Instead, each post has a comments object with a comment ID pointing to the comment data in /comments. We’ve created three top-level keys for posts, comments, and users, and the data structure in Firebase looks like this:
/comments
Each post in our blog will have a title, body, published, user, and comments attributes. Using Ember Data’s relationships, each post belongsTo a user and hasMany comments. We’ll use Ember’s computed properties to format the date to display on the blog post.
App.Post = DS.Model.extend({ title: DS.attr('string'), body: DS.attr('string'), published: DS.attr('number'), publishedDate: function() { return moment(this.get('published')).format('MMMM Do, YYYY'); }.property('published'), user: DS.belongsTo('user', { async: true }), comments: DS.hasMany('comment', { async: true }) });
The comment and user models are set up in a similar way, which you can see on GitHub.
To get all blog posts from Firebase we can use the standard Ember Data way of fetching posts, like the following:
App.PostsIndexRoute = Ember.Route.extend({ model: function() { return this.store.findAll('post'); } });
This will fetch all posts when the page first loads, and will also listen for updates to Firebase. Whenever posts are added, the data store will update automatically and new posts will be displayed in the DOM.
Similar to how we fetched all posts using store.findAll(), we’ll fetch individual posts using store.find():
store.findAll()
store.find()
App.PostRoute = Ember.Route.extend({ model: function(params) { return this.store.find('post', params.post_id); } });
We'll use the following template to display a single post. Then we'll create an Ember.Component to display individual blog posts. We'll pass in the model from the route (post=model) as a property to show the post's title, author, date, and comments:
Ember.Component
<script type="text/x-handlebars" data-template-name="post"> {{ "{"}}{{ "{"}}fire-post post=model onPublishComment="publishComment"}} </script>
Here is the App.FirePostComponent we use to display an individual blog post:
App.FirePostComponent
<script type="text/x-handlebars" data-template-name="components/fire-post"> <div class="post-header"> <img {{ "{"}}{{ "{"}}bind-attr src="post.user.avatar"}} class="avatar post-avatar"/> <div class="post-author">{{ "{"}}{{ "{"}}post.user.id}} <div class="post-date">{{ "{"}}{{ "{"}}post.publishedDate}} <h3 class="post-title">{{ "{"}}{{ "{"}}post.title}}</h3> <div class="post-content">{{ "{"}}{{ "{"}}breaklines post.body}} <ul class="post-comments"> {{ "{"}}{{ "{"}}#each comment in post.comments}} <li class="post-comment"> <img {{ "{"}}{{ "{"}}bind-attr src="comment.user.avatar"}} class="avatar post-comment-avatar" /> <div class="post-comment-body"> <div class="post-comment-meta"> <strong class="post-comment-author">{{ "{"}}{{ "{"}}comment.user.username}}</strong> <em class="post-comment-date">{{ "{"}}{{ "{"}}comment.publishedDate}}</em> {{ "{"}}{{ "{"}}breaklines comment.body}} </li> {{ "{"}}{{ "{"}}/each}} </ul> </script>
Now we have a full-featured blogging application that displays posts and comments in realtime without a server. It’s that simple!
Our goal with the new EmberFire library is to make the process of adding a backend to your Ember app and integrating with Ember Data as seamless as possible. We encourage you to take our new bindings for a spin. Submit a pull request or email me at adam@firebase.com with any feedback. We’re just getting started with this new Ember Data integration, and we’re working hard to make updates based on your suggestions. We’re excited to see what you build with Firebase and Ember.
For those of you going to EmberConf, look for James, Alex, Sara and I in the bright yellow t-shirts!
Tom Larkworthy is the developer of Firesafe, a language wrapper for Security and Firebase Rules. Firesafe eases development of complex protocols.He cares about logical perfection, and finds holes in Firebases using formal methods.
Data integrity is hard to scale in databases. Issues surface quickly in massively multiplayer games, where the players are incentivized to cheat. Firesafe is a language for expressing and checking protocols to solve data integrity problems. In this post, I show how sending an item between players can be implemented safely and deadlock free with a protocol developed using Firesafe.
Data integrity is very hard to manage in scalable, realtime applications. Most massively multiplayer games have been hit with an item duplication bug at some point. This type of bug allows users to clone rare items, consequently trashing the virtual economy with deflation, ruining the reward structure and costing the hosting company millions in lost revenue. If not managed, the virtual world can be damaged for years. This has affected popular games in the past, including World of Warcraft, Runescape, Farmville, and many others.
The dual of the duplication bug is the bug where items disappear. The loss of an item flushes a player's efforts down the drain, makes them very angry, and usually causes them to call support. The major cost in massively multiplayer games is not the operational cost of running servers, but the cost of answering thousands of support requests. Data integrity problems are a major cause of support requests.
These problems affect big companies with big budgets, and these bugs are costing them millions. So why don't companies just develop working code?
The reason why these bugs are widespread is because of a common underlying technical issues. Essentially, it is difficult to ensure that transactions always work correctly in a distributed system (called ACID compliance). ACID compliant transactions are even harder to implement when servicing a real time system.
Firesafe is a language for expressing state machine dynamics that is compiled into Security and Firebase Rules. State machines are a natural programming abstraction used extensively in the telecommunications industry to solve similar hard problems. State machines are attractive for formal verification because their finite sized state space makes enumerating their total behavior tractable.
I have developed a system that verifies the correctness of protocols implemented with Firesafe. This post is a demonstration of one possible use of this technology: the development of a protocol for ACID-like transfer of items between users. Item transfers are a canonical hard dynamic to get right in multiplayer games.
A lot of bugs in multiplayer games are centered around the trading dynamic, when one player sends an item to another player. What should happen is one player loses an item, and another player gains the item. Often this is not the case.
Common errors are both players end up with the item, or both lose the item. It sounds like an easy feature to develop, but the player accounts might be on different computers, and a player could (deliberately) disconnect from the game at any point.
Distributed exchange of state is a similar problem banks have when performing an electronic transfer between financial institutions. In developing a solution for Firebase, I have drawn from the bank's solution of a 2 phase commit protocol (2PC).
In a 2PC, a central authority coordinates the a transaction between party A and B. In this approach, it is critical that the coordinator waits until both A and B have indicated they are willing to commit before sending a message that the transfer is going ahead (or not). Before the commit message, either party can back out the transaction or go offline, in which case the coordinator must announce that the transfer is aborted through a message.
A and B have to wait for the coordinators decision, which is the main issue with 2PC.It is safe, but it blocks if the coordinator fails. There are also problems if a coordinators commit message goes missing. As we will see, due to some of the guarantees provided by Firebase, many of the main issues of 2PC disappear.
Firebase is like no other database. It is completely scalable and realtime, but has no stored procedures and all data must be in a tree structure. It only supports transactions on sub-trees, and it enforces data integrity through a custom security language, which is less expressive than traditional SQL conditions.
It is also a work of genius.
While Firebase may seem like a long list of restrictions, it is clear that every restriction is very well thought out and provides an opportunity for blazing realtime performance and scalability. So Firebase is a hot rod, and you don't even have to host anything! The Firebase team deals with expensive operational issues, which democratizes access to top DB performance.
That said, the drawback is that ensuring data goes in and out the database properly is more difficult, as there is no opportunity to run secured arbitrary application level code before remote clients access the database layer. You can only manage data integrity with the security functions Firebase provides. In the case of transferring data from one user account to another, one problem is that Firebase does not support transactions across sibling branches.
An often touted route around Firebase's limitations is to add an additional privileged server to provide additional functionality. However, that is not ideal, as now you have to manage those servers, and they also add additional latency.
The true way for item transfer, in keeping with the Firebase philosophy, is to have clients carry out the transactions. By leveraging atomic in-branch updates, plus out-of-branch referencing in Security and Firebase Rules, it is possible to sequence concurrency safe protocols. However, this requires carefully thought out Security and Firebase Rules to ensure the system isn't open to exploitation.
Firesafe is a language that lets us encode state machines with managed variables inside a Firebase. It provides hooks to limit how variables are modified based on the machines current state. This is a natural way of expressing concurrent protocols, and allows developers to concentrate on the "clever bits" of the the protocol without the error prone drudgery of managing state transitions.
Inspired by the banking sector, I’ve designed a protocol that lets one player send an item to another player. This post gives a basic overview of a two-phase commit example; for a full explanation with code samples, check out the github wiki.
Note that each player account resides on a different branch of user. So we can't use Firebase's subtree transactions to make it atomic.
This kind of transaction is business critical. Either it works, or the transaction is rolled back to the original state. No dupes. No losses. Furthermore, we want this system to be deadlock free. If one player goes offline, it should not hold the other player in limbo.
I have taken the classic 2PC protocol and adapted it for Firebase. Firebase is only able to verify data with boolean clauses, it can't actually move data around itself, and it can't emit messages like the classic central authority.Thus, the connected clients have to initiate the changes of state, with Firebase simply double checking the validity.To get safe transfers, we have to logically force the clients down a predefined set of state transitions that can only lead to transfer of the item or an aborted and rolled back transfer.
An sketch of a successful integration of the two players is shown below.
The ordered stages of the protocol is:
We need a number of variables for each user:
We then need to add Security and Firebase Rules to ensure a client can only alter those variables in a way that leads to our item transfer dynamics. To avoid deadlocks, a player should be able to push an authorised transfer through, even if the other party goes offline. The permissions have to be carefully set to avoid one player authorising the transfer on the other player's behalf. Either player should be able to abort before confirmation.
The whole protocol, including deadlock free rollbacks even if one player goes offline, is shown below:
The Firesafe source HSM representation is here, the compiled rules file is here.
Firesafe's HSM representation is 50% less lines of code, and a large portion of that is variable and state declarations. The Firesafe language lets you concentrate on the guards and effects whilst alleviating the burden of enforcing state switching. There are features of the HSM language we have not mentioned in this post. HSM supports Hierarchical State Machines which is a feature that reduces space explosion by inheritance.
When I first considered developing sibling transactions on Firebase, I did not know deadlock-free ACID compliant transfers would be possible. Developing this protocol was my own personal test of Firebase's metal. With Firebase passing this test, I am fairly sure you can do anything on Firebase without involving third party servers.
Getting data integrity right is tough. I was only sure I had it right after putting it through formal verification (which did indeed find lots of holes in my original logic). Applying formal methods to Firebase semantics is not straight forward yet. You need to ask yourself this question, if a malicious user exploits your database, can it financially ruin you, or nuke your product? If the answer is yes, then test driven development is not a robust enough methodology, you need formal verification.
If you are interested in having Tom verify your app’s security design, you can email him at tom.larkworthy@gmail.com to inquire about this kind of service. Stay up to date on his verification technology here.
James, Andrew, Sara and I are back from ng-conf -- and we all had a blast! The conference was very well put together (kudos to Domo) and it was inspiring to speak with luminaries from across the Angular community in person. Just in time for ng-conf, we released a new version of AngularFire (0.6), deployed improvements to our account dashboard, and announced a new Firebase + Angular quickstart guide. I was also excited to demo on stage how easily you can wire up a backend to your Angular app using Firebase.
Watch this video of my talk on building realtime apps with Firebase and Angular:
The conference started off with an amazing keynote by the creator of Angular himself, Miško Hevery, along with Brad Green from Google. They gave a bit of history behind Angular, how it came to be, where they are today and what their plans are for the future, including how they want to help the community extend the Angular feature set. We were extremely flattered when Miško said in the keynote that Firebase “completes the original vision” of Angular:
After a long day of excellent talks by several preeminent speakers, it was time for the Firebase sponsored party. The party featured a comedian, magician and an impromptu comedy troupe. A spread of desserts was accompanied with our very own Firebase chocolate. We hope everyone had a great time - conferences are so much better when they’re mixed in with a little live entertainment!
We gave away tons of Firebase swag: T-shirts, light-up bracelets, stickers, and of course, our very own hot sauce! We were very happy with all the interest in Firebase, and were delighted to meet everyone who came to our booth with questions, compliments or feedback.
We wrapped up the conference with a day on the slopes with many other ng-conf attendees. It’s one of the best conferences we’ve participated in so far, and we’re definitely looking forward to the next one!
