Many of our developers are using Firebase with a server to perform tasks like background processing, integrating with third party APIs, or handling advanced authentication requirements. Today, we're introducing Firebase Queue, a fault-tolerant multi-worker job pipeline built on Firebase.
If you're writing your server code in Node, Firebase Queue makes it easy to handle background jobs with a worker queue. We're already using it in our private backups feature, where we reliably handle hundreds of jobs per day across multiple machines.
Once you've installed firebase and firebase-queue, you can get started by creating a new Queue and passing it your Firebase reference and a processing function. This processing function specifies the data transformation that the worker performs and when called, will be passed the following four parameters:
data
progress()
resolve()
reject()
Below is a simple example of what an instance of the queue looks like:
var Queue = require('firebase-queue'), Firebase = require('firebase'); var queueRef = new Firebase('https://<YOUR-FIREBASE-APP>.firebaseio.com/queue'); var queue = new Queue(queueRef, function(data, progress, resolve, reject) { // Read and process task data console.log(data); // Update the progress state of the task setTimeout(function() { progress(50); }, 500); // Finish the job asynchronously setTimeout(function() { resolve(); }, 1000); });
When your Queue runs, the data will be stored in Firebase under a /queue/tasks path and removed when complete. For an item in your queue called example1, the data in Firebase would look like the following:
/queue/tasks
example1
{ "queue": { "tasks": { "example1": { "_owner": "worker_1", // The worker process who owns this task "_progress": 88, // A number from 0 to 100 indicating the item’s status in the queue "_state": "in_progress", // Indicates whether the job has errored, is in progress or finished "_state_changed": 1430937731946, // Timestamp when the state was last updated "your": "data" // Original data in your task } } } }
You can push an object with some data to the /queue/tasks location in your Firebase using any Firebase client or the REST API. Workers listening at that location will automatically pick up and process the job:
Firebase ref = new Firebase("https://<YOUR-FIREBASE-APP>.firebaseio.com/queue/tasks"); Map task = new HashMap(); task.put("your", "data"); ref.push().setValue(task);
Firebase *ref = [[Firebase alloc] initWithUrl:@"https://<YOUR-FIREBASE-APP>.firebaseio.com/queue/tasks"]; [[ref childByAutoId] setValue:@{@"your": @"data"}];
var ref = Firebase(url: "https://<YOUR-FIREBASE-APP>.firebaseio.com/queue/tasks") ref.childByAutoId().setValue(["your": "data"])
var ref = new Firebase("https://<YOUR-FIREBASE-APP>.firebaseio.com/queue/tasks"); ref.push({"your": "data"});
curl -X POST -d '{"your": "data"}' https://<YOUR-FIREBASE-APP>.firebaseio.com/queue/tasks.json"
If you don't want to use the default spec, you can define your own. You can also chain several specs together to create custom jobs. More details on customizing and chaining jobs can be found in the Firebase Queue README.
We have a few examples to help you get started with Firebase Queue. This gist creates a new Queue with 10 workers and a custom task task_1. In this example, the queue sanitizes chat messages to make sure they don’t contain any banned words.
task_1
When you try out firebase-queue in your app I'd like to hear what you think. Send a pull request, file an issue, or reach out to me on Twitter @drTriumph.
