Scheduled Cloud Function Tasks and Cron Jobs in Firebase

In this lesson, we’re going to create our own time-based Cloud Functon Task Scheduler, inspired by Sidekiq and DelayedJob from the Ruby world. It works by creating a task queue, then uses a cron-job to run the tasks every 60 seconds (time granularity can be adjusted).

Cron-jobs are a highly requested feature for Firebase. Abe Haskins has a solid article about cron jobs in Firebase, but this approach is static and limited to tasks known by the developer in advance. You manually configure the cron schedule in cron.yml and deploy it. The goal of this solution is to provide dynamic task scheduling that can be managed both clientside and serverside based on user interaction. Let’s consider some uses cases:

• Scheduled or snooze-able reminder notifications
• Send a happy birthday push notification
• Send welcome email 24 hours after signup
• Retry tasks if they fail due to errors

Here’s what our API will be able to handle from the client side.

• performAt schedules a task at a specific time.
• performIn schedules a task relative to the current time.
• performPeriodic runs a recurring task at a specific interval.

The server will regularly pull from the task queue and execute jobs that are ready to run.

Task Runner Basics

Our task queue has three parts - client (Angular), task queue (RealtimeDB), and server (Cloud Functions).

The basic process goes like this

1. Task is enqueued by writing to the RealtimeDB
2. Endpoint is pinged every 60s, invoking the TaskRunner.
3. All tasks where time < Date.now() will be executed, then removed from the queue (or retried in the case of errors).

The Client (Angular)

We need our client to reliably enqueue and schedule tasks in the background. This boils down to a service that writes some data to the Realtime DB that can be used in the future by a Cloud Function to execute a task. I am assuming you have the Angular CLI and AngularFire2 installed in your project.

ng generate service task

The Task Queue (Realtime DB)

This is a situation where RealtimeDatabase is superior to Cloud Firestore. Why? Our task queue is will receive frequent read/writes, but have a very small storage footprint.

Our task data structure looks like this:

tasks
-- taskID
---- time: number (when to run)
---- interval: number (when to reschedule, only needed for periodic jobs)
---- worker: string (worker function name)
---- opts: object (add extra data you might need at runtime)

A basic task should look like this in the database:

The Server (Cloud Functions)

Our main server infrastructure is an HTTP Firebase Cloud Function, let’s call it our TaskRunner. It will pull items from the queue every 60 seconds and execute them.

The task object contains a worker string property, which is the name of a plain JS function that you define with the code to be executed, i.e. send an email. You can also pass in custom data via opts if needed at runtime.

We need to ping the TaskRunner to look for existing tasks. There are many ways to handle this, but here are the two most common approaches.

Plan A. Use cron-job.org. It’s free and easy, but limited to 60s ping intervals.

Plan B. If you want to ping more often you can deploy an instance to App Engine Standard (I’d use GO or Python).

Angular Task Service API

Let’s build a reusable service to enqueue tasks on the client.

import { Injectable } from '@angular/core';
import { AngularFireDatabase } from 'angularfire2/database';

@Injectable()
export class TaskService {

  constructor(private db: AngularFireDatabase) { }

  // Methods go here
}

Keep in mind that task data needs to be serialized, so only pass data in primitive fom, like plain objects, strings, numbers, etc. You can always query raw data from the Cloud Function.

Relative Time

Let’s imagine you want to perform a task 30 minutes from now. This method will add the the interval to UTC time.

• time number of milliseconds to add to current UTC time.
• worker the name of the worker function to invoke (more on this in the next section).
• opts optional custom data to use in the function at runtime.

performIn(time: number, worker: string, opts = {}) {
  const queueRef = this.db.list('tasks'); 

  time = Date.now() + time
  queueRef.push({ time, worker, opts });
}

Exact Time

Another use case is to schedule a task at a specific time. It works exactly the same, but expects a timestamp for a specific future date.

Cloud Functions run on UTC time. Make sure to convert the task time to UTC for consistent results, typically by using new Date(...).getTime()

performAt(time: number, worker: string, opts = {}) {
  const queueRef = this.db.list('tasks');
  
  queueRef.push({ time, worker, opts });
}

Recurring Periodic Tasks

Periodic jobs introduce a number of extra challenges. I highly recommend that you pass an ID as an argument to prevent duplicate recurring jobs from polluting the task queue. For example, userID_weeklyNewsletter would make a good task ID and prevent your user from accidentally receiving multiple weekly newsletters.

When this job runs successfully, we will add the interval to the original timestamp in the Cloud Function task runner.

performPeriodic(id: string, time: number, interval: number, worker: string, opts = {}) {
  const taskRef = this.db.object(`tasks/${id}`);
  
  taskRef.set({ time, interval, worker, opts });
}

Did you know that Cloud Functions will timeout after 1 minute? Keep that in mind if you’re performing many resource intensive background tasks.

Usage

Now that we have a task service built, let’s look at how it could be used in a component. I recommend setting up helper functions for time (or use something like MomentJS) to make your code readable.

this.taskService.performIn(minutes(30), 'sendWelcomeEmail', { user: uid } )


function minutes(v: number) {
   return v * 60 * 1000;
}

Cloud Function Task Runner

Think of the TaskRunner like a cron job, but one that you can manage dynamically with code. It works by querying the RealtimeDB queue for any tasks that have a timestamp less than the current time. We collect all of the tasks, then execute them concurrently using Promise.all.

All worker functions are just plain JS (not Firebase functions) and they must return a Promise. We attach these functions to an object because that allows us to call them with a string representation of the function name.

Each task will be executed and if successful, it will be removed from the queue. If a task fails, its failures count will be incremented and the error message will be saved to the object to help with debugging.

Initialize Functions

Run the following command to initialize cloud functions and make sure to use TypeScript.

firebase init functions

Full Task Runner Function Code

The code below demonstrates how to execute pending tasks and handle errors. All you need to do is create some meaningful worker functions that perform a useful job based some of the use cases we discussed previously.

admin.initializeApp();

const db = admin.database()

exports.taskRunner = functions.https.onRequest(async (req, res) => {
   // ... pro only
})

const workers = {
  // ... pro only
}

import * as functions from 'firebase-functions';
import * as admin from 'firebase-admin';
admin.initializeApp();

const db = admin.database()

/// WORKERS ///

const workers = {
    testTask,
    sendWelcomeEmail,
}

async function testTask(task) {
    console.log('test worker executed')
}

async function sendWelcomeEmail(task) {
    console.log('email sent')
}

/// TASK RUNNER CLOUD FUNCTION ///

exports.taskRunner = functions.https.onRequest(async (req, res) => {

    const queueRef = db.ref('tasks');
    // Get all tasks that with expired times
    const tasks = await queueRef.orderByChild('time').endAt(Date.now()).once('value')


    if (tasks.exists()) {
        const promises = []
        
        // Execute tasks concurrently
        tasks.forEach( taskSnapshot => {
            promises.push( execute(taskSnapshot) )
        })

        const results = await Promise.all(promises)
        
        // Optional: count success/failure ratio
        const successCount = results.reduce(sum);
        const failureCount = results.length - successCount;
            
        res.status(200).send(`Work complete. ${successCount} succeeded, ${failureCount} failed`)
        
    } else {

        res.status(200).send(`Task queue empty`);
    }



});

/// HELPERS

// Helper to run the task, then clear it from the queue
async function execute(taskSnapshot) {

    const task = taskSnapshot.val();
    const key = taskSnapshot.key;
    const ref = db.ref(`tasks/${key}`);

    try {

        // execute worker for task
        const result = await workers[task.worker](task)
        
        // If the task has an interval then reschedule it, else remove it
        if (task.interval) {
            await ref.update({ 
                time: task.time + task.interval,
                runs: (task.runs || 0) + 1
            })
        } else {
            await ref.remove();
        }
       
        return 1;  // === success

    } catch(err) {
        // If error, update fail count and error message
        await ref.update({ 
            err: err.message,
            failures: (task.failures || 0) + 1
        })

        return 0; // === error
    }
}

// Used to count the number o fail
function sum(acc, num) {
    return acc + num;
}

Now deploy the function and make a note of the deployed URL endpoint.

firebase deploy --only functions

Setting Up a Cron Job

At this point, we need a way for our task runner to be invoked at regular intervals. I will be using cron-job.org because it is dead simple. Just register for an account and pass it the deployed TaskRunner function URL.

The End

You now have a dynamic workflow for enqueuing and executing background tasks with Firebase Cloud Functions at specific moments in time. There are many additional considerations we should think about, such as scaling and fine-tuning the time granularity, but this is a good start. Let me know if you have questions in the comments or reach out on Slack.