Multi-Property Data Filtering With Firebase and Angular 4

Multi-property data filtering an array data is a common need in web applications. In this lesson, I outline the best options for filtering your data when you want to achieve something similar to using multiple WHERE conditions in a SQL database query.

Let’s be perfectly clear… Firebase is somewhat limited when you need to filter your data on multiple conditions server-side.

The Realtime Database can only filter one property value at a time!

Consider the following example.

db.list('/animals', {
  query: {
    orderByChild: 'family',
    equalTo: 'fish'
  }
});

That’s the best we can do (out of the box) when filtering by values on the server. But don’t worry yet, I’m going to share four potential solutions.

Option 1 - Client Side Filtering

You can only filter by one value at a time in Firebase, but you can add additional client-side filtering all other parameters as long as you can fit the data into memory.

Pros

• Best performance
• Unlimited flexibility

Cons

• May not be possible with large datasets

Example of Advanced Client-Side Data Filtering

In this example, I subscribe to a FirebaseListObservable of animals, which emits an array of objects, then use lodash to remove objects based on data filtering rules. We can filter javascript objects with multiple rules by combining filter() and conforms(). Conforms takes an object of functions (our filters) that evaluate to true or false. This allows us to create reusable functions that solve common filtering problems, such as exact string matching, less-than, greater-than, or any other logic you so desire.

Our data looks like this:

And this is how to filter it client-side:

 import { Component, OnInit } from '@angular/core';
import { AngularFireDatabase } from 'angularfire2/database';
import * as _ from 'lodash';

@Component({
  selector: 'animals-list',
  templateUrl: './animals-list.component.html',
  styleUrls: ['./animals-list.component.scss']
})
export class AnimalsListComponent implements OnInit {

  constructor(private db: AngularFireDatabase) { }

  /// unwrapped arrays from firebase
  animals: any;
  filteredAnimals: any;

  /// filter-able properties
  family:     string;
  weight:     number;
  endangered: boolean;

  /// Active filter rules
  filters = {}

  ngOnInit() {
    this.db.list('/animals')
      .subscribe(animals => {
        this.animals = animals;
        this.applyFilters()
    })
  }

  private applyFilters() {
    this.filteredAnimals = _.filter(this.animals, _.conforms(this.filters) )
  }

  /// filter property by equality to rule
  filterExact(property: string, rule: any) {
    this.filters[property] = val => val == rule
    this.applyFilters()
  }

  /// filter  numbers greater than rule
  filterGreaterThan(property: string, rule: number) {
    this.filters[property] = val => val > rule
    this.applyFilters()
  }

  /// filter properties that resolve to true
  filterBoolean(property: string, rule: boolean) {
    if (!rule) this.removeFilter(property)
    else {
      this.filters[property] = val => val
      this.applyFilters()
    }
  }

  /// removes filter
  removeFilter(property: string) {
    delete this.filters[property]
    this[property] = null
    this.applyFilters()
  }
}

In the HTML, we can bind the user inputs using ngModel, then update the filtering rules anytime a value changes. The filtering will be instantaneous because the code is running client side. We still have a realtime connection to Firebase so any new entires will stay in sync.

<h2>Filters</h2>
  <h4>Family</h4>

  <select [(ngModel)]="family" (change)="filterExact('family', family)">
    <option value="bird">Bird</option>
    <option value="mammal">Mammal</option>
    <option value="fish">Fish</option>
  </select>

  <button *ngIf="family"
          (click)="removeFilter('family')">

          Remove filter
  </button>

  <h4>Weight Greater Than</h4>

  <input type="number"
         [(ngModel)]="weight"
         (change)="filterGreaterThan('weight', weight)">

  <button *ngIf="weight"
          (click)="removeFilter('weight')">

          Remove filter
        </button>

  <h4>Endangered Species</h4>

  <input type="checkbox"
         [(ngModel)]="endangered"
         (change)="filterBoolean('endangered', endangered)">

    Endangered?

  <h2>Animals</h2>

  <div *ngFor="let animal of filteredAnimals">
    <h6>{{animal.name}}</h6>
    <img [src]="animal.image">
    <ul>
      <li>Family: {{animal.family}}</li>
      <li>Weight: {{animal.weight}} pounds</li>
      <li>Endangered: {{animal.endangered}}</li>
    </ul>
  </div>

Option 2 - Composite Keys

Another potential option for data filtering is to use composite keys. A composite key just combines the the keys and values between multiple parameters.

data = {
  keyA: "valueA",
  keyB: "valueB",

  keyA_keyB: "valueA_valueB"
}

As you might imagine, trying to implement this strategy for multiple params will lead to a ridiculous amount of composite keys. It follows the pattern Eularian Numbers, shown in the table below:

If we wanted to filter by 9 different parameters, we would need to maintain 502 composite key combinations on every object. Yikes!

Also, check out QueryBase - a library from the Firebase team that attempts to create composite keys automatically. It is not production ready, but may give you some ideas on how to implement this concept into your app.

Pros

• Server-side filtering for up to 3 properties.

Cons

• Additional data maintenance.
• Only practical for 3 filter-able properties or fewer.
• Increases risk of data anomalies.

Example of Composite Keys

In this example, we add 4 composite keys to an object to handle the filtering of 3 different properties. You would want to build this data programmatically in a real app, but this example demonstrates how the end result should look.

createData() {

  const data = {
    name: 'Crane',
    family: 'bird',
    weight: 10,
    endangered: false,

    // composite keys
    endangered_family: 'false_bird',
    endangered_weight: 'false_10',
    family_weight: 'bird_10',
    endangered_family_weight: 'false_bird_10'
  }

  this.db.list('/animals').push(data)

}


getData() {
  this.db.list('/animals', {
    query: {
      orderByChild: 'endangered_family_weight',
      equalTo: 'bird_10_false'
    }
  })
}

Option 3 - Filter by Tags

Another option that can work in some cases is to filter data based on tags. In this case, you demoralize the data to be queried based on tags associated with it.

You query the keys from the selected tags, then compute the intersection of the keys. The database might look like this:

Pros

• Server-side filtering for unlimited properties

Cons

• Additional data maintenance and duplication
• Only works for exact matches (no numeric ranges)
• Increases risk of data anomalies.
• Requires multiple queries

tags
    tagFoo
       keyA: true
       keyB: true
    tagBar
       keyB: true
       keyC: true

Next, you query each tag and find intersection of keys between the the emitted values using the lodash intersection helper, for example _.intersection([tagFoo, tagBar, tagBaz]).

Option 4 - Use a 3rd Party Indexing Solution

Running complex multi-property filtered queries is an inherently challenging problem with any NoSQL database. If this type of feature is integral to your app, you might consider using a third-party data indexing service that specializes in this task. Algolia, ElasticSeach, and Apache Solr are the most well-known solution providers.

I recently posed a two-part lesson about using Algolia with Angular4 and Firebase. It’s free (up to a volume limit) and can be easily synced with your database via Firebase Cloud Functions.

That’s it for advanced data filtering with Firebase. Let me know if you have questions in the comments.