Home Quickly Build a Testable API Service Using Firebase Firestore + Functions
Post
Cancel

Quickly Build a Testable API Service Using Firebase Firestore + Functions

Posted Mar 24, 2021 Updated Apr 13, 2024
Preview Image
By
18 min read

Quickly Build a Testable API Service Using Firebase Firestore + Functions

When push notification statistics meet Firebase Firestore + Functions

Photo by [Carlos Muza](https://unsplash.com/@kmuza?utm_source=unsplash&utm_medium=referral&utm_content=creditCopyText){:target="_blank"}

Photo by Carlos Muza

Introduction

Accurate Push Notification Statistics

Recently, I wanted to introduce a feature to the APP. Before implementation, we could only use the success or failure of posting data to APNS/FCM from the backend as the base for push notifications and record the click-through rate. However, this method is very inaccurate as the base includes many invalid devices. Devices with the APP deleted (which may not immediately become invalid) or with push notifications disabled will still return success when posting from the backend.

After iOS 10, you can implement the Notification Service Extension to secretly call an API for statistics when the push notification banner appears. The advantage is that it is very accurate; it only calls when the user’s push notification banner appears. If the APP is deleted, notifications are turned off, or the banner is not displayed, there will be no action. The banner appearing equals a push notification message, and using this as the base for push notifications and then counting the clicks will give an “accurate click-through rate.”

For detailed principles and implementation methods, refer to the previous article: “iOS ≥ 10 Notification Service Extension Application (Swift)”

Currently, the APP’s loss rate should be 0% based on tests. A common practical application is Line’s point-to-point message encryption and decryption (the push notification message is encrypted and decrypted only when received on the phone).

Problem

The work on the APP side is actually not much. Both iOS/Android only need to implement similar functions (but if considering the Chinese market for Android, it becomes more complicated as you need to implement push notification frameworks for more platforms). The bigger work is on the backend and server pressure handling because when a push notification is sent out, it will simultaneously call the API to return records, which might overwhelm the server’s max connection. If using RDBMS to store records, it could be even more severe. If you find statistical losses, it often happens at this stage.

You can record by writing logs to files and do statistics and display when querying.

Additionally, thinking about the scenario of simultaneous returns, the quantity might not be as large as imagined. Push notifications are not sent out in tens or hundreds of thousands at once but in batches. As long as you can handle the number of simultaneous returns from batch sending, it should be fine!

Prototype

Considering the issues mentioned, the backend needs effort to research and modify, and the market may not care about the results. So, I thought of using available resources to create a prototype to test the waters.

Here, I chose Firebase services, which almost all APPs use, specifically the Functions and Firestore features.

Firebase Functions

Functions is a serverless service provided by Google. You only need to write the program logic, and Google will automatically handle the server, execution environment, and you don’t have to worry about server scaling and traffic issues.

Firebase Functions are essentially Google Cloud Functions but can only be written in JavaScript (node.js). Although I haven’t tried it, if you use Google Cloud Functions and choose to write in another language while importing Firebase services, it should work as well.

For API usage, I can write a node.js file, get a real URL (e.g., my-project.cloudfunctions.net/getUser), and write the logic to obtain Request information and provide the corresponding Response.

I previously wrote an article about Google Functions: Using Python + Google Cloud Platform + Line Bot to Automate Routine Tasks

Firebase Functions must enable the Blaze plan (pay-as-you-go) to use.

Firebase Firestore

Firebase Firestore is a NoSQL database used to store and manage data.

Combined with Firebase Functions, you can import Firestore during a Request to operate the database and then respond to the user, allowing you to build a simple Restful API service!

Let’s get hands-on!

Install node.js Environment

It is recommended to use NVM, a node.js version management tool, for installation and management (similar to pyenv for Python).

Copy the installation shell script from the NVM GitHub project:

1

curl -o- https://raw.githubusercontent.com/nvm-sh/nvm/v0.37.2/install.sh | bash

If errors occur during installation, ensure you have a ~/.bashrc or ~/.zshrc file. If not, you can create one using touch ~/.bashrc or touch ~/.zshrc and then rerun the install script.

Next, you can use nvm install node to install the latest version of node.js.

You can check if npm is installed successfully and its version by running npm --version:

Deploy Firebase Functions

Install Firebase-tools:

1

npm install -g firebase-tools

After successful installation, for the first-time use, enter:

1

firebase login

Complete Firebase login authentication.

Initialize the project:

1

firebase init

Note the path where Firebase init is located:

1

You're about to initialize a Firebase project in this directory:

Here you can choose the Firebase CLI tools to install. Use the “↑” and “↓” keys to navigate and the “spacebar” to select. You can choose to install only “Functions” or both “Functions” and “Firestore”.

=== Functions Setup

  • Select language: JavaScript
  • For “use ESLint to catch probable bugs and enforce style” syntax style check, YES / NO both are fine.
  • Install dependencies with npm? YES

=== Emulators Setup

You can test Functions and Firestore features and settings locally without it counting towards usage and without needing to deploy online to test.

Install as needed. I installed it but didn’t use it… because it’s just a small feature.

Coding!

Go to the path noted above, find the functions folder, and open the index.js file with an editor.

index.js:

1
2
3
4
5
6
7
8
9
10
11
12

const functions = require('firebase-functions');
const admin = require('firebase-admin');
admin.initializeApp();

exports.hello = functions.https.onRequest((req, res) => {
    const targetID = req.query.targetID
    const action = req.body.action
    const name = req.body.name

    res.send({"targetID": targetID, "action": action, "name": name});
    return
})

Paste the above content. We have defined a path interface /hello that will return the URL Query ?targetID=, POST action, and name parameter information.

After modifying and saving, go back to the console and run:

1

firebase deploy

Remember to run the firebase deploy command every time you make changes for them to take effect.

Start verifying & deploying to Firebase…

It may take a while. After Deploy complete!, your first Request & Response webpage is done!

At this point, you can go back to the Firebase -> Functions page:

You will see the interface and URL location you just wrote.

Copy the URL below and test it in PostMan:

Remember to select x-www-form-urlencoded for the POST Body.

Success!

Log

We can use the following in the code to log records:

1

functions.logger.log("log:", value);

And view the log results in Firebase -> Functions -> Logs:

Example Goal

Create an API that can add, modify, delete, query articles, and like them.

We want to achieve the functionality design of a Restful API, so we can’t use the pure Path method from the above example. Instead, we need to use the Express framework.

POST Add Article

index.js:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25

const functions = require('firebase-functions');
const admin = require('firebase-admin');
const express = require('express');
const cors = require('cors');
const app = express();

admin.initializeApp();
app.use(cors({ origin: true }));

// Insert
app.post('/', async (req, res) => { // This POST refers to the HTTP Method POST
    const title = req.body.title;
    const content = req.body.content;
    const author = req.body.author;

    if (title == null || content == null || author == null) {
        return res.status(400).send({"message":"Parameter error!"});
    }

    var post = {"title":title, "content":content, "author": author, "created_at": new Date()};
    await admin.firestore().collection('posts').add(post);
    res.status(201).send({"message":"Added successfully!"});
});

exports.post= functions.https.onRequest(app); // This POST refers to the /post path

Now we use Express to handle network requests. Here, we first add a POST method for the path /. The last line indicates that all paths are under /post. Next, we will add APIs for updating and deleting.

After successfully deploying with firebase deploy, go back to Post Man to test:

After successfully hitting Post Man, you can check in Firebase -> Firestore to see if the data is correctly written:

PUT Update Article

index.js:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28

const functions = require('firebase-functions');
const admin = require('firebase-admin');
const express = require('express');
const cors = require('cors');
const app = express();

admin.initializeApp();
app.use(cors({ origin: true }));

// Update
app.put("/:id", async (req, res) => {
    const title = req.body.title;
    const content = req.body.content;
    const author = req.body.author;
    const doc = await admin.firestore().collection('posts').doc(req.params.id).get();

    if (!doc.exists) {
        return res.status(404).send({"message":"Article not found!"}); 
    } else if (title == null || content == null || author == null) {
        return res.status(400).send({"message":"Invalid parameters!"});
    }

    var post = {"title":title, "content":content, "author": author};
    await admin.firestore().collection('posts').doc(req.params.id).update(post);
    res.status(200).send({"message":"Update successful!"});
});

exports.post= functions.https.onRequest(app);

Deployment & testing method is the same as adding, remember to change the Post Man Http Method to PUT.

DELETE Delete Article

index.js:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22

const functions = require('firebase-functions');
const admin = require('firebase-admin');
const express = require('express');
const cors = require('cors');
const app = express();

admin.initializeApp();
app.use(cors({ origin: true }));

// Delete
app.delete("/:id", async (req, res) => {
    const doc = await admin.firestore().collection('posts').doc(req.params.id).get();

    if (!doc.exists) {
        return res.status(404).send({"message":"Article not found!"});
    }

    await admin.firestore().collection("posts").doc(req.params.id).delete();
    res.status(200).send({"message":"Article deleted successfully!"});
})

exports.post= functions.https.onRequest(app);

Deployment & testing method is the same as adding, remember to change the Post Man Http Method to DELETE.

Adding, modifying, and deleting are done, let’s do the query!

SELECT Query Articles

index.js:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33

const functions = require('firebase-functions');
const admin = require('firebase-admin');
const express = require('express');
const cors = require('cors');
const app = express();

admin.initializeApp();
app.use(cors({ origin: true }));

// Select List
app.get('/', async (req, res) => {
    const posts = await admin.firestore().collection('posts').get();
    var result = [];
    posts.forEach(doc => {
      let id = doc.id;
      let data = doc.data();
      result.push({"id":id, ...data})
    });
    res.status(200).send({"result":result});
});

// Select One
app.get("/:id", async (req, res) => {
    const doc = await admin.firestore().collection('posts').doc(req.params.id).get();

    if (!doc.exists) {
        return res.status(404).send({"message":"Article not found!"});
    }

    res.status(200).send({"result":{"id":doc.id, ...doc.data()}});
});

exports.post= functions.https.onRequest(app);

Deployment & testing method is the same as adding, remember to change the Post Man Http Method to GET and switch Body back to none.

InsertOrUpdate?

Sometimes we need to update when the value exists and add when the value does not exist. In this case, we can use set with merge: true:

index.js:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23

const functions = require('firebase-functions');
const admin = require('firebase-admin');
const express = require('express');
const cors = require('cors');
const app = express();

admin.initializeApp();
app.use(cors({ origin: true }));

// InsertOrUpdate
app.post("/tag", async (req, res) => {
    const name = req.body.name;

    if (name == null) {
        return res.status(400).send({"message":"Invalid parameter!"});
    }

    var tag = {"name":name};
    await admin.firestore().collection('tags').doc(name).set({created_at: new Date()}, {merge: true});
    res.status(201).send({"message":"Added successfully!"});
});

exports.post= functions.https.onRequest(app);

Here, taking adding a tag as an example, the deployment & testing method is the same as adding. You can see that Firestore will not repeatedly add new data.

Article Like Counter

Suppose our article data now has an additional likeCount field to record the number of likes. How should we do it?

index.js:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23

const functions = require('firebase-functions');
const admin = require('firebase-admin');
const express = require('express');
const cors = require('cors');
const app = express();

admin.initializeApp();
app.use(cors({ origin: true }));

// Like Post
app.post("/like/:id", async (req, res) => {
    const doc = await admin.firestore().collection('posts').doc(req.params.id).get();
    const increment = admin.firestore.FieldValue.increment(1)

    if (!doc.exists) {
        return res.status(404).send({"message":"Article not found!"});
    }

    await admin.firestore().collection('posts').doc(req.params.id).set({likeCount: increment}, {merge: true});
    res.status(201).send({"message":"Liked successfully!"});
});

exports.post= functions.https.onRequest(app);

Using the increment variable allows you to directly perform the action of retrieving the value +1.

High Traffic Article Like Counter

Because Firestore has write speed limits:

A document can only be written once per second, so when there are many people liking it; simultaneous requests may become very slow.

The official solution “ Distributed counters “ is actually not very advanced technology, it just uses several distributed likeCount fields to count, and then sums them up when reading.

index.js:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26

const functions = require('firebase-functions');
const admin = require('firebase-admin');
const express = require('express');
const cors = require('cors');
const app = express();

admin.initializeApp();
app.use(cors({ origin: true }));

// Distributed counters Like Post
app.post("/like2/:id", async (req, res) => {
    const doc = await admin.firestore().collection('posts').doc(req.params.id).get();
    const increment = admin.firestore.FieldValue.increment(1)

    if (!doc.exists) {
        return res.status(404).send({"message":"Article not found!"});
    }

    //1~10
    await admin.firestore().collection('posts').doc(req.params.id).collection("likeCounter").doc("likeCount_"+(Math.floor(Math.random()*10)+1).toString())
    .set({count: increment}, {merge: true});
    res.status(201).send({"message":"Like successful!"});
});


exports.post= functions.https.onRequest(app);

The above is to distribute the fields to record Count to avoid slow writing; but if there are too many distributed fields, it will increase the reading cost ($$), but it should still be cheaper than adding a new record for each like.

Using Siege Tool for Stress Testing

Use brew to install siege

1

brew install siege

p.s If you encounter brew: command not found, please install the brew package management tool first:

1

/usr/bin/ruby -e "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install)"

After installation, you can run:

1

siege -c 100 -r 1 -H 'Content-Type: application/json' 'https://us-central1-project.cloudfunctions.net/post/like/id POST {}'

Perform stress testing:

  • -c 100: 100 tasks executed simultaneously
  • -r 1: Each task executes 1 request
  • -H ‘Content-Type: application/json’: Required if it is a POST
  • ‘https://us-central1-project.cloudfunctions.net/post/like/id POST {}’: POST URL, Post Body (ex: {“name”:”1234”})

After execution, you can see the results:

successful_transactions: 100 indicates that all 100 transactions were successful.

You can go back to Firebase -> Firestore to check if there is any Loss Data:

Success!

Complete Example Code

index.js:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120

const functions = require('firebase-functions');
const admin = require('firebase-admin');
const express = require('express');
const cors = require('cors');
const app = express();

admin.initializeApp();
app.use(cors({ origin: true }));

// Insert
app.post('/', async (req, res) => {
    const title = req.body.title;
    const content = req.body.content;
    const author = req.body.author;

    if (title == null || content == null || author == null) {
        return res.status(400).send({"message":"Parameter error!"});
    }

    var post = {"title":title, "content":content, "author": author, "created_at": new Date()};
    await admin.firestore().collection('posts').add(post);
    res.status(201).send({"message":"Successfully added!"});
});

// Update
app.put("/:id", async (req, res) => {
    const title = req.body.title;
    const content = req.body.content;
    const author = req.body.author;
    const doc = await admin.firestore().collection('posts').doc(req.params.id).get();

    if (!doc.exists) {
        return res.status(404).send({"message":"Post not found!"}); 
    } else if (title == null || content == null || author == null) {
        return res.status(400).send({"message":"Parameter error!"});
    }

    var post = {"title":title, "content":content, "author": author};
    await admin.firestore().collection('posts').doc(req.params.id).update(post);
    res.status(200).send({"message":"Successfully updated!"});
});

// Delete
app.delete("/:id", async (req, res) => {
    const doc = await admin.firestore().collection('posts').doc(req.params.id).get();

    if (!doc.exists) {
        return res.status(404).send({"message":"Post not found!"});
    }

    await admin.firestore().collection("posts").doc(req.params.id).delete();
    res.status(200).send({"message":"Post successfully deleted!"});
});

// Select List
app.get('/', async (req, res) => {
    const posts = await admin.firestore().collection('posts').get();
    var result = [];
    posts.forEach(doc => {
      let id = doc.id;
      let data = doc.data();
      result.push({"id":id, ...data})
    });
    res.status(200).send({"result":result});
});

// Select One
app.get("/:id", async (req, res) => {
    const doc = await admin.firestore().collection('posts').doc(req.params.id).get();

    if (!doc.exists) {
        return res.status(404).send({"message":"Post not found!"});
    }

    res.status(200).send({"result":{"id":doc.id, ...doc.data()}});
});

// InsertOrUpdate
app.post("/tag", async (req, res) => {
    const name = req.body.name;

    if (name == null) {
        return res.status(400).send({"message":"Parameter error!"});
    }

    var tag = {"name":name};
    await admin.firestore().collection('tags').doc(name).set({created_at: new Date()}, {merge: true});
    res.status(201).send({"message":"Successfully added!"});
});

// Like Post
app.post("/like/:id", async (req, res) => {
    const doc = await admin.firestore().collection('posts').doc(req.params.id).get();
    const increment = admin.firestore.FieldValue.increment(1)

    if (!doc.exists) {
        return res.status(404).send({"message":"Post not found!"});
    }

    await admin.firestore().collection('posts').doc(req.params.id).set({likeCount: increment}, {merge: true});
    res.status(201).send({"message":"Successfully liked!"});
});

// Distributed counters Like Post
app.post("/like2/:id", async (req, res) => {
    const doc = await admin.firestore().collection('posts').doc(req.params.id).get();
    const increment = admin.firestore.FieldValue.increment(1)

    if (!doc.exists) {
        return res.status(404).send({"message":"Post not found!"});
    }

    //1~10
    await admin.firestore().collection('posts').doc(req.params.id).collection("likeCounter").doc("likeCount_"+(Math.floor(Math.random()*10)+1).toString())
    .set({count: increment}, {merge: true});
    res.status(201).send({"message":"Successfully liked!"});
});


exports.post= functions.https.onRequest(app);

Back to the topic, push notification statistics

Back to what we initially wanted to do, the push notification statistics feature.

index.js:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48

const functions = require('firebase-functions');
const admin = require('firebase-admin');
const express = require('express');
const cors = require('cors');
const app = express();

admin.initializeApp();
app.use(cors({ origin: true }));

const vaildPlatformTypes = ["ios","Android"]
const vaildActionTypes = ["clicked","received"]

// Insert Log
app.post('/', async (req, res) => {
    const increment = admin.firestore.FieldValue.increment(1);
    const platformType = req.body.platformType;
    const pushID = req.body.pushID;
    const actionType =  req.body.actionType;

    if (!vaildPlatformTypes.includes(platformType) || pushID == undefined || !vaildActionTypes.includes(actionType)) {
        return res.status(400).send({"message":"Invalid parameters!"});
    } else {
        await admin.firestore().collection(platformType).doc(actionType+"_"+pushID).collection("shards").doc((Math.floor(Math.random()*10)+1).toString())
        .set({count: increment}, {merge: true})
        res.status(201).send({"message":"Record successful!"});
    }
});

// View Log
app.get('/:type/:id', async (req, res) => {
    // received
    const receivedDocs = await admin.firestore().collection(req.params.type).doc("received_"+req.params.id).collection("shards").get();
    var received = 0;
    receivedDocs.forEach(doc => {
      received += doc.data().count;
    });

    // clicked
    const clickedDocs = await admin.firestore().collection(req.params.type).doc("clicked_"+req.params.id).collection("shards").get();
    var clicked = 0;
    clickedDocs.forEach(doc => {
        clicked += doc.data().count;
    });
    
    res.status(200).send({"received":received,"clicked":clicked});
});

exports.notification = functions.https.onRequest(app);

Add Push Notification Record

View Push Notification Statistics

1

https://us-centra1-xxx.cloudfunctions.net/notification/iOS/1

Additionally, we also created an interface to count push notification numbers.

Pitfalls

Since I am not very familiar with node.js, during the initial exploration, I did not add await when adding data. Coupled with the write speed limit, it led to Data Loss under high traffic conditions…

Pricing

Don’t forget to refer to the pricing strategy for Firebase Functions & Firestore.

Functions

Computation Time

Computation Time

Network

Network

Cloud Functions offers a permanent free tier for computation time resources, which includes GB/seconds and GHz/seconds of computation time. In addition to 2 million invocations, the free tier also provides 400,000 GB/seconds and 200,000 GHz/seconds of computation time, as well as 5 GB of internet egress per month.

Firestore

Prices are subject to change at any time, please refer to the official website for the latest information.

Conclusion

As the title suggests, “for testing”, “for testing”, “for testing” it is not recommended to use the above services in a production environment or as the core of a product launch.

Expensive and Hard to Migrate

I once heard that a fairly large service was built using Firebase services, and later on, with large data and traffic, the charges became extremely expensive; it was also very difficult to migrate, the code was okay but the data was very hard to move; it can only be said that saving a little money in the early stages caused huge losses later on, not worth it.

For Testing Only

For the above reasons, I personally recommend using Firebase Functions + Firestore to build API services only for testing or prototype product demonstrations.

More Features

Functions can also integrate Authentication, Storage, but I haven’t researched this part.

References

Further Reading

If you have any questions or suggestions, feel free to contact me.

===

本文中文版本

===

This article was first published in Traditional Chinese on Medium ➡️ View Here


ZRealm, Dev.
ios-app-development firebase google-cloud-platform notifications ios
This post is licensed under CC BY 4.0 by the author.
Recently Updated
Contents

Password Recovery SMS Verification Code Security Issue

AppStore APP’s Reviews Bot Insights