Using Python+Google Cloud Platform+Line Bot to Automate Routine Tasks

Creating a daily automatic check-in script using a check-in reward app as an example

$Photo by [Paweł Czerwiński](https://unsplash.com/@pawel_czerwinski?utm_source=unsplash&utm_medium=referral&utm_content=creditCopyText){:target="_blank"}$

Photo by Paweł Czerwiński

Origin

I have always had the habit of using Python to create small tools; some are serious, like automatically crawling data and generating reports for work, and some are less serious, like scheduling automatic checks for desired information or delegating tasks that would otherwise be done manually to scripts.

When it comes to automation, I have always been quite straightforward, setting up a computer to run Python scripts continuously; the advantage is simplicity and convenience, but the downside is the need for a device connected to the internet and power. Even a Raspberry Pi consumes a small amount of electricity and internet costs, and it cannot be remotely controlled to start or stop (it actually can, but it’s cumbersome). This time, I took advantage of a work break to explore a free & cloud-based method.

Goal

Move the Python script to the cloud for execution, schedule it to run automatically, and enable it to be started/stopped via the internet.

This article uses a script I wrote for a check-in reward app as an example. The script automatically checks in daily, so I don’t have to open the app manually; it also sends me a notification upon completion.

Completion Notification!

Sections in this Article

Using Proxyman for Man in the Middle Attack API Sniffing
Writing a Python script to fake app API requests (simulate check-in actions)
Moving the Python script to Google Cloud
Setting up automatic scheduling on Google Cloud

Due to the sensitive nature of this topic, this article will not disclose which check-in reward app is used. You can extend this method to your own use.
If you are only interested in how to automate Python execution, you can skip the first part about Man in the Middle Attack API Sniffing and start from Chapter 3.

Tools Used

Proxyman: Man in the Middle Attack API Sniffing
Python: Writing the script
Linebot: Sending notifications of script execution results to myself
Google Cloud Function: Hosting the Python script
Google Cloud Scheduler: Automatic scheduling service

1. Using Proxyman for Man in the Middle Attack API Sniffing

I previously wrote an article titled “The app uses HTTPS for transmission, but the data was still stolen.” The principle is similar, but this time I used Proxyman instead of mitmproxy; it’s also free but more user-friendly.

Go to the official website https://proxyman.io/ to download the Proxyman tool
After downloading, start Proxyman and install the Root certificate (to perform Man in the Middle Attack and unpack HTTPS traffic content)

“Certificate” -> “Install Certificate On this Mac” -> “Installed & Trusted”

After installing the Root certificate on the computer, switch to the mobile:

“Certificate” -> “Install Certificate On iOS” -> “Physical Devices…”

Follow the instructions to set up the Proxy on your mobile and complete the certificate installation and activation.

Open the app on your mobile that you want to sniff the API transmission content for.

At this point, Proxyman on the Mac will show the sniffed traffic. Click on the app API domain under the device IP that you want to view; the first time you view it, you need to click “Enable only this domain” for the subsequent traffic to be unpacked.

After “Enable only this domain,” you will see the newly intercepted traffic showing the original Request and Response information:

We use this method to sniff which API EndPoint is called and what data is sent when performing a check-in operation on the app. Record this information and use Python to simulate the request later.

⚠️ Note that some app token information may change, causing the Python simulated request to fail in the future. You need to understand more about the app token exchange method.

⚠️ If Proxyman is confirmed to be working properly, but the app cannot make requests when Proxyman is enabled, it means the app may have SSL Pinning; currently, there is no solution, and you have to give up.

⚠️ App developers who want to know how to prevent sniffing can refer to the previous article.

Assuming we obtained the following information:

POST /usercenter HTTP/1.1
Host: zhgchg.li
Content-Type: application/x-www-form-urlencoded
Cookie: PHPSESSID=dafd27784f94904dd586d4ca19d8ae62
Connection: keep-alive
Accept: */*
User-Agent: (iPhone12,3;iOS 14.5)
Content-Length: 1076
Accept-Language: zh-tw
Accept-Encoding: gzip, deflate, br
AuthToken: 12345

2. Write a Python script to forge the app API request (simulate the check-in action)

Before writing the Python script, we can first use Postman to debug the parameters and see which parameters are necessary or change over time; but you can also directly copy them.

checkIn.py:

  
import requests
import json

def main(args):
    results = {}
    try:
      data = { "action" : "checkIn" }
      headers = { "Cookie" : "PHPSESSID=dafd27784f94904dd586d4ca19d8ae62", 
      "AuthToken" : "12345",
      "User-Agent" : "(iPhone12,3;iOS 14.5)"
      }
      
      request = requests.post('https://zhgchg.li/usercenter', data = data, headers = headers)
      result = json.loads(request.content)
      if result['status_code'] == 200:
        return "CheckIn Success!"
      else:
        return result['message']
    except Exception as e:
      return str(e)

⚠️ main(args) The purpose of args will be explained later. If you want to test locally, just use main(True).

Using the Requests library to execute HTTP Requests, if you encounter:

ImportError: No module named requests

Please install the library using pip install requests.

Adding Linebot notification for execution results:

I made this part very simple, just for reference, and only to notify myself.

Go to & enable Line Developers Console
Create a Provider

Select “Create a Messaging API channel”

Fill in the basic information in the next step and click “Create” to submit.

After creation, find the “Your user ID” section under the first “Basic settings” Tab. This is your User ID.

After creation, select the “Messaging API” Tab, scan the QRCode to add the bot as a friend.

Scroll down to find the “Channel access token” section, click “Issue” to generate a token.

Copy the generated Token. With this Token, we can send messages to users.

With the User ID and Token, we can send messages to ourselves.

Since we don’t need other functionalities, we don’t even need to install the python line sdk, just send HTTP requests directly.

After integrating with the previous Python script…

checkIn.py:

  
import requests
import json

def main(args):
    results = {}
    try:
      data = { "action" : "checkIn" }
      headers = { "Cookie" : "PHPSESSID=dafd27784f94904dd586d4ca19d8ae62", 
      "AuthToken" : "12345",
      "User-Agent" : "(iPhone12,3;iOS 14.5)"
      }
      
      request = requests.post('https://zhgchg.li/usercenter', data = data, headers = headers)
      result = json.loads(request.content)
      if result['status_code'] == 200:
        sendLineNotification("CheckIn Success!")
        return "CheckIn Success!"
      else:
        sendLineNotification(result['message'])
        return result['message']
    except Exception as e:
      sendLineNotification(str(e))
      return str(e)
      
def sendLineNotification(message):
    data = {
        "to" : "Your User ID here",
        "messages" : [
            {
                "type" : "text",
                "text" : message
            }
        ]
    }
    headers = {
        "Content-Type" : "application/json",
        "Authorization" : "Your channel access token here"
    }
    request = requests.post('https://api.line.me/v2/bot/message/push',json = data, headers = headers)

Test if the notification was sent successfully:

Success!

A small note, I originally wanted to use Gmail SMTP to send emails for notifications, but after uploading to Google Cloud, I found it couldn’t be used…

3. Move the Python script to Google Cloud

After covering the basics, let’s get to the main event of this article: moving the Python script to the cloud.

Initially, I aimed for Google Cloud Run but found it too complicated and didn’t want to spend time researching it because my needs are minimal and don’t require so many features. So, I used Google Cloud Function, a serverless solution; it’s more commonly used to build serverless web services.

If you haven’t used Google Cloud before, please go to the Console to create a new project and set up billing information.
On the project console homepage, click “Cloud Functions” in the resources section.

Select “Create Function” at the top.

Enter basic information.

⚠️ Note down the “ Trigger URL“

Region options:

US-WEST1, US-CENTRAL1, US-EAST1 can enjoy free Cloud Storage service quotas.
asia-east2 (Hong Kong) is closer to us but requires a small Cloud Storage fee.

⚠️ Creating Cloud Functions requires Cloud Storage to store the code.

⚠️ For detailed pricing, please refer to the end of the article.

Trigger type: HTTP

Authentication: Depending on your needs, I want to be able to execute the script from an external link, so I choose “Allow unauthenticated invocations”; if you choose to require authentication, the Scheduler service will also need corresponding settings.

Variables, network, and advanced settings can be set in the variables section for Python to use (this way, if parameters change, you don’t need to modify the Python code):

How to call in Python:

  
import os

def main(request):
  return os.environ.get('test', 'DEFAULT VALUE')

No need to change other settings, just “Save” -> “Next”.

Select “Python 3.x” as the runtime and paste the written Python script, changing the entry point to “main”.

Supplement main(args), as mentioned earlier, this service is more used for serverless web; so args are actually Request objects, from which you can get http get query and http post body data, as follows:

Get GET Query information:
request_args = args.args

example: ?name=zhgchgli => request_args = [“name”:”zhgchgli”]

Get POST Body data:
request_json = request.get_json(silent=True)

example: name=zhgchgli => request_json = [“name”:”zhgchgli”]

If testing POST with Postman, remember to use “Raw+JSON” POST data, otherwise, nothing will be received:

After the code part is OK, switch to “requirements.txt” and enter the dependencies used:

We use the “requests” package to help us make API calls, which is not in the native Python library; so we need to add it here:

requests>=2.25.1

Here is the translated Markdown content:

Specify version ≥ 2.25.1 here, or just enter requests to install the latest version.

Once everything is OK, click “Deploy” to start the deployment.

It takes about 1-3 minutes to complete the deployment.

After the deployment is complete, you can go to the “ Trigger URL “ noted earlier to check if it is running correctly, or use “Actions” -> “Test Function” to test it.

If 500 Internal Server Error appears, it means there is an error in the program. You can click the name to view the “Logs” and find the reason:

UnboundLocalError: local variable 'db' referenced before assignment

After clicking the name, you can also click “Edit” to modify the script content.

If the test is fine, it’s done! We have successfully moved the Python script to the cloud.

Additional Information about Variables

According to our needs, we need a place to store and read the token of the check-in APP; because the token may expire, it needs to be re-requested and written for use in the next execution.

To dynamically pass variables from the outside to the script, the following methods are available:

[Read Only] As mentioned earlier, runtime environment variables
[Temp] Cloud Functions provides a /tmp directory for writing and reading files during execution, but it will be deleted after completion. For details, please refer to the official documentation.
[Read Only] GET/POST data transmission
[Read Only] Include additional files

In the program, using the relative path ./ can read it, only read, cannot dynamically modify; to modify, you can only do it in the console and redeploy.

To read and dynamically modify, you need to connect to other GCP services, such as: Cloud SQL, Google Storage, Firebase Cloud Firestore…

[Read & Write] Here I choose Firebase Cloud Firestore because it currently has a free quota for use.

According to the Getting Started Guide, after creating the Firebase project, enter the Firebase console:

Find “ Cloud Firestore “ in the left menu -> “ Add Collection “

Enter the collection ID.

Enter the data content.

A collection can have multiple documents, and each document can have its own field content; it is very flexible to use.

In Python:

First, go to GCP Console -> IAM & Admin -> Service Accounts, and follow the steps below to download the authentication private key file:

First, select the account:

Below, “Add Key” -> “Create New Key”

Select “JSON” to download the file.

Place this JSON file in the same directory as the Python project.

In the local development environment:

pip install --upgrade firebase-admin

Install the firebase-admin package.

In Cloud Functions, add firebase-admin to requirements.txt.

Once the environment is set up, we can read the data we just added:

firebase_admin.py:

  
import firebase_admin
from firebase_admin import credentials
from firebase_admin import firestore

if not firebase_admin._apps:
  cred = credentials.Certificate('./authentication.json')
  firebase_admin.initialize_app(cred)
# Because initializing the app multiple times will cause the following error
# providing an app name as the second argument. In most cases you only need to call initialize_app() once. But if you do want to initialize multiple apps, pass a second argument to initialize_app() to give each app a unique name.
# So to be safe, check if it is already initialized before calling initialize_app

db = firestore.client()
ref = db.collection(u'example') // Collection name
stream = ref.stream()
for data in stream:
  print("id:"+data.id+","+data.to_dict())

If you are on Cloud Functions, you can either upload the authentication JSON file together or change the connection syntax as follows:

  
cred = credentials.ApplicationDefault()
firebase_admin.initialize_app(cred, {
  'projectId': project_id,
})

db = firestore.client()

If you encounter Failed to initialize a certificate credential., please check if the authentication JSON is correct.

For more operations like adding or deleting, please refer to the official documentation.

4. Set up automatic scheduling in Google Cloud

After having the script, the next step is to make it run automatically to achieve our final goal.

Go to the Google Cloud Scheduler console homepage
Click “Create Job” at the top

Enter the basic job information

Execution frequency: Same as crontab input method. If you are not familiar with crontab syntax, you can directly use crontab.guru this amazing website:

It can clearly translate the actual meaning of the syntax you set. (Click next to see the next execution time)

Here I set 15 1 * * *, because the check-in only needs to be executed once a day, set to execute at 1:15 AM every day.

URL part: Enter the “ trigger URL “ noted earlier

Time zone: Enter “Taiwan”, select Taipei Standard Time

HTTP method: According to the previous Python code, we use Get

If you set “authentication” earlier remember to expand “SHOW MORE” to set up authentication.

After filling everything out, press “ Create “.

After successful creation, you can choose “Run Now” to test if it works properly.

You can view the execution results and the last execution date

⚠️ Please note that the execution result “failure” only refers to web status codes 400~500 or errors in the Python program.

All Done!

We have achieved the goal of uploading the routine task Python script to the cloud and setting it to run automatically.

Pricing

Another very important part is the pricing; Google Cloud and Linebot are not completely free services, so understanding the pricing is crucial. Otherwise, for a small script, paying too much money might not be worth it compared to just running it on a computer.

Linebot

Refer to the official pricing information, which is free for up to 500 messages per month.

Google Cloud Functions

Refer to the official pricing information, which includes 2 million invocations, 400,000 GB-seconds, 200,000 GHz-seconds of compute time, and 5 GB of internet egress per month.

Google Firebase Cloud Firestore

Refer to the official pricing information, which includes 1 GB of storage, 10 GB of data transfer per month, 50,000 reads per day, and 20,000 writes/deletes per day; sufficient for light usage!

Google Cloud Scheduler

Refer to the official pricing information, which allows 3 free jobs per account.

The above free quotas are more than enough for the script!

Google Cloud Storage Conditional Free Usage

Despite all efforts, some services might still incur charges.

After creating Cloud Functions, two Cloud Storage instances will be automatically created:

If you chose US-WEST1, US-CENTRAL1, or US-EAST1 for Cloud Functions, you can enjoy free usage quotas:

I chose US-CENTRAL1, and you can see that the first Cloud Storage instance is indeed in US-CENTRAL1, but the second one is labeled Multiple regions in the US; I estimate this one will incur charges.

Refer to the official pricing information, which varies by region.

The code isn’t large, so I estimate the minimum charge will be around 0.0X0 per month (?)

⚠️ The above information was recorded on 2021/02/21, and the actual prices may vary. This is for reference only.

Budget Control Notifications

Just in case… if the usage exceeds the free quota and starts incurring charges, I want to receive notifications to avoid unexpectedly high bills due to program errors.

Go to the Console
Find the “ Billing “ Card:

Click “View Detailed Deduction Records” to enter.

Expand the left menu and enter the “Budget and Alerts” feature.

Click on the top “Set Budget”

Enter a custom name

Next step.

Amount, enter “Target Amount”, you can enter $1, $10; we don’t want to spend too much on small things.

Next step.

Here you can set the action to trigger a notification when the budget reaches a certain percentage.

Check “Send alerts to billing administrators and users via email”, so that when the condition is triggered, you will receive a notification immediately.

Click “Finish” to submit and save.

When the budget is exceeded, we can know immediately to avoid incurring more costs.

Summary

Human energy is limited. In today’s flood of technological information, every platform and service wants to extract our limited energy. If we can use some automated scripts to share our daily lives, we can save more energy to focus on important things!

Using Python+Google Cloud Platform+Line Bot to Automate Routine Tasks