POC App End-to-End Testing Local Snapshot API Mock Server

[POC] App End-to-End Testing Local Snapshot API Mock Server

Verification of the feasibility of implementing E2E Testing for existing apps and existing API architecture

Photo by freestocks

Introduction

As a project that has been operating online for many years, continuously improving stability is a highly challenging issue.

Unit Testing

Due to the static + compiled + strongly typed nature of the development languages Swift/Kotlin or the dynamic to static transition from Objective-C to Swift, it is almost impossible to add Unit Testing later if testability was not considered during development to cleanly separate interface dependencies. However, the refactoring process can also introduce instability, leading to a chicken-and-egg problem.

UI Testing

Testing UI interactions and buttons; it can be implemented by slightly decoupling data dependencies in new or existing screens.

SnapShot Testing

Verifying whether the UI display content and style are consistent before and after adjustments; similar to UI Testing, it can be implemented by slightly decoupling data dependencies in new or existing screens.

It is very useful for transitioning from Storyboard/XIB to Code Layout or UIView from OC to Swift; you can directly import pointfreeco / swift-snapshot-testing for quick implementation.

Although we can add UI Testing and SnapShot Testing later, the coverage of these tests is very limited; most errors are not UI style issues but process or logic problems that interrupt user operations. If this occurs during the checkout process, involving revenue, the issue becomes very serious.

End-to-End Testing

As mentioned earlier, it is not feasible to easily add unit tests to the current project or to integrate units for integration testing. For logic and process protection, the remaining method is to perform End-to-End black-box testing from the outside, directly from the user’s perspective, to check whether important processes (registration/checkout, etc.) are functioning normally.

For major function refactoring, you can also establish process tests before refactoring and re-verify after refactoring to ensure that the functionality works as expected.

Refactoring along with adding Unit Testing and Integration Testing to increase stability, breaking the chicken-and-egg problem.

QA Team

The most direct and brute-force way of End-to-End Testing is to have a QA Team manually test according to the Test Plan, and then continuously optimize or introduce automated operations. Calculating the cost, it would require at least 2 engineers + 1 Leader spending at least half a year to a year to see results.

Evaluating the time and cost, is there anything we can do in the current situation or prepare for the future QA Team so that when there is a QA Team, we can directly jump to optimization and automation operations, or even introduce AI?

Automation

At this stage, the goal is to introduce automated End-to-End Testing, placed in the CI/CD process for automatic checks. The test content does not need to be too comprehensive; as long as it can prevent major process issues, it is already very valuable. Later, we can gradually iterate the Test Plan to cover more areas.

End-to-End Testing — Technical Challenges

UI Operation Issues

The principle of the App is more like using another test App to operate our tested App, and then finding the target object from the View Hierarchy. During testing, we cannot obtain the Log or Output of the tested App because they are essentially two different Apps.

iOS needs to improve the View Accessibility Identifier to increase efficiency and accuracy and handle Alerts (e.g., push notification requests).

In previous implementations on Android, there was an issue where the target object could not be found when mixing Compose and Fragment, but according to a teammate, the new version of Compose has resolved this.

Besides the common traditional issues mentioned above, a bigger problem is the difficulty of integrating dual platforms (writing one test to run on two platforms). Currently, we are trying to use a new testing tool mobile-dev-inc / maestro:

You can write a Test Plan in YAML and then execute tests on dual platforms. For detailed usage and trial experiences, stay tuned for another teammate’s article sharing cc’ed Alejandra Ts. 😝.

API Data Issues

The biggest testing variable for App E2E Testing is API data. If we cannot provide guaranteed data, it will increase the instability of the tests, leading to false positives, and eventually, everyone will lose confidence in the Test Plan.

For example, in testing the checkout process, if the product might be taken off the shelf or disappear, and these status changes are not controllable by the App, the above situation is very likely to occur.

There are many ways to solve data issues, such as establishing a clean Staging or Testing environment, or an Auto-Gen Mock API Server based on Open API. However, these all rely on the backend and external factors of the API. Additionally, the backend API, like the App, is an online project that has been running for many years, and some specifications are still being restructured and migrated, making it temporarily impossible to have a Mock Server.

Given these factors, if we get stuck here, the problem will remain unchanged, and the chicken-and-egg problem cannot be broken. We really can only “take the risk” and make changes first, dealing with issues as they arise.

Snapshot API Local Mock Server

“As long as the mindset doesn’t slip, there are more solutions than difficulties.”

We can think differently. If the UI can be snapshotted into images for replay verification testing, can the API do the same? Can we save the API Request & Response and replay them for verification testing later?

This introduces the main point of this article: establishing a “Snapshot API Local Mock Server” to record API Requests & Replay Responses, removing the dependency on API data.

This article only provides a Proof of Concept (POC) and has not yet fully implemented high-coverage End-to-End Testing. Therefore, the approach is for reference only. I hope it provides new insights for everyone in the current environment.

Snapshot API Local Mock Server

Core Concept — Record & Replay API Data

[Record] — After completing the development of the End-to-End Testing Test Case, enable the recording parameter and execute the test once. During this process, all API Requests & Responses will be saved in the respective Test Case directories.

[Replay] — When running the Test Case later, the corresponding recorded Response Data will be found from the Test Case directory according to the request to complete the testing process.

Illustration

Suppose we want to test the purchase process. The user opens the App, clicks on the product card on the homepage to enter the product detail page, clicks the purchase button at the bottom, a login box pops up to complete the login, completes the purchase, and a purchase success prompt pops up:

How UI Testing controls button clicks, input box inputs, etc., is not the main focus of this article; you can refer to existing testing frameworks for direct use.

Regular Proxy or Reverse Proxy

To achieve Record & Replay API, a Proxy needs to be added between the App and the API to perform a man-in-the-middle attack. You can refer to my earlier article “The APP uses HTTPS transmission, but the data is still stolen.”

In simple terms, there is an additional proxy transmitter between the App and the API, like passing notes. The requests and responses exchanged between both parties will go through it. It can open the content of the notes and can also forge the content of the notes for both parties without them noticing.

Regular Proxy:

A regular proxy is when the client sends a request to the proxy server, the proxy server forwards the request to the target server, and then returns the response from the target server to the client. In a regular proxy mode, the proxy server initiates the request on behalf of the client. The client needs to explicitly specify the address and port number of the proxy server and send the request to the proxy server.

Reverse Proxy:

A reverse proxy is the opposite of a regular proxy. It sits between the target server and the client. The client sends a request to the reverse proxy server, which forwards the request to the backend target server according to certain rules and returns the response from the target server to the client. For the client, the target server appears to be the reverse proxy server, and the client does not need to know the real address of the target server.

For our needs, either regular or reverse proxy can achieve the goal. The only consideration is the method of proxy setup:

Regular Proxy requires setting up a Proxy in the network settings on the computer, phone, or emulator:

• Android can directly set up a Proxy in the emulator.
• iOS Simulator shares the computer’s network environment and cannot individually set up a Proxy, requiring changes to the computer’s settings to set up a Proxy. All traffic on the computer will go through this Proxy, and if other network tools like Proxyman or Charles are also running, they might forcefully change the Proxy settings to their own, causing it to fail.

Reverse Proxy requires changing the API Host in the Codebase and declaring all API Domains to be proxied:

• The API Host in the Codebase needs to be replaced with the Proxy Server IP during testing.
• When enabling Reverse Proxy, declare which Domains need to be proxied.
• Only declared Domains will go through the Proxy; undeclared ones will go directly out.

For iOS App, the following example uses iOS & Reverse Proxy for POC. The same can be applied to Android.

Letting the iOS App Know It’s Running End-to-End Testing

We need to let the App know it’s running End-to-End Testing to add the API Host replacement logic in the App program:

// UI Testing Target:
let app = XCUIApplication()
app.launchArguments = ["duringE2ETesting"]
app.launch()

We make the judgment and replacement in the Network layer.

This is an unavoidable adjustment. Try to avoid changing the App’s Code just for testing.

Using MITMProxy to Implement Reverse Proxy Server

You can also use Swift to develop a Swift Server to achieve this. This article uses the MITMProxy tool for POC.

[2023–09–04 Update] Mitmproxy-rodo is Now Open Source

The implementation content below has been open-sourced to the mitmproxy-rodo project. Feel free to refer to and use it directly.

Some structures and content of this article have been adjusted, and the following adjustments were made when open-sourced:

• Changed the storage directory structure to host / requestPath / method / hash
• Fixed Header information storage, should be Bytes Data instead of pure JSON String
• Corrected some errors
• Added automatic extension of Set-Cookie expiration functionality

⚠️ The following script is for Demo reference only, subsequent script adjustments will be moved to the open-source project maintenance.

MITMProxy

Follow the MITMProxy official website to complete the installation:

brew install mitmproxy

For detailed usage of MITMProxy, you can refer to my earlier article “The APP uses HTTPS transmission, but the data is still stolen.”

• mitmproxy provides an interactive command-line interface.
• mitmweb provides a browser-based graphical user interface.
• mitmdump provides non-interactive terminal output.

Implementing Record & Replay

Since MITMProxy Reverse Proxy does not natively have the functionality to Record (or dump) requests & Mapping Request Replay, we need to write scripts to achieve this functionality.

mock.py :

"""
Example:
    Record: mitmdump -m reverse:https://yourapihost.com -s mock.py --set record=true --set dumper_folder=loginFlow --set config_file=config.json
    Replay: mitmdump -m reverse:https://yourapihost.com -s mock.py --set dumper_folder=loginFlow --set config_file=config.json
"""

import re
import logging
import mimetypes
import os
import json
import hashlib

from pathlib import Path
from mitmproxy import ctx
from mitmproxy import http

class MockServerHandler:

    def load(self, loader):
        self.readHistory = {}
        self.configuration = {}

        loader.add_option(
            name="dumper_folder",
            typespec=str,
            default="dump",
            help="Response Dump directory, can be created by Test Case Name",
        )

        loader.add_option(
            name="network_restricted",
            typespec=bool,
            default=True,
            help="No Mapping data locally... setting true will return 404, false will make a real request to get data.",
        )

        loader.add_option(
            name="record",
            typespec=bool,
            default=False,
            help="Set true to record Request's Response",
        )

        loader.add_option(
            name="config_file",
            typespec=str,
            default="",
            help="Set file path, example file below",
        )
    
    def configure(self, updated):
        self.loadConfig()

    def loadConfig(self):
        configFile = Path(ctx.options.config_file)
        if ctx.options.config_file == "" or not configFile.exists():
            return

        self.configuration = json.loads(open(configFile, "r").read())

    def hash(self, request):
        query = request.query
        requestPath = "-".join(request.path_components)

        ignoredQueryParameterByPaths = self.configuration.get("ignored", {}).get("paths", {}).get(request.host, {}).get(requestPath, {}).get(request.method, {}).get("queryParamters", [])
        ignoredQueryParameterGlobal = self.configuration.get("ignored", {}).get("global", {}).get("queryParamters", [])

        filteredQuery = []
        if query:
            filteredQuery = [(key, value) for key, value in query.items() if key not in ignoredQueryParameterByPaths + ignoredQueryParameterGlobal]
        
        formData = []
        if request.get_content() != None and request.get_content() != b'':
            formData = json.loads(request.get_content())
        
        # or just formData = request.urlencoded_form
        # or just formData = request.multipart_form
        # depends on your api design

        ignoredFormDataParametersByPaths = self.configuration.get("ignored", {}).get("paths", {}).get(request.host, {}).get(requestPath, {}).get(request.method, {}).get("formDataParameters", [])
        ignoredFormDataParametersGlobal = self.configuration.get("ignored", {}).get("global", {}).get("formDataParameters", [])

        filteredFormData = []
        if formData:
            filteredFormData = [(key, value) for key, value in formData.items() if key not in ignoredFormDataParametersByPaths + ignoredFormDataParametersGlobal]
        
        # Serialize the dictionary to a JSON string
        hashData = {"query":sorted(filteredQuery), "form": sorted(filteredFormData)}
        json_str = json.dumps(hashData, sort_keys=True)

        # Apply SHA-256 hash function
        hash_object = hashlib.sha256(json_str.encode())
        hash_string = hash_object.hexdigest()
        
        return hash_string

    def readFromFile(self, request):
        host = request.host
        method = request.method
        hash = self.hash(request)
        requestPath = "-".join(request.path_components)

        folder = Path(ctx.options.dumper_folder) / host / method / requestPath / hash

        if not folder.exists():
            return None

        content_type = request.headers.get("content-type", "").split(";")[0]
        ext = mimetypes.guess_extension(content_type) or ".json"


        count = self.readHistory.get(host, {}).get(method, {}).get(requestPath, {}) or 0

        filepath = folder / f"Content-{str(count)}{ext}"

        while not filepath.exists() and count > 0:
            count = count - 1
            filepath = folder / f"Content-{str(count)}{ext}"

        if self.readHistory.get(host) is None:
            self.readHistory[host] = {}
        if self.readHistory.get(host).get(method) is None:
            self.readHistory[host][method] = {}
        if self.readHistory.get(host).get(method).get(requestPath) is None:
            self.readHistory[host][method][requestPath] = {}

        if filepath.exists():
            headerFilePath = folder / f"Header-{str(count)}.json"
            if not headerFilePath.exists():
                headerFilePath = None
            
            count += 1
            self.readHistory[host][method][requestPath] = count

            return {"content": filepath, "header": headerFilePath}
        else:
            return None


    def saveToFile(self, request, response):
        host = request.host
        method = request.method
        hash = self.hash(request)
        requestPath = "-".join(request.path_components)

        iterable = self.configuration.get("ignored", {}).get("paths", {}).get(request.host, {}).get(requestPath, {}).get(request.method, {}).get("iterable", False)
        
        folder = Path(ctx.options.dumper_folder) / host / method / requestPath / hash

        # create dir if not exists
        if not folder.exists():
            os.makedirs(folder)

        content_type = response.headers.get("content-type", "").split(";")[0]
        ext = mimetypes.guess_extension(content_type) or ".json"

        repeatNumber = 0
        filepath = folder / f"Content-{str(repeatNumber)}{ext}"
        while filepath.exists() and iterable == False:
            repeatNumber += 1
            filepath = folder / f"Content-{str(repeatNumber)}{ext}"
        
        # dump to file
        with open(filepath, "wb") as f:
            f.write(response.content or b'')
            
        
        headerFilepath = folder / f"Header-{str(repeatNumber)}.json"
        with open(headerFilepath, "wb") as f:
            responseDict = dict(response.headers.items())
            responseDict['_status_code'] = response.status_code
            f.write(json.dumps(responseDict).encode('utf-8'))

        return {"content": filepath, "header": headerFilepath}

    def request(self, flow):
        if ctx.options.record != True:
            host = flow.request.host
            path = flow.request.path

            result = self.readFromFile(flow.request)
            if result is not None:
                content = b''
                headers = {}
                statusCode = 200

                if result.get('content') is not None:
                    content = open(result['content'], "r").read()

                if result.get('header') is not None:
                    headers = json.loads(open(result['header'], "r").read())
                    statusCode = headers['_status_code']
                    del headers['_status_code']

                
                headers['_responseFromMitmproxy'] = '1'
                flow.response = http.Response.make(statusCode, content, headers)
                logging.info("Fullfill response from local with "+str(result['content']))
                return

            if ctx.options.network_restricted == True:
                flow.response = http.Response.make(404, b'', {'_responseFromMitmproxy': '1'})
        
    def response(self, flow):
        if ctx.options.record == True and flow.response.headers.get('_responseFromMitmproxy') != '1':
            result = self.saveToFile(flow.request, flow.response)
            logging.info("Save response to local with "+str(result['content']))

addons = [MockServerHandler()]

You can refer to the official documentation and adjust the script content as needed.

The design logic of this script is as follows:

• File path logic: dumper_folder(a.k.a Test Case Name) / Reverse's api host / HTTP Method / Path join with - (e.g. app/launch -> app-launch) / Hash(Get Query & Post Content) /
• File logic: Response content: Content-0.xxx, Content-1.xxx (the second request of the same request) … and so on; Response Header information: Header-0.json (same Content-x logic)

• When saving, it will be saved sequentially according to the path and file logic; during Replay, it will be retrieved in the same order.
• If the number of times does not match, for example, the same path is hit 3 times during Replay, but the Record only saves data up to the 2nd time; it will still respond with the 2nd time, which is the last result.
• When record is True, it will hit the target Server to get the response and save it according to the above logic; when False, it will only read data locally (equivalent to Replay Mode).
• When network_restricted is False, if there is no Mapping data locally, it will directly respond with 404; when True, it will hit the target Server to get the data.
• _responseFromMitmproxy is used to inform the Response Method that the current response is from Local and can be ignored, _status_code borrows the Header.json field to store the HTTP Response status code.

config_file.json configuration file logic design is as follows:

{
  "ignored": {
    "paths": {
      "yourapihost.com": {
        "add-to-cart": {
          "POST": {
            "queryParamters": [
              "created_timestamp"
            ],
            "formDataParameters": []
          }
        },
        "api-status-checker": {
          "GET": {
            "iterable": true
          }
        }
      }
    },
    "global": {
      "queryParamters": [
        "timestamp"
      ],
      "formDataParameters": []
    }
  }
}

queryParamters & formDataParameters:

Because some API parameters may change with each call, for example, some Endpoints will carry time parameters, at this time according to the Server’s design, the Hash(Query Parameter & Body Content) value will be different during Replay Request, resulting in no Mapping to Local Response. Therefore, an additional config.json is used to handle this situation. You can set certain parameters to be excluded from the Hash by Endpoint Path or Global, so you can get the same Mapping result.

iterable :

Because some polling check APIs may be called repeatedly at regular intervals, according to the Server’s design, many Content-x.xxx & Header-x.json files will be generated; but if we don’t care, we can set it to True, and the Response will continue to be saved and overwritten to the first file Content-0.xxx & Header-0.json.

Enable Reverse Proxy Record Mode:

mitmdump -m reverse:https://yourapihost.com -s mock.py --set record=true --set dumper_folder=loginFlow --set config_file=config.json

Enable Reverse Proxy Replay Mode:

mitmdump -m reverse:https://yourapihost.com -s mock.py --set dumper_folder=loginFlow --set config_file=config.json

Assembly & Proof Of Concept

0. Complete the Host replacement in the Codebase

And ensure that during testing, the API is switched to http://127.0.0.1:8080

1. Start Snapshot API Local Mock Server (a.k.a Reverse Proxy Server) Record Mode

mitmdump -m reverse:https://yourapihost.com -s mock.py --set record=true --set dumper_folder=addCart --set config_file=config.json

2. Perform E2E Testing UI Operations

Using the Pinkoi iOS App as an example, test the following flow:

Launch App -> Home -> Scroll Down -> Similar to Wish List Items Section -> First Product -> Click First Product -> Enter Product Page -> Click Add to Cart -> UI Response Added to Cart -> Test Successful ✅

The method of UI automation operation was mentioned earlier, here we manually test the same flow to verify the results.

3. Obtain Record Results

After the operation is completed, you can press ^ + C to terminate the Snapshot API Mock Server and check the recording results in the file directory:

4. Replay to verify the same flow, start the Server & Using Replay Mode

mitmdump -m reverse:https://yourapihost.com -s mock.py --set dumper_folder=addCart --set config_file=config.json

5. Perform the same UI operation again to verify the results

• Left: Test Successful ✅
• Right: Testing clicking on products other than the recorded ones will result in an Error (because there is no data locally + network_restricted is set to False by default, so it will directly return 404 without fetching data from the network)

6. Proof Of Concept ✅

The proof of concept is successful. We can indeed use the Reverse Proxy Server to store API Requests & Responses and use it as a Mock API Server to respond with data to the App during testing 🎉🎉🎉.

[2023-09-04] mitmproxy-rodo is now open source

Follow-up and Miscellaneous

This article only discusses the proof of concept. There are still many areas to be improved and more features to be implemented.

1. Integration with maestro UI Testing tool
2. CI/CD process integration design (How to automatically start the Reverse Proxy? Where to start it?)
3. How to package MITMProxy into development tools?
4. Verify more complex testing scenarios
5. Verify the sent Tracking Requests, need to implement storing Request Body, then extract which Tracking Event Data was sent, and whether it matches the events that should be sent in the flow

Cookie Issues

#...
    def response(self, flow):
        setCookies = flow.response.headers.get_all("set-cookie")
        # setCookies = ['ad=0; Domain=.xxx.com; expires=Wed, 23 Aug 2023 04:59:07 GMT; Max-Age=1800; Path=/', 'sessionid=xxxx; Secure; HttpOnly; Domain=.xxx.com; expires=Wed, 23 Aug 2023 04:59:07 GMT; Max-Age=1800; Path=/']
        
        # OR Replace Cookie Domain From .xxx.com To 127.0.0.1
        setCookies = [re.sub(r"\s*\.xxx\.com\s*", "127.0.0.1", s) for s in setCookies]

        # AND Remove Security-Related Restrictions
        setCookies = [re.sub(r";\s*Secure\s*", "", s) for s in setCookies]
        setCookies = [re.sub(r";\s*HttpOnly;\s*", "", s) for s in setCookies]

        flow.response.headers.set_all("Set-Cookie", setCookies)

        #...

If you encounter issues with Cookies, such as the API responding with a Cookie but the App not receiving it, you can refer to the adjustments above.

The Last Post on Pinkoi

During my 900+ days at Pinkoi, I realized many of my career aspirations and imaginations regarding iOS/App development and processes. I am grateful to all my teammates for walking through the pandemic and weathering the storms together; the courage to say goodbye is akin to the courage to pursue dreams and join the company initially.

I am embarking on a new life challenge (including but not limited to engineering). If you have suitable opportunities (iOS or engineering management or startup products), please feel free to contact me. 🙏🙏🙏

If you have any questions or feedback, feel free to contact me.

===

本文中文版本

===

This article was first published in Traditional Chinese on Medium ➡️ View Here