What Can Be Done to Commemorate When an App Product Reaches Its End?
Using mitmproxy + apple configurator to keep an App in its pre-removal state forever
Introduction
Jujutsu Kaisen
After working for a long time and handling many products, I have started to encounter products that I once participated in reaching their end (removal). Developing a product from scratch is like nurturing a new life, with the team working together for 3-4 months to bring the child into the world. Although it was later handed over to other caretakers (engineers) for further development, hearing that it is about to reach the end of its product lifecycle still brings some regret.
Life is like this too. We never know if the sun will rise first tomorrow or if an accident will happen. The only thing we can do is cherish the present and do things well.
Commemoration
Every step leaves a trace. We hope to do something before the product reaches its end so that everyone still has a chance to remember it and at least leave proof of its existence. The following methods require the App to still be online; if it has already been removed, then only memories remain.
Non-technical Method — Recording
Besides using the iPhone’s built-in screen recording feature, we can also use QuickTime Player to connect the phone to a Mac for recording and exporting videos.
- Open the QuickTime Player App on the Mac
- In the top left toolbar, select “File” -> “New Movie Recording”
- After the recording interface pops up, click the “v” next to the 🔴, and select your connected phone for the screen and speaker
- The recording interface will now display the phone screen
Click the “🔴” to start recording, and operate the content you want to record on the phone.
During recording, the current video size will be displayed. To stop recording, press the “🔴” again.
You can use the QuickTime Player toolbar to simply trim the video. Finally, press “Command” + “s” to export and save the video to the specified location, completing the recording for commemoration.
The advantage of video commemoration is that future memories are more easily connected than with pictures. The deeper you record, the more detailed the record. If you want to convert specific frames into pictures, you can directly take screenshots, which is very convenient.
Technical Method
Technical backup of an App can be divided into two directions: “bones” and “meat”. The App itself is just a skeleton, while the core content data of the App is composed of API Response Data.
- The bones will disappear as the App is removed from the App Store.
- The meat will disappear as the API host and server shut down.
Therefore, we also divide the technical backup into bones and meat.
Disclaimer
This article is for technical research and sharing only. It does not encourage the use of any technology for illegal or infringing activities.
[Bones] Backup .ipa App Installation File
After an App is removed from the store, as long as the downloaded App is not actively deleted from the phone, it will always exist on that phone. If you change phones using the transfer method, it will also be transferred.
But if we accidentally delete the App or change phones without transferring it, then it will be gone forever. At this time, if we manually back up the .ipa file from the store, we can extend its life again.
A long time ago, the reverse engineering article mentioned this, but this time we only need to back up the .ipa file without jailbreaking, all using tools provided by Apple.
1. Install Apple Configurator 2
First, go to the Mac App Store to download and install Apple Configurator 2.
2. Connect iPhone to Mac and click Trust This Computer
Once connected successfully, the iPhone’s home screen will appear.
3. Ensure your phone has the app installed that you want to back up the .ipa file for
We need to use Apple Configurator 2 to get the .ipa file downloaded to the cache, so we need to make sure the target app is installed on the phone.
4. Go back to Apple Configurator 2 on the Mac
Double-click the iPhone home screen shown above to enter the information page.
Switch to “App” -> top right corner “+ Add” -> “App”
After logging into the App Store account, you can get a list of apps you have purchased before.
Search for the target app you want to back up, select it, and click “Add”.
A waiting window will appear, adding the app on XXX, downloading “XXX”.
5. Extract the .ipa file
Wait for it to finish downloading, a window will pop up asking if you want to replace the existing installed app.
Do not click anything at this time. Do not click anything at this time. Do not click anything at this time.
Open a Finder:
Select “Go” -> “Go to Folder” from the top left toolbar
Paste the following path:
1
~/Library/Group Containers/K36BKF7T3D.group.com.apple.configurator/Library/Caches/Assets/TemporaryItems/MobileApps
You can find the target app .ipa file that is downloaded and ready to be installed:
Copy it out to complete the app .ipa file backup.
After completing the file copy, go back to Apple Configurator 2 and click stop to terminate the operation.
[Bone] Restore .ipa App Installation File
Similarly, connect the phone to be restored to the Mac and open Apple Configurator 2, enter the app addition interface.
For restoration, select “Choose from my Mac…” in the bottom left corner.
Select the backed-up app .ipa file and click “Add”.
Wait for the transfer and installation to complete, then you can reopen the app on your phone, successfully revived!
[Meat] Back Up the Final API Response Data
Here we will use the method and open-source project mentioned in the previous App End-to-End Testing Local Snapshot API Mock Server article (refer to the details and principles).
With the same technique used for recording API Request & Response for E2E Testing, we can also use it to record the last API Request & Response Data before an app is taken down or shut down.
1. Install mitmproxy
1
brew install mitmproxy
mitmproxy is an open-source man-in-the-middle attack and network request sniffing tool.
If you are not familiar with the working principle of Mitmproxy man-in-the-middle attacks, you can refer to my previous article: “The APP uses HTTPS transmission, but the data was still stolen.” or the Mitmproxy official documentation.
If you are using it purely for network request sniffing and are not comfortable with the mitmproxy interface, you can also use “Proxyman” as referenced in another previous article.
2. Complete mitmproxy certificate setup
For HTTPS encrypted connections, we need to use a root certificate swap to perform a man-in-the-middle attack. Therefore, the first time you use it, you need to complete the root certificate download and activation on the mobile end.
*If your App & API Server has implemented SSL Pinning, you also need to add the Pinning certificate to mitmproxy.
- First, ensure that the iPhone and Mac are connected to the same network environment.
- If there is no WiFi and the computer is connected to a physical network, you can also turn on the Mac’s WiFi sharing feature to let the phone connect to the Mac’s network.
Start mitmproxy or mitmweb (Web GUI version) in Terminal.
1
mitmproxy
Seeing this screen means the mitmproxy service has started, and there is no traffic coming in, so it is empty. Keep this screen open and do not close the Terminal.
- Go to the Mac network settings to check the Mac’s IP address.
Go back to the phone’s WiFi settings, click “i” to enter detailed settings, and find “Configure Proxy” at the bottom:
- Enter the Mac’s IP address in the server field.
- Enter 8080 in the port field.
- Save.
Open Safari on the phone and enter: http://mitm.it/
If it shows:
1
If you can see this, traffic is not passing through mitmproxy.
It means the network proxy server on the phone was not set up successfully, or mitmproxy was not started on the Mac.
Under normal circumstances, it will show:
At this point, only HTTP traffic can be sniffed, and HTTPS traffic will report an error. We will continue to set it up.
This means the connection is successful. Find the iOS section and click “Get mitmproxy-ca-cert.pem”.
- Click “Allow”.
After the download is complete, go to the phone’s settings, and you will see “Profile Downloaded”. Click to enter.
- Click to enter, in the upper right corner “Install”, enter the phone password to complete the installation.
Go back to Settings -> “General” -> “About” -> At the bottom “Certificate Trust Settings” -> Enable “mitmproxy”.
- “Continue” to complete the activation.
At this point, we have completed all the preliminary work for the man-in-the-middle attack.
Remember that all the traffic on your phone will go through the proxy from your Mac computer. After the operation is completed, remember to go back to the network settings on your phone and turn off the proxy server settings, otherwise the phone’s WiFi will not be able to connect to the external network.
Go back to Terminal mitmproxy, and while operating the App on your phone, you can see all the captured API request records.
Each request can be entered to view detailed Request & Response content:
The above is the basic setup and actual work of mitmproxy.
3. Sniff and Understand the API Structure
Next, we will use mitmproxy’s mitmdump service combined with the mitmproxy-rodo addons I developed earlier to record and replay requests.
My implementation principle is to calculate the Hash value of the Request parameters. When replaying, the request is taken to calculate the Hash again. If the same Hash value backup Response is found locally, it will be returned. If there are multiple requests with the same Hash value, they will be stored and replayed in order.
We can first use the above method to sniff the App’s API (or use Proxyman), observe which fields might affect Hash Mapping, and record them for later exclusion settings. For example, some APIs always carry the ?ts parameter, which does not affect the returned content but affects the Hash value calculation, making it impossible to find the local backup. We need to pick it out and exclude it in the later settings.
4. Set up mitmproxy-rodo:
Use the open-source recording and replay script I wrote.
For detailed parameter settings, please refer to the instructions of the open-source project.
1
2
git clone git@github.com:ZhgChgLi/mitmproxy-rodo.git
cd mitmproxy-rodo
Fill in the parameters picked out in step 3 into the config.json configuration file:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
{
"ignored": {
"*": {
"*": {
"*": {
"iterable": false,
"enables": [
"query",
"formData"
],
"rules": {
"query": {
"parameters": [
"ts",
"connect_id",
"device_id",
"device_name",
]
},
"formData": {
"parameters": [
"aidck",
"device_id",
"ver_name",
]
}
}
}
}
}
}
}
The above parameters will be excluded when calculating the Hash value, and specific exclusion rules can be set for individual Endpoint paths.
5. Enable recording, and execute in Terminal:
1
mitmdump -s rodo.py --set dumper_folder=zhgchgli --set config_file=config.json --set record=true "~d zhgchg.li"
- The ending
"~d zhgchg.li"means to capture only the traffic of * .zhgchg.li. dumper_folder: Name of the output destination directory
6. Operate the target App on the phone to execute the desired recording process path
- It is recommended to restart and reinstall the App to start with the cleanest state.
- It is recommended to record a video to help remember the reproduction steps.
While operating, you will see many captured API Response Data in the output directory, stored according to Domain -> API path -> HTTP method -> Hash value -> Header-X / Content-X (if the same Hash request is made twice, it will be saved in order).
- To re-record, you can directly delete the output directory and let it capture again.
- If the returned data contains personal information, remember to adjust the captured content to anonymize it.
[Meat] Replay the captured API Response Data
After recording, be sure to try replaying once to test if the data is normal. If the Hash Hit is very low (almost no corresponding Response found during replay), you can repeat the sniffing steps to find the variable that affects the Hash value each time the App is executed and exclude it.
Execute replay:
1
mitmdump -s rodo.py --set dumper_folder=zhgchgli --set config_file=config.json
dumper_folder: Name of the output destination directory- By default, if there is no locally mapped Hash Response Data, it will directly return 404 to make the App blank, so you can know if the captured data is effective.
- The path page that was passed during recording and capturing can be displayed again during replay: OK!
- The path page that was not passed during recording and capturing shows a network error during replay: OK!
Remembrance
At this point, we can reproduce the last moments before the App reached its final station through the restoration of bones and the final meat, to remember the time when everyone worked together to produce it.
This article commemorates the team of my first job and the time when I transitioned from web backend development to iOS App development, learning while doing, and independently producing a product from scratch in 3-4 months, together with Android, design, PM supervisors, and backend colleagues. Although it is about to reach the end of its life cycle, I will always remember the bittersweet moments and the excitement of seeing it go live and being used for the first time.
“Thank you”
Contributions Welcome
If you have the same regrets, I hope this article can help you, because mitmproxy-rodo was initially developed as a POC concept verification tool. Contributions, bug reports, or PRs to fix bugs are welcome.
For any questions or comments, feel free to contact me.
===
===
This article was first published in Traditional Chinese on Medium ➡️ View Here
