The APP uses HTTPS for transmission, but data is still stolen.
iOS+MacOS Using mitmproxy for Man-in-the-middle Attack to Sniff API Data and How to Defend Against It?
Introduction
A while ago, we just held an internal CTF competition at the company. When brainstorming challenges, I recalled a project I worked on during college as a backend (PHP) developer— a points-collecting app. It basically had a task list, and when a trigger condition was met, it would call an API to earn points. The boss thought that since the API call was transmitted over HTTPS, it was very secure—until I showed him a man-in-the-middle attack, directly sniffing the transmitted data and forging API calls to get points….
With the rise of big data in recent years, web crawlers have become widespread; the battle between crawling and anti-crawling has intensified, with numerous tricks used on both sides. It’s truly a game of cat and mouse!
Another target for web scraping is the APP’s API. Without any protection, it’s like leaving the door wide open; it’s easy to operate, the format is clean, and it’s harder to detect and block. So, if the website has put a lot of effort into blocking scraping but data is still being extracted, consider checking for vulnerabilities in the APP’s API.
Because I didn’t know how to include this topic in a CTF competition, I decided to write a separate article as a record. This is just a basic concept — HTTPS can be decrypted by certificate replacement during transmission, and how to enhance security to prevent it. Network theory is not my strong suit, so I’ve left that to the experts. If you’re already familiar with this topic, you don’t need to spend time reading this article; just skip to the end to see how apps should protect themselves!
Practical Operation
Environment: MacOS + iOS
Android users can directly download Packet Capture (free). iOS users can use Surge 4 (paid) to unlock MITM features. MacOS users can also use another paid software, Charles.
This article mainly explains how to use the free mitmproxy on iOS. If you already have the above environment, you don’t need to go through all these steps—just open the app on your phone, connect through a VPN, replace the certificate, and you can perform a man-in-the-middle attack! (Please scroll down to see how to protect yourself!)
[Updated 2021/02/25]: There is a new free graphical interface app for Mac (Proxyman) available, which can be used along with the first part of this reference article.
Install mitmproxy
Install directly using brew:
brew install mitmproxy
Installation Complete!
p.s If you see brew: command not found, please install the brew package manager first:
/usr/bin/ruby -e "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install)"
mitmproxy Usage
After installation, enter the following command in Terminal to activate:
mitmproxy
Startup Successful
Put your phone and Mac on the same local network & get the Mac’s IP address
Method (1) Mac connected to WiFi, phone also uses the same WiFi
Mac’s IP Address = “System Preferences” -> “Network” -> “Wi-Fi” -> “IP Address”
Method (2) Use wired network on Mac, enable Internet Sharing; connect your phone to this hotspot:
“System Preferences” -> “Sharing” -> select “Ethernet” -> check “Wi-Fi” -> enable “Internet Sharing”
Mac’s IP Address = 192.168.2.1 (️️Note ⚠️ This is not the Ethernet IP, but the IP Mac uses as a network sharing hotspot)
Mobile Network Settings WiFi — Proxy Server Information
“Settings” -> “WiFi” -> “HTTP Proxy” -> “Manual” -> “Server enter Mac’s IP address” -> “Port enter 8080” -> “Save”
It’s normal for the webpage to fail to load and show a certificate error at this point; let’s continue…
Install mitmproxy Custom HTTPS Certificate
As mentioned above, the way to perform a man-in-the-middle attack is to use your own certificate to intercept and decrypt the data during communication; therefore, we also need to install this custom certificate on the phone.
1. Open http://mitm.it on your mobile Safari
Left side means Proxy settings are correct ✅ / Right side means Proxy settings are incorrect 🚫
“Apple” -> “Install Profile” -> “Install”
⚠️ It’s not over yet; we still need to enable the profile inside.
“General” -> “About” -> “Certificate Trust Settings” -> Enable “mitmproxy”
Done! Now we can go back to the browser and browse the web normally.
Back to operating mitmproxy on Mac
You can see the data transmission logs from the phone on the mitmproxy Terminal.
Find the record you want to sniff and check the Request (which parameters were sent) / Response (what content was returned)
Common Shortcut Keys:
" ? " = View key operation documentation
" k "/ "⬆" = Up
" j "/ "⬇" = Down
" h "/ "⬅" = Left
" l "/ "➡️" = Right
" space " = Next page
" enter " = Enter to view details
" q " = Go back / Exit
" b " = Export response body to specified path text file
" f " = Filter record conditions
" z " = Clear all records
" e " = Edit Request (cookie, headers, params...)
" r " = Resend Request
Not comfortable with CLI? No worries, you can switch to the Web GUI!
Besides the mitmproxy activation method, we can also change to:
mitmweb
You can use the new Web GUI to operate and observe.
mitmweb
The main event: Sniffing APP data:
After completing and becoming familiar with the above setup, we can move on to the main event: sniffing the data transmission of the APP’s API!
This example uses a specific housing app for academic purposes only, with no malicious intent!
We want to know how the object list API requests and what content it returns!
First, press “z” to clear all records (to avoid confusion)
Open the target APP
Open the target APP and try to “pull down to refresh” or trigger the “load next page” action.
🛑If your target APP won’t open or connect, sorry, it means the APP has protections and this method won’t work for sniffing. Please scroll down to the protection section🛑
mitmproxy Logs
Return to mitmproxy to check the logs, use your detective skills to guess which API request record is the one you want, and then enter to view the details!
Request
In the Request section, you can see which parameters were sent in the request.
Use “e” to edit and “r” to resend, then observe the Response to infer the purpose of each parameter!
Response
Response can also directly obtain the raw returned content.
🛑If the Response content is a bunch of encoded data; sorry, it means the APP might be doing its own encryption/decryption, so this method won’t work for sniffing. Please scroll down to the protection section🛑
Hard to read? Garbled Chinese characters? No problem. You can export as a text file to the desktop using “b”, then copy the content to Json Editor Online for parsing!
**Or directly use mitmweb to browse and operate via the web GUI*
mitmweb
After sniffing, observing, filtering, and testing, you can understand how the APP API works, allowing you to exploit it and use crawlers to collect data.
*After collecting the necessary information, remember to close mitmproxy and set the phone’s network proxy back to automatic to use the internet normally.
How Should an APP Protect Itself?
If the app stops working or returns encoded content after attaching mitmproxy, it means the app has protection in place.
Method (1):
Basically, the certificate information is embedded in the APP. If the current HTTPS certificate does not match the one in the APP, access is denied. For details, see this or look for resources on SSL Pinning. The downside is you need to watch out for the certificate’s expiration date!
https://medium.com/@dzungnguyen.hcm/ios-ssl-pinning-bffd2ee9efc
Method (2):
The app encrypts and encodes data before transmission. The API backend decrypts it to get the original request. The API response is also encrypted and encoded before sending back, and the app decrypts it upon receipt. This process is tedious and resource-consuming but is indeed a method. As far as I know, some digital banks use this approach for protection!
However….
Method 1 still has a bypass: How to Bypass SSL Pinning on iOS 12
Method 2: Obtain the encryption key through reverse engineering the project
⚠️No 100% Security⚠️
Or just dig a hole and let it crawl, collecting all kinds of evidence, then let the legal team handle it (？
Still the same saying:
“NEVER TRUST THE CLIENT”
More Uses of mitmproxy:
1. Using mitmdump
Besides mitmproxy and mitmweb, mitmdump can directly export all logs to a text file.
mitmdump -w /log.txt
And you can use Method (2) Python script to configure and filter traffic:
mitmdump -ns examples/filter.py -r /log.txt -w /result.txt
2. Using Python scripts for request parameter settings, access control, and redirection:
from mitmproxy import http
def request(flow: http.HTTPFlow) -> None:
# pretty_host takes the "Host" header of the request into account,
# which is useful in transparent mode where we usually only have the IP
# otherwise.
# Example of setting request parameters:
flow.request.headers['User-Agent'] = 'MitmProxy'
if flow.request.pretty_host == "123.com.tw":
flow.request.host = "456.com.tw"
# Redirect all visits from 123.com.tw to 456.com.tw
Redirect Example
Enable mitmproxy with the following parameter:
mitmproxy -s /redirect.py
or
mitmweb -s /redirect.py
or
mitmdump -s /redirect.py
Filling a Gap
When using mitmproxy to observe requests with HTTP 1.1 and Accept-Ranges: bytes, Content-Range for long connections fetching resource segments continuously, it waits until the entire response is received before displaying it, instead of showing the segments or continuing the download using a persistent connection!
Further Reading
-
Using a Check-in Reward APP as an Example to Create a Daily Auto Check-in Script
-
Revealing a Clever Website Vulnerability Discovered Years Ago
Epilogue
Since I don’t have domain authority to obtain the SSL certificate info, I can’t implement it. Judging by the code, it doesn’t seem difficult. Although there’s no 100% secure method, adding an extra layer of protection can at least improve security. Further attacks would require a lot of time and research, which should discourage about 90% of crawlers!
This article might have low value, as I neglected Medium for a while (went off to play with a DSLR). It’s mainly a warm-up write-up for this weekend’s (2019/09/21–2019/09/22) iPlayground 2019. Looking forward to this year’s sessions 🤩 and hope to produce more high-quality articles afterward!
[2019/02/22 Article Update] What is the iPlayground 2019 Experience Like?
Comments