The APP uses HTTPS for transmission, but the data was still stolen.

Using mitmproxy on iOS+MacOS to perform a Man-in-the-middle attack to sniff API transmission data and how to prevent it?

Introduction

Recently, we just held an internal CTF competition at the company. While brainstorming for topics, I recalled a project from my university days when I was working on backend (PHP) development. It was a point collection APP with a task list, and upon completing the trigger conditions, it would call an API to earn points. The boss thought that calling the API with HTTPS encrypted transmission was very secure — until I demonstrated a Man-in-the-middle attack, directly sniffing the transmission data and forging API calls to earn points…

In recent years, with the rise of big data, web crawlers are everywhere; the battle between crawlers and anti-crawlers is becoming increasingly intense, with various tricks being used. It’s a constant game of cat and mouse!

Another target for crawlers is the APP’s API. If there are no defenses, it’s almost like leaving the door wide open; it’s not only easy to operate but also clean in format, making it harder to identify and block. So if you’ve exhausted all efforts to block on the web end and data is still being crawled, you might want to check if the APP’s API has any vulnerabilities.

Since I didn’t know how to incorporate this topic into the CTF competition, I decided to write a separate article as a record. This article is just to give a basic concept — HTTPS can be decrypted through certificate replacement and how to enhance security to prevent it. The actual network theory is not my strong suit and has been forgotten, so if you already have a concept of this, you don’t need to spend time reading this article, or just scroll to the bottom to see how to protect your APP!

Practical Operation

Environment: MacOS + iOS

Android users can directly download Packet Capture (free), iOS users can use Surge 4 (paid) to unlock the Man-in-the-middle attack feature, and MacOS users can also use another paid software, Charles.

This article mainly explains how to use the free mitmproxy on iOS. If you have the above environment, you don’t need to go through this trouble. Just open the APP on your phone, mount the VPN, and replace the certificate to perform a Man-in-the-middle attack! (Again, please scroll to the bottom to see how to protect your APP!)

[2021/02/25 Update]: Mac has a new free graphical interface program (Proxyman) that can be used, which can be paired with this article for reference in the first part.

Install mitmproxy

Directly use brew to install:

brew install mitmproxy

Installation complete!

p.s. If you encounter brew: command not found, please first install the brew package management tool:

  
/usr/bin/ruby -e "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install)"

Using mitmproxy

After installation, enter the following command in Terminal to activate:

mitmproxy

Startup Successful

Ensure the phone and Mac are on the same local network & obtain the Mac’s IP address

Method (1) Mac connects to WiFi, and the phone uses the same WiFi Mac’s IP address = “System Preferences” -> “Network” -> “Wi-Fi” -> “IP Address”

Method (2) Mac uses a wired network, enables Internet Sharing; phone connects to the hotspot network:

System Preferences -> Sharing -> Select “Ethernet” -> Check “Wi-Fi” -> Enable “Internet Sharing”

Mac’s IP address = 192.168.2.1 (Note ⚠️ This is not the Ethernet IP, but the IP used by the Mac as a network sharing base station)

Phone network settings WiFi — Proxy server information

Settings -> WiFi -> HTTP Proxy -> Manual -> Enter Mac’s IP address in Server -> Enter 8080 in Port -> Save

At this point, it is normal for web pages not to open and for certificate errors to appear; let’s continue…

Install mitmproxy custom https certificate

As mentioned above, the way a man-in-the-middle attack works is by using its own certificate to decrypt and encrypt data during communication; so we also need to install this custom certificate on the phone.

1. Open http://mitm.it on the phone’s Safari

Left side appears -> Proxy settings ✅ / Right side appears -> Proxy settings error 🚫

Apple -> Install Profile -> Install

⚠️ It’s not over yet, we need to enable the profile in the About section

General -> About -> Certificate Trust Settings -> Enable mitmproxy

Done! Now we can go back to the browser and browse web pages normally.

Back to Mac to operate mitmproxy

You can see the data transfer records from the phone on the mitmproxy Terminal

Find the record you want to sniff and view the Request (what parameters were sent) / Response (what content was returned)

Common operation keys:

" ? " = View key operation documentation
" k " / "⬆" = Up 
" j " / "⬇" = Down 
" h " / "⬅" = Left 
" l " / "➡️" = Right 
" space " = Next page
" enter " = View details
" q " = Go back to the previous page/exit
" b " = Export response body to a specified path text file 
" f " = Filter records
" z " = Clear all records
" e " = Edit Request (cookie, headers, params...)
" r " = Resend Request

Not comfortable with CLI? No worries, you can switch to Web GUI!

Besides the mitmproxy activation method, we can change to:

mitmweb

to use the new Web GUI for operation and observation.

mitmweb

The main event, sniffing APP data:

After setting up and familiarizing yourself with the above environment, you can proceed to our main event; sniffing the data transmission content of the APP API!

Here we use a certain real estate APP as an example, purely for academic exchange with no malicious intent!

We want to know how the API for the object list is requested and what content is returned!

First press “z” to clear all records (to avoid confusion)

Open the target APP

Open the target APP and try “pull to refresh” or trigger the “load next page” action.

🛑If your target APP cannot be opened or connected; sorry, it means the APP has protection measures and cannot be sniffed using this method. Please scroll down to the section on how to protect it🛑

mitmproxy records

Go back to mitmproxy to check the records, use your detective skills to guess which API request record is the one we want and enter to view the details!

Request

In the Request section, you can see what parameters were passed in the request.

With “e” to edit and “r” to resend, and observing the Response, you can guess the purpose of each parameter!

Response

The Response section also directly provides the original returned content.

🛑If the Response content is a bunch of codes; sorry, it means the APP might have its own encryption and decryption, making it impossible to sniff using this method. Please scroll down to the section on how to protect it🛑

Hard to read? Chinese garbled text? No problem, you can use “b” to export it as a text file to the desktop, then copy the content to Json Editor Online for parsing!

Or directly use mitmweb to browse and operate using the web GUI

mitmweb

After sniffing, observing, filtering, and testing, you can understand how the APP API works, and thus use it to scrape data.

After collecting the required information, remember to turn off mitmproxy and change the mobile network proxy server back to automatic to use the internet normally.

How should the APP protect itself?

If after setting up mitmproxy, you find that the APP cannot be used or the returned content is encoded, it means the APP has protection.

Method (1):

Generally, it involves placing a copy of the certificate information in the APP. If the current HTTPS certificate does not match the information in the APP, access is denied. For details, you can see this or find related resources on SSL Pinning. The downside might be the need to pay attention to the certificate’s validity period!

$[https://medium.com/@dzungnguyen.hcm/ios-ssl-pinning-bffd2ee9efc](https://medium.com/@dzungnguyen.hcm/ios-ssl-pinning-bffd2ee9efc){:target="_blank"}$

https://medium.com/@dzungnguyen.hcm/ios-ssl-pinning-bffd2ee9efc

Method (2):

The APP encodes and encrypts the data before transmission. The API backend decrypts it to obtain the original request content. The API response is also encoded and encrypted before being sent back. The APP decrypts the received data to get the response content. This method is cumbersome and inefficient, but it is indeed a way to protect data. As far as I know, some digital banks use this method for protection!

However…

Method 1 still has a way to be cracked: How to Bypass SSL Pinning on iOS 12

Method 2 can also be compromised through reverse engineering to obtain the encryption keys.

⚠️There is no 100% security⚠️

Or simply create a trap to collect evidence and solve it legally (？

As always:

“NEVER TRUST THE CLIENT”

More uses of mitmproxy:

1. Using mitmdump

Besides mitmproxy and mitmweb, mitmdump can directly export all records to a text file:

mitmdump -w /log.txt

You can also use Method (2) with a Python script to set and filter traffic:

  
mitmdump -ns examples/filter.py -r /log.txt -w /result.txt

2. Use a Python script for request parameter settings, access control, and redirection:

  
from mitmproxy import http

def request(flow: http.HTTPFlow) -> None:
    # pretty_host takes the "Host" header of the request into account,
    # which is useful in transparent mode where we usually only have the IP
    # otherwise.
    
    # Request parameter setting Example:
    flow.request.headers['User-Agent'] = 'MitmProxy'
    
    if flow.request.pretty_host == "123.com.tw":
        flow.request.host = "456.com.tw"
    # Redirect all access from 123.com.tw to 456.com.tw

Redirection example

When starting mitmproxy, add the parameter:

mitmproxy -s /redirect.py
or
mitmweb -s /redirect.py
or
mitmdump -s /redirect.py

Filling a gap

When using mitmproxy to observe requests using HTTP 1.1 and Accept-Ranges: bytes, Content-Range for long connection segment continuous resource fetching, it will wait until the entire response is received before displaying, rather than showing segments and using persistent connections to continue downloading!

Details here.

Postscript

Since I don’t have domain permissions, I can’t obtain SSL certificate information, so I can’t implement it. The code doesn’t seem difficult, and although there’s no 100% secure method, adding an extra layer of protection can make it safer. Further attacks would require a lot of time to research, which should deter 90% of crawlers!

This article might be a bit low in value. I’ve neglected Medium for a while (playing with a DSLR). Mainly, this is to warm up for iPlayground 2019 this weekend (2019/09/21–2019/09/22). Looking forward to this year’s sessions 🤩, and hope to produce more quality articles after returning!

[Updated on 2019/02/22] What is the Experience of iPlayground 2019?

If you have any questions or feedback, feel free to contact me.

===

本文中文版本

===

This article was first published in Traditional Chinese on Medium ➡️ View Here

The APP uses HTTPS for transmission, but the data was still stolen.