Exploring Vision — APP Avatar Upload Automatic Face Detection and Cropping (Swift)

Exploring Vision — APP Avatar Upload Automatic Face Detection and Cropping (Swift)

Practical application of Vision

Without further ado, here’s a finished product image:

Before Optimization V.S After Optimization — Wedding App

Recently, with the release of iOS 12, I noticed the newly opened CoreML machine learning framework; it seemed quite interesting, so I started thinking about where it could be applied in our current product.

CoreML introductory article now published: Automatically Predict Article Categories Using Machine Learning, Even Training the Model Yourself

CoreML provides interfaces for training and referencing text and image machine learning models in an app. My initial idea was to use CoreML for face recognition to solve the problem of heads or faces being cropped out in the app, as shown on the left in the image above. If a face appears around the edges, it is easily cut off due to scaling and cropping.

After some online research, I realized my knowledge was limited. This feature was already released in iOS 11: the “Vision” framework, which supports text detection, face detection, image matching, QR code detection, object tracking, and more.

Here, we use the face detection feature, optimized as shown on the right; it finds the face and crops the image centered on it.

Let’s Get Started:

First, let’s create a feature that can mark the face position to get a preliminary understanding of how to use Vision.

Demo APP

The completed image above shows the ability to mark the position of faces in a photo.

p.s It can only mark “faces,” not the entire head including hair 😅

This code is mainly divided into two parts. The first part addresses the issue of leaving blank spaces when scaling the original image size to fit into an ImageView. Simply put, we want the ImageView to be the same size as the Image. If you directly insert the image, it will cause misalignment as shown below.

You might think of directly changing the ContentMode to fill, fit, or redraw, but this will distort or crop the image.

let ratio = UIScreen.main.bounds.size.width
// This is because I set the UIIMAGEVIEW to align left and right with a 0 margin, and the aspect ratio is 1:1

let sourceImage = UIImage(named: "Demo2")?.kf.resize(to: CGSize(width: ratio, height: CGFloat.leastNonzeroMagnitude), for: .aspectFill)
// Using KingFisher's image resizing feature, based on width, with height being flexible

imageView.contentMode = .redraw
// Set contentMode to redraw to fill

imageView.image = sourceImage
// Assign the image

imageViewConstraints.constant = (ratio - (sourceImage?.size.height ?? 0))
imageView.layoutIfNeeded()
imageView.sizeToFit()
// This part changes the imageView's constraints. For details, see the complete example at the end of the article.

Here is the processing for images

The cropping part uses Kingfisher to help us, but you can replace it with other libraries or custom methods.

The second part, let’s get straight to the code

if #available(iOS 11.0, *) {
    // Supported only after iOS 11
    let completionHandle: VNRequestCompletionHandler = { request, error in
        if let faceObservations = request.results as? [VNFaceObservation] {
            // Recognized faces
            
            DispatchQueue.main.async {
                // Operate UIView, switch back to the main thread
                let size = self.imageView.frame.size
                
                faceObservations.forEach({ (faceObservation) in
                    // Coordinate system transformation
                    let translate = CGAffineTransform.identity.scaledBy(x: size.width, y: size.height)
                    let transform = CGAffineTransform(scaleX: 1, y: -1).translatedBy(x: 0, y: -size.height)
                    let transRect =  faceObservation.boundingBox.applying(translate).applying(transform)
                    
                    let markerView = UIView(frame: transRect)
                    markerView.backgroundColor = UIColor.init(red: 0/255, green: 255/255, blue: 0/255, alpha: 0.3)
                    self.imageView.addSubview(markerView)
                })
            }
        } else {
            print("No faces detected")
        }
    }
    
    // Recognition request
    let baseRequest = VNDetectFaceRectanglesRequest(completionHandler: completionHandle)
    let faceHandle = VNImageRequestHandler(ciImage: ciImage, options: [:])
    DispatchQueue.global().async {
        // Recognition takes time, so put it in the background thread to avoid blocking the current screen
        do{
            try faceHandle.perform([baseRequest])
        }catch{
            print("Throws：\(error)")
        }
    }
  
} else {
    //
    print("Not supported")
}

The main thing to note is the coordinate system transformation part; the recognized result is the original coordinates of the image; we need to convert it to the actual coordinates of the enclosing ImageView to use it correctly.

Next, let’s do the main task of today — cropping the correct position of the avatar according to the face position

let ratio = UIScreen.main.bounds.size.width
// This is because I set the UIImageView to align left and right to 0, with an aspect ratio of 1:1. For details, see the complete example at the end of the article

let sourceImage = UIImage(named: "Demo")

imageView.contentMode = .scaleAspectFill
// Use scaleAspectFill mode to fill

imageView.image = sourceImage
// Directly assign the original image, we will operate on it later

if let image = sourceImage,#available(iOS 11.0, *),let ciImage = CIImage(image: image) {
    let completionHandle: VNRequestCompletionHandler = { request, error in
        if request.results?.count == 1,let faceObservation = request.results?.first as? VNFaceObservation {
            // One face
            let size = CGSize(width: ratio, height: ratio)
            
            let translate = CGAffineTransform.identity.scaledBy(x: size.width, y: size.height)
            let transform = CGAffineTransform(scaleX: 1, y: -1).translatedBy(x: 0, y: -size.height)
            let finalRect =  faceObservation.boundingBox.applying(translate).applying(transform)
            
            let center = CGPoint(x: (finalRect.origin.x + finalRect.width/2 - size.width/2), y: (finalRect.origin.y + finalRect.height/2 - size.height/2))
            // Here is to calculate the midpoint position of the face range
            
            let newImage = image.kf.resize(to: size, for: .aspectFill).kf.crop(to: size, anchorOn: center)
            // Crop the image according to the midpoint
            
            DispatchQueue.main.async {
                // Operate UIView, switch back to the main thread
                self.imageView.image = newImage
            }
        } else {
            print("Detected multiple faces or no face detected")
        }
    }
    let baseRequest = VNDetectFaceRectanglesRequest(completionHandler: completionHandle)
    let faceHandle = VNImageRequestHandler(ciImage: ciImage, options: [:])
    DispatchQueue.global().async {
        do{
            try faceHandle.perform([baseRequest])
        }catch{
            print("Throws：\(error)")
        }
    }
} else {
    print("Not supported")
}

The principle is similar to marking the position of a face, but the difference is that the profile picture is of a fixed size (e.g., 300x300), so we skip the first part where the Image needs to fit the ImageView.

Another difference is that we need to calculate the center point of the face area and use this center point as the basis for cropping the image.

The red dot is the center point of the face area

Final effect diagram:

The moment before the pause is the original image position

Complete APP example:

The code has been uploaded to Github: Click here

If you have any questions or comments, feel free to contact me.

===

本文中文版本

===

This article was first published in Traditional Chinese on Medium ➡️ View Here