Comprehensive Guide to Implementing Local Cache with AVPlayer

Comprehensive Guide to Implementing Local Cache with AVPlayer

AVPlayer/AVQueuePlayer with AVURLAsset implementing AVAssetResourceLoaderDelegate

Photo by Tyler Lastovich

[2023/03/12] Update

I have open-sourced my previous implementation, and those in need can use it directly.

• Customizable Cache strategy, can use PINCache or others…
• Externally, just call the make AVAsset factory, input the URL, and the AVAsset will support Caching
• Implemented Data Flow strategy using Combine
• Wrote some tests

Introduction

It’s been more than half a year since the last post “Exploring Methods for Implementing iOS HLS Cache”, and the team has always wanted to implement the cache-while-playing feature because it greatly impacts costs. We are a music streaming platform, and if we have to fetch the entire file every time the same song is played, it would be a huge data drain for us and for users who don’t have unlimited data plans. Although music files are at most a few MB, it all adds up to significant costs!

Additionally, since the Android side has already implemented the cache-while-playing feature, we previously compared the costs and found that after launching on Android, there was a significant reduction in data usage. With relatively more users on iOS, we should see even better data savings.

Based on the experience from the previous post, if we continue to use HLS (.m3u8/.ts) to achieve our goal, things will become very complicated and possibly unachievable. So, we decided to revert to using mp3 files, which allows us to directly use AVAssetResourceLoaderDelegate for implementation.

Goals

• Music that has been played will generate a local Cache backup
• When playing music, first check if there is a local Cache to read from; if so, do not request the file from the server again
• Can set Cache strategies; total capacity limit, start deleting the oldest Cache files when exceeded
• Do not interfere with the original AVPlayer playback mechanism (The fastest method would be to use URLSession to download the mp3 and feed it to AVPlayer, but this would lose the ability to play while downloading, making users wait longer and consuming more data)

Preliminary Knowledge (1) — HTTP/1.1 Range Requests, Connection Keep-Alive

HTTP/1.1 Range Requests

First, we need to understand how data is requested from the server when playing videos or music. Generally, video and music files are very large, and it is not feasible to wait until the entire file is fetched before starting playback. The common approach is to fetch data as it plays, only needing the data for the currently playing segment.

The way to achieve this is through HTTP/1.1 Range, which only returns the specified byte range of data, for example, specifying 0–100 will only return the 100 bytes of data from 0–100. Using this method, data can be fetched in segments and then assembled into a complete file. This method can also be applied to resume interrupted downloads.

How to Apply?

We will first use HEAD to check the Response Header to understand if the server supports Range requests, the total length of the resource, and the file type:

curl -i -X HEAD http://zhgchg.li/music.mp3

Using HEAD, we can get the following information from the Response Header:

• Accept-Ranges: bytes indicates that the server supports Range requests. If this value is missing or is Accept-Ranges: none, it means it does not support it.
• Content-Length: The total length of the resource. We need to know the total length to request data in segments.
• Content-Type: The file type, which is information needed by AVPlayer when playing.

However, sometimes we also use GET Range: bytes=0–1, which means we request data in the range of 0–1, but we don’t actually care about the content of 0–1. We just want to see the Response Header information; the native AVPlayer uses GET to check, so this article will also use it.

But it is more recommended to use HEAD to check. One method is more correct, and if the server does not support the Range function, using GET will force the download of the entire file.

curl -i -X GET http://zhgchg.li/music.mp3 -H "Range: bytes=0–1"

Using GET, we can get the following information from the Response Header:

• Accept-Ranges: bytes indicates that the server supports Range requests. If this value is missing or is Accept-Ranges: none, it means it does not support it.
• Content-Range: bytes 0–1/total length of the resource, the number after the “/” is the total length of the resource. We need to know the total length to request data in segments.
• Content-Type: The file type, which is information needed by AVPlayer when playing.

Knowing that the server supports Range requests, we can initiate segmented Range requests:

curl -i -X GET http://zhgchg.li/music.mp3 -H "Range: bytes=0–100"

The server will return 206 Partial Content:

Content-Range: bytes 0-100/total length
Content-Length: 100
...
(binary content)

At this point, we get the data for Range 0–100 and can continue to make new requests for Range 100–200, 200–300, and so on until the end.

If the requested Range exceeds the total length of the resource, it will return 416 Range Not Satisfiable.

Additionally, to get the complete file data, you can request Range 0-total length or use 0-:

curl -i -X GET http://zhgchg.li/music.mp3 -H "Range: bytes=0–"

You can also request multiple Range data in the same request and set conditions, but we don’t need that. For more details, you can refer here.

Connection Keep-Alive

HTTP 1.1 is enabled by default. This feature allows real-time retrieval of downloaded data, for example, a 5 MB file can be retrieved in 16 KB, 16 KB, 16 KB… increments, without waiting for the entire 5 MB to be downloaded.

Connection: Keep-Alive

What if the server does not support Range or Keep-Alive ?

Then there’s no need to do so much. Just use URLSession to download the mp3 file and feed it to the player… But this is not the result we want, so you can ask the backend to modify the server settings.

Preliminary Knowledge (2) — How does the native AVPlayer handle AVURLAsset resources?

When we use AVURLAsset to initialize with a URL resource and assign it to AVPlayer/AVQueuePlayer to start playing, as mentioned above, it will first use GET Range 0–1 to obtain whether it supports Range requests, the total length of the resource, and the file type.

With the file information, a second request will be initiated to request data from 0 to the total length.

⚠️ AVPlayer will request data from 0 to the total length and will cancel the network request once it feels it has enough data (e.g., 16 kb, 16 kb, 16 kb…) (so it won’t actually fetch the entire file unless the file is very small).

It will continue to request data using Range after resuming playback.

(This part is different from what I previously thought; I assumed it would request 0–100, 100–200, etc.)

AVPlayer Request Example:

1. GET Range 0-1 => Response: Total length 150000 / public.mp3 / true
2. GET 0-150000...
3. 16 kb receive
4. 16 kb receive...
5. cancel() // current offset is 700
6. Continue playback
7. GET 700-150000...
8. 16 kb receive
9. 16 kb receive...
10. cancel() // current offset is 1500
11. Continue playback
12. GET 1500-150000...
13. 16 kb receive
14. 16 kb receive...
16. If seek to...5000
17. cancel(12.) // current offset is 2000
18. GET 5000-150000...
19. 16 kb receive
20. 16 kb receive...
...

⚠️ In iOS ≤12, it will first send a few shorter requests to test (?), and then send a request for the total length; in iOS ≥ 13, it will directly send a request for the total length.

Another side issue is that while observing how resources are fetched, I used the mitmproxy tool for sniffing. It showed errors, waiting for the entire response to come back before displaying it, instead of showing segments and using persistent connections for continued downloads. This scared me! I thought iOS was dumb enough to fetch the entire file each time! Next time, I need to be a bit skeptical when using tools Orz.

Timing of Cancel Initiation

1. As mentioned earlier, the second request, which requests resources from 0 to the total length, will initiate a Cancel request once there is enough data.
2. When seeking, it will first initiate a Cancel request for the previous request.

⚠️ Switching to the next resource in AVQueuePlayer or changing the playback resource in AVPlayer will not initiate a Cancel request for the previous track.

AVQueue Pre-buffering

It also calls the Resource Loader to handle it, but the requested data range will be smaller.

Implementation

With the above preliminary knowledge, let’s look at how to implement the local cache function of AVPlayer.

As mentioned earlier, AVAssetResourceLoaderDelegate allows us to implement the Resource Loader for the Asset.

The Resource Loader is essentially a worker. Whether the player needs file information or file data, and the range, it tells us, and we do it.

I saw an example where a Resource Loader serves all AVURLAssets, which I think is wrong. It should be one Resource Loader serving one AVURLAsset, following the lifecycle of the AVURLAsset, as it belongs to the AVURLAsset.

A Resource Loader serving all AVURLAssets in AVQueuePlayer would become very complex and difficult to manage.

Timing of Entering Custom Resource Loader

Note that implementing your own Resource Loader doesn’t mean it will handle everything. It will only use your Resource Loader when the system cannot recognize or handle the resource.

Therefore, before giving the URL resource to AVURLAsset, we need to change the Scheme to our custom Scheme, not http/https… which the system can handle.

http://zhgchg.li/music.mp3 => cacheable://zhgchg.li/music.mp3

AVAssetResourceLoaderDelegate

Only two methods need to be implemented:

• func resourceLoader(_ resourceLoader: AVAssetResourceLoader, shouldWaitForLoadingOfRequestedResource loadingRequest: AVAssetResourceLoadingRequest) -> Bool:

This method asks us if we can handle this resource. Return true if we can, return false if we cannot (unsupported URL).

We can extract what is being requested from loadingRequest (whether it is the first request for file information or a data request, and if it is a data request, what the Range is). After knowing the request, we initiate our own request to fetch the data. Here we can decide whether to initiate a URLSession or return Data from local storage.

Additionally, we can perform Data encryption and decryption operations here to protect the original data.

• func resourceLoader(_ resourceLoader: AVAssetResourceLoader, didCancel loadingRequest: AVAssetResourceLoadingRequest):

As mentioned earlier, Cancel initiation timing when Cancel is initiated…

We can cancel the ongoing URLSession request here.

Local Cache Implementation

For the Cache part, I directly use PINCache, delegating the Cache work to it, avoiding issues like Cache read/write DeadLock and implementing Cache LRU strategy.

️️⚠️️️️️️️️️️️OOM Warning!

Since this is for caching music files with a size of around 10 MB, PINCache can be used as a local Cache tool. However, this method cannot be used for serving videos (which may require loading several GB of data into memory at once).

For such requirements, you can refer to the approach of using FileHandle’s seek read/write features.

Let’s Get Started!

Without further ado, here is the complete project:

AssetData

Local Cache data object mapping implements NSCoding, as PINCache relies on the archivedData method for encoding/decoding.

import Foundation
import CryptoKit

class AssetDataContentInformation: NSObject, NSCoding {
    @objc var contentLength: Int64 = 0
    @objc var contentType: String = ""
    @objc var isByteRangeAccessSupported: Bool = false
    
    func encode(with coder: NSCoder) {
        coder.encode(self.contentLength, forKey: #keyPath(AssetDataContentInformation.contentLength))
        coder.encode(self.contentType, forKey: #keyPath(AssetDataContentInformation.contentType))
        coder.encode(self.isByteRangeAccessSupported, forKey: #keyPath(AssetDataContentInformation.isByteRangeAccessSupported))
    }
    
    override init() {
        super.init()
    }
    
    required init?(coder: NSCoder) {
        super.init()
        self.contentLength = coder.decodeInt64(forKey: #keyPath(AssetDataContentInformation.contentLength))
        self.contentType = coder.decodeObject(forKey: #keyPath(AssetDataContentInformation.contentType)) as? String ?? ""
        self.isByteRangeAccessSupported = coder.decodeObject(forKey: #keyPath(AssetDataContentInformation.isByteRangeAccessSupported)) as? Bool ?? false
    }
}

class AssetData: NSObject, NSCoding {
    @objc var contentInformation: AssetDataContentInformation = AssetDataContentInformation()
    @objc var mediaData: Data = Data()
    
    override init() {
        super.init()
    }

    func encode(with coder: NSCoder) {
        coder.encode(self.contentInformation, forKey: #keyPath(AssetData.contentInformation))
        coder.encode(self.mediaData, forKey: #keyPath(AssetData.mediaData))
    }
    
    required init?(coder: NSCoder) {
        super.init()
        self.contentInformation = coder.decodeObject(forKey: #keyPath(AssetData.contentInformation)) as? AssetDataContentInformation ?? AssetDataContentInformation()
        self.mediaData = coder.decodeObject(forKey: #keyPath(AssetData.mediaData)) as? Data ?? Data()
    }
}

AssetData contains:

• contentInformation : AssetDataContentInformation AssetDataContentInformation: Contains whether Range requests are supported (isByteRangeAccessSupported), total resource length (contentLength), file type (contentType)
• mediaData : Original audio Data (large files here may cause OOM)

PINCacheAssetDataManager

Encapsulates the logic for storing and retrieving Data in PINCache.

import PINCache
import Foundation

protocol AssetDataManager: NSObject {
    func retrieveAssetData() -> AssetData?
    func saveContentInformation(_ contentInformation: AssetDataContentInformation)
    func saveDownloadedData(_ data: Data, offset: Int)
    func mergeDownloadedDataIfIsContinuted(from: Data, with: Data, offset: Int) -> Data?
}

extension AssetDataManager {
    func mergeDownloadedDataIfIsContinuted(from: Data, with: Data, offset: Int) -> Data? {
        if offset <= from.count && (offset + with.count) > from.count {
            let start = from.count - offset
            var data = from
            data.append(with.subdata(in: start..<with.count))
            return data
        }
        return nil
    }
}

//

class PINCacheAssetDataManager: NSObject, AssetDataManager {
    
    static let Cache: PINCache = PINCache(name: "ResourceLoader")
    let cacheKey: String
    
    init(cacheKey: String) {
        self.cacheKey = cacheKey
        super.init()
    }
    
    func saveContentInformation(_ contentInformation: AssetDataContentInformation) {
        let assetData = AssetData()
        assetData.contentInformation = contentInformation
        PINCacheAssetDataManager.Cache.setObjectAsync(assetData, forKey: cacheKey, completion: nil)
    }
    
    func saveDownloadedData(_ data: Data, offset: Int) {
        guard let assetData = self.retrieveAssetData() else {
            return
        }
        
        if let mediaData = self.mergeDownloadedDataIfIsContinuted(from: assetData.mediaData, with: data, offset: offset) {
            assetData.mediaData = mediaData
            
            PINCacheAssetDataManager.Cache.setObjectAsync(assetData, forKey: cacheKey, completion: nil)
        }
    }
    
    func retrieveAssetData() -> AssetData? {
        guard let assetData = PINCacheAssetDataManager.Cache.object(forKey: cacheKey) as? AssetData else {
            return nil
        }
        return assetData
    }
}

Here, we extract the Protocol because we might use other storage methods to replace PINCache in the future. Therefore, other programs should rely on the Protocol rather than the Class instance when using it.

⚠️ mergeDownloadedDataIfIsContinuted This method is extremely important.

For linear playback, you just need to keep appending new Data to the Cache Data, but the real situation is much more complicated. The user might play Range 0~100 and then directly Seek to Range 200–500 for playback. How to merge the existing 0-100 Data with the new 200–500 Data is a big problem.

⚠️ Data merging issues can lead to terrible playback glitches…

The answer here is, we do not handle non-continuous data; because our project is only for audio, and the files are just a few MB (≤ 10MB), considering the development cost, we didn’t do it. I only handle merging continuous data (for example, currently having 0~100, and the new data is 75~200, after merging it becomes 0~200; if the new data is 150~200, I will ignore it and not merge).

If you want to consider non-continuous merging, besides using other methods for storage (to identify the missing parts), you also need to be able to query which segment needs a network request and which segment is taken locally during the Request. Considering this situation, the implementation will be very complicated.

Image source: iOS AVPlayer Video Cache Design and Implementation

CachingAVURLAsset

AVURLAsset weakly holds the ResourceLoader Delegate, so it is recommended to create an AVURLAsset Class that inherits from AVURLAsset, internally create, assign, and hold the ResourceLoader, allowing it to follow the lifecycle of AVURLAsset. Additionally, you can store information such as the original URL, CacheKey, etc.

class CachingAVURLAsset: AVURLAsset {
    static let customScheme = "cacheable"
    let originalURL: URL
    private var _resourceLoader: ResourceLoader?
    
    var cacheKey: String {
        return self.url.lastPathComponent
    }
    
    static func isSchemeSupport(_ url: URL) -> Bool {
        guard let components = URLComponents(url: url, resolvingAgainstBaseURL: false) else {
            return false
        }
        
        return ["http", "https"].contains(components.scheme)
    }
    
    override init(url URL: URL, options: [String: Any]? = nil) {
        self.originalURL = URL
        
        guard var components = URLComponents(url: URL, resolvingAgainstBaseURL: false) else {
            super.init(url: URL, options: options)
            return
        }
        
        components.scheme = CachingAVURLAsset.customScheme
        guard let url = components.url else {
            super.init(url: URL, options: options)
            return
        }
        
        super.init(url: url, options: options)
        
        let resourceLoader = ResourceLoader(asset: self)
        self.resourceLoader.setDelegate(resourceLoader, queue: resourceLoader.loaderQueue)
        self._resourceLoader = resourceLoader
    }
}

Usage:

if CachingAVURLAsset.isSchemeSupport(url) {
  let asset = CachingAVURLAsset(url: url)
  let avplayer = AVPlayer(asset)
  avplayer.play()
}

Where isSchemeSupport() is used to determine if the URL supports our Resource Loader (excluding file://).

originalURL stores the original resource URL.

cacheKey stores the Cache Key for this resource, here we directly use the file name as the Cache Key.

Please adjust cacheKey according to real-world scenarios. If the file name is not hashed and may be duplicated, it is recommended to hash it first to avoid collisions; if you want to hash the entire URL as the key, also pay attention to whether the URL will change (e.g., using CDN).

Hashing can use md5…sha… iOS ≥ 13 can directly use Apple’s CryptoKit, for others, check Github!

ResourceLoaderRequest

import Foundation
import CoreServices

protocol ResourceLoaderRequestDelegate: AnyObject {
    func dataRequestDidReceive(_ resourceLoaderRequest: ResourceLoaderRequest, _ data: Data)
    func dataRequestDidComplete(_ resourceLoaderRequest: ResourceLoaderRequest, _ error: Error?, _ downloadedData: Data)
    func contentInformationDidComplete(_ resourceLoaderRequest: ResourceLoaderRequest, _ result: Result<AssetDataContentInformation, Error>)
}

class ResourceLoaderRequest: NSObject, URLSessionDataDelegate {
    struct RequestRange {
        var start: Int64
        var end: RequestRangeEnd
        
        enum RequestRangeEnd {
            case requestTo(Int64)
            case requestToEnd
        }
    }
    
    enum RequestType {
        case contentInformation
        case dataRequest
    }
    
    struct ResponseUnExpectedError: Error { }
    
    private let loaderQueue: DispatchQueue
    
    let originalURL: URL
    let type: RequestType
    
    private var session: URLSession?
    private var dataTask: URLSessionDataTask?
    private var assetDataManager: AssetDataManager?
    
    private(set) var requestRange: RequestRange?
    private(set) var response: URLResponse?
    private(set) var downloadedData: Data = Data()
    
    private(set) var isCancelled: Bool = false {
        didSet {
            if isCancelled {
                self.dataTask?.cancel()
                self.session?.invalidateAndCancel()
            }
        }
    }
    private(set) var isFinished: Bool = false {
        didSet {
            if isFinished {
                self.session?.finishTasksAndInvalidate()
            }
        }
    }
    
    weak var delegate: ResourceLoaderRequestDelegate?
    
    init(originalURL: URL, type: RequestType, loaderQueue: DispatchQueue, assetDataManager: AssetDataManager?) {
        self.originalURL = originalURL
        self.type = type
        self.loaderQueue = loaderQueue
        self.assetDataManager = assetDataManager
        super.init()
    }
    
    func start(requestRange: RequestRange) {
        guard isCancelled == false, isFinished == false else {
            return
        }
        
        self.loaderQueue.async { [weak self] in
            guard let self = self else {
                return
            }
            
            var request = URLRequest(url: self.originalURL)
            self.requestRange = requestRange
            let start = String(requestRange.start)
            let end: String
            switch requestRange.end {
            case .requestTo(let rangeEnd):
                end = String(rangeEnd)
            case .requestToEnd:
                end = ""
            }
            
            let rangeHeader = "bytes=\(start)-\(end)"
            request.setValue(rangeHeader, forHTTPHeaderField: "Range")
            
            let session = URLSession(configuration: .default, delegate: self, delegateQueue: nil)
            self.session = session
            let dataTask = session.dataTask(with: request)
            self.dataTask = dataTask
            dataTask.resume()
        }
    }
    
    func cancel() {
        self.isCancelled = true
    }
    
    func urlSession(_ session: URLSession, dataTask: URLSessionDataTask, didReceive data: Data) {
        guard self.type == .dataRequest else {
            return
        }
        
        self.loaderQueue.async {
            self.delegate?.dataRequestDidReceive(self, data)
            self.downloadedData.append(data)
        }
    }
    
    func urlSession(_ session: URLSession, dataTask: URLSessionDataTask, didReceive response: URLResponse, completionHandler: @escaping (URLSession.ResponseDisposition) -> Void) {
        self.response = response
        completionHandler(.allow)
    }
    
    func urlSession(_ session: URLSession, task: URLSessionTask, didCompleteWithError error: Error?) {
        self.isFinished = true
        self.loaderQueue.async {
            if self.type == .contentInformation {
                guard error == nil,
                      let response = self.response as? HTTPURLResponse else {
                    let responseError = error ?? ResponseUnExpectedError()
                    self.delegate?.contentInformationDidComplete(self, .failure(responseError))
                    return
                }
                
                let contentInformation = AssetDataContentInformation()
                
                if let rangeString = response.allHeaderFields["Content-Range"] as? String,
                   let bytesString = rangeString.split(separator: "/").map({String($0)}).last,
                   let bytes = Int64(bytesString) {
                    contentInformation.contentLength = bytes
                }
                
                if let mimeType = response.mimeType,
                   let contentType = UTTypeCreatePreferredIdentifierForTag(kUTTagClassMIMEType, mimeType as CFString, nil)?.takeRetainedValue() {
                    contentInformation.contentType = contentType as String
                }
                
                if let value = response.allHeaderFields["Accept-Ranges"] as? String,
                   value == "bytes" {
                    contentInformation.isByteRangeAccessSupported = true
                } else {
                    contentInformation.isByteRangeAccessSupported = false
                }
                
                self.assetDataManager?.saveContentInformation(contentInformation)
                self.delegate?.contentInformationDidComplete(self, .success(contentInformation))
            } else {
                if let offset = self.requestRange?.start, self.downloadedData.count > 0 {
                    self.assetDataManager?.saveDownloadedData(self.downloadedData, offset: Int(offset))
                }
                self.delegate?.dataRequestDidComplete(self, error, self.downloadedData)
            }
        }
    }
}

Encapsulation for Remote Request, mainly for data requests initiated by ResourceLoader.

RequestType: Used to distinguish whether this Request is the first request for file information (contentInformation) or a data request (dataRequest).

RequestRange: Request Range scope, end can specify to where (requestTo(Int64)) or all (requestToEnd).

File information can be obtained from:

func urlSession(_ session: URLSession, dataTask: URLSessionDataTask, didReceive response: URLResponse, completionHandler: @escaping (URLSession.ResponseDisposition) -> Void)

Get the Response Header from it. Additionally, note that if you want to change to HEAD, it won’t enter here; you need to use other methods to receive it.

• isByteRangeAccessSupported: Check Accept-Ranges == bytes in the Response Header.
• contentType: The file type information required by the player, formatted as a Uniform Type Identifier, not audio/mpeg, but written as public.mp3.
• contentLength: Check Content-Range in the Response Header: bytes 0–1/ total length of the resource.

⚠️ Note that the format given by the server may vary in case sensitivity. It may not be written as Accept-Ranges/Content-Range; some servers use lowercase accept-ranges, Accept-ranges…

Supplement: If you need to consider case sensitivity, you can write an HTTPURLResponse Extension

import CoreServices

extension HTTPURLResponse {
    func parseContentLengthFromContentRange() -> Int64? {
        let contentRangeKeys: [String] = [
            "Content-Range",
            "content-range",
            "Content-range",
            "content-Range"
        ]
        
        var rangeString: String?
        for key in contentRangeKeys {
            if let value = self.allHeaderFields[key] as? String {
                rangeString = value
                break
            }
        }
        
        guard let rangeString = rangeString,
              let contentLengthString = rangeString.split(separator: "/").map({String($0)}).last,
              let contentLength = Int64(contentLengthString) else {
            return nil
        }
        
        return contentLength
    }
    
    func parseAcceptRanges() -> Bool? {
        let contentRangeKeys: [String] = [
            "Accept-Ranges",
            "accept-ranges",
            "Accept-ranges",
            "accept-Ranges"
        ]
        
        var rangeString: String?
        for key in contentRangeKeys {
            if let value = self.allHeaderFields[key] as? String {
                rangeString = value
                break
            }
        }
        
        guard let rangeString = rangeString else {
            return nil
        }
        
        return rangeString == "bytes" || rangeString == "Bytes"
    }
    
    func mimeTypeUTI() -> String? {
        guard let mimeType = self.mimeType,
           let contentType = UTTypeCreatePreferredIdentifierForTag(kUTTagClassMIMEType, mimeType as CFString, nil)?.takeRetainedValue() else {
            return nil
        }
        
        return contentType as String
    }
}

Usage:

• contentLength = response.parseContentLengthFromContentRange()
• isByteRangeAccessSupported = response.parseAcceptRanges()
• contentType = response.mimeTypeUTI()

func urlSession(_ session: URLSession, dataTask: URLSessionDataTask, didReceive data: Data)

As mentioned in the preliminary knowledge, the downloaded data will be obtained in real-time, so this method will keep getting called, receiving Data in fragments; we will append it to downloadedData for storage.

func urlSession(_ session: URLSession, task: URLSessionTask, didCompleteWithError error: Error?)

This method is called when the task is canceled or completed, where the downloaded data will be saved.

As mentioned in the preliminary knowledge about the Cancel mechanism, since the player will initiate a Cancel Request after obtaining enough data, when this method is called, the actual error = NSURLErrorCancelled will be received. Therefore, regardless of the error, we will try to save the data if we have received it.

⚠️ Since URLSession requests data concurrently, please ensure all operations are performed within DispatchQueue to avoid data corruption (data corruption can also result in playback issues).

⚠️ If URLSession does not call finishTasksAndInvalidate or invalidateAndCancel, it will strongly retain objects, causing a Memory Leak. Therefore, whether canceling or completing, we must call these methods to release the Request when the task ends.

⚠️ If you are concerned about downloadedData causing OOM, you can save it locally in didReceive Data.

ResourceLoader

import AVFoundation
import Foundation

class ResourceLoader: NSObject {
    
    let loaderQueue = DispatchQueue(label: "li.zhgchg.resourceLoader.queue")
    
    private var requests: [AVAssetResourceLoadingRequest: ResourceLoaderRequest] = [:]
    private let cacheKey: String
    private let originalURL: URL
    
    init(asset: CachingAVURLAsset) {
        self.cacheKey = asset.cacheKey
        self.originalURL = asset.originalURL
        super.init()
    }

    deinit {
        self.requests.forEach { (request) in
            request.value.cancel()
        }
    }
}

extension ResourceLoader: AVAssetResourceLoaderDelegate {
    func resourceLoader(_ resourceLoader: AVAssetResourceLoader, shouldWaitForLoadingOfRequestedResource loadingRequest: AVAssetResourceLoadingRequest) -> Bool {
        
        let type = ResourceLoader.resourceLoaderRequestType(loadingRequest)
        let assetDataManager = PINCacheAssetDataManager(cacheKey: self.cacheKey)

        if let assetData = assetDataManager.retrieveAssetData() {
            if type == .contentInformation {
                loadingRequest.contentInformationRequest?.contentLength = assetData.contentInformation.contentLength
                loadingRequest.contentInformationRequest?.contentType = assetData.contentInformation.contentType
                loadingRequest.contentInformationRequest?.isByteRangeAccessSupported = assetData.contentInformation.isByteRangeAccessSupported
                loadingRequest.finishLoading()
                return true
            } else {
                let range = ResourceLoader.resourceLoaderRequestRange(type, loadingRequest)
                if assetData.mediaData.count > 0 {
                    let end: Int64
                    switch range.end {
                    case .requestTo(let rangeEnd):
                        end = rangeEnd
                    case .requestToEnd:
                        end = assetData.contentInformation.contentLength
                    }
                    
                    if assetData.mediaData.count >= end {
                        let subData = assetData.mediaData.subdata(in: Int(range.start)..<Int(end))
                        loadingRequest.dataRequest?.respond(with: subData)
                        loadingRequest.finishLoading()
                       return true
                    } else if range.start <= assetData.mediaData.count {
                        // has cache data...but not enough
                        let subEnd = (assetData.mediaData.count > end) ? Int((end)) : (assetData.mediaData.count)
                        let subData = assetData.mediaData.subdata(in: Int(range.start)..<subEnd)
                        loadingRequest.dataRequest?.respond(with: subData)
                    }
                }
            }
        }
        
        let range = ResourceLoader.resourceLoaderRequestRange(type, loadingRequest)
        let resourceLoaderRequest = ResourceLoaderRequest(originalURL: self.originalURL, type: type, loaderQueue: self.loaderQueue, assetDataManager: assetDataManager)
        resourceLoaderRequest.delegate = self
        self.requests[loadingRequest]?.cancel()
        self.requests[loadingRequest] = resourceLoaderRequest
        resourceLoaderRequest.start(requestRange: range)
        
        return true
    }
    
    func resourceLoader(_ resourceLoader: AVAssetResourceLoader, didCancel loadingRequest: AVAssetResourceLoadingRequest) {
        guard let resourceLoaderRequest = self.requests[loadingRequest] else {
            return
        }
        
        resourceLoaderRequest.cancel()
        requests.removeValue(forKey: loadingRequest)
    }
}

extension ResourceLoader: ResourceLoaderRequestDelegate {
    func contentInformationDidComplete(_ resourceLoaderRequest: ResourceLoaderRequest, _ result: Result<AssetDataContentInformation, Error>) {
        guard let loadingRequest = self.requests.first(where: { $0.value == resourceLoaderRequest })?.key else {
            return
        }
        
        switch result {
        case .success(let contentInformation):
            loadingRequest.contentInformationRequest?.contentType = contentInformation.contentType
            loadingRequest.contentInformationRequest?.contentLength = contentInformation.contentLength
            loadingRequest.contentInformationRequest?.isByteRangeAccessSupported = contentInformation.isByteRangeAccessSupported
            loadingRequest.finishLoading()
        case .failure(let error):
            loadingRequest.finishLoading(with: error)
        }
    }
    
    func dataRequestDidReceive(_ resourceLoaderRequest: ResourceLoaderRequest, _ data: Data) {
        guard let loadingRequest = self.requests.first(where: { $0.value == resourceLoaderRequest })?.key else {
            return
        }
        
        loadingRequest.dataRequest?.respond(with: data)
    }
    
    func dataRequestDidComplete(_ resourceLoaderRequest: ResourceLoaderRequest, _ error: Error?, _ downloadedData: Data) {
        guard let loadingRequest = self.requests.first(where: { $0.value == resourceLoaderRequest })?.key else {
            return
        }
        
        loadingRequest.finishLoading(with: error)
        requests.removeValue(forKey: loadingRequest)
    }
}

extension ResourceLoader {
    static func resourceLoaderRequestType(_ loadingRequest: AVAssetResourceLoadingRequest) -> ResourceLoaderRequest.RequestType {
        if let _ = loadingRequest.contentInformationRequest {
            return .contentInformation
        } else {
            return .dataRequest
        }
    }
    
    static func resourceLoaderRequestRange(_ type: ResourceLoaderRequest.RequestType, _ loadingRequest: AVAssetResourceLoadingRequest) -> ResourceLoaderRequest.RequestRange {
        if type == .contentInformation {
            return ResourceLoaderRequest.RequestRange(start: 0, end: .requestTo(1))
        } else {
            if loadingRequest.dataRequest?.requestsAllDataToEndOfResource == true {
                let lowerBound = loadingRequest.dataRequest?.currentOffset ?? 0
                return ResourceLoaderRequest.RequestRange(start: lowerBound, end: .requestToEnd)
            } else {
                let lowerBound = loadingRequest.dataRequest?.currentOffset ?? 0
                let length = Int64(loadingRequest.dataRequest?.requestedLength ?? 1)
                let upperBound = lowerBound + length
                return ResourceLoaderRequest.RequestRange(start: lowerBound, end: .requestTo(upperBound))
            }
        }
    }
}

loadingRequest.contentInformationRequest != nil indicates the first request, where the player asks for file information.

When requesting file information, we need to provide these three pieces of information:

• loadingRequest.contentInformationRequest?.isByteRangeAccessSupported: Whether Range access to Data is supported
• loadingRequest.contentInformationRequest?.contentType: Uniform type identifier
• loadingRequest.contentInformationRequest?.contentLength: Total file length Int64

loadingRequest.dataRequest?.requestedOffset can get the starting offset of the requested Range.

loadingRequest.dataRequest?.requestedLength can get the length of the requested Range.

loadingRequest.dataRequest?.requestsAllDataToEndOfResource == true means that regardless of the requested Range length, it will fetch until the end.

loadingRequest.dataRequest?.respond(with: Data) returns the loaded Data to the player.

loadingRequest.dataRequest?.currentOffset can get the current data offset, and dataRequest?.respond(with: Data) will shift the currentOffset.

loadingRequest.finishLoading() indicates that all data has been loaded and informs the player.

func resourceLoader(_ resourceLoader: AVAssetResourceLoader, shouldWaitForLoadingOfRequestedResource loadingRequest: AVAssetResourceLoadingRequest) -> Bool

When the player requests data, we first check if there is data in the local Cache. If there is, we return it; if only part of the data is available, we return that part. For example, if we have 0–100 locally and the player requests 0–200, we return 0–100 first.

If there is no local Cache or the returned data is insufficient, a ResourceLoaderRequest will be initiated to fetch data from the network.

func resourceLoader(_ resourceLoader: AVAssetResourceLoader, didCancel loadingRequest: AVAssetResourceLoadingRequest)

The player cancels the request, canceling the ResourceLoaderRequest.

You might have noticed resourceLoaderRequestRange offset is based on currentOffset because we first load the downloaded Data from the local dataRequest?.respond(with: Data); so we can directly look at the shifted offset.

func private var requests: [AVAssetResourceLoadingRequest: ResourceLoaderRequest] = [:]

⚠️ Some examples use currentRequest: ResourceLoaderRequest to store requests, which can be problematic. If the current request is fetching data and the user seeks, the old request will be canceled and a new one initiated. Since these actions may not occur in order, using a Dictionary for storage and operations is safer!

⚠️ Ensure all operations are on the same DispatchQueue to prevent data inconsistencies.

Cancel all ongoing requests during deinit Resource Loader Deinit indicates AVURLAsset Deinit, meaning the player no longer needs this resource. Therefore, we can cancel ongoing Requests, and the already loaded data will still be written to Cache.

Supplement and Acknowledgments

Thanks to Lex 汤 for the guidance.

Thanks to 外孫女 for providing development advice and support.

This article is only for small music files

Large video files may encounter Out Of Memory issues in downloadedData, AssetData/PINCacheAssetDataManager.

As mentioned earlier, to solve this problem, use fileHandler seek read/write to operate local Cache read/write (replacing AssetData/PINCacheAssetDataManager); or look for projects on Github that handle large data write/read to file.

Cancel downloading items when switching playback items in AVQueuePlayer

As stated in the preliminary knowledge, changing the playback target will not trigger a Cancel; if it is AVPlayer, it will go through AVURLAsset Deinit, so the download will also be interrupted; but AVQueuePlayer will not, because it is still in the Queue, only the playback target has switched to the next one.

The only way here is to receive the notification of changing the playback target, and then cancel the loading of the previous AVURLAsset after receiving the notification.

asset.cancelLoading()

Audio data encryption and decryption

Audio encryption and decryption can be performed in ResourceLoaderRequest when obtaining Data, and when storing, encryption and decryption can be performed on the Data stored locally in the encode/decode of AssetData.

CryptoKit SHA usage example:

class AssetData: NSObject, NSCoding {
    static let encryptionKeyString = "encryptionKeyExzhgchgli"
    ...
    func encode(with coder: NSCoder) {
        coder.encode(self.contentInformation, forKey: #keyPath(AssetData.contentInformation))
        
        if #available(iOS 13.0, *),
           let encryptionData = try? ChaChaPoly.seal(self.mediaData, using: AssetData.encryptionKey).combined {
            coder.encode(encryptionData, forKey: #keyPath(AssetData.mediaData))
        } else {
          //
        }
    }
    
    required init?(coder: NSCoder) {
        super.init()
        ...
        if let mediaData = coder.decodeObject(forKey: #keyPath(AssetData.mediaData)) as? Data {
            if #available(iOS 13.0, *),
               let sealedBox = try? ChaChaPoly.SealedBox(combined: mediaData),
               let decryptedData = try? ChaChaPoly.open(sealedBox, using: AssetData.encryptionKey) {
                self.mediaData = decryptedData
            } else {
              //
            }
        } else {
            //
        }
    }
}

PINCache related operations

PINCache includes PINMemoryCache and PINDiskCache. PINCache will handle reading from file to Memory or writing from Memory to file for us. We only need to operate on PINCache.

To find the Cache file location in the simulator:

Use NSHomeDirectory() to get the simulator file path

Finder -> Go -> Paste the path

In Library -> Caches -> com.pinterest.PINDiskCache.ResourceLoader is the Resource Loader Cache directory we created.

PINCache(name: “ResourceLoader”) where the name is the directory name.

You can also specify the rootPath, and the directory can be moved under Documents (not afraid of being cleared by the system).

Set the maximum limit for PINCache:

 PINCacheAssetDataManager.Cache.diskCache.byteCount = 300 * 1024 * 1024 // max: 300mb
 PINCacheAssetDataManager.Cache.diskCache.byteLimit = 90 * 60 * 60 * 24 // 90 days

System default limit

Setting it to 0 will not proactively delete files.

Postscript

Initially underestimated the difficulty of this feature, thinking it could be handled quickly; ended up struggling and spent about two more weeks dealing with data storage issues. However, I thoroughly understood the entire Resource Loader operation mechanism, GCD, and Data.

References

Finally, here are the references for how to implement it:

1. iOS AVPlayer 视频缓存的设计与实现 Only explains the principle
2. 基于AVPlayer实现音视频播放和缓存，支持视频画面的同步输出 [ SZAVPlayer ] Includes code (very complete but complex)
3. CachingPlayerItem (Simple implementation, easier to understand but not complete)
4. 可能是目前最好的 AVPlayer 音视频缓存方案 AVAssetResourceLoaderDelegate
5. 仿抖音 Swift 版 [ Github ] (Interesting project, a replica of the Douyin APP; also uses Resource Loader)
6. iOS HLS Cache 實踐方法探究之旅

Extension

• DLCachePlayer (Objective-C version)

If you have any questions or comments, feel free to contact me.

===

本文中文版本

===

This article was first published in Traditional Chinese on Medium ➡️ View Here