Real-world Decode Issues with Codable

Real-world Decode Issues with Codable (Part 1)

From basic to advanced, deeply using Decodable to meet all possible problem scenarios

$Photo by [Gustas Brazaitis](https://unsplash.com/@gustasbrazaitis){:target="_blank"}$

Preface

Due to the backend API upgrade, we need to adjust the API processing architecture. Recently, we took this opportunity to update the original network processing architecture written in Objective-C to Swift. Due to the different languages, it is no longer suitable to use the original Restkit to handle our network layer applications. However, it must be said that Restkit’s functionality is very powerful, and it was used very effectively in the project with almost no major issues. But it is relatively cumbersome, almost no longer maintained, and purely Objective-C. It will inevitably need to be replaced in the future.

Restkit almost handled all the network request-related functions we needed, from basic network processing, API calls, network processing, to response processing JSON String to Object, and even storing objects into Core Data. It was a framework that could handle ten tasks at once.

With the evolution of the times, the current frameworks no longer focus on an all-in-one package but more on flexibility, lightness, and combination, increasing more flexibility and creating more variations. Therefore, when replacing it with Swift, we chose to use Moya as the network processing part of the package, and other functions we needed were combined in other ways.

Main Content

For the JSON String to Object Mapping part, we use Swift’s built-in Codable (Decodable) protocol & JSONDecoder for processing. We split the Entity/Model to enhance responsibility separation, operation, and readability. Additionally, we also need to consider the code base mixing Objective-C and Swift.

＊ The Encodable part is omitted, and the examples only show the implementation of Decodable. They are similar; if you can decode, you can also encode.

Getting Started

Assume our initial API Response JSON String is as follows:

  
{
  "id": 123456,
  "comment": "It's Accusefive, not Five Accuse!",
  "target_object": {
    "type": "song",
    "id": 99,
    "name": "Thinking of You Under the Stars"
  },
  "commenter": {
    "type": "user",
    "id": 1,
    "name": "zhgchgli",
    "email": "zhgchgli@gmail.com"
  }
}

From the above example, we can split it into three entities & models: User, Song, and Comment. For convenience, let’s write the Entity/Model in the same file.

User:

  
// Entity:
struct UserEntity: Decodable {
    var id: Int
    var name: String
    var email: String
}

//Model:
class UserModel: NSObject {
    init(_ entity: UserEntity) {
      self.id = entity.id
      self.name = entity.name
      self.email = entity.email
    }
    var id: Int
    var name: String
    var email: String
}

Song:

  
// Entity:
struct SongEntity: Decodable {
    var id: Int
    var name: String
}

//Model:
class SongModel: NSObject {
    init(_ entity: SongEntity) {
      self.id = entity.id
      self.name = entity.name
    }
    var id: Int
    var name: String
}

Comment:

  
// Entity:
struct CommentEntity: Decodable {
    enum CodingKeys: String, CodingKey {
      case id
      case comment
      case targetObject = "target_object"
      case commenter
    }
    
    var id: Int
    var comment: String
    var targetObject: SongEntity
    var commenter: UserEntity
}

//Model:
class CommentModel: NSObject {
    init(_ entity: CommentEntity) {
      self.id = entity.id
      self.comment = entity.comment
      self.targetObject = SongModel(entity.targetObject)
      self.commenter = UserModel(entity.commenter)
    }
    var id: Int
    var comment: String
    var targetObject: SongModel
    var commenter: UserModel
}

JSONDecoder:

  
let jsonString = "{ \"id\": 123456, \"comment\": \"It's Accusefive, not Five Accuse!\", \"target_object\": { \"type\": \"song\", \"id\": 99, \"name\": \"Thinking of You Under the Stars\" }, \"commenter\": { \"type\": \"user\", \"id\": 1, \"name\": \"zhgchgli\", \"email\": \"zhgchgli@gmail.com\" } }"
let jsonDecoder = JSONDecoder()
do {
    let result = try jsonDecoder.decode(CommentEntity.self, from: jsonString.data(using: .utf8)!)
} catch {
    print(error)
}

CodingKeys Enum?

When our JSON String Key Name does not match the Entity Object Property Name, we can add a CodingKeys enum internally to map them, as we cannot control the naming convention of the backend data source.

  
case PropertyKeyName = "backend_field_name"
case PropertyKeyName // If not specified, the default is to use PropertyKeyName as the backend field name

Once the CodingKeys enum is added, all non-Optional fields must be enumerated, and you cannot just list the keys you want to customize.

Another way is to set the keyDecodingStrategy of JSONDecoder. If the response data fields and property names differ only by snake_case <-> camelCase, you can directly set .keyDecodingStrategy = .convertFromSnakeCase to automatically match the mapping.

  
let jsonDecoder = JSONDecoder()
jsonDecoder.keyDecodingStrategy = .convertFromSnakeCase
try jsonDecoder.decode(CommentEntity.self, from: jsonString.data(using: .utf8)!)

When the returned data is an array:

  
struct SongListEntity: Decodable {
    var songs:[SongEntity]
}

Adding constraints to String:

  
struct SongEntity: Decodable {
  var id: Int
  var name: String
  var type: SongType
  
  enum SongType {
    case rock
    case pop
    case country
  }
}

Applicable to string types with a limited range, writing them as Enums makes it convenient for us to pass and use; if a value appears that is not enumerated, decoding will fail!

Make good use of generics to wrap fixed structures:

Assuming the JSON String returned in multiple instances has a fixed format:

  
{
  "count": 10,
  "offset": 0,
  "limit": 0,
  "results": [
    {
      "type": "song",
      "id": 1,
      "name": "1"
    }
  ]
}

You can wrap it using generics:

  
struct PageEntity<E: Decodable>: Decodable {
    var count: Int
    var offset: Int
    var limit: Int
    var results: [E]
}

Usage: PageEntity<Song>.self

Date/Timestamp automatic decoding:

Set the dateDecodingStrategy of JSONDecoder

.secondsSince1970/.millisecondsSince1970: Unix timestamp
.deferredToDate: Apple’s timestamp, rarely used, different from Unix timestamp, it starts from 2001/01/01
.iso8601: ISO 8601 date format
.formatted(DateFormatter): Decode Date according to the passed-in DateFormatter
.custom: Custom Date Decode logic

.custom example: Assuming the API returns both YYYY/MM/DD and ISO 8601 formats, both need to be decoded:

  
var dateFormatter = DateFormatter()
var iso8601DateFormatter = ISO8601DateFormatter()

let decoder: JSONDecoder = JSONDecoder()
decoder.dateDecodingStrategy = .custom({ (decoder) -> Date in
    let container = try decoder.singleValueContainer()
    let dateString = try container.decode(String.self)
    
    //ISO8601:
    if let date = iso8601DateFormatter.date(from: dateString) {
        return date
    }
    
    //YYYY-MM-DD:
    dateFormatter.dateFormat = "yyyy-MM-dd"
    if let date = dateFormatter.date(from: dateString) {
        return date
    }
    
    throw DecodingError.dataCorruptedError(in: container, debugDescription: "Cannot decode date string \(dateString)")
})

let result = try jsonDecoder.decode(CommentEntity.self, from: jsonString.data(using: .utf8)!)

＊DateFormatter is very performance-consuming when initialized, try to reuse it as much as possible.

Basic Decode knowledge:

The field types (struct/class/enum) within the Decodable Protocol must implement the Decodable Protocol; or assign values when initializing the decoder
Decoding will fail if the field types do not match
If a field in the Decodable Object is set to Optional, it is optional; if provided, it will be decoded
Optional fields can accept: JSON String without the field, provided but given as nil
Blank, 0 is not equal to nil, nil is nil; pay attention to weakly typed backend APIs!
By default, if a non-Optional field in the Decodable Object is an enum and the JSON String does not provide it, decoding will fail (will explain how to handle this later)
By default, decoding failure will directly interrupt and exit, it cannot simply skip erroneous data (will explain how to handle this later)

$[Left: "" Right: nil](https://josjong.com/2017/10/16/null-vs-empty-strings-why-oracle-was-right-and-apple-is-not/){:target="_blank"}$

Left: “” / Right: nil

Advanced Usage

So far, the basic usage has been completed, but the real world is not that simple. Below are some advanced scenarios you might encounter and solutions using Codable. From here on, we can no longer rely on the original Decode to help us with Mapping; we need to implement init(from decoder: Decoder) for custom Decode operations.

＊For now, we will only show the Entity part; the Model is not needed yet.

init(from decoder: Decoder)

init decoder, must assign initial values to all non-Optional fields (that’s init!).

When customizing Decode operations, we need to get the container from the decoder to operate on the values. There are three types of containers to retrieve content from.

First type container(keyedBy: CodingKeys.self) Operate according to CodingKeys:

  
struct SongEntity: Decodable {
    var id: Int
    var name: String
    
    enum CodingKeys: String, CodingKey {
      case id
      case name
    }
    
    init(from decoder: Decoder) throws {
        let container = try decoder.container(keyedBy: CodingKeys.self)
        self.id = try container.decode(Int.self, forKey: .id)
        // Parameter 1 accepts support: class implementing Decodable
        // Parameter 2 CodingKeys
        
        self.name = try container.decode(String.self, forKey: .name)
    }
}

Second type singleValueContainer Retrieve the whole package for operation (single value):

  
enum HandsomeLevel: Decodable {
    case handsome(String)
    case normal(String)
    init(from decoder: Decoder) throws {
        let container = try decoder.singleValueContainer()
        let name = try container.decode(String.self)
        if name == "zhgchgli" {
            self = .handsome(name)
        } else {
            self = .normal(name)
        }
    }
}

struct UserEntity: Decodable {
    var id: Int
    var name: HandsomeLevel
    var email: String
    
    enum CodingKeys: String, CodingKey {
        case id
        case name
        case email
    }
}

Suitable for Associated Value Enum field types, for example, name also carries a level of handsomeness!

Third type unkeyedContainer Treat the whole package as an array:

  
struct ListEntity: Decodable {
    var items:[Decodable]
    init(from decoder: Decoder) throws {
        var unkeyedContainer = try decoder.unkeyedContainer()
        self.items = []
        while !unkeyedContainer.isAtEnd {
            // The internal pointer of unkeyedContainer will automatically point to the next object after the decode operation
            // Until it points to the end, indicating the traversal is complete
            if let id = try? unkeyedContainer.decode(Int.self) {
                items.append(id)
            } else if let name = try? unkeyedContainer.decode(String.self) {
                items.append(name)
            }
        }
    }
}

let jsonString = "[\"test\",1234,5566]"
let jsonDecoder = JSONDecoder()
let result = try jsonDecoder.decode(ListEntity.self, from: jsonString.data(using: .utf8)!)
print(result)

Applicable to array fields of variable types.

Under Container, we can also use nestedContainer / nestedUnkeyedContainer to operate on specific fields:

＊Flatten data fields (similar to flatMap)

  
struct ListEntity: Decodable {
    
    enum CodingKeys: String, CodingKey {
        case items
        case date
        case name
        case target
    }
    
    enum PredictKey: String, CodingKey {
        case type
    }
    
    var date: Date
    var name: String
    var items: [Decodable]
    var target: Decodable
    
    init(from decoder: Decoder) throws {
        let container = try decoder.container(keyedBy: CodingKeys.self)
        
        self.date = try container.decode(Date.self, forKey: .date)
        self.name = try container.decode(String.self, forKey: .name)
        
        let nestedContainer = try container.nestedContainer(keyedBy: PredictKey.self, forKey: .target)
        
        let type = try nestedContainer.decode(String.self, forKey: .type)
        if type == "song" {
            self.target = try container.decode(SongEntity.self, forKey: .target)
        } else {
            self.target = try container.decode(UserEntity.self, forKey: .target)
        }
        
        var unkeyedContainer = try container.nestedUnkeyedContainer(forKey: .items)
        self.items = []
        while !unkeyedContainer.isAtEnd {
            if let song = try? unkeyedContainer.decode(SongEntity.self) {
                items.append(song)
            } else if let user = try? unkeyedContainer.decode(UserEntity.self) {
                items.append(user)
            }
        }
    }
}

Access and decode objects of different levels. The example demonstrates using nestedContainer to flatten out the type for target/items and then decode accordingly based on the type.

Decode & DecodeIfPresent

DecodeIfPresent: Decode only when the response provides the data field (when Codable Property is set to Optional)
Decode: Perform the decode operation. If the response does not provide the data field, it will throw an error.

＊The above is just a brief introduction to the methods and functions of init decoder and container. It’s okay if you don’t understand; we’ll dive directly into real-world scenarios and experience the combined operations in the examples.

Real-world Scenario

Returning to the original example JSON String.

Scenario 1. Suppose today the comment could be on a song or a person. The `targetObject` field could be `User` or `Song`. How should we handle it?

  
{
  "results": [
    {
      "id": 123456,
      "comment": "It's Accusefive, not Five Accuse!",
      "target_object": {
        "type": "song",
        "id": 99,
        "name": "Thinking of You Under the Stars"
      },
      "commenter": {
        "type": "user",
        "id": 1,
        "name": "zhgchgli",
        "email": "zhgchgli@gmail.com"
      }
    },
    {
      "id": 55,
      "comment": "66666!",
      "target_object": {
        "type": "user",
        "id": 1,
        "name": "zhgchgli"
      },
      "commenter": {
        "type": "user",
        "id": 2,
        "name": "aaaa",
        "email": "aaaa@gmail.com"
      }
    }
  ]
}

Method a.

Using Enum as a container for Decode.

  
struct CommentEntity: Decodable {
    
    enum CodingKeys: String, CodingKey {
      case id
      case comment
      case targetObject = "target_object"
      case commenter
    }
    
    var id: Int
    var comment: String
    var targetObject: TargetObject
    var commenter: UserEntity
    
    enum TargetObject: Decodable {
        case song(SongEntity)
        case user(UserEntity)
        
        enum PredictKey: String, CodingKey {
            case type
        }
        
        enum TargetObjectType: String, Decodable {
            case song
            case user
        }
        
        init(from decoder: Decoder) throws {
            let container = try decoder.container(keyedBy: PredictKey.self)
            let singleValueContainer = try decoder.singleValueContainer()
            let targetObjectType = try container.decode(TargetObjectType.self, forKey: .type)
            
            switch targetObjectType {
            case .song:
                let song = try singleValueContainer.decode(SongEntity.self)
                self = .song(song)
            case .user:
                let user = try singleValueContainer.decode(UserEntity.self)
                self = .user(user)
            }
        }
    }
}

We change the targetObject property to an Associated Value Enum, deciding what content to put inside the Enum during Decode.

The core practice is to create a Decodable Enum as a container, decode it by first extracting the key field (the type field in the example JSON String), and if it is Song, use singleValueContainer to decode the whole package into SongEntity, and similarly for User.

Extract from Enum when using:

  
//if case let
if case let CommentEntity.TargetObject.user(user) = result.targetObject {
    print(user)
} else if case let CommentEntity.TargetObject.song(song) = result.targetObject {
    print(song)
}

//switch case let
switch result.targetObject {
case .song(let song):
    print(song)
case .user(let user):
    print(user)
}

Method b.

Declare the field property as Base Class.

  
struct CommentEntity: Decodable {
    enum CodingKeys: String, CodingKey {
      case id
      case comment
      case targetObject = "target_object"
      case commenter
    }
    enum PredictKey: String, CodingKey {
        case type
    }
    
    var id: Int
    var comment: String
    var targetObject: Decodable
    var commenter: UserEntity
    
    init(from decoder: Decoder) throws {
        let container = try decoder.container(keyedBy: CodingKeys.self)
        self.id = try container.decode(Int.self, forKey: .id)
        self.comment = try container.decode(String.self, forKey: .comment)
        self.commenter = try container.decode(UserEntity.self, forKey: .commenter)
        
        //
        let targetObjectContainer = try container.nestedContainer(keyedBy: PredictKey.self, forKey: .targetObject)
        let targetObjectType = try targetObjectContainer.decode(String.self, forKey: .type)
        if targetObjectType == "user" {
            self.targetObject = try container.decode(UserEntity.self, forKey: .targetObject)
        } else {
            self.targetObject = try container.decode(SongEntity.self, forKey: .targetObject)
        }
    }
}

The principle is similar, but here we first use nestedContainer to dive into targetObject to get the type and then decide what type targetObject should be parsed into.

Cast when using:

  
if let song = result.targetObject as? Song {
  print(song)
} else if let user = result.targetObject as? User {
  print(user)
}

Scenario 2. How to decode if the data array contains multiple types of data?

{
  "results": [
    {
      "type": "song",
      "id": 99,
      "name": "Thinking of You Under the Stars"
    },
    {
      "type": "user",
      "id": 1,
      "name": "zhgchgli",
      "email": "zhgchgli@gmail.com"
    }
  ]
}

Combine the nestedUnkeyedContainer mentioned above with the solution from Scenario 1; you can also use Scenario 1’s a. solution, using Associated Value Enum to store values.

Scenario 3. Decode JSON String field only when it has a value

[
  {
    "type": "song",
    "id": 99,
    "name": "Thinking of You Under the Stars"
  },
  {
    "type": "song",
    "id": 11
  }
]

Use decodeIfPresent to decode.

Scenario 4. Skip data that fails to decode in an array

  
{
  "results": [
    {
      "type": "song",
      "id": 99,
      "name": "Thinking of You Under the Stars"
    },
    {
      "error": "error"
    },
    {
      "type": "song",
      "id": 19,
      "name": "Take Me to Find Nightlife"
    }
  ]
}

As mentioned earlier, Decodable by default requires all data to be correctly parsed to map the output; sometimes you may encounter unstable data from the backend, providing a long array but with some entries missing fields or having mismatched field types causing decode failures; resulting in the entire package failing and returning nil.

  
struct ResultsEntity: Decodable {
    enum CodingKeys: String, CodingKey {
        case results
    }
    var results: [SongEntity]
    
    init(from decoder: Decoder) throws {
        let container = try decoder.container(keyedBy: CodingKeys.self)
        var nestedUnkeyedContainer = try container.nestedUnkeyedContainer(forKey: .results)
        
        self.results = []
        while !nestedUnkeyedContainer.isAtEnd {
            if let song = try? nestedUnkeyedContainer.decode(SongEntity.self) {
                self.results.append(song)
            } else {
                let _ = try nestedUnkeyedContainer.decode(EmptyEntity.self)
            }
        }
    }
}

struct EmptyEntity: Decodable { }

struct SongEntity: Decodable {
    var type: String
    var id: Int
    var name: String
}

let jsonString = "{ \"results\": [ { \"type\": \"song\", \"id\": 99, \"name\": \"Thinking of You Under the Stars\" }, { \"error\": \"error\" }, { \"type\": \"song\", \"id\": 19, \"name\": \"Take Me to Find Nightlife\" } ] }"
let jsonDecoder = JSONDecoder()
let result = try jsonDecoder.decode(ResultsEntity.self, from: jsonString.data(using: .utf8)!)
print(result)

The solution is similar to Scenario 2’s solution; nestedUnkeyedContainer iterates through each content and performs try? Decode. If Decode fails, it uses Empty Decode to allow the nestedUnkeyedContainer’s internal pointer to continue executing.

＊This method is somewhat of a workaround because we cannot command nestedUnkeyedContainer to skip, and nestedUnkeyedContainer must successfully decode to continue executing. Therefore, we do it this way. Some in the Swift community have suggested adding moveNext(), but it has not been implemented in the current version.

Scenario 5. Some fields are for internal use in my program, not for Decode

Method a. Entity/Model

Here we need to mention what was said at the beginning about the utility of splitting Entity/Model; Entity is solely responsible for JSON String to Entity (Decodable) Mapping; Model initWith Entity, the actual program transmission, operation, and business logic all use Model.

  
struct SongEntity: Decodable {
    var type: String
    var id: Int
    var name: String
}

class SongModel: NSObject {
    init(_ entity: SongEntity) {
        self.type = entity.type
        self.id = entity.id
        self.name = entity.name
    }
    
    var type: String
    var id: Int
    var name: String
    
    var isSave:Bool = false //business logic
}

Benefits of splitting Entity/Model:

Clear responsibilities, Entity: JSON String to Decodable, Model: business logic
Clear mapping of fields, just look at Entity
Avoid cluttering when there are many fields
Can be used in Objective-C (since Model is just NSObject, struct/Decodable is not visible in Objective-C)
Internal business logic and fields can be placed in Model

Method b. init handling

List CodingKeys and exclude fields for internal use, give default values during init or set fields with default values or make them Optional, but these are not good methods, just runnable ones.

[2020/06/26 Update] — Next Scenario 6. API Response uses 0/1 to represent Bool, how to Decode?

Summary of Decode issues encountered in real use of Codable (Part 2)

[2020/06/26 Update] — Next Scenario 7. Don’t want to rewrite init decoder every time

Summary of Decode issues encountered in real use of Codable (Part 2)

[2020/06/26 Update] — Next Scenario 8. Reasonable handling of Response Null field data

Summary of Decode issues encountered in real use of Codable (Part 2)

Comprehensive Scenario Example

A complete example combining the basic and advanced usage mentioned above:

{
  "count": 5,
  "offset": 0,
  "limit": 10,
  "results": [
    {
      "id": 123456,
      "comment": "It's Accusefive, not Fiveaccuse!",
      "target_object": {
        "type": "song",
        "id": 99,
        "name": "Thinking of You Under the Stars",
        "create_date": "2020-06-13T15:21:42+0800"
      },
      "commenter": {
        "type": "user",
        "id": 1,
        "name": "zhgchgli",
        "email": "zhgchgli@gmail.com",
        "birthday": "1994/07/18"
      }
    },
    {
      "error": "not found"
    },
    {
      "error": "not found"
    },
    {
      "id": 2,
      "comment": "Haha, me too!",
      "target_object": {
        "type": "user",
        "id": 1,
        "name": "zhgchgli",
        "email": "zhgchgli@gmail.com",
        "birthday": "1994/07/18"
      },
      "commenter": {
        "type": "user",
        "id": 1,
        "name": "Passerby A",
        "email": "man@gmail.com",
        "birthday": "2000/01/12"
      }
    }
  ]
}

Output:

zhgchgli: It's Accusefive, not Five Accuse!

Complete example demonstration as above!

(Next) Part & Other Scenarios Updated:

Summary of Decode Issues Encountered in Real Use of Codable (Part 2)

Summary

The benefits of choosing to use Codable, first of all, are because it is native, you don’t have to worry about no one maintaining it in the future, and it looks nice when written; but relatively, the restrictions are stricter, it is less flexible in parsing JSON Strings, otherwise, you have to do more things as described in this article to complete it, and the performance is actually not superior to using other Mapping packages (Decodable still uses NSJSONSerialization from the Objective-C era for parsing). However, I think Apple might optimize this in future updates, so we won’t need to change the program then.

The scenarios and examples in the article may be extreme, but sometimes you can’t help it when you encounter them; of course, we hope that in general situations, simple Codable can meet our needs; but with the above techniques, there should be no unsolvable problems!

Thanks to ＠saiday for technical support.

Real-world Decode Issues with Codable