AppStore APP’s Reviews Bot Insights

AppStore APP’s Reviews Slack Bot Insights

Using Ruby+Fastlane-SpaceShip to build an APP review tracking notification Slack bot

Photo by Austin Distel

Ignorance is bliss

AppReviewBot as an example

I recently discovered that the bot in Slack that forwards the latest APP reviews is a paid service. I always thought this feature was free. The cost ranges from $5 to $200 USD/month because each platform offers more than just the “App Review Bot” feature. They also provide data statistics, records, unified backend, competitor comparisons, etc. The cost is based on the services each platform can provide. The Review Bot is just one part of their offerings, but I only need this feature and nothing else. Paying for it seems wasteful.

Problem

I originally used the free open-source tool TradeMe/ReviewMe for Slack notifications, but this tool has been outdated for a long time. Occasionally, Slack would suddenly send out some old reviews, which was quite alarming (many bugs had already been fixed, making us think there were new issues!). The reason was unclear.

So, I considered finding other tools or methods to replace it.

TL;DR [2022/08/10] Update:

We have now redesigned the App Reviews Bot using the new App Store Connect API and relaunched it as “ ZReviewTender — a free open-source App Reviews monitoring bot “.

====

2022/07/20 Update

App Store Connect API now supports reading and managing Customer Reviews. The App Store Connect API natively supports accessing App reviews, no longer requiring Fastlane — Spaceship to fetch reviews from the backend.

Principle Exploration

With the motivation in place, let’s explore the principles to achieve the goal.

Official API ❌

Apple provides the App Store Connect API, but it does not offer a feature to fetch reviews.

[2022/07/20 Update]: App Store Connect API now supports reading and managing Customer Reviews

Public URL API (RSS) ⚠️

Apple provides a public APP review RSS subscription URL, and it offers both rss xml and json formats.

https://itunes.apple.com/country_code/rss/customerreviews/id=APP_ID/page=1/sortBy=mostRecent/json

• Country code: Refer to this document.
• APP_ID: Go to the App web version, and you will get the URL: https://apps.apple.com/tw/app/APP_NAME/id 12345678, the number after id is the App ID (pure numbers).
• Page: You can request pages 1~10, beyond that you cannot retrieve.
• SortBy: mostRecent/json requests the latest & json format, you can also change it to mostRecent/xml for xml format.

The returned review data is as follows:

rss.json:

{
  "author": {
    "uri": {
      "label": "https://itunes.apple.com/tw/reviews/id123456789"
    },
    "name": {
      "label": "test"
    },
    "label": ""
  },
  "im:version": {
    "label": "4.27.1"
  },
  "im:rating": {
    "label": "5"
  },
  "id": {
    "label": "123456789"
  },
  "title": {
    "label": "Great presence!"
  },
  "content": {
    "label": "Life is worth it~",
    "attributes": {
      "type": "text"
    }
  },
  "link": {
    "attributes": {
      "rel": "related",
      "href": "https://itunes.apple.com/tw/review?id=123456789&type=Purple%20Software"
    }
  },
  "im:voteSum": {
    "label": "0"
  },
  "im:contentType": {
    "attributes": {
      "term": "Application",
      "label": "Application"
    }
  },
  "im:voteCount": {
    "label": "0"
  }
}

Advantages:

1. Public, accessible without authentication steps
2. Simple and easy to use

Disadvantages:

1. This RSS API is very outdated and hasn’t been updated
2. The returned review information is too little (no comment time, edited review?, replied?)
3. Encounter data disorder issues (the last few pages occasionally suddenly spit out old data)
4. Can access up to 10 pages

The biggest problem we encountered is 3; but it is uncertain whether this is an issue with the Bot tool we are using or with the RSS URL data.

Private URL API ✅

This method is somewhat unconventional and was discovered by a sudden inspiration; but after referring to other Review Bot practices, I found that many websites also use it this way, so it should be fine, and I saw tools doing this 4-5 years ago, just didn’t delve into it at the time.

Advantages:

1. Same data as Apple’s backend
2. Complete and up-to-date data
3. Can do more detailed filtering
4. Deeply integrated APP tools also use this method (AppRadar/AppReviewBot…)

Disadvantages:

1. Unofficial method (unconventional)
2. Due to Apple’s implementation of comprehensive two-step login, the login session needs to be updated regularly.

Step 1 — Sniff the API that loads the review section of App Store Connect backend:

Get the Apple backend by hitting:

https://appstoreconnect.apple.com/WebObjects/iTunesConnect.woa/ra/apps/APP_ID/platforms/ios/reviews?index=0&sort=REVIEW_SORT_ORDER_MOST_RECENT

This endpoint retrieves the review list:

index = page offset, up to 100 entries per page.

The returned review data is as follows:

private.json:

{
  "value": {
    "id": 123456789,
    "rating": 5,
    "title": "Great presence!",
    "review": "Life is worth it~",
    "created": null,
    "nickname": "test",
    "storeFront": "TW",
    "appVersionString": "4.27.1",
    "lastModified": 1618836654000,
    "helpfulViews": 0,
    "totalViews": 0,
    "edited": false,
    "developerResponse": null
  },
  "isEditable": true,
  "isRequired": false,
  "errorKeys": null
}

After testing, it was found that you only need to include cookie: myacinfo=<Token> to forge a request and obtain data:

We have the API and the required headers, now we need to automate the retrieval of this cookie information from the backend.

Step Two — The Versatile Fastlane

Since Apple now enforces full Two-Step Verification, automating login verification has become more cumbersome. Fortunately, the clever Fastlane has implemented everything from the official App Store Connect API, iTMSTransporter, to web authentication (including two-step verification). We can directly use Fastlane’s command:

fastlane spaceauth -u <App Store Connect Account (Email)>

This command will complete the web login verification (including two-step verification) and then store the cookie in the FASTLANE_SESSION file.

You will get a string similar to the following:

!ruby/object:HTTP::Cookie
name: myacinfo  value: <token>  
domain: apple.com for_domain: true  path: "/"  
secure: true  httponly: true  expires: max_age: 
created_at: 2021-04-21 20:42:36.818821000 +08:00  
accessed_at: 2021-04-21 22:02:45.923016000 +08:00
!ruby/object:HTTP::Cookie
name: <hash>  value: <token>
domain: idmsa.apple.com for_domain: true  path: "/"
secure: true  httponly: true  expires: max_age: 2592000
created_at: 2021-04-19 23:21:05.851853000 +08:00
accessed_at: 2021-04-21 20:42:35.735921000 +08:00

By including myacinfo = value, you can obtain the review list.

Step Three — SpaceShip

Initially, I thought Fastlane could only help us up to this point, and we would have to manually integrate the flow of obtaining the cookie from Fastlane and then calling the API. However, after some exploration, I discovered that Fastlane’s authentication module SpaceShip has even more powerful features!

SpaceShip

SpaceShip already has a method for fetching the review list Class: Spaceship::TunesClient::get_reviews!

app = Spaceship::Tunes::login(appstore_account, appstore_password)
reviews = app.get_reviews(app_id, platform, storefront, versionId = '')

*storefront = region

Step Four — Assembly

Fastlane and Spaceship are both written in Ruby, so we also need to use Ruby to create this Bot tool.

We can create a reviewBot.rb file, and to compile and execute it, simply enter in the Terminal:

ruby reviewBot.rb

That’s it. ( *For more Ruby environment issues, refer to the tips at the end)

First, since the original get_reviews method’s parameters do not meet our needs; I want review data for all regions and all versions, without filtering, and with pagination support:

extension.rb:

# Extension Spaceship->TunesClient
module Spaceship
  class TunesClient < Spaceship::Client
    def get_recent_reviews(app_id, platform, index)
      r = request(:get, "ra/apps/#{app_id}/platforms/#{platform}/reviews?index=#{index}&sort=REVIEW_SORT_ORDER_MOST_RECENT")
      parse_response(r, 'data')['reviews']
     end
  end
end

So we extend a method in TunesClient, with parameters only including app_id, platform = ios ( all lowercase ), index = pagination offset.

Next, assemble login authentication and fetch the review list:

get_recent_reviews.rb:

index = 0
breakWhile = true
while breakWhile
  app = Spaceship::Tunes::login(APPStoreConnect account (Email), APPStoreConnect password)
  reviews = app.get_recent_reviews($app_id, $platform, index)
  if reviews.length() <= 0
    breakWhile = false
    break
  end
  reviews.each { |review|
    index += 1
    puts review["value"]
  }
end

Use while to traverse all pages, and terminate when there is no content.

Next, add a record of the last latest time, and only notify the latest messages that have not been notified:

lastModified.rb:

lastModified = 0
if File.exists?(".lastModified")
  lastModifiedFile = File.open(".lastModified")
  lastModified = lastModifiedFile.read.to_i
end
newLastModified = lastModified
isFirst = true
messages = []

index = 0
breakWhile = true
while breakWhile
  app = Spaceship::Tunes::login(APPStoreConnect account (Email), APPStoreConnect password)
  reviews = app.get_recent_reviews($app_id, $platform, index)
  if reviews.length() <= 0
    breakWhile = false
    break
  end
  reviews.each { |review|
    index += 1
    if isFirst
      isFirst = false
      newLastModified = review["value"]["lastModified"]
    end

    if review["value"]["lastModified"] > lastModified && lastModified != 0  
      # Do not send notifications the first time
      messages.append(review["value"])
    else
      breakWhile = false
      break
    end
  }
end

messages.sort! { |a, b|  a["lastModified"] <=> b["lastModified"] }
messages.each { |message|
    notify_slack(message)
}

File.write(".lastModified", newLastModified, mode: "w+")

Simply use a .lastModified to record the time obtained during the last execution.

*Do not send notifications the first time, otherwise, it will spam

The final step, assemble the push message & send it to Slack:

slack.rb:

# Slack Bot
def notify_slack(review)
  rating = review["rating"].to_i
  color = rating >= 4 ? "good" : (rating >= 2 ? "warning" : "danger")
  like = review["helpfulViews"].to_i > 0 ? " - #{review["helpfulViews"]} :thumbsup:" : ""
  date = review["edited"] == false ? "Created at: #{Time.at(review["lastModified"].to_i / 1000).to_datetime}" : "Updated at: #{Time.at(review["lastModified"].to_i / 1000).to_datetime}"
  
    
  isResponse = ""
  if review["developerResponse"] != nil && review["developerResponse"]['lastModified'] < review["lastModified"]
    isResponse = " (Response outdated)"
  end
  
  edited = review["edited"] == false ? "" : ":memo: User updated review#{isResponse}:"

  stars = "★" * rating + "☆" * (5 - rating)
  attachments = {
    :pretext => edited,
    :color => color,
    :fallback => "#{review["title"]} - #{stars}#{like}",
    :title => "#{review["title"]} - #{stars}#{like}",
    :text => review["review"],
    :author_name => review["nickname"],
    :footer => "iOS - v#{review["appVersionString"]} - #{review["storeFront"]} - #{date} - <https://appstoreconnect.apple.com/apps/APP_ID/appstore/activity/ios/ratingsResponses|Go To App Store>"
  }
  payload = {
   :attachments => [attachments],
   :icon_emoji => ":storm_trooper:",
   :username => "ZhgChgLi iOS Review Bot"
  }.to_json
  cmd = "curl -X POST --data-urlencode 'payload=#{payload}' SLACK_WEB_HOOK_URL"
  system(cmd, :err => File::NULL)
  puts "#{review["id"]} send Notify Success!"
 end

SLACK_WEB_HOOK_URL = Incoming WebHook URL

Final Result

appreviewbot.rb:

require "Spaceship"
require 'json'
require 'date'

# Config
$slack_web_hook = "Target notification web hook url"
$slack_debug_web_hook = "Notification web hook url when the bot has an error"
$appstore_account = "APPStoreConnect account (Email)"
$appstore_password = "APPStoreConnect password"
$app_id = "APP_ID"
$platform = "ios"

# Extension Spaceship->TunesClient
module Spaceship
  class TunesClient < Spaceship::Client
    def get_recent_reviews(app_id, platform, index)
      r = request(:get, "ra/apps/#{app_id}/platforms/#{platform}/reviews?index=#{index}&sort=REVIEW_SORT_ORDER_MOST_RECENT")
      parse_response(r, 'data')['reviews']
     end
  end
end

# Slack Bot
def notify_slack(review)
  rating = review["rating"].to_i
  color = rating >= 4 ? "good" : (rating >= 2 ? "warning" : "danger")
  like = review["helpfulViews"].to_i > 0 ? " - #{review["helpfulViews"]} :thumbsup:" : ""
  date = review["edited"] == false ? "Created at: #{Time.at(review["lastModified"].to_i / 1000).to_datetime}" : "Updated at: #{Time.at(review["lastModified"].to_i / 1000).to_datetime}"
  
    
  isResponse = ""
  if review["developerResponse"] != nil && review["developerResponse"]['lastModified'] < review["lastModified"]
    isResponse = " (Customer service response is outdated)"
  end
  
  edited = review["edited"] == false ? "" : ":memo: User updated review#{isResponse}:"

  stars = "★" * rating + "☆" * (5 - rating)
  attachments = {
    :pretext => edited,
    :color => color,
    :fallback => "#{review["title"]} - #{stars}#{like}",
    :title => "#{review["title"]} - #{stars}#{like}",
    :text => review["review"],
    :author_name => review["nickname"],
    :footer => "iOS - v#{review["appVersionString"]} - #{review["storeFront"]} - #{date} - <https://appstoreconnect.apple.com/apps/APP_ID/appstore/activity/ios/ratingsResponses|Go To App Store>"
  }
  payload = {
   :attachments => [attachments],
   :icon_emoji => ":storm_trooper:",
   :username => "ZhgChgLi iOS Review Bot"
  }.to_json
  cmd = "curl -X POST --data-urlencode 'payload=#{payload}' #{$slack_web_hook}"
  system(cmd, :err => File::NULL)
  puts "#{review["id"]} send Notify Success!"
 end

begin
    lastModified = 0
    if File.exists?(".lastModified")
      lastModifiedFile = File.open(".lastModified")
      lastModified = lastModifiedFile.read.to_i
    end
    newLastModified = lastModified
    isFirst = true
    messages = []

    index = 0
    breakWhile = true
    while breakWhile
      app = Spaceship::Tunes::login($appstore_account, $appstore_password)
      reviews = app.get_recent_reviews($app_id, $platform, index)
      if reviews.length() <= 0
        breakWhile = false
        break
      end
      reviews.each { |review|
        index += 1
        if isFirst
          isFirst = false
          newLastModified = review["value"]["lastModified"]
        end

        if review["value"]["lastModified"] > lastModified && lastModified != 0  
          # Do not send notification on first use
          messages.append(review["value"])
        else
          breakWhile = false
          break
        end
      }
    end
    
    messages.sort! { |a, b|  a["lastModified"] <=> b["lastModified"] }
    messages.each { |message|
        notify_slack(message)
    }
    
    File.write(".lastModified", newLastModified, mode: "w+")
rescue => error
    attachments = {
        :color => "danger",
        :title => "AppStoreReviewBot Error occurs!",
        :text => error,
        :footer => "*Due to Apple's technical limitations, the precise rating crawling function needs to be re-logged in and set approximately every month. We apologize for the inconvenience."
    }
    payload = {
        :attachments => [attachments],
        :icon_emoji => ":storm_trooper:",
        :username => "ZhgChgLi iOS Review Bot"
    }.to_json
    cmd = "curl -X POST --data-urlencode 'payload=#{payload}' #{$slack_debug_web_hook}"
    system(cmd, :err => File::NULL)
    puts error
end

Additionally, a begin…rescue (try…catch) protection is added. If an error occurs, a Slack notification will be sent for us to check (mostly due to session expiration).

Finally, just add this script to crontab / schedule and other scheduling tools to execute it regularly!

Effect picture:

Free Alternatives

1. AppFollow: Uses Public URL API (RSS), it’s usable at best.
2. feedis.io: Uses Private URL API, requires giving them your account and password.
3. TradeMe/ReviewMe: Self-hosted service (node.js), we originally used this but encountered the aforementioned issues.
4. JonSnow: Self-hosted service (GO), supports one-click deployment to heroku, author: @saiday

Warm Tips

1. ⚠️Private URL API method, if using an account with two-factor authentication, it needs to be re-verified every 30 days at most and currently has no solution; if you can create an account without two-factor authentication, you can use it smoothly.

#important-note-about-session-duration

1. ⚠️Whether free, paid, or self-hosted as mentioned in this article; do not use a developer account, be sure to create a separate App Store Connect account with only “Customer Support” permissions to prevent security issues.

2. It is recommended to use rbenv to manage Ruby, as the system’s built-in version 2.6 can easily cause conflicts.

3. If you encounter GEM or Ruby environment errors on macOS Catalina, you can refer to this reply to solve them.

Problem Solved!

After the above journey, I have a better understanding of how the Slack Bot works and how the iOS App Store crawls review content. I also got to play around with ruby! It feels great to write!

If you have any questions or feedback, feel free to contact me.

===

本文中文版本

===

This article was first published in Traditional Chinese on Medium ➡️ View Here