Overview

This guide explains how to minimize player downtime during full synchronized updates of game clients and servers, while accounting for app store propagation delays when publishing new updates. We mainly address the iOS App Store and Google Play Store for Android, but the concepts should translate to other stores as well.

Apple's App Store unfortunately has a potentially multi-hour delay when publishing already-approved game updates to the public. This can lead to situations where the server has already been updated to a new version while iOS clients aren't yet downloadable, preventing iOS players from accessing the game.

The Google Play store has a much shorter propagation delay and doesn't have this issue.

When to Update?

It is highly recommended to schedule updates for weekday mornings (excluding Fridays). Deploying new game versions always carries some risk, and you'll want to have maximum working hours available in case issues arise.

Pushing an update on Friday afternoon can be highly detrimental to team morale, especially if a significant number of players can't access the game while the development team is leaving for the weekend.

How Much Downtime?

Using the approach outlined in this guide, updates can be published with as little as 5-10 minutes of player downtime. However, it's prudent to allocate at least 30 minutes to accommodate manual testing of new client versions and other verification steps.

Update Steps

1. Prepare the New Client and Server Versions

First, you need to build and deploy the new server version in the staging environment. After that, build the game clients for the new version. You can upload the clients to the app stores for review immediately or first perform internal QA before doing that.

Configure the production server to forward all clients of the new version to be redirected to the staging environment. This can be done from the production server's LiveOps Dashboard settings page: Edit the Client Compatibility Settings, enable the ‘New Client Redirect Enabled' toggle, then type in the staging server's hostname and CDN URL. This is required for the app store reviewers to be able to launch the new game version.

2. Perform Internal QA on the Production Bulds

Following this update process means game clients will be published to players before the server is updated. Since published clients cannot be easily rolled back, you become committed to completing the update. Therefore, thorough testing of the new game version is crucial to prevent additional downtime caused by client-server incompatibility.

After the reviews are accepted, disable the new version redirection from the production server.

3. Publish to iOS

After publishing the iOS build in the App Store, it can take anywhere from a few minutes to multiple hours for the new build to become visible to the players. Fortunately, when the build is visible to the first players, it should be visible to all the players within some 5-10 minutes.

Now, wait for the iOS build to become available in the store. Currently, this step requires manual monitoring, though we plan to add SDK functionality to automate this process. You can either manually refresh the store page or use a script to poll for the new version's availability. Alternatively, monitor Grafana metrics for connection attempts from the new version.

Put the server into maintenance mode with an appropriate heads-up time for the players with the old client. The new clients should see a maintenance mode message already.

4. Publish to Android

Next, publish the application in the Google Play store (and any other stores that propagate the builds quickly).

5. Wait for Maintenance Mode

You should now wait for maintenance mode to activate and players to be disconnected from the servers. Monitor this on the LiveOps Dashboard front page. When the 'Concurrents' metrics approach zero, you can proceed safely.

Note

Inform your team to avoid performing customer support or other tasks via the LiveOps Dashboard during maintenance mode, as changes may be lost during server reboots or database rollbacks.

6. Make Database Backups

It's important to create database backups as a precaution. For production environments hosted by Metaplay, the database is backed up automatically and no further steps are needed. For self-hosted users, there is more information about how to create backups in the Backups and Restoring page.

7. Deploy to Production

Deploy the new server version into production. You should use your CI pipeline to perform the deployment. The server will boot up with maintenance mode enabled. Optionally, publish new versions of the game configs directly to the server. Depending on the game config workflow used, the previous step has already updated the game configs.

It's important to note that the game server image will contain the game configs that are in the repository at the time of building, and any changes after that will not be included automatically.

You can check out the Setup CI Pipeline page for information on how to deploy your game server.

8. Test Updated Clients

Conduct manual testing with updated clients to verify proper functionality. You can grant Developer Status to test devices via the dashboard to enable login during maintenance.

9. Wrapping Up

Now, you can disable maintenance mode. The players should now start returning to the game.

It's important to continue monitoring the servers for several hours to ensure proper operation. The "Metaplay Server" Grafana dashboard provides the most comprehensive system status:

• "Active Connections" should return to roughly the same as it was before the update. Same for "Login Rate" and "Player Actors".
• "CPU Usage" and "Memory Usage" should also stabilize to roughly what they were before the update.
• "EntityAsk Errors" and "Player Incident Reports by Kind" should remain low. Same for "Error Log Rate" and "Warning Log Rate".