|
 |
|
 |
|
Player Metrics Analysis
Replica Island reports simple play statistics back to a server. These statistics communicate three states: when the player has died, when the player has completed a level, and when the player has completed the game. The messages are completely anonymous and users can choose to opt-out of statistics reporting. Using these data we can build a profile of each level that is very useful in finding frustration points and difficulty spikes. This page describes aggregate player statistics as of March 18, 2010, about a week after the game was released.
Unique Player Count as of 3/18/2010: 12954
First, let's take a look at all of the levels as a whole. The two graphs below describe the average number of player deaths per level and the average amount of time required to complete each level (including deaths and restarts).
You can see that the number of deaths starts to spike in the later levels. These levels were intended to be harder, but the spikes might indicate a frustration point. The spike in level 27 is an interesting example. This level
is Memory #34, and if you look at the heat map for that level you can see that all the deaths are concentrated in a single spot. This is a puzzle which the player has seen once before, albeit in a simpler form. Lots and lots of players are trying to solve this puzzle the wrong way and dying as a result. This probably indicates that the game is failing to teach the player which of his powers to apply to this sort of situation.
Our goal in designing these levels was to ensure that they generally get harder as the game progresses yet rarely require more than five minutes to complete. The graphs clearly show that these goals are mostly fullfilled, though the end of the game is marked by pretty major changes in difficulty from level to level. Some of this is intentional, but most of it is not; the end game levels need work to smooth out the level of difficulty here.
Finally, here's a breakdown of the number of players per level. You can see that the number of players drops off quickly at first and then slowly degrades toward the end of the game. The slow degrade is normal and ideal: it's produced by people progressing at a regular rate through the game. The large drop off the first four levels indicates that lots of players quit the game early on. This sort of drop-off is probably inevidible, but it also shows how important it is to make those first few levels interesting. At the point at which this data was captured, only about 30% of players who begin the game make it to the forth level.
The encouraging thing about this graph is that there's no major drop in users after level 4. This means that people who do make it to the fourth level will probably play through the rest of the game. There are a few very small drops (like right after level 27, again telling us that the puzzle in that level is a progression blocker), but generally the degrade is linear, which is good.
Next let's look at individual levels. Below is a chart containing all of the aggregated event data for each level. Fields that have gone over some threshold are colored in red, and represent areas that might need rework. Clicking on memory name will display a heat map of the death events in that level. Thanks very much to this page for the crash course in heat maps and the useful image files (and if you are interested in how these are generated, read the disclaimer at the bottom). Areas where lots of users have died are brighter than areas were no users died. Bright red areas are the spots where lots of players died. Note that looking at these images will spoil parts of the game for you!
Note that Replica Island levels do not play out in chronological order, so the levels in this list are sorted in the order that they occur in the game.
Disclaimer: these heat maps are slightly incorrect. In a correctly renderered heat map, areas of maximum intensity (the red spots) must be areas in which the majority of events occurred. The contribution of each event spot is scaled by the total number of events, so only areas in which there is a lot of overlap result in an intense color. However, when there's a lot of data points contributing to the map, each spot may contribute so little to the resulting image that the calculation cannot be performed with normal image formats. To do the calculation correctly, you need floating point color values for each pixel so that you can represent a color contribution that is less than 1. I wrote my heat map tool in php, which does not support floating point colors, and is also not fast enough to implement the feature in php itself, so I ended up clamping the minimum contribution to 1. This means that some spots are more intense than they should be. However, the data is still highly useful for game tuning.
|
|
 |
|
 |
|
Scan to download!
(Click for bigger image)
|
