Leon Xiao | Keywords in Play, Episode 15

As a joint venture, “Keywords in Play” expands Critical Distance’s commitment to innovative writing and research about games while using a conversational style to bring new and diverse scholarship to a wider audience.

Our goal is to highlight the work of graduate students, early career researchers and scholars from under-represented groups, backgrounds and regions. The primary inspiration comes from sociologist and critic Raymond Williams. In the Preface to his book Keywords: a vocabulary of culture and society, Williams envisaged not a static dictionary but an interactive document, encouraging readers to populate blank pages with their own keywords, notes and amendments. “Keywords in Play” follows Williams in affirming that “The significance is in the selection”, and works towards diversifying the critical terms with which we describe games and game culture.

Please consider supporting Critical Distance at https://www.patreon.com/critdistance

Production Team: Darshana Jayemanne, Zoyander Street, Emilie Reed, Bettina Bodi.

Audio Direction and Engineering: Damian Stewart

Double Bass: Aaron Stewart

Transcription: Charly Harbord

Transcription:

Darshana: You’re listening to Keywords in Play, an interview series about game research supported by Critical Distance and the Digital Games Research Association.

As a joint venture, Keywords in Play expands Critical Distance’s commitment to innovative writing and research about games while using a conversational style to bring new and diverse scholarship to a wider audience.

Leon Zhao, very happy to have you here for this episode of Keywords in Play. Could you please introduce yourself in your own words?

Leon: Sure. I’m a law graduate of Durham University. I recently finished the training course for barristers, well litigation lawyers, I expect to be called to the bar of England and Wales next year. I’m currently teaching Tort law at Queen Mary University of London as a teaching associate. I research the regulation of video games with a focus on these gambling-like mechanics in video games, that are commonly known as loot boxes. Given my background in law, I would describe the kind of research that I do as fundamentally legal research. However, I’ve taken on certain empirical aspects from other disciplines, and have started to explore other research methods. In addition, my Chinese background has allowed me to explore the question of loot boxes in China specifically, being able to speak Chinese has really helped.

Darshana: Can you tell us the title of the paper that we’re going to be talking about today and introduce us to your co-authors and their affiliations?

Leon: We will be discussing a paper entitled ‘What are the odds’ lower compliance with Western loot box probability disclosure, industry self-regulation, then Chinese legal regulation, and this is still a preprint that is undergoing peer review at the moment. This paper was co-written by myself and Laura L. Henderson, who is also a graduate of Durham law school, and Philip W. S. Newell who is currently a postdoctoral researcher at Central Queensland University.

Darshana: Can you tell us a little bit about the phenomenon of loot boxes? One, in particular, led you to a comparative study of the People’s Republic of China and United Kingdom contexts. And what were your hypotheses?

Leon: Loot boxes are these mechanics in video games that can be bought for real-world money. The player obtains randomized rewards whenever they purchase a loot box. Most of the time, the player will get a reward that is perceived to be worth less than the purchase price of the loot box. However, rarely, the player will receive a reward that is very valuable. Players are known to purchase many loot boxes in order to try to obtain or chase after those rare rewards. These mechanics are frequently implemented in video games. Previous research has found that about 60% of the highest-grossing mobile games in Western Countries and about 90% in China contain these loot boxes. These mechanics are also frequently implemented in games deemed suitable for children to play. We know that loot boxes generate a lot of money for the video game industry. I think Electronic Arts financial reports can be interpreted to say that at least 30% of their revenue is coming from loot boxes. Because of loot boxes randomized nature, conceptually and psychologically some academics have argued that they are akin to gambling. Previous research has found the correlation between loot box expenditure and problem gambling severity. Generally, players who spent more on loot boxes tend to have higher problem gambling severity. I noted that we do not know the direction of this correlation in the sense that we do not know whether players who are already problem gamblers tend to spend more on loot boxes, or whether loot boxes indeed make people into problem gamblers. Further research on this still needs to be conducted. However, given the similarity between loot boxes and gambling, many countries have considered or are currently considering whether or not to regulate loot boxes. This is where the country’s start to diverge in China, by law, it is required that video game companies must disclose the probabilities of obtaining randomized rewards from loot boxes. In contrast, in every other country, this is only required by industry self-regulation, there is no similar law in other countries requiring the same. We wanted to do this study comparing China and the UK because we wanted to find out whether it is necessary to have legal regulation in this context to have a higher degree of compliance and a higher degree of consumer protection. Industry self-regulation is generally seen as being more cost-effective than legal regulation because, for example, a self-regulator might understand the industry better, it might in this context, understand game design better. So, it will be easier for a self-regulator to spot non-compliance than a legal regulator who is not used to video game design, for example. So, if industry self-regulation is working just as well in the UK as legal regulation in China, then there is arguably no argument to say that the UK should impose a similar measure as law. However, if the compliance rate in the UK is lower than in China, then there is an argument to say that policymakers and regulators should consider requiring probability disclosures by law so that what was promised to consumers by the industry self-regulation that is not actually achieved, will actually be given to the consumers. So, our first hypothesis was that the disclosure rate that we will find in the UK will be lower than the 95.6% disclosure rate that we observed in China last year. Another second research question that we studied was whether loot box prevalence rate has changed in the last two years. We have a previous study from 2019 conducted by Zendle et al., which found that the loot box prevalence rate was 59% on the iPhone platform in the UK. Loot boxes have received a lot of backlash since that time, many players are realizing that these mechanics are problematic and arguably predatory. And we have also had media reports of certain companies deciding not to implement loot boxes in their games anymore. For example, Epic Games have said this. So, our second hypothesis was that the loot box prevalence rate that we will now find in 2021, will be lower than the 59% that Zendle et al. found in 2019, two years ago.

Darshana: Can you tell us a little bit about your method, and how you coded the ways that games disclosed reward probabilities? Were there any curveballs or cases that stood out in particular?

Leon: I think what you put a name on the method that we use, I think it would be content analysis. Basically, what we did was we went on to App Annie, which is an analytics company that provides the sales data for all the apps on the app stores, and we captured the top ranking list on a particular date. Using that list, which was our sample, we downloaded all the games from the Apple App Store. And we played those games for up to 40 minutes, basically, until we found a loot box. And we also tried to find the probability disclosures for those loot boxes both inside the game and externally on the game’s official websites. We chose this method of finding loot boxes and probability disclosures through gameplay rather than say, for example, watching videos of other players engaging with loot boxes, because very few videos will actually show a player accessing the probability disclosures. So, we basically had to do this through gameplay in order to ensure that we did manage to find the probability disclosures. I think we spent a considerable effort when trying to find the probability disclosures above and beyond what a normal player would find. So, I would say that in our opinion, if we did not find a probability disclosure, it is very likely that that company did not make a probability disclosure. When coding probability disclosures, we had a number of subcategories of the way that that particular probability disclosure can be accessed from our previous study in China. However, we were also prepared to define and add a new subcategories if we found one that did not fall into any of those pre-existing subcategories. One subcategory, for example, might say that this requires the player to interact with a button that specifically says probabilities or drop rates on the purchase page where the loot box can be bought. Whilst another subcategory might say that the player has to engage with a button on the loot box purchase page that is just a question mark or an exclamation mark that does not refer to probabilities specifically. We also dual coded 15% of the sample to ensure that our coding was reliable. We took screenshots of all the loot boxes and all the probability disclosures, and we’ve shared these publicly online. I think open science in this particular area’s very important so everyone can go and scrutinize our screenshots and find out whether we did indeed find those probability disclosures or not. In terms of curveballs, I think the important ones are illustrated in using screenshots in the paper. I will just highlight a few here. I think the first one is in relation to Minecraft and Roblox. Those are sandbox-like games, where players can generate content in those games. So, for those games, the original developer of those two games did not, to our knowledge, put loot boxes into those games. However, it is possible for users or third parties to generate content that are like loot boxes and put them into those games. Roblox specifically recognizes this officially on their forums. So, this is quite interesting in the sense that Apple would require these two games to make probability disclosures. However, they are not the ones who made the original loot box so all that they can do is require people who may the loot boxes, that those third parties to make probability disclosures. We decided to exclude these two games from the probability disclosure related analysis because we thought that it doesn’t really further the aim of our research because we want to find out how video game companies disclose probabilities. We’re sure that there are user-generated contents that disclose probabilities very well. But we’re also quite confident in saying that we can definitely find loot boxes generated by users that do not disclose probabilities. Some other examples that I think is worth highlighting is, for example, the loot box showing figure 7 in the paper. This is the loot box in The Sims Free Play. You’ll see that the disclosure does not add up to 100%. This is quite problematic because you would expect all the probabilities disclosed to add up to 100%. So, in this case, we do not actually know what 0.4% of the time the player will receive. Another example with highlighting is shown in figure 8. This one discloses a range of probabilities. For example, it says the probabilities of obtaining a specific item is between 0.5% and 2%. That’s actually a very big difference on one hand on the 2.0% hand it is four times more likely to receive the reward than on the 0.5% time. So, we don’t actually know what the exact probabilities of obtaining a specific reward is. Finally, I think another one is worth highlighting is actually figure 5 this is a loot box that allowed the player to obtain premium currency. The player was able to buy these premium currencies directly at a set conversion rate. So, they will spend a certain amount of real-world money to get a specific amount of premium currency. However, they can also buy premium currency through this loot box. And some of the time they’ll receive less premium currency and some of the time to receive more premium currency. So, I think this was much closer to gambling in the sense that they’re actually buying something that has monetary worth, arguably. Our sample consisted mostly of the 100, top-grossing iPhone games. We chose this platform in particular, because the previous Chinese study that we did, only looked at the iPhone platform. So, if we looked at another platform, we couldn’t really compare them. We also chose to do the 100 top-grossing games because we thought that these were the games that the most number of players would have engaged with.

Darshana: Many folks who will be familiar with the idea of loot boxes, but less so with the terms pity timer, or sludge, which you discuss in the paper. What do these terms refer to? And why were they important to bring into your study?

Leon: So, pity timer is I would say a sub mechanic of a loot box in the sense that it is implemented as part of some loot boxes and not as part of others. What it means is, I think it’s best to illustrate with an example. So, imagine that there is a 1% chance of getting a rare item from one particular loot box, and there is an 11% chance of getting nine other more common items. So, what a pity time and does is that as the player purchases more loot boxes, the probability of getting that rare item increases so that 1% grows to 2% 3% and those 11 % decreases. So basically, it means that the player will be more likely to receive a rare reward when they have bought more loot boxes in these mechanics are usually implemented such that once the player does obtain a rare reward, the probabilities are reset back to where they were at the start, and the pity accumulation can start again. We thought that this was an important aspect to study because this reflects that the probabilities of obtaining potential rewards from loot boxes changes in the background. In most cases that we found, the probabilities have increased. However, it is not unreasonable to suggest and we indeed have found examples where the probability of obtaining those rare items have decreased. As the player bought more loot boxes, we have seen industry patterns that have recognized that they will like to use players behavioural data to potentially change the probabilities of obtaining specific rewards. For example, if we’re to take a more cynic perspective onto this, perhaps having identified that a player spends a lot of money on loot boxes, because they did so on previous loot boxes, a game company might reduce the probabilities of obtaining rare rewards from newer loot boxes they implement for that player only to generate as much money as they can from that player because they know that that player will continue to buy loot boxes until they get the rare items. Another interesting thing that we thought was that in the context of gambling, there is something known as the gambler’s fallacy. What that means is, say for example, with a roulette wheel. Generally, a lot of people would tend to think that once the ball has landed, say, for example, on red five times in a row, they would tend to think that the probability of the ball landing on red again, is lower because the ball has landed on red multiple times previously. But this is a fallacy because these are actually independent events. If the probability of a landing on red is the same, it has not changed because they landed on red previously. However, that is a fallacy in the context of gambling. But when a pity timer is implemented, this actually becomes true that the player is now more likely to obtain those rare rewards, the probabilities have indeed increased. This is something of course for future study. But is it possible that because players have engaged with these mechanics, that they now have a stronger belief of the gambler’s fallacy, and they may be more harmed when they do gamble? In terms of the, the second term that of sludge, that is a term from the behavioural science discipline, it is the opposite of nudge. What a nudge is, is something that helps people to make better decisions. For example, a nudge might be that we put the more healthy foods, as you walk into the cafeteria, we put those healthy foods closer to the entrance so that the person would tend to pick those things up more than if, if you put the healthy things at the end of the cafeteria. What a sludge is, is the opposite of that. Imagine that, instead of putting the healthy food at the entrance, you instead put the junk food at the entrance to try to encourage people to get those instead, I think it’s important to differentiate between sludge that was positively implemented and sludge that just happened to be there. Sludge that is positively implemented will, for example, be putting the junk food at the entrance or sludge that just happened to be there in the loot box context might just be a link that did not work because for some reason they programmed it wrong. But we thought it was important to identify sludge in the loot box disclosure context because we should find them and remove them to help players get the information as you get the maximum benefit from the probability disclosure that they could have gotten.

Darshana: So, with the proviso that these are provisional findings, this is a preprint that you’ve uploaded, what were the findings? And how would you characterize your thinking about loot box design and regulation in the regions studied with regards to your two hypotheses?

Leon: Sure, I think I would take hypothesis 2 first because that is more general and then we’ll look into our hypothesis 1, which was about the probability disclosures. So, hypothesis 2 was there we were, we predicted that we will find a lower loot box prevalence rate now than two years ago, however, we actually found the opposite. We found that 77% of the 100 highest-grossing iPhone games contained loot boxes, compared to 59% found by Zendle et al. in 2019. This was statistically significantly higher. So, it would seem that at least on the mobile video game market, the player backlash and some companies deciding to stop implementing loot boxes does not reflect a broader trend in the video game industry. Now, players, both adult and children are actually more likely to be exposed to loot boxes than before. As to the second hypothesis, we predicted that the compliance rate with industry self-regulation in the UK will be lower than the disclosure rate found previously in China with the legal regulation. And we indeed found this, we found that only 64% of games containing loot boxes, disclosed probabilities. This was statistically significantly lower than the 95.6% disclosure rate found in China. This meant that we could not find probability disclosures for 36% of games containing loot boxes. This, in our opinion, reflects that the legal regulation in China was more effective at ensuring compliance and consumer protection than industry self-regulation. We say that the current self-regulatory situation in the UK and by extension in most other countries is unsatisfactory because the self-regulation does not impose any negative consequences for noncompliance. A company that has not complied with industry self-regulation is not punished with a fine or by, for example, being delisted from the App Store, nothing happens to them. We examined the highest-grossing iPhone games; these were likely the video games that were the most scrutinized by players. So, we would say that the disclosure rate with the compliance rate amongst lower ranked games is likely even lower because those games are scrutinized less by players and by other companies. We also say that the current self-regulatory situation is arguably worse than if the video game companies did not even say that they would make disclosures. Because what is happening now is companies are basically saying that they have made disclosures when they have not done so because when you upload a game onto the Apple App Store, you actually need to say that you have complied with Apple’s App Review guidelines, which includes the loot box probability disclosure self-regulation requirements. So, what the companies are doing now is that they are purporting to have made disclosures when they have not. This, we think is worse than not saying that you have made disclosures and not making them. And indeed, European consumer protection regulations would say that purporting to have done something without doing it is worse. A third finding though, we would highlight is that even for the disclosures that we did find, most were difficult to access and not prominent, in the sense that we deemed that the most prominent disclosure format is automatically showing the probabilities on the in-game loot box purchase page without requiring the player to do anything. However, most games require the player to tap a button or interact with something before they are shown the disclosure. Companies could have done the automatically disclosing methods, but they’ve chosen not to do that. And we say that the disclosure they have made is therefore suboptimal. Further, we found that only 21.3% of games containing loot boxes disclose probabilities on the game’s official websites. This is bad because there are people who do not play the games who would also be interested in learning about the probability disclosures, for example, parents of child players. We think it will be unfair to force these people, these parents to have to play the game in order to find the disclosure. We should make it so that if they go onto Google, and they search for it, that they will easily be able to find it. In terms of the differing regulation of loot boxes across the world. Belgium has, for example, applied its existing gambling law and has banned all implementations of paid loot boxes. Whilst in China, there has not been a ban, but there was this legal requirement to disclose probabilities. We say that the Chinese approach is more nuanced than the Belgian approach. The Belgium approach is potentially over-regulation in the sense that not all players will be harmed by loot boxes not all players will overspend on loot boxes. However, because how the law has been applied in Belgium, some video games have been removed from the Belgian market. For example, Nintendo has removed Fire Emblem Heroes from Belgium, so Belgian players can no longer play those games. Some Belgian players who would never have been harmed by loot boxes can now no longer play those games. Loot boxes is an alternative or sometimes additional way to monetize video games. And for some players, it helps them to get free entertainment. And because of the way that gambling law has been applied in Belgium, these players ability to play these games have potentially been restricted and deprived. We also say that the company’s commercial interests have also been negatively affected by the ban on loot boxes in Belgium. At the moment, we do really know what regulatory direction the UK will take. And so, I’m not really sure how much I can comment on that. The research currently on loot boxes have generally relied on self-reported data from players. I think it’s very important to develop a more scientific and objective understanding of the effect that loot boxes have on players. And one way of going about doing that is for the industry to share some loot box purchasing data with researchers. So, can we please get industry data?

Darshana: Yeah, I don’t think you’re alone in asking for that! So, Leon, thanks so much for speaking with us today about this paper. Where can folks find out more about your work?

Leon: I am on I think most social medias you can find me on Twitter. You can probably find me on my website as well where you can get access to all of my research paper and data.

Darshana: We hope you have enjoyed this episode of Keywords in Play. For more great ideas around games check out criticaldistance.com. Or take a dive into the DiGRA archives at DiGRA.org.