How to Run a Diary Study With Mobile Games [TATT #43]

the acagamic tip tuesday Dec 13, 2022
An AI-generated image of a character writing a diary entry in Rembrandt lighting.

In today's issue, I will show you how to run a diary study with your mobile game to get a comprehensive view of player experience with your game over time. Lab studies may lack ecological validity as game builds become more complex. Some mobile games are built around requiring people to check in on mechanics over more extended periods of time. For example, players might get different quests and challenges daily or have to check on resources. Players must manage time wisely, returning to the game daily to complete objectives.

Diary studies for mobile games: Get a comprehensive picture of players' experiences over an extended period of time.

Diary studies have limitations though.

  • Managing participants is essential to successful data collection in mobile game diary studies.
  • Sorting and analyzing the data can take many hours, making it challenging in a quick research environment.

However, diary study data offer rich, contextual insights not available from lab studies. A diary study can capture gameplay data in context over multiple days or weeks. In a diary study, participants remotely document their mobile gameplay activities and behaviours. Reports offer contextual insights that are hard to get in the lab. Before diving into creating a diary study though, you should consider your game structure when designing it.

Unfortunately, many designers either don't know about diary studies or do not consider the structure of their game.

Here is how to design a mobile diary study step by step:

1. Decide on the number of participants.

Timeline, resources and research questions limit how many people you can include. You want to uncover themes and patterns while considering that this is large-scale qualitative data. Five participants might generate heaps of insights. It's common to see UX researchers include up to 20 people to guarantee diversity.

2. Plan to give people access to your game.

The easiest way to do this is to invite participants to install the game on their mobile device in person or remotely via TestFlight (iOS) or Google Play (Android).

3. Set the study duration.

Your study length can be fixed to several calendar days. Something like 1-2 weeks is prevalent. Alternatively, you can structure it around your game milestones, such as completing the first stage or chapter of your game.

4. Set the diary writing method.

There are tools like Indeemo that people can use to record their reflections on game sessions. Of course, you can also let them use paper, online surveys, text messages, video logs, or voicemails. The advantage of using a mobile phone is that lots of tools are available for this.

5. Set the diary content.

You want to let people collect a vast array of interesting log data with each entry, such as location, activities, feelings, technical issues, or any other details they feel are important to their experience of that moment. Of course, if your game can timestamp and capture session frequency and duration as well as the overall game progression for that specific player, even better.

6. Set the diary entry frequency.

People's diary entries should provide researchers with necessary context without overwhelming participants. Good reporting times are after game sessions, daily, or upon game milestones.

That's it.

Those are the key details to keep in mind for designing a diary study for mobile games.

Games Research Find of the Week

This short paper discusses lessons from a diary study of Electronic Arts' NHL'16 conducted in August 2015. Their findings suggest high risk but great benefit in terms of actionable data and storytelling artifacts from diary studies for AAA games. To reduce the risk, the authors present guidelines for conducting a diary study in AAA games user research, which has not been explored previously: what and when to research, creating a schedule, recruitment, executing the procedure, data analysis/reporting, and privacy. The paper helped establish diary methods in GUR, providing AAA game examples and results.

Read the full study: Serena Hillman, Tad Stach, Jason Procyk, and Veronica Zammitto. 2016. Diary Methods in AAA Games User Research. In Proceedings of the 2016 CHI Conference Extended Abstracts on Human Factors in Computing Systems (CHI EA '16). Association for Computing Machinery, New York, NY, USA, 1879–1885. https://doi-org.proxy.lib.uwaterloo.ca/10.1145/2851581.2892316 

Join UX rockstars learning about games, research, and writing

Each Tuesday, I will send you tips for UX in game research and design. And once a month, I will send you emails about academic writing.

No hassle, no selling your information, no spam. You can unsubscribe anytime.