How to run a pilot user study on a game [TATT #45]

the acagamic tip tuesday Dec 27, 2022
Stormtroopers running a pilot study in their lab.

In today's issue, I will explain to you how to run a pilot user study on a video game.

Want to ensure that your UX research study is a success? Then it's time to run a pilot study! By gathering a small group of players and testing your study design, you can uncover any potential issues or problems and make necessary adjustments before diving into the full study. This way, you can be confident that your study will go off without a hitch and provide valuable insights when you recruit a larger audience. Don't skip the pilot study step - it could make all the difference in the success of your research!

Unfortunately, many game UX researchers struggle with designing practical pilot studies. There are a few primary reasons for this.

Many studies lack planning and preparation.

Here are some reasons why people struggle with running pilot studies:

  • Lack of planning and preparation: Many people struggle with pilot studies because of a lack of planning and preparation; this includes unclear goals/objectives and recruitment plans.
  • Difficulty gathering and analyzing data: Running pilot studies can be challenging because of a lack of tools/resources and unclear data collection/analysis plans.
  • Difficulty using the results of the pilot study: Many lack a plan or resources to use pilot study results to improve research/design.

Don't let these challenges hold you back! With my step-by-step guide, you can efficiently run a successful pilot study for your video game under development. And if you want to test the waters before bringing in real players, you can even try it out with your friends first. By following my guide, you'll be well on your way to creating a successful and well-tested game.

Here's how, step by step:

Select a game to test

If you are not working for a games company yet, start by picking a simple game to focus on to practice your skills (e.g., pick a single-player computer game). Consider data collection feasibility when selecting a game for your pilot user study. You can choose an existing complete game or one you made yourself, depending on which is easier to test and collect data from. If creating the game, factor in difficulty level, resources, and time needed to ensure the study is achievable within parameters/timeframe. Researching existing complete game mechanics and features helps inform questions during testing for more meaningful results. Avoid testing multiplayer games; it can be tricky to coordinate multiple players.

Define the goals of your study

Before you begin the pilot study, you must clearly understand what you want to accomplish with your actual study. This may include gathering feedback on a specific gameplay mechanic, identifying any bugs or issues, or testing the game's performance on different devices.

Selecting good research questions is vital here. Define specific, actionable research questions relevant to your study's goals and practical enough to answer within a reasonable timeframe.

UX research can answer 'why' questions well by combining observation and feedback from players. Observe player interactions with the game and ask for their thoughts to optimize player experience. Clarifying research questions will help plan the details of your study. Define tasks, data to collect, and questions for players during/after play.

If you are testing an existing game, avoid testing sections close to the end of the game; it may be difficult to find players fresh with that content, and those who have already finished may not provide an insightful take. Choose early gameplay elements to test for this study when testing existing games. In addition, avoid selecting an existing game that's hard to find testers for, or one whose needs differ from yours, or where extra care is needed to ensure a safe test.

Ensure your testing practice won't cause harm. Confirm the game isn't triggering or harmful to testers. Consider ethical implications when selecting a game for your study. Ensure participants are comfortable with a complicated subject matter before starting your test.

Recruit a small sample of participants

Pilot studies are small and there to test how your actual study will run, so you don't need many participants. Ideally, for a real pilot test, recruit the right players for your game and questions (a representative sample). If that's not possible, opt for a 'convenience sample' (whoever is around you). So, recruit expert players only for questions suited to them or those who haven't seen the game before for questions they can answer. Recruit participants via online ads or social media if you want a larger sample.

Remember, you are more interested in testing your research method than getting actionable results, but testing your research question makes sense in conjunction with your method. For a pilot, you can usually get satisfying feedback from 5-7 people and adjust your testing protocol and methods accordingly.

How many participants do you need for a complete study though? Recruitment varies based on study aims and desired data analysis. For usability testing, a small sample of participants can uncover substantial issues. For (user and market) research, you need a larger sample to guarantee valid results.

Decide what information you will collect and where

Develop a clear, detailed plan to avoid costly mistakes and ensure consistency in play sessions. Create a script to keep interactions standardized; this will help ensure accurate feedback on the player experience.

  • Outline the player tasks: Start point, duration and specific game activities. Consider including the tutorial if not starting from the beginning. To schedule people to complete your test, you must tell them how long the process will take. So, you need to plan how long they will be playing and how much is reasonable for them to do in that time. Some players may take longer than others to get through tasks that you set, and you want to allow for that. So, be realistic about how much they can complete. Tie tasks to core questions to focus the study and answer research questions.
  • Player task flow: Start point? Playtime? Orientation? Session length?
  • Data collection: Do you record gameplay? Tools needed for capturing (computer-based games are best for this). Think aloud or not? Collect analytics? Prompt questions during gameplay or an interview/survey at the end? Where are you storing the data?

Create a run-through script and checklist divided into pre-arrival, arrival and post-arrival sections. This ensures consistency throughout the study and reduces bias. Ensure the game is ready to go before the test. Set up a stable build in advance without any last-minute changes if it's your own game. Ensure existing hardware is working and you can access necessary save points. Follow your script in the same place you plan to use for the actual study.

Choose a quiet, comfortable space for your test. Observe in person to address technical issues and encourage participants to think aloud (if desired). Use the same environment for all sessions to reduce feedback variability.

Run people through your study and test analysis

When running people through your study, it is crucial to ensure that each player experience is consistent. You want to follow the pre-defined script and check off all the steps in that script. Making sure to stick to the same process will help minimize possible bias or errors during the actual study. Additionally, when recruiting users for pilot testing, ensure they have similar backgrounds. Hence, their experiences are as comparable as possible - this could include factors such as age range, gender or gaming experience level. Finally, take careful notes during the sessions and debrief with participants afterwards to gain further insights into their reactions and experiences.

Create a lessons learned document

Identify areas that need changing, such as recording, shortening surveys and providing more instructions. You have collected data needed to answer some questions about the game, too. Summarize critical results, evidence and issues from research questions, just like you would for an actual study.

Your document should provide an overview of what went well during the data collection and analysis processes and highlight any areas for improvement. This document aims to ensure any issues are addressed before running the complete study - thus ensuring its success. Going through all data collection and analysis stages, just like in a real-world study, will also help you gain valuable insight into how such studies can be conducted more effectively.

Wrap Up

By following these steps, you can run a practical pilot study that helps you to gather valuable feedback and data on your study and make necessary adjustments before conducting it with a broader audience. With careful planning and execution, you can ensure that your study is the best it can be before your start rolling it out with a larger audience.

Games Research Find of the Week

This paper presents an example of a pilot study using Gamettes, a game-based methodology for collecting rich behavioural data on human decisions. Gamettes are simple games for studying decision-making, but their complexity complicates conducting experiments. The authors present the results from their pilot study using Gamettes in a supply chain task and provide evidence why it's crucial to conduct pilot studies when using Gamettes for behavioural research. Pilot studies are essential for gamified research to identify issues that could bias the experiment and lead to a failed study. Results from the pilot study shed light on the efficacy of the experiment and how Gamettes' players interact with it.

Read the full study: Omid Mohaddesi and Casper Harteveld. 2020. The Importance of Pilot Studies for Gamified Research: Pre-Testing Gamettes to Study Supply Chain Decisions. In Extended Abstracts of the 2020 Annual Symposium on Computer-Human Interaction in Play (CHI PLAY '20). Association for Computing Machinery, New York, NY, USA, 316–320. https://doi.org/10.1145/3383668.3419889

Join UX rockstars learning about games, research, and writing

Each Tuesday, I will send you tips for UX in game research and design. And once a month, I will send you emails about academic writing.

No hassle, no selling your information, no spam. You can unsubscribe anytime.