The first question is why is it important to test. First of all, you want to test in order to evaluate your VR game or app, to see what doesn’t work as intended and get some insights into how to improve it, find possible bugs, to avoid having to table the whole idea after working on it for months and spending large budgets, to prevent rebuilding the project, at least on a big scale. It helps to be more deliberate with the budget! Testing provides an opportunity to identify problems along the way, so there is enough time and resources to iterate. You can also use user testing to generate new ideas. For example, it can be a tactical user research, to understand what to build next.
There can be plenty to discover:
- Do people understand how to play your game or use your app and what’s the learning curve for them? How different demographics interact with it and if patterns appear, what they do or try to do.
- Do people enjoy the game or not? Was it too difficult to play or too easy? Were they matched with players of relevant skills and was the competition fair? Do they feel rewarded?
- Was it comfortable? Physically and mentally.
- What was their favorite part?
- Would people recommend it or play again themselves? How valuable it is for them and why?
Testing is important to ensure that the decisions are right, that these decisions are based on business goals and will provide real value to people who will play this VR game or use this app. It may also help to secure the longevity of the vision and make sure the idea works in the long run.
Also, building for VR is not the same as building for desktop, so there can be unexpected moments. One of the other goals of User Testing is to uncover people’s mental models. what people think about something, what they assume. Even if they didn’t try, or they don’t know much about VR or certain game genres or apps, they already assume and expect something. Those are large parts of what may hold some people back from VR at the moment, for example.
And, finally, when you are aware of it all, you can build a better VR game or experience to make people happy, you will get more downloads, more sales, more 5-star reviews, higher user retention, and so on because you’ll know exactly what the potential users want and what they like.
Next, what is crucial to consider is when to start testing? It should be done early on and the sooner the better.
When a first scene draft is created, it’s a good time to start testing. There is no need to polish or create a detailed prototype in order to test it. You can even test drafts (not interactive prototypes) to preview scenes, storyboards, early blocking (objects’ locations), test some small parts to check the color palette, scales, locations, and approximate distances. It’s easy to test rough volumetric prototypes (grey- or sometimes called white boxing) to see interactions. Traditional media designers create wireframes for web and mobile. To maintain a similar workflow for VR, you can create draft scenes utilizing simple untextured geometry for any object and add mechanics to test if your interaction system is fun, flexible, and reusable. And more VR prototyping tools are emerging to create the first drafts of your game more easily.
Don't forget to test regularly throughout the whole VR development cycle. When a 1-st scene is created, it’s a good time to start testing. Preview it in VR every time you make a change. Test small parts (colors, interactions, scales, proximity, brightness, contrast) to make sure the experience is comfortable, as well as test general ideas and mechanics to see if your approach is fun. Test at all stages of VR product design and development. And keep testing after launch to uncover new features and expand your community.
Testing a VR game or experience early on is essential because in VR it feels very different from how it looks on the desktop. Even very rough small prototypes can confirm the comfort zones, locations, and scales.
Next, it’s important to define what to test, and which part of a VR game or experience should be tested.
Let’s say you already defined why you want to test your VR game. For example, you want to find out why people quit after playing a certain level or after some time in game or app. Your goal is to increase playtime.
To make it happen, depending on the game or app, you may want to test what may cause game interruption. There can be a variety of reasons, e.g., bugs that make the game laggy or just crush it, poor optimization which makes an app too slow to load or function. One of the other drop-off reasons can be an uncomfortable VR environment that causes motion sickness, eye strain, and overall discomfort. It might be a bad user experience or visual design, e.g., too low contrast might cause players to not notice the UI. Or they simply get stuck and don’t know how to proceed further in the game, they get confused, can’t understand what to do next, and eventually quit.
It’s important to know what you want to find out and how this discovery will affect your metrics. Will it help? Then you should test it.
The easiest way to test at the very beginning of design and development is to test yourself or with your team. It’s also important to test with people who have domain expertise in the industry type that your product is in, as well as with VR experts. For example, if you are working on some educational gamified VR app, you would need to test it with people who have expertise in this specific topic (education).
And most importantly, test with people from your Target Audience, otherwise, it all might be useless. Test with those whom this game or app is intended for.
How many testers to select? 5 should be enough to test general UX and usability issues, and some features, but 10 wouldn’t hurt either. To study demographics you would need more people (example - people aged 15-17 y.o, or ppl who play certain types of games). It’s always best to have access to a large pool of testers from your specific Target Audience.
When testing, don’t forget to compare VR users by experience: new VR users (people who have minimal to no knowledge about VR) and experienced VR users (those who use VR regularly, might own multiple VR devices, or even develop for VR). The most wanted testers for you are those who would be potentially buying and using your game or app.
Recruiting testers can be challenging, but get creative, offer something valuable for your potential audience, e.g., early access, or do your research to find out what people from your particular Target Audience could be interested in in exchange for their feedback, if monetary compensation isn’t an option, and try.
There are definitely, some pros and cons with new and experienced VR users. Experienced users can test your app remotely because they have all the hardware and know how to use it. But they might be used to certain VR-specific functionality like various locomotion techniques, so it’s hard to test motion sickness, for example. They may also know how to use some features because they have previous VR experience.
In the case of testing with new or recent VR users, you would need to carry your target device or invite people to your office, cafe, or co-working place in order to conduct testing because they will most likely need a step-by-step guidance on the whole process, especially if you want to share your game via .APK or .EXE file, expect them to record video, audio, and follow certain tasks. Otherwise, getting consistent results (or even any results at all) might be hard. On the other hand, they will have a fresh view of your game as casual users and most likely their opinion will be unbiased.
VR Oxygen solves this problem allowing you to test with both, experienced and new VR users remotely without the burden of setting anything up or the need to provide guidance.
The next challenge is where to find all these people who desire to playtest your game, will show up on time and complete all steps as required? Well, it might be not so easy, but you can try recruiting people in groups like VR Testing.
Members who are interested in helping you out will comment and share their feedback. Participation is voluntary and results purely rely on your expertise with user testing. The downside is that you do not always know what the response rate will be and if you are able to complete the testing on time. The results will be inconsistent among all testers since there is no methodology involved and no way to track or mnage this process. All the members might be absolutely brilliant candidates in various fields but having no process or talking to the wrong target audience may bring the wrong results.
You can also attend expos and events where it will be suitable to showcase your VR game.
It’s also important to consider the context and User Journey - why and where would people want to use your VR game or app? Is it intended for education, productivity, or leisure and entertainment? Where will people want to use it? Is it at home or an arcade experience? Single or multiplayer? User Journey represents the full end-to-end experience including the emotional part, motivations, actions, interactions, touchpoints, emotions, and feelings. What would trigger potential users to engage with your application?
Another detail is that your game sharing method (game codes/keys, channel invites, .EXE or .APK file) may affect where you find testers . WebVR can be more shareable, since there is no need to download it, and you could probably get more responses because it’s very accessible. If your build is already in one of the channels, you may want to share it with a selected number of people you found in advance.
If your game is in app store, sharing keys or game/store links in specific groups can be a solution, but unfortunately, it’s rarely possible to ensure that keys are used properly and won’t be stolen, not mentioning the consistency of the deliverables.
And we definitely don't recommend sharing your files publicly, for security reasons.
With VR Oxygen, you don't need to worry about anything described above.
There are a lot of usability methods and techniques, here are some examples:
- Guerrilla User Testing
- In-person and Remote VR Testing
- VR Analytics, Heatmaps
- Professional VR Testing Services
When selecting a test type, it is crucial to select wisely based on:
- What is being tested
- Platform (e.g., Meta Quest 2, Pico 4, HTC Vive, Valve Index, etc.)
- Genre and type of experience (shooter, educational app, training, multiplayer puzzle, etc)
- Target Audience
- Development stage
- The number of users (and is it a multiplayer or a single player?)
For example, for VR Analytics, having a certain number of users is essential because it won’t make any sense otherwise if a game or app has too few users yet.
Guerrilla user testing is usually a relatively simple and low-cost method that involves a facilitator approaching people and asking them to participate in a test, showing them an app, and asking questions. It’s usually very casual, and participants can be found practically anywhere from a cafe to an expo - wherever you can locate your Target Audience. And you can also select a location based on the User Journey.
Definitely, to be tested with real users this way, it should be a 3D VR prototype with some more or less defined environments and objects. It needs to have the main interactions, sound, and some visual design but doesn’t have to be polished yet.
Attending various VR and non-VR expos and events in order to recruit the testers and specifically the first-time VR users can be helpful. It allows to test in a casual setting, and to observe them, observations play a really big role in VR testing. Post-test interviews and questionnaires are very helpful as well.
Guerrilla testing is perfectly suited to:
- Conduct frequent tests throughout the design and development cycle to identify the usability issues
- To refine the experience
- To manage tests when access to the end-users is limited
Probably the most productive and the most challenging user testing method. To get the best results, the test should be carefully planned.
The test plan should contain all the information such as:
- Testers’ criteria, recruitment, and screening plan.
- Stage of your product and milestones with deadlines.
- What you are testing and why, metrics.
- Specific outcomes you are looking to get (e.g., why players drop off on level 2).
- The methodology you are going to use.
- Scenarios and tasks.
- Questionnaires and when during the course of the test they would be implemented.
- Error and severity assessment plan.
- Logistics.
- Documentation like consent forms and NDA.
How to conduct it: For in-person user testing, you need to bring the testers into your office or wherever your VR setup is, physically, which adds some serious time and maybe even budget constraints.
For in-person testing, you would need a full setting, an office or a room with a set up that includes:
- A headset you are testing with (Meta Quest Pro, Valve Index, Pico 4, etc.)
- Camera for external recording or your phone with a tripod to film the participants, their behaviors, movements, and reactions, as well as sounds and comments.
Since in VR you can not directly study the participants’ view, you should record what they see - ask participants to record their VR View or mirror it onto the desktop via Oculus Mirror, OBS, Steam. There is also an Open VR plugin for OBS to record an eye of choice and adjust settings. Recording just one eye makes sense for games or experiences when you have to aim and want to test with right-and left-handed people.
Another option can be built-in video recording and even live streaming which as well should involve in-VR recording for future evaluations. These options are not always useful due to IP considerations if a game is not launched/confidential. Video recording itself is the whole other topic which we will not cover fully here but there are different options and setups you can use to conduct VR remote user testing.
It’s often recommended to practice a think-aloud protocol. But the cases can vary. Sometimes it’s better to let players relax and just play as they would by themselves while observing them and taking notes. It’s important to not distract testers from the actual test which may happen if they try too hard to not forget to speak aloud instead of just playing the game. Consider all pros and cons and choose wisely.
Always test in the environment and setting that would be natural for your specific software. E.g., if it’s a game it would be most likely played at home, but if it’s a medical training app, it might be used at a University or even Hospital. It can be standing, seated, or room-scale experience. It all will affect the guardian set up, playground areas, and dimensions.
Record both, the gameplay and all actions the user takes in real life, how they move, what they say. It’s really helpful to see the actual players and where they struggle, what they like and don’t like, and when they get tired or feel uncomfortable. You can also include Behavioral and Attitudinal tests - what ppl do vs what they say. E.g., people may say they are fine but by observing them becoming red or pale, or sweating, you can make certain guesses. By observing players’ behavior, you can tell if their feedback is honest. Sometimes people tell the opposite of what they really feel or think because they don’t want to be seen in certain conditions or be judged.
One of the biggest measures of success is that experience is comfortable and easy to navigate, ensuring that a player is able to easily find out what to do inside of VR. Testing learnability is essential.
You may also collect data captured by a wearable device (e.g., Apple Watch, Samsung Gear Fit, Oura ring) to record vitals like heart rate, and blood pressure if needed. It might be useful if you are creating a meditation app, so you can measure if the person is relaxed or stressed.
Before conducting the test, don’t forget to gather and document the demographics, as well as info about the hardware set up of each tester such as HMD, GPU, and CPU for the remote studies.
If you are testing with new users, it’s important to make sure the issue they encountered is not caused by a lack of knowledge about how to use controllers. Make sure you explain it before you test, so you can see that the users are stuck because of in-game settings and behaviors, but not because they don’t know how to push the button inside of VR or how to grab an object. Don’t test the controllers, test your app.
Managing this whole process might be daunting, but with VR Oxygen you won't need to do any of this, all profiles and hardware details for ech tester is provided together with the test results.
Not only the logistics of the whole test should be planned but also its facilitation, and moderation process if applicable. Create a script with a step-by-step plan and make sure all testers have the same experience, same tasks, spend the same amount of time playtesting and get the same questionnaire. Consistency is really important for user testing.
During in-person tests follow these recommendations:
- Put participants at ease.
- Introduce yourself, offer an explanation of the product and any equipment, and make sure they understand.
- Always treat users as experts.
- Be impartial, ensure people can be candid with you and are fine to have good and bad comments. You can mention to the participants that you had limited involvement in the design/ dev. if you feel it’s necessary so that they don’t worry about hurting your feelings. Don’t show you are upset if people don’t like something you designed.
- Look for verbal, and facial cues, and observe. Note everything. Pay attention to health, emotions, and behavior.
Have a script (at least in your head) and be prepared!
Note:
- Health and emotions: People enjoy expressing positive emotions and excitement over VR, but they are less likely to admit to feeling sick, very often we find out about it later, even the next days after the test was conducted. Sometimes new users think it’s normal if it feels uncomfortable.
Sometimes users tell what You (as they think) want to hear, or are afraid if they do something wrong. Watch out for signs of discomfort to make sure they are ok. - Environment: Pay attention to the surroundings, set up everything having enough space from the walls and other objects, and ensure there are no obstacles or hazards near the participant.
- Hygiene: Prepare wet napkins, alcohol sheets, and you can also use Surgical Dressings (patches you can get in Walgreens or anywhere, they are cheap) as a protection, it sticks to the foam.
-
Devices: Cool down headsets and mobile phones between sessions, charge headsets and batteries for controllers, and charge mobile phones if you use them in your testing session.
Another consideration when testing is to prepare the documents needed for tests such as Tasks. People will forget the task as soon as they put a headset on. So spend some time writing the tasks to make sure they are short, clear, and not biased - the words are chosen carefully.
Don’t forget to ask all participants to sign the consent form before the session, and maybe other documents relative to health disclaimers, or NDA, if needed.
Documents to consider:
- NDA
- Consent forms
- Ethical Guidelines (review if needed) - for example, when testing with children or people with different abilities, check some psychological specifics to make sure you comply with all the possible situations.
When testing in person, you can use an app like DropboxSign to let people sign electronically from their phone or tablet and receive an e-mail confirmation. You would need to manually send an NDA to each tester. - Questionnaire after a task or in the end (this is a longer topic as well which we won’t cover in this article but make sure testers fill it out right after completing the test when their memory is still fresh. Timely reminders to complete all steps are important.
And you are all set!
With VR Oxygen, you won't need to worry about the documents - we provide an NDA template if you don't have one, and give you the tools to manage the whole process.
Another method you can use if you have regular active users, is Analytics. Among the tools to measure performance and player engagements are Unity and YouTube, and they both have VR Heatmaps. For YouTube, the video must have 1000 views to enable it, and be 360. So it might be useful to publish some VR scenes on YouTube to receive more statistics. At the time of writing of this material, Unity required to have at least Plus license.
You can measure player engagement, where people looked, how frequently they change position, and use the interaction methods. Discover the most viewed areas of the experience and how long people are looking at a specific part of the video. This method is more suitable for launched applications that already have a user base.
Meta Quest Developer Hub (MQDH) and Steam Diagnostics tools are other essential tools when developing a game or app.
Once you’ve done testing, don’t forget to document all your findings, even if it’s not what you were actually looking for. If you discover something that you were not actually looking for during your tests and made a completely new discovery - get happy about it, every new discovery is a win!
Document all the Research Stages to compare and evaluate later. Documents in any medium you’d like - write, create a gif, or a video.
You can create a Report when the analysis is completed if you want everything to be accurate and organized.
Professional services that specialize in testing would be the most efficient way if you are looking for serious measurable results like sales increase, better reviews, higher user retention, discovering new features to build, understanding the value and replayability, and so on. Depending on the project goals and complexity, one testing round may take from 1 day to 2 weeks.
At VR Oxygen you can fully customize your deliverables and requirements, and create oyur own unique testing workflow. Read more about the process and deliverables here or contact us for more details.
After the tests are done it’s time to study the findings and make some hypotheses. You can create affinity maps or diagrams to help you sort out the discoveries, so you can see if patterns appear.
For example, who and why liked or disliked something. Or connect why they disliked based on what you’ve learned about these participants.
It’s always useful to evaluate app performance (Load Time, Frame Rate, and if there are any drops, lags, bugs), comfort, 3D Art, UI and navigation, gameplay, and game mechanics as well.
When doing tests you can ask test participants to rate the app or rate some features of it. Later you can find out The Net Promoter Score. it’s an index ranging from -100 to 100 that measures the willingness of customers to recommend a company’s products or services to others. It is used to estimate the customer’s overall satisfaction with a company’s product or service and the customer’s loyalty to the brand.
You can rate different components of an app and usability objectives on 1-5 or 1-10 scale depending on what you rate and its complexity. The overall app rating should be on a 1-5 scale as it usually is (like in App Store, Google Play, Oculus Store, Steam, etc.).
Finding both, what people like or dislike is beneficial, because it helps to iterate and move in the right direction.
Let’s check some of the findings now. These are from VR Oxygen’s research very long time ago when the goal was to find out what people do when placed in VR, specifically, first-time users. How they use certain features, their feelings, and thoughts.
People don’t turn around/ look around for cues.
At several expos, VR Oxygen demonstrated Cardboard VR app which was selected as one of the favorites at that time, because of its beautiful graphics. Some of the discoveries were surprising at that time.
Even when people were told that it’s a 360 experience where they can move around and look around, they wouldn’t. They would stay still and wait, some would turn head very slightly to the sides, carefully. And it was because they were not used to it. New Users also didn’t look down, even a little, but looked in front of them.
In this situation the Hypothesis is obvious:
Creating clear visual and audio cues at the very beginning and positioning the UI and important elements in front of the viewer will explain players they can interact with the environment, will guide them. If a scene is engaging, with clear directional cues, more likely people will turn around and explore the full 360. Getting people to understand within the first 5 seconds that they can move around will help to keep them engaged.
People assumed they could perform any gestures, tried to manipulate all the objects.
Some people did that or something similar after encountering that items are interactive in VR, so they tried to manipulate all the objects that were present in the environment. The others assumed that if it’s VR they would see their arms there. They tried to grab things.
In this situation the Hypothesis is obvious:
All the objects should be interactive and useful, and behaviors - are easy to understand in order to support expectations. Creating interactions based on real-life affordances and people’s mental models will provide a smoother learning curve for New Users and may help to predict behaviors.
- Always keep in mind the context, and map a User Journey to see an end-to-end experience. Don’t rush or start the experience automatically, make the user control it. Agency is important.
- Provide users with clear feedback on what’s happening, and how they are doing.
- Use omnidirectional sound in general at all times.
- Recruit your target audience for tests.
- Consider ergonomics (Avoid interactions that can cause “Text Neck”)
- Make sure that labeling is clear for new users
User testing is far too important to be neglected or postponed, it’s really important to start testing early, iterate, and test again until the best solution is found.
More materials about VR Design and development can be found here.
Follow and join us on Twitter, Instagram, LinkedIn, Facebook
Stay tuned!
