The first question is why is it important to test. First of all, you want to test in order to evaluate a VR app, to see what doesn’t work as intended and get some insights into how to improve it, to find possible bugs, to avoid having to table the whole idea after working on it for months and spending large budgets, to prevent rebuilding the project, at least on a big scale. It helps to be more careful with the budget. User Testing provides an opportunity to identify problems along the way, so there is enough time and resources to iterate. You can also use user testing to generate new ideas. For example, it can be tactical user research, to understand what to build next.

In general, when testing there can be plenty to discover such as:

• Do people understand how to use your app and what’s the learning curve for them? It shows how different demographics use your software and if patterns appear (what they do or try to do).
• Do people enjoy the app or not, would they recommend it or play again themselves?
• Was it comfortable?
• What was their favorite part?

Testing is important to ensure that the design decisions are right and are based on business goals and people who will use this VR experience. It may also help to secure the longevity of vision and make sure the app idea works in the long run.

Also, building for VR is not the same as building for desktop, so there can be unexpected moments. One of the other goals of User Testing is to uncover people’s mental models. what people think about something, what they assume. They didn’t try, they don’t know much about VR or certain games or apps, but they already assume and expect something. Those are large parts of what may hold some people back from VR at the moment, for example.

And, finally, when you are aware of it all, you can build a better VR experience to make people happy, and you will get more downloads, more sales, more 5-star reviews, higher user retention and so on because you’ll know what the potential users want and what they like.

Next, what is crucial to consider is when to start testing? It should be done early on and the sooner the better.

When a first scene draft is created, it’s a good time to start testing. There is no need to polish or create a detailed prototype in order to test it. You can even test drafts (not interactive paper prototypes) to preview scenes, paper VR prototypes (skyboxes), storyboards-to-VR, early blocking (objects’ locations), test some small parts to check the color palette, scales, locations, and approximate distances. It’s easy to test rough volumetric prototypes (grey- or sometimes called white boxing) to see interactions. Traditional media designers create wireframes for web and mobile. To maintain a similar workflow for VR, you can create draft scenes utilizing simple untextured geometry for any object and add mechanics to test if your interaction system is fun, flexible, and reusable.

Don't forget to test regularly throughout the whole VR development process. When a 1-st scene is created, it’s a good time to start testing. Preview it in VR every time you make a change. Test small parts (colors, interactions, scales, proximity, brightness, contrast) to make sure the experience is comfortable, as well as test general idea and mechanics to see if your approach is fun. Test at all stages of VR product design. And keep testing after launch to uncover new features and expand your community.

The importance of testing the experience in VR is essential because it always feels very different from the desktop. Even very rough small prototypes can confirm the comfort zones, locations, and scales.

Next, it’s important to define what to test, which part of a VR experience should be tested.

Let’s say you already defined why you want to test an app. For example, you want to find out why people quit after a certain part of the experience or after some time. You want to increase playtime or ensure people continue playing at least until a certain part or level.

To make it happen, depending on the app itself, you may want to test what may cause game interruption. There can be a variety of reasons, e.g., bugs in the code which make the game laggy or just crush it exactly where you think players quit, or poor optimization which makes an app too slow to load or function. One of the other drop-off reasons can as well be an uncomfortable VR environment that causes motion sickness, eye strain, and overall discomfort. It might be a bad user experience or visual design, e.g., too low contrast might cause players to not notice the UI. Or they simply get stuck and don’t know how to proceed further in the game, they get confused, can’t understand what to do next, and eventually quit.

It’s important to know what you want to find out and how it will affect the VR experience after you get the answer to your questions. Will it help? Then you should test it.

The easiest way to test at the very beginning of design and development is to test yourself or with your team. It’s also important to test with people who have domain expertise in the industry type that your product is in. For example, if you are working on some medical-related VR app, you would need to test it with people who have expertise in this specific topic. You can also show your VR app to people who are experts specifically in the VR industry.

And, of course, test with people from your Target Audience, otherwise it all might be useless. Test with those whom this app is intended for.

How many testers to select? 5 should be enough to test general usability issues, some features, but 10 wouldn’t hurt either. To study demographics you would need more people (example - people aged 15-17 y.o, or ppl who play certain types of games). It’s always the best to have access to a large pool of testers from your specific Target Audience.

When testing, don’t forget to compare new VR users (people who have never used VR before, or maybe just tried once, but have no knowledge about it) and experienced VR users (those who use VR regularly and own a VR device or even develop for VR). But the most wanted testers for you are those who would be potentially buying and using your app.

Recruiting testers can be challenging, but get creative, offer something valuable for your potential audience, e.g., early access, or do your research to find out what people from your particular Target Audience could be interested in in exchange for their feedback, if monetary compensation isn’t an option for you, and try.

There are definitely, some pros and cons with new and experienced VR users. Experienced users can test your app remotely because they have all the hardware. But they can be already used to various locomotion techniques, so it’s hard to test motion sickness, for example. They may also know how to use some features because they have a previous VR experience.

In the case of testing with new VR users, you would need to carry your target device or invite people to your office, or cafe, or co-working place in order to conduct testing. On the other hand, they will have a fresh view on your app and most likely their opinion will be unbiased.

There are a lot of usability methods and techniques, but the following are the most useful and widely applicable:

• VR Checklist - VR UX Tool (VR Heuristic Evaluation)
• Guerrilla User Testing
• In-person and Remote VR Testing
• VR Analytics, Heatmaps
• Professional VR Testing Services

When selecting a test type, it is crucial to select wisely based on:

• What is being tested
• Platform (Oculus Rift S, Oculus Quest, HTC Vive, Valve Index, etc.)
• Genre and type of experience (shooter, educational app, training, multiplayer puzzle, etc)
• Target Audience
• Development stage
• The number of users

For example, for VR Analytics, having a certain number of users is essential because it won’t make any sense otherwise if an app has too few players yet.

It’s a VR software evaluation method based on compliance with certain VR principles (the “heuristics”). The heuristics should be selected carefully based on the app specifics, they should be applicable to your VR experience. It’s a great tool to use at a stage before testing with real users, before actually talking to people from your Target Audience. You can evaluate your app yourself, together with your team, and preferably with these who work in the field your product is in (e.g., medicine, education), experts that were mentioned earlier.

It’s a checklist that helps to make sure that the major VR principles are followed. It’s not a set of concrete rules, but a flexible guide which everyone can apply. So it’s more of thought-provoking material to help everyone check if everything was considered. It’s quick and inexpensive feedback. And you can obtain it early in the design process.

Guerrilla user testing is usually a relatively simple and low-cost method that involves a facilitator approaching people and asking to participate in a test, showing them an app, and asking questions. It’s usually very casual, and participants can be found practically anywhere from cafe to an expo - wherever you can locate your Target Audience. And you can also select a location based on the User Journey.

Definitely, for a VR application to be tested with real users, it should be a 3D VR prototype with some more or less defined environments and objects. It needs to have the main interactions, sound, some visual design but doesn’t have to be polished yet.

Attending various VR and non-VR expos and events in order to recruit the testers and specifically the first-time VR users can be helpful. It allows to test in a casual setting, and to observe them, observations play a really big role in VR testing. Post-test interviews and questionnaires are very helpful as well.

Guerrilla testing is perfectly suited to:

• Conduct frequent tests throughout the design and development cycle to identify the usability issues.
• To refine the experience.
• To manage tests when access to the end-users is limited.

Probably the most productive and the most challenging user testing method. To get the best results, the test should be carefully planned.

The test plan should contain all the information such as:

• Testers’ criteria, recruitment, and screening plan.
• Stage of your product and milestones with deadlines.
• What you are testing and why, metrics.
• Specific outcomes you are looking to get (e.g., why players drop off on level 2).
• The methodology you are going to use.
• Scenarios and tasks.
• Questionnaires and when during the course of the test they would be implemented.
• Error and severity assessment plan.
• Logistics.
• Documentation like consent forms and NDA.

How to conduct it: For in-person user testing, you need to bring the testers into your office or wherever your VR setup is, physically, which adds some serious time and maybe even budget constraints.

For in-person testing you would need a full setting, an office or a room with a set up that includes:

• An HMD you are testing with (Oculus Rift, Quest, HTC Vive, etc.)
• Camera for external recording or your phone with a tripod to film the participants, their behaviors, movements, and reactions, as well as sounds and comments.

Since in VR we can not directly study the participants’ view, we should record what they see. You can use Oculus Mirror or VR View if you are on Steam, to mirror the VR experience from headset onto the desktop and to record it using OBS, there is also an Open VR plugin for OBS to record an eye of choice and adjust settings. One of the cases, when it makes sense, are games or experiences when you have to aim and want to test right-and left-handed people.

Another option can be built-in video recording and even live streaming which as well should involve in-VR recording for future evaluations. These options are not always useful due to IP considerations. Video recording itself is the whole other topic which we will not cover fully here but there are different options and setups you can use to conduct VR remote user testing.

It’s often recommended to practice a think-aloud protocol. But the cases can vary. Sometimes it’s better to let players relax and just play as they would by themselves while observing them and taking notes. It’s important to not distract testers from the actual test which may happen if they try too hard to not forget to speak aloud instead of just playing the game. Consider all pros and cons and choose wisely.

Always test in the environment and setting that would be natural for your specific software. E.g., if it’s a game it would be most likely played at home, but if it’s a medical training app, it might be used at a University or even Hospital. It can be standing, seated, or room-scale experience. It all will affect the guardian set up, playground areas, and dimensions.

Record both, the gameplay and all actions the user takes in real life, how they move, what they say. It’s really helpful to see the actual players and where they struggle, what they like and don’t like, when they get tired or feel uncomfortable. You can also include Behavioral and Attitudinal tests - what ppl do vs what they say. E.g., people may say they are fine but by observing them becoming red or pale, or sweating, you can make certain guesses. By observing players’ behavior, you can tell if their feedback is honest. Sometimes people tell the opposite of what they really feel or think because they don’t want to be seen in certain conditions or be judged.

One of the biggest measures of success is that experience is comfortable and easy to navigate, ensuring that a player is able to easily find out what to do inside of VR. Testing learnability is essential.

You may also collect data captured by a wearable device (e.g., Apple Watch, Samsung Gear Fit, Oura ring) to record vitals like heart rate, blood pressure if needed. It might be useful if you are creating a meditation app, so you can measure if the person is relaxed or stressed.

Before conducting the test don’t forget to gather and document the demographics, as well as info about the hardware set up of each tester such as HMD, GPU, CPU for the remote studies. Of course, we are talking about more or less experienced user testers in this case.

If you are testing with new users, it’s important to make sure the issue they encountered is not caused by the lack of knowledge about how to use controllers. Make sure you explain it before you test, so you can see that the users stuck because of in-game settings and behaviors, but not because they don’t know how to push the button inside of VR or how to grab an object. Don’t test the controllers, test your app.

Not only the logistics of the whole test should be planned but also it’s facilitation, and moderation process if applicable. Create a script with a step-by-step plan and make sure all testers have the same experience, same tasks, spend the same amount of time playtesting and get the same questionnaire. Consistency is really important for user testing.

During in-person tests follow these recommendations:

• Put participants at ease.
• Introduce yourself, offer an explanation of the product and any equipment, make sure they understand.
• Always treat users as experts.
• Be impartial, ensure people can be candid with you, and are fine to have good and bad comments. You can mention to the participants that you had limited involvement in the design/ dev. if you feel it’s necessary so that they don’t worry about hurting your feelings. Don’t show you are upset if people don’t like something you designed.
• Look for verbal, facial cues, observe. Note everything. Pay attention to health, emotions, behavior.

Have a script (at least in your head) and be prepared!

Note:

• Health and emotions: People enjoy expressing positive emotions and excitement over VR, but they are less likely to admit to feeling sick, very often we find out about it later, even the next days after the test was conducted. Sometimes new users think it’s normal if it feels uncomfortable.
  Sometimes users tell what You (as they think) want to hear, or are afraid if they do something wrong. Watch out for signs of discomfort to make sure they are ok.
• Environment: Pay attention to the surroundings, set up everything having enough space from the walls and other objects, ensure there are no obstacles or hazards near the participant.
• Hygiene: Prepare wet napkins, alcohol sheets, and you can also use Surgical Dressings (patches you can get in Walgreens or anywhere, they are cheap) as a protection, it sticks to the’ foam.
• Devices: Cool down headsets and mobile phones between sessions, charge headsets and batteries for controllers, charge mobile phones if you use them in your testing session.

Another consideration when testing is to prepare the documents needed for tests such as Tasks. People will forget the task as soon as they put a headset on. So spend some time writing the tasks to make sure they are short, clear, and not biased - the words are chosen carefully.

Don’t forget to ask the participants to sign the consent form before the session, and maybe other documents relative to health disclaimers, or NDA, if needed.

Documents to consider:

• NDA
• Consent forms
• Ethical Guidelines (review if needed) - for example, when testing with children or people with different abilities, check some psychological specifics to make sure you comply with all the possible situations.
  
  When testing in-person, you can use an app like Hello Sign to let people sign electronically from your phone or tablet and receive an e-mail confirmation.
• Questionnaire after a task or in the end (this is a longer topic as well which we won’t cover in this article but make sure testers fill it out right after completing the test when their memory is still fresh.

And you are all set!

Another method you can use is Analytics. Among the tools to measure performance and player engagements are Unity and YouTube, and they both have VR Heatmaps. For YouTube, the video must have 1000 views to enable it, and be 360. So it might be useful to publish some VR scenes on YouTube to receive more statistics. At the time of writing of this material, Unity required to have at least Plus license.

You can measure player engagement, where did people look at, how frequently they change position, and use the interaction methods. Discover the most viewed areas of the experience and how long people are looking at a specific part of the video. This method is more suitable for launched applications that already have a user base.

Once you’ve done testing, don’t forget to document all your findings, even if it’s not what you were actually looking for. If you discover something that you were not actually looking for during your tests and found out a completely new discovery - get happy about it, every new discovery is a win!

Document all the Research Stages to compare and evaluate later. Documents in any medium you’d like - write, create a gif or a video.

You can also create a UX Report in the end when the analysis is made if you want everything to be accurate and organized.

After the tests are done it’s time to study the findings and make some hypotheses. You can create affinity maps or diagrams to help you sort out the discoveries, so you can see if patterns appear.

For example, who and why liked or disliked something. Or connect why they disliked based on what you’ve learned about these participants.

It’s always useful to evaluate app performance (Load Time, Frame Rate, and if there are any drops, lags, bugs), comfort, 3D Art, UI and navigation, gameplay, and game mechanics as well.

When doing tests you can ask test participants to rate the app, or rate some features of it. Later you can find out The Net Promoter Score. it’s an index ranging from -100 to 100 that measures the willingness of customers to recommend a company’s products or services to others. It is used to estimate the customer’s overall satisfaction with a company’s product or service and the customer’s loyalty to the brand.

You can rate different components of an app and usability objectives from 1-5 or 1-10 scale depending on what you rate and its complexity. The overall app rating should be from a 1-5 scale as it usually is (like in App Store, Google Play, Oculus Store, Steam, etc.).

Finding both, what people like or dislike is beneficial, because it helps to iterate and move in the right direction.

Let’s check some of the findings now. These are from VR Oxygen’s research a while ago when the goal was to find out what people do when placed in VR, specifically, first-time users. How they use certain features, their feelings, and thoughts.

People don’t turn around/ look around for cues.

At several expos, VR Oxygen demonstrated Cradboard VR app which was selected as one of the favorites at that time, because of its beautiful graphics. Some of the discoveries were surprising at that time.

Even when people were told that it’s a 360 experience where they can move around and look around, they wouldn’t. They would stay still and wait, some would turn head very slightly to the sides, carefully. And it was because they were not used to it. New Users also didn’t look down, even a little, but looked in front of them.

In this situation the Hypothesis is obvious:

Creating clear visual and audio cues at the very beginning and positioning the UI and important elements in front of the viewer will explain players they can interact with the environment, will guide them. If a scene is engaging, with clear directional cues, more likely people will turn around and explore the full 360. Getting people to understand within the first 5 seconds that they can move around will help to keep them engaged.

People assumed they could perform any gestures, tried to manipulate all the objects.

Some people did that or something similar after encountering that items are interactive in VR, so they tried to manipulate all the objects that were present in the environment. The others assumed that if it’s VR they would see their arms there. They tried to grab things.

In this situation the Hypothesis is obvious:

All the objects should be interactive and useful, behaviors - easy to understand in order to support expectations. Creating interactions based on real-life affordances and people’s mental models will provide a smoother learning curve for New Users and may help to predict behaviors.