This first part of the video explains why testing is important, and when, what, and with whom to test. It's a usability test example that can be conducted independently with the help of the existing tools. If you'd like to get more comprehensive testing done, please contact us.

The second part covers different usability testing methods, what tools and software to use.

In the third part, we emphasize the importance of proper analysis and documentation of your findings and share some tips and discoveries.

Why Test?

The first question is why is it important to test. First of all, you want to test in order to evaluate a VR app, to see what doesn’t work as intended and get some insights into how to improve it, to find possible bugs, to avoid having to table the whole idea after working on it for months and spending large budgets, to prevent rebuilding the project, at least on a big scale. It helps to be more careful with the budget. User Testing provides an opportunity to identify problems along the way, so there is enough time and resources to iterate. You can also use user testing to generate new ideas. For example, it can be tactical user research, to understand what to build next.

In general, when testing there can be plenty to discover such as:


Testing is important to ensure that the design decisions are right and are based on business goals and people who will use this VR experience. It may also help to secure the longevity of vision and make sure the app idea works in the long run.

Also, building for VR is not the same as building for desktop, so there can be unexpected moments. One of the other goals of User Testing is to uncover people’s mental models. what people think about something, what they assume. They didn’t try, they don’t know much about VR or certain games or apps, but they already assume and expect something. Those are large parts of what may hold some people back from VR at the moment, for example.

And, finally, when you are aware of it all, you can build a better VR experience to make people happy, and you will get more downloads, more sales, more 5-star reviews, higher user retention and so on because you’ll know what the potential users want and what they like.

When to Test?

Next, what is crucial to consider is when to start testing? It should be done early on and the sooner the better.

When a first scene draft is created, it’s a good time to start testing. There is no need to polish or create a detailed prototype in order to test it. You can even test drafts (not interactive paper prototypes) to preview scenes, paper VR prototypes (skyboxes), storyboards-to-VR, early blocking (objects’ locations), test some small parts to check the color palette, scales, locations, and approximate distances. It’s easy to test rough volumetric prototypes (grey- or sometimes called white boxing) to see interactions. Traditional media designers create wireframes for web and mobile. To maintain a similar workflow for VR, you can create draft scenes utilizing simple untextured geometry for any object and add mechanics to test if your interaction system is fun, flexible, and reusable.

Don't forget to test regularly throughout the whole VR development process. When a 1-st scene is created, it’s a good time to start testing. Preview it in VR every time you make a change. Test small parts (colors, interactions, scales, proximity, brightness, contrast) to make sure the experience is comfortable, as well as test general idea and mechanics to see if your approach is fun. Test at all stages of VR product design. And keep testing after launch to uncover new features and expand your community.

The importance of testing the experience in VR is essential because it always feels very different from the desktop. Even very rough small prototypes can confirm the comfort zones, locations, and scales.

What to Test?

Next, it’s important to define what to test, which part of a VR experience should be tested.

Let’s say you already defined why you want to test an app. For example, you want to find out why people quit after a certain part of the experience or after some time. You want to increase playtime or ensure people continue playing at least until a certain part or level.

To make it happen, depending on the app itself, you may want to test what may cause game interruption. There can be a variety of reasons, e.g., bugs in the code which make the game laggy or just crush it exactly where you think players quit, or poor optimization which makes an app too slow to load or function. One of the other drop-off reasons can as well be an uncomfortable VR environment that causes motion sickness, eye strain, and overall discomfort. It might be a bad user experience or visual design, e.g., too low contrast might cause players to not notice the UI. Or they simply get stuck and don’t know how to proceed further in the game, they get confused, can’t understand what to do next, and eventually quit.

It’s important to know what you want to find out and how it will affect the VR experience after you get the answer to your questions. Will it help? Then you should test it.

Whom to Test With?

The easiest way to test at the very beginning of design and development is to test yourself or with your team. It’s also important to test with people who have domain expertise in the industry type that your product is in. For example, if you are working on some medical-related VR app, you would need to test it with people who have expertise in this specific topic. You can also show your VR app to people who are experts specifically in the VR industry.

And, of course, test with people from your Target Audience, otherwise it all might be useless. Test with those whom this app is intended for.

How many testers to select? 5 should be enough to test general usability issues, some features, but 10 wouldn’t hurt either. To study demographics you would need more people (example - people aged 15-17 y.o, or ppl who play certain types of games). It’s always the best to have access to a large pool of testers from your specific Target Audience.

When testing, don’t forget to compare new VR users (people who have never used VR before, or maybe just tried once, but have no knowledge about it) and experienced VR users (those who use VR regularly and own a VR device or even develop for VR). But the most wanted testers for you are those who would be potentially buying and using your app.

Recruiting testers can be challenging, but get creative, offer something valuable for your potential audience, e.g., early access, or do your research to find out what people from your particular Target Audience could be interested in in exchange for their feedback, if monetary compensation isn’t an option for you, and try.

There are definitely, some pros and cons with new and experienced VR users. Experienced users can test your app remotely because they have all the hardware. But they can be already used to various locomotion techniques, so it’s hard to test motion sickness, for example. They may also know how to use some features because they have a previous VR experience.

In the case of testing with new VR users, you would need to carry your target device or invite people to your office, or cafe, or co-working place in order to conduct testing. On the other hand, they will have a fresh view on your app and most likely their opinion will be unbiased.

Where to Find the Right Testers?

The next challenge is where to find all these people who desire to playtest your game? Well, it might be not so easy, but you can try recruiting people on Reddit or some Facebook groups and events.

You can make a post about an app you want to get tested and list all your criteria including:


Read the group rules to ensure you acknowledge them. Members who are interested in helping you out will comment and share their feedback. Participation is voluntary and results purely rely on your expertise with user testing. The downside is that you do not always know what the response rate will be and if you will be able to complete testing on time, as well as it might not be consistent among all testers since there won’t be any methodology involved. All the members might be absolutely brilliant candidates in various fields but talking to the wrong target audience may bring the wrong results.

You can also attend non-VR related expos where it will be suitable to showcase your VR app, partner with a retail shop, school, or University.

It’s also important to consider the context and User Journey - why and where would people want to use your VR software? Is it intended for education, productivity, or for leisure and entertainment? Where will people want to use it? Is it at home or an arcade experience? Single or multiplayer? User Journey represents the full end-to-end experience including the emotional part, motivations, actions, interactions, touchpoints, emotions, and feelings. What would trigger potential users to engage with your application?

Another detail is that the way of finding testers may be affected by the type of VR build itself as well as the development stage. WebVR can be more shareable, since there is no need to download, and you could probably get more responses because it’s very accessible. Or if your build is already in one of the channels but not released yet, you c may want to share it with a selected number of people. If your app is already launched, sharing keys or store links in specific groups can be a solution, but unfortunately, it’s rarely possible to ensure that keys are used properly and won’t be stolen.

Methods and Tools

There are a lot of usability methods and techniques, but the following are the most useful and widely applicable:


When selecting a test type, it is crucial to select wisely based on:


For example, for VR Analytics, having a certain number of users is essential because it won’t make any sense otherwise if an app has too few players yet.



VR Checklist - VR UX Tool

It’s a VR software evaluation method based on compliance with certain VR principles (the “heuristics”). The heuristics should be selected carefully based on the app specifics, they should be applicable to your VR experience. It’s a great tool to use at a stage before testing with real users, before actually talking to people from your Target Audience. You can evaluate your app yourself, together with your team, and preferably with these who work in the field your product is in (e.g., medicine, education), experts that were mentioned earlier.

It’s a checklist that helps to make sure that the major VR principles are followed. It’s not a set of concrete rules, but a flexible guide which everyone can apply. So it’s more of thought-provoking material to help everyone check if everything was considered. It’s quick and inexpensive feedback. And you can obtain it early in the design process.

Guerrilla Testing

Guerrilla user testing is usually a relatively simple and low-cost method that involves a facilitator approaching people and asking to participate in a test, showing them an app, and asking questions. It’s usually very casual, and participants can be found practically anywhere from cafe to an expo - wherever you can locate your Target Audience. And you can also select a location based on the User Journey.

Definitely, for a VR application to be tested with real users, it should be a 3D VR prototype with some more or less defined environments and objects. It needs to have the main interactions, sound, some visual design but doesn’t have to be polished yet.

Attending various VR and non-VR expos and events in order to recruit the testers and specifically the first-time VR users can be helpful. It allows to test in a casual setting, and to observe them, observations play a really big role in VR testing. Post-test interviews and questionnaires are very helpful as well.



Guerrilla testing is perfectly suited to:

In-person or Remote Study

Probably the most productive and the most challenging user testing method. To get the best results, the test should be carefully planned.

The test plan should contain all the information such as:


How to conduct it: For in-person user testing, you need to bring the testers into your office or wherever your VR setup is, physically, which adds some serious time and maybe even budget constraints.

For in-person testing you would need a full setting, an office or a room with a set up that includes:


Since in VR we can not directly study the participants’ view, we should record what they see. You can use Oculus Mirror or VR View if you are on Steam, to mirror the VR experience from headset onto the desktop and to record it using OBS, there is also an Open VR plugin for OBS to record an eye of choice and adjust settings. One of the cases, when it makes sense, are games or experiences when you have to aim and want to test right-and left-handed people.

Another option can be built-in video recording and even live streaming which as well should involve in-VR recording for future evaluations. These options are not always useful due to IP considerations. Video recording itself is the whole other topic which we will not cover fully here but there are different options and setups you can use to conduct VR remote user testing.

It’s often recommended to practice a think-aloud protocol. But the cases can vary. Sometimes it’s better to let players relax and just play as they would by themselves while observing them and taking notes. It’s important to not distract testers from the actual test which may happen if they try too hard to not forget to speak aloud instead of just playing the game. Consider all pros and cons and choose wisely.

Always test in the environment and setting that would be natural for your specific software. E.g., if it’s a game it would be most likely played at home, but if it’s a medical training app, it might be used at a University or even Hospital. It can be standing, seated, or room-scale experience. It all will affect the guardian set up, playground areas, and dimensions.

Record both, the gameplay and all actions the user takes in real life, how they move, what they say. It’s really helpful to see the actual players and where they struggle, what they like and don’t like, when they get tired or feel uncomfortable. You can also include Behavioral and Attitudinal tests - what ppl do vs what they say. E.g., people may say they are fine but by observing them becoming red or pale, or sweating, you can make certain guesses. By observing players’ behavior, you can tell if their feedback is honest. Sometimes people tell the opposite of what they really feel or think because they don’t want to be seen in certain conditions or be judged.

One of the biggest measures of success is that experience is comfortable and easy to navigate, ensuring that a player is able to easily find out what to do inside of VR. Testing learnability is essential.

You may also collect data captured by a wearable device (e.g., Apple Watch, Samsung Gear Fit, Oura ring) to record vitals like heart rate, blood pressure if needed. It might be useful if you are creating a meditation app, so you can measure if the person is relaxed or stressed.

Before conducting the test don’t forget to gather and document the demographics, as well as info about the hardware set up of each tester such as HMD, GPU, CPU for the remote studies. Of course, we are talking about more or less experienced user testers in this case.

If you are testing with new users, it’s important to make sure the issue they encountered is not caused by the lack of knowledge about how to use controllers. Make sure you explain it before you test, so you can see that the users stuck because of in-game settings and behaviors, but not because they don’t know how to push the button inside of VR or how to grab an object. Don’t test the controllers, test your app.

Facilitator

Not only the logistics of the whole test should be planned but also it’s facilitation, and moderation process if applicable. Create a script with a step-by-step plan and make sure all testers have the same experience, same tasks, spend the same amount of time playtesting and get the same questionnaire. Consistency is really important for user testing.


During in-person tests follow these recommendations:

Have a script (at least in your head) and be prepared!



Note:

Documents

Another consideration when testing is to prepare the documents needed for tests such as Tasks. People will forget the task as soon as they put a headset on. So spend some time writing the tasks to make sure they are short, clear, and not biased - the words are chosen carefully.

Don’t forget to ask the participants to sign the consent form before the session, and maybe other documents relative to health disclaimers, or NDA, if needed.


Documents to consider:

And you are all set!

VR Analytics and Heatmaps

Another method you can use is Analytics. Among the tools to measure performance and player engagements are Unity and YouTube, and they both have VR Heatmaps. For YouTube, the video must have 1000 views to enable it, and be 360. So it might be useful to publish some VR scenes on YouTube to receive more statistics. At the time of writing of this material, Unity required to have at least Plus license.

You can measure player engagement, where did people look at, how frequently they change position, and use the interaction methods. Discover the most viewed areas of the experience and how long people are looking at a specific part of the video. This method is more suitable for launched applications that already have a user base.

Test report

Once you’ve done testing, don’t forget to document all your findings, even if it’s not what you were actually looking for. If you discover something that you were not actually looking for during your tests and found out a completely new discovery - get happy about it, every new discovery is a win!

Document all the Research Stages to compare and evaluate later. Documents in any medium you’d like - write, create a gif or a video.

You can also create a UX Report in the end when the analysis is made if you want everything to be accurate and organized.

Professional VR Testing Services

Professional services that specialize in testing would be the most efficient way if you are looking for serious measurable results like sales increase, better reviews, higher user retention, discovering new features to build, and so on. Depending on project goals and complexity one testing round may take from 1 day to 3 weeks.

The deliverables at VR Oxygen are usually:
Comprehensive test report, including testers’ profiles, tasks, playtesting videos, questionnaire results. Feel free to contact us and learn more.

Analysis and Evaluation

After the tests are done it’s time to study the findings and make some hypotheses. You can create affinity maps or diagrams to help you sort out the discoveries, so you can see if patterns appear.

For example, who and why liked or disliked something. Or connect why they disliked based on what you’ve learned about these participants.

It’s always useful to evaluate app performance (Load Time, Frame Rate, and if there are any drops, lags, bugs), comfort, 3D Art, UI and navigation, gameplay, and game mechanics as well.

When doing tests you can ask test participants to rate the app, or rate some features of it. Later you can find out The Net Promoter Score. it’s an index ranging from -100 to 100 that measures the willingness of customers to recommend a company’s products or services to others. It is used to estimate the customer’s overall satisfaction with a company’s product or service and the customer’s loyalty to the brand.

You can rate different components of an app and usability objectives from 1-5 or 1-10 scale depending on what you rate and its complexity. The overall app rating should be from a 1-5 scale as it usually is (like in App Store, Google Play, Oculus Store, Steam, etc.).

Finding both, what people like or dislike is beneficial, because it helps to iterate and move in the right direction.

Findings

Let’s check some of the findings now. These are from VR Oxygen’s research a while ago when the goal was to find out what people do when placed in VR, specifically, first-time users. How they use certain features, their feelings, and thoughts.

People don’t turn around/ look around for cues.

At several expos, VR Oxygen demonstrated Cradboard VR app which was selected as one of the favorites at that time, because of its beautiful graphics. Some of the discoveries were surprising at that time.

Even when people were told that it’s a 360 experience where they can move around and look around, they wouldn’t. They would stay still and wait, some would turn head very slightly to the sides, carefully. And it was because they were not used to it. New Users also didn’t look down, even a little, but looked in front of them.

In this situation the Hypothesis is obvious:

Creating clear visual and audio cues at the very beginning and positioning the UI and important elements in front of the viewer will explain players they can interact with the environment, will guide them. If a scene is engaging, with clear directional cues, more likely people will turn around and explore the full 360. Getting people to understand within the first 5 seconds that they can move around will help to keep them engaged.

People assumed they could perform any gestures, tried to manipulate all the objects.

Some people did that or something similar after encountering that items are interactive in VR, so they tried to manipulate all the objects that were present in the environment. The others assumed that if it’s VR they would see their arms there. They tried to grab things.

In this situation the Hypothesis is obvious:

All the objects should be interactive and useful, behaviors - easy to understand in order to support expectations. Creating interactions based on real-life affordances and people’s mental models will provide a smoother learning curve for New Users and may help to predict behaviors.

Tips

Usability testing is far too important to be neglected or postponed, it’s really important to start testing early, iterate, and test again until the best solution is found.

*Find VR Checklist (Heuristic Evaluation Tool/ VR UX Tool) at vroxygen.com. Join VR AR Testing community. More materials about VR Design and development can be found on our YouTube channel. Follow and join us on Twitter, Instagram, LinkedIn, Twitch.

Stay tuned!

Back to Top