Probably the most productive and the most challenging user testing method. To get the best results, the test should be carefully planned.
The test plan should contain all the information such as:
- Testers’ criteria, recruitment, and screening plan.
- Stage of your product and milestones with deadlines.
- What you are testing and why, metrics.
- Specific outcomes you are looking to get (e.g., why players drop off on level 2).
- The methodology you are going to use.
- Scenarios and tasks.
- Questionnaires and when during the course of the test they would be implemented.
- Error and severity assessment plan.
- Logistics.
- Documentation like consent forms and NDA.
How to conduct it: For in-person user testing, you need to bring the testers into your office or wherever your VR setup is, physically, which adds some serious time and maybe even budget constraints.
For in-person testing you would need a full setting, an office or a room with a set up that includes:
- An HMD you are testing with (Oculus Rift, Quest, HTC Vive, etc.)
- Camera for external recording or your phone with a tripod to film the participants, their behaviors, movements, and reactions, as well as sounds and comments.
Since in VR we can not directly study the participants’ view, we should record what they see. You can use Oculus Mirror or VR View if you are on Steam, to mirror the VR experience from headset onto the desktop and to record it using OBS, there is also an Open VR plugin for OBS to record an eye of choice and adjust settings. One of the cases, when it makes sense, are games or experiences when you have to aim and want to test right-and left-handed people.
Another option can be built-in video recording and even live streaming which as well should involve in-VR recording for future evaluations. These options are not always useful due to IP considerations. Video recording itself is the whole other topic which we will not cover fully here but there are different options and setups you can use to conduct VR remote user testing.
It’s often recommended to practice a think-aloud protocol. But the cases can vary. Sometimes it’s better to let players relax and just play as they would by themselves while observing them and taking notes.
It’s important to not distract testers from the actual test which may happen if they try too hard to not forget to speak aloud instead of just playing the game. Consider all pros and cons and choose wisely.
Always test in the environment and setting that would be natural for your specific software. E.g., if it’s a game it would be most likely played at home, but if it’s a medical training app, it might be used at a University or even Hospital. It can be standing, seated, or room-scale experience. It all will affect the guardian set up, playground areas, and dimensions.
Record both, the gameplay and all actions the user takes in real life, how they move, what they say. It’s really helpful to see the actual players and where they struggle, what they like and don’t like, when they get tired or feel uncomfortable. You can also include Behavioral and Attitudinal tests - what ppl do vs what they say. E.g., people may say they are fine but by observing them becoming red or pale, or sweating, you can make certain guesses. By observing players’ behavior, you can tell if their feedback is honest. Sometimes people tell the opposite of what they really feel or think because they don’t want to be seen in certain conditions or be judged.
One of the biggest measures of success is that experience is comfortable and easy to navigate, ensuring that a player is able to easily find out what to do inside of VR. Testing learnability is essential.
You may also collect data captured by a wearable device (e.g., Apple Watch, Samsung Gear Fit, Oura ring) to record vitals like heart rate, blood pressure if needed. It might be useful if you are creating a meditation app, so you can measure if the person is relaxed or stressed.
Before conducting the test don’t forget to gather and document the demographics, as well as info about the hardware set up of each tester such as HMD, GPU, CPU for the remote studies. Of course, we are talking about more or less experienced user testers in this case.
If you are testing with new users, it’s important to make sure the issue they encountered is not caused by the lack of knowledge about how to use controllers. Make sure you explain it before you test, so you can see that the users stuck because of in-game settings and behaviors, but not because they don’t know how to push the button inside of VR or how to grab an object. Don’t test the controllers, test your app.