During lifecycle, any application goes through many phases of iterations and test rounds. Here is an example of the steps to a test conducted in February 2020.

Process:

  • Received client’s requests, defined project goals.
  • Screened test participants, selected testers and created profiles.
  • Created a test plan and tasks.
  • Signed release form (consent form) and NDAs with testers where applicable.
  • Demo test completion, participants’ set up.
  • Playtesting.
  • Questionnaire.
  • Deliverables (vary by project, may include some or all of the following: testers' profiles, tasks, playtest videos (in-VR and in-real-life video recordings), survey results, test analysis report). We work with you to define your deliverables.

Overview:

Masterpiece Studio has two applications, Masterpiece Creator and Masterpiece Motion, that focus on different parts of the creative process in making 3D content for Games & Animation.

Masterpiece Creator is a sculpting program that allows anyone to easily and quickly create 3D content like characters, vehicles, objects, etc. This content can then be used for many purposes, like being part of a game, being in an animated story, being 3D printed, or converted to a 2D illustration.

Goals:

Bring clarity to UI and make it more user-friendly. Determine if the UI and functionality are intuitive, including onboarding and feedback to the user.

Avoid losing users and discover better ways to guide them Discover where users get lost, confused, or frustrated, to avoid drop-offs. To establish how to properly guide the users and avoid losing them.

Understand what new users are trying to achieve and which features are more accessible and intuitive than others.

Devices:
(any of listed)

  • Oculus Rift
  • Rift S
  • HTC Vive
  • Vive Pro
  • Vive Cosmos
  • Valve Index
  • Microsoft Mixed Reality

Metrics:

  • Intuitiveness of functionality
  • Discoverability, UI effectiveness
  • Engagement, level of immersion
  • Potential drop-offs (critical errors)
  • Non-critical errors
  • Satisfaction
  • Task completion rates

Participants:

  • Professional 2D Concept Artists
  • Animators
  • Storytellers
  • Illustrators
  • Creative professionals with little to advanced 3D modeling experience

Methodology and logistics:

Example of a test: Speak aloud (“commentary on”)

Number of participants: 5

Participant eligibility: via Screener.

Location: remote.

Software, tools, and guidelines: provided by VR Oxygen

Training: Demo test completed.

Participant preparation time: 30-60 min.

Test session time per participant: ~1.5 – 2 hr

Tasks:

  • Explore Masterpiece Creator app speaking aloud and explaining what you see, what you are currently doing, and why. (general walkthrough)
  • Use Masterpiece Creator to model an object for your current, past, or upcoming project.
  • Use Masterpiece Creator to model a tree.
  • Use Masterpiece Creator to model a vehicle.

Task duration: 20-30 minutes each.

Total Test duration: 80-120 min.

Test Results Summary:

  • MENU: 5 of 5 participants were confused by the layout, tool labeling, and grouping.
  • TOOLS: 5 of 5 participants had the same feature requests. Liked the same tools. Wanted to have tutorials.
  • BUGS: 3 of 5 participants had bugs and technical problems that prevented them from completing the task.

Key Recommendations:

  • Evaluate the Target Audience and iterate on UI, layout, and tools accordingly.
  • Allow to draw freely everywhere, remove bounding box canvas from the default state, make it selective.
  • Create short tutorials or tool explanations.
  • Menu overall: Improve by making more clear labeling, better layout – make buttons more spaced out, and/or grouping buttons and tools in a better more intuitive way.

Participants’ Quotes:

  • ❤ “Mesh creating itself feels amazing. Mesh works and is pretty cool.”
  • ❤ “Water is really cool. Looks like gel”.
  • ❤ “I liked it a lot” (about using Masterpiece Creator in general)
  • ☹ “I’ve got into the state when I can’t draw… Accidentally resumed working. How?”
  • ☹ “Why did it place it there? And why did it switch to the Mesh layer?” (tools, default canvas shape for Clay)
  • ☹ “It only erases one half of what I draw – it doesn’t work for how I am used to draw. Or can I choose how it erases? If I am drawing something with a large amount of detail.” (Mirror and Erase)

Test Report:

Please contact us if you'd like to learn more about our methodology, process, get a full test report sample, or if you have any questions/suggestions.