VR Interaction Design — Level 3: Advanced

Abstract: This advanced course takes students to professional and exhibition-grade VR production. Over 8 modules, students master Unity XRI architecture, multi-user VR with Photon Fusion, GPU performance optimization, spatial AI, accessibility, and advanced evaluation (IPQ + in-session ESM). The capstone is an exhibition-ready VR experience tested with SUS + IPQ by ≥5 users. Outline: Unity XRI architecture → Multi-user VR → GPU optimization → Spatial AI → Accessibility → IPQ & ESM evaluation → Production readiness → Exhibition preparation.

Learning Objectives

By the end of this course, students will be able to:

1. Build (Bloom: creating) a Unity XRI interaction architecture (XR Origin, interactors, interactables, interaction manager)
2. Implement (creating) a real-time multi-user VR experience with Photon Fusion (NetworkObject, RPC, VoiceConnection)
3. Diagnose and optimize (analyzing/evaluating) VR GPU performance (draw calls, LOD, foveated rendering, Frame Debugger)
4. Integrate (creating) spatial AI components: scene understanding, LLM NPCs, generative content
5. Audit (evaluating) the accessibility of a VR experience using a structured checklist
6. Evaluate (evaluating) presence and immersion with IPQ (Schubert 2001) and ESM in-session questions
7. Produce (creating) an exhibition-ready experience with operator and mediation documentation

Module 1 — Unity XRI Architecture (2.5h)

Concept

Unity XR Interaction Toolkit (XRI) is the standard Unity framework for VR/AR interactions. It separates responsibilities between three components:

• XR Interactor (active — the hands, the ray): the component that performs the interaction
• XR Interactable (passive — the object): the component that can be interacted with
• XR Interaction Manager (orchestrator): intermediary that matches interactions

Complete XRI hierarchy:

XR Origin (Camera rig) ├── Camera Offset │ └── Main Camera (head tracking) ├── Left Controller │ ├── XR Ray Interactor (far interaction) │ ├── XR Direct Interactor (near interaction) │ └── XR Interactor Line Visual └── Right Controller ├── XR Ray Interactor ├── XR Direct Interactor └── XR Interactor Line Visual

XR Interactable types:

Grab configuration:

• Movement Type: Kinematic (stable, no physics) or Tracked (realistic physics)
• Select Mode: Single or Multiple (multiple grabs)
• Interaction Layer Mask: filter which interactors can grab this object

Lab 1.1 — Complete XRI Scene

Steps:

1. Create a new Unity project (URP, Unity 2022.x LTS)
2. Import XR Interaction Toolkit (Package Manager → Unity Registry)
3. Add an XR Origin (Room Scale) to the scene
4. Configure Left and Right Controllers with Ray Interactor + Direct Interactor
5. Create 5 objects: 3 Grab Interactables + 1 Simple Interactable (button) + 1 Socket Interactor
6. Test in Play mode with the XR Device Simulator (no headset required)

Mini-exercise: Add a UI button in world space (3D Canvas, World Space) that changes the color of all objects when clicked. Use Unity Events on XR Simple Interactable.

Module 2 — Multi-User VR with Photon Fusion (3h)

Concept

Multi-user VR requires networked replication of position, orientation, and state in real time.

Photon Fusion (PUN2 successor) is the most widely used Unity SDK for networked VR. Key concepts:

• NetworkRunner: the main Fusion component — manages connection and session
• NetworkObject: a GameObject whose state is synchronized over the network
• NetworkTransform: automatically synchronizes position/rotation
• NetworkBehaviour: base class for networked scripts (replaces MonoBehaviour)
• RPC (Remote Procedure Call): method executed on all clients
• VoiceConnection (Photon Voice): integrated spatialized voice

PhotonAppSettings configuration:

1. Create Fusion AppId at https://dashboard.photonengine.com/
2. In Unity: Photon → Realtime → PhotonServerSettings → paste AppId
3. Create a GameObject with NetworkRunner (Add Peer Mode: Host)
4. Add NetworkObject on each object to synchronize

Grabbable shared object (example): `csharp using Fusion; using UnityEngine; using UnityEngine.XR.Interaction.Toolkit;

public class SharedObject : NetworkBehaviour { [Networked] public bool IsGrabbed { get; set; } [Networked] public NetworkId GrabbedByPlayer { get; set; }

private XRGrabInteractable _grab;

public override void Spawned() { _grab = GetComponent<XRGrabInteractable>(); _grab.selectEntered.AddListener(OnGrabbed); _grab.selectExited.AddListener(OnReleased); }

private void OnGrabbed(SelectEnterEventArgs args) { RPC_Grab(true, Runner.LocalPlayer); }

private void OnReleased(SelectExitEventArgs args) { RPC_Grab(false, default); }

[Rpc(RpcSources.InputAuthority, RpcTargets.All)] public void RPC_Grab(bool grabbed, PlayerRef player) { IsGrabbed = grabbed; GrabbedByPlayer = player.IsValid ? Runner.GetPlayerObject(player).Id : default; } } `

Multi-user avatar: each player needs a prefab with NetworkObject + NetworkTransform on the head and both controllers. The local avatar must be invisible to the local player (deactivate its MeshRenderers), but visible to others.

Lab 2.1 — Multi-User Session

Steps:

1. Configure Photon Fusion (AppId, NetworkRunner)
2. Create a shared room with 4 grabbable objects
3. Test with two instances of Unity (or second computer on the same network)
4. Verify: do objects synchronize correctly? Any latency visible?

Mini-exercise: Implement an RPC that makes an object change color when a player grabs it, and returns to its original color when released.

Module 3 — GPU Optimization for VR (2.5h)

Concept

VR requires maintaining a stable framerate at 90 fps or higher to avoid cybersickness. Unlike desktop games, there is no room for compromise.

Performance targets for Quest 3 (integrated GPU):

Optimization tools:

Optimization techniques:

LOD Group tree example:

MyObject (LOD Group) ├── LOD0 — 5,000 polygons (distance 0–10m) ├── LOD1 — 1,500 polygons (distance 10–25m) ├── LOD2 — 300 polygons (distance 25–50m) └── LOD3 — Culled (distance > 50m)

Lab 3.1 — Performance Profiling

Steps:

1. Open a VR project in Unity with ≥ 5 objects in the scene
2. Open the Frame Debugger: identify the 3 most expensive draw calls
3. Apply Static Batching on all static objects
4. Enable GPU Instancing on the most repeated material
5. Compare draw calls before/after: save a screenshot of each state
6. Add a LOD Group to the most complex object (3 LOD levels)

Mini-exercise: Your VR experience has 180 draw calls. After Static Batching you have 110. After GPU Instancing you have 75. Is that enough for Quest 3? What other optimization would you apply next?

Module 4 — Spatial AI (2.5h)

Concept

Spatial AI combines machine learning and spatial understanding to create VR environments that adapt to the real physical space and incorporate intelligent virtual agents.

Three pillars of spatial AI in VR:

1. Scene understanding (understanding the physical space)

• Meta Quest 3: plane detection, mesh generation, semantic understanding of real objects
• Use: virtual furniture adapting to the real room, digital twins, occupancy detection

1. AI NPCs (Non-Player Characters with LLM)

• LLM (GPT-4, Claude) + speech-to-text (Whisper) + text-to-speech + animation = VR character you can hold a natural conversation with
• Framework: Unity Sentis (local inference), or API REST (OpenAI, Anthropic) — latency to account for

1. Generative content

• Meshy.ai: 3D object generation from text description
• Shap-E (OpenAI): text or image → 3D mesh
• Skybox AI (Blockade Labs): 360° equirectangular panorama generation → <a-sky> in A-Frame

Quest 3 scene understanding (Meta SDK): `csharp using Meta.XR.MRUtilityKit;

public class SceneManager : MonoBehaviour { [SerializeField] private MRUK mruk;

private void Start() { mruk.LoadSceneFromDevice(); mruk.RoomCreatedEvent.AddListener(OnRoomLoaded); }

private void OnRoomLoaded(MRUKRoom room) { // Get all detected planes (walls, floor, ceiling) var planes = room.GetRoomObjects(MRUKAnchor.SceneLabels.WALL_FACE); foreach (var plane in planes) { Debug.Log($"Wall detected: {plane.transform.position}"); } } } `

Lab 4.1 — AI NPC Prototype

Steps:

1. Create a Unity VR scene with a character (Humanoid + Animator)
2. Add a microphone component (Unity Microphone API)
3. Connect to OpenAI Whisper API for transcription
4. Connect to OpenAI GPT-3.5 or Claude API for response (role: "you are a guide in this VR museum")
5. Use a text-to-speech API (ElevenLabs or Azure) to vocalize the response
6. Test with 3 different questions — does the NPC remain coherent?

Mini-exercise: What are the risks of an AI NPC in a public VR experience? Cite at least 3 (hallucinations, inappropriate content, bias) and a concrete mitigation for each.

Module 5 — VR Accessibility (2.5h)

Concept

Accessibility in VR is more critical than in any other medium because immersion can heighten the experience of exclusion for users who are not accommodated.

4 accessibility categories in VR:

1. Universal Comfort

• [ ] Multiple locomotion options (teleportation mandatory)
• [ ] Comfort mode for very sensitive users (snap turn, vignette)
• [ ] Maximum session duration indicated before entry
• [ ] Exit/pause accessible at all times

2. Visual Accessibility

• [ ] Minimum text contrast 4.5:1 (WCAG AA)
• [ ] No information conveyed by color alone
• [ ] Sufficient text size (>0.1m at 1m distance)
• [ ] Colorblind mode (deuteranopia/protanopia/tritanopia)
• [ ] Adjustable font (size, weight)

3. Auditory Accessibility

• [ ] Subtitles for all voice content
• [ ] Text alternatives for interaction sounds
• [ ] Adjustable volume controls
• [ ] No audio-only critical information

4. Motor Accessibility

• [ ] No interaction requiring simultaneous two-handed manipulation
• [ ] Dwell gaze as an alternative to trigger
• [ ] Customizable button mapping
• [ ] Sufficient physical rest time between interactions

Tools and references:

• XR Accessibility User Requirements (W3C): https://www.w3.org/TR/xaur/
• Meta Accessibility Overview: https://developer.oculus.com/resources/design-accessible-vr-apps/
• WCAG 2.1 (AA for text): https://www.w3.org/TR/WCAG21/

Lab 5.1 — Accessibility Audit

Steps:

1. Apply the checklist to your existing VR project (all 4 categories)
2. For each unchecked item: classify P0 / P1 / P2
3. Implement all P0s
4. Optional: test with a person having a visual or motor disability

Mini-exercise: Redesign the locomotion menu of your capstone project so it is accessible to a user with limited fine motor skills in one hand.

Module 6 — IPQ and ESM Evaluation (2.5h)

Concept

Level 3 evaluation goes beyond SUS to specifically measure presence and in-session satisfaction.

IPQ (Igroup Presence Questionnaire)

The IPQ (Schubert et al., 2001) measures the sense of presence in VR across 4 subscales:

Source: Schubert, T., Friedmann, F., & Regenbrecht, H. (2001). The experience of presence: Factor analytic insights. Presence, 10(3), 266–281. https://doi.org/10.1162/105474601300343603 Online tool: http://www.igroup.org/pq/ipq/index.php

IPQ scoring:

• Each item: 7-point Likert scale (−3 to +3)
• Calculate the mean per subscale
• Compare to normed values from the tool's reference data

ESM (Experience Sampling Method) — In-Session Adaptation

The ESM is a research method for collecting feedback during the experience (not just after). In VR, it consists of 3 discreet, brief in-session questions displayed at predefined moments:

1. "How present do you feel in this environment right now?" (1–7)
2. "How engaged are you with this experience?" (1–7)
3. "How comfortable do you feel physically?" (1–7)

Implementation: a UI panel appears at T+2 min, T+7 min, T+15 min via a Unity coroutine. The user responds with 3 buttons (1, 2, 3... 7) or a slider. Responses are logged with timestamp.

Combined reading:

• SUS (post-session) → global usability
• IPQ (post-session) → presence quality
• ESM (in-session) → temporal dynamics (peak moment, comfort breakdown, etc.)

Lab 6.1 — IPQ Test

Steps:

1. Prepare an online IPQ form (Google Forms or paper) from: http://www.igroup.org/pq/ipq/index.php
2. Test with ≥ 3 participants (after SUS if you have time)
3. Calculate the mean per subscale (GP, SP, INV, REAL)
4. Identify the subscale with the lowest score → what does it tell you about the experience?

Mini-exercise: Your experience scores SP=2.1, INV=1.8, REAL=0.9, GP=1.4. The low REAL score — what does this indicate and what design action could improve it?

Module 7 — Production Readiness (2.5h)

Concept

An exhibition-level VR experience is not only one that works — it must work reliably over 6–8 hours per day for several days, be installable in under 30 minutes, and be operable by a non-technical person.

Production readiness checklist:

Troubleshooting guide template:

Lab 7.1 — Production Readiness Audit

Steps:

1. Apply the production readiness checklist to your current project
2. Identify the 3 most critical missing items
3. Write a 2-page operator guide (setup + daily use + troubleshooting)
4. Test with a volunteer who has never seen your project: can they set it up from the guide alone?

Mini-exercise: Your exhibition opens in 3 days. The technical check shows: framerate stable, no crash, but the operator guide doesn't exist and the APK file is not named correctly. Prioritize and distribute tasks in 3 days.

Module 8 — Exhibition Preparation (2.5h)

Concept

A VR exhibition experience differs from a consumer VR game or a school project on multiple dimensions.

Three required exhibition documents:

1. Technical File (4–6 pages)

• Hardware list (headset + cables + PC/Mac if needed, power supply)
• Software (application name, version, APK or executable format)
• Space plan (minimum 3×3m for standalone headset; diagram of equipment)
• Installation procedure step by step (numbered, with screenshots)
• Restart procedure (daily, with and without reboot)
• Troubleshooting guide (min. 5 common problems)
• Contacts (technical responsible: phone + email)

2. Artistic File (2–4 pages)

• Project title, subtitle, short description (30 words)
• Artistic note (~300 words): concept, references, context
• Artist/collective biography (100 words)
• Technical description (accessible to the general public)
• High-resolution images/screenshots (min. 300dpi, 3 images)
• Credits (programming, sound, 3D assets, libraries used)

3. Mediation Protocol (1–2 pages)

• Introduction (30s–1 min, what to say before putting on the headset)
• Safety instructions (Guardian, avoid moving too fast)
• Key interaction instructions (3 things to know to start)
• De-briefing (3 open questions to ask after removing the headset)
• Rotation protocol (multiple users: cleaning headset, adjustment)

Lab 8.1 — Complete Exhibition Documentation

Steps:

1. Draft the technical file for your capstone project (minimum: hardware, installation, 3 troubleshooting items)
2. Draft the artistic file (concept 300 words + brief biography)
3. Write the mediation protocol (30s intro + 3 key instructions + 3 debrief questions)
4. Test the mediation protocol with a colleague playing "mediator" with a non-VR user

Mini-exercise: Adapt your mediation protocol to accommodate a blind user and a deaf user. What changes?

Capstone Project — "Exhibition-Level VR Experience"

Brief

Create a VR experience that can be presented in an exhibition, festival, or school open day context, with real user testing (SUS + IPQ ≥5 participants) and complete exhibition documentation.

Constraints

• Technology: Unity XRI or WebXR Three.js (your choice)
• At least 3 interaction patterns (proximal grab, ray casting, teleportation + optional)
• At least 5 testers: SUS score + IPQ score + ESM (optional) + P0/P1/P2 prioritization
• Complete documentation: technical file + artistic file + mediation protocol
• Operator README: enough for a non-technical person to run the experience for a day

Deliverables

1. Application (APK Quest, WebXR link, or executable)
2. Technical file (PDF, 4–6 pages)
3. Artistic file (PDF, 2–4 pages)
4. Mediation protocol (PDF, 1–2 pages)
5. Evaluation report: SUS (mean + per tester) + IPQ (mean per subscale) + corrections applied
6. Oral presentation (12 min): experience demo + evaluation method + design lessons + production surprises

Evaluation Rubric

Readings & Resources

• Schubert, T. et al. (2001). The experience of presence: Factor analytic insights. Presence, 10(3), 266–281.
• IPQ online tool: http://www.igroup.org/pq/ipq/index.php
• XR Accessibility User Requirements (W3C): https://www.w3.org/TR/xaur/
• Unity XR Interaction Toolkit docs: https://docs.unity3d.com/Packages/[email protected]/
• Photon Fusion docs: https://doc.photonengine.com/fusion/current/
• Meta Quest Developer Hub: https://developer.oculus.com/downloads/package/meta-quest-developer-hub/
• Boletsis, C., & Cedergren, J. E. (2019). VR Locomotion. https://doi.org/10.1155/2019/7420781
• Yee, N., & Bailenson, J. (2007). The Proteus Effect. https://doi.org/10.1111/j.1468-2958.2007.00299.x

→ Previous level: VR Interaction Design — Level 2 → VR Bible: Complete Reference → Custom consulting for your exhibition/institution: Book a 30-min call — free