AR Interaction Design — Level 3: Advanced

Abstract: This advanced course targets production-level AR experience design. Students build complex multi-user AR systems using Unity AR Foundation, integrate cloud anchors for persistent shared AR, apply spatial AI for scene understanding, optimize for performance, and complete an exhibition-ready project with full documentation. Outline: Unity AR Foundation architecture → Cloud anchors & persistence → Multi-user AR → Performance optimization → Spatial AI & generative AR → Accessibility & ethics at scale → Research methods → Exhibition project.

Learning Objectives

By the end of this course, students will be able to:

1. Architect (creating) a multi-user AR experience with persistence via cloud anchors.
2. Optimize (applying) the performance of a Unity AR scene (draw calls, texture compression, occlusion culling).
3. Integrate (applying) spatial AI capabilities (object detection, semantic segmentation) into an AR experience.
4. Design (creating) for accessibility and ethics at the scale of a large, diverse audience.
5. Conduct (evaluating) a rigorous AR evaluation (usability + presence + engagement) and publish the results.
6. Produce (creating) an exhibition-level AR experience: deployable, documented, and maintainable.

Module 1 — Unity AR Foundation Architecture (2.5h)

Concept

Unity AR Foundation is the Unity abstraction layer that unifies ARKit (Apple) and ARCore (Google) under a single API. One project → iOS + Android.

Documentation: https://docs.unity3d.com/Packages/com.unity.xr.arfoundation@latest

Key components:

• ARSession: session lifecycle management
• ARCameraManager: camera access + feature configuration
• ARPlaneManager: flat surface detection
• ARAnchorManager: anchor creation and management
• ARRaycastManager: raycasting against detected planes
• ARMeshManager: space mesh generation (LiDAR)

Recommended architecture (separation of concerns): ARController (session management) ├── PlacementController (tap-to-place, raycasting) ├── AnchorController (creation, persistence, restoration) ├── UIController (UI states according to tracking state) └── ContentController (AR prefab instantiation and management)

Lab 1.1 — Tap-to-Place in Unity

Steps:

1. Create a Unity project (Unity 2022 LTS recommended) + import AR Foundation
2. Configure the scene: ARSession Origin, AR Camera, ARPlaneManager, ARRaycastManager
3. Write a PlacementController: tap → raycast → instantiate prefab at the hit position
4. Add a visual indicator (ring on the ground) that follows the placement cursor before confirmation
5. Build to iOS or Android device and test

Mini-exercise: Add a constraint: only one object can be placed at a time. If the user taps again, the existing object moves. Implement it.

Module 2 — Cloud Anchors and AR Persistence (2.5h)

Concept

Cloud anchors allow storing an anchor's position on a server and restoring it in a future session or on another device. This is the foundation of persistent and shared AR.

Technical options in 2026:

• ARCore Cloud Anchors (Google): upload of a local feature map → unique identifier → restoration by other apps

https://developers.google.com/ar/develop/cloud-anchors

• Azure Spatial Anchors (Microsoft): multiplatform (iOS, Android, HoloLens), fine-grained permission management

https://learn.microsoft.com/azure/spatial-anchors/

• Immersal SDK: high-precision cloud anchors, visual map of the location

https://immersal.com

Cloud anchor lifecycle:

1. Host scans the surface → local map → upload → receives anchor ID
2. Guest receives anchor ID → re-localizes in the same room → AR objects appear in the right place

Lab 2.1 — Persistent Anchor Between Two Sessions

Steps:

1. Configure ARCore Cloud Anchors in Unity AR Foundation
2. Implement: HOST mode (scan → upload → copy the ID) and RESOLVE mode (enter the ID → restore the anchor)
3. Test: session 1, place an object and note the ID. Close the app. Session 2, enter the ID → the object must reappear in the same place.
4. Test at distance in the same room with two phones

Mini-exercise: What happens if the surface has moved between sessions? Write a fallback UX (error message + re-placement option).

Module 3 — Multi-User AR (2.5h)

Concept

Shared multi-user AR allows multiple people to see and interact with the same virtual objects in the same physical space.

Typical architecture:

• Shared cloud anchor (same ID for all participants) → common spatial alignment
• Real-time backend for state synchronization (Firebase, Photon Fusion, Mirror Networking)
• Each client sends its actions → backend broadcasts to all → each one applies locally

Popular stack 2026:

• Unity AR Foundation + Photon Fusion (networking) + Firebase (persistence) → scalable solution
• Unity Netcode for GameObjects + cloud anchors → simpler, less scalable

Specific problems:

• Network latency: objects move differently on each device if sync is slow
• Authority: who controls an object if two users grab it simultaneously?
• Re-localization: each new participant must re-localize the space before seeing the objects

Lab 3.1 — Shared AR Session for 2

Steps:

1. Configure Photon Fusion (free account: https://www.photonengine.com) in a Unity AR Foundation project
2. Implement: Room creation/join + synchronized NetworkObject (position, rotation)
3. Test with 2 smartphones in the same room: do both see the same object?
4. Simulate a conflict: both try to move the object simultaneously

Mini-exercise: Design the multi-user AR signage: how does each user see the other's presence (avatar, cursor, annotation)? Paper prototype.

Module 4 — AR Performance (2.5h)

Concept

AR apps run on mobile with strict constraints: battery, thermal, memory. Excess polygons or draw calls overheat the phone and drain the battery in 15 minutes.

Target metrics (mobile AR):

• Framerate: constant 60 fps (ideal), never below 30 fps
• Draw calls: < 50 per frame (< 100 on recent devices)
• Tri-count: < 100k active triangles in the scene
• Texture memory: < 200 MB (varies by device)
• CPU/GPU split: < 16ms per frame total

Unity optimization tools:

• Unity Profiler: measure CPU/GPU/memory frame by frame
• Frame Debugger: see each draw call
• GPU Instancing: render copies of the same mesh in a single draw call
• Occlusion Culling: don't render what is hidden
• LOD (Level of Detail): high-resolution model up close, low-resolution from afar
• Texture compression: ASTC for iOS, ETC2 for Android

Lab 4.1 — Profiling and Optimization

Steps:

1. Take a Unity AR scene with 5+ objects
2. Build to device + open Unity Profiler (USB connection)
3. Measure: average framerate, peak draw calls, texture memory
4. Apply 2 optimizations (GPU instancing + LOD or texture compression)
5. Re-measure: documented performance gain

Mini-exercise: For your capstone project, establish a "performance budget": max number of active meshes, target texture resolution, guaranteed minimum framerate.

Module 5 — Spatial AI and Generative AR (2.5h)

Concept

Spatial AI allows AR to understand what it "sees" (not just detect flat surfaces) and adapt content accordingly.

Capabilities available in 2026:

Generative AR — emerging use: The user verbally describes an object → a 3D diffusion model generates the mesh → it appears in AR space. Experimental in 2026 but demonstrated by Luma AI, Block, Adobe.

Ethical considerations: Detecting people and facial recognition in public AR raises serious regulatory questions (GDPR article 9, EU AI Act 2024).

Lab 5.1 — Object Detection in an AR Scene

Steps:

1. Integrate ML Kit Object Detection in a Unity project (or native Swift/ARKit)

https://developers.google.com/ml-kit/vision/object-detection

1. At each frame, send the camera image to the detector
2. If an object of the "plant" category is detected → place an AR label on it ("Plant detected")
3. Test with 5 different objects: what detection rates?

Mini-exercise: Imagine 3 concrete use cases for generative AR in your application domain (cultural heritage, education, industry). Identify the ethical risks of each.

Module 6 — Accessibility and Ethics at Scale (2.5h)

Concept

An AR experience designed for a large audience must anticipate very diverse user profiles. Accessibility is not cosmetic — it is a design constraint from the very beginning.

AR accessibility checklist:

• [ ] All interactions have a non-visual alternative (sound, vibration)
• [ ] All interactions have an alternative without precise gestures (voice, dwell)
• [ ] Text contrast ≥ 4.5:1 (WCAG 2.1 AA) in all environments
• [ ] No blinking animation > 3 Hz (epilepsy risk, WCAG 2.3.1)
• [ ] Sessions limited in duration or with a pause option (visual fatigue)
• [ ] Instructions available in multiple languages
• [ ] Color-blind mode (no information communicated by color alone)

GDPR in AR:

• Any space capture with cloud anchors = mapping data → consent required if people or private places are included
• Biometric data (eye tracking, face tracking) = sensitive GDPR category → processing prohibited without explicit consent

Lab 6.1 — Accessibility Audit

Steps:

1. Take your Level 2 prototype or capstone project
2. Go through the checklist above: how many boxes are checked?
3. Identify 2 points to improve
4. Implement at least 1 improvement (non-visual alternative OR color-blind mode)
5. Re-test with a user who has never seen the app

Module 7 — Advanced Research Methods (2.5h)

Concept

For complex or published AR projects, more rigorous research methods validate design choices.

Methods suited to AR:

Witmer, B. G., & Singer, M. J. (1998). Measuring Presence in Virtual Environments: A Presence Questionnaire. Presence, 7(3), 225–240. https://doi.org/10.1162/105474698565686

Lab 7.1 — Evaluation with Presence Questionnaire

Steps:

1. Download the PQ or IPQ (freely available: http://www.igroup.org/pq/ipq/index.php)
2. Test your prototype with 3 people
3. Administer the IPQ after each session
4. Calculate average scores on the 3 subscales (general presence, spatial presence, involvement)
5. Interpret: which aspects of presence are strong/weak in your experience?

Module 8 — Exhibition-Level Project (2.5h workshop)

Concept

An exhibition-level AR project must be robust, maintainable, documented, and accessible to a non-technical audience without supervision.

"Production readiness" criteria:

• Functions without technical intervention for a minimum of 4 hours
• Handles error cases gracefully (tracking lost → message + automatic restart)
• Technical documentation (README for operators)
• User documentation (< 30 sec onboarding)
• Tested on at least 3 different devices

Lab 8.1 — Production Readiness Review

Steps:

1. List all "failure modes" of your project (tracking lost, network cut, low battery, object out of frame)
2. For each: what currently happens? What should happen?
3. Implement the 2 most critical ones
4. Write the operator README (1 A4 page max): startup, restart, common issues

Capstone Project — Exhibition-Level AR Experience

Brief

Design, develop, test, and document an AR experience presentable in a professional context: art gallery, digital festival, public space, professional training, or heritage site.

Constraints

• Unity AR Foundation (iOS or Android) OR high-quality WebAR (8thWall)
• Minimum 3 distinct interaction patterns
• Cloud anchor or session persistence
• Tested with SUS + Presence Questionnaire + 5 users minimum
• At least one documented iteration based on tests
• Production-ready: error handling, operator README, session duration ≥ 4h without intervention

Deliverables

1. Deployable build (APK/IPA file or WebAR link)
2. Design research report (8–12 pages): context, technical choices, architecture, test results (data + analysis), iterations, limitations
3. Operator README (1 page)
4. Presentation (15 min + 10 min Q&A): live demo + data + critical reflection

Evaluation Rubric

Readings & Resources

• Unity AR Foundation: https://docs.unity3d.com/Packages/com.unity.xr.arfoundation@latest
• ARCore Cloud Anchors: https://developers.google.com/ar/develop/cloud-anchors
• Azure Spatial Anchors: https://learn.microsoft.com/azure/spatial-anchors/
• Photon Fusion (networking): https://www.photonengine.com/fusion
• ML Kit Object Detection: https://developers.google.com/ml-kit/vision/object-detection
• iGroup Presence Questionnaire: http://www.igroup.org/pq/ipq/index.php
• Witmer, B. G., & Singer, M. J. (1998). Measuring Presence in Virtual Environments. Presence, 7(3), 225–240. https://doi.org/10.1162/105474698565686
• W3C WCAG 2.1: https://www.w3.org/TR/WCAG21/
• ACM CHI Proceedings (HCI/AR research): https://dl.acm.org/conference/chi
• IEEE ISMAR (AR/MR): https://ismar.net

→ Previous level: AR Interaction Design — Level 2 → Switch to VR: VR Interaction Design — Level 1 → Masterclass or custom workshop: Book a 30-min call — free