Bible — AR Interaction Design

Public reference document. Complete history, principles, techniques, tools (2026 state), case studies, glossary and bibliography. Every fact is sourced.

Chapter 1 — Definitions & Conceptual Framework

1.1 What is Augmented Reality?

Augmented reality (AR) refers to a set of technologies that overlay digital information — visual, auditory, or haptic — onto the perception of the physical world in real time. Unlike virtual reality (VR), which replaces the environment, AR augments it: the user remains anchored in physical space.

The canonical definition remains that of Ronald Azuma (1997): a system qualifies as "augmented reality" if it combines real and virtual, is interactive in real time, and operates in 3D space.

Azuma, R. (1997). A Survey of Augmented Reality. Presence: Teleoperators and Virtual Environments, 6(4), 355–385. https://doi.org/10.1162/pres.1997.6.4.355

1.2 The Reality-Virtuality Continuum (Milgram & Kishino, 1994)

Paul Milgram and Fumio Kishino proposed in 1994 a fundamental conceptual framework: the Reality-Virtuality Continuum. This model places the real environment and virtual environment at opposite ends of a spectrum, with Mixed Reality Environments in between — including Augmented Reality (closer to real) and Augmented Virtuality (closer to virtual).

This continuum originates from two distinct publications (often conflated):

1. Milgram, P., & Kishino, F. (1994). A Taxonomy of Mixed Reality Visual Displays. IEICE Transactions on Information and Systems, E77-D(12), 1321–1329.
2. Milgram, P., Takemura, H., Utsumi, A., & Kishino, F. (1994/1995). Augmented Reality: A Class of Displays on the Reality-Virtuality Continuum. SPIE Proceedings, Vol. 2351, pp. 282–292.

Source: https://en.wikipedia.org/wiki/Reality%E2%80%93virtuality_continuum

1.3 Core Vocabulary

Chapter 2 — Complete History (Chronological, Sourced)

2.1 The Pioneers (1955–1970)

1955 — Morton Heilig, "The Cinema of the Future" Morton Heilig publishes a visionary essay describing a total multisensory cinema. This text prefigures AR/VR but is not yet a functional device. (Note: the commonly cited date of 1957 is incorrect — the essay dates to 1955.)

1962 — Sensorama Heilig builds and introduces the Sensorama, an immersive arcade machine offering stereoscopic images, stereo sound, vibrations, and smells. US Patent No. 3,050,870 is filed the same year.

Source: https://en.wikipedia.org/wiki/Sensorama

1965 — Ivan Sutherland, "The Ultimate Display" Sutherland publishes this foundational text describing an ideal human-machine interface where the screen controls the existence of matter. This text lays the conceptual foundations for AR and VR.

Sutherland, I. (1965). The Ultimate Display. Proceedings of IFIP Congress 1965, vol. 2, pp. 506–508. Source: https://en.wikipedia.org/wiki/Ivan_Sutherland

1968 — "Sword of Damocles" Sutherland, with the help of his student Bob Sproull, creates the first augmented reality headset at Harvard. The device, suspended from the ceiling (hence its nickname), displays simple vector graphics anchored in physical space. It is the first head-tracking system coupled with a display.

Source: https://en.wikipedia.org/wiki/Sword_of_Damocles_(head-mounted_display)

2.2 The Academic Research Era (1970–1995)

1975 — Myron Krueger, Videoplace First public demonstration of Videoplace at the Milwaukee Art Museum. Krueger creates a reactive environment where participants' silhouettes captured by camera interact with graphical entities. Not classical AR, but it inaugurates contactless interaction. (Note: 1974 marks the start of the research, not the public demonstration.)

Sources: https://en.wikipedia.org/wiki/Myron_Krueger · https://en.wikipedia.org/wiki/Videoplace

1985 — NASA VIEW (Virtual Interface Environment Workstation) Developed at NASA Ames Research Center by Scott Fisher and his team, VIEW is a telerobotic simulation system that fuses real video and synthetic graphics. One of the first systems combining head tracking, data glove input, and immersive display.

Source: https://en.wikipedia.org/wiki/Scott_Fisher_(technologist)

1984–1987 — Jaron Lanier & VPL Research Jaron Lanier founds VPL Research in 1984. The company commercializes the DataGlove and the EyePhone (1987), the first commercial VR/AR equipment. These tools become the decade's references for interaction research.

Source: https://en.wikipedia.org/wiki/VPL_Research

1994 — Milgram & Kishino — Reality-Virtuality Continuum Publication of the taxonomic framework that still structures AR/VR/MR research today (see §1.2).

2.3 AR Takes Shape (1997–2010)

1997 — Ronald Azuma, foundational survey Publication of the first academic reference survey on augmented reality, which defines the three canonical criteria of the field.

Azuma, R. (1997). A Survey of Augmented Reality. Presence, 6(4), 355–385. https://doi.org/10.1162/pres.1997.6.4.355

1997 — Steven Feiner, "Touring Machine" Feiner and colleagues at Columbia present the first outdoor mobile AR system enabling exploration of a university campus. Backpack with computer, GPS, compass, and HMD.

Feiner, S., MacIntyre, B., Höllerer, T., & Webster, A. (1997). A touring machine: prototyping 3D mobile augmented reality systems for exploring the urban environment. Digest of Papers. First International Symposium on Wearable Computers (ISWC '97). IEEE, pp. 74–81. https://doi.org/10.1109/ISWC.1997.629922

1999 — Hirokazu Kato, ARToolKit Kato and Billinghurst publish ARToolKit, the first open-source library for marker-based tracking. It radically democratizes AR by enabling development without specialized hardware.

Source: https://en.wikipedia.org/wiki/ARToolKit Kato, H., & Billinghurst, M. (1999). Marker Tracking and HMD Calibration for a Video-Based Augmented Reality Conferencing System. 2nd IEEE/ACM International Workshop on Augmented Reality (IWAR '99).

2009 — Layar Founding of Layar in Amsterdam, the first commercialized AR browser for smartphones. The app overlays geolocated information layers onto the camera view. Launched June 2009.

Source: https://en.wikipedia.org/wiki/Layar

2.4 Consumer AR (2012–2020)

2012–2015 — Google Glass Google announces Google Glass in April 2012 (Sergey Brin publicly wears a prototype on April 5, 2012). The Explorer program opens in June 2012 at $1,500. Distributed to ~8,000 selected developers in April 2013. Consumer sale on April 15, 2014 (sold out the same day). Program discontinued in January 2015.

Source: https://en.wikipedia.org/wiki/Google_Glass

2015–2016 — Microsoft HoloLens Announced January 21, 2015 during the Windows 10 presentation. The Development Edition ($3,000) ships from March 30, 2016. First standalone AR HMD with real-time spatial mapping.

Source: https://en.wikipedia.org/wiki/Microsoft_HoloLens

2017 — ARKit & ARCore Apple launches ARKit with iOS 11 (presented at WWDC on June 5, 2017). Google publishes ARCore in developer preview on August 29, 2017 (stable version 1.0: February 23, 2018). These two SDKs bring AR to hundreds of millions of users' pockets.

Sources: https://en.wikipedia.org/wiki/ARKit · https://en.wikipedia.org/wiki/ARCore

2017–2019 — WebXR The W3C Immersive Web Working Group publishes the first Public Working Draft of the WebXR Device API on February 5, 2019. WebXR standardizes browser-based access to AR/VR capabilities, succeeding WebVR. In 2026, its status is Candidate Recommendation Draft (not yet a final Recommendation).

Source: https://www.w3.org/TR/webxr/

2.5 The Spatial Computing Era (2020–2026)

2024 — Apple Vision Pro Announced at WWDC 2023 (June 5, 2023), launched in the United States on February 2, 2024 (from $3,499). Introduces the concept of spatial computing and visionOS. High-resolution color passthrough. International launch in June–July 2024.

Source: https://en.wikipedia.org/wiki/Apple_Vision_Pro

2024–2026 — AR/VR/AI Convergence Passthrough headsets (Meta Quest 3, Apple Vision Pro) blur the boundary between AR and VR. Generative AI integrates into spatial experiences: real-time 3D object generation, semantic understanding of space, spatial assistants.

Chapter 3 — Core Principles of AR Interaction Design

3.1 Spatial Affordances

In AR, virtual elements must signal their interactivity without the natural haptic feedback of physical objects. Spatial affordances — visual, gestural, auditory — are therefore critical. Don Norman (The Design of Everyday Things, 1988) theorized affordances in the context of physical objects; their transposition to mixed space constitutes an active research field.

Key principles:

• AR objects must suggest how to manipulate them (size, position, cursor, halo)
• Visual transitions signal the interactive state (hover, selection, deactivation)
• Spatial coherence between real and virtual objects reinforces affordance

3.2 Multimodal Feedback

AR is often deprived of haptic feedback. Compensation comes through:

• Visual feedback: deformation, color change, particles, shadow cast on physical surface
• Spatial audio feedback: sound anchored to the AR object's position in space
• Vibrotactile feedback (where the controller or phone permits)

3.3 Presence and Perceptual Anchoring

Presence in AR — the feeling that virtual objects "are really there" — depends on:

• Tracking quality (latency < 20ms, minimal drift)
• Correct occlusion: real objects hide virtual objects according to depth
• Coherent lighting: AR objects receive the same light as the real scene
• Rendering quality: resolution, anti-aliasing, transparency

3.4 AR Comfort and Ergonomics

AR-specific issues:

• Gorilla arm: holding the arm raised to point is tiring (< 2 min for most users)
• Vergence-accommodation conflict: eyes converge on the real object but accommodate for the display — source of eye strain
• Limited Field of View (HoloLens: ~52°, Magic Leap: ~50°): objects exit the FoV if the user doesn't move their head
• Outdoor brightness: additive AR displays are unreadable in intense direct sunlight

Best practices:

• Reduce HMD session duration (< 20 min for general public)
• Place AR elements in the visual comfort zone (1–10 m, eye level)
• Avoid forcing users to raise their arms; prefer low-amplitude gestures
• Offer alternative interaction methods (gaze, voice) for accessibility

3.5 Privacy and Ethics

AR through connected glasses raises unprecedented ethical questions:

• Invisible surveillance: a permanently worn camera can record the environment, faces, and conversations without third parties' knowledge
• Facial recognition: interoperability with identification databases (prohibited by GDPR in Europe without explicit consent)
• Public space pollution: overlaying advertisements or information in shared spaces without consent

These questions are actively debated in standardization bodies (W3C, ISO) and by regulators (EU AI Act, 2024).

3.6 Accessibility

• Visual impairments: AR can visually amplify the environment, but the interfaces themselves must have sufficient contrast and non-visual alternatives
• Motor disabilities: hand tracking must offer alternatives (gaze, voice, switch)
• Epilepsy: avoid flashing animations (WCAG 2.1, criterion 2.3.1)
• Color blindness: never communicate state through color alone

Chapter 4 — AR Interaction Techniques & Patterns

4.1 Input Modes

4.2 Selection and Manipulation

Near interaction (object within reach):

• Pinch-and-grab: grasp a virtual object like a physical one
• Ray + pinch: project a ray from the hand, pinch to select

Far interaction (distant object):

• Raycasting: aiming from the hand or gaze, selection by pinch/click
• Gaze + dwell: looking at an object for N seconds to select it (hands-free)
• Voice + pointing: "this" + gestural designation

3D manipulation:

• Translation (move), rotation, scaling
• Direct manipulation (6DoF) vs. proxy manipulation (handles, gizmos)
• Spatial constraints: snap-to-surface, snap-to-grid, axis constraints

4.3 Anchoring and Persistence

• Local anchor: object anchored to the device's position at creation — disappears if device restarts
• World anchor / Cloud anchor: ARCore Cloud Anchors, ARKit Persistent World Tracking — object survives sessions and is shareable between users
• Image anchor: triggered by detection of a target image (poster, book, logo)
• Surface anchor: anchored to a detected plane (floor, table, wall)
• Spatial anchor service: Azure Spatial Anchors, Google Cloud Anchors — server-side persistence

4.4 Navigation and Orientation

In AR, the user moves through physical space — navigation is physical, not virtual. Challenges include:

• Maintaining anchoring coherence as the user moves far from the origin point
• Indicating the direction of off-screen AR objects (directional arrows, mini-map overlay)
• Managing indoor-to-outdoor transitions (changing tracking conditions)

4.5 Diegetic vs. Non-Diegetic Spatial UI

Best practice: prefer world-locked over head-locked to avoid visual fatigue; reserve head-locked for critical, transient information.

Chapter 5 — Tools & Ecosystem (2026 State)

5.1 Development Engines

5.2 Web & No-Code

5.3 Creative & Experimental Tools

5.4 Hardware (2026 State)

Chapter 6 — Case Studies

6.1 ARToolKit & Interactive Art (1999–2009)

The free release of ARToolKit triggers a decade of artistic experimentation. Artists and researchers use markers to create installations where virtual objects emerge from physical supports: books, paintings, garments. This is first-generation AR — limited in tracking but accessible.

6.2 Pokémon GO (Niantic, 2016)

Launched in July 2016, Pokémon GO is the first massive consumer AR phenomenon. Its strength is not AR quality (basic: no occlusion, no real anchoring) but the geolocated game layer superimposed on the real world. Over 800 million downloads. Demonstrates that AR value can be social and geographic rather than visually sophisticated.

6.3 ZELLIGE ARTCADE — Kamel Ghabte [K — details to confirm]

Itinerant digital cultural mediation installation. Moroccan zellige geometry is transformed into an interactive game accessible to all audiences. Part of the collection Digital Memories of Morocco / Mémoires Numériques du Maroc. 52 international participations since 2017.

[K — technical details of AR/physical interaction to be confirmed by Kamel: exact stack, interaction type, hardware.]

6.4 Museums & Heritage

Institutions such as the Smithsonian (Washington), the British Museum, and the Louvre have experimented with AR layers over their permanent collections — allowing visitors to see fragmented objects reconstructed, animated frescoes, or contextual information superimposed. Most of these experiences use ARKit or ARCore via dedicated applications.

6.5 Industrial Training & Surgery

PTC Vuforia and Microsoft HoloLens have been deployed in industrial training: technicians see assembly steps superimposed directly onto the machine in front of them. In surgery, AR systems overlay MRI scans in real time onto the surgical field (clinical studies ongoing, not yet standard of care).

Chapter 7 — Current & Future Issues

7.1 Generative AI + Spatial Computing

Integrating generative AI (LLM, diffusion, 3D generation) into AR experiences opens unprecedented uses:

• On-demand 3D object generation: the user describes an object, it appears in space (e.g. Luma AI, Shap-E)
• Semantic scene understanding: AR understands what it "sees" (table, chair, person) and adapts content
• Spatial assistants: an AI agent anchored in space answers questions about physical objects around the user (e.g. Copilot on Vision Pro)

7.2 Open Standards & Interoperability

The risk of proprietary silos (Apple vs Meta vs Microsoft ecosystem) hampers adoption. Open standards in progress:

• OpenXR (Khronos Group): cross-runtime API for AR/VR — https://www.khronos.org/openxr/
• WebXR (W3C): AR in the browser (see §2.4)
• USD (Universal Scene Description): interoperable 3D scene format (Apple, NVIDIA, Pixar)
• glTF: lightweight 3D model format for the web (Khronos) — https://www.khronos.org/gltf/

7.3 Heritage & Culture

AR is a powerful tool for cultural transmission:

• Bringing fragmented objects, archaeological sites, and lost crafts back to life
• Making intangible heritage accessible (gestures, sounds, historical contexts)
• Risks: digitization without community consent, appropriation, disembodiment of living heritage

7.4 Passthrough MR and the AR/VR Boundary

With Meta Quest 3 and Apple Vision Pro, the distinction between AR (additive optical display) and VR (total immersion) is blurring. Passthrough MR (high-resolution color cameras integrated into a VR headset) allows real-time switching from total immersion to transparent overlay. This creates new interaction patterns and new design challenges.

Chapter 8 — Glossary (42 terms)

Chapter 8 — Full Bibliography

Primary Sources

• Sutherland, I. (1965). The Ultimate Display. Proceedings of IFIP Congress 1965, vol. 2, pp. 506–508.
• Milgram, P., & Kishino, F. (1994). A Taxonomy of Mixed Reality Visual Displays. IEICE Transactions on Information and Systems, E77-D(12), 1321–1329.
• Milgram, P., Takemura, H., Utsumi, A., & Kishino, F. (1994/1995). Augmented Reality: A Class of Displays on the Reality-Virtuality Continuum. SPIE Proceedings, Vol. 2351, pp. 282–292.
• Azuma, R. T. (1997). A Survey of Augmented Reality. Presence: Teleoperators and Virtual Environments, 6(4), 355–385. https://doi.org/10.1162/pres.1997.6.4.355
• Feiner, S., MacIntyre, B., Höllerer, T., & Webster, A. (1997). A touring machine: prototyping 3D mobile augmented reality systems for exploring the urban environment. Digest of Papers. First International Symposium on Wearable Computers (ISWC '97). IEEE, pp. 74–81. https://doi.org/10.1109/ISWC.1997.629922
• Kato, H., & Billinghurst, M. (1999). Marker Tracking and HMD Calibration for a Video-Based Augmented Reality Conferencing System. 2nd IEEE/ACM International Workshop on Augmented Reality (IWAR '99).

Secondary Sources & Resources

• W3C Immersive Web Working Group. WebXR Device API. https://www.w3.org/TR/webxr/
• Khronos Group. OpenXR. https://www.khronos.org/openxr/
• Apple Developer. ARKit. https://developer.apple.com/arkit/
• Google Developers. ARCore. https://developers.google.com/ar
• Wikipedia. Reality-Virtuality Continuum. https://en.wikipedia.org/wiki/Reality%E2%80%93virtuality_continuum
• Wikipedia. ARToolKit. https://en.wikipedia.org/wiki/ARToolKit
• Wikipedia. Apple Vision Pro. https://en.wikipedia.org/wiki/Apple_Vision_Pro
• Norman, D. A. (2013). The Design of Everyday Things (revised ed.). Basic Books.

Further Reading

• WebXR Samples (Google): https://immersive-web.github.io/webxr-samples/
• A-Frame Docs: https://aframe.io/docs/
• ARCore Fundamentals (Google Codelabs): https://developers.google.com/ar/develop
• Apple Vision Pro Design Principles: https://developer.apple.com/design/human-interface-guidelines/spatial-ui
• ACM CHI Proceedings (annual, reference HCI/design conference): https://dl.acm.org/conference/chi
• IEEE VR / ISMAR (academic AR/VR conferences): https://ieeevr.org · https://ismar.net

Bridge to Training

This bible is a public reference document. To go further with hands-on guidance:

→ AR Interaction Design Masterclass — functional prototype in 2 days → Intensive workshop for your team or students → Book a 30-min call — free