Mixed Reality (MR) and Augmented Reality (AR) create exciting opportunities to engage users in immersive experiences, resulting in natural human-computer interaction.
Many MR interactions are generated around a First-person Point of View (POV). In these cases, the user directs to the environment, which is digitally displayed either through a head-mounted display or a handheld computing device. One drawback of such conventional AR/MR platforms is that the experience is user-specific. Moreover, these platforms require the user to wear and/or hold an expensive device, which can be cumbersome and alter interaction techniques.
We create a solution for multi-user interactions in AR/MR, where a group can share the same augmented environment with any computer generated (CG) asset and interact in a shared story sequence through a third-person POV. Our approach is to instrument the environment leaving the user unburdened of any equipment, creating a seamless walk-up-and-play experience. We demonstrate this technology in a series of vignettes featuring humanoid animals.
Participants can not only see and hear these characters, they can also feel them on the bench through haptic feedback. Many of the characters also interact with users directly, either through speech or touch. In one vignette an elephant hands a participant a glowing orb. This demonstrates HCI in its simplest form: a person walks up to a computer, and the computer hands the person an object.
1 IMPLEMENTATION
We create a 3D reconstruction of a scene using a combination of the depth and color sensors on an off-the-shelf Microsoft Kinect. To do this, we draw polygons using each point in the point cloud as a vertex, creating the appearance of a solid mesh. The mesh is then aligned to the RGB camera feed of the scene from the same Kinect. This alignment gives the mesh color, and completes a 3D reconstructed video feed.
There are several problems that arise with the 3D constructed feed. First, the monocular feed creates “depth shadows” in areas where there is no direct line-of-sight to the depth sensor. Second, the depth camera is laterally offset from the RGB camera (since they cannot physically occupy the same space) and therefore have slightly different viewing angles, creating further depth shadowing. The resulting data feed is sparse and cannot represent the whole scene (see Figure 3. To solve this, we align the 3D depth feed with the 2D RGB feed from the Kinect. By compositing the depth feed over a 2D backdrop, the system effectively masks these depth shadows, creating a seamless composite that can then be populated with 3D CG assets.
This mixed reality platform centers around the simple setting of a bench. The bench works in an novel way to constrain a few problems, such as identifying where a user is and subsequently inferring the direction of the user’s gaze (i.e., toward the screen). It creates a stage with a foreground and background, with the bench occupants in the middle ground. The bench also acts as a controller; the mixed reality experience won’t trigger until at least one person is detected sitting on the bench. Further, different
seating formations on the bench trigger different experiences.
Magic Bench is a custom Software and custom Hardware platform from Disney Research, necessitating a solution to bridge both aspects. Between the two exists a series of patches created in Cycling ’74 Max designed to convert signals sent from the game engine (via OSC) about the positions and states of objects in the scene, into the haptic sensations felt on the bench. Haptic actuators are dynamically driven based on the location of animated content. The driving waveform for each actuator is designed according to the desired feel — in the current setup we can tweak base frequency, frequency of modulation, general amplitude, amplitude envelope, and three-dimensional position. These parameters can be manually tuned and/or adjusted in real time.
2 INSTALLATION OPTIONS
This piece can run as a traditional VR Village installation or as an autonomous piece in an unsuspecting area at SIGGRAPH — imagine sitting on a bench to rest your feet or check your email; in front of you is a screen showing a SIGGRAPH showreel. Once the system detects you, the content switches to a video feed of you,
creating a mirror effect. From there, an unexpected AR experience unfolds.
Learn more: Magic Bench — A Multi-User & Multi-Sensory AR/MR Platform
The Latest on: Holodeck
[google_news title=”” keyword=”holodeck” num_posts=”10″ blurb_length=”0″ show_thumb=”left”]- How Science Fiction Confronts the Real Isolation of Spaceon May 6, 2024 at 2:19 pm
A Space Odyssey to Moon, Project Hail Mary and Star Trek's holodeck, explore the psychological challenges of astronaut loneliness found in sci-fi, drawing parallels to real-life ISS experiences and ...
- 6 Ups & 5 Downs From Star Trek: Discovery 5.6 — Whistlespeakon May 2, 2024 at 8:05 am
In this, the written version of the Ups & Downs, we're going to do something a little different when it comes to Cetacean Observations. Seán will have the fine-eyed and exhaustive list of easter eggs ...
- Make 3D Scenes With A Holodeck-Like Voice Interfaceon April 29, 2024 at 8:30 am
The voice interface for the holodeck in Star Trek had users create objects by saying things like “create a table” and “now make it a metal table” and so forth, all with ...
- Shockingly, People Aren’t Interested In Paying $3,500 For Apple’s Wildly Overpriced Vision Pro VR Headseton April 26, 2024 at 5:49 am
Apple analyst Ming-Chi Kuo reported that the company had slashed its Vision Pro shipments to 400,000-450,000 units, with the market predicting 700,000–800,000 units or more. According to the report, ...
- Best of Last Week—gravity free technology, recreating the Holodeck, why the moon is lopsidedon April 14, 2024 at 5:00 pm
And a team at the University of Pennsylvania recreated Star Trek's holodeck using ChatGPT and video game assets—called Holodeck, the system generates interactive virtual 3D environments. A team of ...
- Star Trek’s Holodeck Recreated As Virtual Training Ground For Next-Gen Robotson April 14, 2024 at 5:00 pm
But the Holodeck system (yep, that’s what it’s called), created by a team at Penn Engineering and their collaborators, can build pretty much any 3D environment you can think up. You just have ...
- Penn Engineers recreate Star Trek’s Holodeck using ChatGPT and video game assetson April 11, 2024 at 9:21 am
Using everyday language, users can prompt Holodeck to generate a virtually infinite variety of 3D spaces, which creates new possibilities for training robots to navigate the world. That paucity of ...
- Star Trek's Holodeck recreated using ChatGPT and video game assetson April 10, 2024 at 5:00 pm
Star Trek's Holodeck is no longer just science fiction. Using AI, engineers have created a tool that can generate 3D environments, prompted by everyday language. In Star Trek: The Next Generation ...
- Holodeck: A Revolutionary AI Tool for Creating Virtual Worlds Unveiledon December 19, 2023 at 12:30 pm
Holodeck’s core functionality hinges on the amalgamation of cutting-edge AI technologies. At its heart lies OpenAI’s GPT-4, a powerhouse model renowned for its adeptness in comprehending and ...
- 5 Best Star Trek Doctors From Every Series And Movieon May 12, 2023 at 8:43 am
The Star Trek franchise is replete with doctors of different types, from the Original Series to Picard and Strange New Worlds. It is also full of strong opinions about each of them-their ...
via Google News and Bing News