My Role Technical and Educational Research, 3D Modeling, UX/UI Design, Sound Production, Game Development

Platform

Vive

Language
C#

Tools
Unity, Maya, Google Blocks, Adobe Audition, Figma

An independent capstone project.

Base on my research on the application of new media in children’s education, I had in my mind a tangible user interface(TUI) platform that aims to encourage exploration and understanding of nature and science.

Unlike traditional Graphical User interfaces(GUI) that only allows limited on-screen interaction, I propose to extend the use of a screen with a trackable periscope and projection mapping. Through active engagement in this immersive digital playground, the children can get an idea of species evolution through the case of whales, as well as the climate and geographical factors that are critical to the process. The immersive, game experience encourages the player to empathize with the marine creatures, to look at the world they are living in not simply as a static existence, but a product of a dynamic change in an extended time span.

Book ReferenceNijholt, Anton. Playful User Interfaces: Interfaces That Invite Social and Physical Interaction. Springer, 2014.

Zimmer, Carl, and Douglas John Emlen. Evolution: Making Sense of Life. Roberts and Company Publishers, 2016.

Kac, Eduardo. Telepresence & Bio Art: Networking Humans, Rabbits, & Robots. University of Michigan Press, 2005.

Bérigny, Caitilin De, et al. “Tangible User Interface Design for Climate Change Education in Interactive Installation Art.” Leonardo, vol. 47, no. 5, Oct. 2014, pp. 451–456., doi:10.1162/leon_a_00710.


Project Reference



A-Vovle by Christa Sommerer & Laurent Mignonneau

Genesis by Eduardo Kac
Plankton Populations Exhibit at Exploratorium
Connecting! Block Town by TeamLab

PoseNet

PoseNet is a vision model that can be used to estimate the pose of a person in an image or video by estimating where key body joints are.

Unity is a real-time development platform. It provides service to create 3D, 2D VR & AR visualizations for Games, Auto, Transportation, Film, Animation, Architecture, Engineering & more.

PRO: accessible to the audience as long as they can download the game and play on PC

CON: not immersive

PRO: gives a possibility to simulate more natural interactions, like shooting arrows, etc

CON: the Vive Tracker should be used accompanied by Vive Controller, and it only gives position feedback so not much input from the player to engage in a game experience

Unity 3D with Vive HMD + Vive Controller

PRO: SteamVR gives a lot of possibilities to program the feedback from the controller to the user. It is well-supported in development.

CON: Vive can only power the game when HMD and Vive Controller are powered together, otherwise the calculation will get inaccurate. This means that it is difficult to let the player only user controller, not wearing the headset to play the game.

I ended up compromising the more engineering-heavy plan, which requires more engineering to create a headset-free immersive experience with controllers.

I made several prototypes and interaction scripts in C# in Unity.

The game development involves some testing in technical possibility within the constraint of time. It scaled down the scope of the virtual world, as well as the limited realization of moving whales due to lack of time to rigging and character animation in Maya exporting to Unity.

I use the development prototypes, I came up with many key insights into the way of interaction. Including how players navigate the space with teleportation, as well as balancing the amount of audio information with roaming time in the virtual world for the player.

Meanwhile, I got important insights into the interaction when tested on small children with VR. I realized the HMD is to heavy for children between 4 to 7. I also realized that the height difference made the game less reliable for children to experience in VR.