2019, Shanghai, China

My Role Technical and Educational Research, 3D Modeling, UX/UI Design, Sound Production, Game Development




Unity, Maya, Google Blocks, Adobe Audition, Figma

3 months

An immersive VR game experience that opens up the conversation in science education.

By providing location-based and contextual guidance throughout the transit journey, our goal was to successfully guide autistic adults that are inexperienced riders from their location to their desired destination. 

An independent capstone project.


According to the 2016 New Media Consortium (NMC) Technology Outlook, the integration of virtual learning experience will be a critical trend for Chinese K-12 education in five years. Science education today prepares the future generation with ethics for technological advancement and empathy for nature, where museums can shape public science literacy with display and novel interaction. In particular, Virtual Reality (VR) and Mixed Reality (MR) have been proven to perform better than any other media technologies in creating psychological presence, reinforcing children’s curiosity, empathy, and engagement.

Oceanscope is an independent capstone project that aims to open up discussions in science education, new media, and HCI:

How can better engage curiosity by creating a virtual space that encourages exploration?

What are the key concepts to pay attention to children engaging in virtual reality?

How can we think out of the box to create new interactions?

Secondary Research

Base on my research on the application of new media in children’s education, I had in my mind a tangible user interface(TUI) platform that aims to encourage exploration and understanding of nature and science.

Unlike traditional Graphical User interfaces(GUI) that only allows limited on-screen interaction, I propose to extend the use of a screen with a trackable periscope and projection mapping. Through active engagement in this immersive digital playground, the children can get an idea of species evolution through the case of whales, as well as the climate and geographical factors that are critical to the process. The immersive, game experience encourages the player to empathize with the marine creatures, to look at the world they are living in not simply as a static existence, but a product of a dynamic change in an extended time span.

Book ReferenceNijholt, Anton. Playful User Interfaces: Interfaces That Invite Social and Physical Interaction. Springer, 2014.

Zimmer, Carl, and Douglas John Emlen. Evolution: Making Sense of Life. Roberts and Company Publishers, 2016.

Kac, Eduardo. Telepresence & Bio Art: Networking Humans, Rabbits, & Robots. University of Michigan Press, 2005.

Bérigny, Caitilin De, et al. “Tangible User Interface Design for Climate Change Education in Interactive Installation Art.” Leonardo, vol. 47, no. 5, Oct. 2014, pp. 451–456., doi:10.1162/leon_a_00710.

Project Reference

A-Vovle by Christa Sommerer & Laurent Mignonneau

Genesis by Eduardo Kac

Plankton Populations Exhibit at Exploratorium

Connecting! Block Town by TeamLab

Technical Exploration

I carried research on technical feasibility for the game, and have explored some technical feasibility of tracking and realization.


Testing Video - Integrating Posenet model into P5.js web application

PoseNet is a vision model that can be used to estimate the pose of a person in an image or video by estimating where key body joints are.

PRO: Very accessible to the audience, because it is browser-based.

CON: Tracking is a lot less stable/reliable. The game scope is very limited since it mainly supports 2D visuals.

Unity 3D  + Keyboard Control

Unity is a real-time development platform. It provides service to create 3D, 2D VR & AR visualizations for Games, Auto, Transportation, Film, Animation, Architecture, Engineering & more.

PRO: accessible to the audience as long as they can download the game and play on PC

CON: not immersive

Unity 3D with Vive HMD + Vive Tracker

image source: https://vvvv.org/sites/default/files/imagecache/large/images/tracker-s01-min_0.png

PRO: gives a possibility to simulate more natural interactions, like shooting arrows, etc
CON: the Vive Tracker should be used accompanied by Vive Controller, and it only gives position feedback so not much input from the player to engage in a game experience

Unity 3D with Vive HMD + Vive Controller

First iteration of improved interaction of teleportation in the game

PRO: SteamVR gives a lot of possibilities to program the feedback from the controller to the user. It is well-supported in development.

CON: Vive can only power the game when HMD and Vive Controller are powered together, otherwise the calculation will get inaccurate. This means that it is difficult to let the player only user controller, not wearing the headset to play the game.
I ended up compromising the more engineering-heavy plan, which requires more engineering to create a headset-free immersive experience with controllers.

Design and Prototyping

The visual design should appeal to chi

ldren and encourage room for imagination. The overall design for the virtual space should be abstract and low-poly.I was trying to create scenes with Google Tilt Brush. However, it ran into a lot of compatibility issues with the SteamVR library (only compatible with Steam VR library v.1). Having tried both Google Tilt Brush and Google Blocks, I decided that Google Brush is more ideal of volume heavy character and object modeling co


I made several prototypes and interaction scripts in C# in Unity.
The game development involves some testing in technical possibility within the constraint of time. It scaled down the scope of the virtual world, as well as the limited realization of moving whales due to lack of time to rigging and character animation in Maya exporting to Unity.

* see my final project on GitHub


I use the development prototypes, I came up with many key insights into the way of interaction. Including how players navigate the space with teleportation, as well as balancing the amount of audio information with roaming time in the virtual world for the player.

Meanwhile, I got important insights into the interaction when tested on small children with VR. I realized the HMD is to heavy for children between 4 to 7. I also realized that the height difference made the game less reliable for children to experience in VR.