With the advancements in VR, the applications now include helping people cope with grief at the loss of a loved one.

In 2020, a 2-part Korean television documentary, called Meeting You (or I Met You depending on translation) was released, that told the story of using VR to reunite a mother with her deceased daughter. Jang Ji-sung, mother of four, lost her seven-year-old daughter Nayeon due to a blood disease. After 3 years, Ji-sung keeps pictures of Nayeon and all her favourite toys around the house, promising to remember her forever. But as time passed, she feared she would start to forget her daughter as her memories become more distant. Hence, she hoped to see her daughter for one last time. 

To fulfil her wish, the production team worked to produce this meeting between mother and daughter in the virtual world. They recreated Nayeon to be as accurate as possible, from the face down to the voice and motions. With a set of VR HMD and gloves, Ji-sung was able to meet Nayeon once again and had an emotional reunion. Nayeon’s father and siblings watched tearfully as her mother finally gotten the closure she was seeking from this final meeting.

To make this demo possible, an HTC Vive Pro was used, along with Noitom’s VR gloves that allowed Ji-sung to interact with her daughter. The virtual Nayeon was created by Korean startup Vive Studios over the course of 8 months using various methods. High resolution photos of Nayeon’s younger sister were taken in a process called photogrammetry to serve as the basis for the 3D model, with further refinement done using old photos to better resemble the 7-year-old Nayeon.  In addition, numerous child actors were brought in to recreate Nayeon’s voice according to old recordings, and motion capture was used to record an adult actor emulating the motions and facial experience based on old videos and photos.

The demo itself was run in Unreal Engine 4, with an NPC using the animations and model of the recreated Nayeon. The model included different levels of detail, allowing the pores on Nayeon’s face to be seen when Ji-sung moved nearer, along with interactable hair and clothes. The short demo also included a basic AI, allowing Ji-sung to experience events with her daughter such as her birthday and putting her down on her bed for a nap. While these events were ultimately scripted, it still elicited a very emotional response from Ji-sung.

Although I have had personal experience with many VR applications such as Superhot and Beat Saber, as well as having developed a VR demo in UE4, it was this recreation of a passed loved one that truly struck a chord with me. It demonstrated the ability of VR to create immersive experiences that resonate on a truly visceral and emotional level.

However, there is the elephant in the room with regards to the ethics of such an application. Many see this as impeding the natural healing process of grief. But such accusations were rebutted by Ji-sung, who maintained that it was a positive expirence. Others criticised the experience as ‘fake’. While it is true, given the virtual Nayeon is not a truly robust AI and all the encounters within the demo are scripted, the emotional response from Ji-sung cannot be denied. And neither can the efforts of the developers in creating a virtual Nayeon that is sufficiently realistic while still avoiding the uncanny valley. For performance, capturing the right motions for Nayeon to move like a convincing 7-year-old and the voice to sound like her are also props I would give to the developers.

As to the question of how such a use case can be improved, I would say that only time will tell. If AI progresses to the point that we can accurately recreate personalities of the deceased and generate interactions in real time, we could truly remake a late loved one. And the latest improvements in creating realistic faces can already be seen in the trailer for The Matrix Awakens made in Unreal Engine 5 which features a highly realistic Keanu Reeves.

The viral video of Ji-sung’s final meeting with Nayeon has touched many hearts in its time on Youtube.