Sweet Shades: Exploring a framework for augmented reality

Interns

William Ou, Edward Gonzalez, Julianna Flores, Erik Rodriguez

Project Description

In our research project, we explored facial detection technology in augmented reality (AR) to bring digital content to the real world. There are many applications of AR, such as in entertainment, education, and productivity. Our project goal is to use facial detection software to detect facial markers and layer an image of our choosing on top of a face. This project would develop our understanding of AR tech. Our main tools were python as our coding language and OpenCV (Open Computer Vision), which is an open-source library that provides tools for computer vision such as face detection and facial key points. We implemented these tools by first writing a python script to take in our video feed, then used OpenCV to detect the faces and create facial keypoints called landmarks. We took two points of interest of our shades and mapped them to our corresponding facial landmarks, in this case, next to the eyes. This resulted in shades being applied onto the face in a live video feed. We tested our facial detection software on a face with various accessories, backgrounds, lighting, distance, and tilt. We found a combination of both tilt of the face and distance were the main limiting factors in the program’s ability to detect all our faces. We successfully applied shades on a quick and consistently running program with a smooth video output. Our exploration allowed us to better understand AR technology and its applications in the real world.

Project Files