Augmented Reality Glasses
Overview
This project aimed to build a spectacle aided with a display and a camera. The glasses projected whatever it sees into the person’s field of view. The main motive was to enable people who have a low vision to see the surrounding clearly. It would seem like an augmented reality display in front of the eyes. Also, the display was connected to a raspberry pi which enabled facial recognition, and the information about that person was directly displayed onto the display. Similar to the one which Iron man wears!
Inspiration
This project was inspired by people having incurable eye disease, who cannot wear normal glasses and correct their vision. So to help them to see things clearly in daily life we thought it would be better for them to have a zoomed-in image of whatever is in his field of vision. With an added benefit of recognizing any person in front of him
The Architecture
We have developed a custom framework for this project. Our code consists mainly of three parts face_recognition module, display module, and camera_driver (also web_server module which runs independently). So the video loop is constantly taking photos from the raspberry camera at 30 FPS. The display module depending on the mode of operation, displays the camera feed at 30 FPS. The face_recognition module works at 1 FPS, as the computations are resource-heavy and we can’t run it at 30 FPS. We have used threading and multiprocessing both. So the main program starts and splits a thread for video loop which runs continuously. Then it starts a Process for face recognition module which runs at 1 FPS. Then it goes into the display loop. The intercommunication between all these is done by the data variable which is shared with every module. A lot of trouble was faced to achieve this. We have worked through many workarounds and hacks to exchange information between threads.
Here are the different layers of the architecture.
Display
In Display module we have used pygame to draw different things on the window. There are two modes introduced, one to display the camera feed and second to display just text and information. We have introduced a generic text class, which further enhances the ease of drawing text in pygames.
Face Recognition:-
We have tunned the dlib library to run fluently on the raspberry pi. Also, we have introduced a structured database to store the recognized faces. This was done so that afterward dynamic databasing can be done easily.
Web Server
We have also made a website that can run on any web browser phone/tablet/pc. The server runs on raspberry pi. You just have to connect to the same wifi network as pi and go to a specific web address. With this, you can control the different aspects of glasses. This can also be used for dynamic databasing and remote face recognition in the future.
Components Used
Raspberry Pi 3
This powers the glasses, all programs run on raspberry pi. Installation of raspbian and activating camera was quite easy, one can find numerous guides online. The problem we faced was to install OpenCV(v3+), we did it by compiling from the source, this was a very long process that took 6-7 hours to complete.
Vufine Display
The biggest challenge was to actually display the screen in front of the eyes (augmented reality window). Also buying a heads up display was difficult as it was too costly, so we contacted the company ‘VUFINE’, and they were ready to give us their heads up display unit for free. I would thank VUFINE for this gesture.
3D Printed Mounts
For holding the Raspberry pi camera right behind the vufine display we needed a few custom made parts. Hence we opted to 3D print the mounts.
DLIB Library
Face Recognition was one of a crucial part of our project, and we tried many different libraries and also standard algorithms from the OpenCV. But finally, we used the dlib library. We used the facial models trained by Adam Geitgey