NAV-TECH

The major motivation for this project was the feeling of difficulty that blind and visually impaired individuals experience while navigating in an unknown indoor spaces such as shopping mall and another major reason being them not having a freedom to move independently without any external expensive assistive technologies

We implemented a camera based navigation mechanism using deep reinforcement learning that helps them navigate in an indoor spaces(for the prototyping phase at least). They will have an airpods that delivers an spatial audio of left, right, front and back so as to help then navigate without any additional accessibility aids. Also with this strategy, we need not make much modifications to the infrastructure, thus making it realistic to implement at relatively low cost.

We for our project implemented a Deep Dyna-Q learning algorithm for path planning that translates the isometric camera perspective into a grid map of walkable spaces and obstacles and with their phone in their pocket we know their precise orientation and with the problem basically resolves to a maze game where we need to navigate them from start to a goal state. We also used YOLO V7 algorithm with Deepsort to track precisely and maintain discrepancy among other members in the room. We built a full fledge react native application that streams the magnetometer readings to the local server and and renders spatial audio to airpods based on the guiding information received from the local server where the model is trained and the camera is connected within a wifi.

One major challenge that we ran into was not being able to use the pre-mount cctv cameras in the building and thus even if we tried to use our phone cameras we were not able to raise it up to the ceiling just as these CCTV cameras to capture the isometric views. Another challenge was the compute power and since we ran the entire model, the motion planning as well as the YOLO algorithm, we didn't get a good frame rate as they are something that needs to run in a dedicated GPU. This lead us not being able to conduct the testing in a realistic manner, however we strongly believe this can be something that's novel compared to prevalent navigation mechanisms in the status-quo.

We were able to complete a full-fledge react native mobile app, the tracking and the motion planning algorithm and being able to generate spatial audio to navigate in an intuitive manner. We still cannot believe we have accomplished so much and it passed our expectations by a significant amount. We are almost done with our project and are really excited to get it published in and top-tier conference.

We still can't believe we learned so much in this 24 hour time. A period of stress, a period of extreme productivity, a period of desire to reach that extra mile and finally brainstorming new ideas on the way and refining our project in a best way we can. I think this has been one of most productive time i had this year.

We're shooting for publication and we aim to apply for a government grant so as to make this technology accessible and deploy in a real life navigation scenario.

Built With

• mobile
• python
• pytorch
• react-native
• sensors
• speech-to-text
• typescript
• websockets