404_Fawn

Initial Page, clicking on the Fawn Extension
Selecting Visual Help
Selecting Get Help
Settings within app extension 1/2
Settings within app extension 2/2
Potential results of extension

Inspiration

The inspiration for Fawn stems from the "Digital Divide" that separates generations. As Computer Science students, we are the designated IT support for our families. We watched our loved ones struggle with banking portals not because of a lack of intelligence, but because the web isn't built for the natural decline in vision and dexterity that comes with aging. We realized that for the 1 in 4 Canadians who will face irreversible vision loss by age 75, the internet is becoming a minefield of "red" and "green" buttons they can’t distinguish and tiny text they can’t read. We wanted to build a bridge a tool that allows us to help them remotely without compromising their most sensitive financial data.

What it does

Fawn is a specialized accessibility layer that overlays directly onto websites. It features:

Peer-to-Peer Remote "Visual Help": A browser-native screen-sharing utility where a helper can "draw" or "click" on their feed, sending X/Y coordinates via WebRTC to trigger native mouse ripples on the host’s screen. To direct them into helping solve their issues without taking over their computer.

Transaction Summarizer: A custom DOM-scraping algorithm that bypasses complex CSS Grids and ARIA tables to extract and categorize spending locally, for easy monthly bank statements.

Sensory Suite: Live injection of Open Dyslexic fonts and SVG color matrices to recalibrate the web for Protanopia, Deuteranopia, Tritanopia, and Monochromacy or a custom visual correction.

Financial Literacy Tools: Automated highlighting of complex terms (e.g., Amortization, APR) with hoverable, simplified tooltips and integrated Text-to-Speech.

How we built it

We architected Fawn as a Manifest V3 Chrome Extension paired with a Node.js Signaling Server.

Frontend: Built with React.js for a modern UI and vanilla JavaScript content scripts for DOM manipulation.

Connectivity: We used Socket.io for signaling to exchange WebRTC offer/answer packets. Once the RTCPeerConnection is established, the server is removed from the loop, ensuring a valid connection.

Interaction: We utilized navigator.mediaDevices.getDisplayMedia for the video stream and RTCDataChannel for transmitting interaction coordinates.

Privacy Infrastructure: We intentionally avoided external databases (like PostgreSQL or AWS). Instead, storing preferences in the encrypted chrome.storage.local sandbox and bundling financial dictionaries as static JSON files which we have no access to.

Challenges we ran into

Building a remote help tool within a browser extension presented a major ethical hurdle: security vs. control. Initially, we aimed for full remote control, but we quickly realized that giving a third party active control over a banking session was a massive security risk. We pivoted to a "Guide-only" model, developing a custom P2P system where helpers "suggest" actions via visual ripples and drawing on the host's screen. This keeps the user in the driver’s seat while providing clear guidance. Technically, this required building a robust WebRTC signaling server that maintains a "Zero-Knowledge" state and a semantic DOM-scraping algorithm to handle the inconsistent HTML structures found across different banking portals.

Accomplishments that we're proud of

We are incredibly proud of the technical milestones we reached during this hackathon, especially considering this was our first time building a Chrome Extension.

Zero-Knowledge Privacy: We successfully architected a feature-rich fintech tool that requires zero external API calls for data parsing. Keeping a user's sensitive financial life entirely on their own machine is a standard of privacy we are proud to uphold.

The "Guide-only" Pivot: We take great pride in our decision to move away from full remote control in favor of a visual guidance system. Implementing this required a more complex custom interaction layer, but it was the right choice for user safety and empowerment.

And that we spent countless hours together, working through the night to master WebRTC signaling. The sheer volume of features we were able to implement—from SVG color matrices to P2P data channels—is a testament to our team's synergy and dedication to making the web more accessible.

What we learned

Security as a Form of Care: Our pivot from a "full remote control" model to a "visual guidance" system taught us an invaluable lesson in digital ethics. We learned that true accessibility means empowering the user to stay in control, not just solving the problem for them.

The Power of Universal Design: Through our research into eye-health and dyslexia statistics, we learned that accessibility isn't just about "fixing" a site—it’s about creating multiple ways to perceive and interact with data. Implementing SVG color matrices and DOM-scraping taught us to look at the web through a more inclusive lens.

What's next for Fawn

We are incredibly passionate about this project and can't wait to keep working on it long after the hackathon ends. Our immediate goal is to refine the codebase for a public release on the Chrome Web Store, ensuring that Fawn can actually reach the audiences who need it most.