MIRNet-TFJS

Inspiration

Just another day, I was browsing through arXiv and I saw the Learning Enriched Features for Real Image Restoration and Enhancement paper by Zamir et al. It seemed quite interesting to me and I read it through. I was pretty fascinated by the idea shared in the paper for the task of image super-resolution and allow to generate spatially robust images. This motivated me to try and implement the project.

What it does

This project shows the TFJS model conversion and inference processes for the MIRNet model as proposed by Learning Enriched Features for Real Image Restoration and Enhancement by Zamir et al. This model is capable of enhancing low-light images up to a great extent.

How we built it

I trained the MIRNet model in Python and then converted it to TFJS format to be able to share it with all through the web. I also built a minimalistic website integrating the model so others could try it out for themselves.

A bit about the architecture

The MIRNet presents a novel architecture with the collective goals of maintaining high-resolution representations through the entire network, and receiving strong contextual information from the low-resolution representations.

The core of this approach is a multi-scale residual block containing the following key elements:

• parallel multi-resolution convolution streams for extracting multi-scale features
• information exchange across the multi-resolution streams
• spatial and channel attention mechanisms for capturing contextual information
• attention-based multi-scale feature aggregation.

Framework of MIRNet

The figure above shows the framework of the proposed network MIRNet that learns enriched feature representations for image restoration and enhancement.

Challenges we ran into

One of the major challenges was to use such a heavy model on the web. The model is almost 152 MB, which is pretty huge to load on the web and then perform inferences on the client-side. To allow people to still try this out with installing a single thing via a website I decided to infer on the server-side. The first day I made it public, there were almost 500 requests to the server after my tweet to do such heavy computation. Not only, did the server crash but I exhausted all of Azure's free tier and promotional credits. I started getting issues and messages saying the website is down. I had to work a lot that day itself to optimize the workloads, request Azure for some more free student credits, and voila then it is was up!

Another major challenge I faced was about inference times, I planned to do the computation on the server side so folks can try using the heavy model even on their mobile phones! But this sometimes resulted in pretty high inference times, and I again had to work on that to allow folks to try it out.

Accomplishments that we're proud of

• 246 stars (250 would be lovely for the moment)
• Invited to talk about the project in detail on Google TensorFlow's YouTube channel
• Accepted to be published on TensorFlow Hub

What we learned

• managing LFS objects
• learned Express

What's next for MIRNet-TFJS

As I mentioned as of now we only develop a minimal website:

I am looking for contributors who would love to help me with the UI, not having any experience of UI development I could only come up with such a simple and minimalistic UI.

Built With

• azure-web-app
• busyboy
• colab
• css3
• express.js
• github-actions
• html5
• javascript
• node.js
• python
• tensorflow
• tensorflow-hub
• tfjs
• tfjs-converter