Neuro Art

Art for the non-technical nerds

Comment

Inspiration

We were inspired by projects which have been done in the past - many stunning examples of NST in action have made their way into the public eye, but there are few accessible implementations of this for end users and beginners. We felt that building an API to provide this model to non-experts was a great way to introduce people to deep learning in a fun, interactive way.

What it does

This project provides two things:

  • An API for image processing
  • A front end application for end users

When users interact with our application via the front end, they can upload an image and process it using deep learning to style it based on the style of a famous artist. The API can also be used directly via POST and PUT requests.

How we built it

Our project mainly consists of three parts. The back end contains the mechanics to interact with TensorFlow to process our images using NST, as well as a Flask server with which to access this functionality via a RESTful API.

The front end is a React app which provides a user-friendly interface capable of interacting with our back end and database via HTTP requests. Before any coding, we made a Figma mock up.

The database and file storage are provided by a Firebase project using Firestore and Firebase Storage, using the Python 3 Firebase SDK.

Challenges we ran into

  • One of our goals to begin with were using BentoML, but we ran into an issue which turned out to be a know bug. We're looking at following up with their team about this, as storing our model and serving it with their library would be very useful.
  • Getting TensorFlow 2 to work with our local machine was quite a pain, and resulted in us using some experimental features which are a known workaround for NVIDIA compatibility.
  • Finding the right balance of speed and quality was an interesting decision to make, and we're still fine tuning it on our own time.
  • We were not able to deploy this project anywhere because hosting providers with enough power to run this software are quite costly, so we settled on running it on and off on a local computer with a high powered graphics card.

Accomplishments that we're proud of

Our results look pretty good! Initially when we were running this service on a CPU, our results took over 2 minutes to process for mediocre results. After working out GPU compatibility, we cut that down to under 20 seconds for much better looking images.

Our API is very usable with POST requests, and can be used with the front end or standalone.

Our front end interface looks great, and interfaces with our back end relatively seamlessly. The user experience does not belie the complex nature of the back end, and our process is fast enough so as not to be a chore.

What we learned

Our team learned a lot about deploying deep learning models, about Firebase, and about developing a useful, helpful set of documentation. Our GitHub is full of comments and Wiki articles which help ourselves and others understand our code and facilitate development moving forward.

Engineering focus: documentation

We did some great job keeping track of our development process by creating various GitHub Projects (Global stuff, backend, frontend, ai model), making issues, writing READMEs through out the project, and setting up the repository Wikis.

As well we make sure to the main files in the backend and frontend codebases are commented.

About deployment

Why are there no other projects like this out there in the wild? Because of GPUs. Computing each image is both time and power consuming, that the free tier in popular deploy services (e.g. GCP) falls short quickly. We weren't able to find better alternatives by the constraints of time of the sprint.

What's next for Neuro Art

In the future, we'd like to try a few interesting things:

  • A PyTorch implementation
  • BentoML integration. Currently, facing this issue.
  • Hyperparameter access at a higher level (via config files and/or client side input)
  • Scaling our server program to take multiple requests simultaneously.
  • Add a gallery section with some cool NST results (this might have some legal considerations first!).

We'll definitely be adding to this project in the future, and we can't wait to see what the community thinks.

Built With

Share this project:

Updates