Find My Professors

Inspiration

As students, we often find ourselves stressing about the uncertainty of our academic achievement in the upcoming semesters. When choosing professors to take, especially for classes that many professors teach, we spend so much of our time copy-pasting, going back and forth between the course registration site and Rate My Professors. This is time wasted; instead of researching the professors' yourselves, let FindMyProfessors do it for you.

What it does

FindMyProfessors is a site that allows students to search for the professors by the course that they want to take, instead of by the professor. This will enable students to save time manually searching for the best professor by using our site which runs an in-depth rating analysis on each professor and presents all the relevant data to you. No more copy-pasting names into RateMyProfessors; use FindMyProfessors to find the best professors.

How we built it

Backend

Utilizing PostgreSQL, Go, and GraphQL (gqlgen) we were able to develop a GraphQL API that allows the frontend to query relevant data pertaining to schools, professors, courses, and ratings. Essentially, the frontend sends a request against the backend's predefined schema, in which the API gets all the requested data from the database, runs rating analysis as necessary, and returns the data to the frontend.

Scraper

The most challenging part of the project was implementing the web scraper. The scraper is broken down into two distinct components: the school scraper and the RMP scraper.

The school scraper is an abstraction that we use to scrape different schools while still getting the same data from each one.

We implemented the RMP scraper to collect all the data from a specific school on RMP.

Frontend

We built the frontend using the latest version of Next.js, Next 13. Thus, we used many of its latest features, including the new app directory, layouts, React server components, and @next/font. We also used Tailwind CSS to handle styling, which allowed us to quickly apply styles as we created each component of the site. To interact with the backend GraphQL API, we used the Apollo GraphQL Client on the frontend to execute queries in our server components. Lastly, to easily display the immense amounts of data we were getting from the back end in a nice fashion, we used React Table and Chart.js.

Challenges we ran into

Scraper

Each school has an entirely different system regarding getting the courses being taught and the professors teaching them, building this out was a difficulty in itself. How are we able to abstract all the data while still being able to be school agnostic? Not all schools format their data the same, and some schools do not provide an easy way of requesting that data.

Scraping 100s of thousands of distinct entities from RMP and the individual school sites provided us with so much data and no way to easily correlate the professors. Unfortunately, some professors have a different name on their school's website than on RMP. Ultimately, we wrote an algorithm that checks for name hyphenation and attempts to correlate professors based on name similarity. For most cases, this works; however, there are some cases where the name is just so unusually different, and in that case, the professor scraped from the school will have to be manually linked to their RMP.

Frontend

Because we were using the latest version of Next.js, we encountered numerous bugs during the development of the frontend. The biggest issue that we have encountered so far is not being able to perform GraphQL queries with the Apollo Client in Next.js client components, as it would never fetch the data and only return an infinite loading state. This problem made it extremely difficult to get the exact data we needed from the backend exactly when we wanted it. Luckily, data fetching on the server didn't seem to be an issue whatsoever, and actually surprised with its performance.

What's next for Find My Professors

• We would like to implement an account system that allows users to build the optimal schedule for classes they would like to take.
• Implement better caching that will allow for better scalability.
• Implement automatic scraping for RMP & Schools for each semester.
• Implement a CI/CD pipeline.
• Move hosting on a Kubernetes cluster.

Built With

• golang
• graphql
• nextjs
• postgresql
• react
• tailwindcss