ScholarLink

Unlock the Power of Knowledge: ScholarLink - Your Personalized Research Paper Recommender

Comment

Inspiration

Research papers are invaluable for staying up-to-date in any field. However, research paper publishing platforms primarily cater to researchers and academics, often lacking user-friendliness for the general public. When individuals seek to explore a particular field, they typically face the arduous task of searching for specific research papers, often resulting in decision fatigue due to the overwhelming number of options presented by the platform. I propose the development of a service that recommends research papers tailored to readers’ specific interests. Such a service would not only connect readers with the most relevant resources but also enhance the accessibility of cutting-edge research to the general public.

What it does

ScholarLink is a user-friendly web application that revolutionizes the way individuals access research papers. By simply providing their interests and subscribing to the service, users unlock a world of knowledge. ScholarLink leverages this information to fetch the most relevant research papers from Springer, a renowned research publication platform. Once a week, subscribers receive a personalized selection of recommended research papers straight to their inboxes.

How I built it

ScholarLink is developed using the MERN stack and designed with a service-oriented architecture. The front end is built using React.js. The web server is implemented in Node.js and Express.js and incorporates essential features such as built-in rate limiting and email validations. The Recommendation Server is developed in Python and uses multi-threading to optimize recommendation processing. The Mailing Server is built with Node.js and utilizes cron jobs and SMTP to ensure the timely delivery of weekly recommended papers directly to the subscriber’s inbox.

To ensure smooth and efficient processing, ScholarLink incorporates a worker queue powered by RabbitMQ mainly between the web server and the recommendation server. This facilitates the asynchronous processing of tasks, enhancing performance and responsiveness. The application’s data is stored and managed using MongoDB, a flexible and scalable NoSQL database.

In crafting the recommendation algorithm, I devised a strategy that incorporates all of the subscriber’s interests equally and selects a research paper published within the past six years.

For simplified deployment and management, ScholarLink can be easily deployed as a multi-container application using Docker. This containerization approach streamlines the setup process, promotes scalability, and enhances the overall portability of the application.

Tech Stack

  • Client: React.js
  • Web Server: Node.js, Express.js, Javascript, cache, rate limiter
  • Database: MongoDB
  • Work Queue: RabbitMQ
  • Recommendation Server: Python, cache, Springer API
  • Mailing Server: Node.js, Javascript, cron, SMTP
  • DevOps: Docker

Scalabilities

  • The Web Server, Recommendation Server, Mailing Server, and Work Queue are all stateless and can be horizontally scaled to accommodate for the increase in volume

Optimizations

  • Rate Limiter: to optimize the performance of a system resource and ensure equal service distribution
  • Web Server cache: cache user interests and interests options

Challenges I ran into

During the development of ScholarLink, I encountered several challenges that required careful consideration. Firstly, I spent a lot of time designing the system with scalability in mind, anticipating potential issues as the user base grows. To address this, I focus on a stateless approach and implemented an asynchronous worker queue, which efficiently processes recommendations.

Another significant challenge I faced was determining the appropriate interest options and creating a robust recommendation system based on these interests. It was crucial to ensure that the recommended research papers align closely with each user’s specific interests.

When selecting a research paper publishing platform, I opted for Springer due to its superior API support. However, I encountered limitations imposed by many research paper platforms, which constrained the usage of their APIs. This made it challenging to build an optimized recommendation system that could fetch and analyze a substantial number of papers. Despite these limitations, I strived to create the most effective recommendation system within the confines of the available resources and API restrictions.

Accomplishments that I am proud of

I am proud of simply starting and finishing a full-stack web application and also being able to build an impactful product that bridges the gap between the readers and valuable intellectual resources and in turn, increases the exposure of cutting-edge research to the wider general public.

What I learned

During the development process, I learned a lot about designing a scalable and fault-tolerant system, using new technologies or libraries that align with my need, understanding the workings of a recommendation system, and also understandings how most research paper publishing platform organized their contents.

What's next for ScholarLink

While ScholarLink currently functions as a working prototype, several key areas require further attention, including the addition of new features, optimizations, security enhancement, thorough testing, comprehensive documentation, etc. By leveraging paid API usage with Springer, ScholarLink can access more extensive datasets and tap into advanced algorithms, resulting in even more precise and tailored recommendations for subscribers. As part of the weekly email delivery, users will have the option to participate in a survey regarding the recommended research paper from the previous week. This survey will provide a platform for users to share their thoughts, opinions, and insights on the recommended paper. By actively engaging users in this feedback process, ScholarLink gains valuable data that can be utilized to refine and improve the recommendation system.

Share this project:

Updates