PseudoGrader

Test cases execution results
Logical evaluation (similar algorithms)
Logical evaluation (feedback)
Converted code
Running the process
Editing the pseudocode

Inspiration

Our inspiration came from noticing the frustration many students face when dealing with unclear grading criteria and inconsistent feedback on coding assignments. It also came from imagining the inconvenience of reading and validating incomprehensible and error-filled pseudocode for TAs and professors. We wanted to ease each party's troubles by creating a website that automatically validates and tests all pseudocode making everyones' lives easier through the use of powerful AI tools.

What it does

We created a simple and sleek website that validates and tests pseudocode with detail-oriented test cases and logic checks and outputting in-depth logic evaluations which critiques the logic of the pseudocode and the results of test cases showing which tests passed and which tests failed and where these tests failed and from what. It also displays translated code from your pseudocode to be tested. It displays all of these through taking in files, pdfs, images/scans of the pseudocode and the website will automatically read it and it also takes in the instructions/context behind the code or what it's TRYING to do.

How we built it

For the frontend, we utilized Figma to create a structured design skeleton that outlined the intended functionality and user experience of our website. Additionally, we crafted mind maps and detailed diagrams to clearly define how frontend and backend components would interact, ensuring smooth communication and logic flow between both ends of the application. Our primary development tools included Visual Studio Code for programming, Git for collaborative version control, and Discord for team communication. The frontend was built using Tailwind CSS along with TypeScript to streamline styling and maintain a cohesive and responsive user interface.

The backend architecture was developed using FastAPI, designed to handle incoming files that represent question contexts and associated pseudocode. These files could be either text or images. For image-based pseudocode, we integrated Google Vision AI to accurately extract text content. Once extracted, the pseudocode was forwarded to Gemini and Cohere APIs to generate executable Python code and corresponding Python test cases. These test cases were executed in isolated Docker containers for safe and consistent evaluation. Furthermore, our backend included an advanced pipeline leveraging ChromaDB, a vector database housing embeddings of over 1000 algorithms and their descriptions generated by the Cohere-embed model. This pipeline assessed similarity between the submitted question context and pseudocode against these stored algorithms, identifying closely related algorithms. Subsequently, these results, along with the initial pseudocode, were evaluated through a Cohere model, providing a comprehensive logic evaluation to ensure correctness, efficiency, and readability of the pseudocode.

Challenges we ran into

There were many issues when setting up APIs with LLM hallucinations. These hallucinations occasionally produced inaccurate or irrelevant code snippets and test cases, complicating debugging and validation processes. We had to implement additional verification and filtering steps to mitigate these inaccuracies, ensuring the quality and reliability of the generated outputs.

There were also a lot of small issues when it came to tweaking small things with the design like with scrolling boxes, and larger issues like restructuring previously created code for more efficient solutions while ensuring the current design and functionality do not change drastically.

Accomplishments that we're proud of

Our implementation of routing converted code to the testing API and then reading the results to display passed and failed testing and the details about each tests. The aesthetic and accuracy of this section is exactly to our preference and it came out really well. Our front-end routing and design overall is something we're very proud of because it turned out how we hoped it would, and our backend works exactly how we intended it to ensuring smooth function of our entire website.

What we learned

Many of us are new to working with APIs and AI, and throughout the creation of PseudoGrader, our usage of AI and developer tools along with putting together our knowledge of routing and using APIs to create website, we learned a lot about backend and frontend routing and routing logic, setting up APIs, and overall how to effectively apply these tools to create sleek and optimized programs and applications for real world scenarios.

What's next for PseudoGrader

We plan to branch out and create options to not only aid teachers in grading code, but also aid in grading mathematics like proofs, calculations, reasoning, and biology which includes creating reports or essays, and other science and mathematic based fields.