GitGauge

Our initial dashboard

Inspiration

If you're a student recruiting for a tech role, you've likely heard of CodeSignal or Leetcode. These programs are a company's way of evaluating a candidate's technical skill. However, these methods of assessments often fail to capture a completely accurate picture of an individual's coding skills. Performance on a sole online assessment does not accurately depict abilities. In addition, job postings often ask applicants to include a GitHub link with their application in order to glean insight into the applicant's past projects.

We sought to operationalize GitHub into metrics available to a recruiter, in addition to already existing programming assessments. CodeSignal and Leetcode give numeric, data-driven evaluations of performance, but examining GitHub projects can paint a more holistic picture of applicant skills.

What it does

From a GitHub link, it assesses candidate strengths in technical fields that are specified by the recruiter. GitGauge then outputs levels of strength in the corresponding technical fields selected for examination. A user can additionally prompt GitGauge in other areas of inquiry about the applicant, and from there more evaluations can generate.

One use case might be: If a sophomore in college has participated in hackathons and worked on developing apps from front to end, however lacks much formal education in algorithmic coding (often the content of Leetcode or CodeSignal interviews). This applicant is already at a significant disadvantage as opposed to a student who is able to have higher education and be trained in these particular skills. Here, online assessment not fully encapsulate skills. Not only is this a more accurate evaluation method, but is more speedy for recruiters (as opposed to having to search through a GitHub project page and lengthy blocks of code).

How we built it

API from GitHub retrieve data, extract relevant code with TogetherAI, LLM for chatbox, algorithm to compute value.

Challenges we ran into

Putting the scheme into action was tough. It seemed clear and doable when we looked at broadly, but we lost a bit of sleep over it! Technical challenges: One challenge we ran into was speed of analyzing all the code. Analyzing everything in sequence was too time consuming, but doing them all in parallel was too resource-intensive. We settled on a compromise of processing queries and repositories in batches, a few per inference call.

Another challenge was fetching the user's data from GitHub. We couldn't just fetch all the user's repos and cloning big repos took too long. To solve this, we had to learn GraphQL to fetch the preliminary data, and we solved the issue of cloning efficiently using git sparse cloning, only cloning files our LLMs recommended.

Accomplishments that we're proud of

One thing we're proud of is how fast we were able to learn all the skills to build the project. Before this, we had next to no experience with FastAPI, GraphQL, Together.ai, Docker, and automating filesystem manipulation.

Another thing we're proud of is the real time updates while the backend is doing the processing. To create these, we connected to Firebase and subscribed to a doc with changing update statuses, all in the span of an hour.

What we learned

GraphQL, FastAPI, Together.ai, Docker

What's next for GitGauge

Improve the relevant code finding algorithm. Right now, the algorithm is fed lots of code and it picks out the relevant ones. In the future, we should be able to give the algorithm a file and have it search for the relevant code by itself.

Improve the speed. This can be done by parallelizing more of the computations, and refining our prompts.