Patch Analyzer

Pre-commit output, displaying feedback to developer.
Editor, to directly apply suggestions and feedback

Patch Analyzer: A Collaborative Venture in AI-Enhanced Commit Messages Inspiration

As a team of two passionate developers, we were inspired to create Patch Analyzer upon recognizing a common challenge in software development: the often overlooked quality of meaningful commit messages. Observing how unclear or context-lacking messages could hinder project documentation and collaboration, we saw an opportunity to make a difference. Our shared interest in the capabilities of OpenAI's GPT models led us to conceptualize a tool that could assist developers in writing better commit messages, leveraging the power of AI.

What We Learned

The journey of developing Patch Analyzer was rich with learning experiences:

Python: Enhancing our Python skills was essential, as the backbone of our tool is written in this language.
Pre-commit: We explored the intricacies of using pre-commit in the Git Workflow, learning how to effectively intercept and process commit messages.
AI Integration and Ethics: Integrating OpenAI's GPT-4 Turbo model was not just a technical challenge; it also involved understanding ethical considerations in AI usage.
User-Centric Design: We focused on creating a user-friendly interaction between the developers and the AI model, which taught us valuable lessons in user experience design, especially in a CLI environment.

How We Built It

The development of Patch Analyzer involved several collaborative steps:

Setting Up the Pre-commit Hook: Our first step was establishing a basic hook for pre-commit, using Python. We want need to edit the commit msg before the commit is finished without interrupting the process.
Integrating the AI Model: We then integrated the OpenAI GPT-4 Turbo model to assess and provide feedback on commit messages. We also use openAI's function calling capability to enforce a JSON format that can be represented in Python Pydantic models.
Developing the Feedback Mechanism: We implemented a system for the tool to interact with the user, presenting AI-generated feedback and suggestions.
Facilitating Message Editing: An integral part of our tool was allowing users to edit their commit messages based on AI suggestions directly in their usual workflow.
Automating Message Replacement: We ensured that any revised commit message was automatically updated as the new commit message.

Challenges We Faced

Relevance of AI Feedback: One of the biggest challenges was ensuring that the AI provided relevant and contextually appropriate feedback. This required continuous tweaking and understanding of the AI model's capabilities.
Designing a User-Friendly CLI Interface: Crafting an intuitive command-line interface that was both user-friendly and functional presented its own set of challenges.
Optimizing Performance: We worked diligently to balance the tool's performance, particularly in terms of the AI model's response time, to ensure a seamless user experience.
Extensive Testing: As a team, we conducted thorough testing to cover various scenarios and ensure the reliability and robustness of our tool.

Examples

Hello JetBrains

  def main():
-     print("Hello World!")
+     print("Hello JetBrains!")

git commit -m "Change main.py"

Analyze Patch............................................................
=======================  ===========================================================================
Completeness             ⚠️ The commit message 'Change main.py' does not correctly reflect the
                         changes made in the git diff. The changes are actually in 'test.py', not
                         'main.py', and the message gives no indication of what was changed
                         within the file.
=======================  ===========================================================================
Clarity and Conciseness  ⚠️ The commit message is too vague and does not describe the nature of the
                         change. The message 'Change main.py' could refer to any number of
                         changes and does not convey the actual content updated in 'test.py'.
=======================  ===========================================================================
Relevance and Context    ⚠️ There is no context provided in the commit message to explain why the
                         print statement was changed from 'Hello World!' to 'Hello JetBrains!'.
                         Furthermore, the message references an incorrect file.
=======================  ===========================================================================
Grammar and Spelling     ✅ .
=======================  ===========================================================================
Specificity              ⚠️ The commit message lacks specificity; it does not specify which file was
                         changed or the nature of the change. A more specific message would help
                         identify the purpose and scope of the changes made.
=======================  ===========================================================================
Do you want to edit the commit message? [y/N]:

Simple input / Hello World example

+ user = input("Enter your username")
+ print("Hello Bernd")

git commit -m "Welcome user with their username"

Analyze Patch............................................................
=======================  ===========================================================================
Completeness             ⚠️ The commit message does not mention that the code change hardcodes the
                         greeting for a specific username, which does not match the feature
                         described in the message.
=======================  ===========================================================================
Clarity and Conciseness  ⚠️ The message is ambiguous as it suggests a dynamic welcome message, while
                         the code change is static and always greets the same username.
=======================  ===========================================================================
Relevance and Context    ⚠️ The commit message does not provide context indicating the code is not
                         finished or the hardcoded value is just for testing purposes.
=======================  ===========================================================================
Grammar and Spelling     ✅ .
=======================  ===========================================================================
Specificity              ⚠️ The message is too vague. It should specify that the current
                         implementation only welcomes a specific user and does not yet use the
                         username variable.
=======================  ===========================================================================
Do you want to edit the commit message? [y/N]:

Function calling with typo

  def foo():
      print("This is the 'foo' function.")
-     bar()
+     foo()

  def bar():
      print("This is the 'bar' function.")

  foo() # Calls foo() -> bar()

git commit -m "Exit method instead of calling bar"

Analyze Patch............................................................
=======================  ===========================================================================
Completeness             ⚠️ The commit message 'Exit method instead of calling bar' suggests that
                         the function 'bar' should not be called and the method should exit
                         instead. However, the diff shows that within the 'foo' function, the
                         call to 'bar()' is replaced by a recursive call to 'foo()' itself, which
                         does not align with the commit message.
=======================  ===========================================================================
Clarity and Conciseness  ⚠️ The commit message is not clear because it suggests an action (exiting a
                         method) that does not correspond to the actual changes in the diff. The
                         diff shows a change that can lead to infinite recursion, which is likely
                         unintended and should be clarified.
=======================  ===========================================================================
Relevance and Context    ⚠️ The commit message does not accurately represent the changes in the
                         diff, therefore lacking relevance. Additionally, there is no context as
                         to why this change is being made, such as a bug fix or a feature
                         implementation, which should be included in the message.
=======================  ===========================================================================
Grammar and Spelling     ✅ .
=======================  ===========================================================================
Specificity              ⚠️ The commit message fails to be specific about the actual change, which
                         is replacing the call to 'bar()' with a recursive call to 'foo()'.
                         Specificity is crucial for understanding the intention behind the code
                         modification.
=======================  ===========================================================================
Do you want to edit the commit message? [y/N]: