[PROPOSAL] Integrate Automated AI-based Code Reviews #342
Description
What/Why
What are you proposing?
LLMs help with many parts of the tedious development work, and they can be used for code reviews as well. This issue proposes the org to add a GitHub apps that leverage AI to automatically go through each pull request diff and leaves comments with potential problems and fixes, and repos that are willing to onboard can use them to help with code reviews.
What users have asked for this feature?
N/A, this is for developers not users.
What problems are you trying to solve?
This helps developers to have automated checks on their PR to catch early issues and speed up code review process.
What is the developer experience going to be?
Developers will send and review pull requests as usual, this does not replace any parts of the process. It only adds additional comments on the PR.
Are there any security considerations?
No, all pull requests are public.
Are there any breaking changes to the API
No
What is the user experience going to be?
No change on user experience
Are there breaking changes to the User Experience?
No change on user experience
Why should it be built? Any reason not to?
It should be built because it should simplify code reviews and improve code quality. It may introduce additional noise or incorrect information due to model and tool limitations, we can disable it if it's not performing well and I think it's worth a try.
What will it take to execute?
Onboarding depends on the solution we use. Most likely we will add some secrets and a GitHub action workflow in repos that are onboarding.
Any remaining open questions?
There are different providers, I prefer the amazon-q-developer app because it's free to try. Here are some I looked into
- https://github.com/apps/amazon-q-developer: preferred as this is free to use for now
- https://docs.github.com/en/copilot/using-github-copilot/code-review/using-copilot-code-review: this requires premium requests with monthly per-person quota
- https://github.com/qodo-ai/pr-agent: this is an open source solution that requires custom API providers