fix: E2E test improvements and CEL error reporting#2448
fix: E2E test improvements and CEL error reporting#2448chmouel merged 4 commits intotektoncd:mainfrom
Conversation
387110b to
054c4c6
Compare
Summary of ChangesHello @chmouel, I'm Gemini Code Assist1! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed! This pull request addresses an issue where invalid Common Expression Language (CEL) expressions in PipelineRuns would lead to duplicate error comments being posted on Pull Requests. By introducing a control mechanism for error reporting within the Highlights
Changelog
Activity
Using Gemini Code AssistThe full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips. Invoking Gemini You can request assistance from Gemini at any point by creating a comment using either
Customization To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a Limitations & Feedback Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here. You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension. Footnotes
|
🔍 PR Lint Feedback
|
There was a problem hiding this comment.
Code Review
This pull request effectively addresses the issue of duplicate error comments for invalid CEL expressions by introducing a reportErrors flag to the MatchPipelinerunByAnnotation function. The implementation is clean and correctly applied at the different call sites. To ensure the new logic is robust and prevent future regressions, I've suggested adding a dedicated test case to verify the behavior of the reportErrors flag, with a refinement to avoid mutating shared state within the test.
There was a problem hiding this comment.
Pull request overview
This PR fixes a bug where CEL validation errors were being reported twice on pull requests. The issue occurred because MatchPipelinerunByAnnotation() was called multiple times during processing, and the first call (before template expansion) was also reporting errors even though it was just a preliminary check.
Changes:
- Added a
reportErrorsboolean parameter toMatchPipelinerunByAnnotation()to control whether CEL validation errors are reported - Updated the first call (preliminary check) to pass
reportErrors=falseto suppress error reporting - Updated subsequent calls and all tests to pass
reportErrors=trueto maintain existing behavior
Reviewed changes
Copilot reviewed 3 out of 3 changed files in this pull request and generated no comments.
| File | Description |
|---|---|
| pkg/matcher/annotation_matcher.go | Added reportErrors parameter to control CEL validation error reporting; only reports errors when parameter is true |
| pkg/pipelineascode/match.go | Updated all three calls to MatchPipelinerunByAnnotation(): first call passes false to suppress duplicate errors, other calls pass true |
| pkg/matcher/annotation_matcher_test.go | Updated test calls to pass reportErrors=true to maintain existing test behavior |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
|
❌ PR-Agent failed to apply 'global' repo settings The configuration file needs to be a valid TOML, please fix it. Error message: Configuration content:[github_app]
pr_commands = [ "/review" ]
[pr_reviewer]
enable_review_labels_security = true
enable_chat_in_code_suggestions=false
enable_chat_in_code_suggestions = false
enable_summary=false
enable_help_text=false
[pr_description]
publish_description_as_comment = true
[pr_code_suggestions]
commitable_code_suggestions = true
[pr_test]
num_tests = 0 |
|
/review |
PR Reviewer Guide 🔍Here are some key observations to aid the review process:
|
340f39a to
eebf159
Compare
Added a `reportErrors` parameter to `MatchPipelinerunByAnnotation` to allow callers to control CEL validation error reporting. Updated `getPipelineRunsFromRepo` to initially match with reporting disabled and then enable it for the final matching, preventing redundant error logs during the process. Tests were also modified to accommodate this new behavior. Signed-off-by: Chmouel Boudjnah <chmouel@redhat.com>
Modified the `NSTearDown` function to append a random suffix to repository URLs before deleting them. This ensures that any outstanding webhooks targeting these repositories do not trigger new runs during the teardown process, preventing potential interference or errors. The change was made because previously, repositories were deleted directly, which could lead to race conditions where webhooks were still active and attempted to create new pipeline runs in a namespace that was being cleaned up. Signed-off-by: Chmouel Boudjnah <chmouel@redhat.com>
a920380 to
26d80cf
Compare
Added extensive debug logging throughout the pipelineascode and related packages. This includes: * Enhanced logging for the matching process, including details on pipelinerun evaluation, annotation checks, and CEL expression evaluation. * Detailed logging for remote resource fetching, including the source (HTTP, catalog, repository) and type of resource. * Improved visibility into the cancellation of pipeline runs, with logs for enabled/disabled states and the selectors used. * More granular logging for client setup, including secret handling and webhook validation. * Logging for access checks and error reporting, providing context on sender and validation outcomes. * Detailed logging for repository and pipeline run matching, including URL matching and commit info fetching. * Comprehensive logging for fetching pipeline configurations, including template parsing and annotation processing. * Logging for the `makeTemplate` function, indicating the number of resolved parameters and changed file groups. * Detailed logging within the `resolve` package, covering the reading of Tekton types, resolution of remote resources, and metadata resolution. * Logging for the `startPR` function, detailing pipelinerun creation, secret management, status updates, and patching operations. * Logging for secret retrieval processes, including details on the repository, provider URL, and secret keys used.
A new job, `e2e-flaky-tests`, was added to the CI workflow to specifically run tests that have previously exhibited flaky behavior. This job runs after the main E2E tests and uses the `always()` condition to ensure it executes even if earlier jobs fail, facilitating the collection of logs for debugging. The `hack/gh-workflow-ci.sh` script was updated to recognize and filter tests tagged as "Flaky". Additionally, test function names in `test/github_pullrequest_test.go` were updated to include "Flaky" to align with this new categorization and allow the script to identify them correctly. This change aims to improve the reliability of the E2E test suite by isolating and addressing flaky tests. Signed-off-by: Chmouel Boudjnah <chmouel@redhat.com>
26d80cf to
7624805
Compare
|
/gemini review |
There was a problem hiding this comment.
Code Review
This pull request introduces several valuable improvements to the E2E testing infrastructure and fixes a bug in CEL error reporting. The change to prevent duplicate CEL validation errors by adding a reportErrors flag to MatchPipelinerunByAnnotation is a clean solution. Moving flaky tests to a dedicated CI job is a good practice for CI stability. The addition of extensive debug logging throughout the codebase will be very helpful for future troubleshooting. The fix in the E2E test teardown to modify repository URLs for better log collection is also a good improvement. I've found one area for improvement in the teardown logic related to a common Go pitfall with loop variables, which I've detailed in a specific comment.
📝 Description of the Change
This PR contains several improvements to the E2E test infrastructure and fixes:
1. Only report CEL validation errors once When someone writes a bad CEL expression in their PipelineRun, the system was posting 2 identical error comments on the PR instead of just 1.
The function
MatchPipelinerunByAnnotation()gets called twice during processing:Added a
reportErrorsparameter to control when errors are reported, preventing duplicate comments.2. Move flaky tests to dedicated CI job
e2e-flaky-testsjob that runs after main E2E tests complete_Z_prefix toFlakynaming convention (e.g.,TestGithubFlakyPullRequestBadYaml)t.Skip()calls so these tests actually runhack/gh-workflow-ci.shto handle the newflakytarget3. Add debug logging for pipelineascode
Added extensive debug logging throughout matching, remote resource fetching, cancellation, client setup, and more.
4. Fix repo URL for log collection
Modified
NSTearDownto append a random suffix to repository URLs before deletion, preventing race conditions with active webhooks.🚀 Type of Change
fix:)feat:)feat!:,fix!:)docs:)chore:)refactor:)enhance:)deps:)🧪 Testing Strategy
🤖 AI Assistance
Which LLM was used?
Extent of AI Assistance:
✅ Submitter Checklist
make testandmake lintlocally.