Skip to content

fix: E2E test improvements and CEL error reporting#2448

Merged
chmouel merged 4 commits intotektoncd:mainfrom
chmouel:match-annotation-only-report-error-once
Feb 4, 2026
Merged

fix: E2E test improvements and CEL error reporting#2448
chmouel merged 4 commits intotektoncd:mainfrom
chmouel:match-annotation-only-report-error-once

Conversation

@chmouel
Copy link
Copy Markdown
Member

@chmouel chmouel commented Feb 4, 2026

📝 Description of the Change

This PR contains several improvements to the E2E test infrastructure and fixes:

1. Only report CEL validation errors once When someone writes a bad CEL expression in their PipelineRun, the system was posting 2 identical error comments on the PR instead of just 1.

The function MatchPipelinerunByAnnotation() gets called twice during processing:

  • First call - Quick check before processing templates
  • Second call - Real check after templates are ready

Added a reportErrors parameter to control when errors are reported, preventing duplicate comments.

2. Move flaky tests to dedicated CI job

  • Added new e2e-flaky-tests job that runs after main E2E tests complete
  • Renamed tests from _Z_ prefix to Flaky naming convention (e.g., TestGithubFlakyPullRequestBadYaml)
  • Removed t.Skip() calls so these tests actually run
  • Updated hack/gh-workflow-ci.sh to handle the new flaky target

3. Add debug logging for pipelineascode

Added extensive debug logging throughout matching, remote resource fetching, cancellation, client setup, and more.

4. Fix repo URL for log collection

Modified NSTearDown to append a random suffix to repository URLs before deletion, preventing race conditions with active webhooks.

🚀 Type of Change

  • 🐛 Bug fix (fix:)
  • ✨ New feature (feat:)
  • 💥 Breaking change (feat!:, fix!:)
  • 📚 Documentation update (docs:)
  • ⚙️ Chore (chore:)
  • 💅 Refactor (refactor:)
  • 🔧 Enhancement (enhance:)
  • 📦 Dependency update (deps:)

🧪 Testing Strategy

  • Unit tests
  • Integration tests
  • End-to-end tests
  • Manual testing
  • Not Applicable

🤖 AI Assistance

  • I have not used any AI assistance for this PR.
  • I have used AI assistance for this PR.

Which LLM was used?

  • GitHub Copilot
  • ChatGPT (OpenAI)
  • Claude (Anthropic)
  • Cursor
  • Gemini (Google)
  • Other: ____________

Extent of AI Assistance:

  • Documentation and research only
  • Unit tests or E2E tests only
  • Code generation (parts of the code)
  • Full code generation (most of the PR)
  • PR description and comments
  • Commit message(s)

✅ Submitter Checklist

  • 📝 My commit messages are clear, informative, and follow the project's commit message guide.
  • ✨ I have ensured my commit message prefix matches the "Type of Change" selected above.
  • ♽ I have run make test and make lint locally.
  • 📖 I have added or updated documentation for any user-facing changes.
  • 🧪 I have added sufficient unit tests for my code changes.
  • 🎁 I have added end-to-end tests where feasible.
  • 🔎 I have addressed any CI test flakiness or provided a clear reason to bypass it.

Copilot AI review requested due to automatic review settings February 4, 2026 10:45
@chmouel chmouel force-pushed the match-annotation-only-report-error-once branch from 387110b to 054c4c6 Compare February 4, 2026 10:46
@gemini-code-assist
Copy link
Copy Markdown

Summary of Changes

Hello @chmouel, I'm Gemini Code Assist1! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request addresses an issue where invalid Common Expression Language (CEL) expressions in PipelineRuns would lead to duplicate error comments being posted on Pull Requests. By introducing a control mechanism for error reporting within the MatchPipelinerunByAnnotation function, the change ensures that only a single, relevant error notification is provided to users, significantly improving the clarity and reducing noise in PR feedback.

Highlights

  • Duplicate CEL Validation Errors: Previously, when a PipelineRun contained an invalid CEL expression, the system would post two identical error comments on the Pull Request. This was due to the MatchPipelinerunByAnnotation() function being invoked twice during the processing flow.
  • Controlled Error Reporting: A new boolean parameter, reportErrors, has been introduced to the MatchPipelinerunByAnnotation() function. This parameter provides an explicit switch to control whether CEL validation errors are reported to GitHub, ensuring that error comments are posted only once at the appropriate stage of processing.
Changelog
  • pkg/matcher/annotation_matcher.go
    • Modified the MatchPipelinerunByAnnotation function signature to accept a new reportErrors boolean parameter.
    • Updated the error reporting logic within MatchPipelinerunByAnnotation to conditionally report CEL validation errors only when reportErrors is true.
  • pkg/matcher/annotation_matcher_test.go
    • Updated existing test cases for MatchPipelinerunByAnnotation to pass true for the newly added reportErrors parameter, maintaining current test behavior.
  • pkg/pipelineascode/match.go
    • Modified the initial, early call to matcher.MatchPipelinerunByAnnotation to pass false for reportErrors, preventing premature error comments.
    • Modified subsequent, real calls to matcher.MatchPipelinerunByAnnotation to pass true for reportErrors, ensuring errors are reported at the correct processing stage.
Activity
  • No specific activity (comments, reviews, etc.) has been recorded for this pull request yet.
Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature Command Description
Code Review /gemini review Performs a code review for the current pull request in its current state.
Pull Request Summary /gemini summary Provides a summary of the current pull request in its current state.
Comment @gemini-code-assist Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help /gemini help Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Footnotes

  1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution.

@pipelines-as-code
Copy link
Copy Markdown

pipelines-as-code bot commented Feb 4, 2026

🔍 PR Lint Feedback

Note: This automated check helps ensure your PR follows our contribution guidelines.

⚠️ Items that need attention:

🎫 Jira reference

Add a Jira reference in the description using one of the following formats:

  • https://issues.redhat.com/browse/SRVKP-<number>

If no SRVKP ticket exists yet, link a GitHub issue instead (e.g., Fixes #123).
Minor housekeeping PRs without Jira coverage can skip this after confirming with reviewers.


🤖 AI attribution

The following commits lack an explicit AI attribution footer:

  • 4d7c9e3 fix: Only report CEL validation errors once
  • ee64d0c fix: modify repo URL for log collection
  • 8a242d4 feat: Add debug logging for pipelineascode
  • 7624805 test: Move flaky tests to its own CI job

If no AI assistance was used for a commit, you can ignore this warning.
Otherwise add an Assisted-by: or Co-authored-by: footer referencing the AI used.


ℹ️ Next Steps

  • Review and address the items above
  • Push new commits to update this PR
  • This comment will be automatically updated when issues are resolved
🔧 Admin Tools (click to expand)

Automated Issue/Ticket Creation:

  • /issue-create - Generate a GitHub issue from this PR content using AI
  • /jira-create - Create a SRVKP Jira ticket from this PR content using AI

⚠️ Important: Always review and edit generated content before finalizing tickets/issues.
The AI-generated content should be used as a starting point and may need adjustments.

These commands are available to maintainers and will post the generated content as PR comments for review.

🤖 This feedback was generated automatically by the PR CI system

Copy link
Copy Markdown

@gemini-code-assist gemini-code-assist bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request effectively addresses the issue of duplicate error comments for invalid CEL expressions by introducing a reportErrors flag to the MatchPipelinerunByAnnotation function. The implementation is clean and correctly applied at the different call sites. To ensure the new logic is robust and prevent future regressions, I've suggested adding a dedicated test case to verify the behavior of the reportErrors flag, with a refinement to avoid mutating shared state within the test.

Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR fixes a bug where CEL validation errors were being reported twice on pull requests. The issue occurred because MatchPipelinerunByAnnotation() was called multiple times during processing, and the first call (before template expansion) was also reporting errors even though it was just a preliminary check.

Changes:

  • Added a reportErrors boolean parameter to MatchPipelinerunByAnnotation() to control whether CEL validation errors are reported
  • Updated the first call (preliminary check) to pass reportErrors=false to suppress error reporting
  • Updated subsequent calls and all tests to pass reportErrors=true to maintain existing behavior

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated no comments.

File Description
pkg/matcher/annotation_matcher.go Added reportErrors parameter to control CEL validation error reporting; only reports errors when parameter is true
pkg/pipelineascode/match.go Updated all three calls to MatchPipelinerunByAnnotation(): first call passes false to suppress duplicate errors, other calls pass true
pkg/matcher/annotation_matcher_test.go Updated test calls to pass reportErrors=true to maintain existing test behavior

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

@qodo-code-review
Copy link
Copy Markdown

qodo-code-review bot commented Feb 4, 2026

PR-Agent failed to apply 'global' repo settings

The configuration file needs to be a valid TOML, please fix it.


Error message:
cannot access local variable 'file_data' where it is not associated with a value

Configuration content:
[github_app]
pr_commands = [ "/review" ]

[pr_reviewer]
enable_review_labels_security = true
enable_chat_in_code_suggestions=false
enable_chat_in_code_suggestions = false
enable_summary=false
enable_help_text=false

[pr_description]
publish_description_as_comment = true

[pr_code_suggestions]
commitable_code_suggestions = true

[pr_test]
num_tests = 0

@chmouel
Copy link
Copy Markdown
Member Author

chmouel commented Feb 4, 2026

/review

@qodo-code-review
Copy link
Copy Markdown

PR Reviewer Guide 🔍

Here are some key observations to aid the review process:

⏱️ Estimated effort to review: 3 🔵🔵🔵⚪⚪
🧪 No relevant tests
🔒 No security concerns identified
⚡ Recommended focus areas for review

API Change

MatchPipelinerunByAnnotation gained a new reportErrors parameter. Since it is an exported function, this is a breaking signature change for any external callers. Consider adding a small backward-compatible wrapper (defaulting to reporting errors) or ensure all in-repo and downstream usages are updated accordingly.

func MatchPipelinerunByAnnotation(ctx context.Context, logger *zap.SugaredLogger, pruns []*tektonv1.PipelineRun, cs *params.Run, event *info.Event, vcx provider.Interface, eventEmitter *events.EventEmitter, repo *apipac.Repository, reportErrors bool) ([]Match, error) {
	matchedPRs := []Match{}
Behavior Coverage

CEL validation errors are now only reported when reportErrors is true. It would be good to validate that the “early check” path does not emit GitHub comments/events while still preserving any required logging/return behavior, and that the “real check” still reports exactly once.

if len(celValidationErrors) > 0 && reportErrors {
	reportCELValidationErrors(ctx, repo, celValidationErrors, eventEmitter, vcx, event)
}
Logic Change

Repository status counting now filters by opts.TargetSHA when provided. Validate that repo.Status entries with nil SHA are expected, that counting semantics match how statuses are appended (e.g., multiple contexts per SHA), and that opts.MinNumberStatus still reflects the intended threshold under filtering.

	statusCount := len(repo.Status)
	if opts.TargetSHA != "" {
		statusCount = 0
		for _, status := range repo.Status {
			if status.SHA != nil && *status.SHA == opts.TargetSHA {
				statusCount++
			}
		}
	}

	clients.Log.Infof("Still waiting for repository status to be updated: %d/%d", statusCount, opts.MinNumberStatus)
	time.Sleep(2 * time.Second)
	return statusCount >= opts.MinNumberStatus, nil
})

@chmouel chmouel force-pushed the match-annotation-only-report-error-once branch 4 times, most recently from 340f39a to eebf159 Compare February 4, 2026 13:08
@pipelines-as-code pipelines-as-code bot added the e2e label Feb 4, 2026
Added a `reportErrors` parameter to `MatchPipelinerunByAnnotation` to
allow callers to control CEL validation error reporting. Updated
`getPipelineRunsFromRepo` to initially match with reporting disabled and
then enable it for the final matching, preventing redundant error logs
during the process. Tests were also modified to accommodate this new
behavior.

Signed-off-by: Chmouel Boudjnah <chmouel@redhat.com>
Modified the `NSTearDown` function to append a random suffix to
repository URLs before deleting them. This ensures that any outstanding
webhooks targeting these repositories do not trigger new runs during the
teardown process, preventing potential interference or errors.

The change was made because previously, repositories were deleted
directly, which could lead to race conditions where webhooks were still
active and attempted to create new pipeline runs in a namespace that was
being cleaned up.

Signed-off-by: Chmouel Boudjnah <chmouel@redhat.com>
@chmouel chmouel force-pushed the match-annotation-only-report-error-once branch 10 times, most recently from a920380 to 26d80cf Compare February 4, 2026 17:44
Added extensive debug logging throughout the pipelineascode and related
packages. This includes:

* Enhanced logging for the matching process, including details on
  pipelinerun evaluation, annotation checks, and CEL expression
  evaluation.
* Detailed logging for remote resource fetching, including the source
  (HTTP, catalog, repository) and type of resource.
* Improved visibility into the cancellation of pipeline runs, with logs
  for enabled/disabled states and the selectors used.
* More granular logging for client setup, including secret handling and
  webhook validation.
* Logging for access checks and error reporting, providing context on
  sender and validation outcomes.
* Detailed logging for repository and pipeline run matching, including
  URL matching and commit info fetching.
* Comprehensive logging for fetching pipeline configurations, including
  template parsing and annotation processing.
* Logging for the `makeTemplate` function, indicating the number of
  resolved parameters and changed file groups.
* Detailed logging within the `resolve` package, covering the reading of
  Tekton types, resolution of remote resources, and metadata resolution.
* Logging for the `startPR` function, detailing pipelinerun creation,
  secret management, status updates, and patching operations.
* Logging for secret retrieval processes, including details on the
  repository, provider URL, and secret keys used.
A new job, `e2e-flaky-tests`, was added to the CI workflow to specifically run
tests that have previously exhibited flaky behavior. This job runs after the
main E2E tests and uses the `always()` condition to ensure it executes even if
earlier jobs fail, facilitating the collection of logs for debugging.

The `hack/gh-workflow-ci.sh` script was updated to recognize and filter tests
tagged as "Flaky". Additionally, test function names in
`test/github_pullrequest_test.go` were updated to include "Flaky" to align with
this new categorization and allow the script to identify them correctly. This
change aims to improve the reliability of the E2E test suite by isolating and
addressing flaky tests.

Signed-off-by: Chmouel Boudjnah <chmouel@redhat.com>
@chmouel chmouel force-pushed the match-annotation-only-report-error-once branch from 26d80cf to 7624805 Compare February 4, 2026 18:28
@chmouel chmouel changed the title fix: Control CEL validation error reporting fix: E2E test improvements and CEL error reporting Feb 4, 2026
@chmouel
Copy link
Copy Markdown
Member Author

chmouel commented Feb 4, 2026

/gemini review

Copy link
Copy Markdown

@gemini-code-assist gemini-code-assist bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request introduces several valuable improvements to the E2E testing infrastructure and fixes a bug in CEL error reporting. The change to prevent duplicate CEL validation errors by adding a reportErrors flag to MatchPipelinerunByAnnotation is a clean solution. Moving flaky tests to a dedicated CI job is a good practice for CI stability. The addition of extensive debug logging throughout the codebase will be very helpful for future troubleshooting. The fix in the E2E test teardown to modify repository URLs for better log collection is also a good improvement. I've found one area for improvement in the teardown logic related to a common Go pitfall with loop variables, which I've detailed in a specific comment.

@chmouel chmouel merged commit 7acb05b into tektoncd:main Feb 4, 2026
29 of 30 checks passed
@chmouel chmouel deleted the match-annotation-only-report-error-once branch February 4, 2026 21:19
@chmouel chmouel restored the match-annotation-only-report-error-once branch February 19, 2026 13:18
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants