Add anti-pattern check for usage of have_admin_callout (matcher strategy) - deprecated#16003
Add anti-pattern check for usage of have_admin_callout (matcher strategy) - deprecated#16003andreslucena wants to merge 6 commits intodevelopfrom
Conversation
decidim-dev/lib/decidim/dev/test/rspec_support/admin_callout_matchers.rb
Fixed
Show fixed
Hide fixed
decidim-dev/lib/decidim/dev/test/rspec_support/admin_callout_matchers.rb
Fixed
Show fixed
Hide fixed
decidim-dev/lib/decidim/dev/test/rspec_support/admin_callout_matchers.rb
Fixed
Show fixed
Hide fixed
decidim-dev/lib/decidim/dev/test/rspec_support/admin_callout_matchers.rb
Fixed
Show fixed
Hide fixed
decidim-dev/lib/decidim/dev/test/rspec_support/admin_callout_matchers.rb
Fixed
Show fixed
Hide fixed
|
Doing a quick run to see all the failures first |
📝 WalkthroughWalkthroughAdds a new RSpec matcher Changes
Sequence Diagram(s)sequenceDiagram
participant Spec as RSpec Example
participant Matcher as have_admin_callout matcher
participant Flash as Admin Flash Messages (DOM)
Spec->>Matcher: invoke have_admin_callout(expected)
Matcher->>Matcher: validate_text(expected) (nil/symbol/length/anti-patterns)
alt validation passes
Matcher->>Flash: within_flash_messages { find expected text }
Flash-->>Matcher: match found
Matcher-->>Spec: pass
else validation fails
Matcher-->>Spec: raise AntipatternError with grep hint
end
Estimated code review effort🎯 3 (Moderate) | ⏱️ ~25 minutes Poem
🚥 Pre-merge checks | ✅ 2 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (2 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing touches
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
There was a problem hiding this comment.
Pull request overview
This PR introduces antipattern detection for the have_admin_callout matcher to prevent developers from using generic, vague flash message assertions (like have_admin_callout("successfully")) that don't verify the actual feedback shown to users. The validation enforces a minimum message length of 12 characters and explicitly blocks common single-word antipatterns.
Changes:
- Replaces the simple
have_admin_callouthelper method with a full RSpec custom matcher that validates message quality - Adds validation rules to detect and reject generic single-word messages and very short strings
- Includes comprehensive test coverage for the antipattern detection logic
Reviewed changes
Copilot reviewed 3 out of 3 changed files in this pull request and generated 6 comments.
| File | Description |
|---|---|
| decidim-dev/lib/decidim/dev/test/rspec_support/admin_callout_matchers.rb | New file implementing the have_admin_callout custom RSpec matcher with antipattern detection logic |
| decidim-dev/spec/lib/decidim/dev/admin_callout_matchers_spec.rb | New test file verifying antipattern detection for various invalid inputs |
| decidim-dev/lib/decidim/dev/test/rspec_support/helpers.rb | Removes the old simple have_admin_callout method (6 lines) that is being replaced by the new matcher |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| MIN_LENGTH_THRESHOLD = 12 | ||
|
|
||
| SINGLE_WORD_ANTIPATTERNS = %w( | ||
| successfully | ||
| successfully. | ||
| problem | ||
| error | ||
| warning | ||
| done | ||
| complete | ||
| finished | ||
| ok | ||
| okay | ||
| saved | ||
| updated | ||
| created | ||
| deleted | ||
| removed | ||
| published | ||
| unpublished | ||
| ).freeze | ||
|
|
||
| RSpec::Matchers.define :have_admin_callout do |expected| | ||
| include Decidim::ComponentTestHelpers | ||
|
|
||
| match do |_page| | ||
| # Short-circuit for nil/symbols to avoid calling within_flash_messages | ||
| return true if expected.nil? || expected.is_a?(Symbol) | ||
|
|
||
| validate_text(expected) | ||
|
|
||
| within_flash_messages do | ||
| have_content expected | ||
| end | ||
| end | ||
|
|
||
| def validate_text(text) | ||
| return true if text.nil? || text.is_a?(Symbol) | ||
|
|
||
| stripped_text = text.to_s.gsub(/[[:punct:]]/, "") | ||
| is_single_word = text.to_s.strip !~ /\s/ | ||
| is_too_short = stripped_text.length < MIN_LENGTH_THRESHOLD | ||
|
|
||
| if is_single_word && is_too_short | ||
| raise <<~ERROR | ||
| Antipattern detected: Using generic single-word text "#{text}" in admin callout assertions. | ||
|
|
||
| Problem: Generic single-word messages don't verify the actual feedback shown to users. | ||
|
|
||
| #{format_find_command(text)} | ||
| ERROR | ||
| end | ||
|
|
||
| if is_single_word && SINGLE_WORD_ANTIPATTERNS.include?(stripped_text.downcase) | ||
| raise <<~ERROR | ||
| Antipattern detected: Using generic single-word text "#{text}" in admin callout assertions. | ||
|
|
||
| Problem: Generic single-word messages don't verify the actual feedback shown to users. | ||
| Single-word messages like "successfully" or "problem" are too vague. | ||
|
|
||
| Example: "Meeting successfully published" is better than just "successfully" | ||
|
|
||
| #{format_find_command(text)} | ||
| ERROR | ||
| end | ||
|
|
||
| if is_too_short | ||
| raise <<~ERROR | ||
| Antipattern detected: Using very short text "#{text}" (#{stripped_text.length} chars) in admin callout assertions. | ||
|
|
||
| Problem: Very short messages are likely generic and don't provide meaningful feedback. | ||
| Messages should be at least #{MIN_LENGTH_THRESHOLD} characters to convey useful information. | ||
|
|
||
| Examples of proper messages: | ||
| - "Meeting successfully published" | ||
| - "Budget successfully created." | ||
| - "There was a problem saving the form" | ||
| - "Your changes have been saved successfully" | ||
|
|
||
| #{format_find_command(text)} | ||
| ERROR | ||
| end | ||
|
|
||
| true | ||
| end | ||
|
|
||
| def format_find_command(text) | ||
| <<~COMMAND | ||
|
|
||
| To find similar instances: | ||
| grep -r 'have_admin_callout.*#{text}' --include="*.rb" decidim-*/ | ||
| COMMAND | ||
| end | ||
|
|
||
| failure_message do | ||
| @failure_message || super() | ||
| end | ||
|
|
||
| diffable |
There was a problem hiding this comment.
The constants MIN_LENGTH_THRESHOLD and SINGLE_WORD_ANTIPATTERNS are defined at the top level (global namespace) instead of being scoped within a module or the matcher definition. This can lead to namespace pollution and potential conflicts with other code that might use similar constant names.
Consider moving these constants inside the matcher definition or wrapping them in a module namespace. For example, you could define them as class-level constants within a module, or define them as constants within the matcher block itself.
| MIN_LENGTH_THRESHOLD = 12 | |
| SINGLE_WORD_ANTIPATTERNS = %w( | |
| successfully | |
| successfully. | |
| problem | |
| error | |
| warning | |
| done | |
| complete | |
| finished | |
| ok | |
| okay | |
| saved | |
| updated | |
| created | |
| deleted | |
| removed | |
| published | |
| unpublished | |
| ).freeze | |
| RSpec::Matchers.define :have_admin_callout do |expected| | |
| include Decidim::ComponentTestHelpers | |
| match do |_page| | |
| # Short-circuit for nil/symbols to avoid calling within_flash_messages | |
| return true if expected.nil? || expected.is_a?(Symbol) | |
| validate_text(expected) | |
| within_flash_messages do | |
| have_content expected | |
| end | |
| end | |
| def validate_text(text) | |
| return true if text.nil? || text.is_a?(Symbol) | |
| stripped_text = text.to_s.gsub(/[[:punct:]]/, "") | |
| is_single_word = text.to_s.strip !~ /\s/ | |
| is_too_short = stripped_text.length < MIN_LENGTH_THRESHOLD | |
| if is_single_word && is_too_short | |
| raise <<~ERROR | |
| Antipattern detected: Using generic single-word text "#{text}" in admin callout assertions. | |
| Problem: Generic single-word messages don't verify the actual feedback shown to users. | |
| #{format_find_command(text)} | |
| ERROR | |
| end | |
| if is_single_word && SINGLE_WORD_ANTIPATTERNS.include?(stripped_text.downcase) | |
| raise <<~ERROR | |
| Antipattern detected: Using generic single-word text "#{text}" in admin callout assertions. | |
| Problem: Generic single-word messages don't verify the actual feedback shown to users. | |
| Single-word messages like "successfully" or "problem" are too vague. | |
| Example: "Meeting successfully published" is better than just "successfully" | |
| #{format_find_command(text)} | |
| ERROR | |
| end | |
| if is_too_short | |
| raise <<~ERROR | |
| Antipattern detected: Using very short text "#{text}" (#{stripped_text.length} chars) in admin callout assertions. | |
| Problem: Very short messages are likely generic and don't provide meaningful feedback. | |
| Messages should be at least #{MIN_LENGTH_THRESHOLD} characters to convey useful information. | |
| Examples of proper messages: | |
| - "Meeting successfully published" | |
| - "Budget successfully created." | |
| - "There was a problem saving the form" | |
| - "Your changes have been saved successfully" | |
| #{format_find_command(text)} | |
| ERROR | |
| end | |
| true | |
| end | |
| def format_find_command(text) | |
| <<~COMMAND | |
| To find similar instances: | |
| grep -r 'have_admin_callout.*#{text}' --include="*.rb" decidim-*/ | |
| COMMAND | |
| end | |
| failure_message do | |
| @failure_message || super() | |
| end | |
| diffable | |
| module Decidim | |
| module Dev | |
| module Test | |
| MIN_LENGTH_THRESHOLD = 12 | |
| SINGLE_WORD_ANTIPATTERNS = %w( | |
| successfully | |
| successfully. | |
| problem | |
| error | |
| warning | |
| done | |
| complete | |
| finished | |
| ok | |
| okay | |
| saved | |
| updated | |
| created | |
| deleted | |
| removed | |
| published | |
| unpublished | |
| ).freeze | |
| RSpec::Matchers.define :have_admin_callout do |expected| | |
| include Decidim::ComponentTestHelpers | |
| match do |_page| | |
| # Short-circuit for nil/symbols to avoid calling within_flash_messages | |
| return true if expected.nil? || expected.is_a?(Symbol) | |
| validate_text(expected) | |
| within_flash_messages do | |
| have_content expected | |
| end | |
| end | |
| def validate_text(text) | |
| return true if text.nil? || text.is_a?(Symbol) | |
| stripped_text = text.to_s.gsub(/[[:punct:]]/, "") | |
| is_single_word = text.to_s.strip !~ /\s/ | |
| is_too_short = stripped_text.length < MIN_LENGTH_THRESHOLD | |
| if is_single_word && is_too_short | |
| raise <<~ERROR | |
| Antipattern detected: Using generic single-word text "#{text}" in admin callout assertions. | |
| Problem: Generic single-word messages don't verify the actual feedback shown to users. | |
| #{format_find_command(text)} | |
| ERROR | |
| end | |
| if is_single_word && SINGLE_WORD_ANTIPATTERNS.include?(stripped_text.downcase) | |
| raise <<~ERROR | |
| Antipattern detected: Using generic single-word text "#{text}" in admin callout assertions. | |
| Problem: Generic single-word messages don't verify the actual feedback shown to users. | |
| Single-word messages like "successfully" or "problem" are too vague. | |
| Example: "Meeting successfully published" is better than just "successfully" | |
| #{format_find_command(text)} | |
| ERROR | |
| end | |
| if is_too_short | |
| raise <<~ERROR | |
| Antipattern detected: Using very short text "#{text}" (#{stripped_text.length} chars) in admin callout assertions. | |
| Problem: Very short messages are likely generic and don't provide meaningful feedback. | |
| Messages should be at least #{MIN_LENGTH_THRESHOLD} characters to convey useful information. | |
| Examples of proper messages: | |
| - "Meeting successfully published" | |
| - "Budget successfully created." | |
| - "There was a problem saving the form" | |
| - "Your changes have been saved successfully" | |
| #{format_find_command(text)} | |
| ERROR | |
| end | |
| true | |
| end | |
| def format_find_command(text) | |
| <<~COMMAND | |
| To find similar instances: | |
| grep -r 'have_admin_callout.*#{text}' --include="*.rb" decidim-*/ | |
| COMMAND | |
| end | |
| failure_message do | |
| @failure_message || super() | |
| end | |
| diffable | |
| end | |
| end | |
| end |
There was a problem hiding this comment.
As I mentioned in my comment :
Doing a quick run to see all the failures first
So, this is actually part of the process :)
| within_flash_messages do | ||
| have_content expected | ||
| end |
There was a problem hiding this comment.
The match block is not correctly evaluating whether the content exists in the flash messages. The code calls have_content expected within the within_flash_messages block, but this just returns a matcher object without evaluating it. Since matcher objects are always truthy, this will always return true (assuming validation passes).
The match block should actually check if the content exists. Consider using one of these approaches:
- Use Capybara's predicate methods:
page.has_content?(expected) - Delegate to Capybara's built-in matcher and return the result of checking the scoped element
- Use an expectation and return true/false based on whether it passes
For example, you could capture the result within the within_flash_messages block by checking if the current scope has the content.
| within_flash_messages do | |
| have_content expected | |
| end | |
| result = nil | |
| within_flash_messages do | |
| result = page.has_content?(expected) | |
| end | |
| result |
decidim-dev/lib/decidim/dev/test/rspec_support/admin_callout_matchers.rb
Outdated
Show resolved
Hide resolved
decidim-dev/lib/decidim/dev/test/rspec_support/admin_callout_matchers.rb
Outdated
Show resolved
Hide resolved
| RSpec::Matchers.define :have_admin_callout do |expected| | ||
| include Decidim::ComponentTestHelpers | ||
|
|
||
| match do |_page| | ||
| # Short-circuit for nil/symbols to avoid calling within_flash_messages | ||
| return true if expected.nil? || expected.is_a?(Symbol) | ||
|
|
||
| validate_text(expected) | ||
|
|
||
| within_flash_messages do | ||
| have_content expected | ||
| end | ||
| end | ||
|
|
||
| def validate_text(text) | ||
| return true if text.nil? || text.is_a?(Symbol) | ||
|
|
||
| stripped_text = text.to_s.gsub(/[[:punct:]]/, "") | ||
| is_single_word = text.to_s.strip !~ /\s/ | ||
| is_too_short = stripped_text.length < MIN_LENGTH_THRESHOLD | ||
|
|
||
| if is_single_word && is_too_short | ||
| raise <<~ERROR | ||
| Antipattern detected: Using generic single-word text "#{text}" in admin callout assertions. | ||
|
|
||
| Problem: Generic single-word messages don't verify the actual feedback shown to users. | ||
|
|
||
| #{format_find_command(text)} | ||
| ERROR | ||
| end | ||
|
|
||
| if is_single_word && SINGLE_WORD_ANTIPATTERNS.include?(stripped_text.downcase) | ||
| raise <<~ERROR | ||
| Antipattern detected: Using generic single-word text "#{text}" in admin callout assertions. | ||
|
|
||
| Problem: Generic single-word messages don't verify the actual feedback shown to users. | ||
| Single-word messages like "successfully" or "problem" are too vague. | ||
|
|
||
| Example: "Meeting successfully published" is better than just "successfully" | ||
|
|
||
| #{format_find_command(text)} | ||
| ERROR | ||
| end | ||
|
|
||
| if is_too_short | ||
| raise <<~ERROR | ||
| Antipattern detected: Using very short text "#{text}" (#{stripped_text.length} chars) in admin callout assertions. | ||
|
|
||
| Problem: Very short messages are likely generic and don't provide meaningful feedback. | ||
| Messages should be at least #{MIN_LENGTH_THRESHOLD} characters to convey useful information. | ||
|
|
||
| Examples of proper messages: | ||
| - "Meeting successfully published" | ||
| - "Budget successfully created." | ||
| - "There was a problem saving the form" | ||
| - "Your changes have been saved successfully" | ||
|
|
||
| #{format_find_command(text)} | ||
| ERROR | ||
| end | ||
|
|
||
| true | ||
| end | ||
|
|
||
| def format_find_command(text) | ||
| <<~COMMAND | ||
|
|
||
| To find similar instances: | ||
| grep -r 'have_admin_callout.*#{text}' --include="*.rb" decidim-*/ | ||
| COMMAND | ||
| end | ||
|
|
||
| failure_message do | ||
| @failure_message || super() | ||
| end | ||
|
|
||
| diffable | ||
| end |
There was a problem hiding this comment.
This PR introduces antipattern detection that will cause hundreds of existing tests to fail. A search of the codebase shows 206+ instances of have_admin_callout("successfully") and many more instances of similar antipatterns like have_admin_callout("updated"), have_admin_callout("created"), etc.
The PR description states "Do the search and see it on 0" suggesting all instances should be fixed, but this PR doesn't include any fixes to the affected test files. This will cause a massive test failure across the entire codebase once merged.
This is a breaking change that requires one of the following approaches:
- Include fixes for all affected tests in this same PR
- Split into two PRs: first fix all the antipattern usages, then add the validation
- Add a temporary flag or environment variable to enable/disable the antipattern detection, allowing for gradual migration
- Provide a clear migration guide and accept that tests will fail until they're updated
Without addressing this, merging this PR will break the CI/CD pipeline and block development until all affected tests are fixed.
There was a problem hiding this comment.
Actionable comments posted: 3
🤖 Fix all issues with AI agents
In `@decidim-dev/lib/decidim/dev/test/rspec_support/admin_callout_matchers.rb`:
- Around line 7-25: Rename the constant SINGLE_WORD_ANTIPATTERNS to a
lint-friendly name like SINGLE_WORD_ANTI_PATTERNS and update any code that
references it (e.g., in admin_callout_matchers.rb) to use the new constant;
replace any user-facing text that uses “antipattern/Antipattern” with
“anti-pattern” and change contractions like “don’t” to “do not” in the
error/message strings so spelling/lint checks pass; update all specs that assert
on these messages to expect the new wording and adjust any other occurrences in
the same file (including the other constants/messages around the block where
SINGLE_WORD_ANTIPATTERNS was defined) to the new names/strings.
- Around line 30-38: The match block currently returns an unevaluated RSpec
matcher (have_content expected) which always passes; inside the
within_flash_messages block (and keeping validate_text(expected) and the
surrounding match do |_page| intact) replace the matcher construction with a
boolean check such as has_text?(expected) (i.e., call has_text?(expected) inside
within_flash_messages) so the block returns a true/false based on actual
presence of the text rather than a matcher object.
In `@decidim-dev/spec/lib/decidim/dev/admin_callout_matchers_spec.rb`:
- Around line 5-12: Update the spec strings and error-message expectations to
use the spelled form "anti-pattern" instead of "antipattern": search for
occurrences in the RSpec example group description ("antipattern detection"),
context/descriptions, and any regex expectations (e.g., the
raise_error(/Antipattern detected/) assertion) and replace them with
"anti-pattern" or a regex matching /Anti-pattern detected/ so they align with
the matcher have_admin_callout's wording; ensure all other examples in the same
file are updated similarly.
decidim-dev/lib/decidim/dev/test/rspec_support/admin_callout_matchers.rb
Outdated
Show resolved
Hide resolved
| match do |_page| | ||
| # Short-circuit for nil/symbols to avoid calling within_flash_messages | ||
| return true if expected.nil? || expected.is_a?(Symbol) | ||
|
|
||
| validate_text(expected) | ||
|
|
||
| within_flash_messages do | ||
| have_content expected | ||
| end |
There was a problem hiding this comment.
🧩 Analysis chain
🏁 Script executed:
# First, let's read the entire file to understand the context
cat -n decidim-dev/lib/decidim/dev/test/rspec_support/admin_callout_matchers.rbRepository: decidim/decidim
Length of output: 3456
🏁 Script executed:
# Search for within_flash_messages definition
rg -n "def within_flash_messages" --type rubyRepository: decidim/decidim
Length of output: 149
🏁 Script executed:
# Search for how within_flash_messages is used elsewhere in the codebase
rg -n "within_flash_messages" --type ruby -A 2 | head -50Repository: decidim/decidim
Length of output: 3688
🏁 Script executed:
# Search for other RSpec.Matchers.define patterns in the codebase to see how match blocks are typically structured
rg -n "RSpec::Matchers\.define" --type ruby -A 10 | head -100Repository: decidim/decidim
Length of output: 9119
🏁 Script executed:
# Look for examples of match blocks in custom matchers
rg -n "match do" --type ruby -A 3 | grep -E "(match do|expect|has_|\.call)" | head -50Repository: decidim/decidim
Length of output: 2511
🏁 Script executed:
# Verify that have_content is an RSpec matcher that needs expect() to be evaluated
rg -n "have_content" decidim-dev/lib/decidim/dev/test/rspec_support/helpers.rbRepository: decidim/decidim
Length of output: 109
Match block returns unevaluated matcher instead of boolean assertion
The have_content expected statement inside within_flash_messages only builds a matcher object and returns it without evaluating the assertion. RSpec matcher objects are truthy, so the block will always pass regardless of whether the text is actually present. Wrap the assertion with expect() or use a boolean method like has_text?() to properly evaluate the condition.
Proposed fix
match do |_page|
# Short-circuit for nil/symbols to avoid calling within_flash_messages
return true if expected.nil? || expected.is_a?(Symbol)
validate_text(expected)
within_flash_messages do
- have_content expected
+ page.has_text?(expected)
end
end🤖 Prompt for AI Agents
In `@decidim-dev/lib/decidim/dev/test/rspec_support/admin_callout_matchers.rb`
around lines 30 - 38, The match block currently returns an unevaluated RSpec
matcher (have_content expected) which always passes; inside the
within_flash_messages block (and keeping validate_text(expected) and the
surrounding match do |_page| intact) replace the matcher construction with a
boolean check such as has_text?(expected) (i.e., call has_text?(expected) inside
within_flash_messages) so the block returns a true/false based on actual
presence of the text rather than a matcher object.
decidim-dev/spec/lib/decidim/dev/admin_callout_matchers_spec.rb
Outdated
Show resolved
Hide resolved
|
Closing this one in favour of #16011 - that seems much cleaner |
🎩 What? Why?
While giving an eye to #15994, I found the usage of this antipattern:
This is something that is well extended in the codebase, and it actually can also ignore a minor bug in the process: as the spec was copy and pasted, the flash can also be copy pasted and we can be showing a wrong message in the flash (something like "Meeting successfully updated" when it should really be a Proposal).
As this is really extended through the codebase, I think we should introduce a rubocop or a failure in the CI for this. I think just adding the failure in the matcher is the cleanest way of doing it.
📌 Related Issues
Testing
Summary by CodeRabbit
Tests
Chores