I don't use junit at the moment but I know when I've used Jenkins in the past there was reporting that identified changes in individual tests over time - should we be worried about a risk that renaming the tests would break people's reports / workflows?

I've come to realise something else. At work we create custom reports that reuse some of this functionality (title, example values etc).

We realised that the nicest solution (in our opinion) is to replace placeholders in the title - instead of add #<number> or in this case (<example values>).

Consider this feature:

feature:
    Scenario Outline: <count> items should <passOrFail>
        When I have <count> items
        Then I should <passOrFail>
    Examples:
        | count  | passOrFail |
        | 0      | pass       |
        | 1      | pass       |
        | 2      | fail       |

• Currently, the result will be:

      <count> items should <passOrFail> #1
      <count> items should <passOrFail> #2
      <count> items should <passOrFail> #3
  

  The number is not particularly useful.
• With this PR, it would look like:

      <count> items should <passOrFail> (count: 0, passOrFail: pass)
      <count> items should <passOrFail> (count: 1, passOrFail: pass)
      <count> items should <passOrFail> (count: 2, passOrFail: fail)
  

  This is more useful, but not very readable and easily/unavoidably verbose
• With my latest proposal:

      0 items should pass
      1 items should pass
      2 items should fail
  

  Couldn't get any better. :)

I think the 3rd would be the best overall approach, but I've a feeling this is not supported in gherkin though (although I guess it's not contradicting it either). What do you think @stof, @ciaranmcnulty?

I like the idea of the third approach (it's similar to how e.g. it.each( works in jest / vitest).

Off the top of my head I don't know how feasible it is with gherkin syntax - although presumably could find a placeholder string that didn't contain anything gherkin considers to be special characters.

The other question would be what you do if the title doesn't contain placeholders - would you try to detect that and append the values as in this PR, or would you stick with the current #0 etc suffix, or something else?

The other question would be what you do if the title doesn't contain placeholders - would you try to detect that and append the values as in this PR, or would you stick with the current #0 etc suffix, or something else?

I was hoping I wouldn't have to answer that. 😆

When I think about it, it might make sense to add the numbering when the title already exists - it would be backward compatible and somewhat unsurprising.

    private $outlineTitles = [];

    private function buildExampleName(ExampleNode $scenario): string
    {
        $currentOutline = $this->outlineStoreListener->getCurrentOutline($scenario);

        $title = $this->replacePlaceholders($scenario->getOutlineTitle(), $scenario->getTokens());

        $number = $this->outlineTitles[$title] ?? 0;

        $this->outlineTitles[$title] = $number + 1;

        if ($number > 0) {
            $title .= " #$number";
        }

        if ($currentOutline === $this->lastOutline) {
            $this->outlineTitles = [];
        }

        return $title;
    }

That could work quite well, so it becomes a definitely-BC update path:

• If you do nothing, you'll get the same test names as before (including that the numbers are inconsistent between initial and re-run?)
• But to get consistent - and clearer - naming, you just have to reword your scenario outlines to include placeholders

That's a really nice idea if you can find a syntax for it that works in gherkin. Using <title> would seem logical as it follows the pattern of how the parameters are referenced in the steps within the outline. But might be risky (now or in future) because of that.

I don't think {} has any special meaning in gherkin, so that might be a possibility...

so it becomes a definitely-BC update path

Exactly.

But might be risky (now or in future) because of that.

If I didn't misunderstand, it is supported/implemented in cucumber already: cucumber/common#2149 and cucumber/cucumber-js#267. The request here seems to lend credence to that too: https://youtrack.jetbrains.com/issue/IDEA-320963/Cucumber-example-variable-usage-in-scenario-outline-title

On the whole, I think I'd go for the <...> syntax since it seems to also be what people there were expecting.

I see mixed signals in Behat though: #1024

Oh perfect, if cucumber is already using the tags like that for this purpose then that's exactly what we need. There's also an example of it in cucumber/gherkin.

If I've understood right, most of the mixed signals in #1024 look to me like they're specific to the pretty output format (where I can see there's no logical place to do the substitution in the current output, because it only prints the Scenario Outline block once and just renders the results into and below the Examples: table).

The suggestion this should be implemented in Gherkin is probably right, but:

• I'm not sure why this comment is linking to OutlineNode::getKeyword as I'm pretty sure that will (correctly) always just return Scenario Outline.
• It can't change the return of OutlineNode::getTitle() because that needs to be the title that represents all the examples e.g. without replacing any placeholders, so that e.g. the pretty formatter can use it as above.

I actually think the logical place to do this is inside ExampleNode. That is the class that's responsible for replacing placeholders in the step definitions, so replacing placeholders in the title there as well feels sensible.

ExampleNode already has a getOutlineTitle which currently just returns the parent title as-is. I can't find any usages of that method in behat - it could of course be used by other consumers. I don't know if we'd consider changing the string content a BC break - the safest option of course would be to add a new getOutlineTitleWithPlaceholders (with a better name 😆) and perhaps deprecate the existing getOutlineTitle.

The other advantage of doing it in ExampleNode is that when it's created by OutlineNode we presumably know which index it is in the gherkin file - I think filtering what will actually run comes later than parsing. So we could pass that into ExampleNode as well and use that to append a consistent number to the end of the title which would also then solve the JUnit bug even if people don't put placeholders into their titles.

WDYT?

I also feel the best place to put it would be in ExampleNode. I'm a bit hesitant with the semantics though.

• getTitle - this feels like it should return the definitive title, e.g. to be shown in reports and so on. In the ideal world, I think this should be the one replacing placeholders and adding numbers for deduplication.
• getOutlineTitle - Internally (e.g. even in gherkin), it seems that there shouldn't be a concept of a scenario outline - instead they should be flattened into regular scenarios. With that in mind, I think the "outline title" should be the original (parent?) title - since that parent is the "scenario outline".
• getOutlineTitleWithPlaceholders (& co.) - I would go for such a method if we want to be absolutely BC safe. That said, I do feel that the majority of end users would rather have the BC break than the current approach, but that's my hypothesis.

PS: with regards to getTitle, if it did behave like that, #1470 might have been avoided. I'd appreciate @cavo789's thoughts on this as well.

The challenge is getTitle is the title of the "example" and is currently the content of the table row for the example e.g. 1 | 5 | 4 and existing formatters rely on it

Regarding collapsing scenario and scenario outline - that could also be valid : gherkin now treats them as synonyms. But again existing formatters display them differently.

Maybe we should move this to an issue in gherkin to keep the discussion visible & clear?

I've updated the PR to fix conflicts and apply the last review suggestions, so that is can be "frozen" while a discussion can take place in behat/gherkin. Done: Behat/Gherkin#270