test-sbt: [bug] match tests on both short and prefixed names. by dubinsky · Pull Request #9756 · zio/zio

dubinsky · 2025-04-06T20:12:25Z

If a test is defined as suite("suite")(test("test"){}), it seems reasonable that we should be able to ask for it to be executed by either its short name test or by its full name suite - test.

When outer suite is run via sbt.testing and tests to run specified using sbt.testing.Test[Wildcard]Selector, this can be achieved by stripping the suite prefix "suite -" from the names in the Selectors. However, this approach breaks down in the presence of nested suites; e.g. if a test is defined as suite("outer")(suite("inner")(test("test"){})), when the outer suite is run:

selector test does executes the test - as it should;
selector outer - test does execute the test (prefix gets wrongly stripped) - but should not, since test with such a full name does not exist;
selector inner - test does not execute the test (wrong prefix gets stripped) - but should;
selector outer - inner - test does not execute the test (inner prefix does not get stripped) - but should.

When outer suite is run directly and tests to run specified using search terms (-t):

search term inner - test does not execute the test - but should;
search term outer - inner - test does not execute the test - but should.

To make this work correctly, an overload of Spec.filterLabels() is introduced that keep track of the accumulated suite prefix and matches the search terms against prefixed names. This overload is used in FilteredSpec.

kyri-petrou · 2025-04-26T07:12:49Z

test/shared/src/main/scala/zio/test/Spec.scala

   * an empty suite if this is a suite or `None` otherwise.
   */
-  final def filterLabels(f: String => Boolean)(implicit trace: Trace): Option[Spec[R, E]] =
+  final def filterLabels(f: String => Boolean, prefix: String = "", accumulatePrefix: Boolean = true)(implicit


This isn't binary compatible (which is why CI is failing). You'll have to introduce a new method called filterLabels with the additional arguments, and proxy to it from the original method. i.e.,:

final def filterLabels(f: String => Boolean)(implicit trace: Trace): Option[Spec[R, E]] = filterLabels(f, "") final def filterLabels(f: String => Boolean, prefix: String, accumulatePrefix: Boolean = true)(implicit trace: Trace): Option[Spec[R, E]] = { // implementation goes here }

You'll have to introduce a new method ...

Thank you for the clue!!
Done.

dubinsky · 2025-04-30T02:26:51Z

@kyri-petrou any thoughts? Thanks!

hearnadam

Seems alright to me

dubinsky · 2025-05-18T02:00:29Z

Seems alright to me

Thank you, @hearnadam!
Is there anything I need to do for this to get merged?

dubinsky · 2025-06-05T21:23:18Z

@kyri-petrou @hearnadam Is there anything I need to do for this to get merged?

If a test is defined as `suite("suite")(test("test"){})` it seems reasonable that we should be able to ask for it to be executed by either its short name "test" or by its full name "suite - test". This *can* be achieved by stripping the suite prefix "suite -" from the names from the selectors. However, this approach breaks down in the presence of nested suites; e.g. if a test is defined as `suite("outer")(suite("inner)(test("test"){}))`, when the outer suite is run: - selector "test" still executes the test - as it should; - selector "inner - test" does not execute the test (wrong prefix gets stipped) - but should; - selector "outer - inner - test" does not execute the test (inner prefix does not get stripped) - but should; - selector "outer - test" does execute the test - but shouldn't, since test with such a full name does not exist. To make this work correctly, `Spec.filterLabels()` has to be adjusted to keep track of the accumulated suite prefix and match the search terms against the full names. Once that is done, we do not need to strip anything here; in fact, we do not even need know what the label of the spec is.

hearnadam · 2025-06-16T21:52:02Z

@dubinsky good to merge?

dubinsky · 2025-06-16T21:53:21Z

@dubinsky good to merge?

Yes. Thanks you!

/closes #9756 `exitCode` operator is dangerous as it swallows all errors. Since this is no longer necessary for creating a ZIOApp, it should no longer be used.

* Deprecated `ZIO#exitCode` /closes #9756 `exitCode` operator is dangerous as it swallows all errors. Since this is no longer necessary for creating a ZIOApp, it should no longer be used. * fix brittle test

@kyri-petrou

* [test-sbt] More uniformity. Currently, code implementing `sbt.testing.Framework` for JVM, Scala.js and Scala Native suffers from severe code triplication: code doing the same thing is present in three places. This pull request removes this gratuitous redundancy. An unfortunate - and inevitable - consequence of code duplication is code drift: different copies of the code do things differently for no good reason. Backends `sbt.testing.Framework` runs on _do_ have inherent differences: tests running on Scala.js and Scala Native are not running on JVM, and thus their outcomes need to be communicated to the process managing the test run; Scala.js is single-threaded by nature, which affects the way tests have to be run. Code differences not warranted by the inherent differences between the backends seem gratuitous, and this pull request removes some of them. One area where implementations of the `sbt.testing.Framework` diverged from one another is the machinery to test them: only JVM implementation is currently testable. With this pull request, `sbt.testing.Framework` tests run on all the backends. Another divergence is in the storage of test summaries: JVM uses `AtomicReference[Vector]`, Scala.js uses `Buffer`, Scala Native uses `ConcurrentLinkedQueue`; this pull request unifies summary storage for all backends: `AtomicReference[Vector]` works for all of them. Another divergence is the way test summary is rendered: counts of tests executed, ignored, or failed are included in the summary on JVM but not on Scala.js or Scala Native; setting test renderer to IntelliJ in the test arguments is honored only on JVM. This pull request makes test summary uniform across the backends. Another divergence is: currently, signal handlers are installed only on JVM. This pull request installs them on all backends. Another divergence affects a recent [enhancement](#9756), which turned out to be effective only on JVM; this pull request applies it to all backends (and tests that it works ;)). @kyri-petrou, your [remark](#9629 (comment)) > I gave it a go at this (basically trying to unify the JVM / JS / Native implementations so that we can reuse the existing event test suites that exist for the JVM, but it doesn't seem possible. > ... writing a full test suite from scratch (which frankly it's going to be extremely time consuming) is on target: this was not trivial ;) * [test-sbt] More uniformity. Address review comments. * [test-sbt] More uniformity. Address more review comments. * [test-sbt] More uniformity. Address more review comments.

dubinsky mentioned this pull request Apr 6, 2025

ZioTest ignores test wildcard selectors dubinsky/scalajs-gradle#45

Closed

dubinsky force-pushed the test-sbt-match-tests-on-both-short-and-prefixed-names branch from 2d9d681 to 7c60c5c Compare April 6, 2025 20:19

dubinsky changed the title ~~test-sbt: match tests on both short and prefixed names.~~ WIP: test-sbt: match tests on both short and prefixed names. Apr 6, 2025

dubinsky marked this pull request as draft April 6, 2025 20:43

dubinsky force-pushed the test-sbt-match-tests-on-both-short-and-prefixed-names branch 3 times, most recently from d39a623 to f9d3b6f Compare April 7, 2025 00:12

dubinsky changed the title ~~WIP: test-sbt: match tests on both short and prefixed names.~~ test-sbt: match tests on both short and prefixed names. Apr 7, 2025

dubinsky marked this pull request as ready for review April 7, 2025 00:19

dubinsky changed the title ~~test-sbt: match tests on both short and prefixed names.~~ test-sbt: [bug] match tests on both short and prefixed names. Apr 7, 2025

kyri-petrou reviewed Apr 26, 2025

View reviewed changes

dubinsky force-pushed the test-sbt-match-tests-on-both-short-and-prefixed-names branch 4 times, most recently from 568aab0 to a514126 Compare April 29, 2025 00:36

hearnadam approved these changes May 17, 2025

View reviewed changes

dubinsky force-pushed the test-sbt-match-tests-on-both-short-and-prefixed-names branch from 328208a to d5c9101 Compare June 16, 2025 21:04

hearnadam approved these changes Jun 16, 2025

View reviewed changes

hearnadam merged commit a729ee6 into zio:series/2.x Jun 16, 2025
19 checks passed

hearnadam added a commit that referenced this pull request Jun 16, 2025

Deprecated ZIO#exitCode

4cea445

/closes #9756 `exitCode` operator is dangerous as it swallows all errors. Since this is no longer necessary for creating a ZIOApp, it should no longer be used.

hearnadam added a commit that referenced this pull request Jun 16, 2025

Deprecated ZIO#exitCode

eeedc45

/closes #9756 `exitCode` operator is dangerous as it swallows all errors. Since this is no longer necessary for creating a ZIOApp, it should no longer be used.

hearnadam mentioned this pull request Jun 16, 2025

Deprecated ZIO#exitCode #9978

Merged

hearnadam added a commit that referenced this pull request Jun 21, 2025

Deprecated ZIO#exitCode

b9af307

/closes #9756 `exitCode` operator is dangerous as it swallows all errors. Since this is no longer necessary for creating a ZIOApp, it should no longer be used.

hearnadam added a commit that referenced this pull request Jun 21, 2025

Deprecated ZIO#exitCode

9768fcf

/closes #9756 `exitCode` operator is dangerous as it swallows all errors. Since this is no longer necessary for creating a ZIOApp, it should no longer be used.

dubinsky mentioned this pull request Aug 19, 2025

[test-sbt] More uniformity. #10120

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

test-sbt: [bug] match tests on both short and prefixed names.#9756

test-sbt: [bug] match tests on both short and prefixed names.#9756
hearnadam merged 1 commit intozio:series/2.xfrom
dubinsky:test-sbt-match-tests-on-both-short-and-prefixed-names

dubinsky commented Apr 6, 2025 •

edited

Loading

Uh oh!

kyri-petrou Apr 26, 2025

Uh oh!

dubinsky Apr 27, 2025

Uh oh!

dubinsky commented Apr 30, 2025

Uh oh!

hearnadam left a comment

Uh oh!

dubinsky commented May 18, 2025

Uh oh!

dubinsky commented Jun 5, 2025

Uh oh!

hearnadam commented Jun 16, 2025

Uh oh!

dubinsky commented Jun 16, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

dubinsky commented Apr 6, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

kyri-petrou Apr 26, 2025

Choose a reason for hiding this comment

Uh oh!

dubinsky Apr 27, 2025

Choose a reason for hiding this comment

Uh oh!

dubinsky commented Apr 30, 2025

Uh oh!

hearnadam left a comment

Choose a reason for hiding this comment

Uh oh!

dubinsky commented May 18, 2025

Uh oh!

dubinsky commented Jun 5, 2025

Uh oh!

hearnadam commented Jun 16, 2025

Uh oh!

dubinsky commented Jun 16, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

dubinsky commented Apr 6, 2025 •

edited

Loading