Context
If you run test suites with x.py test (sub)command(s) in environments we don't run it in in CI, they may fail. That's natural and expected, of course, but the tooling output doesn't make it particularly clear what usage is or isn't intended.
If the user is on a weird platform or such, this might be expected anyway; it becomes surprising if the thing that caused a faulure is merely a different bootstrap.toml setup. Most notably if the setup happens to be exactly one of the presets that x.py setup offers.
As an extreme example of the confusion this may lead to: A plain x.py test without any extra arguments, with a basic profile = "dist" profile (created in x.py setup), on a normal x86_64-unknown-linux-gnu (Ubuntu) setup, used to1 be prominently displaying an ICE error message as part the first failing test.
An initial search for similar/related kinds of issues had me finding
which seems to also fit this category of problems (though in this case, it's more obviously a case where things should actually just be fixed).
Questions
On one hand, I have some remaining questions about the dist profile: Is the dist profile in particular even meant to be used with commands like x test? I also wonder (i.e. haven't quite figured out yet) what about it makes it more prone to test failures in particular?
On the other hand, I would be interested in judging potential ways forward so that someone who sees a test failure wouldn't even need to wonder about such fundamental things as “is this supposed to work in the first place?”.
👉 expand/collapse the list - [I’m trying not to take away all the focus from the first question]
-
certain profiles could "disable"2 certain x.py subcommands that aren't intended/meant to work anyway, to stop them from running before they even begin
- this would make the failure quicker, and the error message more consistent than "whatever happens to be the first thing that breaks"
- the error would provide information on how to change
bootstrap.toml, or maybe pass an extra argument, to run the test anyway
-
or instead, nothing should change - reasons for this may include:
- there's not really any issue; the test suite is mostly for our internal usage anyway
- it's too much work to implement and maintain this (for too little benefit)
- we need better documentation for contributors, and with better documentation, things will be clear enough
x test shouldn't fail on dist in the first place because [xyz], we should just fix this
-
or instead, certain profiles could emit extra warnings/notes with certain x.py subcommands
- warnings can work similar to out-of-date-
change-id warnings, showing both at the start and end of invocation
- could set expectations through case-specific messages (document known/expected failure; point to context/workarounds for known issues3; or conversely possibly encourage reporting issues for cases that really should work [but not enforced by CI])
-
or instead, something else? Or a combination of things?
Context
If you run test suites with
x.py test(sub)command(s) in environments we don't run it in in CI, they may fail. That's natural and expected, of course, but the tooling output doesn't make it particularly clear what usage is or isn't intended.If the user is on a weird platform or such, this might be expected anyway; it becomes surprising if the thing that caused a faulure is merely a different
bootstrap.tomlsetup. Most notably if the setup happens to be exactly one of the presets thatx.py setupoffers.As an extreme example of the confusion this may lead to: A plain
x.py testwithout any extra arguments, with a basicprofile = "dist"profile (created inx.py setup), on a normalx86_64-unknown-linux-gnu(Ubuntu) setup, used to1 be prominently displaying an ICE error message as part the first failing test.An initial search for similar/related kinds of issues had me finding
./x test library/stdfails with library profile #142505which seems to also fit this category of problems (though in this case, it's more obviously a case where things should actually just be fixed).
Questions
On one hand, I have some remaining questions about the
distprofile: Is thedistprofile in particular even meant to be used with commands likex test? I also wonder (i.e. haven't quite figured out yet) what about it makes it more prone to test failures in particular?On the other hand, I would be interested in judging potential ways forward so that someone who sees a test failure wouldn't even need to wonder about such fundamental things as “is this supposed to work in the first place?”.
👉 expand/collapse the list - [I’m trying not to take away all the focus from the first question]
certain profiles could "disable"2 certain
x.pysubcommands that aren't intended/meant to work anyway, to stop them from running before they even beginbootstrap.toml, or maybe pass an extra argument, to run the test anywayor instead, nothing should change - reasons for this may include:
x testshouldn't fail ondistin the first place because[xyz], we should just fix thisor instead, certain profiles could emit extra warnings/notes with certain
x.pysubcommandschange-idwarnings, showing both at the start and end of invocationor instead, something else? Or a combination of things?
Footnotes
at the time of this writing waiting for the merge queue ↩
that is, always produce an error message when it's used ↩
it would ideally be easy to configure certain configuration+command usage patterns, if "known issues" (like
./x test library/stdon library profile) should be handled, so contributors can keep them up-to-date easily ↩