[SWT-NNNN] Make the Issues API public & provide a public Issue.record instance method.#481
[SWT-NNNN] Make the Issues API public & provide a public Issue.record instance method.#481younata wants to merge 8 commits intoswiftlang:mainfrom
Issue.record instance method.#481Conversation
…scoped to just making the issues api public
|
One minor thing that worries me is that this allows creating issues of type As well, should we consider deprecating the static |
| public func record() -> Self { | ||
| record(configuration: nil) | ||
| } |
There was a problem hiding this comment.
For the purposes of a proposal, you can omit the body of the method
| ## Integration with supporting tools | ||
|
|
||
| Third-party assertion tools would be able to directly create an `Issue` | ||
| and then report it using the `Issue.report()` instance method. `Issue.report()` |
There was a problem hiding this comment.
These say report() but the actual proposed method is named record(), can you update those to be consistent throughout the document?
|
One thing which is likely to pose a challenge with using this API as proposed is that other related types don't yet expose a In order to correctly represent an expectation failure, a client would need to first programmatically construct an I'd recommend looking through the |
Co-authored-by: Stuart Montgomery <smontgomery@apple.com>
Co-authored-by: Stuart Montgomery <smontgomery@apple.com>
Thanks for the review! I think a decent next-steps for me would be to go and implement this in my branch, and check out what needs to be modified to actually do this. |
- Makes the Issue initializer public, with a slight cleanup. - Creates the Issue.record() instance method, which just calls out to Issue.record(configuration:) - Marks the Issue.Kind.system case as only available for tools integrations. - Issue.record(configuration:) no longer has a default value of nil, to avoid confusion with the new Issue.record() methods. - Creates a public initializer for Expectation, which creates an empty Expression - Adds an explicit internal initializer for Expectation, where we can pass in Expressions - Allow third parties to read & write to the Expectation.mismatchedErrorDescription and Expectation.differenceDescription properties
|
Update: I added an implementation, which I used with Nimble to actually create an integration. You can check out the basic-swift-testing-support branch on Nimble if you'd like to try it out. Both the xcodeproj and the Package.swift use this branch of swift-testing. Run the |
| /// | ||
| /// If this expectation passed, the value of this property is `nil` because no | ||
| /// error mismatch occurred. | ||
| @_spi(ForToolsIntegrationOnly) |
There was a problem hiding this comment.
This property should probably stay SPI for now as it's a bit hacky and probably not the final design for this data.
| /// If this expectation passed, the value of this property is `nil` because | ||
| /// the difference is only computed when necessary to assist with diagnosing | ||
| /// test failures. | ||
| @_spi(ForToolsIntegrationOnly) |
There was a problem hiding this comment.
This property should probably stay SPI for now as it's a bit hacky and probably not the final design for this data.
| return issue.record(configuration: configuration) | ||
| } | ||
|
|
||
| /// Record this issue by wrapping it in an ``Event`` and passing it to the |
There was a problem hiding this comment.
I don't think we really discuss events or event handlers in our API documentation because they're not otherwise part of our API surface. How about simply:
/// Record this issue.| /// Record this issue by wrapping it in an ``Event`` and passing it to the | ||
| /// current event handler. | ||
| /// | ||
| /// - Returns: The issue that was recorded (`self` or a modified copy of it.) |
There was a problem hiding this comment.
We should make sure the API documentation explains what might be modified (the SPI documentation is, and this is a technical term here, "lazy".)
|
|
||
| /// An issue due to a failure in the underlying system, not due to a failure | ||
| /// within the tests being run. | ||
| @_spi(ForToolsIntegrationOnly) |
There was a problem hiding this comment.
We should leave .system as SPI for now since we don't generally want external components to generate system issues. These represent problems with Swift Testing itself. (We could maybe add an additional case e.g. case integratedToolFailed(description: String), what do you think @stmontgomery ?)
There was a problem hiding this comment.
Just so I understand, you're saying this line is correct? Or is there a more-correct @_spi annotation than ForToolsIntegrationOnly? The SPI doc only lists Experimental and ForToolsIntegrationOnly.
There was a problem hiding this comment.
Yes, adding the SPI marker here is probably the right thing to do, but I'd like @stmontgomery to chime in too.
There was a problem hiding this comment.
I think it's OK to leave .system without an @_spi attribute. Although it's true that users shouldn't generally record system issues themselves, I do think it can be useful for them to match against issues of that kind using other APIs like withKnownIssue. So personally, I don't think we need to add it as part of this proposal/PR.
| /// calling ``SourceContext/init(backtrace:sourceLocation:)`` with zero | ||
| /// arguments. | ||
| init( | ||
| public init( |
There was a problem hiding this comment.
This is fine but will produce a source context for the issue that points into Swift Testing rather than the call site, which is an oversight of the existing code to be fair. As well, we're contemplating lowering Backtrace and SourceContext to SPI since the Swift standard library has its own backtrace type now that we'll eventually want to adopt and we don't want a naming conflict.
How about we expose a different initializer with the signature:
init(
_ kind: Kind,
comments: [Comment] = [],
sourceLocation: SourceLocation = #_sourceLocation
)(Whether or not kind deserves a label is something we can bikeshed.)
| kind: Kind, | ||
| comments: [Comment] = [], | ||
| sourceContext: SourceContext = .init() | ||
| ) {} |
There was a problem hiding this comment.
Proposals don't need to include the bodies of methods/initializers/etc. You can leave the squiggly brackets out here.
| } | ||
| ``` | ||
|
|
||
| Furthermore, to support passing in any `Issue.Kind` to `Issue`, `Expectation` |
There was a problem hiding this comment.
It is intentional that Expectation does not have a public initializer: it is always supposed to be the product of #expect or #require. If you record an issue that does not come from one of those macros, it should not be producing an instance of this type.
|
|
||
| ## Acknowledgments | ||
|
|
||
| I'd like to thank [Jonathon Grynspan](https://github.com/grynspan) and |
| self.isPassing = isPassing | ||
| self.isRequired = isRequired | ||
| self.sourceLocation = sourceLocation | ||
| } |
There was a problem hiding this comment.
I'd like to explore ideas for ways to structure this that avoid needing to publicly expose an init on Expectation. I know I brought that idea up in an earlier comment on this PR, but after reviewing this type again, and taking @grynspan 's comments about the two properties above into consideration, I'm wary to open this type up for general API use right now until it's a bit better designed.
As one alternative, I wonder if we could expose a static func on Issue.Kind which takes a generic message, as well as one or more values of the evaluated expression, and internally construct an Expectation with the relevant Expression and Expression.Value values representing all of the passed-in values? Something like
extension Issue.Kind {
public static func expectationFailed<each T>(
_ description: String,
values: repeat each T,
isRequired: Bool,
sourceLocation: SourceLocation
) -> Self
}This would require some internal change to __Expression but that seems doable, and it would allow capturing rich details of the left- and right-hand side values of binary expressions.
|
If we were to proceed with #490, would we still need this API change? |
With that in mind, I'd like to suggest we close out this PR as well as #474, and focus on that PR. (If you agree that it's an approach that works for you.) We'll want to do an API PR and forum post for that one too, if you're interested in writing it? |
Ok, sure. I'll take that on. |
See #474 for the related bug.
Read the full proposal here!
Motivation
To better facilitate integrating Swift Testing with third party assertion tools, Swift Testing should allow public access to creating and recording Issues directly.
Modifications:
At this stage, the PR is just the pitch doc, and I'm more than happy to contribute the small change as we discuss it. I figured it would be better to agree on the pitch first?
Result:
Once merged, this will open up third-party assertion tools to better integrate with Swift Testing.
Checklist: