Skip to content

Store test content in a custom metadata section.#880

Merged
grynspan merged 6 commits into
mainfrom
jgrynspan/custom-metadata-section
Mar 11, 2025
Merged

Store test content in a custom metadata section.#880
grynspan merged 6 commits into
mainfrom
jgrynspan/custom-metadata-section

Conversation

@grynspan

@grynspan grynspan commented Jan 2, 2025

Copy link
Copy Markdown
Contributor

This PR uses the experimental symbol linkage margers feature in the Swift compiler to emit metadata about tests (and exit tests) into a dedicated section of the test executable being built. At runtime, we discover that section and read out the tests from it.

This has several benefits over our current model, which involves walking Swift's type metadata table looking for types that conform to a protocol:

  1. We don't need to define that protocol as public API in Swift Testing,
  2. We don't need to emit type metadata (much larger than what we really need) for every test function,
  3. We don't need to duplicate a large chunk of the Swift ABI sources in order to walk the type metadata table correctly, and
  4. Almost all the new code is written in Swift, whereas the code it is intended to replace could not be fully represented in Swift and needed to be written in C++.

This change will be necessary to support Embedded Swift because there is no type metadata section emitted for embedded targets.

The change also opens up the possibility of supporting generic types in the future because we can emit metadata without needing to emit a nested type (which is not always valid in a generic context.) That's a "future direction" and not covered by this PR specifically.

I've defined a layout for entries in the new swift5_tests section that should be flexible enough for us in the short-to-medium term and which lets us define additional arbitrary test content record types. The layout of this section is covered in depth in the new TestContent.md article.

This functionality is only available if a test target enables the experimental "SymbolLinkageMarkers" feature and only if Swift Testing is used as a package (not as a toolchain component.) We continue to emit protocol-conforming types for now—that code will be removed if and when the experimental feature is properly supported (modulo us adopting relevant changes to the feature's API.)

See Also

#735
#764
swiftlang/swift#76698
swiftlang/swift#78411

Checklist:

  • Code and documentation should follow the style of the Style Guide.
  • If public symbols are renamed or modified, DocC references should be updated.

@grynspan grynspan added enhancement New feature or request tools integration 🛠️ Integration of swift-testing into tools/IDEs windows 🪟 Windows support linux 🐧 Linux support (all distros) darwin 🍎 macOS, iOS, watchOS, tvOS, and visionOS support performance 🏎️ Performance issues embedded-swift 📟 Embedded Swift issues wasi/wasm 🧭 WebAssembly support android 🤖 Android support freebsd 😈 FreeBSD support labels Jan 2, 2025
@grynspan grynspan added this to the Swift 6.x milestone Jan 2, 2025
@grynspan grynspan self-assigned this Jan 2, 2025
@grynspan grynspan force-pushed the jgrynspan/custom-metadata-section branch from e98f3bc to bb2526a Compare January 3, 2025 20:22
Comment thread Documentation/ABI/TestContent.md Outdated
Comment thread Documentation/ABI/TestContent.md Outdated
Comment thread Documentation/ABI/TestContent.md Outdated
Comment thread Documentation/ABI/TestContent.md
Comment thread Documentation/ABI/TestContent.md
Comment thread Documentation/ABI/TestContent.md
Comment thread Documentation/ABI/TestContent.md Outdated
Comment thread Documentation/ABI/TestContent.md Outdated
Comment thread Documentation/ABI/TestContent.md Outdated
Comment thread Documentation/Porting.md Outdated
@grynspan grynspan requested a review from compnerd January 5, 2025 17:54
@grynspan

grynspan commented Jan 5, 2025

Copy link
Copy Markdown
Contributor Author

@swift-ci test

Comment thread Documentation/ABI/TestContent.md Outdated
Comment thread Documentation/ABI/TestContent.md Outdated
@grynspan

grynspan commented Jan 6, 2025

Copy link
Copy Markdown
Contributor Author

@swift-ci test

@grynspan

grynspan commented Jan 6, 2025

Copy link
Copy Markdown
Contributor Author

@swift-ci test Linux

@grynspan

grynspan commented Jan 6, 2025

Copy link
Copy Markdown
Contributor Author

@swift-ci test Windows

1 similar comment
@grynspan

grynspan commented Jan 6, 2025

Copy link
Copy Markdown
Contributor Author

@swift-ci test Windows

@grynspan grynspan force-pushed the jgrynspan/custom-metadata-section branch 2 times, most recently from fe3dbab to 713b032 Compare January 6, 2025 20:17
@grynspan

grynspan commented Jan 6, 2025

Copy link
Copy Markdown
Contributor Author

@swift-ci test

@grynspan

grynspan commented Jan 6, 2025

Copy link
Copy Markdown
Contributor Author

@swift-ci test macOS

@grynspan

Copy link
Copy Markdown
Contributor Author

@swift-ci test

@grynspan grynspan force-pushed the jgrynspan/custom-metadata-section branch from 266b53b to 9a93f24 Compare March 5, 2025 13:49
@grynspan

grynspan commented Mar 5, 2025

Copy link
Copy Markdown
Contributor Author

@swift-ci test

@grynspan grynspan force-pushed the jgrynspan/custom-metadata-section branch from 9a93f24 to e3e5cfe Compare March 7, 2025 22:42
@grynspan

grynspan commented Mar 7, 2025

Copy link
Copy Markdown
Contributor Author

@swift-ci test

@grynspan grynspan force-pushed the jgrynspan/custom-metadata-section branch 2 times, most recently from 950c904 to 54a7db9 Compare March 10, 2025 19:20
This PR uses the experimental symbol linkage margers feature in the Swift
compiler to emit metadata about tests (and exit tests) into a dedicated section
of the test executable being built. At runtime, we discover that section and
read out the tests from it.

This has several benefits over our current model, which involves walking Swift's
type metadata table looking for types that conform to a protocol:

1. We don't need to define that protocol as public API in Swift Testing,
1. We don't need to emit type metadata (much larger than what we really need)
   for every test function,
1. We don't need to duplicate a large chunk of the Swift ABI sources in order to
   walk the type metadata table correctly, and
1. Almost all the new code is written in Swift, whereas the code it is intended
   to replace could not be fully represented in Swift and needed to be written
   in C++.

The change also opens up the possibility of supporting generic types in the
future because we can emit metadata without needing to emit a nested type (which
is not always valid in a generic context.) That's a "future direction" and not
covered by this PR specifically.

I've defined a layout for entries in the new `swift5_tests` section that should
be flexible enough for us in the short-to-medium term and which lets us define
additional arbitrary test content record types. The layout of this section is
covered in depth in the new [TestContent.md](Documentation/ABI/TestContent.md)
article.

This functionality is only available if a test target enables the experimental
`"SymbolLinkageMarkers"` feature. We continue to emit protocol-conforming types
for now—that code will be removed if and when the experimental feature is
properly supported (modulo us adopting relevant changes to the feature's API.)

#735
swiftlang/swift#76698
swiftlang/swift#78411
@grynspan grynspan force-pushed the jgrynspan/custom-metadata-section branch from 54a7db9 to abc1825 Compare March 10, 2025 20:59
@grynspan

Copy link
Copy Markdown
Contributor Author

@swift-ci test

Comment thread Documentation/Porting.md Outdated
@grynspan

Copy link
Copy Markdown
Contributor Author

@swift-ci test

@grynspan

Copy link
Copy Markdown
Contributor Author

@swift-ci test

@grynspan

Copy link
Copy Markdown
Contributor Author

@swift-ci test

@grynspan

Copy link
Copy Markdown
Contributor Author

@swift-ci test

@grynspan

Copy link
Copy Markdown
Contributor Author

@swift-ci test macOS

@grynspan

Copy link
Copy Markdown
Contributor Author

@swift-ci test

@grynspan

Copy link
Copy Markdown
Contributor Author

@swift-ci test Linux

@grynspan

Copy link
Copy Markdown
Contributor Author

@swift-ci test macOS

@grynspan

Copy link
Copy Markdown
Contributor Author

@swift-ci test Windows

@grynspan

Copy link
Copy Markdown
Contributor Author

@swift-ci test macOS

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

android 🤖 Android support darwin 🍎 macOS, iOS, watchOS, tvOS, and visionOS support discovery 🔎 test content discovery embedded-swift 📟 Embedded Swift issues enhancement New feature or request freebsd 😈 FreeBSD support less-c++ 🐀 Work to reduce the size of our C++ codebase and/or dependencies linux 🐧 Linux support (all distros) performance 🏎️ Performance issues tools integration 🛠️ Integration of swift-testing into tools/IDEs wasi/wasm 🧭 WebAssembly support windows 🪟 Windows support

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants