Skip to content

swift 5.4.2#73984

Closed
Bo98 wants to merge 1 commit intoHomebrew:masterfrom
Bo98:swift-5.3.3
Closed

swift 5.4.2#73984
Bo98 wants to merge 1 commit intoHomebrew:masterfrom
Bo98:swift-5.3.3

Conversation

@Bo98
Copy link
Copy Markdown
Member

@Bo98 Bo98 commented Mar 26, 2021

  • Have you followed the guidelines for contributing?
  • Have you checked that there aren't other open pull requests for the same formula update/change?
  • Have you built your formula locally with brew install --build-from-source <formula>, where <formula> is the name of the formula you're submitting?
  • Is your test running fine brew test <formula>, where <formula> is the name of the formula you're submitting?
  • Does your build pass brew audit --strict <formula> (after doing brew install <formula>)?

Oh boy, it's debugging Swift time.

This contains some Linux changes which I have not tested at all yet (but will do over at linuxbrew-core).

@Bo98 Bo98 added the swift Swift use is a significant feature of the PR or issue label Mar 26, 2021
@BrewTestBot BrewTestBot removed the swift Swift use is a significant feature of the PR or issue label Mar 26, 2021
@carlocab carlocab added CI-force-arm [DEPRECATED] Don't pass --skip-unbottled-arm to brew test-bot. swift Swift use is a significant feature of the PR or issue labels Mar 26, 2021
@BrewTestBot BrewTestBot removed the swift Swift use is a significant feature of the PR or issue label Mar 26, 2021
@carlocab
Copy link
Copy Markdown
Member

ninja: error: 'stdlib/swift-test-stdlib-macosx-x86_64', needed by 'stdlib/CMakeFiles/swift-test-stdlib', missing and no known rule to make it
Building the standard library for: swift-stdlib-macosx-arm64
ERROR: command terminated with a non-zero exit status 1, aborting

@Bo98
Copy link
Copy Markdown
Member Author

Bo98 commented Mar 26, 2021

Will need swiftlang/swift@e06bccd on Big Sur. Will be interesting to see how Catalina does.

I might wait for Swift 5.4, since I suspect it will release very soon - very likely this coming week. It might make the Linux side of things easier.

@Bo98
Copy link
Copy Markdown
Member Author

Bo98 commented Mar 27, 2021

Might as well try the 5.4 snapshots and get those working rather than waste time on 5.3.3.

@Bo98 Bo98 added the prerelease-testing Pull request from upstream, testing a pre-release with homebrew dependencies label Mar 27, 2021
@Bo98 Bo98 changed the title swift 5.3.3 swift 5.4 Mar 27, 2021
@Bo98 Bo98 force-pushed the swift-5.3.3 branch 4 times, most recently from c356c8e to bab37ee Compare March 27, 2021 03:56
@Bo98
Copy link
Copy Markdown
Member Author

Bo98 commented Mar 27, 2021

11.0 ARM fails on: Building llbuild for target: x86_64-apple-macos10.15
10.15 Intel fails on: Building llbuild for target: arm64-apple-macos10.15

A bit backwards?

@carlocab
Copy link
Copy Markdown
Member

Well, that swift isn't effortless to build on a Mac is also a bit backward, I think, so...

@Bo98 Bo98 force-pushed the swift-5.3.3 branch 3 times, most recently from 3e26aab to 8067310 Compare March 28, 2021 01:33
@BrewTestBot BrewTestBot added the python Python use is a significant feature of the PR or issue label Mar 28, 2021
@Bo98 Bo98 force-pushed the swift-5.3.3 branch 2 times, most recently from a5797f9 to 7a6adb9 Compare March 28, 2021 04:02
@carlocab
Copy link
Copy Markdown
Member

Nice. What're the chances your changes here would just work for swift 5.3.3?

@Bo98
Copy link
Copy Markdown
Member Author

Bo98 commented Mar 28, 2021

Intel: maybe if I delete a couple resources.
Apple Silicon: not without patches.
Linux: not a chance - 5.3 requires Python 2.

I expect 5.4 to be released in a couple days anyway.

@Bo98
Copy link
Copy Markdown
Member Author

Bo98 commented Aug 22, 2021

In fact, I'll fill that time by updating this to 5.4.2 so I can see the latest logs.

@Bo98 Bo98 changed the title swift 5.4 swift 5.4.2 Aug 22, 2021
@BrewTestBot BrewTestBot added the automerge-skip `brew pr-automerge` will skip this pull request label Aug 22, 2021
@Bo98 Bo98 added the CI-linux-self-hosted Build on Linux self-hosted runner label Aug 23, 2021
@Bo98
Copy link
Copy Markdown
Member Author

Bo98 commented Aug 23, 2021

Building a stdlib is problematic on macOS and to be honest is probably unnecessary. Xcode toolchains no longer include it and rely on those shipped with the OS.

Trying a build now without it.

@Bo98
Copy link
Copy Markdown
Member Author

Bo98 commented Aug 31, 2021

Ok, everything now works except arm64.

I do not have an arm64 machine to properly debug this at the moment. I might borrow time from one of our CI machines or something.

@carlocab
Copy link
Copy Markdown
Member

carlocab commented Aug 31, 2021

Here's the error log:

2021-08-31T02:17:04.5908990Z --- bootstrap: note: Building TSC
2021-08-31T02:17:04.5909620Z -- The C compiler identification is AppleClang 12.0.5.12050022
2021-08-31T02:17:04.5910320Z -- The Swift compiler identification is Apple 5.4.2
2021-08-31T02:17:04.5910910Z -- Detecting C compiler ABI info
2021-08-31T02:17:04.5911470Z -- Detecting C compiler ABI info - done
2021-08-31T02:17:04.5912200Z -- Check for working C compiler: /opt/homebrew/Library/Homebrew/shims/mac/super/clang - skipped
2021-08-31T02:17:04.5912990Z -- Detecting C compile features
2021-08-31T02:17:04.5913560Z -- Detecting C compile features - done
2021-08-31T02:17:04.5914360Z -- Check for working Swift compiler: /opt/homebrew/Cellar/swift/5.4.2/Swift-5.4.2.xctoolchain/usr/bin/swiftc
2021-08-31T02:17:04.5915410Z -- Check for working Swift compiler: /opt/homebrew/Cellar/swift/5.4.2/Swift-5.4.2.xctoolchain/usr/bin/swiftc - broken
2021-08-31T02:17:04.5916360Z CMake Error at /opt/homebrew/Cellar/cmake/3.21.2/share/cmake/Modules/CMakeTestSwiftCompiler.cmake:44 (message):
2021-08-31T02:17:04.5917070Z   The Swift compiler
2021-08-31T02:17:04.5917280Z 
2021-08-31T02:17:04.5917870Z     "/opt/homebrew/Cellar/swift/5.4.2/Swift-5.4.2.xctoolchain/usr/bin/swiftc"
2021-08-31T02:17:04.5918260Z 
2021-08-31T02:17:04.5918570Z   is not able to compile a simple test program.
2021-08-31T02:17:04.5918860Z 
2021-08-31T02:17:04.5919170Z   It fails with the following output:
2021-08-31T02:17:04.5919440Z 
2021-08-31T02:17:04.5920240Z     Change Dir: /tmp/swift-20210830-42193-16wc9og/build/swiftpm-macosx-arm64/arm64-apple-macosx/tsc/CMakeFiles/CMakeTmp
2021-08-31T02:17:04.5920910Z     
2021-08-31T02:17:04.5921390Z     Run Build Command(s):/opt/homebrew/opt/ninja/bin/ninja cmTC_0460a && [1/1][100%][0.024s] Linking Swift executable cmTC_0460a
2021-08-31T02:17:04.5922090Z     FAILED: cmTC_0460a CMakeFiles/cmTC_0460a.dir/main.swift.o cmTC_0460a.swiftmodule 
2021-08-31T02:17:04.5926570Z     : && /opt/homebrew/Cellar/swift/5.4.2/Swift-5.4.2.xctoolchain/usr/bin/swiftc -output-file-map CMakeFiles/cmTC_0460a.dir//output-file-map.json -incremental -j 8 -emit-executable -o cmTC_0460a -emit-dependencies  -sdk /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX11.3.sdk -module-cache-path "/private/tmp/swift-20210830-42193-16wc9og/build/swiftpm-macosx-arm64/arm64-apple-macosx/tsc/module-cache"  -sdk /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX11.3.sdk -sdk /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX11.3.sdk -module-cache-path "/private/tmp/swift-20210830-42193-16wc9og/build/swiftpm-macosx-arm64/arm64-apple-macosx/tsc/module-cache"  -sdk /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX11.3.sdk  /tmp/swift-20210830-42193-16wc9og/build/swiftpm-macosx-arm64/arm64-apple-macosx/tsc/CMakeFiles/CMakeTmp/main.swift    && :
2021-08-31T02:17:04.5931280Z     <unknown>:0: error: unable to load standard library for target 'arm64-apple-macosx11.0'
2021-08-31T02:17:04.5931910Z     ninja: build stopped: subcommand failed.
2021-08-31T02:17:04.5932240Z     
2021-08-31T02:17:04.5932460Z     
2021-08-31T02:17:04.5932620Z 
2021-08-31T02:17:04.5932830Z   
2021-08-31T02:17:04.5932990Z 
2021-08-31T02:17:04.5933450Z   CMake will not be able to correctly generate this project.
2021-08-31T02:17:04.5933980Z Call Stack (most recent call first):
2021-08-31T02:17:04.5934410Z   CMakeLists.txt:3 (project)
2021-08-31T02:17:04.5934690Z 
2021-08-31T02:17:04.5934860Z 
2021-08-31T02:17:04.5935400Z -- Configuring incomplete, errors occurred!
2021-08-31T02:17:04.5936650Z See also "/tmp/swift-20210830-42193-16wc9og/build/swiftpm-macosx-arm64/arm64-apple-macosx/tsc/CMakeFiles/CMakeOutput.log".
2021-08-31T02:17:04.5938080Z See also "/tmp/swift-20210830-42193-16wc9og/build/swiftpm-macosx-arm64/arm64-apple-macosx/tsc/CMakeFiles/CMakeError.log".

The referenced CMakeOutput.log and CMakeError.log should be in the logs artifact uploaded by CI. This was a lie; their build script put things outside of buildpath

carlocab
carlocab previously approved these changes Aug 31, 2021
Copy link
Copy Markdown
Member

@carlocab carlocab left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this is good to merge, as this formula never had an ARM bottle to begin with.

@Bo98
Copy link
Copy Markdown
Member Author

Bo98 commented Aug 31, 2021

unable to load standard library for target 'arm64-apple-macosx11.0'

Doesn't really make sense since this should be shipped by macOS in /usr/lib/swift. The x86_64 build finds it without issue.

(We can build the stdlib ourselves, but there's something not right and inconsistent here.)

@finagolfin
Copy link
Copy Markdown

The Swift compiler is now very particular about the target passed in, to the point where it started rejecting armv7-none-linux-androideabi last summer, swiftlang/swift#34919, but would accept it as a synonym for armv7-unknown-linux-androideabi before. Where is that target coming from, are you passing it in yourself?

I checked a macOS toolchain build log for a pull of mine and the CI doesn't build TSC with CMake as you are here, even though it cross-compiles the toolchain for Mac arm64 from x86_64 in that CI run. If you're natively compiling the Swift toolchain on Mac arm64, it's possible that's not fully open-sourced by Apple yet, as with swiftlang/swift#34998.

@Bo98
Copy link
Copy Markdown
Member Author

Bo98 commented Aug 31, 2021

The target is correct. In fact, the triplets in swift -print-target-info match that of Apple's build exactly. Apple Silicon host patches like swiftlang/swift#34998 did make it into Swift 5.4, some at a very late stage.

The issue is one not obvious in the triplets themselves. Apple's build, although it reports itself as arm64 in the triplet, is built as arm64e. The Swift modules shipped in the SDK are also arm64e. It turns out that an arm64 build does not check for arm64e modules and so as such our build cannot find them in the SDK (and building for arm64e is not supported outside Apple).

This is being fixed upstream in swiftlang/swift#39083. Applying that patch here (with a minor tweak to make it work on the 5.4 branch) fixes the problem.

I also found a second, separate issue to do with triplets over in LLDB, which is invoked if you start Swift in REPL mode. It would fail to handle the arm64e CPU subtype properly and return a triplet of unknown-apple-macosx11.4--. This was fixed by applying swiftlang/llvm-project@479b672. This patch has not been backported to the Swift 5.5 branch so we'll need to do it ourselves for a while (it however applies cleanly and doesn't require modification).

@carlocab
Copy link
Copy Markdown
Member

carlocab commented Sep 1, 2021

Is this ready @Bo98? LGTM

@carlocab carlocab mentioned this pull request Sep 1, 2021
@Bo98
Copy link
Copy Markdown
Member Author

Bo98 commented Sep 1, 2021

If you're happy with how all the changes look, then I'm happy for this to be merged.

I managed to make the macOS bottles relocatable which is good. Linux bottles are not yet - I'll try look into getting that sorted (if possible) before we start using it as a dependency because it's not exactly a quick thing to build.

@Bo98 Bo98 marked this pull request as ready for review September 1, 2021 11:57
Comment on lines +151 to +155
resources.each do |r|
next if r.name == "six"

r.stage(workspace/r.name)
end
Copy link
Copy Markdown
Contributor

@bayandin bayandin Sep 1, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Since we don't use six as a resource anymore we can revert this part back (could be done later to not to retrigger the build)

Suggested change
resources.each do |r|
next if r.name == "six"
r.stage(workspace/r.name)
end
resources.each { |r| r.stage(workspace/r.name) }

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I could probably cheat and just amend the commit while merging via pr-pull.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Bottles are 3 GB. pr-pull might take a while.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've dealt with LLVM back when it was all statically linked and we didn't have pr-publish, it'll probably be fine.

@Bo98 Bo98 removed help wanted Task(s) needing PRs from the community or maintainers do not merge in progress Stale bot should stay away labels Sep 1, 2021
@Bo98 Bo98 closed this in 5043011 Sep 1, 2021
@github-actions github-actions bot added the outdated PR was locked due to age label Oct 2, 2021
@github-actions github-actions bot locked as resolved and limited conversation to collaborators Oct 2, 2021
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

automerge-skip `brew pr-automerge` will skip this pull request CI-force-arm [DEPRECATED] Don't pass --skip-unbottled-arm to brew test-bot. CI-force-linux [DEPRECATED] Don't pass --skip-unbottled-linux to brew test-bot. CI-linux-self-hosted Build on Linux self-hosted runner CI-long-timeout [DEPRECATED] Use longer GitHub Actions CI timeout. CI-no-fail-fast Continue CI tests despite failing GitHub Actions matrix builds. outdated PR was locked due to age python Python use is a significant feature of the PR or issue

Projects

None yet

Development

Successfully merging this pull request may close these issues.

7 participants