Skip to content

Add a Swift parser library that is written in Swift#616

Merged
DougGregor merged 3 commits intomainfrom
parser-library
Aug 22, 2022
Merged

Add a Swift parser library that is written in Swift#616
DougGregor merged 3 commits intomainfrom
parser-library

Conversation

@ahoppen
Copy link
Copy Markdown
Member

@ahoppen ahoppen commented Aug 22, 2022

This starts an effort to implement the parser for the Swift compiler in Swift itself.

For more information, please see the announcement forum post at: https://forums.swift.org/t/a-new-swift-parser-for-swiftsyntax/59813

This starts an effort to implement the parser for the Swift compiler in Swift itself.

Co-authored-by: Robert Widmann <devteam.codafi@gmail.com>
Co-authored-by: Doug Gregor <dgregor@apple.com>
Co-authored-by: Rintaro Ishizaki <rishizaki@apple.com>
@ahoppen
Copy link
Copy Markdown
Member Author

ahoppen commented Aug 22, 2022

@swift-ci Please test

@etcwilde
Copy link
Copy Markdown
Member

@compnerd, if you have a cycle or two, it's probably worth making sure this works on Windows.

@compnerd
Copy link
Copy Markdown
Member

Doesn't seem to build on Windows (which is a regression). At the very least we will need swiftlang/swift-docc-plugin#26.

@CodaFi
Copy link
Copy Markdown
Contributor

CodaFi commented Aug 22, 2022

The docc plugin is not required for this repository in general, and is not a dependency for this PR specifically. Docc support is built into Xcode, I am not sure why we have a plugin dependency as well.

@compnerd
Copy link
Copy Markdown
Member

I couldn't answer why we have that. We should invert the condition so that it is not used by default.

@DougGregor
Copy link
Copy Markdown
Member

I'm fine with removing the DocC plugin dependency or inverting the condition. We shouldn't land this in a form that breaks the Windows build.

@CodaFi
Copy link
Copy Markdown
Contributor

CodaFi commented Aug 22, 2022

Let's strike it for now. We shouldn't have merged it without verifying that it worked for Windows

#617

Copy link
Copy Markdown
Member

@compnerd compnerd left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This at least allows this PR to be built on Windows with:

swift build -Xcc -IC:\Library\Developer\Toolchains\unknown-Asserts-development.xctoolchain\usr\include

Note that building this requires a locally built toolchain as there was a fix for SPM that is needed that has not had a new toolchain snapshot built due to other failures introduced by the rebranch.

Thank you @compnerd!

Co-authored-by: Saleem Abdulrasool <compnerd@compnerd.org>
@DougGregor
Copy link
Copy Markdown
Member

@swift-ci please test

1 similar comment
@DougGregor
Copy link
Copy Markdown
Member

@swift-ci please test

@DougGregor
Copy link
Copy Markdown
Member

@swift-ci please test

@@ -0,0 +1,70 @@
## Overview
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

(not a blocker) The existence of this file triggers a warning, at least with 5.7:

warning: 'swift-syntax': found 1 file(s) which are unhandled; explicitly declare them as resources or exclude from the target
    Sources/SwiftParser/README.md

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'll merge and we'll fix it up afterward. Thanks!

@wu-s-john
Copy link
Copy Markdown

Nice. Swift would really benefit from code generating it's own code rather than C++.

@broken-bytes
Copy link
Copy Markdown

I can assure you that this does currently not work on Windows. I just tried using this framework in an SPM project on Windows.

I am getting the following error:

[41/88] Compiling SwiftSyntax AtomicCounter.swift
Please submit a bug report (https://swift.org/contributing/#reporting-bugs) and include the project and the crash backtrace.
Stack dump:
0.      Program arguments: C:\\Library\\Developer\\Toolchains\\unknown-Asserts-development.xctoolchain\\usr\\bin\\swiftc.exe -frontend -c H:\\Projects\\Cyanite-Rewrite\\cyanite-engine\\cyanite\\api\\.build\\checkouts\\swift-syntax\\Sources\\SwiftSyntax\\AbsolutePosition.swift H:\\Projects\\Cyanite-Rewrite\\cyanite-engine\\cyanite\\api\\.build\\checkouts\\swift-syntax\\Sources\\SwiftSyntax\\AtomicCounter.swift H:\\Projects\\Cyanite-Rewrite\\cyanite-engine\\cyanite\\api\\.build\\checkouts\\swift-syntax\\Sources\\SwiftSyntax\\CNodes.swift H:\\Projects\\Cyanite-Rewrite\\cyanite-engine\\cyanite\\api\\.build\\checkouts\\swift-syntax\\Sources\\SwiftSyntax\\IncrementalParseTransition.swift H:\\Projects\\Cyanite-Rewrite\\cyanite-engine\\cyanite\\api\\.build\\checkouts\\swift-syntax\\Sources\\SwiftSyntax\\RawSyntax.swift H:\\Projects\\Cyanite-Rewrite\\cyanite-engine\\cyanite\\api\\.build\\checkouts\\swift-syntax\\Sources\\SwiftSyntax\\SourceLength.swift -primary-file H:\\Projects\\Cyanite-Rewrite\\cyanite-engine\\cyanite\\api\\.build\\checkouts\\swift-syntax\\Sources\\SwiftSyntax\\SourceLocation.swift -primary-file H:\\Projects\\Cyanite-Rewrite\\cyanite-engine\\cyanite\\api\\.build\\checkouts\\swift-syntax\\Sources\\SwiftSyntax\\SourcePresence.swift H:\\Projects\\Cyanite-Rewrite\\cyanite-engine\\cyanite\\api\\.build\\checkouts\\swift-syntax\\Sources\\SwiftSyntax\\Syntax.swift H:\\Projects\\Cyanite-Rewrite\\cyanite-engine\\cyanite\\api\\.build\\checkouts\\swift-syntax\\Sources\\SwiftSyntax\\SyntaxChildren.swift H:\\Projects\\Cyanite-Rewrite\\cyanite-engine\\cyanite\\api\\.build\\checkouts\\swift-syntax\\Sources\\SwiftSyntax\\SyntaxClassifier.swift H:\\Projects\\Cyanite-Rewrite\\cyanite-engine\\cyanite\\api\\.build\\checkouts\\swift-syntax\\Sources\\SwiftSyntax\\SyntaxData.swift H:\\Projects\\Cyanite-Rewrite\\cyanite-engine\\cyanite\\api\\.build\\checkouts\\swift-syntax\\Sources\\SwiftSyntax\\SyntaxOtherNodes.swift H:\\Projects\\Cyanite-Rewrite\\cyanite-engine\\cyanite\\api\\.build\\checkouts\\swift-syntax\\Sources\\SwiftSyntax\\SyntaxVerifier.swift H:\\Projects\\Cyanite-Rewrite\\cyanite-engine\\cyanite\\api\\.build\\checkouts\\swift-syntax\\Sources\\SwiftSyntax\\Utils.swift H:\\Projects\\Cyanite-Rewrite\\cyanite-engine\\cyanite\\api\\.build\\checkouts\\swift-syntax\\Sources\\SwiftSyntax\\_SyntaxParserInterop.swift H:\\Projects\\Cyanite-Rewrite\\cyanite-engine\\cyanite\\api\\.build\\checkouts\\swift-syntax\\Sources\\SwiftSyntax\\gyb_generated\\Misc.swift H:\\Projects\\Cyanite-Rewrite\\cyanite-engine\\cyanite\\api\\.build\\checkouts\\swift-syntax\\Sources\\SwiftSyntax\\gyb_generated\\SyntaxAnyVisitor.swift H:\\Projects\\Cyanite-Rewrite\\cyanite-engine\\cyanite\\api\\.build\\checkouts\\swift-syntax\\Sources\\SwiftSyntax\\gyb_generated\\SyntaxBaseNodes.swift H:\\Projects\\Cyanite-Rewrite\\cyanite-engine\\cyanite\\api\\.build\\checkouts\\swift-syntax\\Sources\\SwiftSyntax\\gyb_generated\\SyntaxBuilders.swift H:\\Projects\\Cyanite-Rewrite\\cyanite-engine\\cyanite\\api\\.build\\checkouts\\swift-syntax\\Sources\\SwiftSyntax\\gyb_generated\\SyntaxClassification.swift H:\\Projects\\Cyanite-Rewrite\\cyanite-engine\\cyanite\\api\\.build\\checkouts\\swift-syntax\\Sources\\SwiftSyntax\\gyb_generated\\SyntaxCollections.swift H:\\Projects\\Cyanite-Rewrite\\cyanite-engine\\cyanite\\api\\.build\\checkouts\\swift-syntax\\Sources\\SwiftSyntax\\gyb_generated\\SyntaxEnum.swift H:\\Projects\\Cyanite-Rewrite\\cyanite-engine\\cyanite\\api\\.build\\checkouts\\swift-syntax\\Sources\\SwiftSyntax\\gyb_generated\\SyntaxFactory.swift H:\\Projects\\Cyanite-Rewrite\\cyanite-engine\\cyanite\\api\\.build\\checkouts\\swift-syntax\\Sources\\SwiftSyntax\\gyb_generated\\SyntaxKind.swift H:\\Projects\\Cyanite-Rewrite\\cyanite-engine\\cyanite\\api\\.build\\checkouts\\swift-syntax\\Sources\\SwiftSyntax\\gyb_generated\\SyntaxRewriter.swift H:\\Projects\\Cyanite-Rewrite\\cyanite-engine\\cyanite\\api\\.build\\checkouts\\swift-syntax\\Sources\\SwiftSyntax\\gyb_generated\\SyntaxTraits.swift H:\\Projects\\Cyanite-Rewrite\\cyanite-engine\\cyanite\\api\\.build\\checkouts\\swift-syntax\\Sources\\SwiftSyntax\\gyb_generated\\SyntaxVisitor.swift H:\\Projects\\Cyanite-Rewrite\\cyanite-engine\\cyanite\\api\\.build\\checkouts\\swift-syntax\\Sources\\SwiftSyntax\\gyb_generated\\TokenKind.swift H:\\Projects\\Cyanite-Rewrite\\cyanite-engine\\cyanite\\api\\.build\\checkouts\\swift-syntax\\Sources\\SwiftSyntax\\gyb_generated\\Trivia.swift H:\\Projects\\Cyanite-Rewrite\\cyanite-engine\\cyanite\\api\\.build\\checkouts\\swift-syntax\\Sources\\SwiftSyntax\\gyb_generated\\syntax_nodes\\SyntaxDeclNodes.swift H:\\Projects\\Cyanite-Rewrite\\cyanite-engine\\cyanite\\api\\.build\\checkouts\\swift-syntax\\Sources\\SwiftSyntax\\gyb_generated\\syntax_nodes\\SyntaxExprNodes.swift H:\\Projects\\Cyanite-Rewrite\\cyanite-engine\\cyanite\\api\\.build\\checkouts\\swift-syntax\\Sources\\SwiftSyntax\\gyb_generated\\syntax_nodes\\SyntaxNodes.swift H:\\Projects\\Cyanite-Rewrite\\cyanite-engine\\cyanite\\api\\.build\\checkouts\\swift-syntax\\Sources\\SwiftSyntax\\gyb_generated\\syntax_nodes\\SyntaxPatternNodes.swift H:\\Projects\\Cyanite-Rewrite\\cyanite-engine\\cyanite\\api\\.build\\checkouts\\swift-syntax\\Sources\\SwiftSyntax\\gyb_generated\\syntax_nodes\\SyntaxStmtNodes.swift H:\\Projects\\Cyanite-Rewrite\\cyanite-engine\\cyanite\\api\\.build\\checkouts\\swift-syntax\\Sources\\SwiftSyntax\\gyb_generated\\syntax_nodes\\SyntaxTypeNodes.swift -emit-module-path H:\\Projects\\Cyanite-Rewrite\\cyanite-engine\\cyanite\\api\\.build\\x86_64-unknown-windows-msvc\\debug\\SwiftSyntax.build\\SourceLocation~partial.swiftmodule -emit-module-path H:\\Projects\\Cyanite-Rewrite\\cyanite-engine\\cyanite\\api\\.build\\x86_64-unknown-windows-msvc\\debug\\SwiftSyntax.build\\SourcePresence~partial.swiftmodule -emit-module-doc-path H:\\Projects\\Cyanite-Rewrite\\cyanite-engine\\cyanite\\api\\.build\\x86_64-unknown-windows-msvc\\debug\\SwiftSyntax.build\\SourceLocation~partial.swiftdoc -emit-module-doc-path H:\\Projects\\Cyanite-Rewrite\\cyanite-engine\\cyanite\\api\\.build\\x86_64-unknown-windows-msvc\\debug\\SwiftSyntax.build\\SourcePresence~partial.swiftdoc -emit-module-source-info-path H:\\Projects\\Cyanite-Rewrite\\cyanite-engine\\cyanite\\api\\.build\\x86_64-unknown-windows-msvc\\debug\\SwiftSyntax.build\\SourceLocation~partial.swiftsourceinfo -emit-module-source-info-path H:\\Projects\\Cyanite-Rewrite\\cyanite-engine\\cyanite\\api\\.build\\x86_64-unknown-windows-msvc\\debug\\SwiftSyntax.build\\SourcePresence~partial.swiftsourceinfo -emit-dependencies-path H:\\Projects\\Cyanite-Rewrite\\cyanite-engine\\cyanite\\api\\.build\\x86_64-unknown-windows-msvc\\debug\\SwiftSyntax.build\\SourceLocation.d -emit-dependencies-path H:\\Projects\\Cyanite-Rewrite\\cyanite-engine\\cyanite\\api\\.build\\x86_64-unknown-windows-msvc\\debug\\SwiftSyntax.build\\SourcePresence.d -emit-reference-dependencies-path H:\\Projects\\Cyanite-Rewrite\\cyanite-engine\\cyanite\\api\\.build\\x86_64-unknown-windows-msvc\\debug\\SwiftSyntax.build\\SourceLocation.swiftdeps -emit-reference-dependencies-path H:\\Projects\\Cyanite-Rewrite\\cyanite-engine\\cyanite\\api\\.build\\x86_64-unknown-windows-msvc\\debug\\SwiftSyntax.build\\SourcePresence.swiftdeps -target x86_64-unknown-windows-msvc -disable-objc-interop -sdk C:\\Library\\Developer\\Platforms\\Windows.platform\\Developer\\SDKs\\Windows.sdk -I H:\\Projects\\Cyanite-Rewrite\\cyanite-engine\\cyanite\\api\\.build\\x86_64-unknown-windows-msvc\\debug -I C:\\Library\\Developer\\Platforms\\Windows.platform\\Developer\\Library\\XCTest-development\\usr\\lib\\swift\\windows\\x86_64 -enable-testing -g -module-cache-path H:\\Projects\\Cyanite-Rewrite\\cyanite-engine\\cyanite\\api\\.build\\x86_64-unknown-windows-msvc\\debug\\ModuleCache -swift-version 5 -Onone -D SWIFT_PACKAGE -D DEBUG -enable-anonymous-context-mangled-names -Xcc -fmodule-map-file=H:\\Projects\\Cyanite-Rewrite\\cyanite-engine\\cyanite\\api\\.build\\x86_64-unknown-windows-msvc\\debug\\_CSwiftSyntax.build\\module.modulemap -Xcc -I -Xcc H:\\Projects\\Cyanite-Rewrite\\cyanite-engine\\cyanite\\api\\.build\\checkouts\\swift-syntax\\Sources\\_CSwiftSyntax\\include -autolink-library oldnames -autolink-library msvcrt -Xcc -D_MT -Xcc -D_DLL -parse-as-library -module-name SwiftSyntax -o H:\\Projects\\Cyanite-Rewrite\\cyanite-engine\\cyanite\\api\\.build\\x86_64-unknown-windows-msvc\\debug\\SwiftSyntax.build\\SourceLocation.swift.o -o H:\\Projects\\Cyanite-Rewrite\\cyanite-engine\\cyanite\\api\\.build\\x86_64-unknown-windows-msvc\\debug\\SwiftSyntax.build\\SourcePresence.swift.o -index-store-path H:\\Projects\\Cyanite-Rewrite\\cyanite-engine\\cyanite\\api\\.build\\x86_64-unknown-windows-msvc\\debug\\index\\store -index-system-modules
1.      compnerd.org Swift version 5.6.3 (swift-5.6.3-RELEASE)
2.      Compiling with the current language version
3.      While evaluating request IRGenRequest(IR Generation for file "H:\Projects\Cyanite-Rewrite\cyanite-engine\cyanite\api\.build\checkouts\swift-syntax\Sources\SwiftSyntax\SourceLocation.swift")
 #0 0x000001b55a432310
 #1 0x00007ff61db030d9 (C:\Library\Developer\Toolchains\unknown-Asserts-development.xctoolchain\usr\bin\swiftc.exe+0x38530d9)
 #2 0x00007ff61da6f0d3 (C:\Library\Developer\Toolchains\unknown-Asserts-development.xctoolchain\usr\bin\swiftc.exe+0x37bf0d3)
 #3 0x00007ff61da2365f (C:\Library\Developer\Toolchains\unknown-Asserts-development.xctoolchain\usr\bin\swiftc.exe+0x377365f)
 #4 0x00007ff61e3ba844 (C:\Library\Developer\Toolchains\unknown-Asserts-development.xctoolchain\usr\bin\swiftc.exe+0x410a844)
 #5 0x00007ff61e3c0fe0 (C:\Library\Developer\Toolchains\unknown-Asserts-development.xctoolchain\usr\bin\swiftc.exe+0x4110fe0)
 #6 0x00007ff61a812bda (C:\Library\Developer\Toolchains\unknown-Asserts-development.xctoolchain\usr\bin\swiftc.exe+0x562bda)
 #7 0x00007ff61a75ee1b (C:\Library\Developer\Toolchains\unknown-Asserts-development.xctoolchain\usr\bin\swiftc.exe+0x4aee1b)
 #8 0x00007ff61a74cabf (C:\Library\Developer\Toolchains\unknown-Asserts-development.xctoolchain\usr\bin\swiftc.exe+0x49cabf)
 #9 0x00007ff61a756e06 (C:\Library\Developer\Toolchains\unknown-Asserts-development.xctoolchain\usr\bin\swiftc.exe+0x4a6e06)
#10 0x00007ff61a744f90 (C:\Library\Developer\Toolchains\unknown-Asserts-development.xctoolchain\usr\bin\swiftc.exe+0x494f90)
#11 0x00007ff61a750bd8 (C:\Library\Developer\Toolchains\unknown-Asserts-development.xctoolchain\usr\bin\swiftc.exe+0x4a0bd8)
#12 0x00007ff61a3766c0 (C:\Library\Developer\Toolchains\unknown-Asserts-development.xctoolchain\usr\bin\swiftc.exe+0xc66c0)
#13 0x00007ff61a37705e (C:\Library\Developer\Toolchains\unknown-Asserts-development.xctoolchain\usr\bin\swiftc.exe+0xc705e)
#14 0x00007ff61a375cbb (C:\Library\Developer\Toolchains\unknown-Asserts-development.xctoolchain\usr\bin\swiftc.exe+0xc5cbb)
#15 0x00007ff61a378097 (C:\Library\Developer\Toolchains\unknown-Asserts-development.xctoolchain\usr\bin\swiftc.exe+0xc8097)
#16 0x00007ff61a3398ec (C:\Library\Developer\Toolchains\unknown-Asserts-development.xctoolchain\usr\bin\swiftc.exe+0x898ec)
#17 0x00007ff61a33951e (C:\Library\Developer\Toolchains\unknown-Asserts-development.xctoolchain\usr\bin\swiftc.exe+0x8951e)
#18 0x00007ff61f015fbc (C:\Library\Developer\Toolchains\unknown-Asserts-development.xctoolchain\usr\bin\swiftc.exe+0x4d65fbc)
#19 0x00007ff8939d54e0 (C:\Windows\System32\KERNEL32.DLL+0x154e0)
#20 0x00007ff8948c485b (C:\Windows\SYSTEM32\ntdll.dll+0x485b)

@compnerd
Copy link
Copy Markdown
Member

I can assure you that this does currently not work on Windows. I just tried using this framework in an SPM project on Windows.

Please use a build from main. At the very least, a recent snapshot.

@broken-bytes
Copy link
Copy Markdown

broken-bytes commented Sep 13, 2022

I used the most recent version.

dependencies: [
        .package(
            url: "https://github.com/apple/swift-syntax.git", from: "0.50600.1"),
    ],

As far as I am concerned, this is the most recent snapshot

@compnerd
Copy link
Copy Markdown
Member

5.6 should be buildable - https://github.com/compnerd/swift-build/releases/tag/swift-format-5.6-RELEASE was built and would pull in swift-syntax 5.6. I wonder if the .1 has anything to do with it 🤔

@DougGregor DougGregor deleted the parser-library branch September 13, 2022 16:49
@DougGregor
Copy link
Copy Markdown
Member

@broken-bytes If you're looking to try the new parser that's the subject of this pull request, you'll need main.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

10 participants