Skip to content

Add InProc Option for Sourcekitd#728

Merged
jpsim merged 5 commits intojpsim:mainfrom
juozasvalancius:feature/inproc_sourcekitd
Mar 15, 2022
Merged

Add InProc Option for Sourcekitd#728
jpsim merged 5 commits intojpsim:mainfrom
juozasvalancius:feature/inproc_sourcekitd

Conversation

@juozasvalancius
Copy link
Copy Markdown
Contributor

There are two versions of sourcekitd that come with Xcode:

  • sourcekitd.framework, which uses XPC to connect to the sourcekit daemon and parse swift files out-of-process.
  • sourcekitdInProc.framework, which does the parsing in-process.

SourceKitten uses the XPC version on macOS. However, there may be cases, when the in-process version might be preferred. So this PR adds the possibility to configure this using the following environment variable:

USE_INPROC_SOURCEKIT=YES

Specifically, this will be needed to support running SourceKitten in Swift Package Manager build tool plugins. It is a new feature (SE-0303) in Swift 5.6, that allows Swift packages to execute custom build commands when building. So for example, you could have swiftlint run during a build of your Swift package.

As of this writing, Swift 5.6 is available in the Release Candidate of Xcode 13.3. I have experimented with it and found that SPM build commands run in a sandbox that prevent the use of XPC and crash when using the regular version of sourcekitd, but it works normally with sourcekitdInProc.

@johnfairh
Copy link
Copy Markdown
Collaborator

Do you think it's intentional that the XPC capability is disabled in the sandbox? Doesn't seem to be mentioned.

Can we tell we're in an SPM-plugin / sandboxed environment and automatically use the in-proc version? Something like SWIFT_PACKAGE but in the environment?

Should we just always use the in-proc version if it's available? I guess that would be a bold move...

Need to document the setting in the README.

@juozasvalancius
Copy link
Copy Markdown
Contributor Author

When you build a Swift package that uses SPM plugins, you can find the helper script that Xcode uses to execute custom build commands in the build log. The script uses sandbox-exec with some rules that specify which directories the command may write files to. It doesn't explicitly specify any XPC rules, so I guess that's just a side effect of using a sandbox.

I looked at detecting if we're in an SPM plugin, the only thing that I found was this environment variable set by Xcode:

export GCC_PREPROCESSOR_DEFINITIONS\=\ SWIFT_PACKAGE\ DEBUG\=1

So it would be possible to parse it at runtime, and if SWIFT_PACKAGE is part of this environment variable, then we could assume that we're running in an SPM plugin. Though I'm not sure if it is a good solution.

We could also try testing if we have write access to some directories to test if we're in a sandbox, but that also feels like a workaround.

The solution in this PR isn't ideal either, because setting environment variables in a .buildCommand in an SPM build tool plugin doesn't seem to work in the current Release Candidate of Xcode. But it can be worked around, for example, swiftlint could have a command line argument, and if that is set, then it could set the environment variable at runtime, before calling SourceKitten.

Regarding always using the in-proc version, that might be not great for people who use SourceKittenFramework in a GUI app, because then if sourcekit crashes, so will the GUI app. So there are advantages in using the XPC version.

So, which solution is the best? I feel that using an environment variable to give the users a choice is the most universal way.

@jpsim
Copy link
Copy Markdown
Owner

jpsim commented Mar 13, 2022

An environment variable for this is fine, but I think we’d additionally want a programmatic option for use in SwiftLint.

@juozasvalancius juozasvalancius force-pushed the feature/inproc_sourcekitd branch from fd2e3c1 to 3c27f6c Compare March 13, 2022 22:08
@juozasvalancius
Copy link
Copy Markdown
Contributor Author

Alright, I've added the description of the new environment variable in the README.

For a programmatic option, we could make this a public var. But it would only have an effect if it is set before SourceKit is used.

-let useInProcSourceKit = envBool("USE_INPROC_SOURCEKIT")
+public var useInProcSourceKit = envBool("USE_INPROC_SOURCEKIT")

@jpsim
Copy link
Copy Markdown
Owner

jpsim commented Mar 15, 2022

Thanks @juozasvalancius! I made a few small changes on your branch, I hope you don't mind:

  • Moved useInProcSourceKit from a global to a public static mutable var in a SourceKittenConfiguration namespace. In a public API, I prefer not exposing global variables or free functions. This provides the programmatic interface I had asked for earlier.
  • Renamed the env var from USE_INPROC_SOURCEKIT to IN_PROCESS_SOURCEKIT, which I feel is a bit more descriptive without being any longer

I'll be merging this once CI passes.

Thanks for working on this, and please keep me up to date as you make progress on your SwiftPM build tool integration!

@jpsim jpsim merged commit cbb4b38 into jpsim:main Mar 15, 2022
@jpsim
Copy link
Copy Markdown
Owner

jpsim commented Mar 15, 2022

I tested with SwiftLint to see if there was any performance impact and it looks like the answer is no.

$ hyperfine \
  "IN_PROCESS_SOURCEKIT=1 swiftlint --quiet --no-cache" \
  "IN_PROCESS_SOURCEKIT=0 swiftlint --quiet --no-cache"
Benchmark 1: IN_PROCESS_SOURCEKIT=1 swiftlint --quiet --no-cache
  Time (mean ± σ):      1.201 s ±  0.032 s    [User: 5.793 s, System: 0.281 s]
  Range (min … max):    1.146 s …  1.252 s    10 runs
 
Benchmark 2: IN_PROCESS_SOURCEKIT=0 swiftlint --quiet --no-cache
  Time (mean ± σ):      1.181 s ±  0.009 s    [User: 5.678 s, System: 0.220 s]
  Range (min … max):    1.167 s …  1.199 s    10 runs
 
Summary
  'IN_PROCESS_SOURCEKIT=0 swiftlint --quiet --no-cache' ran
    1.02 ± 0.03 times faster than 'IN_PROCESS_SOURCEKIT=1 swiftlint --quiet --no-cache'

I confirmed via htop that a SourceKitService process is running when IN_PROCESS_SOURCEKIT=0 but not when IN_PROCESS_SOURCEKIT=1 so this seems to be working as intended.

jpsim added a commit to jpsim/homebrew-core that referenced this pull request Mar 15, 2022
As of jpsim/SourceKitten#728 there's now a way to run
SourceKitten while avoiding XPC communication with SourceKit.

This should prevent the sandbox violations we were seeing before.
BrewTestBot pushed a commit to Homebrew/homebrew-core that referenced this pull request Mar 16, 2022
* sourcekitten 0.32.0
* Requires Xcode 12.0 to build
* Improve SourceKitten test
  As of jpsim/SourceKitten#728 there's now a way to run
  SourceKitten while avoiding XPC communication with SourceKit.

  This should prevent the sandbox violations we were seeing before.
* Fix style
* Use ENV instead
* Try with Xcode 13+
* Only run syntax test when Xcode 13 is selected
* Use MacOs::Xcode.version

Closes #96932.

Co-authored-by: Carlo Cabrera <30379873+carlocab@users.noreply.github.com>
Signed-off-by: Carlo Cabrera <30379873+carlocab@users.noreply.github.com>
Signed-off-by: BrewTestBot <1589480+BrewTestBot@users.noreply.github.com>
@juozasvalancius
Copy link
Copy Markdown
Contributor Author

@jpsim thanks for the improvements! And I'll be sure to update you if I have more code to share about SPM plugins.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants