SQLCipher (Official) Swift Package Manager Integration#1827
SQLCipher (Official) Swift Package Manager Integration#1827R4N wants to merge 17 commits intogroue:developmentfrom
Conversation
… package trait is enabled - Package.swift -- Updates swift-tools-version to 6.1 to support traits -- Adds https://github.com/sqlcipher/SQLCipher.swift dependency -- Adds SQLCipher trait -- Adds SQLCipher dependency to GRDB target when SQLCipher trait is enabled -- Adds SQLCipherConfig library target to expose SQLCipher_config.h functions to swift with pass through to C variadic functions -- Adds SQLCipherConfig library depdency to GRDB target when SQLCipher trait is enabled -- Adds SQLCIPHER_HAS_CODEC swiftSettings/cSettings to GRDB target when SQLCipher trait is enabled -- Adds SQLCIPHER swiftSettings to GRDB target when SQLCipher trait is enabled -- Adds GRDBCIPHER_USE_ENCRYPTION to GRDBTests target when SQLCipher trait is enabled - Adjusts imports to check `#if SQLCIPHER` before `#if SWIFT_PACKAGE` - Adds Database+SQLCipher extension with SQLCipher operations, enabled by `#if SQLITE_HAS_CODEC` - Adds test_SPM_SQLCipher Makefile task and adds it as a dependent task of smokeTest Resolves groue#1772
… enabled (SQLCIPHER swiftSetting is set).
…bled - Removes SQLCipher related methods from Database as they are now moved to Database+SQLCipher - Adds SQLITE_DISABLE_SNAPSHOT swiftSetting to Package.swift when SQLCipher trait is enabled
…to test installing GRDB with SQLCipher trait - Adds test_install_SPM_SQLCipher Makefile task called as dependent task of test_install_SPM - Adjusts README.md SQLCipher example AppDependencies Package.swift to match GRDB platform versions
|
🥳 Thank you @R4N! I'm very happy we are converging. I'll review this great PR shortly! |
- Removes exposing SQLCipherConfig shim package publicly in Package.swift - Calls dropAllDatabaseObjects from Database.erase() method when `#if SQLITE_HAS_CODEC` (SQLCipher enabled) - Adds `--traits SQLCipher` to swift build commands in test_SPM_SQLCipher Makefile task - Fixes formatting of XCODEBUILD commands in test_install_SPM_SQLCipher Makefile task - Removes setting SQLCIPHER swiftSetting in favor of using SQLCipher trait directly in import statements - Removes unneeded `#if SQLITE_VERSION_NUMBER >= 3029000` from SQLCipherConfig shim - Adjusts SQLCipher Information Accessors example code to use try variants in README.md - Adds details/summary to cipher_logging Example output in README.md - Adds cipherVersion display to sqlcipher SPM install test project - Removes references to Database+SQLCipher from GRDB.xcodeproj (only used for SPM) - Removes duplicate reference to AppDependencies in sqlcipher.xcodeproj (SQLCipher SPM example project)
…et/sqlcipher in README.md SQLCipher encryption section
…tion This is necessary for inheriting the SQLCipher passphrase. Also, update DatabaseConfigurationTests.testPrepareDatabase() so that it accurately counts the number of prepareDatabase invocations.
I did not take the time to answer to this paragraph yet! Thank you very much! @R4N and @sjlombardo, you both have wonderfully found your way in this library. |
|
@R4N, Bad news. The test ( To reproduce, start from a clean stage (run You can also open This is a blocker 😬 [EDIT]
[EDIT] With Xcode 16.4, Indeed the demo app is linked against SQLCipher: // prints "4.11.0 community"
let version = try String.fetchOne(db, sql: "PRAGMA cipher_version")
print(version)In summary, we have two blockers:
The first blocker is not due to this pull request. It was already there in #1826, where package traits were introduced. My personal focus is now to restore this basic functionality. Everything else is secondary (this pull request, #1825 and #1708). Unfortunately, we just can not go faster than Xcode. |
|
This seems like another scenario where importing packages within the Xcode UI doesn't play nice with package traits (even the default ones specified) yet. When I try running that make task using Xcode 26.x it complained about the missing module The issue seems to be that the default trait is not be getting setup properly so the dependency condition for GRDBSQLite isn't being met and hence not added: https://github.com/R4N/GRDB.swift/blob/4683362cf87e14a09ea13a581d2be694f6d5e614/Package.swift#L88 I had previously tested using GRDB with some of my modifications to include the SQLCipher trait both with the trait enabled (using the wrapper package) and without the trait enabled (both with the wrapper package and directly import using the Xcode UI which both worked prior). This was prior to the addition of the default GRDBSQLite trait. One immediate option that I see is removing the i.e. Remove the And then always include With these adjustments in place, I was able to successfully run
This configuration will:
I hope that Xcode will improve their support for adding swift Packages with traits in the near future (I'm surprised it's not already in Xcode 26!) When I ran This is why both One approach would be to update the swift-tools-version to 6.2, although that would require Xcode 26+ for consumers whereas 6.1 requires Xcode 16.4. |
|
I don't see how linking SQLCipher (or even a custom build of SQLite) with GRDB can be done reliably considering how dyld and swift currently works. |
|
@R4N If you decide to create a fork of GRDB to experiment with this alternative structure, I would genuinely love to try it out in production right away. It would be a huge help for teams who need a working SQLCipher integration now, without waiting for Apple to ship reliable trait support. I’d be happy to contribute testing or feedback if you move forward with it. @groue If you decide to open a Feedback Assistant report or a public thread about this SwiftPM traits situation, I’ll gladly support it and upvote it. But at the same time, I have to admit — based on experience, I don’t have much faith in Apple fixing these things quickly. In 2023 I reported a serious bug related to interactive widgets in iOS 17. Apple ignored the report for months, and only partially fixed the issue a year later in iOS 18. So yeah… I’m not optimistic about waiting for Apple to solve this in the short term. That’s why I really hope we can find a practical workaround together as a community — something we can use today so development doesn’t get blocked for months by tooling instability. GRDB is an amazing library and I’d love to see it continue moving forward despite Apple’s roadblocks. |
|
@R4N The separate GRDBSQLCipher target you describe above would not bring SQLCipher to related libraries (GRDBQuery for SwiftUI helpers, GRDBSnapshotTesting for testing helpers, SQLiteData for iCloud sync, etc) unless those related libraries would make a similar dance. Your idea is clever and has benefits, but it is gaming SPM and playing against it. That's why I'm not thrilled. |
|
And I'm not mentioning that SPM downloads and builds unneeded dependencies, which means that users who do not need SQLCipher would have to download it, see SQLCipher built for no reason, see SQLCipher in their package list in Xcode, etc. So many bugs to report (some are reported already). With no real hope that they are fixed one day - in SPM, and in Xcode. I'm afraid we're back to the beginning: the only clean solution is a fork (as well as forks of related repositories). |
|
Another issue with the separate GRDBSQLCipher target is that it does not scale: Linux and Android versions need a distinct flavor of SQLCipher, which can be built from source (#1708). That would be another target, another dependency, and more SPM useless work (download, build, etc.) |
Thanks for your support, @mezhevikin. I agree that hoping for a fast response from Apple won't lead us anywhere. Until someone at Apple is able to tackle the many issues we are facing (both in will and in action), nothing will happen, and our complaints will just be received as useless whining from toxic people. I don't have anyone from Apple in my contacts, and anyway, I don't think they're organized to do what we need. I also read here and there signs of fossilization of SPM which do not make me optimistic. And waiting for 2, 3, ♾️ years is not quite ideal. One possible way forward could be:
|
|
Hello @R4N, @marcprux. I'd be happy to get your feedback on the previous message at some point. The general idea is that supporting SQLCipher is a team effort, with contributions from Zetetic for the official SQLCipher distribution, from the Swift Android Workgroup for the alternative SQLCipher distribution that builds on Linux and Android, and from your humble servant for the general integration of these new features in the existing GRDB landscape. It would be beneficial if we would all agree of some key points:
The first two points are not subject to debate. That's how GRDB is managed, by me. The third one is very disappointing, but I also do not think that it is open to debate. We tried, and saw the issues. The fourth point is more subjective, and I can imagine that some people would not care about those undesired SPM behaviors, or the presence of extra dependencies. I'm willing to stand by it, though. My quality ideals have well served the users of this library for many years, and are part of the reasons why you're here in this discussion today. So. That's how I'm currently describing our playground. If you think I am missing something, please tell! If I do not, I do not see any other solution than the one I outlined above. It's a solution that's imperfect in many ways for the GRDB/SQLCipher users, and that gives me a lot of extra work. Again, I may be missing something. In all cases, and because you all showed a lovely engagement in helping bringing GRDB to new places, I have to ask your opinion 🙏 |
I have been following these ongoing threads, but have been unable to engage much or offer any help due to time constraints. I share your dismay at the state of Swift package traits and Xcode and regret that it doesn't look like it will be a viable solution in the near future (maybe Swift 6.3 will improve things). Additionally I see how the inability to exclude the presence of a package dependency using traits also makes this an undesirable route, since I agree that vestigial dependencies have bad optics. Since you asked about alternative solutions, I was thinking more on my predecessor to #1708 and your proposal for a I don't even think we need the complexity of parameterizing the GRDB types with the protocol (like the suggestion for It might be worthwhile to take another look at #1701 and think more about how it could be made to work. IIRC, it was mostly working and had good performance, but some of the "specialty" SQLite structs like those supporting FTS5 made it tricky to abstract into a protocol. Obviously, it would be a big refactoring of GRDB, but I remember that I had a shim implementation that left global-looking functions Anyway, that's my 2 centimes. If you're interested in dusting off that PR and giving it another go, I'd be happy to help get it back into shape for testing and evaluation. I'd also like to hear from @R4N about whether such a solution might work for the SQLCipher team. |
Agreed, I was hopeful that traits would be better supported. Your thoughts are inline with the conclusions we've come to during our work to attempt to integrate via traits. Too many workarounds required to resolve bugs and no direct Xcode integration for trait selection (or using the default specified traits) are real blockers there. For number 4, we are definitely open to utilizing an official fork if that's the preferred path forward. We do have a couple of comments/questions related to the conclusions there for using a fork vs using separate products/targets:
This definitely occurs with the approach for using separate products/targets (in the core repo), but would have also occurred when using Package traits. When adding a Package, Swift Package Manager will download all defined dependencies (and transitive dependencies) to prioritze the complete availability of the dependency graph. Because of this, I would think this is the "acceptable" behavior. I could see it adding to some confusion for folks not utilizing SQLCipher though, it would be nice at some point if SPM optimized that behavior to de-prioritize downloading all dependencies and only download dependencies which are referenced in products/targets which are consumed by the consuming project.
I couldn't find any references which specify this behavior. My understanding was that the explicit dependencies drive what is built. So if in Xcode, you choose to add the standard GRDB product (when adding the GRDB Package), it will only build that (and not the SQLCipher product). Similar to what is described in this Apple documentation: https://developer.apple.com/documentation/xcode/building-your-project-with-explicit-module-dependencies For the preferred path moving forwards, if we roll with an official GRDB fork to support Official SQLCipher, would we also re-use the repo for the Swift Android Workgroup's product/target? Or a separate fork? If we re-use the same repo, would we be back in a similar situation as we are currently where we'd need to use separate products/targets in the same forked repo since traits are still pretty broken? If we use a separate fork (for each SQLCipher flavor), it seems like a lot of maintanence for you. We are happy to proceed however you see fit (fork or standalone products/targets in the core repo), but just wanted to add our feedback on your comments and bring up that the benefits of a separate fork might not constitute the extra maintanence required. |
|
I'll address your feedback with more details soon @R4N 👍 Just a quick note on two points:
I understand. I also think that the negative consequences for GRDB users that I described are real. The tolerance to "acceptable"-with-quotes is, IMHO, one of the problems we have to deal with.
Various occurrences of this behavior can be found in the comments of this forum thread https://forums.swift.org/t/swift-test-tries-to-build-all-targets-instead-of-just-those-needed-for-testing/82803. I wholeheartedly relate to this quote:
|
Agreed. When first looking at package traits, I was surprised that you couldn't restrict the dependencies of the package based on trait so that they weren't even downloaded.
Thanks for linking that forum thread over with the comments on incorrect behavior related to building random targets for no reason. That is definitely undesirable behavior!
I may be missing the difference, but wouldn't there have been a similar situation if traits worked as expected? i.e. Packages which rely on GRDB as a dependency would need to specify the "SQLCipher" trait in their Package definition to bring in the SQLCipher dependency. With the separate GRDBSQLCipher product/target instead they would need to specify the GRDBSQLCipher product. And then with the forked approach, would the other related libraries need to maintain their own forks which use the forked url for the dependency? So for example if traits worked, GRDBSnapshotTesting would need to specify the trait here (or with the forked approach specify the forked url): Whereas with the separate product/target they would need modify the product name wherever used as a dependency: This approach looks versatile and should work to integrate with the Official SQLCipher SPM as well. The feedback I have on it is similar to @groue's in the initial PR:
I think your comment here: #1701 (comment) to keep the default GRDB target/product as the "-inline" variant (i.e. SQLite included) might resolve 2 + 3. Our suggestion is to take a two-pronged approach: Short Term SolutionSeparate Product/Target solution proposed earlier in this thread or official separate fork Long Term SolutionRe-work GRDB according to @marcprux's suggestion to support BYO SQLite Taking this approach fills the gap caused by the deprecation of CocoaPods while allowing flexibility implementing the larger re-work of GRDB without pressure/time constraints. |
Yes, I definitely think there should only be a single target, and that GRDB should implicitly use its own system SQLite implementation. To sketch out what I'm envisioning, it would look something like:
Definitely! I got it largely done in
Yes. So you (SQLCipher) would need both the base graph TD
App --> GRDB
App --> SQLCipher
GRDB --> SQLInterface
SQLCipher --> SQLInterface
%% Define packages in a style for clarity
classDef package fill:stroke:#333,stroke-width:2px;
class App,GRDB,SQLCipher,SQLInterface package;
%% Define a class for the top-level app
classDef app fill:#ccf,stroke:#333,stroke-width:2px;
class App app;
That might be over-engineering things, though 😉 |
|
Hey @marcprux 👋 I really like your proposed solution — it looks clean and makes a lot of sense! Something like this: import GRDB
import GRDBSQLCipher
var configuration = Configuration()
configuration.implementation = GRDBSQLCipher()
configuration.prepareDatabase { db in
try db.usePassphrase("secret")
}
let queue = try DatabaseQueue(
path: "database.sqlite",
configuration: configuration
)P.S. I don’t know GRDB’s architecture deeply, so apologies if this doesn’t make sense 🙂 |
Yes, that probably would be ideal. TBH, my proposal was trying to optimize for ease-of-implementation and not having to touch every part of the GRDB codebase. When there is a single global |
|
(Thank you all for looking for a solution - I will be slow to answer again) |
|
So I gave a spin at @marcprux's ideas (#1827 (comment)), where all SQLite APIs are wrapped in a protocol. Last time I checked, GRDB is close to the speed of light in optimized scenarios (i.e. the maximum possible speed, which is the speed of SQLite itself). This is an important design goal of the library — no one should ever feel hindered for choosing GRDB, even people who need to work with large amounts of data. Those optimized scenarios are very well integrated with other GRDB features, the ORM and database observation features: the optimisation is not forcing the user to call raw C SQLite functions, it is giving GRDB enough information for going straight to the point. (To run the relevant performance tests, open Tests/Performance/GRDBPerformance/GRDBPerformance.xcodeproj, focus on FetchPositionalValuesTests and FetchRecordOptimizedTests in the GRDBOSXPerformanceTests target). Accessing SQLite APIs through a global existential ( I thus tried to see what it would take to make GRDB generic. Unlike existentials, generics are well optimized by the compiler, through specialization and inlining. Instead of an existential global, the particular I spend a few hours playing the generics score. Ideally, importing This ideal picture faces a few hiccups and breaking changes, though. In particular:
None of those problems are unsolvable, and we can probably reduce the breaking changes to a minimum set of rarely used apis, by defining a lot of facades. There are other negative consequences:
This is not a surprise: with generics, we are making GRDB able to open connections to multiple SQLite interfaces at the same time. It's time to remember that this is not, precisely, the feature people are asking for. The feature people are asking for is the ability to connect to SQLite, or SQLCipher, but not both. This discards the generics solution. To conclude, thank you very much Marc for your idea. It was quite worth exploring. But time has passed since our early discussions, I gained experience, and I'm much more reluctant now to degrade the library because of our lacking tooling. My current position has not changed: I do not see any other viable solution than a fork (or helping people forking). |
|
Thanks for posting the results of your investigations into Marc's proposed solution. Understandable that you would like to avoid degradation because of lacking tooling.
We'd be happy to help contribute to this effort. Revisiting your previous comment, you had proposed a few bullet points on a possible way forwards:
As you mentioned, we could use a nice chunk of the changes in this PR and modify the Package.swift to no longer use traits and instead just have the GRDB target/product have a direct dependency on SQLCipher.swift.
It would be preferable to setup an official fork that can be assured to be updated regularly, maintained, and use best practices.
My initial inclination is to adjust the product name to GRDBSQLCipher and leave the target name as is (GRDB). This would allow consumers to switch from GRDB without encryption to GRDB with SQLCipher without having to change around any existing import statements while being explicit that GRDBSQLCipher is the included product (when selecting the Package product from the Xcode interface for example).
Having you host it under your username makes sense to us as well. You have a more intimate familiarity with the inner workings and feature developments as GRDB evolves. We're happy to assist with maintenance, contributions, or issues that arise. If you'd prefer for us to host it under SQLCipher, we would also be ok with that. |
|
Hi @groue, thanks again for the work on this PR. I wanted to clarify our current situation and ask specifically about the SPM roadmap. We were previously using GRDB via Swift Package Manager (v7.6.1), but since SPM does not currently support SQLCipher, we had to switch package managers when encryption became a requirement. At the moment, we’re using CocoaPods with the unreleased GRDB 7 branch to get SQLCipher support. However, since CocoaPods support has now been discontinued and no new versions will be published, this puts us in a difficult position: SPM is our preferred (and long-term) dependency manager SQLCipher is a hard requirement for us Using an unreleased branch indefinitely is not ideal for production Could you share whether there is an expected plan or rough timeline for an official GRDB + SQLCipher solution via Swift Package Manager? Thanks again for GRDB, it’s been excellent to work with overall, and we really appreciate the transparency around supported configurations. |
|
Hello @mansi-prajapati-simformsolutions
There is none. Apple tooling makes this impossible. GRDB + SQLCipher with SPM is theoretically possible with package traits. Indeed package traits can have GRDB link against the system SQLite, OR one SQLCipher package (the official one, or an alternative one for Linux, for example), OR another SQLite library. All ORs must be exclusive, or linker errors are lurking. Unfortunately, Xcode does not support package traits:
The last two defects are blockers. Many GRDB users use Xcode. I won't ship a package that is not correctly supported. So far, your only solution to avoid CocoaPods is to:
|
|
I've created a separate pull request which includes some of the changes in this pull request to upstream these modifications: This will allow forkers to enable SQLCipher and limit the diff to adjustments to I've got work prepped on my fork on a separate branch which uses GRDB+SQLCipher : https://github.com/R4N/GRDB.swift/tree/sqlcipher-package We're happy to host the repo under SQLCipher org's github if that's preferable, where we could add you as a collaborator. Alternatively, we're ok if you'd prefer to take ownership. Benefits of this approach:
|
|
Thanks @R4N! I'm having a look at the PR shortly :-)
It's really appreciated that you take some time testing it. It will greatly help me reviewing the PR.
The first step is to recommended the "fork and adjust" technique. It requires documentation. A sample repo can surely help. There's no real hosting question at that step. The second step would be "GRDB+SQLCipher is officially supported at [some url]". This does not invalidate the first step, because some users still need other SQLCipher variants (Linux, Android). So let's reach the first step to begin with.
I'm happy that we are clear on one point: traits that select an SQLite variant will not enter the main GRDB repo until several years. They will when GRDB raises the minimum version of Xcode to a version that supports traits - so far, none does. I do not hold my breath :-)
True. We could also say that since it's a fork, it could drop all backward compatibility guarantees, use traits, and force the wrapper package technique onto the Xcode users. But, yeah, it takes courage™ to tell Xcode users "Sorry guys we made it difficult for you" ;-) |
|
Thanks for the comments.
This is what I've done so far:
Let me know if you'd like me to run some more extensive testing of any kind.
I've created a PoC branch for forking w/ BYO SQLCipher which is a working example of using the amalgamation in place of the "official" fork. I've also created some preliminary documentation for forkers in that same branch If this documentation looks good to you, I could integrate them into the existing documentation somewhere (let me know where you think is best) as an add on for my existing upstream pull request or a separate pull request.
Understood. One other option could be to have folks fork from the official GRDB+SQLCipher repo (after it's established). The benefit of that is the GRDBSQLCipher shim would already be setup, and a couple of the swiftSettings/cSettings would already be established. They would just need to:
I don't think there's enough benefit to using the traits if we're using the dedicated fork that only supports SQLCipher. Not having to use the wrapper package is pretty nice for consumers. Both of these forking options are predicated on the upstream pull request to support the necessary conditional SQLCipher imports, reducing the diff with the core repo, and simplifying auditing. |
|
I wonder if you could check whether this forking mechanism will also work with a pure source build of SQLCipher, like https://github.com/skiptools/swift-sqlcipher? That package does provide a "SQLCipher" product, so in theory it might "just work"… |
|
Yes, I think generally it will work with subbing in It's imported in the upstream PR here: So I had remove that import to get swift-sqlcipher to work. Aside from that, just adding swift-sqlcipher as a dependency: Then setting it as a dependency for the GRDB target: And adjusting the swiftSettings/cSettings as outlined in that preliminary doc I've posted worked to run for me. You probably don't need all the cSettings I posted in your case as some of them are applied to your package, This minimal set allowed it to run and produce the correct I was trying to reduce the import variations between upstream and the forks, but perhaps having the "official" fork have that diff isn't too bad. That would involve:
That might be the preferable approach, but I've gotta think about it a little more. |
|
To follow up on my previous comment, I've
These adjustments will:
That last one is a minor compromise, but I think the flexibility/versatility of not including the GRDBSQLCipher import in the upstream pull request to allow forks to determine how they want to handle it outweighs the compromise. @marcprux This adjustment should allow a drop-in replacement for
|
|
Closing in favor of #1845 |
Adds SQLCipher.swift Swift Package Manager integration when SQLCipher package trait is enabled
Changes
#if SQLITE_HAS_CODECResolves #1772
Pull Request Checklist
developmentbranch.make smokeTestterminal command runs without failure.Ongoing Support
The SQLCipher Team is committed to updating and supporting the official
SQLCipher.swiftSwift Package. We're happy to assist with any GitHub issues related to integration or troubleshooting usingGRDB.swiftwithSQLCipher.swiftpackage dependency. Please feel free to raise an issue in the Official SQLCipher.swift repo