chore(etc): add comprehensive sandboxing#10421
Conversation
Update the existing minimal service hardening with a comprehensive sandbox to minimize blast damage from service compromise. Please see the detailed code comments for an explanation of what is sandboxed. Roughly, we limit: /dev, /proc, /tmp, AF_UNIX, AF_PACKET, execution of _any_ binary other than "/usr/bin/syncthing" and "/usr/lib", uncommon syscalls plus io_uring, tons of kernel internals and more. We also enable a bunch of kernel namespaces for isolation. In short, pretty much everything is sandboxed and specifically tuned for syncthing's behavior. Sadly, we cannot use ProtectSystem=strict by default because we don't know the directories that the user will be sharing. There's a big comment block explaining how users can enable it for "extra credit". :) If the user did add the following options as the unit file recommends: - ProtectSystem=strict - ReadWritePaths=/my/shared/dir1 /my/shared/dir2 - ProtectHome=true Then the user would end up with a *far* more comprehensive sandbox than anything a container runtime (like Docker/Podman/whatever) would provide. Much (but not all) of these options could be ported to the user/syncthing.service file, BUT it would require work. Systemd does not allow all of these options to be used with the user service manager, although using PrivateUsers=true would help with most of it. I cannot justify the time investment to develop, audit and test the port to user/syncthing.service so I leave that for interested contributors. Tested on Debian Trixie (13) with the following versions: - v1.29.5, Linux (64-bit Intel/AMD) - latest HEAD (d3d3fc2 committed on Mon Oct 6 01:42:58 2025) Signed-off-by: Val Markovic <val@markovic.io>
calmh
left a comment
There was a problem hiding this comment.
Does this work everywhere Syncthing currently runs as a systemd service?
It should. Systemd only works on Linux and nothing in this PR is distro specific. It should work on any Linux distro that uses systemd. It should even work on older versions of systemd that don't support some of these options; they will be ignored (with possibly a warning printed in the journal). If systemd supports the option but the kernel does not (for whatever reason), systemd will ignore the option. Thus the sandbox is "best effort", as the systemd docs recommend it be used. |
|
Also, none of these options introduce external dependencies. These are all pure systemd+kernel features (ignored when not supported, as mentioned). |
Syncthing supports calling user-provided binaries which can be in arbitrary locations in the filesystem. Signed-off-by: Val Markovic <val@markovic.io>
|
I pushed f48a680 which removes Advice for maintainers: since this now makes it possible for an attacker to escalate access to the rest of the system by calling other binaries already present (LOTL), it would be prudent to use internal app sandboxing to limit which binaries syncthing is capable of executing. For instance, it would be a great idea for syncthing code to sandbox the syncthing process so that it can only execute allow-listed binaries like syncthing, Setting up the necessary custom mount namespacing is a non-trivial amount of work though. But there are sandboxing libraries that can help with this. Tough sell, I know. |
|
I have already been using most of these for some time, but sadly never found the time to contribute something similar, so kudos. I've added in those I wasn't using and will feedback if I have any problems. |
It should never exist, but if it does, syncthing running with a system user would implicitly get access to /nonexistent on Debian-like systems because it's the standard HOME folder for system users. Signed-off-by: Val Markovic <val@markovic.io>
Signed-off-by: Val Markovic <val@markovic.io>
Files with setuid/setgid bits are designed to be used for privilege escalation. Such files are *exceptionally* dangerous. Signed-off-by: Val Markovic <val@markovic.io>
|
I've made a small tweak to prevent a systemd warning about @ProactiveServices Any further comments? |
|
I don't really use the systemd setup much and can't vouch for or maintain this, so I'd like someone else to step up and say they've tested it, that it won't cause any problems for the users already using it on upgrade, and that they'll make sure it works for future versions. |
|
I don't understand the hate for |
|
(Apologies for the delays, I'm down with COVID. I'll respond to comments when I've recovered.) |
|
@CoelacanthusHex I added a note about AF_NETLINK, thanks! I want to thank both of you for leaving comments because it pinged me and made me realize I had, uh, forgotten about this PR. 😅 @ProactiveServices I don't see the error message we both noticed ( @calmh @ProactiveServices I'd be happy to mantain this service file going forward. I've been using syncthing for many years now and I have no intention to stop using it. I exclusively use the systemd service file on my servers. If there are any issues, an at-valloric ping on Github should reliably reach me. Meanwhile, I've been using this sandboxed setup for several months now across several devices with zero issues. I think the PR is ready to merge.
As mentioned, I've extensively tested this on my systems. This sandbox has been designed to be as widely applicable as possible. I of course cannot promise the absence of bugs or upgrading issues; people have all sorts of weird configurations. What I can commit to is fixing any reasonable1 problems people bring up after upgrading. Footnotes
|
Needed when binding the web interface to a Unix socket. Signed-off-by: Val Markovic <val@markovic.io>
| # Example drop-in file location (assuming user "syncthing"): | ||
| # /etc/systemd/system/syncthing@syncthing.service.d/override.conf | ||
| # | ||
| ## Elevated permissions to sync ownership (disabled by default), |
There was a problem hiding this comment.
Will these work and be enough with the stuff above in place? Some of it sure sounds like it would interfere with syncing ownership, like PrivateUsers.
There was a problem hiding this comment.
Good catch!
The user would have to change PrivateUsers to false as well. I have re-examined all the other properties this PR is setting and I do not see anything else that would interfere.
UMask=7027 looks at first glance that it might be an issue, but that only sets more restricted file permissions at file creation time. It's meant to prevent accidentally wider-than-intended file access.
UMask= does not restrict chmod operations at all. I'm assuming syncthing downloads the files (creating them on disk) and then uses chmod/fchmod/fchmodat/etc syscalls to change the file permissions to something else (if the user is using that feature). UMask= would not interfere with that approach.
No other properties seem relevant.
I have updated the commented-out section to clarify this.
There was a problem hiding this comment.
One more thing: RestrictSUIDSGID=true prevents setting the setuserid/setgroupid bits on any files. I'm assuming syncthing wouldn't sync that bit today as a security precaution, but if it did sync that bit today, this property would prevent it.
This would IMO be an excellent result because nobody in their right mind should be syncing files with SUID/SGID bits set. That's a security breach waiting to happen.
If syncthing would quietly sync this bit without this sandbox being enabled, I'd personally report that as a security issue.
There was a problem hiding this comment.
Syncthing doesn't set suid/sgid bits.
However on the privateuser point, I think that means it needs to default to false or we'll break users who use ownership syncing and have applied the current recommended setup as an overlay.
There was a problem hiding this comment.
Ah, I see your point.
I have removed PrivateUsers=true. I'd like to mention that PrivateUsers=true is the single strongest element of the sandbox this PR provides. Removing it meaningfully hurts the security of every user for the benefit of a handful that are using a non-standard setup.
If you ever change your mind, I'd be happy to put it back in.
Syncthing doesn't set suid/sgid bits.
(Thank you! ❤️ )
|
Just an FYI that this PR was my "this change broke my workflow" moment (https://xkcd.com/1172/). "Silently" changing the umask wasn't that awesome for me. I relied on having some files world-readable on a folder so that folder had "ignore permissions" enabled in order for syncthing to rely on the umask (instead of whatever permissions the other side uses), but with this new default you can't even stat the dir if you're not the owner. For folders where permissions are synced the umask doesn't matter anyway, but this is a rather significant change in permissions for all folders where ignore permissions is set. (systemd override obviously fixed it, just a surprising behaviour change given that this was released as a minor patch) |
|
@GermanCoding I'm sorry this change affected your workflow, but a world-readable umask for a folder containing unknown PII is not a safe default. If you need it, you should set it explicitly in a systemd unit override (as you've discovered). |
|
These changes broke the "inherit ownership" setting. To get it working again, I had to override the sandbox configuration to allow access to the Thank you for the contribution though; this looks like a good change. |
|
@calmh Thoughts on the CAP_CHOWN/CAP_FOWNER above? These Linux capabilities give very broad powers to the syncthing process that could be easily used for privilege escalation attacks. CAP_CHOWN allows for "arbitrary changes to file UIDs and GIDs" and CAP_FOWNER is even broader. I don't think a folder syncing service should have the access rights to change/ignore filesystem ownership across the entire OS. These rights cannot be scoped to a folder. I'm not sure what this "inherit ownership" setting is though, but IMO it cannot be worth this level of access. WDYT? I'm wary of breaking existing functionality. Can we "solve" this with better documentation? Maybe something like "users wishing to use 'inherit ownership' on Linux will need to add the following properties to a systemd override file while keeping in mind the security exposure". |
|
It requires what it requires. These capabilities weren't default earlier either and just need the proper docs to enable.
I don't know what you're suggesting or implying here, but I suspect you need to chill. We're not going to remove features people are using just because you don't like the required capabilities. |
|
@calmh I want to call out that a position of "zero breakage of shipped features" is not unreasonable. If that's the call syncthing maintainers wish to make, I'll carve out the necessary access.
This is fair.
This is unnecessary. :) It's not about personal distaste/opinions, it's about user security exposure. CAP_CHOWN/CAP_FOWNER provide significant opportunity for privilege escalation. |
Now I'm confused... how did this PR then break the "inherit ownership" setting? @agoeckner Could you provide more info on the breakage you saw in an issue report? Please at-tag me in your issue. |
|
Presumably because the previous additions to enable them are no longer enough given the additional sandboxing. Or they were running the service as root. |
Could you explain "previous additions" more? I'm not sure what you're referring too.
This PR changes |
That makes perfect sense! So this feature always required the user to add extra permissions. We can fix this retroactively by adding: in I'll send a PR. |
Syncthing docs in https://docs.syncthing.net/users/autostart.html#permissions tell the user to set `AmbientCapabilities=CAP_CHOWN CAP_FOWNER` if the user wishes to use the `syncOwnership` option. syncthing#10421 broke `syncOwnership` for users that followed that advice because the PR introduced `CapabilityBoundingSet=` which cancels out any additional capabilities granted with `AmbientCapabilities`. (`AmbientCapabilities` _adds_ capabilities; `CapabilityBoundingSet` _limits_ maximum capabilities to the specified set. Setting `CapabilityBoundingSet` to an empty list prevents any capabilities from being acquired in any way.) This PR fixes the breakage by explicitly setting CapabilityBoundingSet=CAP_CHOWN CAP_FOWNER and also adding `chown` to `SystemCallFilter`. This does _not_ grant any additional access rights to syncthing if the user is not explicitly setting `AmbientCapabilities` as well, but it does loosen the sandbox _a bit_. An attacker compromising the syncthing process could now more easily expand their access to include CAP_CHOWN/CAP_FOWNER even if the user is not setting `AmbientCapabilities`.
Syncthing docs in https://docs.syncthing.net/users/autostart.html#permissions tell the user to set `AmbientCapabilities=CAP_CHOWN CAP_FOWNER` if the user wishes to use the `syncOwnership` option. syncthing#10421 broke `syncOwnership` for users that followed that advice because the PR introduced `CapabilityBoundingSet=` which cancels out any additional capabilities granted with `AmbientCapabilities`. (`AmbientCapabilities` _adds_ capabilities; `CapabilityBoundingSet` _limits_ maximum capabilities to the specified set. Setting `CapabilityBoundingSet` to an empty list prevents any capabilities from being acquired in any way.) This PR fixes the breakage by explicitly setting CapabilityBoundingSet=CAP_CHOWN CAP_FOWNER and also adding `chown` to `SystemCallFilter`. This does _not_ grant any additional access rights to syncthing if the user is not explicitly setting `AmbientCapabilities` as well, but it does loosen the sandbox _a bit_. An attacker compromising the syncthing process could now more easily expand their access to include CAP_CHOWN/CAP_FOWNER even if the user is not setting `AmbientCapabilities`. Signed-off-by: Val Markovic <val@markovic.io>
|
Sent #10602 |
Syncthing docs in https://docs.syncthing.net/users/autostart.html#permissions tell the user to set `AmbientCapabilities=CAP_CHOWN CAP_FOWNER` if the user wishes to use the `syncOwnership` option. syncthing#10421 broke `syncOwnership` for users that followed that advice because the PR introduced `CapabilityBoundingSet=` which cancels out any additional capabilities granted with `AmbientCapabilities`. (`AmbientCapabilities` _adds_ capabilities; `CapabilityBoundingSet` _limits_ maximum capabilities to the specified set. Setting `CapabilityBoundingSet` to an empty list prevents any capabilities from being acquired in any way.) This PR fixes the breakage by explicitly setting CapabilityBoundingSet=CAP_CHOWN CAP_FOWNER This does _not_ grant any additional access rights to syncthing if the user is not explicitly setting `AmbientCapabilities` as well, but it does loosen the sandbox _a bit_. An attacker compromising the syncthing process could now more easily expand their access to include CAP_CHOWN/CAP_FOWNER even if the user is not setting `AmbientCapabilities`. Signed-off-by: Val Markovic <val@markovic.io>
Syncthing docs in https://docs.syncthing.net/users/autostart.html#permissions tell the user to set `AmbientCapabilities=CAP_CHOWN CAP_FOWNER` if the user wishes to use the `syncOwnership` option. #10421 broke `syncOwnership` for users that followed that advice because the PR introduced `CapabilityBoundingSet=` which cancels out any additional capabilities granted with `AmbientCapabilities`. (`AmbientCapabilities` _adds_ capabilities; `CapabilityBoundingSet` _limits_ maximum capabilities to the specified set. Setting `CapabilityBoundingSet` to an empty list prevents any capabilities from being acquired in any way.) This PR fixes the breakage by explicitly setting CapabilityBoundingSet=CAP_CHOWN CAP_FOWNER This does _not_ grant any additional access rights to syncthing if the user is not explicitly setting `AmbientCapabilities` as well, but it does loosen the sandbox _a bit_. An attacker compromising the syncthing process could now more easily expand their access to include CAP_CHOWN/CAP_FOWNER even if the user is not setting `AmbientCapabilities`. Signed-off-by: Val Markovic <val@markovic.io>
|
Hey @Valloric! The PR #10602 doesn't actually address what I reported, and it contradicts your previous work, where you had already included the The minimal change required is to add I suppose that you could also set the |
No, I wrote every word in every comment and PR. I despise LLM-speak. Please don't "jump the gun" and accuse people of LLM-speak with zero evidence.
Your report contained very little information. Please provide more information in the future so we don't have to reverse-engineer the issue we think you might have encountered. Consider filing an issue report instead of leaving comments on closed PRs.
Yes, this is what @calmh and I were guessing were the issue you encountered. CapabilityBoundingSet would have broken any override that sets just AmbientCapabilities, as the docs stated.
As the discussion in #10602 pointed out, Did you try out the fix in #10602? Please confirm if the problem persist after those changes are applied. Footnotes
|
|
Sorry, I'm just so used to people putting everything through an LLM. It's a perpetual peeve of mine! Initially, I only used the override suggested in the systemd unit comments/documentation: However, the inherit ownership feature did not work until I added the SystemCallFilter override as well: I'm not sure what's going on here. You're right that I wonder if it's being removed by this line: https://github.com/syncthing/syncthing/pull/10421/changes#diff-9da7d7beb75aa041e0bd9698a0b479b0e769baa8ea50f816dca52f77fd16096fR124 edit: Yep, that's the case. |
You might be right! Very surprising behavior from systemd here. Docs: So what does "merged" mean then? Does the order of I'll need to read some systemd source code and experiment, the docs are definitely not clear on this. @agoeckner Could you file an issue to track this? 😃 |
|
Besides the
Comment says it includes fchmod et al, but the linked code only showns chown - chmod is part of |
|
Very true, but And sure @Valloric let me create one. Regarding chmod vs chown... Let me double-check which one I actually added to my server's configuration once I get home. Perhaps I made a mistake when reporting my solution :) |
|
New issue created: #10603. I can also confirm that I did include |
IFF the user enables the `syncOwnership` feature AND sets `AmbientCapabilities=CAP_CHOWN CAP_FOWNER` as the docs in https://docs.syncthing.net/users/autostart.html#permissions state, THEN syncthing needs to use the `chown` syscall. PR syncthing#10421 added a comprehensive sandbox that breaks `syncOwnership`. In PR syncthing#10602 we fixed one part, which is expanding the default `CapabilityBoundingSet` (see the PR for details). But there's a very subtle bug that this PR fixes. PR syncthing#10421 sets the following properties: SystemCallFilter=@System-service SystemCallFilter=~@PRIVILEGED io_uring_enter io_uring_register io_uring_setup (Systemd merges `SystemCallFilter` values; we had to set the property twice because to negate syscalls, the whole list has to start with `~`.) The goal was to allow all syscalls in the `@system-service` set, BUT disallow any `@privileged` syscalls and the `io_uring*` syscalls. But the sets are not disjoint; `chown` is in both `@system-service` and in `@privileged`, so it is removed from the allow list by the second property value. This property is also parsed in a very peculiar way. From systemd docs: > If you specify both types of this option (i.e. allow-listing and > deny-listing), the first encountered will take precedence and will > dictate the default action (termination or approval of a system call). > Then the next occurrences of this option will add or delete the listed > system calls from the set of the filtered system calls, depending of its > type and the default action. (For example, if you have started with an > allow list rule for read() and write(), and right after it add a deny > list rule for write(), then write() will be removed from the set.) Not only does the order of `SystemCallFilter` properties matter (later ones can undo effects of prior ones), but the _type_ of the _first_ property sets the overall behavior of the syscall filter: if the first `SystemCallFilter` value is an allow list, then all syscalls that are not specified are disallowed by default (and reverse if the first value is a deny list). Of course, this is completely different from how other allow/deny lists are implemented in systemd; for example, `IPAddress[Allow|Deny]` properties don't work like this at all. >:( Since this complexity has bit us once, we're removing the additional deny list of syscalls and sticking with just `SystemCallFilter=@system-service`. This leaves some privileged syscalls in the allow list. Other options would require entering the "deny list by default" mode and deny lists are less secure than allow lists in general because they have to be maintained (the kernel always adds new syscalls). The rest of the sandbox (capability bounds) should be sufficient. Fixes syncthing#10603 Signed-off-by: Val Markovic <val@markovic.io>
fix(systemd): Add back chown allowed syscalls IFF the user enables the `syncOwnership` feature AND sets `AmbientCapabilities=CAP_CHOWN CAP_FOWNER` as the docs in https://docs.syncthing.net/users/autostart.html#permissions state, THEN syncthing needs to use the `chown` syscall. PR #10421 added a comprehensive sandbox that breaks `syncOwnership`. In PR #10602 we fixed one part, which is expanding the default `CapabilityBoundingSet` (see the PR for details). But there's a very subtle bug that this PR fixes. PR #10421 sets the following properties: SystemCallFilter=@System-service SystemCallFilter=~@PRIVILEGED io_uring_enter io_uring_register io_uring_setup (Systemd merges `SystemCallFilter` values; we had to set the property twice because to negate syscalls, the whole list has to start with `~`.) The goal was to allow all syscalls in the `@system-service` set, BUT disallow any `@privileged` syscalls and the `io_uring*` syscalls. But the sets are not disjoint; `chown` is in both `@system-service` and in `@privileged`, so it is removed from the allow list by the second property value. This property is also parsed in a very peculiar way. From systemd docs: > If you specify both types of this option (i.e. allow-listing and > deny-listing), the first encountered will take precedence and will > dictate the default action (termination or approval of a system call). > Then the next occurrences of this option will add or delete the listed > system calls from the set of the filtered system calls, depending of its > type and the default action. (For example, if you have started with an > allow list rule for read() and write(), and right after it add a deny > list rule for write(), then write() will be removed from the set.) Not only does the order of `SystemCallFilter` properties matter (later ones can undo effects of prior ones), but the _type_ of the _first_ property sets the overall behavior of the syscall filter: if the first `SystemCallFilter` value is an allow list, then all syscalls that are not specified are disallowed by default (and reverse if the first value is a deny list). Of course, this is completely different from how other allow/deny lists are implemented in systemd; for example, `IPAddress[Allow|Deny]` properties don't work like this at all. >:( Since this complexity has bit us once, we're removing the additional deny list of syscalls and sticking with just `SystemCallFilter=@system-service`. This leaves some privileged syscalls in the allow list. Other options would require entering the "deny list by default" mode and deny lists are less secure than allow lists in general because they have to be maintained (the kernel always adds new syscalls). The rest of the sandbox (capability bounds) should be sufficient. Fixes #10603 Signed-off-by: Val Markovic <val@markovic.io>
* main: (45 commits) chore(gui, man, authors): update docs, translations, and contributors chore(sqlite): reduce max open connections, keep them open permanently (fixes syncthing#10592) (syncthing#10596) fix(systemd): add back chown allowed syscalls (syncthing#10605) fix(systemd): support overrides for syncOwnership (syncthing#10602) chore(gui, man, authors): update docs, translations, and contributors fix(protocol): verify compressed message length before decompression (syncthing#10595) build(deps): update dependencies (syncthing#10588) chore: trigger rebuild chore(gui, man, authors): update docs, translations, and contributors chore(gui, man, authors): update docs, translations, and contributors chore(gui, man, authors): update docs, translations, and contributors chore(config, connections): use same reconnection interval for QUIC and TCP (fixes syncthing#10507) (syncthing#10573) chore: build with Go 1.26; use Go 1.25 features (syncthing#10570) chore(etc): add more comprehensive systemd sandboxing (syncthing#10421) fix(gui): remove width limit for language select items (syncthing#10531) fix(gui): show restarting modal during upgrade restart (fixes syncthing#1248) (syncthing#10566) chore(db): add ability to wait for programmatically started database maintenance, query last maintenance time (syncthing#10565) chore(gui, man, authors): update docs, translations, and contributors chore(gui): add id and name to Stay logged in checkbox for password managers (syncthing#10558) refactor: remove unused support for Azure blob stores ...
Update the existing minimal service hardening with a comprehensive sandbox to minimize blast damage from service compromise.
Please see the detailed code comments for an explanation of what is sandboxed.
Roughly, we limit:
/dev,/proc,/sys,/tmp,AF_UNIX,AF_PACKET, uncommon syscalls plus anythingio_uring1, kernel modules, clock, hostname, domain, cgroups, tons of kernel internals and more. We also enable a bunch of kernel namespaces for isolation.In short: if there's a Systemd sandboxing option, it's enabled and tuned for syncthing's behavior (to the extent supported by my knowledge of syncthing).
Sadly, we cannot use
ProtectSystem=strictby default because we don't know the directories that the user will be sharing. There's a big comment block explaining how users can enable it for "extra credit". :)If the user did add the following options in a systemd drop-in file as the service file comments recommend:
ProtectSystem=strictReadWritePaths=/my/shared/dir1 /my/shared/dir2ProtectHome=trueNoExecPaths=/ExecPaths=/usr/bin/syncthing /usr/libThen the user would end up with a far more comprehensive sandbox than anything a container runtime (like Docker/Podman/whatever) would provide.
Much (but not all) of these options could be ported to the
user/syncthing.servicefile, BUT it would require work. Systemd does not allow all of these options to be used with the user service manager, although usingPrivateUsers=truewould help with most of it.I cannot justify the time investment to develop, audit and test the port to
user/syncthing.serviceso I leave that for future PRs by interested contributors.Tested on Debian Trixie (13) with the following versions:
Documentation changes
None? This doesn't have to be user-visible, but it may make sense to tout the strong sandbox (and possible "extra credit" work users can do) if there's a desire for it. Up to maintainers.
Footnotes
It's still a large source of exploits and syncthing doesn't use those syscalls at all. If either of the two changes, the syscall filter can be easily updated. ↩