At rest encryption: TLS keys and wrappers around the etcd wal/snap packages#1598
At rest encryption: TLS keys and wrappers around the etcd wal/snap packages#1598aaronlehmann merged 3 commits intomoby:masterfrom
Conversation
cb41245 to
53e0dfa
Compare
Current coverage is 54.82% (diff: 72.08%)@@ master #1598 diff @@
==========================================
Files 97 101 +4
Lines 15481 16475 +994
Methods 0 0
Messages 0 0
Branches 0 0
==========================================
+ Hits 8539 9033 +494
- Misses 5808 6301 +493
- Partials 1134 1141 +7
|
|
Looks good so far. My preference would be to wait for the whole encryption feature to be ready before merging this, if that works for you. It will let people review it in context and make sure it fits the model. I think the biggest unresolved question is how to distinguish encrypted and unencrypted WALs/snapshots. For a clean slate approach, I think the best way would be to put encryption metadata this in the protobuf we store at the beginning of each WAL, and the protobuf we serialize into each snapshot file. However, it's not obvious that there's a good backward-compatible way to do this. For example, right now we serialize a So maybe some other mechanism, like file naming, is better. I guess we should start by asking what exactly information we want to store.
|
|
Is there a way to simply encrypt the stream, rather than having the encryption work at the message-level? This approach intertwines the encryption format with the message structure, which is a level of coupling that isn't really necessary. We can see this in the fact that term and index aren't actually encrypted. Ideally, we should be able to hand in an |
@aaronlehmann Sure, no worries - was just trying to keep the PRs smaller. :)
I was originally thinking the encryption algorithm, the hash if we're not going to do GCM and hence need to provide our own digest. We could possibly the ID of the key, if we store the encryption key as a secret. Possibly the IV or nonce, if one is used.
@stevvooe I agree it's not great to tie things to the The WAL records and snapshots are serialized inside the Another approach @NathanMcCauley suggested is that we can encrypt above the raft layer. So we can do that at the We can pass a This also means that the snapshots and raft logs will also be sent encrypted (beyond just TLS) between peers, which isn't a feature we need but not necessarily a bad thing. Each node would have to be bootstrapped with the encryption key before being able read the The problem is that new encryption key distribution to all the nodes using the Possibly this can be dealt with using the 3 key network rotation thing, and just let users rotate key encryption keys. Or maybe we can use the manager TLS keys to encrypt the I think that wrapping the |
|
I don't really like the idea of encrypting raft log entries before they are distributed.
Given the structure of the wal package, I think wrapping it like this is the best approach, as discussed previously. Otherwise we would have to maintain a fork.
|
|
|
||
| func (m *meowCoder) Decode(index, term uint64, orig []byte) ([]byte, error) { | ||
| if !bytes.HasSuffix(orig, []byte("🐱")) { | ||
| return nil, fmt.Errorf("not meowcoded") |
@cyli one possibility is an out-of-raft key distribution mechanism, such as an API that when called with a manager certificate returns the current key. I still agree that the best way forward is the current path (wrapper around the WAL), but the best solution is we encrypt above raft. |
|
Other random thoughts that may affect the decision:
|
|
I agree with @diogomonica that encrypting Raft entries is the way to go. I assumed that's how it would work from the start. This has a few advantages, including e.g. that you don't really care if it's Raft anymore. Do you actually need key rotation? If so, why? I can think of a few reasons, which all have different answers. If your primitive won't let you encrypt large amounts of data, derive a new key, or pick a better primitive, or both. In the case of compromise, you should be nuking the secret store from orbit anyway and rotate all of the secrets stored inside it. (Plus, that's ideally a rare case, so I'm not sure if I see the value for optimizing it.) If you're worried about algorithmic agility: a) pick a strong primitive and don't mess it up (I'm volunteering to help with that) b) that should also be a rare event, so just moving all the data over should still be an acceptable solution. I'm not saying that key rotation is intrinsically bad or that you can't have it; just that it doesn't come free, and it's not obvious to me why you'd want it. New nodes already need some kind of secret to be able to join the cluster (the join token), and once you get a 256 bit secret you should be set for the lifetime of the cluster. You might not be able to bootstrap the secret from he join token (those apparently are easy to regenerate and the key should be constant for a cluster), but whatever the thing is that issues certificates could just tell you about the key as well. |
Not having to re-bootstrap the entire swarm cluster and thus losing all stored data (and rotating out the individual secrets for instance) would be good. But as you and @diogomonica mention there are other solutions for the key distribution. |
|
@cyli I'm not sure I follow. Wouldn't you be able to do that regardless just by writing new entries into the Raft log? That doesn't seem like it'd require changing the encryption key for that cluster. |
|
@lvh Er, sorry, I mean if the encryption key is ever compromised, not having to have to create an entirely new swarm cluster would be good (rotate encryption key and rotate your secrets individually) |
6fac38a to
767c8f7
Compare
|
Might be ready for another look. This does the encryption part of the wrapping, and handles forcing a new snapshot if the key is rotated. Couple of caveats:
This also may also be a little tricky because order of raft operations has to be right (I tested as best I could - I've probably missed something obvious). Going to also try out the above-raft implementation too to see if it's simpler, especially since it turns out this also has some key-bootstrap issues, and we can see which one we like. |
|
I'll defer to you and Diogo on KDF vs. raw key. I'm not yet convinced that a shared key is a good idea. Storing the key in raft causes a lot of problems. I think it would be better to let each node use its own key. If the admin wants to use the same key everywhere, they're free to do that - or they can generate a separate key for every node, which potentially improves security. Automatically distributing a shared key definitely worries me. I also get concerned when I see discussion of possibilities like "require that joining a cluster also takes a lock key" and "refuse to write any WALs to disk unless there is a non-empty encryption key". I think it's really important that this feature not cause extra hurdles for normal users. I expect very, very few people will make meaningful use of the encryption-at-rest feature, since it will be a big operational hassle to unlock the encrypted files after a restart. I'm okay with adding the feature as long as it's opt-in and doesn't cause problems for normal users. But I see this moving in the direction of becoming more tightly integrated and user-impacting and TBH I don't really like that. |
api/specs.proto
Outdated
| // TaskDefaults specifies the default values to use for task creation. | ||
| TaskDefaults task_defaults = 7 [(gogoproto.nullable) = false]; | ||
|
|
||
| // LockKey is the encryption key for the raft data |
There was a problem hiding this comment.
For now, as I understand, this is limited to raft. Do we have plans on opening this up to "at rest" data? Would it be more accurate to say this?
There was a problem hiding this comment.
Also, do we need to store the new and old key here in case the migration is interrupted?
There was a problem hiding this comment.
For now, as I understand, this is limited to raft. Do we have plans on opening this up to "at rest" data? Would it be more accurate to say this?
Sorry, yes, this is intended for everything at rest - I just hadn't implemented that part yet, and though I was going to do that in another PR, so I was intended on updating it then when the feature was in. After some discussion with @aaronlehmann this is PR is going to implement everything nuts to bolts, so I'll update the comment. :)
Also, do we need to store the new and old key here in case the migration is interrupted?
I'm not sure the old key inside the ClusterSpec itself is necessary, since we don't need to be able to read old data except on starting up. And on starting up, we won't be able to read data in here anyway. :) We just need the new key in order to write new data.
I think we are going to not use a shared key though, so I guess it won't be distributed via raft anymore. If we have a data encryption key + key encryption key, then yes if the data encryption key is rotated we'll have to keep the old one around.
manager/state/raft/raft.go
Outdated
| func (l *lockKeyRotator) MaybeUpdateKey(newKey []byte) { | ||
| l.mu.Lock() | ||
| defer l.mu.Unlock() | ||
| if !bytes.Equal(l.currentKey, newKey) { |
There was a problem hiding this comment.
What happens if the key is updated while already in the midst of a rotation?
There was a problem hiding this comment.
The rotation should happen as a callback inside the DoRotation function, which grabs a write lock. I think this should prevent the key from actually being updated whilst in the middle of a rotation? See https://github.com/docker/swarmkit/pull/1598/files#diff-fcccb5a38e5148a94f6ecfa031233c31R261. I didn't lock the entire doSnapshot call, because it only matters when we actually save a snapshot to disk.
So what will happen if you rotate and rotate again is (at least this is what I was going for...):
MaybeUpdateKeygrabs a lock. Currently no other rotation happening, so it updates the key.- The logic that calls
MaybeUpdateKeysees if there's a snapshot in progress. Let's say there isn't for now, so it callsdoSnapshot - Before
doSnapshotactually saves a snapshot to disk, it checks if it needs to rote, and if so, actually does rotate out the keys by changing the encoder in side the wrapped snapshotter and WAL writer. It grabs a lock while doing so. saveSnapshotnow starts writing stuff to disk.- Let's say another cluster update comes in.
MaybeUpdateKeyis triggered again. If it happens in the middle of the encoder rotation being done insaveSnapshot, it will wait until that's done. Then it updates the key. - Because a snapshot is already in progress, nothing happens - no new snapshot is saved. The key has been rotated, but it's not being used because the encoders haven't been rotated.
- When the snapshot is done, the
snapshotInProgresschannel is written to, then set to nil. Then it checks to see if a rotation needs to be done. If so, it triggers anotherdoSnapshot. - Which calls
saveSnapshot, which will rotate the key as necessary, etc.
So if you rotate a key over and over, snapshots will be written over and over to disk, but the cached keys in keyRotator can change independently of the actual key used for encryption. The keyRotator is more of an indication of whether the key currently used for encryption needs to change.
| } | ||
|
|
||
| // the wal/snap directories in decreasing order of preference/version | ||
| var versionedWalSnapDirs = []walSnapDirs{ |
| return err | ||
| } | ||
|
|
||
| if err := e.snapshotter.SaveSnap(snapshot); err != nil { |
There was a problem hiding this comment.
Does this need to hold the lock and block key rotation while the snapshot is actually saved? Could we do something like:
snapshotter := e.snapshotter
e.encoderMu.RUnlock()
if err := snapshotter.SaveSnap(snapshot); err != nil {| // ReadRepairWAL opens a WAL for reading, and attempts to read it. If we can't read it, attempts to repair | ||
| // and read again. | ||
| func ReadRepairWAL(ctx context.Context, walDir string, walsnap walpb.Snapshot, factory WALFactory) ( | ||
| WAL, WALData, error) { |
There was a problem hiding this comment.
Don't break the line here
84f0f79 to
7815619
Compare
node/node.go
Outdated
| close(n.certificateRequested) | ||
| if securityConfig == nil { | ||
| switch { | ||
| case n.config.JoinAddr == "": |
There was a problem hiding this comment.
nit: I'd format this as an if/else
ca/keyreadwriter.go
Outdated
| err := os.MkdirAll(filepath.Dir(k.paths.Key), 0755) | ||
| if err != nil { | ||
| // current assumption is that the cert and key will be in the same directory - we | ||
| // create a temporary directory |
There was a problem hiding this comment.
This doesn't look like a temporary directory. Is the comment wrong, or is the path wrong?
ca/server_test.go
Outdated
| cluster = store.GetCluster(tx, cluster.ID) | ||
| }) | ||
|
|
||
| time.Sleep(250 * time.Millisecond) // server needs to get update event |
There was a problem hiding this comment.
Can we poll for the correct key instead? (sorry)
e3394c7 to
bca338c
Compare
|
LGTM |
18c71ce to
c8f7d60
Compare
1. There is a single object that knows how to read/write TLS keys and certs. This object knows how
to encrypt/decrypt the TLS key, and preserve any headers stored on the key as specified by a
PEMKeyHeaders interface.
2. No longer write the Root CA key to disk - it is stored inside raft only.
Refactor the node TLS loading/bootstrapping as well to live in a single function.
Signed-off-by: cyli <ying.li@docker.com>
…y so that so that TLS certificates are never written to disk after they are downloaded from the swarm CA. Also, update the cluster spec and object so that a user can enable and disable autolocking managers. If autolocking is enabled, a shared key will be used by all the managers to encrypt TLS keys and raft DEKs. Also, provide the control API and swarmd/swarmct commands to enable/disable autolocking, and rotate lock keys. Signed-off-by: cyli <ying.li@docker.com>
… snapshot into a storage package that knows how to do encryption and rotate encryption keys. Handle raft DEK encryption rotation in the raft node and in the manager. Ensure that when a node starts up, it uses a PEMKeyHeader interface that knows how to handle raft DEKs. Signed-off-by: cyli <ying.li@docker.com>
c8f7d60 to
31b94a7
Compare
An implementation of a wrapper that would implement the on-disk snapshot encryption and on-disk WAL encryption as described here: https://docs.google.com/document/d/1YxMH2oIv-mtRcs1djRkm0ndLfiteo0UzBUFKYhLKYkQ/edit?usp=sharing.
This is going to rely on
raft.goorstorage.goto tell it whether something is encrypted, and with which algorithm. For WALs, such information can be included in the metadata of the file, but no such metadata exists for snapshots. Another option is to base this on the snapshot and WAL directory locations/names, or on a metadata file that lives alongside the raft store.This wraps every data object in an
EncryptedRecordobject which specifies the nonce, the algorithm, etc.encryption key KDF using lock key and node ID<-- we can just take the full key for now, this can be in a separate PR