Skip to content

[encryption] Manage file key by the file to encrypt itself#16

Merged
acelyc111 merged 9 commits intopegasus-kv:v8.3.2-pegasus-encryptfrom
acelyc111:pk_enc_new
Sep 15, 2023
Merged

[encryption] Manage file key by the file to encrypt itself#16
acelyc111 merged 9 commits intopegasus-kv:v8.3.2-pegasus-encryptfrom
acelyc111:pk_enc_new

Conversation

@acelyc111
Copy link
Copy Markdown
Member

@acelyc111 acelyc111 commented Aug 16, 2023

apache/incubator-pegasus#1575

After all encryption related patches been cherry-picked from
tikv and
merged, now we will improve the encrytion, including:

  • Fix action job build-linux-encrypted_env-no_compression-no_openssl
    to build binaries without openssl and compression libs correctly.
  • Fix action job build-linux-encrypted_env-openssl to export the
    ENCRYPTED_ENV enviroment variable correctly.
  • Don not skip tests which are skipped by TiKV.
  • Refactor AESCTRCipherStream and AESEncryptionProvider to support
    manage file key by the file itself, according to the design docs in
    Data at rest encryption.
  • Remove all KeyManager related codes.
  • Replace KeyManager tests by AES encryption tests.
  • Refactor encryption/encryption_test.cc and add more tests.
  • Make it possible to construct AESEncryptionProvider object via
    EncryptionProvider::CreateFromString() by registering a
    factory in "encryption" library.
    It's possible to construct an object by URI: AES, AES://test or
    AES:<instance_key>,<EncryptionMethod>.
  • ldb tool support to parse --fs_uri flags as the URI mentioned above.
  • Add tests to create AESEncryptionProvider object in
    CreateEncryptedEnvTest.CreateEncryptedFileSystem
  • db_bench support to run benchmark with encryption enabled, by adding
    new flags for db_bench, they are encryption_method and encryption_instance_key.
  • Move code from the exported header directory (i.e. include/rocksdb/encryption.h)
    to rocksdb internal (i.e. encryption/encryption.h), do not expose them to users.
  • Code format.

Review hint: #17 shows all the code changes
from the base branch (i.e. pegasus-kv:v8.3.2-pegasus), you can review it together to
make sure the request branch acelyc111:pk_enc_new doesn't have vice effect on the base.

Manual test:

// Generate some data.
./db_bench --encryption_method=AES128CTR --encryption_instance_key=test_instance_key  --num=10000

// Dump WAL OK
./tools/ldb --fs_uri="provider=AES; id=EncryptedFileSystem" dump_wal --walfile=/tmp/rocksdbtest-1000/dbbench/000004.log
./tools/ldb --fs_uri="provider=AES://test; id=EncryptedFileSystem" dump_wal --walfile=/tmp/rocksdbtest-1000/dbbench/000004.log
./tools/ldb --fs_uri="provider=AES:test_instance_key,AES128CTR; id=EncryptedFileSystem" dump_wal --walfile=/tmp/rocksdbtest-1000/dbbench/000004.log

// Dump WAL failed. Pass bad provider parameters to --fs_uri, e.g.
./tools/ldb --fs_uri="provider=AES1:test_instance_key,1AES128CTR; id=EncryptedFileSystem" dump_wal --walfile=/tmp/rocksdbtest-1000/dbbench/000004.log
./tools/ldb --fs_uri="provider=AES:bad_test_instance_key,AES128CTR; id=EncryptedFileSystem" dump_wal --walfile=/tmp/rocksdbtest-1000/dbbench/000004.log
./tools/ldb --fs_uri="provider=AES:test_instance_key,AES192CTR; id=EncryptedFileSystem" dump_wal --walfile=/tmp/rocksdbtest-1000/dbbench/000004.log

// The same to other ldb tools.

@empiredan empiredan enabled auto-merge (squash) September 15, 2023 03:25
@acelyc111 acelyc111 disabled auto-merge September 15, 2023 03:48
@acelyc111 acelyc111 merged commit e9206ed into pegasus-kv:v8.3.2-pegasus-encrypt Sep 15, 2023
acelyc111 added a commit that referenced this pull request Sep 15, 2023
apache/incubator-pegasus#1575

After all encryption related patches been cherry-picked from
[tikv](https://github.com/tikv/rocksdb/commits/6.29.tikv) and
merged, now we will improve the encrytion, including:
- Fix action job `build-linux-encrypted_env-no_compression-no_openssl`
  to build binaries without openssl and compression libs correctly.
- Fix action job `build-linux-encrypted_env-openssl` to export the
  `ENCRYPTED_ENV` enviroment variable correctly.
- Don not skip tests which are skipped by TiKV.
- Refactor `AESCTRCipherStream` and `AESEncryptionProvider` to support
  manage file key by the file itself, according to the design docs in
[Data at rest
encryption](apache/incubator-pegasus#1575).
- Remove all KeyManager related codes.
- Replace KeyManager tests by AES encryption tests.
- Refactor encryption/encryption_test.cc and add more tests.
- Make it possible to construct AESEncryptionProvider object via
  `EncryptionProvider::CreateFromString()` by registering a
  factory in "encryption" library.
  It's possible to construct an object by URI: `AES`, `AES://test` or
  `AES:<instance_key>,<EncryptionMethod>`.
- `ldb` tool support to parse `--fs_uri` flags as the URI mentioned
above.
- Add tests to create AESEncryptionProvider object in
  `CreateEncryptedEnvTest.CreateEncryptedFileSystem`
- `db_bench` support to run benchmark with encryption enabled, by adding
new flags for `db_bench`, they are `encryption_method` and
`encryption_instance_key`.
- Move code from the exported header directory (i.e.
include/rocksdb/encryption.h)
to rocksdb internal (i.e. encryption/encryption.h), do not expose them
to users.
- Code format.

Review hint: #17 shows all the
code changes
from the base branch (i.e. `pegasus-kv:v8.3.2-pegasus`), you can review
it together to
make sure the request branch `acelyc111:pk_enc_new` doesn't have vice
effect on the base.

Manual test:
```
// Generate some data.
./db_bench --encryption_method=AES128CTR --encryption_instance_key=test_instance_key  --num=10000

// Dump WAL OK
./tools/ldb --fs_uri="provider=AES; id=EncryptedFileSystem" dump_wal --walfile=/tmp/rocksdbtest-1000/dbbench/000004.log
./tools/ldb --fs_uri="provider=AES://test; id=EncryptedFileSystem" dump_wal --walfile=/tmp/rocksdbtest-1000/dbbench/000004.log
./tools/ldb --fs_uri="provider=AES:test_instance_key,AES128CTR; id=EncryptedFileSystem" dump_wal --walfile=/tmp/rocksdbtest-1000/dbbench/000004.log

// Dump WAL failed. Pass bad provider parameters to --fs_uri, e.g.
./tools/ldb --fs_uri="provider=AES1:test_instance_key,1AES128CTR; id=EncryptedFileSystem" dump_wal --walfile=/tmp/rocksdbtest-1000/dbbench/000004.log
./tools/ldb --fs_uri="provider=AES:bad_test_instance_key,AES128CTR; id=EncryptedFileSystem" dump_wal --walfile=/tmp/rocksdbtest-1000/dbbench/000004.log
./tools/ldb --fs_uri="provider=AES:test_instance_key,AES192CTR; id=EncryptedFileSystem" dump_wal --walfile=/tmp/rocksdbtest-1000/dbbench/000004.log

// The same to other ldb tools.

```
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants