-
Notifications
You must be signed in to change notification settings - Fork 1.4k
Add metadata for Encrypted Backups #12354
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add metadata for Encrypted Backups #12354
Conversation
Result of foundationdb-pr-clang-ide on Linux RHEL 9
|
Result of foundationdb-pr-clang-arm on Linux CentOS 7
|
Result of foundationdb-pr on Linux RHEL 9
|
Result of foundationdb-pr-cluster-tests on Linux RHEL 9
|
Result of foundationdb-pr-clang on Linux RHEL 9
|
046a9fb
to
60993cb
Compare
Result of foundationdb-pr on Linux RHEL 9
|
Result of foundationdb-pr-clang-ide on Linux RHEL 9
|
Result of foundationdb-pr-clang on Linux RHEL 9
|
Result of foundationdb-pr-cluster-tests on Linux RHEL 9
|
Result of foundationdb-pr-clang-arm on Linux CentOS 7
|
Result of foundationdb-pr-macos-m1 on macOS Ventura 13.x
|
60993cb
to
3bbfad1
Compare
Result of foundationdb-pr-macos on macOS Ventura 13.x
|
Result of foundationdb-pr on Linux RHEL 9
|
Result of foundationdb-pr-clang-ide on Linux RHEL 9
|
Result of foundationdb-pr-clang on Linux RHEL 9
|
Result of foundationdb-pr-clang-arm on Linux CentOS 7
|
Result of foundationdb-pr-cluster-tests on Linux RHEL 9
|
Result of foundationdb-pr-macos-m1 on macOS Ventura 13.x
|
Result of foundationdb-pr-clang-ide on Linux RHEL 9
|
Result of foundationdb-pr-macos-m1 on macOS Ventura 13.x
|
Result of foundationdb-pr-clang-arm on Linux CentOS 7
|
Result of foundationdb-pr-cluster-tests on Linux RHEL 9
|
Result of foundationdb-pr-clang on Linux RHEL 9
|
Result of foundationdb-pr on Linux RHEL 9
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM! Thanks!
May be in a different PR, there should be some encryption unit tests, add these checks of not passing encryption file and validating the errors
Reference<IAsyncFile> f = makeReference<AsyncFileS3BlobStoreRead>(m_bstore, m_bucket, dataPath(path)); | ||
|
||
if (usesEncryption()) { | ||
if (usesEncryption() && !StringRef(path).startsWith("properties/"_sr)) { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Were the properties file previously encrypted?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yes, initially they were encrypted.
So if fdbbackup describe
is called from command line then no encryption of properties and
and if fdbrestore
is called then properties are encrypted.
* Add metadata for Encrypted backups * Format files
* Add metadata for Encrypted backups * Format files
This PR adds
file_level_encryption
metadata file inproperties
folder inside backup directory during backup creation that contains whether backup files are encrypted or not. It contains 0 or 1.It also excludes encryption of other metadata files in properties to keep it consistent and avoid conflicts for
fdbackup describe
andfdbrestore
code path.fdbbackup describe shows file level encryption as:
Testing
100k correctness test (Completed)
20250909-173704-ak_bkup1-fd31f1ec86ab12af compressed=True data_size=41538538 duration=6039823 ended=100000 fail=1 fail_fast=10 max_runs=100000 pass=99999 priority=100 remaining=0 runtime=1:30:32 sanity=False started=100000 stopped=20250909-190736 submitted=20250909-173704 timeout=5400 username=ak_bkup1
One failure in
RandomSeed="905797951" SourceVersion="161fa8060c161b6e36e5ad1abb78b3f4c3ef9686" Time="1757442771" BuggifyEnabled="1" DeterminismCheck="0" FaultInjectionEnabled="1" TestFile="tests/slow/FastTriggeredWatches.toml"
- Not related to this change or backup/restoreNOTE:
It throws error when backup is not encrypted and during restore encryption is passed. It can be changed in separate PR if needed as it's not straightforward and container is opened with encryption key and encryption parameters are set before it reads metadata.
Code-Reviewer Section
The general pull request guidelines can be found here.
Please check each of the following things and check all boxes before accepting a PR.
For Release-Branches
If this PR is made against a release-branch, please also check the following:
release-branch
ormain
if this is the youngest branch)