Skip to content

Redact log (release-4.0)#1333

Merged
JaySon-Huang merged 8 commits intopingcap:release-4.0from
JaySon-Huang:redact_log_4.0
Dec 31, 2020
Merged

Redact log (release-4.0)#1333
JaySon-Huang merged 8 commits intopingcap:release-4.0from
JaySon-Huang:redact_log_4.0

Conversation

@JaySon-Huang
Copy link
Contributor

@JaySon-Huang JaySon-Huang commented Dec 29, 2020 •
edited
Loading

cherry-pick of #1282, #1306

What problem does this PR solve?

Issue Number: related to #1277

Problem Summary: Avoid printing user data (key range and value in RSFilter) to logging file. This PR doesn't avoid printing all user data but greatly reduces those logs.

What is changed and how it works?

  • Add a new config security.redact_info_log which avoids printing user data to log
  • Add a class Redact to control whether we will convert the sensitive value to ? or not
  • Apply Redact::keyToDebugString for HandleRange,RowKeyRange,TiKVKey,TiKVValue,RawTiDBPK
  • Add a class FieldVisitorToDebugString. It Mainly to avoid printing value to log (e.g. RSFilter)
  • Change some method name from toString to toDebugString, indicating that the value will be converted to '?' if redact-log is on
  • Update client-c and set redact flag for client-c to redact the key in error log

Note that the PRs that redact logs with some sensitive values generated by protobuf structure DebugString/ShortDebugString will be file later

Related changes

  • PR to update pingcap/docs/pingcap/docs-cn:
  • Need to cherry-pick to the release branch

Check List

Tests

  • Unit test
  • Manual test (add detailed scripts or steps below)
  • Run some queries and check the log file
    image

Side effects

  • Performance regression
    • Consumes more CPU

Release note

  • Add security.redact_info_log config, which redacts user data from logs

JaySon-Huang and others added 3 commits December 29, 2020 13:33
@JaySon-Huang
[Redact log] Redact key range and value in RSFilter from logging (pin…
840fb1c 
…gcap#1282)

Signed-off-by: JaySon-Huang <tshent@qq.com>
 Conflicts:
	dbms/src/Debug/dbgFuncRegion.cpp
	dbms/src/Storages/DeltaMerge/DeltaMerge.h
	dbms/src/Storages/DeltaMerge/DeltaMergeStore.cpp
	dbms/src/Storages/DeltaMerge/File/DMFilePackFilter.h
	dbms/src/Storages/DeltaMerge/File/DMFileReader.h
	dbms/src/Storages/DeltaMerge/RangeUtils.cpp
	dbms/src/Storages/DeltaMerge/RegionSplit.cpp
	dbms/src/Storages/DeltaMerge/ReorganizeBlockInputStream.h
	dbms/src/Storages/DeltaMerge/RowKeyRange.cpp
	dbms/src/Storages/DeltaMerge/RowKeyRange.h
	dbms/src/Storages/DeltaMerge/Segment.cpp
	dbms/src/Storages/DeltaMerge/tests/dm_basic_include.h
	dbms/src/Storages/DeltaMerge/tests/gtest_dm_file.cpp
	dbms/src/Storages/StorageDeltaMerge.cpp
	dbms/src/Storages/Transaction/LearnerRead.cpp
	dbms/src/Storages/Transaction/RegionCFDataBase.cpp
	dbms/src/Storages/Transaction/RegionData.cpp
	dbms/src/Storages/Transaction/RegionState.cpp
	dbms/src/Storages/Transaction/TiKVKeyValue.h
	dbms/src/Storages/Transaction/TiKVRecordFormat.h
	tests/delta-merge-test/raft/region_merge_common_handle.test
@JaySon-Huang
Fix compile error
021adae 
Signed-off-by: JaySon-Huang <tshent@qq.com>
@JaySon-Huang
Fix test cases
95c2d30
@JaySon-Huang JaySon-Huang added the CHERRY-PICK cherry pick label Dec 29, 2020
@JaySon-Huang JaySon-Huang self-assigned this Dec 29, 2020
JaySon-Huang and others added 2 commits December 29, 2020 14:43
@JaySon-Huang
Log redact for client-c (pingcap#1306)
11fb025 
* Redact log for client-c

Signed-off-by: JaySon-Huang <tshent@qq.com>
@JaySon-Huang
Update client-c
49318bb
@JaySon-Huang
Copy link
Contributor Author

JaySon-Huang commented Dec 29, 2020

/run-all-tests

@JaySon-Huang JaySon-Huang changed the title [DNM] Redact log (release-4.0) Redact log (release-4.0) Dec 29, 2020
@JaySon-Huang
Copy link
Contributor Author

JaySon-Huang commented Dec 29, 2020

This will update client-c, which upgrades the kvproto library. I think there is no backward compatibility issue on upgrading it on the release-4.0 branch? @solotzg

@JaySon-Huang
Copy link
Contributor Author

JaySon-Huang commented Dec 30, 2020

/run-all-tests

@JaySon-Huang JaySon-Huang added this to the v4.0.10 milestone Dec 30, 2020
@JaySon-Huang
Copy link
Contributor Author

JaySon-Huang commented Dec 30, 2020

/run-all-tests

@JaySon-Huang
Copy link
Contributor Author

JaySon-Huang commented Dec 30, 2020

/run-all-tests

@JaySon-Huang
Copy link
Contributor Author

JaySon-Huang commented Dec 31, 2020 •
edited
Loading

@zanmato1984 @flowbehappy PTAL

@flowbehappy flowbehappy self-requested a review December 31, 2020 05:28
zanmato1984
zanmato1984 approved these changes Dec 31, 2020
View reviewed changes
Copy link
Contributor

@zanmato1984 zanmato1984 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@ti-srebot ti-srebot added the status/LGT1 Indicates that a PR has LGTM 1. label Dec 31, 2020
@JaySon-Huang JaySon-Huang merged commit d54685e into pingcap:release-4.0 Dec 31, 2020
@JaySon-Huang JaySon-Huang deleted the redact_log_4.0 branch December 31, 2020 05:46
@JaySon-Huang JaySon-Huang mentioned this pull request Mar 9, 2021
Merged
11 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@zanmato1984 zanmato1984 zanmato1984 approved these changes

@flowbehappy flowbehappy Awaiting requested review from flowbehappy

Assignees

@JaySon-Huang JaySon-Huang

Labels

CHERRY-PICK cherry pick status/LGT1 Indicates that a PR has LGTM 1.

Projects

None yet

Milestone

v4.0.10

Development

Successfully merging this pull request may close these issues.

3 participants

@JaySon-Huang @zanmato1984 @ti-srebot