Skip to content

[Redact log] Redact key range and value in RSFilter from logging#1282

Merged
JaySon-Huang merged 12 commits intopingcap:masterfrom
JaySon-Huang:redact_log
Dec 15, 2020
Merged

[Redact log] Redact key range and value in RSFilter from logging#1282
JaySon-Huang merged 12 commits intopingcap:masterfrom
JaySon-Huang:redact_log

Conversation

@JaySon-Huang
Copy link
Contributor

@JaySon-Huang JaySon-Huang commented Dec 14, 2020 •
edited
Loading

What problem does this PR solve?

Issue Number: related to #1277

Problem Summary: Avoid printing user data (key range and value in RSFilter) to logging file. This PR doesn't avoid printing all user data but greatly reduces those logs.

What is changed and how it works?

  • Add a new config security.redact_info_log which avoids printing user data to log
  • Add a class Redact to control whether we will convert the sensitive value to ? or not
  • Apply Redact::keyToDebugString for HandleRange,RowKeyRange,TiKVKey,TiKVValue,RawTiDBPK
  • Add a class FieldVisitorToDebugString. It Mainly to avoid printing value to log (e.g. RSFilter)
  • Change some method name from toString to toDebugString, indicating that the value will be converted to '?' if redact-log is on

Note that the PRs that redact logs with some sensitive values generated by protobuf structure DebugString/ShortDebugString will be file later

Related changes

  • PR to update pingcap/docs/pingcap/docs-cn:
  • Need to cherry-pick to the release branch

Check List

Tests

  • Unit test
  • Manual test (add detailed scripts or steps below)
    Run some queries and check the log file
    image

Side effects

  • Performance regression
    • Consumes more CPU

Release note

  • Add security.redact_info_log config, which redacts user data from logs

@JaySon-Huang JaySon-Huang changed the title [DNM][Redact log] Redact key range and value in RSFilter from logging [Redact log] Redact key range and value in RSFilter from logging Dec 14, 2020
@JaySon-Huang JaySon-Huang mentioned this pull request Dec 14, 2020
Closed
7 tasks
@JaySon-Huang
Copy link
Contributor Author

JaySon-Huang commented Dec 14, 2020

/run-all-tests

JaySon-Huang and others added 12 commits December 15, 2020 15:08
@JaySon-Huang
Redact key
583dbe3 
Signed-off-by: JaySon-Huang <tshent@qq.com>
@JaySon-Huang
Explicit toDebugString
5fc7646 
Signed-off-by: JaySon-Huang <tshent@qq.com>
@JaySon-Huang
Make it more easy to print DecodedTiKVRange
9ae66d4 
Signed-off-by: JaySon-Huang <tshent@qq.com>
@JaySon-Huang
Add configuration
31b964b 
Signed-off-by: JaySon-Huang <tshent@qq.com>
@JaySon-Huang
Refine format functions for RowKeyRange
ca5eed2 
Signed-off-by: JaySon-Huang <tshent@qq.com>
@JaySon-Huang
add test cases
0db54d4 
Signed-off-by: JaySon-Huang <tshent@qq.com>
@JaySon-Huang
Add unittest
61d5745 
Signed-off-by: JaySon-Huang <tshent@qq.com>
@JaySon-Huang
Format code
5ec10d4 
Signed-off-by: JaySon-Huang <tshent@qq.com>
@JaySon-Huang
Redact the value in RSOperator
fc8a232 
Signed-off-by: JaySon-Huang <tshent@qq.com>
@JaySon-Huang
Fix mock test
1d3965d
@JaySon-Huang
Restore redact flags after tests
445c6ed 
Signed-off-by: JaySon-Huang <tshent@qq.com>
@JaySon-Huang
Align option name style
d075581 
Signed-off-by: JaySon-Huang <tshent@qq.com>
@JaySon-Huang
Copy link
Contributor Author

JaySon-Huang commented Dec 15, 2020

/run-all-tests

@flowbehappy flowbehappy self-requested a review December 15, 2020 09:53
flowbehappy
flowbehappy approved these changes Dec 15, 2020
View reviewed changes
Copy link
Contributor

@flowbehappy flowbehappy left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@ti-srebot ti-srebot added the status/LGT1 Indicates that a PR has LGTM 1. label Dec 15, 2020
@JaySon-Huang JaySon-Huang merged commit 5c40a7e into pingcap:master Dec 15, 2020
@JaySon-Huang JaySon-Huang deleted the redact_log branch December 15, 2020 09:58
@JaySon-Huang JaySon-Huang added this to the v5.0-alpha milestone Dec 15, 2020
JaySon-Huang added a commit to JaySon-Huang/tiflash that referenced this pull request Dec 29, 2020
@JaySon-Huang
[Redact log] Redact key range and value in RSFilter from logging (pin…
840fb1c 
…gcap#1282)

Signed-off-by: JaySon-Huang <tshent@qq.com>
 Conflicts:
	dbms/src/Debug/dbgFuncRegion.cpp
	dbms/src/Storages/DeltaMerge/DeltaMerge.h
	dbms/src/Storages/DeltaMerge/DeltaMergeStore.cpp
	dbms/src/Storages/DeltaMerge/File/DMFilePackFilter.h
	dbms/src/Storages/DeltaMerge/File/DMFileReader.h
	dbms/src/Storages/DeltaMerge/RangeUtils.cpp
	dbms/src/Storages/DeltaMerge/RegionSplit.cpp
	dbms/src/Storages/DeltaMerge/ReorganizeBlockInputStream.h
	dbms/src/Storages/DeltaMerge/RowKeyRange.cpp
	dbms/src/Storages/DeltaMerge/RowKeyRange.h
	dbms/src/Storages/DeltaMerge/Segment.cpp
	dbms/src/Storages/DeltaMerge/tests/dm_basic_include.h
	dbms/src/Storages/DeltaMerge/tests/gtest_dm_file.cpp
	dbms/src/Storages/StorageDeltaMerge.cpp
	dbms/src/Storages/Transaction/LearnerRead.cpp
	dbms/src/Storages/Transaction/RegionCFDataBase.cpp
	dbms/src/Storages/Transaction/RegionData.cpp
	dbms/src/Storages/Transaction/RegionState.cpp
	dbms/src/Storages/Transaction/TiKVKeyValue.h
	dbms/src/Storages/Transaction/TiKVRecordFormat.h
	tests/delta-merge-test/raft/region_merge_common_handle.test
@JaySon-Huang JaySon-Huang mentioned this pull request Dec 29, 2020
Merged
JaySon-Huang added a commit that referenced this pull request Dec 31, 2020
@JaySon-Huang
Redact log (release-4.0) (#1333)
d54685e 
* [Redact log] Redact key range and value in RSFilter from logging (#1282)
* Log redact for client-c (#1306)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@flowbehappy flowbehappy flowbehappy approved these changes

Assignees

No one assigned

Labels

status/LGT1 Indicates that a PR has LGTM 1.

Projects

None yet

Milestone

v5.0-alpha

Development

Successfully merging this pull request may close these issues.

3 participants

@JaySon-Huang @flowbehappy @ti-srebot