Skip to content

log redaction: reduce overly conservative log redaction policies #86316

@abarganier

Description

@abarganier

Is your feature request related to a problem? Please describe.
Support staff have long relied on debug zip bundles to help folks running CockroachDB debug & diagnose issues. A key piece of these debug zip bundles are the dumps of all the logs available on each node. These logs are a core piece of observability data for operators and support staff alike.

However, as CockroachDB matures, so do the compliance policies that it must meet. With compliance comes a need for us to expand our redaction to all sources of data that flow out of CockroachDB, especially those used by folks who don't actually own the running cluster (e.g. support staff).

debug zip has recently been updated to support a --redact flag, which has taken over from the now deprecated --redact-logs flag. A redacted debug zip will redact all of its logs - a feature that has existed for some time now, but has not been utilized much as our compliance policies didn't mandate it.

Today, our compliance policies are expanding, and in the future a redacted debug zip bundle may be all that our support staff are able to gain access to.

Unfortunately, support has made it very clear that redacted logs as they stand today are insufficient for debugging & diagnosing support escalations. Redacted logs feel "overly redacted" to the point where operational information that is unrelated to customer data (e.g. NodeIDs, operation priorities, ClusterID, queries that already have constants stripped, etc).

Describe the solution you'd like
If in the future we are going to enforce that our support staff is only able to access logs in redacted form, we need to make sure that those logs are redacting data thoughtfully, as opposed to the overly-conservative redaction of today.

We need to define what "sensitive data" means in the context of log redaction, and draw clear lines around what's redacted & what's left unredacted. We can then make targeted improvements to log redaction by implementing SafeFormatter where appropriate, or explicitly marking objects interpolated into logs as "safe".

Additional context
In other redaction work aimed at meeting compliance requirements, all customer data has been identified as "sensitive". The exception here is key data, which may contain row-level data but has been deemed necessary to support CockroachDB. Can the same definitions be applied to log redaction?

Jira issue: CRDB-18694

Epic CRDB-12732

Metadata

Metadata

Assignees

Labels

A-observability-infC-enhancementSolution expected to add code/behavior + preserve backward-compat (pg compat issues are exception)

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions