Skip to content

Online DDL: ensure message is valid utf8 in updateMigrationMessage()#11914

Merged
shlomi-noach merged 4 commits intovitessio:mainfrom
planetscale:onlineddl-report-error-message-binary
Dec 12, 2022
Merged

Online DDL: ensure message is valid utf8 in updateMigrationMessage()#11914
shlomi-noach merged 4 commits intovitessio:mainfrom
planetscale:onlineddl-report-error-message-binary

Conversation

@shlomi-noach
Copy link
Copy Markdown
Contributor

@shlomi-noach shlomi-noach commented Dec 8, 2022

Description

Fixes #11913

This PR ensures to clean up non-utf8 characters from the error message reported in _vt.schema_migrations. If writing the message still fails for whatever reason (the text is too long?), we opt for an alternative, generic error message.

Related Issue(s)

#11913
#6926

Checklist

  • "Backport to:" labels have been added if this change should be back-ported
  • Tests were added or are not required
  • Documentation was added or is not required

Deployment Notes

@shlomi-noach
OnlineDDL: handle binary/invalid data in updateMigrationMessage()
e003c08 
Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>
@shlomi-noach
Ensure error message is valid utf8
8b5d8e8 
Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>
@shlomi-noach shlomi-noach requested a review from dbussink December 8, 2022 12:03
@vitess-bot
Copy link
Copy Markdown
Contributor

vitess-bot bot commented Dec 8, 2022 •
edited by dbussink
Loading

Review Checklist

Hello reviewers! 👋 Please follow this checklist when reviewing this Pull Request.

General

  • Ensure that the Pull Request has a descriptive title.
  • If this is a change that users need to know about, please apply the release notes (needs details) label so that merging is blocked unless the summary release notes document is included.

If a new flag is being introduced:

  • Is it really necessary to add this flag?
  • Flag names should be clear and intuitive (as far as possible)
  • Help text should be descriptive.
  • Flag names should use dashes (-) as word separators rather than underscores (_).

If a workflow is added or modified:

  • Each item in Jobs should be named in order to mark it as required.
  • If the workflow should be required, the maintainer team should be notified.

Bug fixes

  • There should be at least one unit or end-to-end test.
  • The Pull Request description should include a link to an issue that describes the bug.

Non-trivial changes

  • There should be some code comments as to why things are implemented the way they are.

New/Existing features

  • Should be documented, either by modifying the existing documentation or creating new documentation.
  • New features should have a link to a feature request issue or an RFC that documents the use cases, corner cases and test cases.

Backward compatibility

  • Protobuf changes should be wire-compatible.
  • Changes to _vt tables and RPCs need to be backward compatible.
  • vtctl command output order should be stable and awk-able.
  • RPC changes should be compatible with vitess-operator
  • If a flag is removed, then it should also be removed from VTop, if used there.

@dbussink
Copy link
Copy Markdown
Member

dbussink commented Dec 8, 2022

If writing the message still fails for whatever reason (the text is too long?), we opt for an alternative, generic error message.

If we already know length to be a potential issue, should we already always truncate the string then anyway? And do that change here as well?

dbussink
dbussink reviewed Dec 8, 2022
View reviewed changes
go/vt/vttablet/onlineddl/executor.go Outdated
if err != nil {

update := func(message string) error {
message = strings.ToValidUTF8(message, "")
Copy link
Copy Markdown
Member

@dbussink dbussink Dec 8, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we want to use � as the replacement here since that's the canonical UTF-8 replacement character (and also what a console / log today would already print).

See also https://www.fileformat.info/info/unicode/char/fffd/index.htm

Copy link
Copy Markdown
Contributor Author

@shlomi-noach shlomi-noach Dec 8, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@dbussink sounds good. I'm down for maintenance, would you like to push that change?

Copy link
Copy Markdown
Contributor Author

@shlomi-noach shlomi-noach Dec 8, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done

@shlomi-noach
cap length; special replacement character
8e1563e 
Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>
@shlomi-noach
Copy link
Copy Markdown
Contributor Author

shlomi-noach commented Dec 8, 2022

truncating length to 2048

@shlomi-noach shlomi-noach requested a review from a team December 8, 2022 16:56
@dbussink
Copy link
Copy Markdown
Member

dbussink commented Dec 11, 2022 •
edited
Loading

@systay Looks like your recent fixes for the planner are triggered in the upgrade tests (from https://github.com/vitessio/vitess/actions/runs/3648727440/jobs/6203228592):

 === RUN   TestSubqueriesExists
[654](https://github.com/vitessio/vitess/actions/runs/3648727440/jobs/6203228592#step:18:655)
    subquery_test.go:69: 
[655](https://github.com/vitessio/vitess/actions/runs/3648727440/jobs/6203228592#step:18:656)
        	Error Trace:	/home/runner/work/vitess/vitess/go/test/endtoend/vtgate/queries/subquery/cmp.go:194
[656](https://github.com/vitessio/vitess/actions/runs/3648727440/jobs/6203228592#step:18:657)
        	            				/home/runner/work/vitess/vitess/go/test/endtoend/vtgate/queries/subquery/cmp.go:66
[657](https://github.com/vitessio/vitess/actions/runs/3648727440/jobs/6203228592#step:18:658)
        	            				/home/runner/work/vitess/vitess/go/test/endtoend/vtgate/queries/subquery/subquery_test.go:69
[658](https://github.com/vitessio/vitess/actions/runs/3648727440/jobs/6203228592#step:18:659)
        	Error:      	Received unexpected error:
[659](https://github.com/vitessio/vitess/actions/runs/3648727440/jobs/6203228592#step:18:660)
        	            	Gen4 failed while v3 did not: unsupported: unable to split predicates to derived table: :__sq_has_values1 (errno 1815) (sqlstate HY000) during query: select * from (select 1) as tmp where exists(select 1 from t1 where id1 = 1)
[660](https://github.com/vitessio/vitess/actions/runs/3648727440/jobs/6203228592#step:18:661)
        	Test:       	TestSubqueriesExists
[661](https://github.com/vitessio/vitess/actions/runs/3648727440/jobs/6203228592#step:18:662)
        	Messages:   	[Vitess Error] for query: select * from (select 1) as tmp where exists(select 1 from t1 where id1 = 1)

Not sure if that would be fixed if this PR is updated with latest main or what would need to happen to fix that?

The issue is not related to the changes here in this PR though.

dbussink
dbussink reviewed Dec 11, 2022
View reviewed changes
go/vt/vttablet/onlineddl/executor.go
message = strings.ToValidUTF8(message, "�")
if len(message) > maxlen {
message = message[0:maxlen]
}
Copy link
Copy Markdown
Member

@dbussink dbussink Dec 11, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@shlomi-noach Realizing that we should change the order here I think. Since slicing this can then break the last character if the slicing happens on a word boundary and it still creates invalidly encoded data.

Copy link
Copy Markdown
Member

@dbussink dbussink Dec 11, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Since message is a TEXT, 16383 would be a safe maxlen. But we still then need to flip the order, so first truncate and then convert to valid UTF-8. It could lead to a � at the end if then we break the last potentially valid unicode char, but I think that's a totally ok trade off if the message is that long already anyway.

Copy link
Copy Markdown
Contributor Author

@shlomi-noach shlomi-noach Dec 11, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done

@shlomi-noach
swap order, so that utf8 compliance of string is ensured
d33d7d3 
Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>
rohit-nayak-ps
rohit-nayak-ps approved these changes Dec 12, 2022
View reviewed changes
Copy link
Copy Markdown
Member

@rohit-nayak-ps rohit-nayak-ps left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm

@shlomi-noach shlomi-noach merged commit 2d08a98 into vitessio:main Dec 12, 2022
@shlomi-noach shlomi-noach deleted the onlineddl-report-error-message-binary branch December 12, 2022 14:36
@systay
Copy link
Copy Markdown
Collaborator

systay commented Dec 12, 2022

@systay Looks like your recent fixes for the planner are triggered in the upgrade tests (from https://github.com/vitessio/vitess/actions/runs/3648727440/jobs/6203228592):

#11923 should have fixed this

@shlomi-noach shlomi-noach mentioned this pull request Jul 13, 2023
Merged
4 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@dbussink dbussink dbussink approved these changes

@rohit-nayak-ps rohit-nayak-ps rohit-nayak-ps approved these changes

@mattlord mattlord Awaiting requested review from mattlord mattlord is a code owner

Assignees

No one assigned

Labels

Component: Query Serving Type: Bug

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

OnlineDDL: updateMigrationMessage() can fail with binary input

4 participants

@shlomi-noach @dbussink @systay @rohit-nayak-ps