expression/expressions: replace iconv-go with text/transform#28
Closed
unknwon wants to merge 1 commit intopingcap:masterfrom
unknwon:master
Closed
expression/expressions: replace iconv-go with text/transform#28unknwon wants to merge 1 commit intopingcap:masterfrom unknwon:master
unknwon wants to merge 1 commit intopingcap:masterfrom
unknwon:master
Conversation
Remove CGO requirement for compiling iconv-go currently is using. Replace it with native implementation of Go. Signed-off-by: Unknwon <u@gogs.io>
Contributor
Author
|
Thank god, passed this time. |
|
LGTM. But I cannot press the merge button. :( |
Contributor
Author
|
Godep is killing GitHub diff. So, you can't even find where did I changed. |
|
@unknwon I usually keep them into separate commits:
|
Contributor
Author
|
godep save doesn't work with tidb out of box, so I changed everything manually 😂 really don't want to go through hell again. |
Contributor
Author
|
OK... there is the diff: diff --git a/expression/expressions/convert.go b/expression/expressions/convert.go
index b636cb5..8fc7918 100644
--- a/expression/expressions/convert.go
+++ b/expression/expressions/convert.go
@@ -17,11 +17,12 @@ import (
"fmt"
"strings"
- iconv "github.com/djimenez/iconv-go"
"github.com/juju/errors"
"github.com/ngaut/log"
"github.com/pingcap/tidb/context"
"github.com/pingcap/tidb/expression"
+ "golang.org/x/net/html/charset"
+ "golang.org/x/text/transform"
)
// FunctionConvert provides a way to convert data between different character sets.
@@ -74,7 +75,13 @@ func (f *FunctionConvert) Eval(ctx context.Context, args map[interface{}]interfa
} else if strings.ToLower(f.Charset) == "utf8mb4" {
return value, nil
}
- target, err := iconv.ConvertString(str, "utf-8", f.Charset)
+
+ encoding, _ := charset.Lookup(f.Charset)
+ if encoding == nil {
+ return nil, fmt.Errorf("unknow char decoder %s", f.Charset)
+ }
+
+ target, _, err := transform.String(encoding.NewDecoder(), str)
if err != nil {
log.Errorf("Convert %s to %s with error: %v", str, f.Charset, err)
return nil, errors.Trace(err) |
Member
Member
|
@unknwon this PR introduced too much dependencies, can you remove the depedency of "golang.org/x/net/html/charset", all we need is a map from charset name to encoding. |
Contributor
Author
SunRunAway
pushed a commit
to SunRunAway/tidb
that referenced
this pull request
Oct 15, 2020
YuJuncen
pushed a commit
to YuJuncen/tidb
that referenced
this pull request
Apr 23, 2021
YuJuncen
pushed a commit
to YuJuncen/tidb
that referenced
this pull request
Apr 23, 2021
* add version subcommand Signed-off-by: Neil Shen <overvenus@gmail.com> * restore: fix typo Signed-off-by: Neil Shen <overvenus@gmail.com> * Init cmd on subcommands Signed-off-by: Neil Shen <overvenus@gmail.com> * tests: clean up tikv-importer Signed-off-by: Neil Shen <overvenus@gmail.com>
xhebox
pushed a commit
to xhebox/tidb
that referenced
this pull request
Sep 28, 2021
xhebox
pushed a commit
to xhebox/tidb
that referenced
this pull request
Oct 8, 2021
ti-chi-bot
pushed a commit
that referenced
this pull request
Oct 9, 2021
okJiang
pushed a commit
to okJiang/tidb
that referenced
this pull request
Oct 19, 2021
Connor1996
pushed a commit
to Connor1996/tidb
that referenced
this pull request
Dec 14, 2022
* set resource group ID for read request * replace group id with name
ywqzzy
pushed a commit
to ywqzzy/tidb
that referenced
this pull request
Jul 26, 2023
fix write metric
guoshouyan
pushed a commit
to guoshouyan/tidb
that referenced
this pull request
Mar 5, 2024
…cap#28) close pingcap#48899 Co-authored-by: Naman Gupta <naman.gupta@gmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Remove CGO requirement for compiling iconv-go currently is using.
Replace it with native implementation of Go.
maketest has passed, but test cases don't look very detailed onexpression, so not sure if everything will go well.