close tikv#16974

Signed-off-by: ti-chi-bot <ti-community-prow-bot@tidb.io>

close #16974

expression: fix casting REAL type to STRING type
- unify the behavior of converting real types to strings in tidb#53129 and tikv.
- use ryu lib in tikv to implement casting real type to str type.
- `if abs(num) >= 1e15 or (num is not zero and abs(num) < 1e-15)` then use decimal exponent format

Signed-off-by: TONG, Zhigao <tongzhigao@pingcap.com>

Co-authored-by: TONG, Zhigao <tongzhigao@pingcap.com>

close #16974

Co-authored-by: ti-chi-bot[bot] <108142056+ti-chi-bot[bot]@users.noreply.github.com>

…ranch (tikv#2160)

* copr: fix cast_bytes_like_as_duration encoding failed (tikv#15819)

close tikv#15820

fix cast_bytes_like_as_duration encoding failed

Signed-off-by: wshwsh12 <793703860@qq.com>

Co-authored-by: ti-chi-bot[bot] <108142056+ti-chi-bot[bot]@users.noreply.github.com>

* grafana: Fix wrong scheduler command variables of grafana in cloud env (tikv#15833)

close tikv#15832

Fix wrong scheduler command variables of grafana in cloud env by adding a `\b` to regex to make sure it's at the word boundary.

Signed-off-by: Connor1996 <zbk602423539@gmail.com>

* metrics: fix TiKV Detail command regex escape issue (tikv#15858)

ref tikv#15832

Signed-off-by: Neil Shen <overvenus@gmail.com>

* metrics: fix 2 incorrect grafana expression (tikv#15860)

close tikv#15859

Signed-off-by: glorv <glorvs@163.com>

Co-authored-by: ti-chi-bot[bot] <108142056+ti-chi-bot[bot]@users.noreply.github.com>

* Copr: do not change expression type for `Lower` (tikv#16010)

close tikv#16009

Copr: do not change expression type for `Lower`

Signed-off-by: gengliqi <gengliqiii@gmail.com>

* tidb_query_expr: fix incorrect intdiv for decimal type (tikv#16025)

close tikv#16024

Signed-off-by: wjhuang2016 <huangwenjun1997@gmail.com>

Co-authored-by: iosmanthus <dengliming@pingcap.com>

* coprocessor: rewrite `Upper` and `Lower` functions by porting the implementation from Go library (tikv#16160)

close tikv#16159

rewrite `Upper` and `Lower` functions by porting the implementation from Go library

Signed-off-by: gengliqi <gengliqiii@gmail.com>

Co-authored-by: ti-chi-bot[bot] <108142056+ti-chi-bot[bot]@users.noreply.github.com>

* coprocessor: add query digest tracing in tikv slow query (tikv#16172)

ref tikv#15513

coprocessor: add query digest tracing in tikv slow query

Signed-off-by: nolouch <nolouch@gmail.com>

* coprocessor: make the error format the same as tidb (tikv#16404)

close tikv#16407

Signed-off-by: xufei <xufeixw@mail.ustc.edu.cn>

* executor: fix incorrect handling expr eval results in projection executor (tikv#16590)

close tikv#16589

Expression's evaluation results can be three types:

1. Scalar: ScalarValue
2. Generated: RpnStackNodeVectorValue::Generated
3. Ref: RpnStackNodeVectorValue
For Ref type, it maintains a physical vector ref and an logical index vector ref, it is expected to only use indexes in logical index vector to access physical vector.
For Generated type, it maintains only a physical vector, all elements can be accessed.
So for a query like select col0, cast(col0 as signed) from t0 where col0 > 100, the 'col0' projection expression will produce Ref which reuses input column's logical index vector, while 'cast(col0 as signed)' will produce Generated.
However, in current projection executor, we will use input column's logical index vector for both two expression results,  causing index out of range for Generated one.
Change to transform Ref to Generated, and recreate logical index vector to [0...N) to fix the issue.

Signed-off-by: yibin <huyibin@pingcap.com>

Co-authored-by: Liqi Geng <gengliqiii@gmail.com>

* tidb_query_datatype, tidb_query_expr: Add div_precision_increment support in dag request (tikv#16622)

close tikv#16627, ref pingcap/tidb#51501

Add support for div_precision_increment in dag request.

Signed-off-by: yibin <huyibin@pingcap.com>

* expr: add json_array_append support (tikv#16690)

close tikv#16703

Signed-off-by: dbsid <chenhuansheng@pingcap.com>

* tidb_query_expr: fix overflow in elt function (tikv#14826)

close tikv#14827

Changed `i + 1 > raw_args.len()` to `i >= raw_args.len()` to avoid overflow

Signed-off-by: Hangjie Mo <mohangjie1995@gmail.com>

Co-authored-by: MyonKeminta <9948422+MyonKeminta@users.noreply.github.com>

* components: only check output schema field type when set resp encode type (tikv#14817)

close tikv#14818

Signed-off-by: Hangjie Mo <mohangjie1995@gmail.com>
Signed-off-by: Jason Mo <mohangjie1995@gmail.com>

* expr: Support json_merge_patch pushdown (tikv#16771)

close tikv#16770

Signed-off-by: dbsid <chenhuansheng@pingcap.com>

Co-authored-by: ti-chi-bot[bot] <108142056+ti-chi-bot[bot]@users.noreply.github.com>

* statistics: add comments for FmSketch (tikv#16968)

ref tikv#16463

Add more comments for FmSketch.

Signed-off-by: hi-rustin <rustin.liu@gmail.com>

Co-authored-by: ti-chi-bot[bot] <108142056+ti-chi-bot[bot]@users.noreply.github.com>

* expr: fix panic in json_array_append (tikv#16929)

close tikv#16930

mysql> drop table if exists t;
Query OK, 0 rows affected, 1 warning (0.05 sec)

mysql> create table t(a json, index idx((cast(json_array_append(a, '$.a', 1) as char(300)))));
Query OK, 0 rows affected (0.12 sec)

mysql> insert into t (a) values ('{\"a\": 1}');
Query OK, 1 row affected (0.07 sec)

mysql> insert into t (a) values ('{\"b\": 1}');
Query OK, 1 row affected (0.05 sec)

mysql> insert into t (a) values ('{\"a\": {\"b\": 1}}');
Query OK, 1 row affected (0.07 sec)

mysql> insert into t (a) values ('{\"a\": {\"b\": [1,2,3]}}');
Query OK, 1 row affected (0.07 sec)

mysql> insert into t (a) values ('[1,2,3]');
Query OK, 1 row affected (0.06 sec)

mysql> insert into t (a) values ('[\"a\",\"b\"]');
Query OK, 1 row affected (0.03 sec)

mysql> insert into t (a) values ('\"abc\"');
Query OK, 1 row affected (0.07 sec)

mysql> insert into t (a) values ('123');
Query OK, 1 row affected (0.07 sec)

mysql> insert into t (a) values ('true');
Query OK, 1 row affected (0.08 sec)

mysql> insert into t (a) values ('false');
Query OK, 1 row affected (0.04 sec)

mysql> insert into t (a) values (NULL);
Query OK, 1 row affected (0.08 sec)

mysql>
mysql> select * from t ignore index(idx) where cast(json_array_append(a, '$.a', 1) as char(300)) = '{"a": [1, 1]}';
+----------+
| a        |
+----------+
| {"a": 1} |
+----------+
1 row in set (0.06 sec)

Signed-off-by: dbsid <chenhuansheng@pingcap.com>

Co-authored-by: ti-chi-bot[bot] <108142056+ti-chi-bot[bot]@users.noreply.github.com>

* expression: fix casting REAL type to STRING type (tikv#16975)

close tikv#16974

expression: fix casting REAL type to STRING type
- unify the behavior of converting real types to strings in tidb#53129 and tikv.
- use ryu lib in tikv to implement casting real type to str type.
- `if abs(num) >= 1e15 or (num is not zero and abs(num) < 1e-15)` then use decimal exponent format

Signed-off-by: TONG, Zhigao <tongzhigao@pingcap.com>

* expression: reverts tikv#16975 (tikv#17013)

close tikv#16974

Co-authored-by: ti-chi-bot[bot] <108142056+ti-chi-bot[bot]@users.noreply.github.com>

* statistics: update comments for FMSketch (tikv#17073)

 

Update comments for FMSketch

Signed-off-by: hi-rustin <rustin.liu@gmail.com>

* executor: Optimize projection executor to avoid deep copy for no dup column ref (tikv#16856)

close tikv#16858

Reconginize no-dup column ref only projections, and avoid deep VectorValue copy for such cases.

Signed-off-by: yibin <huyibin@pingcap.com>

Co-authored-by: Liqi Geng <gengliqiii@gmail.com>

* copr: let `ExtraPhysTblIDCol` column return partition id when it's a global indexScan (tikv#17141)

close tikv#17138

let `ExtraPhysTblIDCol` column return partition id when it's a global indexScan

Signed-off-by: Jason Mo <mohangjie1995@gmail.com>
Signed-off-by: Hangjie Mo <mohangjie1995@gmail.com>

Co-authored-by: ti-chi-bot[bot] <108142056+ti-chi-bot[bot]@users.noreply.github.com>

* copr: add a `force_no_pad` argument to `sort_compare`, and fix `like` behavior (tikv#17337)

close tikv#17332

Signed-off-by: Yang Keao <yangkeao@chunibyo.icu>

Co-authored-by: ti-chi-bot[bot] <108142056+ti-chi-bot[bot]@users.noreply.github.com>

* charset,collation: implement `GB18030` and 2 collations (tikv#17467)

close tikv#17470

* Add a new charset `gb18030`
* Add two related collations `gb18030_chinese_ci` and `gb18030_bin`

Signed-off-by: cbcwestwolf <1004626265@qq.com>

Co-authored-by: ti-chi-bot[bot] <108142056+ti-chi-bot[bot]@users.noreply.github.com>

* Support date_add/date_sub expression functions (tikv#17541)

close tikv#17529

Support date_add/date_sub expression functions

Signed-off-by: gengliqi <gengliqiii@gmail.com>

Co-authored-by: ti-chi-bot[bot] <108142056+ti-chi-bot[bot]@users.noreply.github.com>

* expression: date_add/date_sub handle overflow calculation properly (tikv#17609)

close tikv#17608

expression: date_add/date_sub handle overflow calculation properly

Signed-off-by: gengliqi <gengliqiii@gmail.com>

Co-authored-by: ti-chi-bot[bot] <108142056+ti-chi-bot[bot]@users.noreply.github.com>

* expr: fix panic when using radians and degree (tikv#17853) (tikv#17854)

close tikv#17852

expr: fix panic when using radians and degree

Signed-off-by: gengliqi <gengliqiii@gmail.com>

Co-authored-by: gengliqi <gengliqiii@gmail.com>

* fix build

Signed-off-by: Connor1996 <zbk602423539@gmail.com>

* fix clippy

Signed-off-by: Connor1996 <zbk602423539@gmail.com>

* fix format

Signed-off-by: Connor1996 <zbk602423539@gmail.com>

---------

Signed-off-by: Connor1996 <zbk602423539@gmail.com>
Signed-off-by: Neil Shen <overvenus@gmail.com>
Signed-off-by: gengliqi <gengliqiii@gmail.com>
Signed-off-by: nolouch <nolouch@gmail.com>
Signed-off-by: xufei <xufeixw@mail.ustc.edu.cn>
Signed-off-by: yibin <huyibin@pingcap.com>
Signed-off-by: dbsid <chenhuansheng@pingcap.com>
Signed-off-by: Hangjie Mo <mohangjie1995@gmail.com>
Signed-off-by: Jason Mo <mohangjie1995@gmail.com>
Signed-off-by: TONG, Zhigao <tongzhigao@pingcap.com>
Signed-off-by: hi-rustin <rustin.liu@gmail.com>
Co-authored-by: Shenghui Wu <793703860@qq.com>
Co-authored-by: ti-chi-bot[bot] <108142056+ti-chi-bot[bot]@users.noreply.github.com>
Co-authored-by: Neil Shen <overvenus@gmail.com>
Co-authored-by: glorv <glorvs@163.com>
Co-authored-by: Liqi Geng <gengliqiii@gmail.com>
Co-authored-by: wjHuang <huangwenjun1997@gmail.com>
Co-authored-by: iosmanthus <dengliming@pingcap.com>
Co-authored-by: ShuNing <nolouch@gmail.com>
Co-authored-by: xufei <xufeixw@mail.ustc.edu.cn>
Co-authored-by: yibin <huyibin@pingcap.com>
Co-authored-by: dbsid <chenhuansheng@pingcap.com>
Co-authored-by: Hangjie Mo <mohangjie1995@gmail.com>
Co-authored-by: MyonKeminta <9948422+MyonKeminta@users.noreply.github.com>
Co-authored-by: 二手掉包工程师 <rustin.liu@gmail.com>
Co-authored-by: TONG, Zhigao <tongzhigao@pingcap.com>
Co-authored-by: YangKeao <yangkeao@chunibyo.icu>
Co-authored-by: CbcWestwolf <1004626265@qq.com>
Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io>

…ranch (tikv#2160) (tikv#2794)

* copr: fix cast_bytes_like_as_duration encoding failed (tikv#15819)

close tikv#15820

fix cast_bytes_like_as_duration encoding failed





* grafana: Fix wrong scheduler command variables of grafana in cloud env (tikv#15833)

close tikv#15832

Fix wrong scheduler command variables of grafana in cloud env by adding a `\b` to regex to make sure it's at the word boundary.



* metrics: fix TiKV Detail command regex escape issue (tikv#15858)

ref tikv#15832



* metrics: fix 2 incorrect grafana expression (tikv#15860)

close tikv#15859





* Copr: do not change expression type for `Lower` (tikv#16010)

close tikv#16009

Copr: do not change expression type for `Lower`



* tidb_query_expr: fix incorrect intdiv for decimal type (tikv#16025)

close tikv#16024





* coprocessor: rewrite `Upper` and `Lower` functions by porting the implementation from Go library (tikv#16160)

close tikv#16159

rewrite `Upper` and `Lower` functions by porting the implementation from Go library





* coprocessor: add query digest tracing in tikv slow query (tikv#16172)

ref tikv#15513

coprocessor: add query digest tracing in tikv slow query



* coprocessor: make the error format the same as tidb (tikv#16404)

close tikv#16407



* executor: fix incorrect handling expr eval results in projection executor (tikv#16590)

close tikv#16589

Expression's evaluation results can be three types:

1. Scalar: ScalarValue
2. Generated: RpnStackNodeVectorValue::Generated
3. Ref: RpnStackNodeVectorValue
For Ref type, it maintains a physical vector ref and an logical index vector ref, it is expected to only use indexes in logical index vector to access physical vector.
For Generated type, it maintains only a physical vector, all elements can be accessed.
So for a query like select col0, cast(col0 as signed) from t0 where col0 > 100, the 'col0' projection expression will produce Ref which reuses input column's logical index vector, while 'cast(col0 as signed)' will produce Generated.
However, in current projection executor, we will use input column's logical index vector for both two expression results,  causing index out of range for Generated one.
Change to transform Ref to Generated, and recreate logical index vector to [0...N) to fix the issue.





* tidb_query_datatype, tidb_query_expr: Add div_precision_increment support in dag request (tikv#16622)

close tikv#16627, ref pingcap/tidb#51501

Add support for div_precision_increment in dag request.



* expr: add json_array_append support (tikv#16690)

close tikv#16703



* tidb_query_expr: fix overflow in elt function (tikv#14826)

close tikv#14827

Changed `i + 1 > raw_args.len()` to `i >= raw_args.len()` to avoid overflow





* components: only check output schema field type when set resp encode type (tikv#14817)

close tikv#14818




* expr: Support json_merge_patch pushdown (tikv#16771)

close tikv#16770





* statistics: add comments for FmSketch (tikv#16968)

ref tikv#16463

Add more comments for FmSketch.





* expr: fix panic in json_array_append (tikv#16929)

close tikv#16930

mysql> drop table if exists t;
Query OK, 0 rows affected, 1 warning (0.05 sec)

mysql> create table t(a json, index idx((cast(json_array_append(a, '$.a', 1) as char(300)))));
Query OK, 0 rows affected (0.12 sec)

mysql> insert into t (a) values ('{\"a\": 1}');
Query OK, 1 row affected (0.07 sec)

mysql> insert into t (a) values ('{\"b\": 1}');
Query OK, 1 row affected (0.05 sec)

mysql> insert into t (a) values ('{\"a\": {\"b\": 1}}');
Query OK, 1 row affected (0.07 sec)

mysql> insert into t (a) values ('{\"a\": {\"b\": [1,2,3]}}');
Query OK, 1 row affected (0.07 sec)

mysql> insert into t (a) values ('[1,2,3]');
Query OK, 1 row affected (0.06 sec)

mysql> insert into t (a) values ('[\"a\",\"b\"]');
Query OK, 1 row affected (0.03 sec)

mysql> insert into t (a) values ('\"abc\"');
Query OK, 1 row affected (0.07 sec)

mysql> insert into t (a) values ('123');
Query OK, 1 row affected (0.07 sec)

mysql> insert into t (a) values ('true');
Query OK, 1 row affected (0.08 sec)

mysql> insert into t (a) values ('false');
Query OK, 1 row affected (0.04 sec)

mysql> insert into t (a) values (NULL);
Query OK, 1 row affected (0.08 sec)

mysql>
mysql> select * from t ignore index(idx) where cast(json_array_append(a, '$.a', 1) as char(300)) = '{"a": [1, 1]}';
+----------+
| a        |
+----------+
| {"a": 1} |
+----------+
1 row in set (0.06 sec)





* expression: fix casting REAL type to STRING type (tikv#16975)

close tikv#16974

expression: fix casting REAL type to STRING type
- unify the behavior of converting real types to strings in tidb#53129 and tikv.
- use ryu lib in tikv to implement casting real type to str type.
- `if abs(num) >= 1e15 or (num is not zero and abs(num) < 1e-15)` then use decimal exponent format



* expression: reverts tikv#16975 (tikv#17013)

close tikv#16974



* statistics: update comments for FMSketch (tikv#17073)

 

Update comments for FMSketch



* executor: Optimize projection executor to avoid deep copy for no dup column ref (tikv#16856)

close tikv#16858

Reconginize no-dup column ref only projections, and avoid deep VectorValue copy for such cases.





* copr: let `ExtraPhysTblIDCol` column return partition id when it's a global indexScan (tikv#17141)

close tikv#17138

let `ExtraPhysTblIDCol` column return partition id when it's a global indexScan






* copr: add a `force_no_pad` argument to `sort_compare`, and fix `like` behavior (tikv#17337)

close tikv#17332





* charset,collation: implement `GB18030` and 2 collations (tikv#17467)

close tikv#17470

* Add a new charset `gb18030`
* Add two related collations `gb18030_chinese_ci` and `gb18030_bin`





* Support date_add/date_sub expression functions (tikv#17541)

close tikv#17529

Support date_add/date_sub expression functions





* expression: date_add/date_sub handle overflow calculation properly (tikv#17609)

close tikv#17608

expression: date_add/date_sub handle overflow calculation properly





* expr: fix panic when using radians and degree (tikv#17853) (tikv#17854)

close tikv#17852

expr: fix panic when using radians and degree





* fix build



* fix clippy



* fix format



---------

Signed-off-by: Connor1996 <zbk602423539@gmail.com>
Signed-off-by: Neil Shen <overvenus@gmail.com>
Signed-off-by: gengliqi <gengliqiii@gmail.com>
Signed-off-by: nolouch <nolouch@gmail.com>
Signed-off-by: xufei <xufeixw@mail.ustc.edu.cn>
Signed-off-by: yibin <huyibin@pingcap.com>
Signed-off-by: dbsid <chenhuansheng@pingcap.com>
Signed-off-by: Hangjie Mo <mohangjie1995@gmail.com>
Signed-off-by: Jason Mo <mohangjie1995@gmail.com>
Signed-off-by: TONG, Zhigao <tongzhigao@pingcap.com>
Signed-off-by: hi-rustin <rustin.liu@gmail.com>
Co-authored-by: Connor <zbk602423539@gmail.com>
Co-authored-by: Shenghui Wu <793703860@qq.com>
Co-authored-by: ti-chi-bot[bot] <108142056+ti-chi-bot[bot]@users.noreply.github.com>
Co-authored-by: Neil Shen <overvenus@gmail.com>
Co-authored-by: glorv <glorvs@163.com>
Co-authored-by: Liqi Geng <gengliqiii@gmail.com>
Co-authored-by: wjHuang <huangwenjun1997@gmail.com>
Co-authored-by: iosmanthus <dengliming@pingcap.com>
Co-authored-by: ShuNing <nolouch@gmail.com>
Co-authored-by: xufei <xufeixw@mail.ustc.edu.cn>
Co-authored-by: yibin <huyibin@pingcap.com>
Co-authored-by: dbsid <chenhuansheng@pingcap.com>
Co-authored-by: Hangjie Mo <mohangjie1995@gmail.com>
Co-authored-by: MyonKeminta <9948422+MyonKeminta@users.noreply.github.com>
Co-authored-by: 二手掉包工程师 <rustin.liu@gmail.com>
Co-authored-by: TONG, Zhigao <tongzhigao@pingcap.com>
Co-authored-by: YangKeao <yangkeao@chunibyo.icu>
Co-authored-by: CbcWestwolf <1004626265@qq.com>
Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io>