parser: cleanup parser/scanner error.

…ingcap#145)

* *: parse the data source directly into data and skip the KV encoder

This skips the more complex pingcap/parser, and speeds up parsing speed
by 50%.

We have also refactored the KV delivery mechanism to use channels
directly, and revamped metrics:

- Make the metrics about engines into its own `engines` counter. The
  `tables` counter is exclusively about tables now.
- Removed `block_read_seconds`, `block_read_bytes`, `block_encode_seconds`
  since the concept of "block" no longer applies. Replaced by the
  equivalents named `row_***`.
- Removed `chunk_parser_read_row_seconds` for being overlapping with
  `row_read_seconds`.
- Changed `block_deliver_bytes` into a histogram vec, with kind=index
  or kind=data. Introduced `block_deliver_kv_pairs`.

* tests,restore: prevent spurious error in checkpoint_chunks test

Only kill Lightning if the whole chunk is imported exactly. The chunk
checkpoint may be recorded before a chunk is fully written, and this will
hit the failpoint more than 5 times.

* kv: use composed interface to simplify some types

* kv: properly handle the SQL mode

* common: disable IsContextCanceledError() when log level = debug

This helps debugging some mysterious cancellation where the log is
inhibited.

Added IsReallyContextCanceledError() for code logic affected by error
type.

* restore: made some log more detailed

* restore: made the SlowDownImport failpoint apply to index engines too

* restore: do not open a write stream when there are no KV pairs to send

* tests: ensure we drop the checkpoints DB before re-run

* mydump: fixed various off-by-one errors in the CSV parser

* *: rename `!IsContextCanceledError` to `ShouldLogError`

* *: addressed comments

* restore: zero the checksums and column permutations on initialization

* *: addressed comments

* tests: add back a missing license header

* tests: improve a comment.

…gcap#158 (pingcap#159)

* tests: fix a test failure due to conflict between pingcap#145 and pingcap#158

* restore: apply the row count limit to failpoint KillIfImportedChunk too

…ase 6.4` rate limit (pingcap#155)

* merge pr#130,pingcap#133 into 6.4

Signed-off-by: zeminzhou <zhouzemin@pingcap.com>

* fix start limit rate

Signed-off-by: zeminzhou <zhouzemin@pingcap.com>

* default config

Signed-off-by: zeminzhou <zhouzemin@pingcap.com>

* keyspace: use `user_storage_size` instead of `storage_size` for ratelimit (pingcap#145)

* use user_storage_size instead of storage_size for ratelimit

Signed-off-by: zeminzhou <zhouzemin@pingcap.com>

* update golang.org/x/text to 0.3.8

Signed-off-by: zeminzhou <zhouzemin@pingcap.com>

* Fix error handle when loadKeyspace failed at startup (pingcap#143)

Signed-off-by: yongman <yming0221@gmail.com>

Signed-off-by: yongman <yming0221@gmail.com>

* add keyspace-activate mode (pingcap#142)

* fix loadkeyspace nil (pingcap#144)

* fix loadkeyspace nil

Signed-off-by: ystaticy <y_static_y@sina.com>

* retry load keyspace

Signed-off-by: David <8039876+AmoebaProtozoa@users.noreply.github.com>

* address comments

Signed-off-by: David <8039876+AmoebaProtozoa@users.noreply.github.com>

Signed-off-by: ystaticy <y_static_y@sina.com>
Signed-off-by: David <8039876+AmoebaProtozoa@users.noreply.github.com>
Co-authored-by: David <8039876+AmoebaProtozoa@users.noreply.github.com>

* Compatible with old pd

Signed-off-by: zeminzhou <zhouzemin@pingcap.com>

Signed-off-by: zeminzhou <zhouzemin@pingcap.com>
Signed-off-by: yongman <yming0221@gmail.com>
Signed-off-by: ystaticy <y_static_y@sina.com>
Signed-off-by: David <8039876+AmoebaProtozoa@users.noreply.github.com>
Co-authored-by: yongman <yming0221@gmail.com>
Co-authored-by: better0332 <better0332@163.com>
Co-authored-by: ystaticy <y_static_y@sina.com>
Co-authored-by: David <8039876+AmoebaProtozoa@users.noreply.github.com>

* make check

Signed-off-by: zeminzhou <zhouzemin@pingcap.com>

* fix confict

Signed-off-by: zeminzhou <zhouzemin@pingcap.com>

* make check

Signed-off-by: zeminzhou <zhouzemin@pingcap.com>

Signed-off-by: zeminzhou <zhouzemin@pingcap.com>
Signed-off-by: yongman <yming0221@gmail.com>
Signed-off-by: ystaticy <y_static_y@sina.com>
Signed-off-by: David <8039876+AmoebaProtozoa@users.noreply.github.com>
Co-authored-by: yongman <yming0221@gmail.com>
Co-authored-by: better0332 <better0332@163.com>
Co-authored-by: ystaticy <y_static_y@sina.com>
Co-authored-by: David <8039876+AmoebaProtozoa@users.noreply.github.com>

Reviewed-on: https://git.pingcap.net/pingkai/tidb/pulls/145
Reviewed-by: zhaoyilin <zhaoyilin@pingcap.cn>