*: enhance the table regions HTTP API by tiancaiamao · Pull Request #12357 · pingcap/tidb

tiancaiamao · 2019-09-24T09:32:07Z

What problem does this PR solve?

This API is used to get the table regions http://localhost:10080/tables/{DB}/{Table}/regions

In current implementation, getRegionsMeta is used to get the region information.
That function first gets the regions in the key range, and then call GetRegionByID one by one for each region id.

The problem is that when the table is huge, it contains so many regions, calling the API is very slow because current implementation visits PD for each region.

What is changed and how it works?

Use ScanRegions API to implement http://localhost:10080/tables/{DB}/{Table}/regions
ScanRegions accepts a key range and returns the region information, we need just one call to get the result instead of calling GetRegionByID for every region.

Check List

Tests

Manual test (add detailed scripts or steps below)

I test it manually on TiKV.

CREATE TABLE IF NOT EXISTS TEST_HOTSPOT(
      id                   BIGINT PRIMARY KEY,
      age                INT,
      user_name  VARCHAR(32),
      email 	 VARCHAR(128)
)

Split the table to get more regions:

SPLIT TABLE TEST_HOTSPOT BETWEEN (0) AND (9223372036854775807) REGIONS 128;

See the table regions:

mysql> show table TEST_HOTSPOT regions;
+-----------+----------------------------+----------------------------+-----------+-----------------+-------+------------+
| REGION_ID | START_KEY                  | END_KEY                    | LEADER_ID | LEADER_STORE_ID | PEERS | SCATTERING |
+-----------+----------------------------+----------------------------+-----------+-----------------+-------+------------+
|        48 | t_47_                      | t_47_r_72057594037927935   |        49 |               1 | 49    |          0 |
|        50 | t_47_r_72057594037927935   | t_47_r_144115188075855870  |        51 |               1 | 51    |          0 |
|        52 | t_47_r_144115188075855870  | t_47_r_216172782113783805  |        53 |               1 | 53    |          0 |
|        54 | t_47_r_216172782113783805  | t_47_r_288230376151711740  |        55 |               1 | 55    |          0 |
|        56 | t_47_r_288230376151711740  | t_47_r_360287970189639675  |        57 |               1 | 57    |          0 |
|        58 | t_47_r_360287970189639675  | t_47_r_432345564227567610  |        59 |               1 | 59    |          0 |
|        60 | t_47_r_432345564227567610  | t_47_r_504403158265495545  |        61 |               1 | 61    |          0 |
|        62 | t_47_r_504403158265495545  | t_47_r_576460752303423480  |        63 |               1 | 63    |          0 |
|        64 | t_47_r_576460752303423480  | t_47_r_648518346341351415  |        65 |               1 | 65    |          0 |
|        66 | t_47_r_648518346341351415  | t_47_r_720575940379279350  |        67 |               1 | 67    |          0 |
|        68 | t_47_r_720575940379279350  | t_47_r_792633534417207285  |        69 |               1 | 69    |          0 |
|        70 | t_47_r_792633534417207285  | t_47_r_864691128455135220  |        71 |               1 | 71    |          0 |
|        72 | t_47_r_864691128455135220  | t_47_r_936748722493063155  |        73 |               1 | 73    |          0 |
|        74 | t_47_r_936748722493063155  | t_47_r_1008806316530991090 |        75 |               1 | 75    |          0 |
|        76 | t_47_r_1008806316530991090 | t_47_r_1080863910568919025 |        77 |               1 | 77    |          0 |
|         2 | t_47_r_1080863910568919025 |                            |         3 |               1 | 3     |          0 |
+-----------+----------------------------+----------------------------+-----------+-----------------+-------+------------+
16 rows in set (0.03 sec)

Compare it with the result from the HTTP API:

{
 "name": "TEST_HOTSPOT",
 "id": 47,
 "record_regions": [
  {
   "region_id": 48,
   "leader": {
    "id": 49,
    "store_id": 1
   },
   "peers": [
    {
     "id": 49,
     "store_id": 1
    }
   ],
   "region_epoch": {
    "conf_ver": 1,
    "version": 24
   }
  },
  {
   "region_id": 50,
   "leader": {
    "id": 51,
    "store_id": 1
   },
   "peers": [
    {
     "id": 51,
     "store_id": 1
    }
   ],
   "region_epoch": {
    "conf_ver": 1,
    "version": 25
   }
  },
  {
   "region_id": 52,
   "leader": {
    "id": 53,
    "store_id": 1
   },
   "peers": [
    {
     "id": 53,
     "store_id": 1
    }
   ],
   "region_epoch": {
    "conf_ver": 1,
    "version": 26
   }
  },
  {
   "region_id": 54,
   "leader": {
    "id": 55,
    "store_id": 1
   },
   "peers": [
    {
     "id": 55,
     "store_id": 1
    }
   ],
   "region_epoch": {
    "conf_ver": 1,
    "version": 27
   }
  },
  {
   "region_id": 56,
   "leader": {
    "id": 57,
    "store_id": 1
   },
   "peers": [
    {
     "id": 57,
     "store_id": 1
    }
   ],
   "region_epoch": {
    "conf_ver": 1,
    "version": 28
   }
  },
  {
   "region_id": 58,
   "leader": {
    "id": 59,
    "store_id": 1
   },
   "peers": [
    {
     "id": 59,
     "store_id": 1
    }
   ],
   "region_epoch": {
    "conf_ver": 1,
    "version": 29
   }
  },
  {
   "region_id": 60,
   "leader": {
    "id": 61,
    "store_id": 1
   },
   "peers": [
    {
     "id": 61,
     "store_id": 1
    }
   ],
   "region_epoch": {
    "conf_ver": 1,
    "version": 30
   }
  },
  {
   "region_id": 62,
   "leader": {
    "id": 63,
    "store_id": 1
   },
   "peers": [
    {
     "id": 63,
     "store_id": 1
    }
   ],
   "region_epoch": {
    "conf_ver": 1,
    "version": 31
   }
  },
  {
   "region_id": 64,
   "leader": {
    "id": 65,
    "store_id": 1
   },
   "peers": [
    {
     "id": 65,
     "store_id": 1
    }
   ],
   "region_epoch": {
    "conf_ver": 1,
    "version": 32
   }
  },
  {
   "region_id": 66,
   "leader": {
    "id": 67,
    "store_id": 1
   },
   "peers": [
    {
     "id": 67,
     "store_id": 1
    }
   ],
   "region_epoch": {
    "conf_ver": 1,
    "version": 33
   }
  },
  {
   "region_id": 68,
   "leader": {
    "id": 69,
    "store_id": 1
   },
   "peers": [
    {
     "id": 69,
     "store_id": 1
    }
   ],
   "region_epoch": {
    "conf_ver": 1,
    "version": 34
   }
  },
  {
   "region_id": 70,
   "leader": {
    "id": 71,
    "store_id": 1
   },
   "peers": [
    {
     "id": 71,
     "store_id": 1
    }
   ],
   "region_epoch": {
    "conf_ver": 1,
    "version": 35
   }
  },
  {
   "region_id": 72,
   "leader": {
    "id": 73,
    "store_id": 1
   },
   "peers": [
    {
     "id": 73,
     "store_id": 1
    }
   ],
   "region_epoch": {
    "conf_ver": 1,
    "version": 36
   }
  },
  {
   "region_id": 74,
   "leader": {
    "id": 75,
    "store_id": 1
   },
   "peers": [
    {
     "id": 75,
     "store_id": 1
    }
   ],
   "region_epoch": {
    "conf_ver": 1,
    "version": 37
   }
  },
  {
   "region_id": 76,
   "leader": {
    "id": 77,
    "store_id": 1
   },
   "peers": [
    {
     "id": 77,
     "store_id": 1
    }
   ],
   "region_epoch": {
    "conf_ver": 1,
    "version": 38
   }
  },
  {
   "region_id": 2,
   "leader": {
    "id": 3,
    "store_id": 1
   },
   "peers": [
    {
     "id": 3,
     "store_id": 1
    }
   ],
   "region_epoch": {
    "conf_ver": 1,
    "version": 38
   }
  }
 ],
 "indices": [
  {
   "name": "i_email",
   "id": 1,
   "regions": [
    {
     "region_id": 48,
     "leader": {
      "id": 49,
      "store_id": 1
     },
     "peers": [
      {
       "id": 49,
       "store_id": 1
      }
     ],
     "region_epoch": {
      "conf_ver": 1,
      "version": 24
     }
    }
   ]
  }
 ]
}

disksing · 2019-09-24T09:45:24Z

go.sum

-github.com/pingcap/pd v0.0.0-20190712044914-75a1f9f3062b h1:oS9PftxQqgcRouKhhdaB52tXhVLEP7Ng3Qqsd6Z18iY=
-github.com/pingcap/pd v0.0.0-20190712044914-75a1f9f3062b/go.mod h1:3DlDlFT7EF64A1bmb/tulZb6wbPSagm5G4p1AlhaEDs=
+github.com/pingcap/pd v1.1.0-beta.0.20190923032047-5c648dc365e0 h1:GIEq+wZfrl2bcJxpuSrEH4H7/nlf5YdmpS+dU9lNIt8=
+github.com/pingcap/pd v1.1.0-beta.0.20190923032047-5c648dc365e0/go.mod h1:G/6rJpnYwM0LKMec2rI82/5Kg6GaZMvlfB+e6/tvYmI=


weird, any idea why it changes to v1.1.0?

I use the latest pd in go.mod
https://github.com/pingcap/tidb/pull/12357/files#diff-37aff102a57d3d7b797f152915a6dc16R44

server/http_handler.go

store/tikv/store_test.go

codecov · 2019-09-25T06:39:31Z

Codecov Report

Merging #12357 into master will increase coverage by 0.3579%.
The diff coverage is n/a.

@@               Coverage Diff                @@
##             master     #12357        +/-   ##
================================================
+ Coverage   79.7501%   80.1081%   +0.3579%     
================================================
  Files           462        462                
  Lines        101917     103962      +2045     
================================================
+ Hits          81279      83282      +2003     
- Misses        14773      14818        +45     
+ Partials       5865       5862         -3

store/mockstore/mocktikv/cluster.go

tiancaiamao · 2019-09-25T07:07:15Z

/run-all-tests

disksing

LGTM

tiancaiamao · 2019-09-25T07:40:58Z

/run-all-tests

tiancaiamao · 2019-09-25T08:03:16Z

PTAL @lysu @jackysp

tiancaiamao · 2019-09-26T03:56:45Z

PTAL @lysu @jackysp

lysu

LGTM

sre-bot · 2019-09-26T06:41:59Z

Your auto merge job has been accepted, waiting for 12348

sre-bot · 2019-09-26T06:48:33Z

/run-all-tests

sre-bot · 2019-09-26T07:05:01Z

@tiancaiamao merge failed.

tiancaiamao · 2019-09-26T12:59:43Z

/run-integration-common-test

lysu · 2019-09-26T13:06:12Z

/merge

sre-bot · 2019-09-26T13:08:33Z

/run-all-tests

tiancaiamao added 3 commits September 24, 2019 16:42

*: fix the table regions HTTP API

5dc0173

Merge branch 'master' into fix-table-regions

8492f9f

Merge branch 'master' into fix-table-regions

4f12c94

tiancaiamao added type/enhancement The issue or PR belongs to an enhancement. component/server labels Sep 24, 2019

disksing reviewed Sep 24, 2019

View reviewed changes

address comment

e35df15

disksing reviewed Sep 25, 2019

View reviewed changes

store/mockstore/mocktikv/cluster.go Outdated Show resolved Hide resolved

tiancaiamao added 2 commits September 25, 2019 15:00

address comment

67deccb

Merge branch 'master' into fix-table-regions

471a412

disksing approved these changes Sep 25, 2019

View reviewed changes

Merge branch 'master' into fix-table-regions

2fdca2a

tiancaiamao added the status/LGT1 Indicates that a PR has LGTM 1. label Sep 25, 2019

tiancaiamao added the status/all tests passed label Sep 25, 2019

lysu reviewed Sep 26, 2019

View reviewed changes

lysu added status/can-merge Indicates a PR has been approved by a committer. and removed status/LGT1 Indicates that a PR has LGTM 1. labels Sep 26, 2019

lysu added the status/LGT2 Indicates that a PR has LGTM 2. label Sep 26, 2019

Merge branch 'master' into fix-table-regions

9164765

Merge branch 'master' into fix-table-regions

8ef5685

sre-bot merged commit db1eb1c into pingcap:master Sep 26, 2019

tiancaiamao deleted the fix-table-regions branch September 26, 2019 13:58

tiancaiamao added a commit to tiancaiamao/tidb that referenced this pull request Nov 25, 2019

*: enhance the table regions HTTP API (pingcap#12357)

5549df6

tiancaiamao mentioned this pull request Nov 25, 2019

*: enhance the table regions HTTP API (#12357) #13725

Merged

sre-bot pushed a commit that referenced this pull request Nov 25, 2019

*: enhance the table regions HTTP API (#12357) (#13725)

4546a1f

Conversation

tiancaiamao commented Sep 24, 2019

What problem does this PR solve?

What is changed and how it works?

Check List

Uh oh!

disksing Sep 24, 2019

Choose a reason for hiding this comment

Uh oh!

tiancaiamao Sep 25, 2019

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

codecov bot commented Sep 25, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Uh oh!

tiancaiamao commented Sep 25, 2019

Uh oh!

disksing left a comment

Choose a reason for hiding this comment

Uh oh!

tiancaiamao commented Sep 25, 2019

Uh oh!

tiancaiamao commented Sep 25, 2019

Uh oh!

tiancaiamao commented Sep 26, 2019

Uh oh!

lysu left a comment

Choose a reason for hiding this comment

Uh oh!

sre-bot commented Sep 26, 2019

Uh oh!

sre-bot commented Sep 26, 2019

Uh oh!

sre-bot commented Sep 26, 2019

Uh oh!

tiancaiamao commented Sep 26, 2019

Uh oh!

lysu commented Sep 26, 2019

Uh oh!

sre-bot commented Sep 26, 2019

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

codecov bot commented Sep 25, 2019 •

edited

Loading