Skip to content

PageStorage: Ref page lifetime mechanism#4174

Merged
ti-chi-bot merged 20 commits intopingcap:masterfrom
JaySon-Huang:ref_page_life_matters
Mar 8, 2022
Merged

PageStorage: Ref page lifetime mechanism#4174
ti-chi-bot merged 20 commits intopingcap:masterfrom
JaySon-Huang:ref_page_life_matters

Conversation

@JaySon-Huang
Copy link
Contributor

@JaySon-Huang JaySon-Huang commented Mar 3, 2022

What problem does this PR solve?

Issue Number: close #4017, ref #3594

Problem Summary:

What is changed and how it works?

Introduce VarEntry to manage the lifetime of PageEntries/ExternalPages in PageDirectory with RefPage mechanism

Check List

Tests

  • Unit test
  • Integration test
  • Manual test (add detailed scripts or steps below)
  • No code

Side effects

  • Performance regression: Consumes more CPU
  • Performance regression: Consumes more Memory
  • Breaking backward compatibility

Documentation

  • Affects user behaviors
  • Contains syntax changes
  • Contains variable changes
  • Contains experimental features
  • Changes MySQL compatibility

Release note

None

@ti-chi-bot
Copy link
Member

ti-chi-bot commented Mar 3, 2022

[REVIEW NOTIFICATION]

This pull request has been approved by:

  • jiaqizho
  • lidezhu

To complete the pull request process, please ask the reviewers in the list to review by filling /cc @reviewer in the comment.
After your PR has acquired the required number of LGTMs, you can assign this pull request to the committer in the list by filling /assign @committer in the comment to help you merge this pull request.

The full list of commands accepted by this bot can be found here.

Details

Reviewer can indicate their review by submitting an approval review.
Reviewer can cancel approval by submitting a request changes review.

@ti-chi-bot ti-chi-bot added do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. release-note-none Denotes a PR that doesn't merit a release note. size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. labels Mar 3, 2022
@JaySon-Huang JaySon-Huang force-pushed the ref_page_life_matters branch from 07e21aa to cb4b1ca Compare March 3, 2022 07:23
@JaySon-Huang
Copy link
Contributor Author

/run-unit-test

1 similar comment
@JaySon-Huang
Copy link
Contributor Author

/run-unit-test

@sre-bot
Copy link
Collaborator

sre-bot commented Mar 3, 2022

Coverage for changed files

Filename                                   Regions    Missed Regions     Cover   Functions  Missed Functions  Executed       Lines      Missed Lines     Cover    Branches   Missed Branches     Cover
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
DeltaMerge/DeltaMergeStore.cpp                1278               435    65.96%          65                 5    92.31%        1858               413    77.77%         746               351    52.95%
Page/PageStorage.h                              16                 8    50.00%          16                 8    50.00%          53                20    62.26%           0                 0         -
Page/V2/PageStorage.cpp                        518               146    71.81%          36                 3    91.67%         880               163    81.48%         350               144    58.86%
Page/V2/PageStorage.h                           41                 9    78.05%          16                 3    81.25%          44                 3    93.18%          20                 9    55.00%
Page/V2/tests/gtest_page_storage.cpp          4855               980    79.81%          32                 2    93.75%        1044                 5    99.52%        1498               712    52.47%
Page/V3/BlobStore.cpp                          344               121    64.83%          39                 6    84.62%         722               222    69.25%         214                89    58.41%
Page/V3/BlobStore.h                              7                 2    71.43%           7                 2    71.43%          25                11    56.00%           0                 0         -
Page/V3/MapUtils.h                              12                 1    91.67%           3                 0   100.00%          18                 1    94.44%           6                 1    83.33%
Page/V3/PageDirectory.cpp                      339               100    70.50%          28                 5    82.14%         849               239    71.85%         276                93    66.30%
Page/V3/PageDirectory.h                         17                 5    70.59%          13                 5    61.54%          49                20    59.18%           6                 0   100.00%
Page/V3/PageDirectoryFactory.cpp                31                26    16.13%           2                 0   100.00%          74                61    17.57%          38                35     7.89%
Page/V3/PageDirectoryFactory.h                   1                 0   100.00%           1                 0   100.00%           4                 0   100.00%           0                 0         -
Page/V3/PageEntriesEdit.h                       48                 8    83.33%          36                 6    83.33%         161                32    80.12%          12                 6    50.00%
Page/V3/PageStorageImpl.cpp                     67                22    67.16%          18                 7    61.11%         163                63    61.35%          38                21    44.74%
Page/V3/PageStorageImpl.h                        8                 4    50.00%           8                 4    50.00%           8                 4    50.00%           0                 0         -
Page/V3/WAL/WALReader.cpp                       66                16    75.76%           9                 0   100.00%         166                28    83.13%          50                18    64.00%
Page/V3/WAL/WALReader.h                         12                 3    75.00%           4                 1    75.00%          19                 7    63.16%           6                 2    66.67%
Page/V3/WAL/serialize.cpp                       47                11    76.60%          15                 1    93.33%         190                41    78.42%          52                17    67.31%
Page/V3/WALStore.cpp                            63                37    41.27%           8                 1    87.50%         103                32    68.93%          28                17    39.29%
Page/V3/WALStore.h                               1                 0   100.00%           1                 0   100.00%           3                 0   100.00%           0                 0         -
Page/V3/tests/entries_helper.h                  65                36    44.62%          10                 3    70.00%         237               111    53.16%          32                21    34.38%
Page/V3/tests/gtest_blob_store.cpp            3478               559    83.93%          16                 0   100.00%         650                 0   100.00%        1080               508    52.96%
Page/V3/tests/gtest_page_directory.cpp        6397              1167    81.76%          36                 0   100.00%        1084                 6    99.45%        1960               990    49.49%
Page/V3/tests/gtest_page_storage.cpp          2452               921    62.44%          27                 8    70.37%         631               161    74.48%         786               474    39.69%
Page/V3/tests/gtest_wal_store.cpp             1764               244    86.17%          10                 0   100.00%         408                12    97.06%         582               284    51.20%
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
TOTAL                                        21927              4861    77.83%         456                70    84.65%        9443              1655    82.47%        7780              3792    51.26%

Coverage summary

Functions  MissedFunctions  Executed  Lines   MissedLines  Cover
16728      9463             43.43%    187654  95901        48.89%

full coverage report (for internal network access only)

@JaySon-Huang JaySon-Huang force-pushed the ref_page_life_matters branch from dd2c101 to 4b68bdc Compare March 4, 2022 02:30
@JaySon-Huang
Copy link
Contributor Author

/run-unit-test

@sre-bot
Copy link
Collaborator

sre-bot commented Mar 4, 2022

Coverage for changed files

Filename                                   Regions    Missed Regions     Cover   Functions  Missed Functions  Executed       Lines      Missed Lines     Cover    Branches   Missed Branches     Cover
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
DeltaMerge/DeltaMergeStore.cpp                1278               435    65.96%          65                 5    92.31%        1858               413    77.77%         746               351    52.95%
Page/PageStorage.h                              16                 8    50.00%          16                 8    50.00%          53                20    62.26%           0                 0         -
Page/V2/PageStorage.cpp                        518               146    71.81%          36                 3    91.67%         880               163    81.48%         350               144    58.86%
Page/V2/PageStorage.h                           41                 9    78.05%          16                 3    81.25%          44                 3    93.18%          20                 9    55.00%
Page/V2/tests/gtest_page_storage.cpp          4855               980    79.81%          32                 2    93.75%        1044                 5    99.52%        1498               712    52.47%
Page/V3/BlobStore.cpp                          344               121    64.83%          39                 6    84.62%         722               222    69.25%         214                89    58.41%
Page/V3/BlobStore.h                              7                 2    71.43%           7                 2    71.43%          25                11    56.00%           0                 0         -
Page/V3/MapUtils.h                              12                 1    91.67%           3                 0   100.00%          18                 1    94.44%           6                 1    83.33%
Page/V3/PageDirectory.cpp                      449               143    68.15%          29                 6    79.31%         963               319    66.87%         376               144    61.70%
Page/V3/PageDirectory.h                         12                 4    66.67%          12                 4    66.67%          38                15    60.53%           0                 0         -
Page/V3/PageDirectoryFactory.cpp                31                26    16.13%           2                 0   100.00%          74                61    17.57%          38                35     7.89%
Page/V3/PageDirectoryFactory.h                   1                 0   100.00%           1                 0   100.00%           4                 0   100.00%           0                 0         -
Page/V3/PageEntriesEdit.h                       47                 7    85.11%          32                 5    84.38%         137                26    81.02%          14                 6    57.14%
Page/V3/PageStorageImpl.cpp                     67                22    67.16%          18                 7    61.11%         162                63    61.11%          38                21    44.74%
Page/V3/PageStorageImpl.h                        8                 4    50.00%           8                 4    50.00%           8                 4    50.00%           0                 0         -
Page/V3/WAL/WALReader.cpp                       66                16    75.76%           9                 0   100.00%         166                28    83.13%          50                18    64.00%
Page/V3/WAL/WALReader.h                         12                 3    75.00%           4                 1    75.00%          19                 7    63.16%           6                 2    66.67%
Page/V3/WAL/serialize.cpp                       47                11    76.60%          15                 1    93.33%         190                41    78.42%          52                17    67.31%
Page/V3/WALStore.cpp                            63                37    41.27%           8                 1    87.50%         103                32    68.93%          28                17    39.29%
Page/V3/WALStore.h                               1                 0   100.00%           1                 0   100.00%           3                 0   100.00%           0                 0         -
Page/V3/tests/entries_helper.h                  65                36    44.62%          10                 3    70.00%         237               111    53.16%          32                21    34.38%
Page/V3/tests/gtest_blob_store.cpp            3478               559    83.93%          16                 0   100.00%         650                 0   100.00%        1080               508    52.96%
Page/V3/tests/gtest_page_directory.cpp        6397              1167    81.76%          36                 0   100.00%        1084                 6    99.45%        1960               990    49.49%
Page/V3/tests/gtest_page_storage.cpp          2571               934    63.67%          30                 8    73.33%         666               161    75.83%         826               494    40.19%
Page/V3/tests/gtest_wal_store.cpp             1764               244    86.17%          10                 0   100.00%         408                12    97.06%         582               284    51.20%
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
TOTAL                                        22150              4915    77.81%         455                69    84.84%        9556              1724    81.96%        7916              3863    51.20%

Coverage summary

Functions  MissedFunctions  Executed  Lines   MissedLines  Cover
16727      9462             43.43%    187767  95990        48.88%

full coverage report (for internal network access only)

@JaySon-Huang JaySon-Huang changed the title [WIP] PageStorage: Ref page lifetime mechanism PageStorage: Ref page lifetime mechanism Mar 4, 2022
@ti-chi-bot ti-chi-bot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Mar 4, 2022
@JaySon-Huang JaySon-Huang requested review from jiaqizho and lidezhu March 4, 2022 03:16
@JaySon-Huang JaySon-Huang self-assigned this Mar 4, 2022
@JaySon-Huang
Copy link
Contributor Author

/run-all-tests

@sre-bot
Copy link
Collaborator

sre-bot commented Mar 4, 2022

Coverage for changed files

Filename                                   Regions    Missed Regions     Cover   Functions  Missed Functions  Executed       Lines      Missed Lines     Cover    Branches   Missed Branches     Cover
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
DeltaMerge/DeltaMergeStore.cpp                1278               435    65.96%          65                 5    92.31%        1858               413    77.77%         746               351    52.95%
Page/PageStorage.h                              16                 8    50.00%          16                 8    50.00%          53                20    62.26%           0                 0         -
Page/V2/PageStorage.cpp                        518               146    71.81%          36                 3    91.67%         880               163    81.48%         350               144    58.86%
Page/V2/PageStorage.h                           41                 9    78.05%          16                 3    81.25%          44                 3    93.18%          20                 9    55.00%
Page/V2/tests/gtest_page_storage.cpp          4855               980    79.81%          32                 2    93.75%        1044                 5    99.52%        1498               712    52.47%
Page/V3/BlobStore.cpp                          344               121    64.83%          39                 6    84.62%         722               222    69.25%         214                89    58.41%
Page/V3/BlobStore.h                              7                 2    71.43%           7                 2    71.43%          25                11    56.00%           0                 0         -
Page/V3/MapUtils.h                              12                 1    91.67%           3                 0   100.00%          18                 1    94.44%           6                 1    83.33%
Page/V3/PageDirectory.cpp                      449               143    68.15%          29                 6    79.31%         963               319    66.87%         376               144    61.70%
Page/V3/PageDirectory.h                         12                 4    66.67%          12                 4    66.67%          38                15    60.53%           0                 0         -
Page/V3/PageDirectoryFactory.cpp                31                26    16.13%           2                 0   100.00%          74                61    17.57%          38                35     7.89%
Page/V3/PageDirectoryFactory.h                   1                 0   100.00%           1                 0   100.00%           4                 0   100.00%           0                 0         -
Page/V3/PageEntriesEdit.h                       47                 7    85.11%          32                 5    84.38%         137                26    81.02%          14                 6    57.14%
Page/V3/PageStorageImpl.cpp                     67                22    67.16%          18                 7    61.11%         162                63    61.11%          38                21    44.74%
Page/V3/PageStorageImpl.h                        8                 4    50.00%           8                 4    50.00%           8                 4    50.00%           0                 0         -
Page/V3/WAL/WALReader.cpp                       66                16    75.76%           9                 0   100.00%         166                28    83.13%          50                18    64.00%
Page/V3/WAL/WALReader.h                         12                 3    75.00%           4                 1    75.00%          19                 7    63.16%           6                 2    66.67%
Page/V3/WAL/serialize.cpp                       47                11    76.60%          15                 1    93.33%         190                41    78.42%          52                17    67.31%
Page/V3/WALStore.cpp                            63                37    41.27%           8                 1    87.50%         103                32    68.93%          28                17    39.29%
Page/V3/WALStore.h                               1                 0   100.00%           1                 0   100.00%           3                 0   100.00%           0                 0         -
Page/V3/tests/entries_helper.h                  65                36    44.62%          10                 3    70.00%         237               111    53.16%          32                21    34.38%
Page/V3/tests/gtest_blob_store.cpp            3478               559    83.93%          16                 0   100.00%         650                 0   100.00%        1080               508    52.96%
Page/V3/tests/gtest_page_directory.cpp        6397              1167    81.76%          36                 0   100.00%        1084                 6    99.45%        1960               990    49.49%
Page/V3/tests/gtest_page_storage.cpp          2571               934    63.67%          30                 8    73.33%         666               161    75.83%         826               494    40.19%
Page/V3/tests/gtest_wal_store.cpp             1764               244    86.17%          10                 0   100.00%         408                12    97.06%         582               284    51.20%
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
TOTAL                                        22150              4915    77.81%         455                69    84.84%        9556              1724    81.96%        7916              3863    51.20%

Coverage summary

Functions  MissedFunctions  Executed  Lines   MissedLines  Cover
16727      9462             43.43%    187767  95972        48.89%

full coverage report (for internal network access only)

@JaySon-Huang JaySon-Huang force-pushed the ref_page_life_matters branch from 38a19eb to ce8448f Compare March 7, 2022 05:37
@JaySon-Huang
Copy link
Contributor Author

/run-all-tests

Copy link
Contributor

@jiaqizho jiaqizho left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@ti-chi-bot ti-chi-bot added the status/LGT1 Indicates that a PR has LGTM 1. label Mar 7, 2022
@JaySon-Huang
Copy link
Contributor Author

/run-all-tests

@sre-bot
Copy link
Collaborator

sre-bot commented Mar 7, 2022

Coverage for changed files

Filename                                   Regions    Missed Regions     Cover   Functions  Missed Functions  Executed       Lines      Missed Lines     Cover    Branches   Missed Branches     Cover
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
DeltaMerge/DeltaMergeStore.cpp                1317               476    63.86%          65                 5    92.31%        1881               430    77.14%         766               371    51.57%
Page/PageStorage.h                              16                 8    50.00%          16                 8    50.00%          53                20    62.26%           0                 0         -
Page/V2/PageStorage.cpp                        518               146    71.81%          36                 3    91.67%         880               163    81.48%         350               144    58.86%
Page/V2/PageStorage.h                           41                 9    78.05%          16                 3    81.25%          44                 3    93.18%          20                 9    55.00%
Page/V2/tests/gtest_page_storage.cpp          4855               980    79.81%          32                 2    93.75%        1044                 5    99.52%        1498               712    52.47%
Page/V3/BlobStore.cpp                          353               108    69.41%          39                 3    92.31%         764               201    73.69%         222                84    62.16%
Page/V3/BlobStore.h                              7                 1    85.71%           7                 1    85.71%          25                10    60.00%           0                 0         -
Page/V3/MapUtils.h                              12                 1    91.67%           3                 0   100.00%          18                 1    94.44%           6                 1    83.33%
Page/V3/PageDirectory.cpp                      437               134    69.34%          30                 6    80.00%         965               312    67.67%         364               132    63.74%
Page/V3/PageDirectory.h                         20                 5    75.00%          20                 5    75.00%          89                22    75.28%           0                 0         -
Page/V3/PageDirectoryFactory.cpp                31                26    16.13%           2                 0   100.00%          74                61    17.57%          38                35     7.89%
Page/V3/PageDirectoryFactory.h                   1                 0   100.00%           1                 0   100.00%           4                 0   100.00%           0                 0         -
Page/V3/PageEntriesEdit.h                       43                 9    79.07%          28                 7    75.00%         118                41    65.25%          14                 6    57.14%
Page/V3/PageStorageImpl.cpp                     67                22    67.16%          18                 7    61.11%         162                63    61.11%          36                21    41.67%
Page/V3/PageStorageImpl.h                        8                 4    50.00%           8                 4    50.00%           8                 4    50.00%           0                 0         -
Page/V3/WAL/WALReader.cpp                       66                16    75.76%           9                 0   100.00%         166                28    83.13%          50                18    64.00%
Page/V3/WAL/WALReader.h                         12                 3    75.00%           4                 1    75.00%          19                 7    63.16%           6                 2    66.67%
Page/V3/WAL/serialize.cpp                       47                11    76.60%          15                 1    93.33%         190                41    78.42%          52                17    67.31%
Page/V3/WALStore.cpp                            63                37    41.27%           8                 1    87.50%         103                32    68.93%          28                17    39.29%
Page/V3/WALStore.h                               1                 0   100.00%           1                 0   100.00%           3                 0   100.00%           0                 0         -
Page/V3/tests/entries_helper.h                  65                36    44.62%          10                 3    70.00%         237               111    53.16%          32                21    34.38%
Page/V3/tests/gtest_blob_store.cpp            3549               584    83.54%          17                 0   100.00%         688                 0   100.00%        1102               518    52.99%
Page/V3/tests/gtest_page_directory.cpp        6619              1206    81.78%          36                 0   100.00%        1098                 6    99.45%        2038              1029    49.51%
Page/V3/tests/gtest_page_storage.cpp          2571               934    63.67%          30                 8    73.33%         666               161    75.83%         826               494    40.19%
Page/V3/tests/gtest_wal_store.cpp             1764               244    86.17%          10                 0   100.00%         408                12    97.06%         582               284    51.20%
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
TOTAL                                        22483              5000    77.76%         461                68    85.25%        9707              1734    82.14%        8030              3915    51.25%

Coverage summary

Functions  MissedFunctions  Executed  Lines   MissedLines  Cover
16741      9450             43.55%    188265  95897        49.06%

full coverage report (for internal network access only)

@JaySon-Huang
Copy link
Contributor Author

/run-all-tests

@sre-bot
Copy link
Collaborator

sre-bot commented Mar 7, 2022

Coverage for changed files

Filename                                   Regions    Missed Regions     Cover   Functions  Missed Functions  Executed       Lines      Missed Lines     Cover    Branches   Missed Branches     Cover
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
DeltaMerge/DeltaMergeStore.cpp                1317               466    64.62%          65                 5    92.31%        1881               426    77.35%         766               367    52.09%
Page/PageStorage.h                              16                 8    50.00%          16                 8    50.00%          53                20    62.26%           0                 0         -
Page/V2/PageStorage.cpp                        518               146    71.81%          36                 3    91.67%         880               163    81.48%         350               144    58.86%
Page/V2/PageStorage.h                           41                 9    78.05%          16                 3    81.25%          44                 3    93.18%          20                 9    55.00%
Page/V2/tests/gtest_page_storage.cpp          4855               980    79.81%          32                 2    93.75%        1044                 5    99.52%        1498               712    52.47%
Page/V3/BlobStore.cpp                          353               108    69.41%          39                 3    92.31%         764               201    73.69%         222                84    62.16%
Page/V3/BlobStore.h                              7                 1    85.71%           7                 1    85.71%          25                10    60.00%           0                 0         -
Page/V3/MapUtils.h                              12                 1    91.67%           3                 0   100.00%          18                 1    94.44%           6                 1    83.33%
Page/V3/PageDirectory.cpp                      444                70    84.23%          30                 3    90.00%         981               199    79.71%         368                90    75.54%
Page/V3/PageDirectory.h                         20                 4    80.00%          20                 4    80.00%          89                15    83.15%           0                 0         -
Page/V3/PageDirectoryFactory.cpp                39                18    53.85%           4                 0   100.00%         100                37    63.00%          44                17    61.36%
Page/V3/PageDirectoryFactory.h                   1                 0   100.00%           1                 0   100.00%           4                 0   100.00%           0                 0         -
Page/V3/PageEntriesEdit.h                       42                 4    90.48%          27                 2    92.59%         114                 5    95.61%          14                 6    57.14%
Page/V3/PageStorageImpl.cpp                     67                22    67.16%          18                 7    61.11%         162                63    61.11%          36                21    41.67%
Page/V3/PageStorageImpl.h                        8                 4    50.00%           8                 4    50.00%           8                 4    50.00%           0                 0         -
Page/V3/WAL/WALReader.cpp                       66                16    75.76%           9                 0   100.00%         166                28    83.13%          50                18    64.00%
Page/V3/WAL/WALReader.h                         12                 3    75.00%           4                 1    75.00%          19                 7    63.16%           6                 2    66.67%
Page/V3/WAL/serialize.cpp                       47                11    76.60%          15                 1    93.33%         190                41    78.42%          52                17    67.31%
Page/V3/WALStore.cpp                            63                37    41.27%           8                 1    87.50%         103                32    68.93%          28                17    39.29%
Page/V3/WALStore.h                               1                 0   100.00%           1                 0   100.00%           3                 0   100.00%           0                 0         -
Page/V3/tests/entries_helper.h                  65                36    44.62%          10                 3    70.00%         237               111    53.16%          32                21    34.38%
Page/V3/tests/gtest_blob_store.cpp            3549               584    83.54%          17                 0   100.00%         688                 0   100.00%        1102               518    52.99%
Page/V3/tests/gtest_page_directory.cpp        8161              1417    82.64%          43                 0   100.00%        1428                 6    99.58%        2570              1297    49.53%
Page/V3/tests/gtest_page_storage.cpp          2571               934    63.67%          30                 8    73.33%         666               161    75.83%         826               494    40.19%
Page/V3/tests/gtest_wal_store.cpp             1764               244    86.17%          10                 0   100.00%         408                12    97.06%         582               284    51.20%
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
TOTAL                                        24039              5123    78.69%         469                59    87.42%       10075              1550    84.62%        8572              4119    51.95%

Coverage summary

Functions  MissedFunctions  Executed  Lines   MissedLines  Cover
16749      9441             43.63%    188633  95739        49.25%

full coverage report (for internal network access only)

@JaySon-Huang
Copy link
Contributor Author

/run-all-tests

Comment on lines +508 to +523
// wal->compactLogs();
// wal.reset();

// // After logs compacted, they should be written as one edit.
// num_edits_read = 0;
// num_pages_read = 0;
// wal = WALStore::create(
// [&](PageEntriesEdit && edit) {
// num_pages_read += edit.size();
// EXPECT_EQ(page_id, edit.size()) << fmt::format("at idx={}", num_edits_read);
// num_edits_read += 1;
// },
// provider,
// delegator);
// EXPECT_EQ(num_edits_read, 1);
// EXPECT_EQ(num_pages_read, page_id);
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These test need to be rewrite in the following PR

@sre-bot
Copy link
Collaborator

sre-bot commented Mar 7, 2022

Coverage for changed files

Filename                                   Regions    Missed Regions     Cover   Functions  Missed Functions  Executed       Lines      Missed Lines     Cover    Branches   Missed Branches     Cover
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
DeltaMerge/DeltaMergeStore.cpp                1317               467    64.54%          65                 5    92.31%        1881               426    77.35%         766               368    51.96%
Page/PageStorage.h                              16                 8    50.00%          16                 8    50.00%          53                20    62.26%           0                 0         -
Page/V2/PageStorage.cpp                        518               146    71.81%          36                 3    91.67%         880               163    81.48%         350               144    58.86%
Page/V2/PageStorage.h                           41                 9    78.05%          16                 3    81.25%          44                 3    93.18%          20                 9    55.00%
Page/V2/tests/gtest_page_storage.cpp          4855               980    79.81%          32                 2    93.75%        1044                 5    99.52%        1498               712    52.47%
Page/V3/BlobStore.cpp                          353               108    69.41%          39                 3    92.31%         764               201    73.69%         222                84    62.16%
Page/V3/BlobStore.h                              7                 1    85.71%           7                 1    85.71%          25                10    60.00%           0                 0         -
Page/V3/MapUtils.h                              12                 1    91.67%           3                 0   100.00%          18                 1    94.44%           6                 1    83.33%
Page/V3/PageDirectory.cpp                      444                70    84.23%          30                 3    90.00%         981               199    79.71%         368                90    75.54%
Page/V3/PageDirectory.h                         20                 4    80.00%          20                 4    80.00%          89                15    83.15%           0                 0         -
Page/V3/PageDirectoryFactory.cpp                39                18    53.85%           4                 0   100.00%         100                37    63.00%          44                17    61.36%
Page/V3/PageDirectoryFactory.h                   1                 0   100.00%           1                 0   100.00%           4                 0   100.00%           0                 0         -
Page/V3/PageEntriesEdit.h                       42                 4    90.48%          27                 2    92.59%         114                 5    95.61%          14                 6    57.14%
Page/V3/PageStorageImpl.cpp                     67                22    67.16%          18                 7    61.11%         162                63    61.11%          36                21    41.67%
Page/V3/PageStorageImpl.h                        8                 4    50.00%           8                 4    50.00%           8                 4    50.00%           0                 0         -
Page/V3/WAL/WALReader.cpp                       66                16    75.76%           9                 0   100.00%         166                28    83.13%          50                18    64.00%
Page/V3/WAL/WALReader.h                         12                 3    75.00%           4                 1    75.00%          19                 7    63.16%           6                 2    66.67%
Page/V3/WAL/serialize.cpp                       47                11    76.60%          15                 1    93.33%         190                41    78.42%          52                17    67.31%
Page/V3/WALStore.cpp                            63                37    41.27%           8                 1    87.50%         103                32    68.93%          28                17    39.29%
Page/V3/WALStore.h                               1                 0   100.00%           1                 0   100.00%           3                 0   100.00%           0                 0         -
Page/V3/tests/entries_helper.h                  65                36    44.62%          10                 3    70.00%         237               111    53.16%          32                21    34.38%
Page/V3/tests/gtest_blob_store.cpp            3549               584    83.54%          17                 0   100.00%         688                 0   100.00%        1102               518    52.99%
Page/V3/tests/gtest_page_directory.cpp        8161              1417    82.64%          43                 0   100.00%        1428                 6    99.58%        2570              1297    49.53%
Page/V3/tests/gtest_page_storage.cpp          2571               934    63.67%          30                 8    73.33%         666               161    75.83%         826               494    40.19%
Page/V3/tests/gtest_wal_store.cpp             1764               244    86.17%          10                 0   100.00%         408                12    97.06%         582               284    51.20%
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
TOTAL                                        24039              5124    78.68%         469                59    87.42%       10075              1550    84.62%        8572              4120    51.94%

Coverage summary

Functions  MissedFunctions  Executed  Lines   MissedLines  Cover
16749      9441             43.63%    188633  95742        49.24%

full coverage report (for internal network access only)

@ti-chi-bot ti-chi-bot added status/LGT2 Indicates that a PR has LGTM 2. and removed status/LGT1 Indicates that a PR has LGTM 1. labels Mar 7, 2022
@JaySon-Huang
Copy link
Contributor Author

/merge

@ti-chi-bot
Copy link
Member

@JaySon-Huang: It seems you want to merge this PR, I will help you trigger all the tests:

/run-all-tests

You only need to trigger /merge once, and if the CI test fails, you just re-trigger the test that failed and the bot will merge the PR for you after the CI passes.

If you have any questions about the PR merge process, please refer to pr process.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the ti-community-infra/tichi repository.

@ti-chi-bot
Copy link
Member

This pull request has been accepted and is ready to merge.

DetailsCommit hash: bf76e3e

@ti-chi-bot ti-chi-bot added the status/can-merge Indicates a PR has been approved by a committer. label Mar 8, 2022
@sre-bot
Copy link
Collaborator

sre-bot commented Mar 8, 2022

Coverage for changed files

Filename                                   Regions    Missed Regions     Cover   Functions  Missed Functions  Executed       Lines      Missed Lines     Cover    Branches   Missed Branches     Cover
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
DeltaMerge/DeltaMergeStore.cpp                1317               466    64.62%          65                 5    92.31%        1881               426    77.35%         766               366    52.22%
Page/PageStorage.h                              16                 8    50.00%          16                 8    50.00%          53                20    62.26%           0                 0         -
Page/V2/PageStorage.cpp                        518               146    71.81%          36                 3    91.67%         880               163    81.48%         350               144    58.86%
Page/V2/PageStorage.h                           41                 9    78.05%          16                 3    81.25%          44                 3    93.18%          20                 9    55.00%
Page/V2/tests/gtest_page_storage.cpp          4855               980    79.81%          32                 2    93.75%        1044                 5    99.52%        1498               712    52.47%
Page/V3/BlobStore.cpp                          353               108    69.41%          39                 3    92.31%         764               201    73.69%         222                84    62.16%
Page/V3/BlobStore.h                              7                 1    85.71%           7                 1    85.71%          25                10    60.00%           0                 0         -
Page/V3/MapUtils.h                              12                 1    91.67%           3                 0   100.00%          18                 1    94.44%           6                 1    83.33%
Page/V3/PageDirectory.cpp                      444                70    84.23%          30                 3    90.00%         981               199    79.71%         368                90    75.54%
Page/V3/PageDirectory.h                         20                 4    80.00%          20                 4    80.00%          89                15    83.15%           0                 0         -
Page/V3/PageDirectoryFactory.cpp                39                18    53.85%           4                 0   100.00%         100                37    63.00%          44                17    61.36%
Page/V3/PageDirectoryFactory.h                   1                 0   100.00%           1                 0   100.00%           4                 0   100.00%           0                 0         -
Page/V3/PageEntriesEdit.h                       42                 4    90.48%          27                 2    92.59%         114                 5    95.61%          14                 6    57.14%
Page/V3/PageStorageImpl.cpp                     67                22    67.16%          18                 7    61.11%         162                63    61.11%          36                21    41.67%
Page/V3/PageStorageImpl.h                        8                 4    50.00%           8                 4    50.00%           8                 4    50.00%           0                 0         -
Page/V3/WAL/WALReader.cpp                       66                16    75.76%           9                 0   100.00%         166                28    83.13%          50                18    64.00%
Page/V3/WAL/WALReader.h                         12                 3    75.00%           4                 1    75.00%          19                 7    63.16%           6                 2    66.67%
Page/V3/WAL/serialize.cpp                       47                11    76.60%          15                 1    93.33%         190                41    78.42%          52                17    67.31%
Page/V3/WALStore.cpp                            63                37    41.27%           8                 1    87.50%         103                32    68.93%          28                17    39.29%
Page/V3/WALStore.h                               1                 0   100.00%           1                 0   100.00%           3                 0   100.00%           0                 0         -
Page/V3/tests/entries_helper.h                  65                36    44.62%          10                 3    70.00%         237               111    53.16%          32                21    34.38%
Page/V3/tests/gtest_blob_store.cpp            3549               584    83.54%          17                 0   100.00%         688                 0   100.00%        1102               518    52.99%
Page/V3/tests/gtest_page_directory.cpp        8161              1417    82.64%          43                 0   100.00%        1428                 6    99.58%        2570              1297    49.53%
Page/V3/tests/gtest_page_storage.cpp          2571               934    63.67%          30                 8    73.33%         666               161    75.83%         826               494    40.19%
Page/V3/tests/gtest_wal_store.cpp             1764               244    86.17%          10                 0   100.00%         408                12    97.06%         582               284    51.20%
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
TOTAL                                        24039              5123    78.69%         469                59    87.42%       10075              1550    84.62%        8572              4118    51.96%

Coverage summary

Functions  MissedFunctions  Executed  Lines   MissedLines  Cover
16755      9447             43.62%    188614  95721        49.25%

full coverage report (for internal network access only)

@ti-chi-bot ti-chi-bot merged commit a7be22f into pingcap:master Mar 8, 2022
@JaySon-Huang JaySon-Huang deleted the ref_page_life_matters branch March 8, 2022 04:22
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

release-note-none Denotes a PR that doesn't merit a release note. size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. status/can-merge Indicates a PR has been approved by a committer. status/LGT2 Indicates that a PR has LGTM 2.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Ref page lifetime management is broken

5 participants