Reduce time cost about ci test by solotzg · Pull Request #1849 · pingcap/tiflash

solotzg · 2021-04-30T04:29:25Z

Signed-off-by: Zhigao Tong tongzhigao@pingcap.com

What problem does this PR solve?

Problem Summary:

What is changed and how it works?

Proposal: https://internal.pingcap.net/jira/browse/FD-53

What's Changed:

reduce time cost about ci test by parallel
add -DNO_WERROR=ON to cmake config for release-darwin build

Check List

Tests

Unit test
Integration test

Release note

No release note

solotzg · 2021-04-30T04:59:25Z

/rebuild

ilovesoup · 2021-05-05T16:14:37Z

what is this...

solotzg · 2021-05-08T08:36:47Z

/run-all-tests

solotzg · 2021-05-08T10:07:01Z

/run-all-tests

solotzg · 2021-05-09T04:05:07Z

/run-all-tests

solotzg · 2021-05-09T05:09:24Z

/run-all-tests

solotzg · 2021-05-10T01:34:45Z

/run-all-tests

solotzg · 2021-05-10T07:51:54Z

/run-all-tests

solotzg · 2021-05-10T11:23:47Z

/run-all-tests

solotzg · 2021-05-10T13:29:23Z

/run-all-tests

Signed-off-by: Zhigao Tong <tongzhigao@pingcap.com>

solotzg · 2021-05-10T14:37:56Z

/run-all-tests

birdstorm

LGTM

Signed-off-by: Zhigao Tong <tongzhigao@pingcap.com>

solotzg · 2021-05-11T02:23:02Z

/rebuild

solotzg · 2021-05-11T03:28:09Z

/run-all-tests

solotzg · 2021-05-11T04:54:49Z

/run-all-tests

solotzg · 2021-05-11T06:18:51Z

/run-all-tests

solotzg · 2021-05-11T07:05:52Z

/run-all-tests

solotzg · 2021-05-11T07:52:48Z

/run-all-tests

solotzg · 2021-05-11T08:02:37Z

/run-all-tests

Signed-off-by: Zhigao Tong <tongzhigao@pingcap.com>

solotzg · 2021-05-11T11:25:58Z

/run-all-tests

solotzg · 2021-05-11T12:05:36Z

/run-all-tests

solotzg · 2021-05-11T13:27:29Z

/run-all-tests

solotzg · 2021-05-11T13:49:29Z

/run-all-tests

ti-srebot · 2021-05-11T14:40:01Z

cherry pick to release-4.0 failed

ti-srebot · 2021-05-11T14:51:48Z

cherry pick to release-5.0 failed

solotzg · 2021-05-12T07:24:31Z

/cherry-pick

* reduce time cost about ci test by parallel * add `-DNO_WERROR=ON` to cmake config for release-darwin build * Fix tidb_ghpr_tics_test fail (#1895) Signed-off-by: Zhigao Tong <tongzhigao@pingcap.com>

* reduce time cost about ci test by parallel * add `-DNO_WERROR=ON` to cmake config for release-darwin build * Fix tidb_ghpr_tics_test fail Signed-off-by: Zhigao Tong <tongzhigao@pingcap.com>

Signed-off-by: ti-srebot <ti-srebot@pingcap.com> Co-authored-by: Flowyi <flowbehappy@gmail.com> cherry pick pingcap#1821 to release-5.0 (pingcap#1827) Signed-off-by: ti-srebot <ti-srebot@pingcap.com> Co-authored-by: xufei <xufei@pingcap.com> Remove ReplicatedMergeTree family (pingcap#1805) (pingcap#1826) cherry pick pingcap#1835 to release-5.0 (pingcap#1840) Signed-off-by: ti-srebot <ti-srebot@pingcap.com> Co-authored-by: xufei <xufei@pingcap.com> Use file path as cache key (pingcap#1852) (pingcap#1862) Lazily initializing DeltaMergeStore (pingcap#1423) (pingcap#1751) (pingcap#1868) Add background gc check thread for DeltaTree storage (pingcap#1742) (pingcap#1828) * cherry pick pingcap#1742 to release-5.0 Signed-off-by: ti-srebot <ti-srebot@pingcap.com> * fix conflict Co-authored-by: lidezhu <47731263+lidezhu@users.noreply.github.com> Fix calculate stable property (pingcap#1839) (pingcap#1878) * cherry pick pingcap#1839 to release-5.0 Signed-off-by: ti-srebot <ti-srebot@pingcap.com> * fix compile and cherry pick a small fix Co-authored-by: lidezhu <47731263+lidezhu@users.noreply.github.com> Fix client-c mistake setting current_ts to 0 when resolving lock for async commit (pingcap#1856) (pingcap#1870) support constant folding in dbg coprocessor (pingcap#1881) (pingcap#1884) Optimize & Reduce time cost about ci test (pingcap#1849) (pingcap#1894) * reduce time cost about ci test by parallel * add `-DNO_WERROR=ON` to cmake config for release-darwin build * Fix tidb_ghpr_tics_test fail (pingcap#1895) Signed-off-by: Zhigao Tong <tongzhigao@pingcap.com> Fix panic with feature `compaction filter` in release-5.0 (pingcap#1891) Init store in background task. (pingcap#1843,pingcap#1896) (pingcap#1874) Push down filter on timestamp type column to storage layer (pingcap#1875) (pingcap#1906) * cherry pick pingcap#1875 to release-5.0 Signed-off-by: ti-srebot <ti-srebot@pingcap.com> * remove irrevalant code Co-authored-by: lidezhu <47731263+lidezhu@users.noreply.github.com> Fix ExchangeSender: remove duplicated write stream operation (pingcap#1379) (pingcap#1883) cherry pick pingcap#1917 to release-5.0 (pingcap#1923) Signed-off-by: ti-srebot <ti-srebot@pingcap.com> Co-authored-by: Fu Zhe <fuzhe1989@gmail.com> Pre-handle SSTFiles to DTFiles when applying snapshots (pingcap#1439) (pingcap#1867) * Add `raft.snapshot.method` in configuration file to control the method of applying snapshots - "block" decode snapshot data as a block - "file1" decode snapshot data as DTFiles (directory mode) [**By default**] * Add a stream `SSTFilesToBlockInputStream` to decode snapshot data into blocks * Add a stream `BoundedSSTFilesToBlockInputStream` to bound the blocks read from SSTFilesToBlockInputStream by column `_tidb_rowid` * Add a stream `SSTFilesToDTFilesOutputStream` that accept `BoundedSSTFilesToBlockInputStream` as input stream to write blocks into DTFiles * Make `STORAGE_FORMAT_CURRENT` to be `STORAGE_FORMAT_V2` by default to support ingest DTFile into DT engine * Fix the bug that the block decoded from SSTFile is not sorted by primary key and version (pingcap#1888) * Fix panic with feature compaction filter with DTFile * Fix ingest write amplification after snapshot apply optimization (pingcap#1913) Co-authored-by: Zhigao Tong <tongzhigao@pingcap.com> Signed-off-by: JaySon-Huang <jayson.hjs@gmail.com> Cast int/real as real (pingcap#1911) (pingcap#1928) Fix the problem that old dmfile is not removed atomically (pingcap#1918) (pingcap#1925) Eliminate the table lock between reading,writing and DDL operators for TiFlash (pingcap#1736) (pingcap#1858) * Eliminate the table lock between reading,writing and DDL operators for TiFlash (pingcap#1736) 1. Port the latest `RWLock` for `IStorage` from Clickhouse. "phase-fair rwlocks have been shown to provide more predictable timings compared to task-fair rwlocks for certain workloads (for relatively low share of writes, <25%). Brandenburg B, Anderson J, (2010)" 2. Refactor the lock model for `IStorage` to eliminate the lock between reading, writing, and DDL operations. 3. Acquire table lock by QueryID/threadName instead of function name 4. Tombstone the database after receiving a "drop" DDL from TiDB. Physically drop the databases / tables after gc safe point. 5. Remove some outdated tests under `tests/mutable-test/txn_schema` (all those tests are ported into `tests/delta-merge-test/raft/schema`) * Mark FIXME for conflict codes Conflicts: dbms/src/Flash/Coprocessor/DAGQueryBlockInterpreter.cpp Signed-off-by: JaySon-Huang <jayson.hjs@gmail.com> cached tics code incase of clone same code repeatedly (pingcap#1994) (pingcap#2001) Add row & bytes threshold config for CompactLog (pingcap#1997) (pingcap#2005) * cherry pick pingcap#1997 to release-5.0 Signed-off-by: ti-srebot <ti-srebot@pingcap.com> Co-authored-by: Zhigao Tong <tongzhigao@pingcap.com> Fix the bug that incomplete write batches are not truncated (pingcap#1934) (pingcap#2003) Revert "Push down filter on timestamp type column to storage layer (pingcap#1875) (pingcap#1906)" (pingcap#2010) This reverts commit 630b612. Revert Lazily Init Store (pingcap#2011) * Revert "Init store in background task. (pingcap#1843,pingcap#1896) (pingcap#1874)" This reverts commit 0882461. Conflicts: dbms/src/Storages/StorageDeltaMerge.cpp * format code * Revert "Lazily initializing DeltaMergeStore (pingcap#1423) (pingcap#1751) (pingcap#1868)" This reverts commit bbce050. Conflicts: dbms/src/Storages/DeltaMerge/tests/gtest_dm_storage_delta_merge.cpp dbms/src/Storages/StorageDeltaMerge.cpp dbms/src/Storages/StorageDeltaMerge.h Co-authored-by: JinheLin <linjinhe@pingcap.com> Co-authored-by: Flowyi <flowbehappy@gmail.com> Revert "background gc thread" (pingcap#2015) * Revert "Fix calculate stable property (pingcap#1839) (pingcap#1878)" This reverts commit 6a9eb58. * Revert "Add background gc check thread for DeltaTree storage (pingcap#1742) (pingcap#1828)" This reverts commit cbbbd09. * remove code * format code * format code * fix test compile * fix comment Fix the potential concurrency problem when clone the shared delta index (pingcap#2030) (pingcap#2033) * cherry pick pingcap#2030 to release-5.0 Signed-off-by: ti-srebot <ti-srebot@pingcap.com> * Update dbms/src/Storages/DeltaMerge/DeltaIndex.h Co-authored-by: JaySon <jayson.hjs@gmail.com> Co-authored-by: lidezhu <47731263+lidezhu@users.noreply.github.com> Co-authored-by: JaySon <jayson.hjs@gmail.com> Fix bug for unexpected deleted ingest file (pingcap#2047) (pingcap#2048) * Only enable GC for DTFiles after they get applied to all segments. * Refine some loggings Signed-off-by: JaySon-Huang <jayson.hjs@gmail.com> Fix cast int as time (pingcap#1788) (pingcap#1893) cherry pick pingcap#1903 to release-5.0 (pingcap#1915) Signed-off-by: ti-srebot <ti-srebot@pingcap.com> Fix ingesting file may add ref page to deleted page (pingcap#2054) (pingcap#2055) 1. Put the ingest file id into `storage_pool.data` before ingesting them into segments 2. Add ref pages to the ingest files for each segment 3. Delete the original page id after all Signed-off-by: JaySon-Huang <jayson.hjs@gmail.com> Fix compile error on Mac Clang 12.0.5 (pingcap#2058) (pingcap#2070) cherry pick pingcap#2017 pingcap#2020 pingcap#2051 to release-5.0 (pingcap#2074) Revert "cherry pick pingcap#2017 pingcap#2020 pingcap#2051 to release-5.0 (pingcap#2074)" This reverts commit 8e0712c. Revert "Fix compile error on Mac Clang 12.0.5 (pingcap#2058) (pingcap#2070)" This reverts commit 3c31b92. Revert "Fix ingesting file may add ref page to deleted page (pingcap#2054) (pingcap#2055)" This reverts commit c773d61. Revert "Fix bug for unexpected deleted ingest file (pingcap#2047) (pingcap#2048)" This reverts commit b329618. Revert "Pre-handle SSTFiles to DTFiles when applying snapshots (pingcap#1439) (pingcap#1867)" This reverts commit c947fd6. Conflicts: dbms/src/Common/FailPoint.cpp dbms/src/Storages/DeltaMerge/File/DMFileBlockOutputStream.h dbms/src/Storages/DeltaMerge/File/DMFileWriter.h dbms/src/Storages/DeltaMerge/ReorganizeBlockInputStream.h dbms/src/Storages/DeltaMerge/SSTFilesToBlockInputStream.cpp dbms/src/Storages/DeltaMerge/SSTFilesToBlockInputStream.h dbms/src/Storages/DeltaMerge/SSTFilesToDTFilesOutputStream.cpp dbms/src/Storages/DeltaMerge/Segment.cpp dbms/src/Storages/DeltaMerge/StableValueSpace.cpp dbms/src/Storages/DeltaMerge/tests/gtest_dm_file.cpp dbms/src/Storages/DeltaMerge/tests/gtest_dm_storage_delta_merge.cpp dbms/src/Storages/StorageDeltaMerge.cpp dbms/src/Storages/StorageDeltaMerge.h dbms/src/Storages/Transaction/ApplySnapshot.cpp dbms/src/Storages/Transaction/PartitionStreams.cpp tests/delta-merge-test/raft/schema/drop_on_restart.test Fix comment Signed-off-by: JaySon-Huang <jayson.hjs@gmail.com>

Reduce time cost about ci test

38e4a71

Signed-off-by: Zhigao Tong <tongzhigao@pingcap.com>

solotzg changed the title ~~[DNM]~~ Reduce time cost about ci test May 10, 2021

solotzg added type/enhancement The issue or PR belongs to an enhancement. needs-cherry-pick-release-4.0 PR which needs to be cherry-picked to release-4.0 needs-cherry-pick-release-5.0 PR which needs to be cherry-picked to release-5.0 labels May 10, 2021

birdstorm approved these changes May 10, 2021

View reviewed changes

ti-srebot added the status/LGT1 Indicates that a PR has LGTM 1. label May 10, 2021

optmize & refine

a0cc5ab

Signed-off-by: Zhigao Tong <tongzhigao@pingcap.com>

fix bug

807e25b

Signed-off-by: Zhigao Tong <tongzhigao@pingcap.com>

solotzg merged commit fe088de into pingcap:master May 11, 2021

solotzg deleted the _tmp branch May 11, 2021 14:20

solotzg mentioned this pull request May 11, 2021

Fix the issue that null_literal.test costs more than 10 minutes #1882

Closed

JaySon-Huang mentioned this pull request May 12, 2021

Speed up CI testing #1394

Closed

JaySon-Huang linked an issue May 12, 2021 that may be closed by this pull request

Speed up CI testing #1394

Closed

JaySon-Huang mentioned this pull request May 25, 2021

CI: put pipeline scripts to this repo #549

Closed

JaySon-Huang mentioned this pull request Jun 8, 2021

Check the diff between revert_snapshot and pick need commit JaySon-Huang/tiflash#8

Merged

Conversation

solotzg commented Apr 30, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What problem does this PR solve?

What is changed and how it works?

Check List

Release note

Uh oh!

solotzg commented Apr 30, 2021

Uh oh!

ilovesoup commented May 5, 2021

Uh oh!

solotzg commented May 8, 2021

Uh oh!

solotzg commented May 8, 2021

Uh oh!

solotzg commented May 9, 2021

Uh oh!

solotzg commented May 9, 2021

Uh oh!

solotzg commented May 10, 2021

Uh oh!

solotzg commented May 10, 2021

Uh oh!

solotzg commented May 10, 2021

Uh oh!

solotzg commented May 10, 2021

Uh oh!

solotzg commented May 10, 2021

Uh oh!

birdstorm left a comment

Choose a reason for hiding this comment

Uh oh!

solotzg commented May 11, 2021

Uh oh!

solotzg commented May 11, 2021

Uh oh!

solotzg commented May 11, 2021

Uh oh!

solotzg commented May 11, 2021

Uh oh!

solotzg commented May 11, 2021

Uh oh!

solotzg commented May 11, 2021

Uh oh!

solotzg commented May 11, 2021

Uh oh!

solotzg commented May 11, 2021

Uh oh!

solotzg commented May 11, 2021

Uh oh!

solotzg commented May 11, 2021

Uh oh!

solotzg commented May 11, 2021

Uh oh!

ti-srebot commented May 11, 2021

Uh oh!

ti-srebot commented May 11, 2021

Uh oh!

solotzg commented May 12, 2021

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

solotzg commented Apr 30, 2021 •

edited

Loading