Optimize tics_ghpr_build: compile without debug info & accelerate by ccache by solotzg · Pull Request #2020 · pingcap/tiflash

solotzg · 2021-05-28T07:29:32Z

Signed-off-by: Zhigao Tong tongzhigao@pingcap.com

What problem does this PR solve?

close #1390

Problem Summary:

What is changed and how it works?

compile without debug info during ci build
ccache
time cost about compiling is about 4mins if 99.88% cache hit

Check List

Tests

Unit test
Integration test
Manual test (add detailed scripts or steps below)
No code

Side effects

Release note

No release note

solotzg · 2021-05-28T15:19:14Z

/rebuild

solotzg · 2021-05-29T05:37:59Z

/run-all-tests

JaySon-Huang

Great job!

I have some questions about

How often should we update the ccache files?
How do we update the ccache files? By running this script on our server? If we need to update the ccache files weekly or monthly, can we make it a Jenkins workflow and automatically update it later?

release-centos7/build/build-tiflash-ci.sh

solotzg · 2021-05-30T04:26:14Z

Great job!

I have some questions about

How often should we update the ccache files?

How do we update the ccache files? By running this script on our server? If we need to update the ccache files weekly or monthly, can we make it a Jenkins workflow and automatically update it later?

https://cd.pingcap.net/blue/organizations/jenkins/build_tiflash_multi_branch/activity/
Each time after a new commit merged into target branch, a task about nightly build will be triggered. BUILD_UPDATE_CI_CCACHE is set true in order to build and upload ccache.

solotzg · 2021-05-31T02:02:20Z

/run-all-tests

CMakeLists.txt

solotzg · 2021-05-31T08:32:45Z

/run-all-tests

solotzg · 2021-05-31T10:57:12Z

/run-all-tests

solotzg · 2021-05-31T11:16:58Z

/run-all-tests

JaySon-Huang · 2021-05-31T11:17:08Z

.ci/util.groovy

https://github.com/pingcap/tics/pull/2020/files#diff-1c2005d09a3a545f2ac1cfeeb0052743c1fb6ce187fb3fb5583b957e0dd8601fR33-R41

Aren't the files downloaded outside "parallel" and pop by "unstash" in L91? Why do we need to download them for each test?

The variables DOWNLOAD_TAR is used to decide whether tiflash binary target should be downloaded. Each task in parallel are independent. If we put binary target in stashed dir, it seems much slower than downloading in each task.

release-centos7/build/build-tiflash-ci.sh

solotzg · 2021-05-31T14:02:09Z

/run-all-tests

solotzg · 2021-05-31T14:25:49Z

/run-all-tests

solotzg · 2021-06-01T02:27:07Z

/run-all-tests

solotzg · 2021-06-01T03:26:48Z

/run-all-tests

solotzg · 2021-06-01T09:58:00Z

@zanmato1984 @JaySon-Huang can we merge this pr ?

solotzg · 2021-06-01T10:10:28Z

/run-all-tests

tests/docker/mock-test-dt.yaml

zanmato1984 · 2021-06-01T10:28:35Z

tests/docker/run.sh

What does -tagged-image even mean?

It means in this yaml, image of tiflash will be hub.pingcap.net/tiflash/tics:${TAG:-master}

So why not name it something like tiflash.yaml?

To describe it precisely.

What about "tiflash-override.yaml"?

Personally, I don't think it's a appropriate name for such case.

zanmato1984 · 2021-06-01T10:30:09Z

release-centos7/build/build-tiflash-ci.sh

Can we install ccache in the builder image so that it doesn't get installed every time for a PR build.

After this pr merged, we should install ccache in builder images by default.

Can we do it now? I don't think updating builder image depends on this PR.

Install ccache only cost few seconds. We should not affect other branches.

It WON'T affect other branches if we just install ccache, will it?

Maybe not. I think we shall remake image later because clickhouse make ccache enabled by default.

CMakeLists.txt

JaySon-Huang

LGTM with minor comment

JaySon-Huang · 2021-06-01T11:00:08Z

release-centos7/build/fetch-ci-build.sh

Maybe use "${SCRIPTPATH}/commit-hash", "${SCRIPTPATH}/tiflash.tar.gz", so that we don't need to care about whether the PWD equals with "${SCRIPPATH}" or not. Those temporary files are always generated under the directory this script in.

Maybe you mistake the working path, it's specified outside https://github.com/pingcap/tics/pull/2020/files#diff-e451f4d1767f8fc40cfd70e4f180a772c48c02f034ad79948e6cfb6670efd5d3R12

JaySon-Huang · 2021-06-01T11:00:28Z

release-centos7/build/upload-ci-build.sh

https://github.com/pingcap/tics/pull/2020/files/7aa5ff0d834b35017dd9a961f0e9be28b42c14b6#diff-86da04371790378c0d0102c67c84a2ea95fabf60873a025b287de33a5c53589cR11

solotzg · 2021-06-01T14:00:44Z

/run-all-tests

solotzg · 2021-06-02T06:07:19Z

/run-all-tests

* compile without debug info during ci build * use `ccache` to accelerate speed * remove docker build in ci Signed-off-by: Zhigao Tong <tongzhigao@pingcap.com>

solotzg · 2021-06-02T06:23:31Z

/rebuild

solotzg · 2021-06-02T06:31:30Z

/run-all-tests

solotzg · 2021-06-02T07:18:43Z

/rebuild

solotzg · 2021-06-02T07:23:20Z

/rebuild

zanmato1984

LGTM

solotzg · 2021-06-02T08:48:01Z

/run-all-tests

solotzg · 2021-06-02T09:16:38Z

/run-all-tests

solotzg · 2021-06-02T09:18:46Z

/run-all-tests

…-5.0 (pingcap#2074)" This reverts commit 8e0712c.

Signed-off-by: ti-srebot <ti-srebot@pingcap.com> Co-authored-by: Flowyi <flowbehappy@gmail.com> cherry pick pingcap#1821 to release-5.0 (pingcap#1827) Signed-off-by: ti-srebot <ti-srebot@pingcap.com> Co-authored-by: xufei <xufei@pingcap.com> Remove ReplicatedMergeTree family (pingcap#1805) (pingcap#1826) cherry pick pingcap#1835 to release-5.0 (pingcap#1840) Signed-off-by: ti-srebot <ti-srebot@pingcap.com> Co-authored-by: xufei <xufei@pingcap.com> Use file path as cache key (pingcap#1852) (pingcap#1862) Lazily initializing DeltaMergeStore (pingcap#1423) (pingcap#1751) (pingcap#1868) Add background gc check thread for DeltaTree storage (pingcap#1742) (pingcap#1828) * cherry pick pingcap#1742 to release-5.0 Signed-off-by: ti-srebot <ti-srebot@pingcap.com> * fix conflict Co-authored-by: lidezhu <47731263+lidezhu@users.noreply.github.com> Fix calculate stable property (pingcap#1839) (pingcap#1878) * cherry pick pingcap#1839 to release-5.0 Signed-off-by: ti-srebot <ti-srebot@pingcap.com> * fix compile and cherry pick a small fix Co-authored-by: lidezhu <47731263+lidezhu@users.noreply.github.com> Fix client-c mistake setting current_ts to 0 when resolving lock for async commit (pingcap#1856) (pingcap#1870) support constant folding in dbg coprocessor (pingcap#1881) (pingcap#1884) Optimize & Reduce time cost about ci test (pingcap#1849) (pingcap#1894) * reduce time cost about ci test by parallel * add `-DNO_WERROR=ON` to cmake config for release-darwin build * Fix tidb_ghpr_tics_test fail (pingcap#1895) Signed-off-by: Zhigao Tong <tongzhigao@pingcap.com> Fix panic with feature `compaction filter` in release-5.0 (pingcap#1891) Init store in background task. (pingcap#1843,pingcap#1896) (pingcap#1874) Push down filter on timestamp type column to storage layer (pingcap#1875) (pingcap#1906) * cherry pick pingcap#1875 to release-5.0 Signed-off-by: ti-srebot <ti-srebot@pingcap.com> * remove irrevalant code Co-authored-by: lidezhu <47731263+lidezhu@users.noreply.github.com> Fix ExchangeSender: remove duplicated write stream operation (pingcap#1379) (pingcap#1883) cherry pick pingcap#1917 to release-5.0 (pingcap#1923) Signed-off-by: ti-srebot <ti-srebot@pingcap.com> Co-authored-by: Fu Zhe <fuzhe1989@gmail.com> Pre-handle SSTFiles to DTFiles when applying snapshots (pingcap#1439) (pingcap#1867) * Add `raft.snapshot.method` in configuration file to control the method of applying snapshots - "block" decode snapshot data as a block - "file1" decode snapshot data as DTFiles (directory mode) [**By default**] * Add a stream `SSTFilesToBlockInputStream` to decode snapshot data into blocks * Add a stream `BoundedSSTFilesToBlockInputStream` to bound the blocks read from SSTFilesToBlockInputStream by column `_tidb_rowid` * Add a stream `SSTFilesToDTFilesOutputStream` that accept `BoundedSSTFilesToBlockInputStream` as input stream to write blocks into DTFiles * Make `STORAGE_FORMAT_CURRENT` to be `STORAGE_FORMAT_V2` by default to support ingest DTFile into DT engine * Fix the bug that the block decoded from SSTFile is not sorted by primary key and version (pingcap#1888) * Fix panic with feature compaction filter with DTFile * Fix ingest write amplification after snapshot apply optimization (pingcap#1913) Co-authored-by: Zhigao Tong <tongzhigao@pingcap.com> Signed-off-by: JaySon-Huang <jayson.hjs@gmail.com> Cast int/real as real (pingcap#1911) (pingcap#1928) Fix the problem that old dmfile is not removed atomically (pingcap#1918) (pingcap#1925) Eliminate the table lock between reading,writing and DDL operators for TiFlash (pingcap#1736) (pingcap#1858) * Eliminate the table lock between reading,writing and DDL operators for TiFlash (pingcap#1736) 1. Port the latest `RWLock` for `IStorage` from Clickhouse. "phase-fair rwlocks have been shown to provide more predictable timings compared to task-fair rwlocks for certain workloads (for relatively low share of writes, <25%). Brandenburg B, Anderson J, (2010)" 2. Refactor the lock model for `IStorage` to eliminate the lock between reading, writing, and DDL operations. 3. Acquire table lock by QueryID/threadName instead of function name 4. Tombstone the database after receiving a "drop" DDL from TiDB. Physically drop the databases / tables after gc safe point. 5. Remove some outdated tests under `tests/mutable-test/txn_schema` (all those tests are ported into `tests/delta-merge-test/raft/schema`) * Mark FIXME for conflict codes Conflicts: dbms/src/Flash/Coprocessor/DAGQueryBlockInterpreter.cpp Signed-off-by: JaySon-Huang <jayson.hjs@gmail.com> cached tics code incase of clone same code repeatedly (pingcap#1994) (pingcap#2001) Add row & bytes threshold config for CompactLog (pingcap#1997) (pingcap#2005) * cherry pick pingcap#1997 to release-5.0 Signed-off-by: ti-srebot <ti-srebot@pingcap.com> Co-authored-by: Zhigao Tong <tongzhigao@pingcap.com> Fix the bug that incomplete write batches are not truncated (pingcap#1934) (pingcap#2003) Revert "Push down filter on timestamp type column to storage layer (pingcap#1875) (pingcap#1906)" (pingcap#2010) This reverts commit 630b612. Revert Lazily Init Store (pingcap#2011) * Revert "Init store in background task. (pingcap#1843,pingcap#1896) (pingcap#1874)" This reverts commit 0882461. Conflicts: dbms/src/Storages/StorageDeltaMerge.cpp * format code * Revert "Lazily initializing DeltaMergeStore (pingcap#1423) (pingcap#1751) (pingcap#1868)" This reverts commit bbce050. Conflicts: dbms/src/Storages/DeltaMerge/tests/gtest_dm_storage_delta_merge.cpp dbms/src/Storages/StorageDeltaMerge.cpp dbms/src/Storages/StorageDeltaMerge.h Co-authored-by: JinheLin <linjinhe@pingcap.com> Co-authored-by: Flowyi <flowbehappy@gmail.com> Revert "background gc thread" (pingcap#2015) * Revert "Fix calculate stable property (pingcap#1839) (pingcap#1878)" This reverts commit 6a9eb58. * Revert "Add background gc check thread for DeltaTree storage (pingcap#1742) (pingcap#1828)" This reverts commit cbbbd09. * remove code * format code * format code * fix test compile * fix comment Fix the potential concurrency problem when clone the shared delta index (pingcap#2030) (pingcap#2033) * cherry pick pingcap#2030 to release-5.0 Signed-off-by: ti-srebot <ti-srebot@pingcap.com> * Update dbms/src/Storages/DeltaMerge/DeltaIndex.h Co-authored-by: JaySon <jayson.hjs@gmail.com> Co-authored-by: lidezhu <47731263+lidezhu@users.noreply.github.com> Co-authored-by: JaySon <jayson.hjs@gmail.com> Fix bug for unexpected deleted ingest file (pingcap#2047) (pingcap#2048) * Only enable GC for DTFiles after they get applied to all segments. * Refine some loggings Signed-off-by: JaySon-Huang <jayson.hjs@gmail.com> Fix cast int as time (pingcap#1788) (pingcap#1893) cherry pick pingcap#1903 to release-5.0 (pingcap#1915) Signed-off-by: ti-srebot <ti-srebot@pingcap.com> Fix ingesting file may add ref page to deleted page (pingcap#2054) (pingcap#2055) 1. Put the ingest file id into `storage_pool.data` before ingesting them into segments 2. Add ref pages to the ingest files for each segment 3. Delete the original page id after all Signed-off-by: JaySon-Huang <jayson.hjs@gmail.com> Fix compile error on Mac Clang 12.0.5 (pingcap#2058) (pingcap#2070) cherry pick pingcap#2017 pingcap#2020 pingcap#2051 to release-5.0 (pingcap#2074) Revert "cherry pick pingcap#2017 pingcap#2020 pingcap#2051 to release-5.0 (pingcap#2074)" This reverts commit 8e0712c. Revert "Fix compile error on Mac Clang 12.0.5 (pingcap#2058) (pingcap#2070)" This reverts commit 3c31b92. Revert "Fix ingesting file may add ref page to deleted page (pingcap#2054) (pingcap#2055)" This reverts commit c773d61. Revert "Fix bug for unexpected deleted ingest file (pingcap#2047) (pingcap#2048)" This reverts commit b329618. Revert "Pre-handle SSTFiles to DTFiles when applying snapshots (pingcap#1439) (pingcap#1867)" This reverts commit c947fd6. Conflicts: dbms/src/Common/FailPoint.cpp dbms/src/Storages/DeltaMerge/File/DMFileBlockOutputStream.h dbms/src/Storages/DeltaMerge/File/DMFileWriter.h dbms/src/Storages/DeltaMerge/ReorganizeBlockInputStream.h dbms/src/Storages/DeltaMerge/SSTFilesToBlockInputStream.cpp dbms/src/Storages/DeltaMerge/SSTFilesToBlockInputStream.h dbms/src/Storages/DeltaMerge/SSTFilesToDTFilesOutputStream.cpp dbms/src/Storages/DeltaMerge/Segment.cpp dbms/src/Storages/DeltaMerge/StableValueSpace.cpp dbms/src/Storages/DeltaMerge/tests/gtest_dm_file.cpp dbms/src/Storages/DeltaMerge/tests/gtest_dm_storage_delta_merge.cpp dbms/src/Storages/StorageDeltaMerge.cpp dbms/src/Storages/StorageDeltaMerge.h dbms/src/Storages/Transaction/ApplySnapshot.cpp dbms/src/Storages/Transaction/PartitionStreams.cpp tests/delta-merge-test/raft/schema/drop_on_restart.test Fix comment Signed-off-by: JaySon-Huang <jayson.hjs@gmail.com>

* Revert "cherry pick #2017 #2020 #2051 to release-5.0 (#2074)" * Revert "Fix compile error on Mac Clang 12.0.5 (#2058) (#2070)" * Revert "Fix ingesting file may add ref page to deleted page (#2054) (#2055)" * Revert "Fix bug for unexpected deleted ingest file (#2047) (#2048)" * Revert "Pre-handle SSTFiles to DTFiles when applying snapshots (#1439) (#1867)" This reverts commit 8e0712c , 3c31b92 , c773d61 , b329618 , c947fd6 Conflicts: dbms/src/Common/FailPoint.cpp dbms/src/Storages/DeltaMerge/File/DMFileBlockOutputStream.h dbms/src/Storages/DeltaMerge/File/DMFileWriter.h dbms/src/Storages/DeltaMerge/ReorganizeBlockInputStream.h dbms/src/Storages/DeltaMerge/SSTFilesToBlockInputStream.cpp dbms/src/Storages/DeltaMerge/SSTFilesToBlockInputStream.h dbms/src/Storages/DeltaMerge/SSTFilesToDTFilesOutputStream.cpp dbms/src/Storages/DeltaMerge/Segment.cpp dbms/src/Storages/DeltaMerge/StableValueSpace.cpp dbms/src/Storages/DeltaMerge/tests/gtest_dm_file.cpp dbms/src/Storages/DeltaMerge/tests/gtest_dm_storage_delta_merge.cpp dbms/src/Storages/StorageDeltaMerge.cpp dbms/src/Storages/StorageDeltaMerge.h dbms/src/Storages/Transaction/ApplySnapshot.cpp dbms/src/Storages/Transaction/PartitionStreams.cpp tests/delta-merge-test/raft/schema/drop_on_restart.test Signed-off-by: JaySon-Huang <jayson.hjs@gmail.com>

solotzg changed the title ~~[DNM] test~~ Compile without debug info for build-tiflash-ci May 28, 2021

solotzg changed the title ~~Compile without debug info for build-tiflash-ci~~ tmp May 28, 2021

solotzg changed the title ~~tmp~~ Compile without debug info for build-tiflash-ci & Use ccache to speed up compile May 29, 2021

solotzg changed the title ~~Compile without debug info for build-tiflash-ci & Use ccache to speed up compile~~ Optimize tics_ghpr_build: compile without debug info & enable ccache May 29, 2021

solotzg mentioned this pull request May 29, 2021

Clean up redundant code for TiFlash #2019

Open

18 tasks

solotzg requested a review from birdstorm May 29, 2021 09:38

JaySon-Huang reviewed May 29, 2021

View reviewed changes

release-centos7/build/build-tiflash-ci.sh Outdated Show resolved Hide resolved

release-centos7/build/build-tiflash-ci.sh Outdated Show resolved Hide resolved

release-centos7/build/build-tiflash-ci.sh Outdated Show resolved Hide resolved

solotzg changed the title ~~Optimize tics_ghpr_build: compile without debug info & enable ccache~~ Optimize tics_ghpr_build: compile without debug info & accelerate by ccache May 30, 2021

JaySon-Huang reviewed May 31, 2021

View reviewed changes

CMakeLists.txt Outdated Show resolved Hide resolved

solotzg requested review from LittleFall and zanmato1984 May 31, 2021 08:48

JaySon-Huang reviewed May 31, 2021

View reviewed changes

zanmato1984 reviewed May 31, 2021

View reviewed changes

release-centos7/build/build-tiflash-ci.sh Outdated Show resolved Hide resolved

zanmato1984 reviewed Jun 1, 2021

View reviewed changes

tests/docker/mock-test-dt.yaml Outdated Show resolved Hide resolved

zanmato1984 reviewed Jun 1, 2021

View reviewed changes

CMakeLists.txt Outdated Show resolved Hide resolved

zanmato1984 reviewed Jun 1, 2021

View reviewed changes

CMakeLists.txt Outdated Show resolved Hide resolved

JaySon-Huang reviewed Jun 1, 2021

View reviewed changes

Optimize tics_ghpr_build

8a6ccf5

* compile without debug info during ci build * use `ccache` to accelerate speed * remove docker build in ci Signed-off-by: Zhigao Tong <tongzhigao@pingcap.com>

zanmato1984 approved these changes Jun 2, 2021

View reviewed changes

ti-srebot added the status/LGT1 Indicates that a PR has LGTM 1. label Jun 2, 2021

solotzg merged commit cee3aff into pingcap:master Jun 2, 2021

solotzg mentioned this pull request Jun 2, 2021

Optimize tiflash_multi_branch build #2051

Merged

ti-srebot pushed a commit that referenced this pull request Jun 3, 2021

cherry pick #2017 #2020 #2051 to release-4.0 (#2073)

ad87508

ti-srebot pushed a commit that referenced this pull request Jun 3, 2021

cherry pick #2017 #2020 #2051 to release-5.0 (#2074)

8e0712c

JaySon-Huang added a commit to JaySon-Huang/tiflash that referenced this pull request Jun 8, 2021

Revert "cherry pick pingcap#2017 pingcap#2020 pingcap#2051 to release…

c998f44

…-5.0 (pingcap#2074)" This reverts commit 8e0712c.

JaySon-Huang mentioned this pull request Jun 8, 2021

Check the diff between revert_snapshot and pick need commit JaySon-Huang/tiflash#8

Merged

Conversation

solotzg commented May 28, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What problem does this PR solve?

What is changed and how it works?

Check List

Release note

Uh oh!

solotzg commented May 28, 2021

Uh oh!

solotzg commented May 29, 2021

Uh oh!

JaySon-Huang left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

solotzg commented May 30, 2021

Uh oh!

solotzg commented May 31, 2021

Uh oh!

Uh oh!

solotzg commented May 31, 2021

Uh oh!

solotzg commented May 31, 2021

Uh oh!

solotzg commented May 31, 2021

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

solotzg commented May 31, 2021

Uh oh!

solotzg commented May 31, 2021

Uh oh!

solotzg commented Jun 1, 2021

Uh oh!

solotzg commented Jun 1, 2021

Uh oh!

solotzg commented Jun 1, 2021

Uh oh!

solotzg commented Jun 1, 2021

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

solotzg Jun 1, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

solotzg Jun 1, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

JaySon-Huang left a comment

Choose a reason for hiding this comment

Uh oh!

solotzg commented May 28, 2021 •

edited

Loading

solotzg Jun 1, 2021 •

edited

Loading

solotzg Jun 1, 2021 •

edited

Loading

JaySon-Huang Jun 1, 2021 •

edited

Loading

solotzg Jun 2, 2021 •

edited

Loading