*: use SimSIMD for vectors by Lloyd-Pottiger · Pull Request #9372 · pingcap/tiflash

Lloyd-Pottiger · 2024-08-26T10:13:27Z

What problem does this PR solve?

Issue Number: ref #9032

Problem Summary:

What is changed and how it works?

Pick https://github.com/tidbcloud/tiflash-cse/pull/221, https://github.com/tidbcloud/tiflash-cse/pull/224, https://github.com/tidbcloud/tiflash-cse/pull/262 and https://github.com/tidbcloud/tiflash-cse/pull/245

*: use SimSIMD for vectors

Check List

Tests

Unit test
Integration test
Manual test (add detailed scripts or steps below)
No code

Side effects

Performance regression: Consumes more CPU
Performance regression: Consumes more Memory
Breaking backward compatibility

Documentation

Release note

None

Signed-off-by: Wish <breezewish@outlook.com>

Signed-off-by: Wish <breezewish@outlook.com> Co-authored-by: JaySon <tshent@qq.com>

Signed-off-by: Lloyd-Pottiger <yan1579196623@gmail.com>

JaySon-Huang · 2024-08-27T03:03:07Z

In order to resolve the compile error on gtests_libdaemon below, you need to pick these changes: https://github.com/tidbcloud/tiflash-cse/pull/262 @Lloyd-Pottiger

ld.lld: error: undefined symbol: DB::DM::VectorIndexHNSWSIMDFeatures::get()
>>> referenced by TiFlashBuildInfo.cpp
>>>               TiFlashBuildInfo.cpp.o:(TiFlashBuildInfo::getEnabledFeatures()) in archive dbms/libtiflash_common_iod.a
ld.lld: error: undefined symbol: DB::VectorDistanceSIMDFeatures::get()
>>> referenced by TiFlashBuildInfo.cpp
>>>               TiFlashBuildInfo.cpp.o:(TiFlashBuildInfo::getEnabledFeatures()) in archive dbms/libtiflash_common_iod.a

.devcontainer/README.md

release-centos7-llvm/env/README.md

Signed-off-by: Lloyd-Pottiger <yan1579196623@gmail.com>

JaySon-Huang

LGTM

dbms/src/Storages/DeltaMerge/Index/VectorIndexHNSW/Index.cpp

Signed-off-by: Lloyd-Pottiger <yan1579196623@gmail.com>

dbms/src/TiDB/Decode/Vector.cpp

CalvinNeo · 2024-08-27T07:22:18Z

dbms/src/VectorSearch/DistanceSIMDFeatures.cpp

+    simsimd_capability_t cap_l2 = simsimd_details::actual_capability(simsimd_datatype_f32_k, simsimd_metric_l2sq_k);
+    simsimd_capability_t cap_cos = simsimd_details::actual_capability(simsimd_datatype_f32_k, simsimd_metric_cos_k);
+
+    auto cap_to_string = [](simsimd_capability_t cap) -> std::string {


Could there be cap == simsimd_cap_sve_k & simsimd_cap_sve2_k?

https://github.com/ashvardanian/SimSIMD/blob/96891ce929e3ec6b3600f12c82c383cf1d076548/include/simsimd/simsimd.h#L444

seems impossible.

Then just use == rather than &?
If it could be possible in further versions, then move sve2 in the front of sve? Because sve2 is extension of sve, we should return the "newest"

Signed-off-by: Lloyd-Pottiger <yan1579196623@gmail.com>

CalvinNeo · 2024-08-27T08:14:27Z

libs/libdaemon/src/BaseDaemon.cpp

+
+    std::string line;
+    while (std::getline(ss, line, '\n'))
+        LOG_INFO(log, "{}", line);


What is the difference?

print multiple logs

Signed-off-by: Lloyd-Pottiger <yan1579196623@gmail.com>

CalvinNeo

lgtm

ti-chi-bot · 2024-08-27T08:51:59Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: CalvinNeo, JaySon-Huang

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

~~OWNERS~~ [CalvinNeo,JaySon-Huang]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

ti-chi-bot · 2024-08-27T08:52:00Z

[LGTM Timeline notifier]

Timeline:

2024-08-27 06:15:50.554564015 +0000 UTC m=+850945.689014134: ☑️ agreed by JaySon-Huang.
2024-08-27 08:52:00.222849934 +0000 UTC m=+860315.357300057: ☑️ agreed by CalvinNeo.

Lloyd-Pottiger · 2024-08-27T09:25:12Z

/retest

breezewish and others added 3 commits August 26, 2024 17:59

compute: Use SimSIMD for vectors (pingcap#221)

feda156

Signed-off-by: Wish <breezewish@outlook.com>

storage: Enable simsimd for usearch (pingcap#224)

5e96b35

Signed-off-by: Wish <breezewish@outlook.com> Co-authored-by: JaySon <tshent@qq.com>

revert

2b2beea

Signed-off-by: Lloyd-Pottiger <yan1579196623@gmail.com>

ti-chi-bot bot added release-note-none Denotes a PR that doesn't merit a release note. size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. labels Aug 26, 2024

format

b87cf2c

Signed-off-by: Lloyd-Pottiger <yan1579196623@gmail.com>

JaySon-Huang reviewed Aug 27, 2024

View reviewed changes

.devcontainer/README.md Show resolved Hide resolved

release-centos7-llvm/env/README.md Show resolved Hide resolved

*: Fix compile error under clang 17 (pingcap#262)

302553c

Signed-off-by: Lloyd-Pottiger <yan1579196623@gmail.com>

Lloyd-Pottiger force-pushed the cherry-pick-5 branch from 6aacc40 to 302553c Compare August 27, 2024 06:09

JaySon-Huang approved these changes Aug 27, 2024

View reviewed changes

dbms/src/Storages/DeltaMerge/Index/VectorIndexHNSW/Index.cpp Show resolved Hide resolved

ti-chi-bot bot added needs-1-more-lgtm Indicates a PR needs 1 more LGTM. approved labels Aug 27, 2024

Pick tidbcloud/tiflash-cse#245 to fix UT

4ea7767

Signed-off-by: Lloyd-Pottiger <yan1579196623@gmail.com>

CalvinNeo reviewed Aug 27, 2024

View reviewed changes

dbms/src/TiDB/Decode/Vector.cpp Outdated Show resolved Hide resolved

CalvinNeo reviewed Aug 27, 2024

View reviewed changes

fix race condition

b427bee

Signed-off-by: Lloyd-Pottiger <yan1579196623@gmail.com>

CalvinNeo reviewed Aug 27, 2024

View reviewed changes

Lloyd-Pottiger added 2 commits August 27, 2024 16:18

address comments

d7aadb8

Signed-off-by: Lloyd-Pottiger <yan1579196623@gmail.com>

revert

50d0581

Signed-off-by: Lloyd-Pottiger <yan1579196623@gmail.com>

CalvinNeo approved these changes Aug 27, 2024

View reviewed changes

ti-chi-bot bot added the lgtm label Aug 27, 2024

ti-chi-bot bot removed the needs-1-more-lgtm Indicates a PR needs 1 more LGTM. label Aug 27, 2024

ti-chi-bot bot merged commit 5f08ae6 into pingcap:feature/vector-index Aug 27, 2024

Lloyd-Pottiger deleted the cherry-pick-5 branch August 28, 2024 02:34

JaySon-Huang mentioned this pull request Sep 30, 2024

*: support vector search #9486

Merged

12 tasks

Conversation

Lloyd-Pottiger commented Aug 26, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What problem does this PR solve?

What is changed and how it works?

Check List

Release note

Uh oh!

JaySon-Huang commented Aug 27, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

JaySon-Huang left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

CalvinNeo Aug 27, 2024

Choose a reason for hiding this comment

Uh oh!

Lloyd-Pottiger Aug 27, 2024

Choose a reason for hiding this comment

Uh oh!

CalvinNeo Aug 27, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Lloyd-Pottiger Aug 27, 2024

Choose a reason for hiding this comment

Uh oh!

CalvinNeo Aug 27, 2024

Choose a reason for hiding this comment

Uh oh!

Lloyd-Pottiger Aug 27, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Lloyd-Pottiger Aug 27, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

CalvinNeo left a comment

Choose a reason for hiding this comment

Uh oh!

ti-chi-bot bot commented Aug 27, 2024

Uh oh!

ti-chi-bot bot commented Aug 27, 2024

[LGTM Timeline notifier]

Uh oh!

Lloyd-Pottiger commented Aug 27, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Lloyd-Pottiger commented Aug 26, 2024 •

edited

Loading

JaySon-Huang commented Aug 27, 2024 •

edited

Loading

CalvinNeo Aug 27, 2024 •

edited

Loading

Lloyd-Pottiger Aug 27, 2024 •

edited

Loading

Lloyd-Pottiger Aug 27, 2024 •

edited

Loading