[LV] Disable interleaving via hints for uncountable early exit loops#145877
Merged
Conversation
Currently if the user enables interleaving during vectorisation of uncountable early exit loops via the interleave_count pragma and the enable-early-exit-vectorization option, it will miscompile. There is ongoing work to fix this, but for now it seems safer to ignore the hint until it is supported.
Member
|
@llvm/pr-subscribers-llvm-transforms Author: David Sherwood (david-arm) ChangesCurrently if the user enables interleaving during vectorisation of uncountable early exit loops via the interleave_count pragma and the enable-early-exit-vectorization option, it will miscompile. There is ongoing work to fix this, but for now it seems safer to ignore the hint until it is supported. Full diff: https://github.com/llvm/llvm-project/pull/145877.diff 2 Files Affected:
diff --git a/llvm/lib/Transforms/Vectorize/LoopVectorize.cpp b/llvm/lib/Transforms/Vectorize/LoopVectorize.cpp
index f4259d3d69880..ac65884f341a5 100644
--- a/llvm/lib/Transforms/Vectorize/LoopVectorize.cpp
+++ b/llvm/lib/Transforms/Vectorize/LoopVectorize.cpp
@@ -10101,6 +10101,13 @@ bool LoopVectorizePass::processLoop(Loop *L) {
// Get user vectorization factor and interleave count.
ElementCount UserVF = Hints.getWidth();
unsigned UserIC = Hints.getInterleave();
+ if (LVL.hasUncountableEarlyExit() && UserIC != 1 &&
+ !VectorizerParams::isInterleaveForced()) {
+ UserIC = 1;
+ reportVectorizationInfo("Interleaving not currently supported for loops "
+ "with uncountable early exits",
+ "InterleaveEarlyExitDisabled", ORE, L);
+ }
// Plan how to best vectorize.
LVP.plan(UserVF, UserIC);
diff --git a/llvm/test/Transforms/LoopVectorize/single-early-exit-interleave-hint.ll b/llvm/test/Transforms/LoopVectorize/single-early-exit-interleave-hint.ll
new file mode 100644
index 0000000000000..c00ce09713279
--- /dev/null
+++ b/llvm/test/Transforms/LoopVectorize/single-early-exit-interleave-hint.ll
@@ -0,0 +1,88 @@
+; NOTE: Assertions have been autogenerated by utils/update_test_checks.py UTC_ARGS: --version 5
+; RUN: opt -p loop-vectorize -enable-early-exit-vectorization -force-vector-width=4 \
+; RUN: -debug-only=loop-vectorize -S %s 2>%t | FileCheck --check-prefix=VF4IC4 %s
+; RUN: cat %t | FileCheck --check-prefix=DEBUG %s
+
+declare void @init_mem(ptr, i64);
+
+; DEBUG: Interleaving not currently supported for loops with uncountable early exits
+
+define i64 @multi_exiting_to_different_exits_live_in_exit_values() {
+; VF4IC4-LABEL: define i64 @multi_exiting_to_different_exits_live_in_exit_values() {
+; VF4IC4-NEXT: [[ENTRY:.*]]:
+; VF4IC4-NEXT: [[SRC:%.*]] = alloca [128 x i32], align 4
+; VF4IC4-NEXT: call void @init_mem(ptr [[SRC]])
+; VF4IC4-NEXT: br i1 false, label %[[SCALAR_PH:.*]], label %[[VECTOR_PH:.*]]
+; VF4IC4: [[VECTOR_PH]]:
+; VF4IC4-NEXT: br label %[[VECTOR_BODY:.*]]
+; VF4IC4: [[VECTOR_BODY]]:
+; VF4IC4-NEXT: [[INDEX:%.*]] = phi i64 [ 0, %[[VECTOR_PH]] ], [ [[INDEX_NEXT:%.*]], %[[VECTOR_BODY]] ]
+; VF4IC4-NEXT: [[TMP0:%.*]] = getelementptr inbounds i32, ptr [[SRC]], i64 [[INDEX]]
+; VF4IC4-NEXT: [[TMP1:%.*]] = getelementptr inbounds i32, ptr [[TMP0]], i32 0
+; VF4IC4-NEXT: [[WIDE_LOAD:%.*]] = load <4 x i32>, ptr [[TMP1]], align 4
+; VF4IC4-NEXT: [[TMP2:%.*]] = icmp eq <4 x i32> [[WIDE_LOAD]], splat (i32 10)
+; VF4IC4-NEXT: [[INDEX_NEXT]] = add nuw i64 [[INDEX]], 4
+; VF4IC4-NEXT: [[TMP3:%.*]] = call i1 @llvm.vector.reduce.or.v4i1(<4 x i1> [[TMP2]])
+; VF4IC4-NEXT: [[TMP4:%.*]] = icmp eq i64 [[INDEX_NEXT]], 128
+; VF4IC4-NEXT: [[TMP5:%.*]] = or i1 [[TMP3]], [[TMP4]]
+; VF4IC4-NEXT: br i1 [[TMP5]], label %[[MIDDLE_SPLIT:.*]], label %[[VECTOR_BODY]], !llvm.loop [[LOOP0:![0-9]+]]
+; VF4IC4: [[MIDDLE_SPLIT]]:
+; VF4IC4-NEXT: br i1 [[TMP3]], label %[[VECTOR_EARLY_EXIT:.*]], label %[[MIDDLE_BLOCK:.*]]
+; VF4IC4: [[MIDDLE_BLOCK]]:
+; VF4IC4-NEXT: br i1 true, label %[[E2:.*]], label %[[SCALAR_PH]]
+; VF4IC4: [[VECTOR_EARLY_EXIT]]:
+; VF4IC4-NEXT: br label %[[E1:.*]]
+; VF4IC4: [[SCALAR_PH]]:
+; VF4IC4-NEXT: [[BC_RESUME_VAL:%.*]] = phi i64 [ 128, %[[MIDDLE_BLOCK]] ], [ 0, %[[ENTRY]] ]
+; VF4IC4-NEXT: br label %[[LOOP_HEADER:.*]]
+; VF4IC4: [[LOOP_HEADER]]:
+; VF4IC4-NEXT: [[IV:%.*]] = phi i64 [ [[INC:%.*]], %[[LOOP_LATCH:.*]] ], [ [[BC_RESUME_VAL]], %[[SCALAR_PH]] ]
+; VF4IC4-NEXT: [[GEP_SRC:%.*]] = getelementptr inbounds i32, ptr [[SRC]], i64 [[IV]]
+; VF4IC4-NEXT: [[L:%.*]] = load i32, ptr [[GEP_SRC]], align 4
+; VF4IC4-NEXT: [[C_1:%.*]] = icmp eq i32 [[L]], 10
+; VF4IC4-NEXT: br i1 [[C_1]], label %[[E1]], label %[[LOOP_LATCH]]
+; VF4IC4: [[LOOP_LATCH]]:
+; VF4IC4-NEXT: [[INC]] = add nuw i64 [[IV]], 1
+; VF4IC4-NEXT: [[C_2:%.*]] = icmp eq i64 [[INC]], 128
+; VF4IC4-NEXT: br i1 [[C_2]], label %[[E2]], label %[[LOOP_HEADER]], !llvm.loop [[LOOP3:![0-9]+]]
+; VF4IC4: [[E1]]:
+; VF4IC4-NEXT: [[P1:%.*]] = phi i64 [ 0, %[[LOOP_HEADER]] ], [ 0, %[[VECTOR_EARLY_EXIT]] ]
+; VF4IC4-NEXT: ret i64 [[P1]]
+; VF4IC4: [[E2]]:
+; VF4IC4-NEXT: [[P2:%.*]] = phi i64 [ 1, %[[LOOP_LATCH]] ], [ 1, %[[MIDDLE_BLOCK]] ]
+; VF4IC4-NEXT: ret i64 [[P2]]
+;
+entry:
+ %src = alloca [128 x i32]
+ call void @init_mem(ptr %src)
+ br label %loop.header
+
+loop.header:
+ %iv = phi i64 [ %inc, %loop.latch ], [ 0, %entry ]
+ %gep.src = getelementptr inbounds i32, ptr %src, i64 %iv
+ %l = load i32, ptr %gep.src
+ %c.1 = icmp eq i32 %l, 10
+ br i1 %c.1, label %e1, label %loop.latch
+
+loop.latch:
+ %inc = add nuw i64 %iv, 1
+ %c.2 = icmp eq i64 %inc, 128
+ br i1 %c.2, label %e2, label %loop.header, !llvm.loop !0
+
+e1:
+ %p1 = phi i64 [ 0, %loop.header ]
+ ret i64 %p1
+
+e2:
+ %p2 = phi i64 [ 1, %loop.latch ]
+ ret i64 %p2
+}
+
+!0 = distinct !{!0, !1}
+!1 = !{!"llvm.loop.interleave.count", i32 4}
+;.
+; VF4IC4: [[LOOP0]] = distinct !{[[LOOP0]], [[META1:![0-9]+]], [[META2:![0-9]+]]}
+; VF4IC4: [[META1]] = !{!"llvm.loop.isvectorized", i32 1}
+; VF4IC4: [[META2]] = !{!"llvm.loop.unroll.runtime.disable"}
+; VF4IC4: [[LOOP3]] = distinct !{[[LOOP3]], [[META2]], [[META1]]}
+;.
|
Member
|
@llvm/pr-subscribers-vectorizers Author: David Sherwood (david-arm) ChangesCurrently if the user enables interleaving during vectorisation of uncountable early exit loops via the interleave_count pragma and the enable-early-exit-vectorization option, it will miscompile. There is ongoing work to fix this, but for now it seems safer to ignore the hint until it is supported. Full diff: https://github.com/llvm/llvm-project/pull/145877.diff 2 Files Affected:
diff --git a/llvm/lib/Transforms/Vectorize/LoopVectorize.cpp b/llvm/lib/Transforms/Vectorize/LoopVectorize.cpp
index f4259d3d69880..ac65884f341a5 100644
--- a/llvm/lib/Transforms/Vectorize/LoopVectorize.cpp
+++ b/llvm/lib/Transforms/Vectorize/LoopVectorize.cpp
@@ -10101,6 +10101,13 @@ bool LoopVectorizePass::processLoop(Loop *L) {
// Get user vectorization factor and interleave count.
ElementCount UserVF = Hints.getWidth();
unsigned UserIC = Hints.getInterleave();
+ if (LVL.hasUncountableEarlyExit() && UserIC != 1 &&
+ !VectorizerParams::isInterleaveForced()) {
+ UserIC = 1;
+ reportVectorizationInfo("Interleaving not currently supported for loops "
+ "with uncountable early exits",
+ "InterleaveEarlyExitDisabled", ORE, L);
+ }
// Plan how to best vectorize.
LVP.plan(UserVF, UserIC);
diff --git a/llvm/test/Transforms/LoopVectorize/single-early-exit-interleave-hint.ll b/llvm/test/Transforms/LoopVectorize/single-early-exit-interleave-hint.ll
new file mode 100644
index 0000000000000..c00ce09713279
--- /dev/null
+++ b/llvm/test/Transforms/LoopVectorize/single-early-exit-interleave-hint.ll
@@ -0,0 +1,88 @@
+; NOTE: Assertions have been autogenerated by utils/update_test_checks.py UTC_ARGS: --version 5
+; RUN: opt -p loop-vectorize -enable-early-exit-vectorization -force-vector-width=4 \
+; RUN: -debug-only=loop-vectorize -S %s 2>%t | FileCheck --check-prefix=VF4IC4 %s
+; RUN: cat %t | FileCheck --check-prefix=DEBUG %s
+
+declare void @init_mem(ptr, i64);
+
+; DEBUG: Interleaving not currently supported for loops with uncountable early exits
+
+define i64 @multi_exiting_to_different_exits_live_in_exit_values() {
+; VF4IC4-LABEL: define i64 @multi_exiting_to_different_exits_live_in_exit_values() {
+; VF4IC4-NEXT: [[ENTRY:.*]]:
+; VF4IC4-NEXT: [[SRC:%.*]] = alloca [128 x i32], align 4
+; VF4IC4-NEXT: call void @init_mem(ptr [[SRC]])
+; VF4IC4-NEXT: br i1 false, label %[[SCALAR_PH:.*]], label %[[VECTOR_PH:.*]]
+; VF4IC4: [[VECTOR_PH]]:
+; VF4IC4-NEXT: br label %[[VECTOR_BODY:.*]]
+; VF4IC4: [[VECTOR_BODY]]:
+; VF4IC4-NEXT: [[INDEX:%.*]] = phi i64 [ 0, %[[VECTOR_PH]] ], [ [[INDEX_NEXT:%.*]], %[[VECTOR_BODY]] ]
+; VF4IC4-NEXT: [[TMP0:%.*]] = getelementptr inbounds i32, ptr [[SRC]], i64 [[INDEX]]
+; VF4IC4-NEXT: [[TMP1:%.*]] = getelementptr inbounds i32, ptr [[TMP0]], i32 0
+; VF4IC4-NEXT: [[WIDE_LOAD:%.*]] = load <4 x i32>, ptr [[TMP1]], align 4
+; VF4IC4-NEXT: [[TMP2:%.*]] = icmp eq <4 x i32> [[WIDE_LOAD]], splat (i32 10)
+; VF4IC4-NEXT: [[INDEX_NEXT]] = add nuw i64 [[INDEX]], 4
+; VF4IC4-NEXT: [[TMP3:%.*]] = call i1 @llvm.vector.reduce.or.v4i1(<4 x i1> [[TMP2]])
+; VF4IC4-NEXT: [[TMP4:%.*]] = icmp eq i64 [[INDEX_NEXT]], 128
+; VF4IC4-NEXT: [[TMP5:%.*]] = or i1 [[TMP3]], [[TMP4]]
+; VF4IC4-NEXT: br i1 [[TMP5]], label %[[MIDDLE_SPLIT:.*]], label %[[VECTOR_BODY]], !llvm.loop [[LOOP0:![0-9]+]]
+; VF4IC4: [[MIDDLE_SPLIT]]:
+; VF4IC4-NEXT: br i1 [[TMP3]], label %[[VECTOR_EARLY_EXIT:.*]], label %[[MIDDLE_BLOCK:.*]]
+; VF4IC4: [[MIDDLE_BLOCK]]:
+; VF4IC4-NEXT: br i1 true, label %[[E2:.*]], label %[[SCALAR_PH]]
+; VF4IC4: [[VECTOR_EARLY_EXIT]]:
+; VF4IC4-NEXT: br label %[[E1:.*]]
+; VF4IC4: [[SCALAR_PH]]:
+; VF4IC4-NEXT: [[BC_RESUME_VAL:%.*]] = phi i64 [ 128, %[[MIDDLE_BLOCK]] ], [ 0, %[[ENTRY]] ]
+; VF4IC4-NEXT: br label %[[LOOP_HEADER:.*]]
+; VF4IC4: [[LOOP_HEADER]]:
+; VF4IC4-NEXT: [[IV:%.*]] = phi i64 [ [[INC:%.*]], %[[LOOP_LATCH:.*]] ], [ [[BC_RESUME_VAL]], %[[SCALAR_PH]] ]
+; VF4IC4-NEXT: [[GEP_SRC:%.*]] = getelementptr inbounds i32, ptr [[SRC]], i64 [[IV]]
+; VF4IC4-NEXT: [[L:%.*]] = load i32, ptr [[GEP_SRC]], align 4
+; VF4IC4-NEXT: [[C_1:%.*]] = icmp eq i32 [[L]], 10
+; VF4IC4-NEXT: br i1 [[C_1]], label %[[E1]], label %[[LOOP_LATCH]]
+; VF4IC4: [[LOOP_LATCH]]:
+; VF4IC4-NEXT: [[INC]] = add nuw i64 [[IV]], 1
+; VF4IC4-NEXT: [[C_2:%.*]] = icmp eq i64 [[INC]], 128
+; VF4IC4-NEXT: br i1 [[C_2]], label %[[E2]], label %[[LOOP_HEADER]], !llvm.loop [[LOOP3:![0-9]+]]
+; VF4IC4: [[E1]]:
+; VF4IC4-NEXT: [[P1:%.*]] = phi i64 [ 0, %[[LOOP_HEADER]] ], [ 0, %[[VECTOR_EARLY_EXIT]] ]
+; VF4IC4-NEXT: ret i64 [[P1]]
+; VF4IC4: [[E2]]:
+; VF4IC4-NEXT: [[P2:%.*]] = phi i64 [ 1, %[[LOOP_LATCH]] ], [ 1, %[[MIDDLE_BLOCK]] ]
+; VF4IC4-NEXT: ret i64 [[P2]]
+;
+entry:
+ %src = alloca [128 x i32]
+ call void @init_mem(ptr %src)
+ br label %loop.header
+
+loop.header:
+ %iv = phi i64 [ %inc, %loop.latch ], [ 0, %entry ]
+ %gep.src = getelementptr inbounds i32, ptr %src, i64 %iv
+ %l = load i32, ptr %gep.src
+ %c.1 = icmp eq i32 %l, 10
+ br i1 %c.1, label %e1, label %loop.latch
+
+loop.latch:
+ %inc = add nuw i64 %iv, 1
+ %c.2 = icmp eq i64 %inc, 128
+ br i1 %c.2, label %e2, label %loop.header, !llvm.loop !0
+
+e1:
+ %p1 = phi i64 [ 0, %loop.header ]
+ ret i64 %p1
+
+e2:
+ %p2 = phi i64 [ 1, %loop.latch ]
+ ret i64 %p2
+}
+
+!0 = distinct !{!0, !1}
+!1 = !{!"llvm.loop.interleave.count", i32 4}
+;.
+; VF4IC4: [[LOOP0]] = distinct !{[[LOOP0]], [[META1:![0-9]+]], [[META2:![0-9]+]]}
+; VF4IC4: [[META1]] = !{!"llvm.loop.isvectorized", i32 1}
+; VF4IC4: [[META2]] = !{!"llvm.loop.unroll.runtime.disable"}
+; VF4IC4: [[LOOP3]] = distinct !{[[LOOP3]], [[META2]], [[META1]]}
+;.
|
Co-authored-by: Paul Walker <paul.walker@arm.com>
| ElementCount UserVF = Hints.getWidth(); | ||
| unsigned UserIC = Hints.getInterleave(); | ||
| if (LVL.hasUncountableEarlyExit() && UserIC != 1 && | ||
| !VectorizerParams::isInterleaveForced()) { |
Contributor
There was a problem hiding this comment.
Based on #133099 (review) Florian would prefer not to be able to override this. Do we have tests that specifically care about interleaving? or can the restriction be removed?
Contributor
Author
There was a problem hiding this comment.
I think the problem then is that it becomes extremely difficult to write IR tests for any fixes. Any fixes would have to be unit tests.
Contributor
There was a problem hiding this comment.
For now this should be fine, if interleaving isn't fixed before the release-branch we should disable for all cases where UserIC != 1.
paulwalker-arm
approved these changes
Jun 26, 2025
fhahn
approved these changes
Jun 26, 2025
| ElementCount UserVF = Hints.getWidth(); | ||
| unsigned UserIC = Hints.getInterleave(); | ||
| if (LVL.hasUncountableEarlyExit() && UserIC != 1 && | ||
| !VectorizerParams::isInterleaveForced()) { |
Contributor
There was a problem hiding this comment.
For now this should be fine, if interleaving isn't fixed before the release-branch we should disable for all cases where UserIC != 1.
Contributor
Author
|
I suspect I've broken a post-commit build because I've forgotten (again) to add "REQUIRES: asserts". :face_palm. I'll fix it asap. |
Contributor
may be better to check the remark, if at all; they should also be available w/o assertions |
david-arm
added a commit
to david-arm/llvm-project
that referenced
this pull request
Jun 27, 2025
david-arm
added a commit
that referenced
this pull request
Jun 27, 2025
Contributor
Author
Yes, that's probably even better! I hadn't thought of that. I've already committed a patch to add "REQUIRES: asserts" to fix the build quickly, but I'm happy to follow this up with a new patch to check the remark. |
Contributor
Author
|
I was sure I'd fixed test build failures with #146041, however I still saw this https://lab.llvm.org/buildbot/#/builders/63/builds/7492 over the weekend. I'll try to find out what's going on. |
fhahn
added a commit
to fhahn/llvm-project
that referenced
this pull request
Jul 26, 2025
This patch adds a new ExtractLane VPInstruction which extracts across multiple parts using a wide index, to be used in combination with FirstActiveLane. The patch updates early-exit codegen to use it instead ExtractElement, which is only per-part. With this change, interleaving should work correctly with early-exit loops. The patch removes the restrictions added in 6f43754 (llvm#145877), but does not yet automatically select interleave counts > 1 for early-exit loops. I'll share a patch as follow-up. The cost of extracting a lane adds non-trivial overhead in the exit block, so that should be considered when picking the interleave count.
fhahn
added a commit
that referenced
this pull request
Jul 27, 2025
…8817) This patch adds a new ExtractLane VPInstruction which extracts across multiple parts using a wide index, to be used in combination with FirstActiveLane. The patch updates early-exit codegen to use it instead ExtractElement, which is only per-part. With this change, interleaving should work correctly with early-exit loops. The patch removes the restrictions added in 6f43754 (#145877), but does not yet automatically select interleave counts > 1 for early-exit loops. I'll share a patch as follow-up. The cost of extracting a lane adds non-trivial overhead in the exit block, so that should be considered when picking the interleave count. PR: #148817
mahesh-attarde
pushed a commit
to mahesh-attarde/llvm-project
that referenced
this pull request
Jul 28, 2025
…m#148817) This patch adds a new ExtractLane VPInstruction which extracts across multiple parts using a wide index, to be used in combination with FirstActiveLane. The patch updates early-exit codegen to use it instead ExtractElement, which is only per-part. With this change, interleaving should work correctly with early-exit loops. The patch removes the restrictions added in 6f43754 (llvm#145877), but does not yet automatically select interleave counts > 1 for early-exit loops. I'll share a patch as follow-up. The cost of extracting a lane adds non-trivial overhead in the exit block, so that should be considered when picking the interleave count. PR: llvm#148817
cardigan1008
pushed a commit
to cardigan1008/llvm-project
that referenced
this pull request
Mar 28, 2026
…m#148817) This patch adds a new ExtractLane VPInstruction which extracts across multiple parts using a wide index, to be used in combination with FirstActiveLane. The patch updates early-exit codegen to use it instead ExtractElement, which is only per-part. With this change, interleaving should work correctly with early-exit loops. The patch removes the restrictions added in 6f43754 (llvm#145877), but does not yet automatically select interleave counts > 1 for early-exit loops. I'll share a patch as follow-up. The cost of extracting a lane adds non-trivial overhead in the exit block, so that should be considered when picking the interleave count. PR: llvm#148817
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
5 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Currently if the user enables interleaving during vectorisation of uncountable early exit loops via the interleave_count pragma and the enable-early-exit-vectorization option, it will miscompile. There is ongoing work to fix this, but for now it seems safer to ignore the hint until it is supported.