[VPlan] Remove constant branches early.#183397
Merged
Merged
Conversation
Member
|
@llvm/pr-subscribers-vectorizers @llvm/pr-subscribers-backend-risc-v Author: Florian Hahn (fhahn) ChangesSimplify constant branches early, after introducing the check in the This removes any trivial branches in the input CFG (e.g. over-reduced This requires to remove recipes in the new unreachable blocks, as otherwise we would fail during verification, due to uses in unreachable blocks. Alternatively, we may be able to skip verification for uses in unreachable blocks. Depends on #181252 (included in PR) Patch is 238.99 KiB, truncated to 20.00 KiB below, full version: https://github.com/llvm/llvm-project/pull/183397.diff 36 Files Affected:
diff --git a/llvm/lib/Transforms/Vectorize/LoopVectorize.cpp b/llvm/lib/Transforms/Vectorize/LoopVectorize.cpp
index 0fd425c23c7aa..f545681d1b545 100644
--- a/llvm/lib/Transforms/Vectorize/LoopVectorize.cpp
+++ b/llvm/lib/Transforms/Vectorize/LoopVectorize.cpp
@@ -7364,21 +7364,41 @@ static void fixReductionScalarResumeWhenVectorizingEpilog(
vputils::findRecipe(BackedgeVal, IsaPred<VPReductionPHIRecipe>));
if (!EpiRedHeaderPhi) {
match(BackedgeVal,
- VPlanPatternMatch::m_Select(VPlanPatternMatch::m_VPValue(),
- VPlanPatternMatch::m_VPValue(BackedgeVal),
- VPlanPatternMatch::m_VPValue()));
- EpiRedHeaderPhi = cast<VPReductionPHIRecipe>(
+ m_Select(m_VPValue(), m_VPValue(BackedgeVal), m_VPValue()));
+ EpiRedHeaderPhi = cast_if_present<VPReductionPHIRecipe>(
vputils::findRecipe(BackedgeVal, IsaPred<VPReductionPHIRecipe>));
}
+ // Look through Broadcast or ReductionStartVector to get the underlying
+ // start value.
+ auto GetStartValue = [&](VPValue *V) -> Value * {
+ VPValue *Start;
+ if (match(V, m_VPInstruction<VPInstruction::ReductionStartVector>(
+ m_VPValue(Start), m_VPValue(), m_VPValue())) ||
+ match(V, m_Broadcast(m_VPValue(Start))))
+ return Start->getUnderlyingValue();
+ return V->getUnderlyingValue();
+ };
+
Value *MainResumeValue;
- if (auto *VPI = dyn_cast<VPInstruction>(EpiRedHeaderPhi->getStartValue())) {
- assert((VPI->getOpcode() == VPInstruction::Broadcast ||
- VPI->getOpcode() == VPInstruction::ReductionStartVector) &&
- "unexpected start recipe");
- MainResumeValue = VPI->getOperand(0)->getUnderlyingValue();
- } else
- MainResumeValue = EpiRedHeaderPhi->getStartValue()->getUnderlyingValue();
+ if (EpiRedHeaderPhi) {
+ MainResumeValue = GetStartValue(EpiRedHeaderPhi->getStartValue());
+ } else {
+ // The epilogue vector loop was dissolved (single-iteration). The
+ // reduction header phi was replaced by its start value. Look for a
+ // Broadcast or ReductionStartVector in BackedgeVal or its operands.
+ MainResumeValue = GetStartValue(BackedgeVal);
+ if (MainResumeValue == BackedgeVal->getUnderlyingValue() &&
+ BackedgeVal->getDefiningRecipe()) {
+ for (VPValue *Op : BackedgeVal->getDefiningRecipe()->operands()) {
+ Value *V = GetStartValue(Op);
+ if (V != Op->getUnderlyingValue()) {
+ MainResumeValue = V;
+ break;
+ }
+ }
+ }
+ }
if (EpiRedResult->getOpcode() == VPInstruction::ComputeAnyOfResult) {
[[maybe_unused]] Value *StartV =
EpiRedResult->getOperand(0)->getLiveInIRValue();
@@ -7460,6 +7480,8 @@ DenseMap<const SCEV *, Value *> LoopVectorizationPlanner::executePlan(
VPlanTransforms::expandBranchOnTwoConds(BestVPlan);
// Convert loops with variable-length stepping after regions are dissolved.
VPlanTransforms::convertToVariableLengthStep(BestVPlan);
+ // Remove dead edges for single-iteration loops with BranchOnCond(true).
+ VPlanTransforms::removeBranchOnConst(BestVPlan);
VPlanTransforms::materializeBackedgeTakenCount(BestVPlan, VectorPH);
VPlanTransforms::materializeVectorTripCount(
BestVPlan, VectorPH, CM.foldTailByMasking(),
@@ -7491,8 +7513,9 @@ DenseMap<const SCEV *, Value *> LoopVectorizationPlanner::executePlan(
// 1. Set up the skeleton for vectorization, including vector pre-header and
// middle block. The vector loop is created during VPlan execution.
State.CFG.PrevBB = ILV.createVectorizedLoopSkeleton();
- replaceVPBBWithIRVPBB(BestVPlan.getScalarPreheader(),
- State.CFG.PrevBB->getSingleSuccessor(), &BestVPlan);
+ if (VPBasicBlock *ScalarPH = BestVPlan.getScalarPreheader())
+ replaceVPBBWithIRVPBB(ScalarPH, State.CFG.PrevBB->getSingleSuccessor(),
+ &BestVPlan);
VPlanTransforms::removeDeadRecipes(BestVPlan);
assert(verifyVPlanIsValid(BestVPlan) && "final VPlan is invalid");
@@ -8176,6 +8199,7 @@ VPlanPtr LoopVectorizationPlanner::tryToBuildVPlanWithVPRecipes(
VPlanTransforms::handleEarlyExits(*Plan, Legal->hasUncountableEarlyExit());
VPlanTransforms::addMiddleCheck(*Plan, RequiresScalarEpilogueCheck,
CM.foldTailByMasking());
+ VPlanTransforms::removeBranchOnConst(*Plan);
VPlanTransforms::createLoopRegions(*Plan);
diff --git a/llvm/lib/Transforms/Vectorize/VPlan.cpp b/llvm/lib/Transforms/Vectorize/VPlan.cpp
index f233f0dc1b025..26d9eda5e3b23 100644
--- a/llvm/lib/Transforms/Vectorize/VPlan.cpp
+++ b/llvm/lib/Transforms/Vectorize/VPlan.cpp
@@ -946,7 +946,7 @@ void VPlan::execute(VPTransformState *State) {
BasicBlock *ScalarPh = State->CFG.ExitBB;
VPBasicBlock *ScalarPhVPBB = getScalarPreheader();
- if (ScalarPhVPBB->hasPredecessors()) {
+ if (ScalarPhVPBB) {
// Disconnect scalar preheader and scalar header, as the dominator tree edge
// will be updated as part of VPlan execution. This allows keeping the DTU
// logic generic during VPlan execution.
@@ -961,7 +961,7 @@ void VPlan::execute(VPTransformState *State) {
Block->execute(State);
// If the original loop is unreachable, delete it and all its blocks.
- if (!ScalarPhVPBB->hasPredecessors()) {
+ if (!ScalarPhVPBB) {
// DeleteDeadBlocks will remove single-entry phis. Remove them from the exit
// VPIRBBs in VPlan as well, otherwise we would retain references to deleted
// IR instructions.
@@ -975,7 +975,7 @@ void VPlan::execute(VPTransformState *State) {
Loop *OrigLoop =
State->LI->getLoopFor(getScalarHeader()->getIRBasicBlock());
auto Blocks = OrigLoop->getBlocksVector();
- Blocks.push_back(cast<VPIRBasicBlock>(ScalarPhVPBB)->getIRBasicBlock());
+ Blocks.push_back(ScalarPh);
for (auto *BB : Blocks)
State->LI->removeBlock(BB);
DeleteDeadBlocks(Blocks, &State->CFG.DTU);
@@ -1661,7 +1661,8 @@ void LoopVectorizationPlanner::updateLoopMetadataAndProfileInfo(
// Update the metadata of the scalar loop. Skip the update when vectorizing
// the epilogue loop to ensure it is updated only once. Also skip the update
// when the scalar loop became unreachable.
- if (Plan.getScalarPreheader()->hasPredecessors() && !VectorizingEpilogue) {
+ auto *ScalarPH = Plan.getScalarPreheader();
+ if (ScalarPH && !VectorizingEpilogue) {
std::optional<MDNode *> RemainderLoopID =
makeFollowupLoopID(OrigLoopID, {LLVMLoopVectorizeFollowupAll,
LLVMLoopVectorizeFollowupEpilogue});
@@ -1723,7 +1724,7 @@ void LoopVectorizationPlanner::updateLoopMetadataAndProfileInfo(
AverageVectorTripCount = SE.getSmallConstantTripCount(VectorLoop);
if (ProfcheckDisableMetadataFixes || !AverageVectorTripCount)
return;
- if (Plan.getScalarPreheader()->hasPredecessors())
+ if (ScalarPH)
RemainderAverageTripCount =
SE.getSmallConstantTripCount(OrigLoop) % EstimatedVFxUF;
// Setting to 1 should be sufficient to generate the correct branch weights.
@@ -1739,7 +1740,7 @@ void LoopVectorizationPlanner::updateLoopMetadataAndProfileInfo(
OrigLoopInvocationWeight);
}
- if (Plan.getScalarPreheader()->hasPredecessors()) {
+ if (ScalarPH) {
setLoopEstimatedTripCount(OrigLoop, RemainderAverageTripCount,
OrigLoopInvocationWeight);
}
diff --git a/llvm/lib/Transforms/Vectorize/VPlan.h b/llvm/lib/Transforms/Vectorize/VPlan.h
index 7455129ff91e8..9732f553f3309 100644
--- a/llvm/lib/Transforms/Vectorize/VPlan.h
+++ b/llvm/lib/Transforms/Vectorize/VPlan.h
@@ -4624,7 +4624,8 @@ class VPlan {
/// Return the VPBasicBlock for the preheader of the scalar loop.
VPBasicBlock *getScalarPreheader() const {
- return cast<VPBasicBlock>(getScalarHeader()->getSinglePredecessor());
+ return dyn_cast_or_null<VPBasicBlock>(
+ getScalarHeader()->getSinglePredecessor());
}
/// Return the VPIRBasicBlock wrapping the header of the scalar loop.
@@ -4869,8 +4870,8 @@ class VPlan {
/// that this relies on unneeded branches to the scalar tail loop being
/// removed.
bool hasScalarTail() const {
- return !(!getScalarPreheader()->hasPredecessors() ||
- getScalarPreheader()->getSinglePredecessor() == getEntry());
+ auto *ScalarPH = getScalarPreheader();
+ return ScalarPH && ScalarPH->getSinglePredecessor() != getEntry();
}
};
diff --git a/llvm/lib/Transforms/Vectorize/VPlanConstruction.cpp b/llvm/lib/Transforms/Vectorize/VPlanConstruction.cpp
index 3101b42625c53..af2ae283e8092 100644
--- a/llvm/lib/Transforms/Vectorize/VPlanConstruction.cpp
+++ b/llvm/lib/Transforms/Vectorize/VPlanConstruction.cpp
@@ -1288,32 +1288,34 @@ bool VPlanTransforms::handleMaxMinNumReductions(VPlan &Plan) {
// Update resume phis for inductions in the scalar preheader. If AnyNaNLane is
// true, the resume from the start of the last vector iteration via the
// canonical IV, otherwise from the original value.
- for (auto &R : Plan.getScalarPreheader()->phis()) {
- auto *ResumeR = cast<VPPhi>(&R);
- VPValue *VecV = ResumeR->getOperand(0);
- if (RdxResults.contains(VecV))
- continue;
- if (auto *DerivedIV = dyn_cast<VPDerivedIVRecipe>(VecV)) {
- if (DerivedIV->getNumUsers() == 1 &&
- DerivedIV->getOperand(1) == &Plan.getVectorTripCount()) {
- auto *NewSel =
- MiddleBuilder.createSelect(AnyNaNLane, LoopRegion->getCanonicalIV(),
- &Plan.getVectorTripCount());
- DerivedIV->moveAfter(&*MiddleBuilder.getInsertPoint());
- DerivedIV->setOperand(1, NewSel);
+ if (auto *ScalarPH = Plan.getScalarPreheader()) {
+ for (auto &R : ScalarPH->phis()) {
+ auto *ResumeR = cast<VPPhi>(&R);
+ VPValue *VecV = ResumeR->getOperand(0);
+ if (RdxResults.contains(VecV))
continue;
+ if (auto *DerivedIV = dyn_cast<VPDerivedIVRecipe>(VecV)) {
+ if (DerivedIV->getNumUsers() == 1 &&
+ DerivedIV->getOperand(1) == &Plan.getVectorTripCount()) {
+ auto *NewSel =
+ MiddleBuilder.createSelect(AnyNaNLane, LoopRegion->getCanonicalIV(),
+ &Plan.getVectorTripCount());
+ DerivedIV->moveAfter(&*MiddleBuilder.getInsertPoint());
+ DerivedIV->setOperand(1, NewSel);
+ continue;
+ }
}
+ // Bail out and abandon the current, partially modified, VPlan if we
+ // encounter resume phi that cannot be updated yet.
+ if (VecV != &Plan.getVectorTripCount()) {
+ LLVM_DEBUG(dbgs() << "Found resume phi we cannot update for VPlan with "
+ "FMaxNum/FMinNum reduction.\n");
+ return false;
+ }
+ auto *NewSel = MiddleBuilder.createSelect(
+ AnyNaNLane, LoopRegion->getCanonicalIV(), VecV);
+ ResumeR->setOperand(0, NewSel);
}
- // Bail out and abandon the current, partially modified, VPlan if we
- // encounter resume phi that cannot be updated yet.
- if (VecV != &Plan.getVectorTripCount()) {
- LLVM_DEBUG(dbgs() << "Found resume phi we cannot update for VPlan with "
- "FMaxNum/FMinNum reduction.\n");
- return false;
- }
- auto *NewSel = MiddleBuilder.createSelect(
- AnyNaNLane, LoopRegion->getCanonicalIV(), VecV);
- ResumeR->setOperand(0, NewSel);
}
auto *MiddleTerm = MiddleVPBB->getTerminator();
diff --git a/llvm/lib/Transforms/Vectorize/VPlanPredicator.cpp b/llvm/lib/Transforms/Vectorize/VPlanPredicator.cpp
index dbc2e71c785ee..2e41dbbad8bf5 100644
--- a/llvm/lib/Transforms/Vectorize/VPlanPredicator.cpp
+++ b/llvm/lib/Transforms/Vectorize/VPlanPredicator.cpp
@@ -325,8 +325,11 @@ void VPlanTransforms::introduceMasksAndLinearize(VPlan &Plan, bool FoldTail) {
Plan.getMiddleBlock() &&
"the exit block must have middle block as single predecessor");
- VPBuilder B(Plan.getMiddleBlock()->getTerminator());
- for (VPRecipeBase &R : *Plan.getMiddleBlock()) {
+ auto *MiddleVPBB = Plan.getMiddleBlock();
+ auto *Term = MiddleVPBB->getTerminator();
+ auto IP = Term ? Term->getIterator() : MiddleVPBB->end();
+ VPBuilder B(MiddleVPBB, IP);
+ for (VPRecipeBase &R : *MiddleVPBB) {
VPValue *Op;
if (!match(&R, m_CombineOr(
m_ExitingIVValue(m_VPValue(), m_VPValue(Op)),
diff --git a/llvm/lib/Transforms/Vectorize/VPlanRecipes.cpp b/llvm/lib/Transforms/Vectorize/VPlanRecipes.cpp
index f7810c65d000e..d122bda19bb15 100644
--- a/llvm/lib/Transforms/Vectorize/VPlanRecipes.cpp
+++ b/llvm/lib/Transforms/Vectorize/VPlanRecipes.cpp
@@ -128,6 +128,7 @@ bool VPRecipeBase::mayReadFromMemory() const {
return cast<VPWidenIntrinsicRecipe>(this)->mayReadFromMemory();
case VPBranchOnMaskSC:
case VPDerivedIVSC:
+ case VPCurrentIterationPHISC:
case VPFirstOrderRecurrencePHISC:
case VPReductionPHISC:
case VPPredInstPHISC:
@@ -165,6 +166,7 @@ bool VPRecipeBase::mayHaveSideEffects() const {
return cast<VPExpressionRecipe>(this)->mayHaveSideEffects();
case VPActiveLaneMaskPHISC:
case VPDerivedIVSC:
+ case VPCurrentIterationPHISC:
case VPFirstOrderRecurrencePHISC:
case VPReductionPHISC:
case VPPredInstPHISC:
diff --git a/llvm/lib/Transforms/Vectorize/VPlanTransforms.cpp b/llvm/lib/Transforms/Vectorize/VPlanTransforms.cpp
index c16149303b180..3e9da6a5a4204 100644
--- a/llvm/lib/Transforms/Vectorize/VPlanTransforms.cpp
+++ b/llvm/lib/Transforms/Vectorize/VPlanTransforms.cpp
@@ -1467,12 +1467,6 @@ static void simplifyRecipe(VPSingleDefRecipe *Def, VPTypeAnalysis &TypeInfo) {
return;
}
- if (auto *Phi = dyn_cast<VPFirstOrderRecurrencePHIRecipe>(Def)) {
- if (Phi->getOperand(0) == Phi->getOperand(1))
- Phi->replaceAllUsesWith(Phi->getOperand(0));
- return;
- }
-
// Simplify MaskedCond with no block mask to its single operand.
if (match(Def, m_VPInstruction<VPInstruction::MaskedCond>()) &&
!cast<VPInstruction>(Def)->isMasked())
@@ -1521,9 +1515,15 @@ static void simplifyRecipe(VPSingleDefRecipe *Def, VPTypeAnalysis &TypeInfo) {
return;
}
- if (isa<VPPhi, VPWidenPHIRecipe>(Def)) {
- if (Def->getNumOperands() == 1)
+ if (isa<VPPhi, VPWidenPHIRecipe, VPHeaderPHIRecipe>(Def)) {
+ if (Def->getNumOperands() == 1) {
Def->replaceAllUsesWith(Def->getOperand(0));
+ return;
+ }
+ if (auto *Phi = dyn_cast<VPFirstOrderRecurrencePHIRecipe>(Def)) {
+ if (Phi->getOperand(0) == Phi->getOperand(1))
+ Phi->replaceAllUsesWith(Phi->getOperand(0));
+ }
return;
}
@@ -2095,72 +2095,16 @@ static bool simplifyBranchConditionForVFAndUF(VPlan &Plan, ElementCount BestVF,
return false;
}
- // The vector loop region only executes once. If possible, completely remove
- // the region, otherwise replace the terminator controlling the latch with
- // (BranchOnCond true).
- // TODO: VPWidenIntOrFpInductionRecipe is only partially supported; add
- // support for other non-canonical widen induction recipes (e.g.,
- // VPWidenPointerInductionRecipe).
- // TODO: fold branch-on-constant after dissolving region.
- auto *Header = cast<VPBasicBlock>(VectorRegion->getEntry());
- if (all_of(Header->phis(), [](VPRecipeBase &Phi) {
- if (auto *R = dyn_cast<VPWidenIntOrFpInductionRecipe>(&Phi))
- return R->isCanonical();
- return isa<VPCanonicalIVPHIRecipe, VPCurrentIterationPHIRecipe,
- VPFirstOrderRecurrencePHIRecipe, VPPhi>(&Phi);
- })) {
- for (VPRecipeBase &HeaderR : make_early_inc_range(Header->phis())) {
- if (auto *R = dyn_cast<VPWidenIntOrFpInductionRecipe>(&HeaderR)) {
- VPBuilder Builder(Plan.getVectorPreheader());
- VPValue *StepV = Builder.createNaryOp(VPInstruction::StepVector, {},
- R->getScalarType());
- HeaderR.getVPSingleValue()->replaceAllUsesWith(StepV);
- HeaderR.eraseFromParent();
- continue;
- }
- auto *Phi = cast<VPPhiAccessors>(&HeaderR);
- HeaderR.getVPSingleValue()->replaceAllUsesWith(Phi->getIncomingValue(0));
- HeaderR.eraseFromParent();
- }
-
- VPBlockBase *Preheader = VectorRegion->getSinglePredecessor();
- SmallVector<VPBlockBase *> Exits = to_vector(VectorRegion->getSuccessors());
- VPBlockUtils::disconnectBlocks(Preheader, VectorRegion);
- for (VPBlockBase *Exit : Exits)
- VPBlockUtils::disconnectBlocks(VectorRegion, Exit);
-
- for (VPBlockBase *B : vp_depth_first_shallow(VectorRegion->getEntry()))
- B->setParent(nullptr);
-
- VPBlockUtils::connectBlocks(Preheader, Header);
-
- for (VPBlockBase *Exit : Exits)
- VPBlockUtils::connectBlocks(ExitingVPBB, Exit);
-
- // Replace terminating branch-on-two-conds with branch-on-cond to early
- // exit.
- if (Exits.size() != 1) {
- assert(match(Term, m_BranchOnTwoConds()) && Exits.size() == 2 &&
- "BranchOnTwoConds needs 2 remaining exits");
- VPBuilder(Term).createNaryOp(VPInstruction::BranchOnCond,
- Term->getOperand(0));
- }
- VPlanTransforms::simplifyRecipes(Plan);
- } else {
- // The vector region contains header phis for which we cannot remove the
- // loop region yet.
-
- // For BranchOnTwoConds, set the latch exit condition to true directly.
- if (match(Term, m_BranchOnTwoConds())) {
- Term->setOperand(1, Plan.getTrue());
- return true;
- }
-
- auto *BOC = new VPInstruction(VPInstruction::BranchOnCond, {Plan.getTrue()},
- {}, {}, Term->getDebugLoc());
- ExitingVPBB->appendRecipe(BOC);
+ // The vector loop region only executes once. Convert terminator of the
+ // exiting block to exit in the first iteration.
+ if (match(Term, m_BranchOnTwoConds())) {
+ Term->setOperand(1, Plan.getTrue());
+ return true;
}
+ auto *BOC = new VPInstruction(VPInstruction::BranchOnCond, Plan.getTrue(), {},
+ {}, Term->getDebugLoc());
+ ExitingVPBB->appendRecipe(BOC);
Term->eraseFromParent();
return true;
@@ -2760,6 +2704,7 @@ void VPlanTransforms::truncateToMinimalBitwidths(
}
void VPlanTransforms::removeBranchOnConst(VPlan &Plan) {
+ SmallVector<VPBlockBase *> DeadSuccs;
for (VPBasicBlock *VPBB : VPBlockUtils::blocksOnly<VPBasicBlock>(
vp_depth_first_shallow(Plan.getEntry()))) {
VPValue *Cond;
@@ -2781,15 +2726,43 @@ void VPlanTransforms::removeBranchOnConst(VPlan &Plan) {
cast<VPBasicBlock>(VPBB->getSuccessors()[RemovedIdx]);
assert(count(RemovedSucc->getPredecessors(), VPBB) == 1 &&
"There must be a single edge between VPBB and its successor");
- // Values coming from VPBB into phi recipes of RemoveSucc are removed from
+ // Values coming from VPBB into phi recipes of RemovedSucc are removed from
// these recipes.
for (VPRecipeBase &R : RemovedSucc->phis())
cast<VPPhiAccessors>(&R)->removeIncomingValueFor(VPBB);
- // Disconnect blocks and remove the terminator. RemovedSucc will be deleted
- // automatically on VPlan destruction if it becomes unreachable.
+ // Disconnect blocks and remove the terminator.
VPBlockUtils::disconnectBlocks(VPBB, RemovedSucc);
VPBB->back().eraseFromParent();
+ DeadSuccs.push_back(RemovedSucc);
+ }
+
+ // Transitively disconnect all blocks that became unreachable, removing
+ // incoming values from phi recipes and replacing remaining uses with poison.
+ VPTypeAnalysis TypeInfo(Plan);
+ while (!DeadSuccs.empty()) {
+ VPBlockBase *DeadBlock = DeadSuccs.pop_back_val();
+ if (!DeadBlock->getPredecessors().empty())
+ continue;
+ for (VPBlockBase *Succ :
+ SmallVector<VPBlockBase *>(DeadBlock->successors())) {
+ if (auto *SuccBB = dyn_cast<VPBasicBlock>(Succ))
+ for (VPRecipeBase &R : SuccBB->phis())
+ cast<VPPhiAccessors>(&R)->removeIncomingValueFor(DeadBlock);
+ VPBlockUtils::disconnectBlocks(DeadBlock, Succ);
+ if (Succ->getPredecessors().empty())
+ DeadSuccs.push_back(Succ);
+ }
+ auto *DeadBB = dyn_cast<VPBasicBlock>(DeadBlock);
+ if (!DeadBB)
+ continue;
+ for (VPRecipeBase &R : make_early_inc_range(*DeadBB)) {
+ fo...
[truncated]
|
Member
|
@llvm/pr-subscribers-llvm-transforms Author: Florian Hahn (fhahn) ChangesSimplify constant branches early, after introducing the check in the This removes any trivial branches in the input CFG (e.g. over-reduced This requires to remove recipes in the new unreachable blocks, as otherwise we would fail during verification, due to uses in unreachable blocks. Alternatively, we may be able to skip verification for uses in unreachable blocks. Depends on #181252 (included in PR) Patch is 238.99 KiB, truncated to 20.00 KiB below, full version: https://github.com/llvm/llvm-project/pull/183397.diff 36 Files Affected:
diff --git a/llvm/lib/Transforms/Vectorize/LoopVectorize.cpp b/llvm/lib/Transforms/Vectorize/LoopVectorize.cpp
index 0fd425c23c7aa..f545681d1b545 100644
--- a/llvm/lib/Transforms/Vectorize/LoopVectorize.cpp
+++ b/llvm/lib/Transforms/Vectorize/LoopVectorize.cpp
@@ -7364,21 +7364,41 @@ static void fixReductionScalarResumeWhenVectorizingEpilog(
vputils::findRecipe(BackedgeVal, IsaPred<VPReductionPHIRecipe>));
if (!EpiRedHeaderPhi) {
match(BackedgeVal,
- VPlanPatternMatch::m_Select(VPlanPatternMatch::m_VPValue(),
- VPlanPatternMatch::m_VPValue(BackedgeVal),
- VPlanPatternMatch::m_VPValue()));
- EpiRedHeaderPhi = cast<VPReductionPHIRecipe>(
+ m_Select(m_VPValue(), m_VPValue(BackedgeVal), m_VPValue()));
+ EpiRedHeaderPhi = cast_if_present<VPReductionPHIRecipe>(
vputils::findRecipe(BackedgeVal, IsaPred<VPReductionPHIRecipe>));
}
+ // Look through Broadcast or ReductionStartVector to get the underlying
+ // start value.
+ auto GetStartValue = [&](VPValue *V) -> Value * {
+ VPValue *Start;
+ if (match(V, m_VPInstruction<VPInstruction::ReductionStartVector>(
+ m_VPValue(Start), m_VPValue(), m_VPValue())) ||
+ match(V, m_Broadcast(m_VPValue(Start))))
+ return Start->getUnderlyingValue();
+ return V->getUnderlyingValue();
+ };
+
Value *MainResumeValue;
- if (auto *VPI = dyn_cast<VPInstruction>(EpiRedHeaderPhi->getStartValue())) {
- assert((VPI->getOpcode() == VPInstruction::Broadcast ||
- VPI->getOpcode() == VPInstruction::ReductionStartVector) &&
- "unexpected start recipe");
- MainResumeValue = VPI->getOperand(0)->getUnderlyingValue();
- } else
- MainResumeValue = EpiRedHeaderPhi->getStartValue()->getUnderlyingValue();
+ if (EpiRedHeaderPhi) {
+ MainResumeValue = GetStartValue(EpiRedHeaderPhi->getStartValue());
+ } else {
+ // The epilogue vector loop was dissolved (single-iteration). The
+ // reduction header phi was replaced by its start value. Look for a
+ // Broadcast or ReductionStartVector in BackedgeVal or its operands.
+ MainResumeValue = GetStartValue(BackedgeVal);
+ if (MainResumeValue == BackedgeVal->getUnderlyingValue() &&
+ BackedgeVal->getDefiningRecipe()) {
+ for (VPValue *Op : BackedgeVal->getDefiningRecipe()->operands()) {
+ Value *V = GetStartValue(Op);
+ if (V != Op->getUnderlyingValue()) {
+ MainResumeValue = V;
+ break;
+ }
+ }
+ }
+ }
if (EpiRedResult->getOpcode() == VPInstruction::ComputeAnyOfResult) {
[[maybe_unused]] Value *StartV =
EpiRedResult->getOperand(0)->getLiveInIRValue();
@@ -7460,6 +7480,8 @@ DenseMap<const SCEV *, Value *> LoopVectorizationPlanner::executePlan(
VPlanTransforms::expandBranchOnTwoConds(BestVPlan);
// Convert loops with variable-length stepping after regions are dissolved.
VPlanTransforms::convertToVariableLengthStep(BestVPlan);
+ // Remove dead edges for single-iteration loops with BranchOnCond(true).
+ VPlanTransforms::removeBranchOnConst(BestVPlan);
VPlanTransforms::materializeBackedgeTakenCount(BestVPlan, VectorPH);
VPlanTransforms::materializeVectorTripCount(
BestVPlan, VectorPH, CM.foldTailByMasking(),
@@ -7491,8 +7513,9 @@ DenseMap<const SCEV *, Value *> LoopVectorizationPlanner::executePlan(
// 1. Set up the skeleton for vectorization, including vector pre-header and
// middle block. The vector loop is created during VPlan execution.
State.CFG.PrevBB = ILV.createVectorizedLoopSkeleton();
- replaceVPBBWithIRVPBB(BestVPlan.getScalarPreheader(),
- State.CFG.PrevBB->getSingleSuccessor(), &BestVPlan);
+ if (VPBasicBlock *ScalarPH = BestVPlan.getScalarPreheader())
+ replaceVPBBWithIRVPBB(ScalarPH, State.CFG.PrevBB->getSingleSuccessor(),
+ &BestVPlan);
VPlanTransforms::removeDeadRecipes(BestVPlan);
assert(verifyVPlanIsValid(BestVPlan) && "final VPlan is invalid");
@@ -8176,6 +8199,7 @@ VPlanPtr LoopVectorizationPlanner::tryToBuildVPlanWithVPRecipes(
VPlanTransforms::handleEarlyExits(*Plan, Legal->hasUncountableEarlyExit());
VPlanTransforms::addMiddleCheck(*Plan, RequiresScalarEpilogueCheck,
CM.foldTailByMasking());
+ VPlanTransforms::removeBranchOnConst(*Plan);
VPlanTransforms::createLoopRegions(*Plan);
diff --git a/llvm/lib/Transforms/Vectorize/VPlan.cpp b/llvm/lib/Transforms/Vectorize/VPlan.cpp
index f233f0dc1b025..26d9eda5e3b23 100644
--- a/llvm/lib/Transforms/Vectorize/VPlan.cpp
+++ b/llvm/lib/Transforms/Vectorize/VPlan.cpp
@@ -946,7 +946,7 @@ void VPlan::execute(VPTransformState *State) {
BasicBlock *ScalarPh = State->CFG.ExitBB;
VPBasicBlock *ScalarPhVPBB = getScalarPreheader();
- if (ScalarPhVPBB->hasPredecessors()) {
+ if (ScalarPhVPBB) {
// Disconnect scalar preheader and scalar header, as the dominator tree edge
// will be updated as part of VPlan execution. This allows keeping the DTU
// logic generic during VPlan execution.
@@ -961,7 +961,7 @@ void VPlan::execute(VPTransformState *State) {
Block->execute(State);
// If the original loop is unreachable, delete it and all its blocks.
- if (!ScalarPhVPBB->hasPredecessors()) {
+ if (!ScalarPhVPBB) {
// DeleteDeadBlocks will remove single-entry phis. Remove them from the exit
// VPIRBBs in VPlan as well, otherwise we would retain references to deleted
// IR instructions.
@@ -975,7 +975,7 @@ void VPlan::execute(VPTransformState *State) {
Loop *OrigLoop =
State->LI->getLoopFor(getScalarHeader()->getIRBasicBlock());
auto Blocks = OrigLoop->getBlocksVector();
- Blocks.push_back(cast<VPIRBasicBlock>(ScalarPhVPBB)->getIRBasicBlock());
+ Blocks.push_back(ScalarPh);
for (auto *BB : Blocks)
State->LI->removeBlock(BB);
DeleteDeadBlocks(Blocks, &State->CFG.DTU);
@@ -1661,7 +1661,8 @@ void LoopVectorizationPlanner::updateLoopMetadataAndProfileInfo(
// Update the metadata of the scalar loop. Skip the update when vectorizing
// the epilogue loop to ensure it is updated only once. Also skip the update
// when the scalar loop became unreachable.
- if (Plan.getScalarPreheader()->hasPredecessors() && !VectorizingEpilogue) {
+ auto *ScalarPH = Plan.getScalarPreheader();
+ if (ScalarPH && !VectorizingEpilogue) {
std::optional<MDNode *> RemainderLoopID =
makeFollowupLoopID(OrigLoopID, {LLVMLoopVectorizeFollowupAll,
LLVMLoopVectorizeFollowupEpilogue});
@@ -1723,7 +1724,7 @@ void LoopVectorizationPlanner::updateLoopMetadataAndProfileInfo(
AverageVectorTripCount = SE.getSmallConstantTripCount(VectorLoop);
if (ProfcheckDisableMetadataFixes || !AverageVectorTripCount)
return;
- if (Plan.getScalarPreheader()->hasPredecessors())
+ if (ScalarPH)
RemainderAverageTripCount =
SE.getSmallConstantTripCount(OrigLoop) % EstimatedVFxUF;
// Setting to 1 should be sufficient to generate the correct branch weights.
@@ -1739,7 +1740,7 @@ void LoopVectorizationPlanner::updateLoopMetadataAndProfileInfo(
OrigLoopInvocationWeight);
}
- if (Plan.getScalarPreheader()->hasPredecessors()) {
+ if (ScalarPH) {
setLoopEstimatedTripCount(OrigLoop, RemainderAverageTripCount,
OrigLoopInvocationWeight);
}
diff --git a/llvm/lib/Transforms/Vectorize/VPlan.h b/llvm/lib/Transforms/Vectorize/VPlan.h
index 7455129ff91e8..9732f553f3309 100644
--- a/llvm/lib/Transforms/Vectorize/VPlan.h
+++ b/llvm/lib/Transforms/Vectorize/VPlan.h
@@ -4624,7 +4624,8 @@ class VPlan {
/// Return the VPBasicBlock for the preheader of the scalar loop.
VPBasicBlock *getScalarPreheader() const {
- return cast<VPBasicBlock>(getScalarHeader()->getSinglePredecessor());
+ return dyn_cast_or_null<VPBasicBlock>(
+ getScalarHeader()->getSinglePredecessor());
}
/// Return the VPIRBasicBlock wrapping the header of the scalar loop.
@@ -4869,8 +4870,8 @@ class VPlan {
/// that this relies on unneeded branches to the scalar tail loop being
/// removed.
bool hasScalarTail() const {
- return !(!getScalarPreheader()->hasPredecessors() ||
- getScalarPreheader()->getSinglePredecessor() == getEntry());
+ auto *ScalarPH = getScalarPreheader();
+ return ScalarPH && ScalarPH->getSinglePredecessor() != getEntry();
}
};
diff --git a/llvm/lib/Transforms/Vectorize/VPlanConstruction.cpp b/llvm/lib/Transforms/Vectorize/VPlanConstruction.cpp
index 3101b42625c53..af2ae283e8092 100644
--- a/llvm/lib/Transforms/Vectorize/VPlanConstruction.cpp
+++ b/llvm/lib/Transforms/Vectorize/VPlanConstruction.cpp
@@ -1288,32 +1288,34 @@ bool VPlanTransforms::handleMaxMinNumReductions(VPlan &Plan) {
// Update resume phis for inductions in the scalar preheader. If AnyNaNLane is
// true, the resume from the start of the last vector iteration via the
// canonical IV, otherwise from the original value.
- for (auto &R : Plan.getScalarPreheader()->phis()) {
- auto *ResumeR = cast<VPPhi>(&R);
- VPValue *VecV = ResumeR->getOperand(0);
- if (RdxResults.contains(VecV))
- continue;
- if (auto *DerivedIV = dyn_cast<VPDerivedIVRecipe>(VecV)) {
- if (DerivedIV->getNumUsers() == 1 &&
- DerivedIV->getOperand(1) == &Plan.getVectorTripCount()) {
- auto *NewSel =
- MiddleBuilder.createSelect(AnyNaNLane, LoopRegion->getCanonicalIV(),
- &Plan.getVectorTripCount());
- DerivedIV->moveAfter(&*MiddleBuilder.getInsertPoint());
- DerivedIV->setOperand(1, NewSel);
+ if (auto *ScalarPH = Plan.getScalarPreheader()) {
+ for (auto &R : ScalarPH->phis()) {
+ auto *ResumeR = cast<VPPhi>(&R);
+ VPValue *VecV = ResumeR->getOperand(0);
+ if (RdxResults.contains(VecV))
continue;
+ if (auto *DerivedIV = dyn_cast<VPDerivedIVRecipe>(VecV)) {
+ if (DerivedIV->getNumUsers() == 1 &&
+ DerivedIV->getOperand(1) == &Plan.getVectorTripCount()) {
+ auto *NewSel =
+ MiddleBuilder.createSelect(AnyNaNLane, LoopRegion->getCanonicalIV(),
+ &Plan.getVectorTripCount());
+ DerivedIV->moveAfter(&*MiddleBuilder.getInsertPoint());
+ DerivedIV->setOperand(1, NewSel);
+ continue;
+ }
}
+ // Bail out and abandon the current, partially modified, VPlan if we
+ // encounter resume phi that cannot be updated yet.
+ if (VecV != &Plan.getVectorTripCount()) {
+ LLVM_DEBUG(dbgs() << "Found resume phi we cannot update for VPlan with "
+ "FMaxNum/FMinNum reduction.\n");
+ return false;
+ }
+ auto *NewSel = MiddleBuilder.createSelect(
+ AnyNaNLane, LoopRegion->getCanonicalIV(), VecV);
+ ResumeR->setOperand(0, NewSel);
}
- // Bail out and abandon the current, partially modified, VPlan if we
- // encounter resume phi that cannot be updated yet.
- if (VecV != &Plan.getVectorTripCount()) {
- LLVM_DEBUG(dbgs() << "Found resume phi we cannot update for VPlan with "
- "FMaxNum/FMinNum reduction.\n");
- return false;
- }
- auto *NewSel = MiddleBuilder.createSelect(
- AnyNaNLane, LoopRegion->getCanonicalIV(), VecV);
- ResumeR->setOperand(0, NewSel);
}
auto *MiddleTerm = MiddleVPBB->getTerminator();
diff --git a/llvm/lib/Transforms/Vectorize/VPlanPredicator.cpp b/llvm/lib/Transforms/Vectorize/VPlanPredicator.cpp
index dbc2e71c785ee..2e41dbbad8bf5 100644
--- a/llvm/lib/Transforms/Vectorize/VPlanPredicator.cpp
+++ b/llvm/lib/Transforms/Vectorize/VPlanPredicator.cpp
@@ -325,8 +325,11 @@ void VPlanTransforms::introduceMasksAndLinearize(VPlan &Plan, bool FoldTail) {
Plan.getMiddleBlock() &&
"the exit block must have middle block as single predecessor");
- VPBuilder B(Plan.getMiddleBlock()->getTerminator());
- for (VPRecipeBase &R : *Plan.getMiddleBlock()) {
+ auto *MiddleVPBB = Plan.getMiddleBlock();
+ auto *Term = MiddleVPBB->getTerminator();
+ auto IP = Term ? Term->getIterator() : MiddleVPBB->end();
+ VPBuilder B(MiddleVPBB, IP);
+ for (VPRecipeBase &R : *MiddleVPBB) {
VPValue *Op;
if (!match(&R, m_CombineOr(
m_ExitingIVValue(m_VPValue(), m_VPValue(Op)),
diff --git a/llvm/lib/Transforms/Vectorize/VPlanRecipes.cpp b/llvm/lib/Transforms/Vectorize/VPlanRecipes.cpp
index f7810c65d000e..d122bda19bb15 100644
--- a/llvm/lib/Transforms/Vectorize/VPlanRecipes.cpp
+++ b/llvm/lib/Transforms/Vectorize/VPlanRecipes.cpp
@@ -128,6 +128,7 @@ bool VPRecipeBase::mayReadFromMemory() const {
return cast<VPWidenIntrinsicRecipe>(this)->mayReadFromMemory();
case VPBranchOnMaskSC:
case VPDerivedIVSC:
+ case VPCurrentIterationPHISC:
case VPFirstOrderRecurrencePHISC:
case VPReductionPHISC:
case VPPredInstPHISC:
@@ -165,6 +166,7 @@ bool VPRecipeBase::mayHaveSideEffects() const {
return cast<VPExpressionRecipe>(this)->mayHaveSideEffects();
case VPActiveLaneMaskPHISC:
case VPDerivedIVSC:
+ case VPCurrentIterationPHISC:
case VPFirstOrderRecurrencePHISC:
case VPReductionPHISC:
case VPPredInstPHISC:
diff --git a/llvm/lib/Transforms/Vectorize/VPlanTransforms.cpp b/llvm/lib/Transforms/Vectorize/VPlanTransforms.cpp
index c16149303b180..3e9da6a5a4204 100644
--- a/llvm/lib/Transforms/Vectorize/VPlanTransforms.cpp
+++ b/llvm/lib/Transforms/Vectorize/VPlanTransforms.cpp
@@ -1467,12 +1467,6 @@ static void simplifyRecipe(VPSingleDefRecipe *Def, VPTypeAnalysis &TypeInfo) {
return;
}
- if (auto *Phi = dyn_cast<VPFirstOrderRecurrencePHIRecipe>(Def)) {
- if (Phi->getOperand(0) == Phi->getOperand(1))
- Phi->replaceAllUsesWith(Phi->getOperand(0));
- return;
- }
-
// Simplify MaskedCond with no block mask to its single operand.
if (match(Def, m_VPInstruction<VPInstruction::MaskedCond>()) &&
!cast<VPInstruction>(Def)->isMasked())
@@ -1521,9 +1515,15 @@ static void simplifyRecipe(VPSingleDefRecipe *Def, VPTypeAnalysis &TypeInfo) {
return;
}
- if (isa<VPPhi, VPWidenPHIRecipe>(Def)) {
- if (Def->getNumOperands() == 1)
+ if (isa<VPPhi, VPWidenPHIRecipe, VPHeaderPHIRecipe>(Def)) {
+ if (Def->getNumOperands() == 1) {
Def->replaceAllUsesWith(Def->getOperand(0));
+ return;
+ }
+ if (auto *Phi = dyn_cast<VPFirstOrderRecurrencePHIRecipe>(Def)) {
+ if (Phi->getOperand(0) == Phi->getOperand(1))
+ Phi->replaceAllUsesWith(Phi->getOperand(0));
+ }
return;
}
@@ -2095,72 +2095,16 @@ static bool simplifyBranchConditionForVFAndUF(VPlan &Plan, ElementCount BestVF,
return false;
}
- // The vector loop region only executes once. If possible, completely remove
- // the region, otherwise replace the terminator controlling the latch with
- // (BranchOnCond true).
- // TODO: VPWidenIntOrFpInductionRecipe is only partially supported; add
- // support for other non-canonical widen induction recipes (e.g.,
- // VPWidenPointerInductionRecipe).
- // TODO: fold branch-on-constant after dissolving region.
- auto *Header = cast<VPBasicBlock>(VectorRegion->getEntry());
- if (all_of(Header->phis(), [](VPRecipeBase &Phi) {
- if (auto *R = dyn_cast<VPWidenIntOrFpInductionRecipe>(&Phi))
- return R->isCanonical();
- return isa<VPCanonicalIVPHIRecipe, VPCurrentIterationPHIRecipe,
- VPFirstOrderRecurrencePHIRecipe, VPPhi>(&Phi);
- })) {
- for (VPRecipeBase &HeaderR : make_early_inc_range(Header->phis())) {
- if (auto *R = dyn_cast<VPWidenIntOrFpInductionRecipe>(&HeaderR)) {
- VPBuilder Builder(Plan.getVectorPreheader());
- VPValue *StepV = Builder.createNaryOp(VPInstruction::StepVector, {},
- R->getScalarType());
- HeaderR.getVPSingleValue()->replaceAllUsesWith(StepV);
- HeaderR.eraseFromParent();
- continue;
- }
- auto *Phi = cast<VPPhiAccessors>(&HeaderR);
- HeaderR.getVPSingleValue()->replaceAllUsesWith(Phi->getIncomingValue(0));
- HeaderR.eraseFromParent();
- }
-
- VPBlockBase *Preheader = VectorRegion->getSinglePredecessor();
- SmallVector<VPBlockBase *> Exits = to_vector(VectorRegion->getSuccessors());
- VPBlockUtils::disconnectBlocks(Preheader, VectorRegion);
- for (VPBlockBase *Exit : Exits)
- VPBlockUtils::disconnectBlocks(VectorRegion, Exit);
-
- for (VPBlockBase *B : vp_depth_first_shallow(VectorRegion->getEntry()))
- B->setParent(nullptr);
-
- VPBlockUtils::connectBlocks(Preheader, Header);
-
- for (VPBlockBase *Exit : Exits)
- VPBlockUtils::connectBlocks(ExitingVPBB, Exit);
-
- // Replace terminating branch-on-two-conds with branch-on-cond to early
- // exit.
- if (Exits.size() != 1) {
- assert(match(Term, m_BranchOnTwoConds()) && Exits.size() == 2 &&
- "BranchOnTwoConds needs 2 remaining exits");
- VPBuilder(Term).createNaryOp(VPInstruction::BranchOnCond,
- Term->getOperand(0));
- }
- VPlanTransforms::simplifyRecipes(Plan);
- } else {
- // The vector region contains header phis for which we cannot remove the
- // loop region yet.
-
- // For BranchOnTwoConds, set the latch exit condition to true directly.
- if (match(Term, m_BranchOnTwoConds())) {
- Term->setOperand(1, Plan.getTrue());
- return true;
- }
-
- auto *BOC = new VPInstruction(VPInstruction::BranchOnCond, {Plan.getTrue()},
- {}, {}, Term->getDebugLoc());
- ExitingVPBB->appendRecipe(BOC);
+ // The vector loop region only executes once. Convert terminator of the
+ // exiting block to exit in the first iteration.
+ if (match(Term, m_BranchOnTwoConds())) {
+ Term->setOperand(1, Plan.getTrue());
+ return true;
}
+ auto *BOC = new VPInstruction(VPInstruction::BranchOnCond, Plan.getTrue(), {},
+ {}, Term->getDebugLoc());
+ ExitingVPBB->appendRecipe(BOC);
Term->eraseFromParent();
return true;
@@ -2760,6 +2704,7 @@ void VPlanTransforms::truncateToMinimalBitwidths(
}
void VPlanTransforms::removeBranchOnConst(VPlan &Plan) {
+ SmallVector<VPBlockBase *> DeadSuccs;
for (VPBasicBlock *VPBB : VPBlockUtils::blocksOnly<VPBasicBlock>(
vp_depth_first_shallow(Plan.getEntry()))) {
VPValue *Cond;
@@ -2781,15 +2726,43 @@ void VPlanTransforms::removeBranchOnConst(VPlan &Plan) {
cast<VPBasicBlock>(VPBB->getSuccessors()[RemovedIdx]);
assert(count(RemovedSucc->getPredecessors(), VPBB) == 1 &&
"There must be a single edge between VPBB and its successor");
- // Values coming from VPBB into phi recipes of RemoveSucc are removed from
+ // Values coming from VPBB into phi recipes of RemovedSucc are removed from
// these recipes.
for (VPRecipeBase &R : RemovedSucc->phis())
cast<VPPhiAccessors>(&R)->removeIncomingValueFor(VPBB);
- // Disconnect blocks and remove the terminator. RemovedSucc will be deleted
- // automatically on VPlan destruction if it becomes unreachable.
+ // Disconnect blocks and remove the terminator.
VPBlockUtils::disconnectBlocks(VPBB, RemovedSucc);
VPBB->back().eraseFromParent();
+ DeadSuccs.push_back(RemovedSucc);
+ }
+
+ // Transitively disconnect all blocks that became unreachable, removing
+ // incoming values from phi recipes and replacing remaining uses with poison.
+ VPTypeAnalysis TypeInfo(Plan);
+ while (!DeadSuccs.empty()) {
+ VPBlockBase *DeadBlock = DeadSuccs.pop_back_val();
+ if (!DeadBlock->getPredecessors().empty())
+ continue;
+ for (VPBlockBase *Succ :
+ SmallVector<VPBlockBase *>(DeadBlock->successors())) {
+ if (auto *SuccBB = dyn_cast<VPBasicBlock>(Succ))
+ for (VPRecipeBase &R : SuccBB->phis())
+ cast<VPPhiAccessors>(&R)->removeIncomingValueFor(DeadBlock);
+ VPBlockUtils::disconnectBlocks(DeadBlock, Succ);
+ if (Succ->getPredecessors().empty())
+ DeadSuccs.push_back(Succ);
+ }
+ auto *DeadBB = dyn_cast<VPBasicBlock>(DeadBlock);
+ if (!DeadBB)
+ continue;
+ for (VPRecipeBase &R : make_early_inc_range(*DeadBB)) {
+ fo...
[truncated]
|
|
✅ With the latest revision this PR passed the C/C++ code formatter. |
87015fd to
93f1033
Compare
93f1033 to
f153ed5
Compare
lukel97
reviewed
Mar 30, 2026
lukel97
reviewed
Mar 30, 2026
lukel97
reviewed
Apr 16, 2026
lukel97
approved these changes
Apr 16, 2026
lukel97
left a comment
Contributor
There was a problem hiding this comment.
LGTM, just one minor comment
Comment on lines
+2927
to
+2928
| for (VPBasicBlock *DeadBB : | ||
| VPBlockUtils::blocksOnly<VPBasicBlock>(vp_depth_first_deep(B))) { |
Contributor
There was a problem hiding this comment.
Do we even need to do the DFS here? I would have thought that we would have reached each block via AllBlocks anyway
Contributor
Author
There was a problem hiding this comment.
Currently this is needed as AllBlocks does not do a deep traversal yet
llvm-sync Bot
pushed a commit
to arm/arm-toolchain
that referenced
this pull request
Apr 17, 2026
Simplify constant branches early, after introducing the check in the middle block. This removes any trivial branches in the input CFG (e.g. over-reduced test cases) early and also folds branches on true/false created by addMiddleChecks. This allows to check if there's a scalar tail instead to check if the tail has been folded, as mentioned in llvm/llvm-project#182507 This requires to remove recipes in the new unreachable blocks, as otherwise we would fail during verification, due to uses in unreachable blocks. Alternatively, we may be able to skip verification for uses in unreachable blocks. Depends on llvm/llvm-project#181252. PR: llvm/llvm-project#183397
cpullvm-upstream-sync Bot
pushed a commit
to navaneethshan/cpullvm-toolchain-1
that referenced
this pull request
Apr 17, 2026
Simplify constant branches early, after introducing the check in the middle block. This removes any trivial branches in the input CFG (e.g. over-reduced test cases) early and also folds branches on true/false created by addMiddleChecks. This allows to check if there's a scalar tail instead to check if the tail has been folded, as mentioned in llvm/llvm-project#182507 This requires to remove recipes in the new unreachable blocks, as otherwise we would fail during verification, due to uses in unreachable blocks. Alternatively, we may be able to skip verification for uses in unreachable blocks. Depends on llvm/llvm-project#181252. PR: llvm/llvm-project#183397
alexfh
pushed a commit
to alexfh/llvm-project
that referenced
this pull request
Apr 18, 2026
Simplify constant branches early, after introducing the check in the middle block. This removes any trivial branches in the input CFG (e.g. over-reduced test cases) early and also folds branches on true/false created by addMiddleChecks. This allows to check if there's a scalar tail instead to check if the tail has been folded, as mentioned in llvm#182507 This requires to remove recipes in the new unreachable blocks, as otherwise we would fail during verification, due to uses in unreachable blocks. Alternatively, we may be able to skip verification for uses in unreachable blocks. Depends on llvm#181252. PR: llvm#183397
llvm-upstreamsync Bot
pushed a commit
to qualcomm/cpullvm-toolchain
that referenced
this pull request
Apr 24, 2026
Simplify constant branches early, after introducing the check in the middle block. This removes any trivial branches in the input CFG (e.g. over-reduced test cases) early and also folds branches on true/false created by addMiddleChecks. This allows to check if there's a scalar tail instead to check if the tail has been folded, as mentioned in llvm/llvm-project#182507 This requires to remove recipes in the new unreachable blocks, as otherwise we would fail during verification, due to uses in unreachable blocks. Alternatively, we may be able to skip verification for uses in unreachable blocks. Depends on llvm/llvm-project#181252. PR: llvm/llvm-project#183397
KHicketts
pushed a commit
to KHicketts/llvm-project
that referenced
this pull request
Apr 30, 2026
Simplify constant branches early, after introducing the check in the middle block. This removes any trivial branches in the input CFG (e.g. over-reduced test cases) early and also folds branches on true/false created by addMiddleChecks. This allows to check if there's a scalar tail instead to check if the tail has been folded, as mentioned in llvm#182507 This requires to remove recipes in the new unreachable blocks, as otherwise we would fail during verification, due to uses in unreachable blocks. Alternatively, we may be able to skip verification for uses in unreachable blocks. Depends on llvm#181252. PR: llvm#183397
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Simplify constant branches early, after introducing the check in the
middle block.
This removes any trivial branches in the input CFG (e.g. over-reduced
test cases) early and also folds branches on true/false created by
addMiddleChecks. This allows to check if there's a scalar tail instead
to check if the tail has been folded, as mentioned in
#182507
This requires to remove recipes in the new unreachable blocks, as otherwise we would fail during verification, due to uses in unreachable blocks. Alternatively, we may be able to skip verification for uses in unreachable blocks.
Depends on #181252.