Skip to content

feat: Implement LLM Triage & Smart Routing System#24

Merged
Kavirubc merged 8 commits intomainfrom
feature/llm-triage-routing
Feb 4, 2026
Merged

feat: Implement LLM Triage & Smart Routing System#24
Kavirubc merged 8 commits intomainfrom
feature/llm-triage-routing

Conversation

@Kavirubc
Copy link
Copy Markdown
Contributor

@Kavirubc Kavirubc commented Feb 4, 2026

Overview

This PR implements an intelligent LLM-based triage system that enhances issue routing, quality assessment, and duplicate detection.

Key Features

1. LLM-Based Repository Routing 🎯

  • Analyzes issue intent vs repository descriptions
  • Confidence-based actions:
    • High (≥0.9): Auto-transfer
    • Medium (0.6-0.9): Suggest in comment
    • Low (<0.6): Silent
  • Runs before rule-based transfer (rules can override)

2. Quality Assessment

  • Evaluates issue completeness and clarity
  • Provides 0-10 scoring scale
  • Identifies missing elements
  • Offers actionable suggestions

3. Semantic Duplicate Detection 🔍

  • Analyzes top 3 similar issues using LLM
  • High-confidence threshold (≥0.8)
  • Explains reasoning
  • Distinguishes related vs duplicate issues

4. Comprehensive Triage Summaries 📊

  • BREAKING: Always posts triage summary (no silent failures)
  • Structured sections: quality, labels, transfer, similar issues, duplicates
  • Visual indicators (✅, ⚠️, 🔄)
  • AI-powered insights

Changes

Configuration

  • Added Description field to RepositoryConfig for LLM routing
  • Added LLMRoutingEnabled, HighConfidence, MediumConfidence to TransferConfig
  • Extended pipeline Result struct with quality, transfer, and duplicate fields

LLM Integration

  • Added RouteIssue() method for repository routing
  • Added AssessQuality() method for quality assessment
  • Added DetectDuplicate() method for duplicate detection
  • All methods use JSON-structured prompts

Pipeline Steps

  • NEW: llm_router - LLM-based repository routing
  • NEW: quality_checker - Issue quality assessment
  • NEW: duplicate_detector - Semantic duplicate detection
  • REWRITTEN: response_builder - Comprehensive triage summaries
  • SIMPLIFIED: triage - Focus on label suggestion only

Pipeline Order

gatekeeper → vectordb_prep → llm_router → transfer_check →
similarity_search → duplicate_detector → quality_checker →
triage → response_builder → action_executor →
pending_action_scheduler → indexer

Configuration Example

repositories:
  - org: "similigh"
    repo: "simili-bot"
    description: "GitHub bot for issue triage, routing, and duplicate detection"
    enabled: true
  - org: "similigh"
    repo: "frontend"
    description: "React-based web UI for dashboard and user management"
    enabled: true

transfer:
  enabled: true
  llm_routing_enabled: true
  high_confidence: 0.9    # Auto-transfer threshold
  medium_confidence: 0.6  # Suggestion threshold

Testing

  • ✅ Build successful
  • ✅ All existing tests passing
  • ✅ Graceful degradation when LLM unavailable

Cost Implications

LLM calls per issue: ~4

  • 1x RouteIssue (if enabled)
  • 1x AssessQuality
  • 1x DetectDuplicate (if similar issues found)
  • 1x AnalyzeIssue (labels)

Estimated cost: $0.01-0.02 per issue with gemini-2.0-flash-lite

Next Steps

  1. Add repository descriptions to .simili.yaml
  2. Enable llm_routing_enabled: true
  3. Test with real issues
  4. Fine-tune confidence thresholds if needed

Commits

  • feat(config): add LLM routing configuration and pipeline result extensions
  • feat(gemini): add LLM methods for routing, quality, and duplicate detection
  • feat(steps): add three new pipeline steps for intelligent triage
  • feat(steps): rewrite response builder for comprehensive triage summaries
  • refactor(steps): simplify triage step to focus on label suggestion
  • feat(pipeline): integrate new steps into issue-triage workflow

…mented transfer rules engine and start a new development session.
…sions

- Add Description field to RepositoryConfig for LLM routing
- Add LLMRoutingEnabled, HighConfidence, MediumConfidence to TransferConfig
- Extend pipeline Result struct with quality, transfer, and duplicate fields
- Set default confidence thresholds (0.9 high, 0.6 medium)
…ection

- Add RouteIssue() method for repository routing with confidence scores
- Add AssessQuality() method for issue quality assessment (0.0-1.0 scale)
- Add DetectDuplicate() method for semantic duplicate detection
- Add corresponding prompt builders with JSON-structured outputs
- Define data structures: RouterResult, QualityResult, DuplicateResult
- Add llm_router step for LLM-based repository routing
  - Confidence-based actions: auto-transfer (≥0.9), suggest (0.6-0.9), silent (<0.6)
  - Graceful degradation when LLM unavailable
- Add quality_checker step for issue quality assessment
  - Evaluates completeness and clarity
  - Provides actionable suggestions
- Add duplicate_detector step for semantic duplicate detection
  - Analyzes top 3 similar issues
  - High-confidence threshold (≥0.8)
BREAKING CHANGE: Response builder now ALWAYS posts triage summaries

- Remove early return logic - post summary on every issue
- Add quality assessment section with score and suggestions
- Add labels applied section
- Add transfer suggestion section with confidence display
- Add similar issues section with similarity percentages
- Add duplicate warning section with reasoning
- Format as structured markdown with visual indicators (✅, ⚠️, 🔄)
- Remove quality assessment logic (moved to quality_checker step)
- Focus solely on label suggestion using LLM
- Update error handling for graceful degradation
- Register llm_router, quality_checker, duplicate_detector steps
- Update issue-triage preset workflow order:
  gatekeeper → vectordb_prep → llm_router → transfer_check →
  similarity_search → duplicate_detector → quality_checker →
  triage → response_builder → action_executor →
  pending_action_scheduler → indexer
Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR implements an intelligent LLM-based triage system that enhances issue routing, quality assessment, and duplicate detection using Gemini AI.

Changes:

  • Added three new pipeline steps: llm_router for intelligent repository routing, quality_checker for issue quality assessment, and duplicate_detector for semantic duplicate detection
  • Rewrote response_builder to generate comprehensive triage summaries with visual indicators and structured sections
  • Extended configuration with LLM routing settings (LLMRoutingEnabled, confidence thresholds) and repository descriptions
  • Extended pipeline Result struct with quality, transfer, and duplicate detection fields

Reviewed changes

Copilot reviewed 14 out of 14 changed files in this pull request and generated 5 comments.

Show a summary per file
File Description
internal/core/config/config.go Added Description field to RepositoryConfig and LLM routing configuration fields (LLMRoutingEnabled, HighConfidence, MediumConfidence) with defaults
internal/core/pipeline/pipeline.go Extended Result struct with quality assessment, transfer routing, and duplicate detection fields
internal/core/pipeline/registry.go Updated issue-triage workflow to include llm_router, duplicate_detector, and quality_checker steps in appropriate order
internal/integrations/gemini/llm.go Added RouteIssue, AssessQuality, and DetectDuplicate methods with JSON-structured responses
internal/integrations/gemini/prompts.go Added prompts for routing, quality assessment, and duplicate detection with clear guidelines and JSON schemas
internal/steps/llm_router.go New step for LLM-based repository routing with confidence-based actions (auto-transfer, suggest, or silent)
internal/steps/quality_checker.go New step for issue quality assessment storing results in context metadata
internal/steps/duplicate_detector.go New step for semantic duplicate detection using top 3 similar issues with 0.8 confidence threshold
internal/steps/triage.go Simplified to focus only on label suggestion, updated comments and GitHub username
internal/steps/response_builder.go Complete rewrite to build comprehensive triage summaries with quality, labels, transfers, similar issues, and duplicate sections
internal/steps/register.go Registered three new pipeline steps
.claude/sessions/*.md Development session documentation updates

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread internal/steps/response_builder.go Outdated

// Score and assessment
scoreDisplay := fmt.Sprintf("%.1f/10", qualityResult.Score*10)
assessment := strings.Title(qualityResult.Assessment)
Copy link

Copilot AI Feb 4, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The strings.Title function has been deprecated since Go 1.18 and should not be used. According to the Go documentation, strings.Title uses simple case-folding which doesn't handle Unicode correctly.

For title-casing a single word like "good", "excellent", etc., use cases.Title(language.Und).String(qualityResult.Assessment) from the golang.org/x/text/cases package, or for simple English strings, implement a custom function like:

func capitalizeFirst(s string) string {
    if s == "" {
        return s
    }
    return strings.ToUpper(s[:1]) + s[1:]
}

Note that the AssessQuality method in internal/integrations/gemini/llm.go already normalizes the assessment to lowercase (line 279), so this formatting should be consistently applied.

Copilot uses AI. Check for mistakes.
Comment on lines 36 to 44
func (s *ResponseBuilder) Run(ctx *pipeline.Context) error {
if len(ctx.SimilarIssues) == 0 && ctx.TransferTarget == "" {
log.Printf("[response_builder] No similar issues or transfer target, skipping comment")
return nil
}

// Try AI response if LLM is available and we have similar issues
if s.llm != nil && len(ctx.SimilarIssues) > 0 {
aiComment := s.generateAIResponse(ctx)
if aiComment != "" {
ctx.Metadata["comment"] = aiComment
log.Printf("[response_builder] Built AI comment")
return nil
}
}
log.Printf("[response_builder] Building comprehensive triage summary for issue #%d", ctx.Issue.Number)

// Fallback to template-based response
comment := s.buildTemplateResponse(ctx)
comment := s.buildTriageSummary(ctx)
ctx.Metadata["comment"] = comment
log.Printf("[response_builder] Built template comment with %d similar issues", len(ctx.SimilarIssues))

log.Printf("[response_builder] Built triage summary")
return nil
}
Copy link

Copilot AI Feb 4, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The response builder always creates a comment with at least the header and footer, even when all content sections are empty. This could result in posting comments with minimal value (just "## 🤖 Simili Triage Report" and the footer) when no quality assessment, labels, transfers, similar issues, or duplicates are found.

Consider adding a check to only build and store the comment if at least one content section is non-empty. For example:

func (s *ResponseBuilder) Run(ctx *pipeline.Context) error {
    // Check if there's any content to report
    hasContent := s.hasAnyContent(ctx)
    if !hasContent {
        log.Printf("[response_builder] No triage content to report, skipping comment")
        return nil
    }
    
    comment := s.buildTriageSummary(ctx)
    ctx.Metadata["comment"] = comment
    return nil
}

func (s *ResponseBuilder) hasAnyContent(ctx *pipeline.Context) bool {
    return s.buildQualitySection(ctx) != "" ||
           s.buildLabelsSection(ctx) != "" ||
           s.buildTransferSection(ctx) != "" ||
           s.buildSimilarSection(ctx) != "" ||
           s.buildDuplicateSection(ctx) != ""
}

This ensures the bot only comments when it has actionable information to provide.

Copilot uses AI. Check for mistakes.
Comment thread internal/steps/duplicate_detector.go Outdated
ctx.Metadata["duplicate_result"] = result

// Mark as duplicate if high confidence (>= 0.8)
if result.IsDuplicate && result.Confidence >= 0.8 {
Copy link

Copilot AI Feb 4, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The duplicate confidence threshold (0.8) is hardcoded in the duplicate detector step. This threshold determines when an issue is marked as a duplicate and should ideally be configurable to allow administrators to tune the sensitivity based on their needs.

Consider adding a DuplicateConfidenceThreshold field to the config (similar to HighConfidence and MediumConfidence in TransferConfig) to make this value configurable. A reasonable default would be 0.8, but teams may want to adjust it based on their tolerance for false positives/negatives.

Copilot uses AI. Check for mistakes.
Comment thread internal/steps/triage.go
@@ -1,7 +1,7 @@
// Author: Kaviru Hapuarachchi
// GitHub: https://github.com/Kavirubc
// GitHub: https://github.com/kavirubc
Copy link

Copilot AI Feb 4, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The GitHub username has been changed from "Kavirubc" to "kavirubc" (lowercase) in the modified files. This creates an inconsistency with the rest of the codebase, where 30+ files still use "Kavirubc" with uppercase K.

For consistency, either:

  1. Update all files to use the lowercase version if that's the correct username, or
  2. Keep the existing uppercase version

GitHub usernames are case-insensitive for URL purposes, but it's better to be consistent across the codebase.

Copilot uses AI. Check for mistakes.
Comment thread internal/steps/response_builder.go Outdated
}

var parts []string
parts = append(parts, "### Labels Applied")
Copy link

Copilot AI Feb 4, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The section title is "Labels Applied" but at the time the response builder runs, the labels haven't actually been applied yet - they're just suggested. The actual application happens later in the action_executor step.

Consider changing the title to "Suggested Labels" or "Labels To Apply" to accurately reflect that these are suggestions that will be applied, not labels that have already been applied. This would be more accurate and prevent user confusion.

Suggested change
parts = append(parts, "### Labels Applied")
parts = append(parts, "### Suggested Labels")

Copilot uses AI. Check for mistakes.
- Replace deprecated strings.Title with custom capitalizeFirst function
- Make duplicate confidence threshold configurable via TransferConfig
- Change 'Labels Applied' to 'Suggested Labels' for accuracy
- Add DuplicateConfidenceThreshold field with 0.8 default
@Kavirubc Kavirubc merged commit c9ffc2d into main Feb 4, 2026
3 checks passed
@Kavirubc Kavirubc deleted the feature/llm-triage-routing branch February 4, 2026 07:25
@Kavirubc Kavirubc linked an issue Feb 4, 2026 that may be closed by this pull request
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[v0.1.0] Implement LLM Triage & Smart Routing

2 participants