feat: Implement LLM Triage & Smart Routing System#24
Conversation
…mented transfer rules engine and start a new development session.
…sions - Add Description field to RepositoryConfig for LLM routing - Add LLMRoutingEnabled, HighConfidence, MediumConfidence to TransferConfig - Extend pipeline Result struct with quality, transfer, and duplicate fields - Set default confidence thresholds (0.9 high, 0.6 medium)
…ection - Add RouteIssue() method for repository routing with confidence scores - Add AssessQuality() method for issue quality assessment (0.0-1.0 scale) - Add DetectDuplicate() method for semantic duplicate detection - Add corresponding prompt builders with JSON-structured outputs - Define data structures: RouterResult, QualityResult, DuplicateResult
- Add llm_router step for LLM-based repository routing - Confidence-based actions: auto-transfer (≥0.9), suggest (0.6-0.9), silent (<0.6) - Graceful degradation when LLM unavailable - Add quality_checker step for issue quality assessment - Evaluates completeness and clarity - Provides actionable suggestions - Add duplicate_detector step for semantic duplicate detection - Analyzes top 3 similar issues - High-confidence threshold (≥0.8)
BREAKING CHANGE: Response builder now ALWAYS posts triage summaries - Remove early return logic - post summary on every issue - Add quality assessment section with score and suggestions - Add labels applied section - Add transfer suggestion section with confidence display - Add similar issues section with similarity percentages - Add duplicate warning section with reasoning - Format as structured markdown with visual indicators (✅,⚠️ , 🔄)
- Remove quality assessment logic (moved to quality_checker step) - Focus solely on label suggestion using LLM - Update error handling for graceful degradation
- Register llm_router, quality_checker, duplicate_detector steps - Update issue-triage preset workflow order: gatekeeper → vectordb_prep → llm_router → transfer_check → similarity_search → duplicate_detector → quality_checker → triage → response_builder → action_executor → pending_action_scheduler → indexer
There was a problem hiding this comment.
Pull request overview
This PR implements an intelligent LLM-based triage system that enhances issue routing, quality assessment, and duplicate detection using Gemini AI.
Changes:
- Added three new pipeline steps:
llm_routerfor intelligent repository routing,quality_checkerfor issue quality assessment, andduplicate_detectorfor semantic duplicate detection - Rewrote
response_builderto generate comprehensive triage summaries with visual indicators and structured sections - Extended configuration with LLM routing settings (
LLMRoutingEnabled, confidence thresholds) and repository descriptions - Extended pipeline Result struct with quality, transfer, and duplicate detection fields
Reviewed changes
Copilot reviewed 14 out of 14 changed files in this pull request and generated 5 comments.
Show a summary per file
| File | Description |
|---|---|
| internal/core/config/config.go | Added Description field to RepositoryConfig and LLM routing configuration fields (LLMRoutingEnabled, HighConfidence, MediumConfidence) with defaults |
| internal/core/pipeline/pipeline.go | Extended Result struct with quality assessment, transfer routing, and duplicate detection fields |
| internal/core/pipeline/registry.go | Updated issue-triage workflow to include llm_router, duplicate_detector, and quality_checker steps in appropriate order |
| internal/integrations/gemini/llm.go | Added RouteIssue, AssessQuality, and DetectDuplicate methods with JSON-structured responses |
| internal/integrations/gemini/prompts.go | Added prompts for routing, quality assessment, and duplicate detection with clear guidelines and JSON schemas |
| internal/steps/llm_router.go | New step for LLM-based repository routing with confidence-based actions (auto-transfer, suggest, or silent) |
| internal/steps/quality_checker.go | New step for issue quality assessment storing results in context metadata |
| internal/steps/duplicate_detector.go | New step for semantic duplicate detection using top 3 similar issues with 0.8 confidence threshold |
| internal/steps/triage.go | Simplified to focus only on label suggestion, updated comments and GitHub username |
| internal/steps/response_builder.go | Complete rewrite to build comprehensive triage summaries with quality, labels, transfers, similar issues, and duplicate sections |
| internal/steps/register.go | Registered three new pipeline steps |
| .claude/sessions/*.md | Development session documentation updates |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
|
|
||
| // Score and assessment | ||
| scoreDisplay := fmt.Sprintf("%.1f/10", qualityResult.Score*10) | ||
| assessment := strings.Title(qualityResult.Assessment) |
There was a problem hiding this comment.
The strings.Title function has been deprecated since Go 1.18 and should not be used. According to the Go documentation, strings.Title uses simple case-folding which doesn't handle Unicode correctly.
For title-casing a single word like "good", "excellent", etc., use cases.Title(language.Und).String(qualityResult.Assessment) from the golang.org/x/text/cases package, or for simple English strings, implement a custom function like:
func capitalizeFirst(s string) string {
if s == "" {
return s
}
return strings.ToUpper(s[:1]) + s[1:]
}Note that the AssessQuality method in internal/integrations/gemini/llm.go already normalizes the assessment to lowercase (line 279), so this formatting should be consistently applied.
| func (s *ResponseBuilder) Run(ctx *pipeline.Context) error { | ||
| if len(ctx.SimilarIssues) == 0 && ctx.TransferTarget == "" { | ||
| log.Printf("[response_builder] No similar issues or transfer target, skipping comment") | ||
| return nil | ||
| } | ||
|
|
||
| // Try AI response if LLM is available and we have similar issues | ||
| if s.llm != nil && len(ctx.SimilarIssues) > 0 { | ||
| aiComment := s.generateAIResponse(ctx) | ||
| if aiComment != "" { | ||
| ctx.Metadata["comment"] = aiComment | ||
| log.Printf("[response_builder] Built AI comment") | ||
| return nil | ||
| } | ||
| } | ||
| log.Printf("[response_builder] Building comprehensive triage summary for issue #%d", ctx.Issue.Number) | ||
|
|
||
| // Fallback to template-based response | ||
| comment := s.buildTemplateResponse(ctx) | ||
| comment := s.buildTriageSummary(ctx) | ||
| ctx.Metadata["comment"] = comment | ||
| log.Printf("[response_builder] Built template comment with %d similar issues", len(ctx.SimilarIssues)) | ||
|
|
||
| log.Printf("[response_builder] Built triage summary") | ||
| return nil | ||
| } |
There was a problem hiding this comment.
The response builder always creates a comment with at least the header and footer, even when all content sections are empty. This could result in posting comments with minimal value (just "## 🤖 Simili Triage Report" and the footer) when no quality assessment, labels, transfers, similar issues, or duplicates are found.
Consider adding a check to only build and store the comment if at least one content section is non-empty. For example:
func (s *ResponseBuilder) Run(ctx *pipeline.Context) error {
// Check if there's any content to report
hasContent := s.hasAnyContent(ctx)
if !hasContent {
log.Printf("[response_builder] No triage content to report, skipping comment")
return nil
}
comment := s.buildTriageSummary(ctx)
ctx.Metadata["comment"] = comment
return nil
}
func (s *ResponseBuilder) hasAnyContent(ctx *pipeline.Context) bool {
return s.buildQualitySection(ctx) != "" ||
s.buildLabelsSection(ctx) != "" ||
s.buildTransferSection(ctx) != "" ||
s.buildSimilarSection(ctx) != "" ||
s.buildDuplicateSection(ctx) != ""
}This ensures the bot only comments when it has actionable information to provide.
| ctx.Metadata["duplicate_result"] = result | ||
|
|
||
| // Mark as duplicate if high confidence (>= 0.8) | ||
| if result.IsDuplicate && result.Confidence >= 0.8 { |
There was a problem hiding this comment.
The duplicate confidence threshold (0.8) is hardcoded in the duplicate detector step. This threshold determines when an issue is marked as a duplicate and should ideally be configurable to allow administrators to tune the sensitivity based on their needs.
Consider adding a DuplicateConfidenceThreshold field to the config (similar to HighConfidence and MediumConfidence in TransferConfig) to make this value configurable. A reasonable default would be 0.8, but teams may want to adjust it based on their tolerance for false positives/negatives.
| @@ -1,7 +1,7 @@ | |||
| // Author: Kaviru Hapuarachchi | |||
| // GitHub: https://github.com/Kavirubc | |||
| // GitHub: https://github.com/kavirubc | |||
There was a problem hiding this comment.
The GitHub username has been changed from "Kavirubc" to "kavirubc" (lowercase) in the modified files. This creates an inconsistency with the rest of the codebase, where 30+ files still use "Kavirubc" with uppercase K.
For consistency, either:
- Update all files to use the lowercase version if that's the correct username, or
- Keep the existing uppercase version
GitHub usernames are case-insensitive for URL purposes, but it's better to be consistent across the codebase.
| } | ||
|
|
||
| var parts []string | ||
| parts = append(parts, "### Labels Applied") |
There was a problem hiding this comment.
The section title is "Labels Applied" but at the time the response builder runs, the labels haven't actually been applied yet - they're just suggested. The actual application happens later in the action_executor step.
Consider changing the title to "Suggested Labels" or "Labels To Apply" to accurately reflect that these are suggestions that will be applied, not labels that have already been applied. This would be more accurate and prevent user confusion.
| parts = append(parts, "### Labels Applied") | |
| parts = append(parts, "### Suggested Labels") |
- Replace deprecated strings.Title with custom capitalizeFirst function - Make duplicate confidence threshold configurable via TransferConfig - Change 'Labels Applied' to 'Suggested Labels' for accuracy - Add DuplicateConfidenceThreshold field with 0.8 default
Overview
This PR implements an intelligent LLM-based triage system that enhances issue routing, quality assessment, and duplicate detection.
Key Features
1. LLM-Based Repository Routing 🎯
2. Quality Assessment ✅
3. Semantic Duplicate Detection 🔍
4. Comprehensive Triage Summaries 📊
Changes
Configuration
Descriptionfield toRepositoryConfigfor LLM routingLLMRoutingEnabled,HighConfidence,MediumConfidencetoTransferConfigResultstruct with quality, transfer, and duplicate fieldsLLM Integration
RouteIssue()method for repository routingAssessQuality()method for quality assessmentDetectDuplicate()method for duplicate detectionPipeline Steps
llm_router- LLM-based repository routingquality_checker- Issue quality assessmentduplicate_detector- Semantic duplicate detectionresponse_builder- Comprehensive triage summariestriage- Focus on label suggestion onlyPipeline Order
Configuration Example
Testing
Cost Implications
LLM calls per issue: ~4
Estimated cost: $0.01-0.02 per issue with gemini-2.0-flash-lite
Next Steps
.simili.yamlllm_routing_enabled: trueCommits