Skip to content

feat(plugins): add error tracking metrics and Grafana dashboard to llm-guard plugin#3406

Open
tedhabeck wants to merge 17 commits intoIBM:mainfrom
tedhabeck:issue-1961
Open

feat(plugins): add error tracking metrics and Grafana dashboard to llm-guard plugin#3406
tedhabeck wants to merge 17 commits intoIBM:mainfrom
tedhabeck:issue-1961

Conversation

@tedhabeck
Copy link
Copy Markdown
Collaborator

@tedhabeck tedhabeck commented Mar 2, 2026 •
edited
Loading

🔗 Related Issue

Closes #1961

📝 Summary

Overview

Added comprehensive error tracking metrics to the LLM Guard plugin to monitor error rates by type across all components, with corresponding Grafana dashboard visualizations.

Key Features

  • Comprehensive Labels: Enable filtering and aggregation by error type, component, and operation
  • Fail-Safe Error Handling: Errors are logged and tracked without disrupting service
  • Real-Time Monitoring: Dashboard auto-refreshes every 10 seconds
  • Visual Alerts: Color-coded thresholds (green/yellow/red) for quick issue identification
  • Historical Analysis: Error summary table shows trends over the last hour
  • Cache expiry updates
  • Use lazy evaluation instead of f-strings for logging.

Platform compatibility changes

  • Replace references to loopback ip address 127.0.0.1 with localhost for SSRF compatibility

🏷️ Type of Change

  • Bug fix
  • Feature / Enhancement
  • Documentation
  • Refactor
  • Chore (deps, CI, tooling)
  • Other (describe below)

🧪 Verification

Check Command Status
Lint suite make lint ✅
Unit tests make test ✅
Coverage ≥ 80% make coverage ✅

✅ Checklist

  • Code formatted (make black isort pre-commit)
  • Documentation updated (if applicable)
  • No secrets or credentials committed

📓 Notes (optional)

Requires application of changes in this PR for successful testing: #3211

@tedhabeck
fix: use lazy evaluation instead of f-strings. Add scanner initializa…
a0461de 
…tion metrics.

Signed-off-by: habeck <habeck@us.ibm.com>
@tedhabeck
Merge branch 'main' into issue-1961
3692dc3
@tedhabeck
fix: tests complaining about missing await
81ca418 
Signed-off-by: habeck <habeck@us.ibm.com>
@tedhabeck
fix: i before e except after c
198cee8 
Signed-off-by: habeck <habeck@us.ibm.com>
@tedhabeck
fix: Build scanner name → scanner index mapping during initialization
59b77f7 
Signed-off-by: habeck <habeck@us.ibm.com>
@tedhabeck
Merge branch 'main' into issue-1961
afdcdd1
@tedhabeck
Merge branch 'main' into issue-1961
8b03a25
@tedhabeck
fix: remove unnecessary logging, skip vault retrieval when TTL=0 (dis…
fb7b6fe 
…abled)

Signed-off-by: habeck <habeck@us.ibm.com>
@tedhabeck
fix: Cache expiry timestamp instead of recalculating each time
1b033e9 
Signed-off-by: habeck <habeck@us.ibm.com>
@tedhabeck
Merge branch 'main' into issue-1961
1e0df4c
@tedhabeck
fix: SSRF does not allow using the loopback IP address. Update doc to…
a0483cd 
… use localhost instead of 127.0.0.1.

Signed-off-by: habeck <habeck@us.ibm.com>
@tedhabeck
Merge branch 'main' into issue-1961
d9c8037
@tedhabeck
fix: SSRF does not allow ip address in URL, replace with hostname and…
ef30ae8 
… provide example for LLMGuardPlugin

Signed-off-by: habeck <habeck@us.ibm.com>
@tedhabeck
fix: add prometheus error rate by type metrics and example Grafana da…
763a352 
…sboard to visualize Scanner initialization time, Per-scanner execution time and Error rates by type

Signed-off-by: habeck <habeck@us.ibm.com>
@tedhabeck
chore: lint fix
653cc6a 
Signed-off-by: habeck <habeck@us.ibm.com>
@tedhabeck
chore: replace refeences to loopback ip address with localhost for SS…
bc96f9f 
…RF compatibility.

Signed-off-by: habeck <habeck@us.ibm.com>
@tedhabeck tedhabeck marked this pull request as ready for review March 2, 2026 19:58
@crivetimihai crivetimihai changed the title Issue 1961 - Fix minor performance issues in llm-guard plugin feat(plugins): add error tracking metrics and Grafana dashboard to llm-guard plugin Mar 5, 2026
@crivetimihai crivetimihai added enhancement New feature or request plugins COULD P3: Nice-to-have features with minimal impact if left out; included if time permits performance Performance related items labels Mar 5, 2026
@crivetimihai crivetimihai added this to the Release 1.1.0 milestone Mar 5, 2026
@crivetimihai
Copy link
Copy Markdown
Member

crivetimihai commented Mar 5, 2026

Thanks @tedhabeck — comprehensive error tracking for #1961. The metrics labels, fail-safe error handling, and Grafana dashboard are well-designed. The lazy evaluation logging fix is a nice bonus. LGTM.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@crivetimihai crivetimihai Awaiting requested review from crivetimihai crivetimihai is a code owner

@kevalmahajan kevalmahajan Awaiting requested review from kevalmahajan kevalmahajan is a code owner

@madhav165 madhav165 Awaiting requested review from madhav165 madhav165 is a code owner

At least 1 approving review is required to merge this pull request.

Assignees

No one assigned

Labels

COULD P3: Nice-to-have features with minimal impact if left out; included if time permits enhancement New feature or request performance Performance related items plugins

Projects

None yet

Milestone

Release 1.1.0

Development

Successfully merging this pull request may close these issues.

[BUG][PERFORMANCE]: Fix minor performance issues in llm-guard plugin

2 participants

@tedhabeck @crivetimihai