Summary

This PR introduces comprehensive performance profiling tooling for the plugin framework using Python's cProfile. The new test suite enables systematic profiling of individual plugin hooks to identify performance bottlenecks and measure optimization impact.

Closes #2061

Changes

New Files

• tests/performance/test_plugins_performance.py

  • Main profiling script using cProfile
  • Profiles each enabled plugin's hooks individually (1000 iterations per hook)
  • Generates .prof files for detailed analysis
  • Prints summary table showing average execution time per plugin per hook
  • Supports --details flag for immediate detailed output

• tests/performance/PLUGIN_PROFILING.md

  • Complete usage guide and documentation
  • Analysis techniques and examples
  • Optimization strategies and best practices
  • Troubleshooting guide

• tests/performance/plugins/config.yaml

  • Test configuration with all plugins enabled
  • Used by performance profiler for consistent testing

• tests/performance/utils/analyze_profiles.py

  • Utility functions to analyze and compare plugins performance profiles

Modified Files

• tests/performance/Makefile

  • Added convenience targets for running plugin performance tests
  • Added profile analysis commands

• tests/performance/README.md

  • Updated with plugin profiling documentation reference

Features

Performance Profiling

• Profiles invoke_hook() calls only (plugin loading excluded)
• Generates one .prof file per plugin-hook combination
• Calculates average execution time using actual call counts from profiler
• Summary table with plugins as rows, hook types as columns
• Empty cells for unimplemented hooks
• Uses plugins/config.yaml for plugin configuration

Output Modes

Default (summary only):

python tests/performance/test_plugins_performance.py

Detailed (with per-hook profiling output):

python tests/performance/test_plugins_performance.py --details

Sample Output

Plugin                       P:post       P:pre      T:post       T:pre
-----------------------------------------------------------------------
PIIFilterPlugin             0.058ms     0.103ms     0.078ms     0.632ms
ReplaceBadWordsPlugin       0.056ms     0.103ms     0.078ms     0.635ms
SPARCStaticValidator              —           —           —     0.625ms
VaultPlugin                       —           —           —     0.636ms

Legend:

• P: = Prompt hooks (pre/post)
• T: = Tool hooks (pre/post invoke)
• R: = Resource hooks (pre/post fetch)
• — = Hook not implemented by plugin

Usage

Run Performance Tests

# From project root
LOG_LEVEL=ERROR \
python tests/performance/test_plugins_performance.py 2>/dev/null

Analyze Specific Profile

python -c "import pstats; \
  p = pstats.Stats('prof/PIIFilterPlugin_tool_pre_invoke.prof'); \
  p.strip_dirs().sort_stats('cumulative').print_stats(20)"

Visualize with SnakeViz

pip install snakeviz
snakeviz prof/PIIFilterPlugin_tool_pre_invoke.prof

Create call graph

gprof2dot -f pstats plugins/prof/SecretsDetection_prompt_pre_fetch.prof | dot -Tsvg -o plugins/prof/SecretsDetection_prompt_pre_fetch.svg 

Create flame graph

flameprof plugins/prof/PIIFilterPlugin_tool_pre_invoke.prof > plugins/prof/flamegraph.svg

Benefits

1. Identify Bottlenecks: Pinpoint slow functions in plugin execution
2. Measure Optimizations: Compare before/after profiles quantitatively
3. Regression Detection: Track performance over time in CI/CD
4. Plugin Development: Optimize new plugins during development
5. Framework Optimization: Identify framework overhead vs plugin work

Documentation

Complete documentation available in:

• tests/performance/PLUGIN_PROFILING.md