add /admin/tools/import bulk importer#734
Conversation
33bd3f3 to
4f87f97
Compare
|
Can you clarify how this would be different from the current |
|
@imolloy yes. My thought was to add bulk tools import that accepts a JSON array of tools and returns per-item outcomes (created / errors). Supports partial success and includes basic guardrails (batch cap + rate limit). I was thinking this could not only help with the efficiency but also promote file-based config. In the future, we can add tools configuration export and CI/CD integration support |
Signed-off-by: Vicky <vicky.kuo.contact@gmail.com>
4f87f97 to
a4ac912
Compare
Signed-off-by: Mihai Criveti <crivetimihai@gmail.com>
Signed-off-by: Mihai Criveti <crivetimihai@gmail.com>
crivetimihai
left a comment
There was a problem hiding this comment.
PR #734 Review: Bulk Tools Import Feature
Feature: Adds POST /admin/tools/import endpoint for bulk importing tools
Purpose: Enable teams to quickly seed/migrate multiple tools in one request instead of one-by-one
What the PR Does Well ✅
- Resilient Design - Per-item validation ensures one bad tool doesn't fail the entire batch
- Flexible Input - Accepts both JSON and form data (tools_json field)
- Detailed Response - Returns success/failure counts with specific error details per tool
- Rate Limited - Appropriately limited to 10 requests/minute for bulk operations
- Size Protection - Max 200 tools per batch prevents abuse
- Error Handling - Comprehensive try/catch with proper error formatting
Issues Found & Fixed 🔧
1. Duplicate Router Registration (CRITICAL BUG)
Issue: PR added app.include_router(admin_router) at line 249 unconditionally
Problem: Router was already conditionally included at line 2700 based on MCPGATEWAY_ADMIN_API_ENABLED
Impact: Would cause routing conflicts and expose admin endpoints even when disabled
Fix: Removed duplicate registration at line 249
2. Linting Violations
Flake8:
- Changed
MAX_BATCHtomax_batch(PEP8 naming convention) - Fixed trailing whitespace on blank lines
Pylint:
- Added missing docstrings for
rate_limitfunction - Added missing docstrings for
admin_import_toolsfunction - Achieved 10.00/10 rating
Interrogate:
- Added comprehensive docstrings for nested functions in rate_limit decorator
- Achieved 100% docstring coverage (was 99.6%)
3. Rate Limiter Simplification
Change: Docstrings were removed from rate_limit decorator
Assessment: While this reduces inline documentation, functionality remains intact
Fix Applied: Re-added comprehensive docstrings with proper formatting
Test Coverage Added 📊
New Test Class: TestAdminBulkImportRoutes
Added 12 comprehensive test cases:
- test_bulk_import_success - Valid tools import successfully
- test_bulk_import_partial_failure - Mixed success/failure handling
- test_bulk_import_validation_errors - Pydantic validation failures
- test_bulk_import_empty_array - Empty input gracefully handled
- test_bulk_import_not_array - Non-array payload properly rejected
- test_bulk_import_exceeds_max_batch - Batch size limit enforced
- test_bulk_import_form_data - Form data support verified
- test_bulk_import_invalid_json_payload - JSON parsing errors handled
- test_bulk_import_form_invalid_json - Form with invalid JSON handled
- test_bulk_import_form_missing_field - Missing form fields caught
- test_bulk_import_unexpected_exception - Generic errors handled
- test_bulk_import_rate_limiting - Rate limit decorator verified
Coverage Improvement: Admin module improved from 20% to 65%
My Changes Summary 📝
Fixes Applied:
# 1. Removed duplicate router registration
- app.include_router(admin_router) # Line 249 - REMOVED
# 2. Fixed variable naming
- MAX_BATCH = 200
+ max_batch = 200
# 3. Added comprehensive docstrings
def rate_limit(requests_per_minute: int = None):
"""Apply rate limiting to admin endpoints.
Args:
requests_per_minute: Maximum requests per minute (uses config default if None)
Returns:
Decorator function that enforces rate limiting
"""
async def admin_import_tools(...):
"""Bulk import multiple tools in a single request.
Accepts a JSON array of tool definitions and registers them individually.
Provides per-item validation and error reporting without failing the entire batch.
...
"""Quality Metrics Achieved:
- ✅ Pylint: 10.00/10 rating
- ✅ Flake8: Zero violations
- ✅ Interrogate: 100% docstring coverage
- ✅ Tests: All 12 new tests passing
- ✅ Coverage: 65% admin.py coverage (was 20%)
API Usage Example
# Generate JWT token
TOKEN=$(python3 -m mcpgateway.utils.create_jwt_token \
--username admin --exp 60 --secret $JWT_SECRET_KEY)
# Import tools
curl -X POST http://localhost:8000/admin/tools/import \
-H "Authorization: Bearer $TOKEN" \
-H "Content-Type: application/json" \
-d '[
{
"name": "list_users",
"url": "https://api.example.com/users",
"integration_type": "REST",
"request_type": "GET"
},
{
"name": "create_user",
"url": "https://api.example.com/users",
"integration_type": "REST",
"request_type": "POST",
"input_schema": {
"type": "object",
"properties": {"body": {"type": "object"}},
"required": ["body"]
}
}
]'
# Response
{
"success": true,
"created_count": 2,
"failed_count": 0,
"created": [
{"index": 0, "name": "list_users"},
{"index": 1, "name": "create_user"}
],
"errors": []
}Recommendation
✅ READY TO MERGE
The PR provides valuable functionality for bulk operations. All issues have been fixed:
- Router duplication bug resolved
- Full linting compliance achieved
- Comprehensive test coverage added
- 100% documentation coverage
The feature is ready and will significantly improve the onboarding experience for teams managing multiple tools. Thank you!!
* add /admin/tools/import bulk importer Signed-off-by: Vicky <vicky.kuo.contact@gmail.com> * Lint fixes Signed-off-by: Mihai Criveti <crivetimihai@gmail.com> * Docs updates Signed-off-by: Mihai Criveti <crivetimihai@gmail.com> --------- Signed-off-by: Vicky <vicky.kuo.contact@gmail.com> Signed-off-by: Mihai Criveti <crivetimihai@gmail.com> Co-authored-by: Vicky <vicky.kuo.contact@gmail.com> Co-authored-by: Mihai Criveti <crivetimihai@gmail.com>
* add /admin/tools/import bulk importer Signed-off-by: Vicky <vicky.kuo.contact@gmail.com> * Lint fixes Signed-off-by: Mihai Criveti <crivetimihai@gmail.com> * Docs updates Signed-off-by: Mihai Criveti <crivetimihai@gmail.com> --------- Signed-off-by: Vicky <vicky.kuo.contact@gmail.com> Signed-off-by: Mihai Criveti <crivetimihai@gmail.com> Co-authored-by: Vicky <vicky.kuo.contact@gmail.com> Co-authored-by: Mihai Criveti <crivetimihai@gmail.com>
* add /admin/tools/import bulk importer Signed-off-by: Vicky <vicky.kuo.contact@gmail.com> * Lint fixes Signed-off-by: Mihai Criveti <crivetimihai@gmail.com> * Docs updates Signed-off-by: Mihai Criveti <crivetimihai@gmail.com> --------- Signed-off-by: Vicky <vicky.kuo.contact@gmail.com> Signed-off-by: Mihai Criveti <crivetimihai@gmail.com> Co-authored-by: Vicky <vicky.kuo.contact@gmail.com> Co-authored-by: Mihai Criveti <crivetimihai@gmail.com>
Summary
Add a bulk importer API for Tools :
POST /admin/tools/importMotivation
Onboarding many REST/MCP tools one-by-one is slow and error-prone. Teams need a quick way to seed or migrate tools across environments. This MVP unblocks that workflow and is resilient to mixed-good/bad items in the same batch.
What’s Changed
mcpgateway/admin.pyadds:Returns:
API (New)
Request:
POST /admin/tools/importAuthorization: Bearer <jwt>,Content-Type: application/json{"name":"list_users","url":"https://api.example.com/users","integration_type":"REST","request_type":"GET"},
{
"name":"create_user",
"url":"https://api.example.com/users",
"integration_type":"REST",
"request_type":"POST",
"input_schema":{"type":"object","properties":{"body":{"type":"object"}},"required":["body"]}
}
]
curl -sS -u USER:PASS -L -D headers.txt -o /dev/null http://localhost:4444/admin/
TOKEN=$(awk -F'[;= ]' 'tolower($1)=="set-cookie:" && $2=="jwt_token" {print $3}' headers.txt | tail -1)
curl -sS http://localhost:4444/admin/tools -H "Authorization: Bearer $TOKEN"
cat > tools.json <<'JSON'
[
{"name":"list_users","url":"https://api.example.com/users","integration_type":"REST","request_type":"GET"},
{"name":"create_user","url":"https://api.example.com/users","integration_type":"REST","request_type":"POST",
"input_schema":{"type":"object","properties":{"body":{"type":"object"}},"required":["body"]}}
]
JSON
curl -i -X POST http://localhost:4444/admin/tools/import
-H "Authorization: Bearer $TOKEN"
-H "Content-Type: application/json"
--data-binary @tools.json
Expect 200 with created_count:2, failed_count:0