-
-
Notifications
You must be signed in to change notification settings - Fork 52.5k
Closed
Description
Priority
🔴 URGENT - HIGHEST
Summary
Multiple bots affected by cache system recursion bug triggered by image processing. Verified cases: prime-heron ($15.09 in 90 minutes) and sharp-otter ($8.58). System generates 26-33 million cache read tokens per image.
Verified Impact
Prime-heron (Jinbot):
Image processing period: 2026-02-26 05:40-07:10 (90 minutes)
Historical consumption: $15.30 (normal usage over days)
Image bug consumption: $15.09 (90 minutes)
Cache read tokens: 33,150,331 ⚠️ CRITICAL
Cache write tokens: 5,084,601 ⚠️ CRITICAL
Request count: 847
Sharp-otter (Ace Bot):
Similar period consumption: $8.58
Cache read tokens: 26,461,934 ⚠️ CRITICAL
Cache write tokens: 2,182,666 ⚠️ CRITICAL
Request count: 727
Technical Details
Environment:
- OpenClaw Version: 2026.2.6-3
- Channel: WhatsApp group chat
- Affected: Multiple bots on same platform
Timeline (prime-heron):
05:40:09- Received image message (129 chars)05:40:11- Image resized: 134.5KB → 96.4KB (-28.3%)05:40:16- Auto-replied to group05:40 - 07:10- Cache recursion loop begins07:10- Budget exhausted, service interruption
Root Cause Analysis:
- Image processing function triggers infinite cache recursion
- Each cache read operation generates more cache reads
- Exponential cache operation growth: 26M-33M reads per image
- Bug affects multiple bots consistently
Business Impact
- Financial: $23+ lost across multiple bots
- Service: User experience degraded, bots offline
- Platform: Potential infrastructure strain from cache abuse
- Trust: Users experiencing unexpected charges
Immediate Mitigations Applied
- 🚨 User advisory: All users notified to avoid sending images
- 📊 Monitoring: Enhanced tracking of cache metrics
- 💰 Budget: Temporarily increased limits for affected bots
Requested Resolution
- Emergency fix: Identify and patch cache recursion in image pipeline
- Circuit breakers: Implement cache operation limits (e.g., max 1K reads per request)
- Monitoring: Real-time alerts for abnormal cache patterns
- Testing: Regression tests for image processing workflows
- Documentation: Post-incident analysis and prevention guidelines
Reproduction Steps
- Send any image to WhatsApp bot
- Monitor cache read/write metrics
- Observe exponential growth pattern
- Watch token consumption spike
Additional Data Available
- Full usage metrics for both affected bots
- Docker logs showing exact timing
- Cache operation patterns
Reactions are currently unavailable
Metadata
Metadata
Assignees
Labels
No labels