Skip to content

Optimize Celery tasks by implementing connection pooling #88

Merged
tarekio merged 2 commits intomainfrom
celery-sqlalchemy-connection-optimization
Mar 28, 2025
Merged

Optimize Celery tasks by implementing connection pooling #88
tarekio merged 2 commits intomainfrom
celery-sqlalchemy-connection-optimization

Conversation

@level09
Copy link
Collaborator

@level09 level09 commented Mar 20, 2025

This PR enhances performance of Celery tasks by implementing connection pooling. It reuses the Flask application instance across worker processes instead of creating a new one for each task execution.

Key improvements:

  • Reduces database connection overhead
  • Minimizes Flask app initialization costs
  • Uses worker_process_init signal to initialize app at worker startup
  • Provides fallback lazy loading for edge cases
  • These changes significantly reduce resource consumption for high-frequency task processing.

Jira Issue

  1. [Add links to jira issues]

Description

[Brief description of changes]

Checklist

  • Tests added/updated
  • Documentation updated (if needed)
  • New strings prepared for translations

API Changes (if applicable)

  • Permissions checked
  • Endpoint tests added

Additional Notes

[Any other relevant information]

…p reuse across worker processes to reduce overhead and improve performance of database operations
@level09 level09 self-assigned this Mar 20, 2025
@tarekio tarekio self-requested a review March 21, 2025 20:21
@tarekio tarekio merged commit 390dae3 into main Mar 28, 2025
6 checks passed
@tarekio tarekio deleted the celery-sqlalchemy-connection-optimization branch March 28, 2025 21:13
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants