Skip to content

feat(daemon): allows daemonization options to be fetched from app settings#8553

Merged
Nusnus merged 13 commits intocelery:mainfrom
noirbizarre:cli/daemon-options-from-config
Jan 17, 2024
Merged

feat(daemon): allows daemonization options to be fetched from app settings#8553
Nusnus merged 13 commits intocelery:mainfrom
noirbizarre:cli/daemon-options-from-config

Conversation

@noirbizarre
Copy link
Contributor

Description

This pull-requests allows to provide daemonization settings (--logfile, --pidfile, --uid, --gid, --umask and --executable) from your config file with the following new settings:

  • beat_logfile
  • beat_pidfile
  • beat_uid
  • beat_gid
  • beat_umask
  • beat_executable
  • worker_logfile
  • worker_pidfile
  • worker_uid
  • worker_gid
  • worker_umask
  • worker_executable
  • events_logfile
  • events_pidfile
  • events_uid
  • events_gid
  • events_umask
  • events_executable

This PR is backward compatible, tested and documented. Missing CLI documentation for daemonization settings have been added in the process.

Copy link
Member

@auvipy auvipy left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

thanks for the PR. can you check the CI failures please?

@noirbizarre noirbizarre force-pushed the cli/daemon-options-from-config branch 2 times, most recently from efc6a15 to 22f28a4 Compare October 3, 2023 17:29
@codecov
Copy link

codecov bot commented Oct 3, 2023

Codecov Report

All modified and coverable lines are covered by tests ✅

Comparison is base (53f3000) 87.45% compared to head (2e818c3) 87.36%.
Report is 81 commits behind head on main.

Additional details and impacted files
@@            Coverage Diff             @@
##             main    #8553      +/-   ##
==========================================
- Coverage   87.45%   87.36%   -0.10%     
==========================================
  Files         148      148              
  Lines       18499    18525      +26     
  Branches     3158     3165       +7     
==========================================
+ Hits        16179    16184       +5     
- Misses       2032     2056      +24     
+ Partials      288      285       -3     
Flag Coverage Δ
unittests 87.33% <100.00%> (-0.10%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@noirbizarre
Copy link
Contributor Author

noirbizarre commented Oct 3, 2023

There it is, PR fixed, Ci is all green 👍🏼

@noirbizarre noirbizarre force-pushed the cli/daemon-options-from-config branch 5 times, most recently from 865aaab to 639f8a4 Compare October 3, 2023 23:57
@noirbizarre noirbizarre force-pushed the cli/daemon-options-from-config branch from 639f8a4 to dc774f6 Compare October 4, 2023 00:54
@auvipy auvipy self-requested a review October 4, 2023 10:38
@auvipy
Copy link
Member

auvipy commented Oct 4, 2023

What extra benefits the changes will provide?

@noirbizarre
Copy link
Contributor Author

noirbizarre commented Oct 4, 2023

Today, the only way to provide those values are from cli flag which prevent retrieving those values from any registry (unless you have a way to fetch those values before running those daemons and passing the processed result to the cli, which is not possible in my case).
This change allows to specify those values from settings (like most of the settings you can set on cli), meaning from Python, allowing for dynamic values (which is my use case) and any kind of preprocessing.
This also works for static settings in the case you have a templated celeryconf.py loaded with Celery.config_from_object() (the exact case which is tested in this PR and was not possible without this PR, cf. https://github.com/celery/celery/pull/8553/files#diff-4e22bffa23e4e274ffb04c27895f3bec22ce4c0e41174f5e4dac197c18116a82)

@auvipy
Copy link
Member

auvipy commented Oct 4, 2023

yeah that looks nice. I am in favor of it. Lets us wait for other maintainers to look into it. I would appreciate if in the mean time you could review this PR #8489

auvipy and others added 10 commits November 7, 2023 20:34
Co-authored-by: Omer Katz <omer.katz@kcg.tech>
Co-authored-by: Omer Katz <omer.katz@kcg.tech>
Co-authored-by: Omer Katz <omer.katz@kcg.tech>
Co-authored-by: Omer Katz <omer.katz@kcg.tech>
Co-authored-by: Omer Katz <omer.katz@kcg.tech>
Co-authored-by: Omer Katz <omer.katz@kcg.tech>
Co-authored-by: Omer Katz <omer.katz@kcg.tech>
Co-authored-by: Omer Katz <omer.katz@kcg.tech>
Co-authored-by: Omer Katz <omer.katz@kcg.tech>
Co-authored-by: Omer Katz <omer.katz@kcg.tech>
@auvipy auvipy added this to the 5.4 milestone Nov 7, 2023
Co-authored-by: Omer Katz <omer.katz@kcg.tech>
Copy link
Member

@Nusnus Nusnus left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@Nusnus
Copy link
Member

Nusnus commented Dec 6, 2023

In addition, all those new configuration settings should be marked with .. versionadded:: 5.4

@noirbizarre thank you for this PR!
Please add the missing 5.4 doc, and I'll merge it right away.

@Nusnus
Copy link
Member

Nusnus commented Jan 10, 2024

@noirbizarre the version doc is important for understanding changes on a release and IIRC that's the only gap on this PR.

As we want to improve our communication on releases, I'd appreciate it if you could please close the small gap so we can merge it in, these are nice changes overall and the finishing line is very near! 😉

@noirbizarre
Copy link
Contributor Author

Hi 👋🏼
Sorry for the lack of response lately (overbooked, holidays, children... life). I'll read the entire thread and update the PR accordingly by the end of week (maybe tonight if the remaining tasks are quick) !

@Nusnus
Copy link
Member

Nusnus commented Jan 10, 2024

Hi 👋🏼 Sorry for the lack of response lately (overbooked, holidays, children... life). I'll read the entire thread and update the PR accordingly by the end of week (maybe tonight if the remaining tasks are quick) !

Amazing!
Tag me when you’re done and I’ll finish the job on our side.

Thank you!

@noirbizarre
Copy link
Contributor Author

There it is ! Version added markers for 5.4 added on all settings 👍🏼

Thanks for the review and the merge 🙏🏼

@Nusnus
Copy link
Member

Nusnus commented Jan 11, 2024

There it is ! Version added markers for 5.4 added on all settings 👍🏼

Thanks for the review and the merge 🙏🏼

Thank you!!
Unfortunately there's one last detail my friend, the version needs to be first under each title and not after the documentation text.

It's important to keep consistency (see other places) so I hope it's not too annoying to ask for this last fix please @noirbizarre 🙏

Copy link
Member

@auvipy auvipy left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

looks good. just the remaining review issues need to be addressed

@Nusnus Nusnus merged commit 2576e83 into celery:main Jan 17, 2024
Nusnus added a commit to Katz-Consulting-Group/celery that referenced this pull request Jan 17, 2024
@Nusnus
Copy link
Member

Nusnus commented Jan 17, 2024

looks good. just the remaining review issues need to be addressed

Fixed my own comment here: #8802

@noirbizarre all good ❤️

@Nusnus
Copy link
Member

Nusnus commented Jan 17, 2024

😃

make -C docs changes
open docs/_build/changes/index.html

CleanShot 2024-01-18 at 01 37 34@2x

@noirbizarre noirbizarre deleted the cli/daemon-options-from-config branch September 23, 2024 12:41
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

Status: Done

Development

Successfully merging this pull request may close these issues.

3 participants