[Pfc] Implementation of ResourceMonitor and PurgePlugin #2436
Conversation
|
For me it's ok to add this as a single commit. However, I suggest something like shown below for the commit message to give both authors proper credit. |
|
Thank you, will update the commit message before the final version of the PR. |
|
The 5.8 merge window is now, I have already merged most things that will go into the next release. There's still lots to do before we release, however, so there's still time to test this before the final merge. |
|
OK, thanks! I will get this ready before the coming Monday then, including testing at UCSD in production, have it mostly setup already. |
|
This is actually out in production at a handful of caches -- so far, no major issues in the basic XCache functionality. Still have yet to enable the purge plugin (off by default) functionality thoroughly. |
1620b01 to
cbe0973
Compare
|
@amadio This should now be ready to go. I have removed the accidental change in readme and formatted the commit message as advised. Thank you very much for your help!!! |
|
@osschar Thank you! I see a couple of typos in the first commit (e.g. |
89a8217 to
983bf61
Compare
|
Thank you -- done as requested, this time with spell checker :) |
Resource Monitor (main class XrdPfcResourceMonitor) provides detailed and
accurate snapshots of state and usage of existing directories. It allows
optimization of per-directory statistics information collection and storage,
and reduces the number of required cache namespace traversals. It can
periodically export the state into JSON and binary files for viewing of
current state and for further processing or aggregation. This
information also serves as input to purge process.
Purge plugin provides a class-based API that allows users to implement their
own steering of the purge priority selection. The purge is still directory
driven, i.e., the purge plugin is given a per-directory snapshot of resource
monitor state and responds with a list of { directory-path, bytes_to_remove }
structures. Pfc still drives the actual low-level directory traversal and file
purge -- though this could be relaxed in the future. If the plugin does not
schedule sufficient removals, the classic LRU algorithm kicks in to satisfy
the high/low watermark and file-based limits.
The purge-plugin API is defined in class Pfc::PurgePin. Reference
implementation, directory-quota based purging, is provided in the core PFC
library class XrdPfcPurgeQuota and enabled as:
pfc.purgelib libXrdPfcPurgeQuota /etc/xrootd/quota.cfg
with entries in quota cfg file looking like:
/store/mc 10T
/store/data 12T
Co-authored-by: Alja Mrak-Tadel <amraktadel@ucsd.edu>
983bf61 to
a2b4940
Compare
Resource Monitor (main class
XrdPfcResourceMonitor) provides detailed andaccurate snapshots of state and usage of existing directories. It allows
optimization of per-directory statistics information collection and storage,
and reduces the number of required cache namespace traversals. It can
periodically export the state into json and binary files for viewing of
current state and for furhter processing or aggregation. Further, this
information also serves as input to purge process.
Purge plugin provides a class-based API that allows users to implement their
own steering of the purge priority selection. The purge is still directory
driven, i.e., the purge plugin is given a per-directory snapshot of resource
monitor state and responds with a list of
{ directory-path, bytes_to_remove }structures. Pfc still drives the actual low-level directory traversal and file
purge -- though this could be relaxed in the future. If the plugin does not
schedule sufficient removals, the classic LRU algorithm kicks in to satisfy
the high/low watermark and file-based limits.
The purge-plugin API is defined in class
Pfc::PurgePin. Referenceimplementation, directory-quota based purging, is provided in the core PFC
library class
XrdPfcPurgeQuotaand can be enabled as:with entries in quota cfg file looking like:
This is squashed into a single commit, over several rebases. For full history of commits see these branches: