Skip to content

feat: enable client-side metrics scraping for JindoCache engine #4331

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
fluid-e2e-bot merged 12 commits into from
Oct 8, 2024
Merged

feat: enable client-side metrics scraping for JindoCache engine #4331

fluid-e2e-bot merged 12 commits into from
Oct 8, 2024

Conversation

@TrafalgarZZZ
Copy link
Member

@TrafalgarZZZ TrafalgarZZZ commented Sep 29, 2024

Ⅰ. Describe what this PR does

Three parts of code changes have been made in this pull request:

  • Inject Fuse sidecar in a determined lexicographic order based on datasets' names. That is, we can make sure the mapping between sidecar container's name and the runtimes will be fixed when there are multiple FUSE sidecars to inject.
  • Add spec.fuse.metrics to JindoRuntime's CRD. See below for some examples.
  • Some inner logic used to support spec.fuse.metrics

Feature API

apiVersion: data.fluid.io/v1alpha1
kind: JindoRuntime
...
spec:
  fuse:
    metrics:
      enabled: true
      scrapeTarget: All

Two fields can be specified under spec.fuse.metrics in JindoRuntime CRD.

  • With enabled: true, Fluid exposes metrics for all FUSE components, including FUSE mount pods and FUSE sidecar containers.
  • scrapeTarget has four valid values:
    • "": The default value, indicating Fluid will not automatically scrape FUSE metrics. (Under the hood, will not add special annotation that makes Prometheus discover FUSE components.)
    • MountPodOnly: Only FUSE mount pods will be discovered by Prometheus and metrics will be scraped.
    • SidecarOnly: Only FUSE sidecar containers will be discovered by Prometheus and metrics will be scraped.
    • All: Both FUSE mount pods and FUSE sidecar containers will be discovered by Prometheus and metrics will be scraped.

Discovered FUSE mount pod example with enabled metrics

apiVersion: v1
kind: Pod
metadata:
  annotations:
    prometheus.fuse.fluid.io/scrape: "true"
...
spec:
  containers:
  - name: jindofs-fuse
    ...
    ports:
    - name: jindo-metrics
      containerPort: 10001 # (default in ContainerNetwork) or <allocated random port> (in HostNetwork)
    args:
    - -okernel_cache
    - -oro
    - -oattr_timeout=3600
    - -oentry_timeout=3600
    - -ometrics_port=10001 # or <allocated randome port>

Discovered FUSE sidecar container example with enabled metrics

apiVersion: v1
kind: Pod
metadata:
  annotations:
    prometheus.fuse.fluid.io/scrape: "true"
  labels:
    container-dataset-mapping.sidecar.fluid.io/fluid-fuse-0: default_demo-dataset # <namespace>_<dataset_name>, used to identify the relationship between dataset and sidecar container.
spec:
  containers:
  - name: fluid-fuse-0
    args:
    - -okernel_cache
    - -oro
    - -oattr_timeout=3600
    - -oentry_timeout=3600
    - -ometrics_port=10001 # or <allocated randome port>
    ports:
    - name: jindo-metrics
      containerPort: 10001 # (default in ContainerNetwork) or <allocated random port> (in HostNetwork)

Ⅱ. Does this pull request fix one issue?

NONE

Ⅲ. List the added test cases (unit test/integration test) if any, please explain if no tests are needed.

Ⅳ. Describe how to verify it

Ⅴ. Special notes for reviews

@TrafalgarZZZ
Copy link
Member Author

TrafalgarZZZ commented Sep 29, 2024

/hold

@codecov
Copy link

codecov bot commented Sep 29, 2024

Codecov Report

Attention: Patch coverage is 21.21212% with 78 lines in your changes missing coverage. Please review.

Project coverage is 58.24%. Comparing base (22e808c) to head (408d2d4).
Report is 31 commits behind head on master.

Files with missing lines Patch % Lines
...application/inject/fuse/mutator/mutator_default.go 0.00% 27 Missing ⚠️
pkg/ddc/jindocache/transform.go 29.41% 16 Missing and 8 partials ⚠️
pkg/ddc/jindocache/sync_runtime.go 15.38% 8 Missing and 3 partials ⚠️
...cation/inject/fuse/mutator/mutator_unprivileged.go 0.00% 8 Missing ⚠️
pkg/utils/map.go 0.00% 6 Missing ⚠️
pkg/ddc/base/runtime.go 77.77% 2 Missing ⚠️
Additional details and impacted files 
@@            Coverage Diff             @@
##           master    #4331      +/-   ##
==========================================
- Coverage   62.90%   58.24%   -4.67%     
==========================================
  Files         483      563      +80     
  Lines       28570    31367    +2797     
==========================================
+ Hits        17972    18269     +297     
- Misses       8393    10837    +2444     
- Partials     2205     2261      +56
Flag Coverage Δ
58.24% <21.21%> (?)

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@TrafalgarZZZ
Implement JindoRuntime.spec.fuse.metrics.scrapeTarget
c0c861b 
Signed-off-by: trafalgarzzz <trafalgarz@outlook.com>
@TrafalgarZZZ
Sync metrics scrape annotation according to runtime.Spec.Fuse.Metrics…
89fc969 
….ScrapeTarget

Signed-off-by: trafalgarzzz <trafalgarz@outlook.com>
@TrafalgarZZZ
Sync metrics scrape annotation according to runtime.Spec.Fuse.Metrics…
4fa1105 
….ScrapeTarget

Signed-off-by: trafalgarzzz <trafalgarz@outlook.com>
@TrafalgarZZZ
Support injecting annotation for fuse sidecars
29ce7c6 
Signed-off-by: trafalgarzzz <trafalgarz@outlook.com>
@TrafalgarZZZ
Support injecting annotation for fuse sidecars
30aa51d 
Signed-off-by: trafalgarzzz <trafalgarz@outlook.com>
@TrafalgarZZZ
Support injecting annotation for fuse sidecars
e5a5130 
Signed-off-by: trafalgarzzz <trafalgarz@outlook.com>
@TrafalgarZZZ
Remove container port when scrape target does not include sidecars
e8fef05 
Signed-off-by: trafalgarzzz <trafalgarz@outlook.com>
@TrafalgarZZZ
Inject FUSE sidecar in alphanumberic order & fix unit tests
f17d73f 
Signed-off-by: trafalgarzzz <trafalgarz@outlook.com>
@TrafalgarZZZ
Update openapi
fbef9f7 
Signed-off-by: trafalgarzzz <trafalgarz@outlook.com>
@TrafalgarZZZ
Add unit tests
9fc5386 
Signed-off-by: trafalgarzzz <trafalgarz@outlook.com>
@TrafalgarZZZ
Add unit tests
b0e7f7f 
Signed-off-by: trafalgarzzz <trafalgarz@outlook.com>
@TrafalgarZZZ TrafalgarZZZ force-pushed the feat/enable_jindo_cache_fuse_metrics branch from d60b25b to b0e7f7f Compare October 8, 2024 02:32
@TrafalgarZZZ
Copy link
Member Author

TrafalgarZZZ commented Oct 8, 2024

/unhold

cheyang
cheyang reviewed Oct 8, 2024
View reviewed changes
cheyang
cheyang reviewed Oct 8, 2024
View reviewed changes
pkg/ddc/jindocache/sync_runtime.go
}

metricsEnabled, exists := fusesToUpdate.Spec.Template.ObjectMeta.Annotations[common.AnnotationPrometheusFuseMetricsScrapeKey]
if exists && metricsEnabled != common.True {
Copy link
Collaborator

@cheyang cheyang Oct 8, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

How about letting the code exit early if the annotation exists but is not equal to common.True, reducing nesting?

if exists && metricsEnabled != common.True {
    e.Log.V(1).Info(fmt.Sprintf("Found user-defined annotation %s != %s, skip syncing.", common.AnnotationPrometheusFuseMetricsScrapeKey, common.True))
    return
}

Copy link
Member Author

@TrafalgarZZZ TrafalgarZZZ Oct 8, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@cheyang The nesting is necessary. L235-L248 updates FUSE's daemonset if changed==true. If the code exits early, the update will be skipped when runtime.Spec.Fuse.Resources is modified(at L213).

@TrafalgarZZZ
fix error message
d31f851 
Signed-off-by: trafalgarzzz <trafalgarz@outlook.com>
@sonarqubecloud
Copy link

sonarqubecloud bot commented Oct 8, 2024

Quality Gate Passed Quality Gate passed

Issues
10 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
60.0% Duplication on New Code

See analysis details on SonarCloud

cheyang
cheyang approved these changes Oct 8, 2024
View reviewed changes
Copy link
Collaborator

@cheyang cheyang left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

/lgtm
/approve

@fluid-e2e-bot fluid-e2e-bot bot added the lgtm label Oct 8, 2024
@fluid-e2e-bot
Copy link

fluid-e2e-bot bot commented Oct 8, 2024

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: cheyang

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@fluid-e2e-bot fluid-e2e-bot bot merged commit c84a499 into fluid-cloudnative:master Oct 8, 2024
@TrafalgarZZZ TrafalgarZZZ mentioned this pull request Oct 10, 2024
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@cheyang cheyang cheyang approved these changes

Assignees

@cheyang cheyang

Labels

approved lgtm

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@TrafalgarZZZ @cheyang