Pinging @elastic/elastic-agent (Team:Elastic-Agent)

This pull request does not have a backport label.
If this is a bug or security fix, could you label this PR @fearful-symmetry? 🙏.
For such, you'll need to label your PR with:

• The upcoming major version of the Elastic Stack
• The upcoming minor version of the Elastic Stack (if you're not pushing a breaking change)

To fixup this pull request, you need to add the backport labels for the needed
branches, such as:

• backport-v8./d.0 is the label to automatically backport to the 8./d branch. /d is the digit

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

❕ Flaky test report

No test was executed to be analysed.

🤖 GitHub comments

@cmacknz if I understand #38708, all the rest of the the implementation of this is in integrations/dashboards. Is someone on the integrations side doing that, or are we doing the dashboards too?

I would update the integration as part of this work. If you can't for some reason, create a separate issue to do it so it isn't lost.

Now that we can know the queue size in bytes, I'm just wondering if we should make it more clear that the percentage is calculated from the number of events.

The queue size limit is still configured in units of events and the metric here should match the units of the maximum size limit.

Pinging @elastic/elastic-agent-data-plane (Team:Elastic-Agent-Data-Plane)

Huh, it's possible for activeEvents > max_events:

{"clients":2,"events":{"active":3201,"published":99200,"total":99200},"queue":{"acked":99200,"filled":{"pct":{"events":1.0003125}},"max_events":3200}}

This happens occasionally, in the metrics, and it's always max_queue+1. I assume it's just a result of the queue itself not being completely synced up with the metrics reporter. That, or an off-by-one. @faec you seen this before?

This happens occasionally, in the metrics, and it's always max_queue+1. I assume it's just a result of the queue itself not being completely synced up with the metrics reporter. That, or an off-by-one. @faec you seen this before?

Yeah, I've seen this, it's just an artifact of the metrics receiver getting notified of acks slightly after the queue unblocks so it sees the new event in flight before decrementing the old ones. I've never seen the pipeline report something that's actually wrong, just slightly behind on acks, though I did see recently that the output subtree sometimes has a number that's just flat-out wrong #39146

Alright, added rounding. I decided to keep the percent calculations "raw" (0.0-1.0, not 0.0-100.0), since that's what we do for the system metrics percentages, and I figure we might as well keep it consistent?