docker metrics (read metrics from cgroups for specified container) by monsterzz · Pull Request #8886 · moby/moby

monsterzz · 2014-10-31T14:45:04Z

Fixes #8842.

This is first and very simple implementation of docker metrics command. A lot of work needed to push it to production.

Roadmap:

make find of cgroup paths more reliable (not just try to search predefined locations, but get exact value from procfs)
specify set of metrics (from cpuacct, memory, blkio, ... controllers)
store metrics in struct instead of map[string]string (typed data)
-f FORMAT support
documentation

PS. PR created for further discussion and this branch will be constantly updated.

/cc @shykes @tobegit3hub @thaJeztah

Signed-off-by: Gleb M Borisov <borisov.gleb@gmail.com>

erikh · 2014-10-31T19:22:10Z

api/client/commands.go

use json.Unmarshal and a struct or map for this, please.

It's copy-paste from CmdInspect. I've marked both functions with TODO (will
extract common things and make their code less ugly).

Thanks for the tip, just a first week with Go :)
On Fri 31 Oct 2014 at 10:22 pm Erik Hollensbe notifications@github.com
wrote:

In api/client/commands.go:

@@ -812,6 +812,46 @@ func (cli *DockerCli) CmdInspect(args ...string) error {
return nil
}

+func (cli *DockerCli) CmdMetrics(args ...string) error {

cmd := cli.Subcmd("metrics", "CONTAINER", "Return runtime information on a container")

if err := cmd.Parse(args); err != nil {

return nil

}

if cmd.NArg() < 1 {

cmd.Usage()

return nil

}

indented := new(bytes.Buffer)

use json.Unmarshal and a struct or map for this, please.

—
Reply to this email directly or view it on GitHub
https://github.com/docker/docker/pull/8886/files#r19688050.

cool. thanks. just let us know when it's ready for review again.

SvenDowideit · 2014-11-03T06:19:21Z

needs documentation in cli.md, a man page (in docs/man) and in runmetrics.md

jeremyeder · 2014-11-13T20:53:12Z

Hi @crosbymichael and @monsterzz this is along the lines of what we're doing with nsinit at the moment. Nice to see it potentially in docker itself. This patch currently does not work on RHEL-like systems, and I think @monsterzz eluded to the cgroup path config needing some work, which is fine. I didn't look into it deeply but hope that can be resolved.

    "Metrics": {
        "CpuUsage": "Error: cgroup subsystem 'cpuacct' directory not found for container 'a7412e396560bcc886009472fdcd0433203598b0c4a11a9dfd4223e33c1c9173'",
        "MemoryLimit": "Error: cgroup subsystem 'memory' directory not found for container 'a7412e396560bcc886009472fdcd0433203598b0c4a11a9dfd4223e33c1c9173'",
        "MemoryMaxUsage": "Error: cgroup subsystem 'memory' directory not found for container 'a7412e396560bcc886009472fdcd0433203598b0c4a11a9dfd4223e33c1c9173'",
        "MemoryUsage": "Error: cgroup subsystem 'memory' directory not found for container 'a7412e396560bcc886009472fdcd0433203598b0c4a11a9dfd4223e33c1c9173'"
    },

I would expect a monitoring system will want a global "watch" on all containers, existing and new. Do you agree ? Basically docker would continuously pump all it's stats to a websocket at a certain interval, and the monitoring system would consume at it's own pace.

You could have i.e. " docker metrics --all" for the cli.

Agreed the stat calculation interval should be tunable; if I could suggest you default to 10s rather than 1s. 1s stat gathering on busy, dense nodes is a costly burden we should reserve for field-debug/troubleshooting. Even 10s might be too aggressive depending on the business.

The other thing is the patch itself decides on arbitrary names that closely resemble their cgroup counterparts. Why not get rid of all ambiguity and use the precise names used by the kernel ?

vishh · 2014-11-13T21:57:30Z

I wonder if it might be better to fork a separate process from the docker
daemon that handles stats acquisition and processing. This new process can
share code with docker daemon and can remain an internal component. This
might help scale docker daemon in the long run. We can continue to have a
single API and have the docker daemon be the source of metrics. WDYT
@crosbymichael?

On Thu, Nov 13, 2014 at 12:53 PM, Jeremy Eder notifications@github.com
wrote:

Hi @crosbymichael https://github.com/crosbymichael and @monsterzz
https://github.com/monsterzz this is along the lines of what we're
doing with nsinit at the moment. Nice to see it potentially in docker
itself. This patch currently does not work on RHEL-like systems, and I
think @monsterzz https://github.com/monsterzz eluded to the cgroup path
config needing some work, which is fine. I didn't look into it deeply but
hope that can be resolved.
"Metrics": {
    "CpuUsage": "Error: cgroup subsystem 'cpuacct' directory not found for container 'a7412e396560bcc886009472fdcd0433203598b0c4a11a9dfd4223e33c1c9173'",
    "MemoryLimit": "Error: cgroup subsystem 'memory' directory not found for container 'a7412e396560bcc886009472fdcd0433203598b0c4a11a9dfd4223e33c1c9173'",
    "MemoryMaxUsage": "Error: cgroup subsystem 'memory' directory not found for container 'a7412e396560bcc886009472fdcd0433203598b0c4a11a9dfd4223e33c1c9173'",
    "MemoryUsage": "Error: cgroup subsystem 'memory' directory not found for container 'a7412e396560bcc886009472fdcd0433203598b0c4a11a9dfd4223e33c1c9173'"
},
I would expect a monitoring system will want a global "watch" on all
containers, existing and new. Do you agree ? Basically docker would
continuously pump all it's stats to a websocket at a certain interval, and
the monitoring system would consume at it's own pace.

You could have i.e. " docker metrics --all" for the cli.

Agreed the stat calculation interval should be tunable; if I could suggest
you default to 10s rather than 1s. 1s stat gathering on busy, dense nodes
is a costly burden we should reserve for field-debug/troubleshooting. Even
10s might be too aggressive depending on the business.

The other thing is the patch itself decides on arbitrary names that
closely resemble their cgroup counterparts. Why not get rid of all
ambiguity and use the precise names used by the kernel ?

—
Reply to this email directly or view it on GitHub
#8886 (comment).

crosbymichael · 2014-11-14T02:19:12Z

@vishh do you not think the Go scheduler could handle this? As long as this is async I think go should be able to handle the load within one process. We can always break this out into another process later if we find that this is not the case and we are having performance issues.

tobegit3hub · 2014-12-17T10:01:40Z

We really need this. Can anyone help to review and merge it 😃

monsterzz · 2014-12-17T10:11:19Z

Unfortunately I have no time to implement push approach at this time. I
think I will have more free time to work on this after Christmas.
On Wed 17 Dec 2014 at 1:02 pm tobe notifications@github.com wrote:

We really need this. Can anyone help to review and merge it [image:
😃]

—
Reply to this email directly or view it on GitHub
#8886 (comment).

crosbymichael · 2015-01-14T21:02:28Z

@monsterzz no problem. I'll take care of finishing this feature.

Closing this in favor of #9984

Initial impl of docker metrics

ccbf200

Signed-off-by: Gleb M Borisov <borisov.gleb@gmail.com>

erikh reviewed Oct 31, 2014
View reviewed changes

SvenDowideit added the /project/doc label Nov 3, 2014

crosbymichael self-assigned this Nov 13, 2014

thaJeztah mentioned this pull request Nov 14, 2014

Proposal: Container and Engine telemetry #9130

Closed

tobegit3hub mentioned this pull request Dec 15, 2014

[Suggestion]: UI mustafaakin/docker-resource-reporter#1

Open

jessfraz added the UX label Jan 6, 2015

SvenDowideit mentioned this pull request Jan 13, 2015

docker stats live container resource metrics #9984

Merged

crosbymichael closed this Jan 14, 2015

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

docker metrics (read metrics from cgroups for specified container)#8886

docker metrics (read metrics from cgroups for specified container)#8886
monsterzz wants to merge 1 commit intomoby:masterfrom
monsterzz:8842-docker-metrics

monsterzz commented Oct 31, 2014

Uh oh!

erikh Oct 31, 2014

Uh oh!

monsterzz Oct 31, 2014

Uh oh!

erikh Oct 31, 2014

Uh oh!

SvenDowideit commented Nov 3, 2014

Uh oh!

jeremyeder commented Nov 13, 2014

Uh oh!

vishh commented Nov 13, 2014

Uh oh!

crosbymichael commented Nov 14, 2014

Uh oh!

tobegit3hub commented Dec 17, 2014

Uh oh!

monsterzz commented Dec 17, 2014

Uh oh!

crosbymichael commented Jan 14, 2015

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

10 participants

Conversation

monsterzz commented Oct 31, 2014

Uh oh!

erikh Oct 31, 2014

Choose a reason for hiding this comment

Uh oh!

monsterzz Oct 31, 2014

Choose a reason for hiding this comment

Uh oh!

erikh Oct 31, 2014

Choose a reason for hiding this comment

Uh oh!

SvenDowideit commented Nov 3, 2014

Uh oh!

jeremyeder commented Nov 13, 2014

Uh oh!

vishh commented Nov 13, 2014

Uh oh!

crosbymichael commented Nov 14, 2014

Uh oh!

tobegit3hub commented Dec 17, 2014

Uh oh!

monsterzz commented Dec 17, 2014

Uh oh!

crosbymichael commented Jan 14, 2015

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

10 participants