Skip to content

Docker monitoring demo with cadvisor and influx#23

Merged
mstemm merged 1 commit intomasterfrom
add-demo-cadvisor-#15
Jan 22, 2016
Merged

Docker monitoring demo with cadvisor and influx#23
mstemm merged 1 commit intomasterfrom
add-demo-cadvisor-#15

Conversation

@dmehra
Copy link
Collaborator

@dmehra dmehra commented Jan 4, 2016

connects to #24

Our ability to run this demo is blocked on 2 things:

  • InfluxDB v0.9 support · Issue #743 · google_cadvisor - should be unblocked, it's merged, although not yet in the official release (use canary image)
  • proper support for periodic reads to the backend in "live" mode, which was hacked up with '-lag' option in the pre-open version of this demo. The real influx adapter doesn't support '-lag' so the juttle here cannot run.

Don't merge these demo docs until the above are resolved, or else switch the demo to use a different backend (say Redis instead of Influx) or use something else (REST API?) instead of cadvisor to get the docker container stats.

@demmer @mstemm as we discussed... I will leave what's done here, and file an issue to pick up this work: #24

@dmehra
Copy link
Collaborator Author

dmehra commented Jan 8, 2016

With cadvisor support for InfluxDB 0.9 merged this should be unblocked.

I tried running google/cadvisor-canary image as part of this demo setup, however cadvisor kept logging

failed to write stats to influxDb - 404 page not found

@mstemm could you take over and get this demo into working shape? It also needs to use outrigger itself as a docker container, instead of relying on local install.

@demmer demmer force-pushed the add-demo-cadvisor-#15 branch 4 times, most recently from 4525e1f to 6eaa918 Compare January 20, 2016 01:18
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you take this XXX out?

@dmehra
Copy link
Collaborator Author

dmehra commented Jan 21, 2016

Ok this basically works, but the README needs much cleanup, which I'll do now and push up for review, and the specific cadvisor-canary build tag is no longer available, we'll have to take our chances with latest (the main release still doesn't include 0.9).

@dmehra dmehra changed the title Checkpointing docker monitoring demo with cadvisor and influx, but it's blocked Docker monitoring demo with cadvisor and influx Jan 22, 2016
@dmehra
Copy link
Collaborator Author

dmehra commented Jan 22, 2016

Cadvisor canary build gave us a 1-minute delay in the demo, see google/cadvisor#1074

The README should now be correct, @mstemm please review again.

The juttle program should be good enough for a public facing example, @rlgomes mind taking a look?

@mstemm
Copy link
Collaborator

mstemm commented Jan 22, 2016

The README changes look good. I'm also going to make the other changes I suggested (renaming juttle-influx-adapter to just influx in config files, not mapping the examples directory by default, removing the copy of the config file in the influx subdirectory).

@mstemm
Copy link
Collaborator

mstemm commented Jan 22, 2016

Also gonna rebase so I can verify compat w/ juttle 0.3.0.

@mstemm mstemm force-pushed the add-demo-cadvisor-#15 branch from 160b6b7 to 804d0e0 Compare January 22, 2016 16:59
@mstemm
Copy link
Collaborator

mstemm commented Jan 22, 2016

I'll squash this down but @demmer/@dmehra can you look at the final 2 commits I made?

@dmehra
Copy link
Collaborator Author

dmehra commented Jan 22, 2016

@mstemm your changes look good to me.

examples/cadvisor-influx/dc-cadvisor-influx.yml is a docker-compose
file that when combined with dc-outrigger.yml results in a properly
linked cadvisor, influxdb, and outrigger.

examples/cadvisor-influx/README.md includes notes on config, setup,
and a brief overview of the programs.

Add necessary config to examples/juttle-config.json to refer to the
services from the docker-compose setup.

Add entries to examples/index.juttle to refer to cadvisor-influx
directory.

Also ran into a problem with specifying more than one override .yml
file during docker-compose up. It turns out that the links directive
is not merged between multiple files, but overwritten for each
additional file.

This meant that a links: entry in dc-elastic.yml was being overwritten
with the links: entry in dc-cadvisor-influx.yml, resulting in
outrigger not being able to reach elasticsearch.

The fix for this is to use external_links, which *is* merged across
multiple .yml files. When using external_links, however, you have to
refer to explicit container names instead of services from the .yml
file, so add some explicit names for the elasticsearch and influxdb
containers, and use those names in the juttle-config files.

dc-elastic/dc-cadvisor-influx keep the links: entries, but that's only
used to make sure the containers are started up in the right
order. the external_links entry is what actually ensures the
containers can communicate with each other.
@mstemm mstemm force-pushed the add-demo-cadvisor-#15 branch from 9bc4f25 to f4cbe75 Compare January 22, 2016 22:15
mstemm pushed a commit that referenced this pull request Jan 22, 2016
Docker monitoring demo with cadvisor and influx
@mstemm mstemm merged commit 498a105 into master Jan 22, 2016
@mstemm mstemm deleted the add-demo-cadvisor-#15 branch January 22, 2016 22:31
@mstemm mstemm mentioned this pull request Jan 26, 2016
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants