monitoring: automatically deploy grafana notifiers (round 2)#11483
Conversation
Codecov Report
@@ Coverage Diff @@
## master #11483 +/- ##
=======================================
Coverage 46.85% 46.85%
=======================================
Files 1392 1392
Lines 78777 78777
Branches 6762 6762
=======================================
Hits 36913 36913
Misses 38359 38359
Partials 3505 3505
|
Co-authored-by: Stephen Gutekanst <stephen.gutekanst@gmail.com>
This is the cluster IP of Grafana and its external service port, configured here: https://github.com/sourcegraph/deploy-sourcegraph-dot-com/blob/release/base/grafana/grafana.Service.yaml#L13 What is stranger is the fact that it ended up in the URL at all, that URL should be: Not:
|
|
Confirmed working in:
|
|
I think it means that |
|
Feel free to deploy your test images on k8s.sgdev.org via a PR to https://github.com/sourcegraph/deploy-sourcegraph-dogfood-k8s (no need to wait for review, just make sure you undo it after you're finished testing) |
| # The (internal) http port to use | ||
| # In `sourcegraph/grafana`, 3370 is reverse-proxied to 3371 - if you change this, | ||
| # make sure you set GRAFANA_INTERNAL_PORT on your Grafana instance. | ||
| http_port = 3371 |
There was a problem hiding this comment.
The config file we use in single-container sourcegraph/server deployments like sourcegraph.sgdev.org was not updated to be 3371:
Maybe that was the cause?
There was a problem hiding this comment.
I think you mentioned in the previous PR that the single-container deployments use the grafana binary directly? also, I'm planning to keep single-server disabled for the time being (https://github.com/sourcegraph/sourcegraph/issues/11473)
There was a problem hiding this comment.
what i mean is to avoid having to add another configuration to every deployment, I've kept the container "interface" the same by putting a reverse proxy on 3370 and having grafana be served internally on 3371, so this configuration does not need to change for single-container at the moment
There was a problem hiding this comment.
Right, as long as you have this disabled for server it does not matter and this PR as-is LGTM - but if you are looking for the cause of the issue this may have been it.
Effectively, Grafana was serving on 3370 not 3371 as you intended - I have no idea what the exact behavior would’ve been for either Grafana itself or Grafana-wrapper in this case (I.e Grafana 3370 was already bound)
|
alright... I'm pretty sure this won't behave badly anymore - https://github.com/sourcegraph/sourcegraph/pull/11483#issuecomment-643844736 |
The changes in #11483 made it difficult to start Grafana within a Linux development environment, and #11486 obscured the error in the goreman output. This restores the previous behaviour around the UID the container will run as, and ensures that logs will be dumped if the container exits unexpectedly.
* dev: fix Grafana startup on Linux The changes in #11483 made it difficult to start Grafana within a Linux development environment, and #11486 obscured the error in the goreman output. This restores the previous behaviour around the UID the container will run as, and ensures that logs will be dumped if the container exits unexpectedly. * Always overwrite grafana.log
Why https://github.com/sourcegraph/sourcegraph/pull/11479 was reverted: https://github.com/sourcegraph/sourcegraph/pull/11479
This PR:
sourcegraph/serverGRAFANA_PORTmight be incorrectly set, so rename it toGRAFANA_INTERNAL_PORT(more descriptive anyway)relevant changes here, this branch starts from a cherry-picked #11427 (already reviewed)
testing in: