Skip to content

[clickhouse] Internal monitoring endpoint#7114

Merged
karencfv merged 18 commits into
oxidecomputer:mainfrom
karencfv:monitoring-endpoints
Nov 26, 2024
Merged

[clickhouse] Internal monitoring endpoint#7114
karencfv merged 18 commits into
oxidecomputer:mainfrom
karencfv:monitoring-endpoints

Conversation

@karencfv

@karencfv karencfv commented Nov 20, 2024

Copy link
Copy Markdown
Contributor

Overview

This commit adds an endpoint to retrieve timeseries from the system database. For the time being we will only add support for the metric_log and asynchronous_metric_log tables.

This endpoint is still a bit bare bones, but will be a good start to begin monitoring the ClickHouse servers.

Examples

Queries per second

$ curl "http://[::1]:8888/timeseries/metric_log/ProfileEvent_Query/avg" | jq
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  2883  100  2883    0     0   5912      0 --:--:-- --:--:-- --:--:--  5907
[
  {
    "time": "2024-11-20T04:09:00Z",
    "value": 0.0
  },
  {
    "time": "2024-11-20T04:10:00Z",
    "value": 0.0
  },
  {
    "time": "2024-11-20T04:11:00Z",
    "value": 0.06666666666666667
  }
]

Disk usage

$ curl "http://[::1]:8888/timeseries/asynchronous_metric_log/DiskUsed_default/avg?interval=120&time_range=3600&timestamp_format=unix_epoch" | jq
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  1786  100  1786    0     0   3728      0 --:--:-- --:--:-- --:--:--  3728
[
 {
    "time": "1732513320",
    "value": 120491427254.85716
  },
  {
    "time": "1732513440",
    "value": 120382774033.06668
  },
  {
    "time": "1732513560",
    "value": 120364752622.93332
  }
]

Related: #6953

@karencfv

Copy link
Copy Markdown
Contributor Author

Have been playing around with this. I think we can get some sort of Ok looking charts with this data:

Screenshot 2024-11-22 at 2 55 30 PM

@karencfv karencfv changed the title [clickhouse] WIP: Internal monitoring endpoints [clickhouse] Internal monitoring endpoint Nov 25, 2024
@karencfv karencfv marked this pull request as ready for review November 25, 2024 06:11
@karencfv karencfv requested a review from bnaecker November 25, 2024 06:12

@andrewjstone andrewjstone left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks great!

Comment thread clickhouse-admin/api/src/lib.rs Outdated
rqctx: RequestContext<Self::Context>,
) -> Result<HttpResponseUpdatedNoContent, HttpError>;

/// Retrieve time series from the system database. The value the average of all

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The second sentence appears to be typo'd. Not sure what you are trying to say.

Also, I'd move that sentence after a blank line, so that the doc comments have a short first line. Same for the method above.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lol, idk what was happening there 😄

Comment thread clickhouse-admin/src/clickhouse_cli.rs Outdated
let log = self.log.clone().unwrap();
let query = settings.query_avg();

info!(&log, "Querying system database"; "query" => &query);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Does this need to be an info log? What about a debug or trace log instead?

@karencfv karencfv enabled auto-merge (squash) November 25, 2024 23:07
@karencfv karencfv merged commit 1f83f07 into oxidecomputer:main Nov 26, 2024
@karencfv karencfv deleted the monitoring-endpoints branch November 26, 2024 05:06
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants