Skip to content

all: Add Prometheus-style metrics to expose some internal performance counters (fixes #5175) #9003

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
calmh merged 26 commits into from
Aug 4, 2023
Merged

all: Add Prometheus-style metrics to expose some internal performance counters (fixes #5175) #9003

calmh merged 26 commits into from
Aug 4, 2023

Conversation

@calmh
Copy link
Member

@calmh calmh commented Jul 25, 2023 •
edited
Loading

This adds a /metrics endpoint on the REST API which exposes a number of performance counters. The purpose is to allow better monitoring of a Syncthing instance, using standard tooling such as Prometheus and Grafana. The metrics endpoint lives behind the same authentication (or lack thereof) configured on the rest of the API. For the avoidance of doubt, this data is just made available for consumption by the user or administrator -- it is not sent as part of the usage report or otherwise communicated to outside observers.

I have added a basic set of metrics that I'm interested in; this could and should be added to in the future as needs arise. Currently it can answer questions such as how much data is transferred to/from each device, with which compression ratio; how much data and how many files are in each folder; how much time is spent pulling or scanning, and how much data is pulled (from where) or scanned; how many filesystem operations do we performance and what's the average time for each such operation; how much data is read or written to the filesystem; how many events of each type are created. Additionally, some Go internals (memory, GC, goroutines, etc) are exposed by default by the Prometheus library.

There are many more things that could be tracked, including things mentioned in the original ticket but not implemented here. I don't want to make an exhaustive attempt, rather keep this one contained and reasonable and let actual needs dictate further additions.

Sample dashboards, somewhat boring from my rather idle home setup:

Screenshot 2023-07-25 at 21 45 06 Screenshot 2023-07-25 at 21 45 18

Here is an example of what the current data can look like:

% curl -s http://localhost:8082/metrics|grep syncthing_
# HELP syncthing_events_total Total number of created/forwarded/dropped events
# TYPE syncthing_events_total counter
syncthing_events_total{event="ClusterConfigReceived",state="created"} 1
syncthing_events_total{event="ClusterConfigReceived",state="delivered"} 2
syncthing_events_total{event="DeviceConnected",state="created"} 1
syncthing_events_total{event="DeviceConnected",state="delivered"} 2
syncthing_events_total{event="DeviceDiscovered",state="created"} 2
syncthing_events_total{event="DeviceDiscovered",state="delivered"} 2
syncthing_events_total{event="FolderCompletion",state="created"} 3
syncthing_events_total{event="FolderCompletion",state="delivered"} 3
syncthing_events_total{event="FolderSummary",state="created"} 2
syncthing_events_total{event="FolderSummary",state="delivered"} 2
syncthing_events_total{event="ItemFinished",state="created"} 17
syncthing_events_total{event="ItemFinished",state="delivered"} 17
syncthing_events_total{event="ItemStarted",state="created"} 17
syncthing_events_total{event="ItemStarted",state="delivered"} 17
syncthing_events_total{event="ListenAddressesChanged",state="created"} 2
syncthing_events_total{event="ListenAddressesChanged",state="delivered"} 2
syncthing_events_total{event="LocalChangeDetected",state="created"} 10
syncthing_events_total{event="LocalChangeDetected",state="delivered"} 10
syncthing_events_total{event="LocalIndexUpdated",state="created"} 2
syncthing_events_total{event="LocalIndexUpdated",state="delivered"} 8
syncthing_events_total{event="RemoteChangeDetected",state="created"} 17
syncthing_events_total{event="RemoteChangeDetected",state="delivered"} 17
syncthing_events_total{event="RemoteIndexUpdated",state="created"} 2
syncthing_events_total{event="RemoteIndexUpdated",state="delivered"} 4
syncthing_events_total{event="Starting",state="created"} 1
syncthing_events_total{event="Starting",state="delivered"} 1
syncthing_events_total{event="StartupComplete",state="created"} 1
syncthing_events_total{event="StartupComplete",state="delivered"} 1
syncthing_events_total{event="StateChanged",state="created"} 43
syncthing_events_total{event="StateChanged",state="delivered"} 86
# HELP syncthing_fs_operation_bytes_total Total number of filesystem bytes transferred, per filesystem root and operation
# TYPE syncthing_fs_operation_bytes_total counter
syncthing_fs_operation_bytes_total{operation="read",root="/Users/jb/dev/syncthing/syncthing/test/s2"} 1.4115015e+08
syncthing_fs_operation_bytes_total{operation="readat",root="/Users/jb/dev/syncthing/syncthing/test/s2"} 6.7778346e+07
syncthing_fs_operation_bytes_total{operation="write",root="/Users/jb/dev/syncthing/syncthing/test/h2"} 825
syncthing_fs_operation_bytes_total{operation="writeat",root="/Users/jb/dev/syncthing/syncthing/test/s2"} 1.71491858e+08
# HELP syncthing_fs_operation_seconds_total Total time spent in filesystem operations, per filesystem root and operation
# TYPE syncthing_fs_operation_seconds_total counter
syncthing_fs_operation_seconds_total{operation="chmod",root="/Users/jb/dev/syncthing/syncthing/test/h2"} 6.8958e-05
syncthing_fs_operation_seconds_total{operation="chmod",root="/Users/jb/dev/syncthing/syncthing/test/s2"} 0.00025108300000000004
syncthing_fs_operation_seconds_total{operation="chtimes",root="/Users/jb/dev/syncthing/syncthing/test/s2"} 0.0012470820000000003
syncthing_fs_operation_seconds_total{operation="close",root="/Users/jb/dev/syncthing/syncthing/test/h2"} 5.5333e-05
syncthing_fs_operation_seconds_total{operation="close",root="/Users/jb/dev/syncthing/syncthing/test/s2"} 0.000968793000000003
syncthing_fs_operation_seconds_total{operation="dirnames",root="/Users/jb/dev/syncthing/syncthing/test/s12-2"} 0.00039025
syncthing_fs_operation_seconds_total{operation="dirnames",root="/Users/jb/dev/syncthing/syncthing/test/s2"} 0.001180666
syncthing_fs_operation_seconds_total{operation="dirnames",root="/Users/jb/dev/syncthing/syncthing/test/s23-2"} 0.00030774900000000007
syncthing_fs_operation_seconds_total{operation="glob",root="/Users/jb/dev/syncthing/syncthing/test/h2"} 0.00017899999999999993
syncthing_fs_operation_seconds_total{operation="hide",root="/Users/jb/dev/syncthing/syncthing/test/s12-2"} 1.5126e-05
syncthing_fs_operation_seconds_total{operation="hide",root="/Users/jb/dev/syncthing/syncthing/test/s2"} 0.001308501
syncthing_fs_operation_seconds_total{operation="hide",root="/Users/jb/dev/syncthing/syncthing/test/s23-2"} 1.25e-05
syncthing_fs_operation_seconds_total{operation="lstat",root="/Users/jb/dev/syncthing/syncthing/test/h2"} 8.9333e-05
syncthing_fs_operation_seconds_total{operation="lstat",root="/Users/jb/dev/syncthing/syncthing/test/s12-2"} 0.000684165
syncthing_fs_operation_seconds_total{operation="lstat",root="/Users/jb/dev/syncthing/syncthing/test/s2"} 0.011043747999999994
syncthing_fs_operation_seconds_total{operation="lstat",root="/Users/jb/dev/syncthing/syncthing/test/s23-2"} 0.000849624
syncthing_fs_operation_seconds_total{operation="mkdirall",root="/Users/jb/dev/syncthing/syncthing/test/h2"} 5.916e-06
syncthing_fs_operation_seconds_total{operation="name",root="/Users/jb/dev/syncthing/syncthing/test/h2"} 8.001e-06
syncthing_fs_operation_seconds_total{operation="open",root="/Users/jb/dev/syncthing/syncthing/test/h2"} 5.9209e-05
syncthing_fs_operation_seconds_total{operation="open",root="/Users/jb/dev/syncthing/syncthing/test/s12-2"} 0.000110208
syncthing_fs_operation_seconds_total{operation="open",root="/Users/jb/dev/syncthing/syncthing/test/s2"} 0.007641950999999998
syncthing_fs_operation_seconds_total{operation="open",root="/Users/jb/dev/syncthing/syncthing/test/s23-2"} 9.3707e-05
syncthing_fs_operation_seconds_total{operation="openfile",root="/Users/jb/dev/syncthing/syncthing/test/h2"} 0.000332874
syncthing_fs_operation_seconds_total{operation="openfile",root="/Users/jb/dev/syncthing/syncthing/test/s2"} 0.0016239590000000003
syncthing_fs_operation_seconds_total{operation="options",root="/Users/jb/dev/syncthing/syncthing/test/h2"} 1.9128000000000004e-05
syncthing_fs_operation_seconds_total{operation="options",root="/Users/jb/dev/syncthing/syncthing/test/h2/index-v0.14.0.db"} 4.2000000000000004e-05
syncthing_fs_operation_seconds_total{operation="options",root="/Users/jb/dev/syncthing/syncthing/test/s12-2"} 0.000124501
syncthing_fs_operation_seconds_total{operation="options",root="/Users/jb/dev/syncthing/syncthing/test/s2"} 0.000125291
syncthing_fs_operation_seconds_total{operation="options",root="/Users/jb/dev/syncthing/syncthing/test/s23-2"} 8.733300000000001e-05
syncthing_fs_operation_seconds_total{operation="read",root="/Users/jb/dev/syncthing/syncthing/test/s2"} 0.06606509799999799
syncthing_fs_operation_seconds_total{operation="readat",root="/Users/jb/dev/syncthing/syncthing/test/s2"} 0.06407341799999998
syncthing_fs_operation_seconds_total{operation="remove",root="/Users/jb/dev/syncthing/syncthing/test/h2"} 6.5708e-05
syncthing_fs_operation_seconds_total{operation="remove",root="/Users/jb/dev/syncthing/syncthing/test/s2"} 0.021861794000000007
syncthing_fs_operation_seconds_total{operation="rename",root="/Users/jb/dev/syncthing/syncthing/test/h2"} 0.002632375
syncthing_fs_operation_seconds_total{operation="rename",root="/Users/jb/dev/syncthing/syncthing/test/s2"} 0.002672331
syncthing_fs_operation_seconds_total{operation="seek",root="/Users/jb/dev/syncthing/syncthing/test/s2"} 0.0003906189999999999
syncthing_fs_operation_seconds_total{operation="stat",root="/Users/jb/dev/syncthing/syncthing/test/h2"} 8.542e-06
syncthing_fs_operation_seconds_total{operation="stat",root="/Users/jb/dev/syncthing/syncthing/test/s12-2"} 0.001152291
syncthing_fs_operation_seconds_total{operation="stat",root="/Users/jb/dev/syncthing/syncthing/test/s2"} 0.0035226219999999996
syncthing_fs_operation_seconds_total{operation="stat",root="/Users/jb/dev/syncthing/syncthing/test/s23-2"} 0.000907167
syncthing_fs_operation_seconds_total{operation="sync",root="/Users/jb/dev/syncthing/syncthing/test/h2"} 0.024849916
syncthing_fs_operation_seconds_total{operation="sync",root="/Users/jb/dev/syncthing/syncthing/test/s2"} 0.19004337299999996
syncthing_fs_operation_seconds_total{operation="truncate",root="/Users/jb/dev/syncthing/syncthing/test/s2"} 0.000249373
syncthing_fs_operation_seconds_total{operation="type",root="/Users/jb/dev/syncthing/syncthing/test/s12-2"} 6.25e-07
syncthing_fs_operation_seconds_total{operation="type",root="/Users/jb/dev/syncthing/syncthing/test/s2"} 1.1791000000000003e-05
syncthing_fs_operation_seconds_total{operation="type",root="/Users/jb/dev/syncthing/syncthing/test/s23-2"} 5.42e-07
syncthing_fs_operation_seconds_total{operation="unhide",root="/Users/jb/dev/syncthing/syncthing/test/s2"} 0.000282835
syncthing_fs_operation_seconds_total{operation="uri",root="/Users/jb/dev/syncthing/syncthing/test/s12-2"} 5e-07
syncthing_fs_operation_seconds_total{operation="uri",root="/Users/jb/dev/syncthing/syncthing/test/s2"} 9.079e-06
syncthing_fs_operation_seconds_total{operation="uri",root="/Users/jb/dev/syncthing/syncthing/test/s23-2"} 3.875e-06
syncthing_fs_operation_seconds_total{operation="usage",root="/Users/jb/dev/syncthing/syncthing/test/h2/index-v0.14.0.db"} 0.000473626
syncthing_fs_operation_seconds_total{operation="usage",root="/Users/jb/dev/syncthing/syncthing/test/s2"} 0.00014337299999999997
syncthing_fs_operation_seconds_total{operation="write",root="/Users/jb/dev/syncthing/syncthing/test/h2"} 0.000307336
syncthing_fs_operation_seconds_total{operation="writeat",root="/Users/jb/dev/syncthing/syncthing/test/s2"} 0.060683214999999985
# HELP syncthing_fs_operations_total Total number of filesystem operations, per filesystem root and operation
# TYPE syncthing_fs_operations_total counter
syncthing_fs_operations_total{operation="chmod",root="/Users/jb/dev/syncthing/syncthing/test/h2"} 2
syncthing_fs_operations_total{operation="chmod",root="/Users/jb/dev/syncthing/syncthing/test/s2"} 17
syncthing_fs_operations_total{operation="chtimes",root="/Users/jb/dev/syncthing/syncthing/test/s2"} 17
syncthing_fs_operations_total{operation="close",root="/Users/jb/dev/syncthing/syncthing/test/h2"} 4
syncthing_fs_operations_total{operation="close",root="/Users/jb/dev/syncthing/syncthing/test/s2"} 566
syncthing_fs_operations_total{operation="dirnames",root="/Users/jb/dev/syncthing/syncthing/test/s12-2"} 5
syncthing_fs_operations_total{operation="dirnames",root="/Users/jb/dev/syncthing/syncthing/test/s2"} 4
syncthing_fs_operations_total{operation="dirnames",root="/Users/jb/dev/syncthing/syncthing/test/s23-2"} 4
syncthing_fs_operations_total{operation="glob",root="/Users/jb/dev/syncthing/syncthing/test/h2"} 11
syncthing_fs_operations_total{operation="hide",root="/Users/jb/dev/syncthing/syncthing/test/s12-2"} 3
syncthing_fs_operations_total{operation="hide",root="/Users/jb/dev/syncthing/syncthing/test/s2"} 20
syncthing_fs_operations_total{operation="hide",root="/Users/jb/dev/syncthing/syncthing/test/s23-2"} 3
syncthing_fs_operations_total{operation="lstat",root="/Users/jb/dev/syncthing/syncthing/test/h2"} 7
syncthing_fs_operations_total{operation="lstat",root="/Users/jb/dev/syncthing/syncthing/test/s12-2"} 10
syncthing_fs_operations_total{operation="lstat",root="/Users/jb/dev/syncthing/syncthing/test/s2"} 557
syncthing_fs_operations_total{operation="lstat",root="/Users/jb/dev/syncthing/syncthing/test/s23-2"} 8
syncthing_fs_operations_total{operation="mkdirall",root="/Users/jb/dev/syncthing/syncthing/test/h2"} 1
syncthing_fs_operations_total{operation="name",root="/Users/jb/dev/syncthing/syncthing/test/h2"} 6
syncthing_fs_operations_total{operation="open",root="/Users/jb/dev/syncthing/syncthing/test/h2"} 2
syncthing_fs_operations_total{operation="open",root="/Users/jb/dev/syncthing/syncthing/test/s12-2"} 6
syncthing_fs_operations_total{operation="open",root="/Users/jb/dev/syncthing/syncthing/test/s2"} 572
syncthing_fs_operations_total{operation="open",root="/Users/jb/dev/syncthing/syncthing/test/s23-2"} 5
syncthing_fs_operations_total{operation="openfile",root="/Users/jb/dev/syncthing/syncthing/test/h2"} 2
syncthing_fs_operations_total{operation="openfile",root="/Users/jb/dev/syncthing/syncthing/test/s2"} 17
syncthing_fs_operations_total{operation="options",root="/Users/jb/dev/syncthing/syncthing/test/h2"} 14
syncthing_fs_operations_total{operation="options",root="/Users/jb/dev/syncthing/syncthing/test/h2/index-v0.14.0.db"} 16
syncthing_fs_operations_total{operation="options",root="/Users/jb/dev/syncthing/syncthing/test/s12-2"} 13
syncthing_fs_operations_total{operation="options",root="/Users/jb/dev/syncthing/syncthing/test/s2"} 40
syncthing_fs_operations_total{operation="options",root="/Users/jb/dev/syncthing/syncthing/test/s23-2"} 12
syncthing_fs_operations_total{operation="read",root="/Users/jb/dev/syncthing/syncthing/test/s2"} 24468
syncthing_fs_operations_total{operation="readat",root="/Users/jb/dev/syncthing/syncthing/test/s2"} 529
syncthing_fs_operations_total{operation="remove",root="/Users/jb/dev/syncthing/syncthing/test/h2"} 2
syncthing_fs_operations_total{operation="remove",root="/Users/jb/dev/syncthing/syncthing/test/s2"} 34
syncthing_fs_operations_total{operation="rename",root="/Users/jb/dev/syncthing/syncthing/test/h2"} 2
syncthing_fs_operations_total{operation="rename",root="/Users/jb/dev/syncthing/syncthing/test/s2"} 17
syncthing_fs_operations_total{operation="seek",root="/Users/jb/dev/syncthing/syncthing/test/s2"} 314
syncthing_fs_operations_total{operation="stat",root="/Users/jb/dev/syncthing/syncthing/test/h2"} 1
syncthing_fs_operations_total{operation="stat",root="/Users/jb/dev/syncthing/syncthing/test/s12-2"} 13
syncthing_fs_operations_total{operation="stat",root="/Users/jb/dev/syncthing/syncthing/test/s2"} 102
syncthing_fs_operations_total{operation="stat",root="/Users/jb/dev/syncthing/syncthing/test/s23-2"} 11
syncthing_fs_operations_total{operation="sync",root="/Users/jb/dev/syncthing/syncthing/test/h2"} 4
syncthing_fs_operations_total{operation="sync",root="/Users/jb/dev/syncthing/syncthing/test/s2"} 18
syncthing_fs_operations_total{operation="truncate",root="/Users/jb/dev/syncthing/syncthing/test/s2"} 17
syncthing_fs_operations_total{operation="type",root="/Users/jb/dev/syncthing/syncthing/test/s12-2"} 1
syncthing_fs_operations_total{operation="type",root="/Users/jb/dev/syncthing/syncthing/test/s2"} 35
syncthing_fs_operations_total{operation="type",root="/Users/jb/dev/syncthing/syncthing/test/s23-2"} 1
syncthing_fs_operations_total{operation="unhide",root="/Users/jb/dev/syncthing/syncthing/test/s2"} 17
syncthing_fs_operations_total{operation="uri",root="/Users/jb/dev/syncthing/syncthing/test/s12-2"} 1
syncthing_fs_operations_total{operation="uri",root="/Users/jb/dev/syncthing/syncthing/test/s2"} 35
syncthing_fs_operations_total{operation="uri",root="/Users/jb/dev/syncthing/syncthing/test/s23-2"} 1
syncthing_fs_operations_total{operation="usage",root="/Users/jb/dev/syncthing/syncthing/test/h2/index-v0.14.0.db"} 16
syncthing_fs_operations_total{operation="usage",root="/Users/jb/dev/syncthing/syncthing/test/s2"} 17
syncthing_fs_operations_total{operation="write",root="/Users/jb/dev/syncthing/syncthing/test/h2"} 25
syncthing_fs_operations_total{operation="writeat",root="/Users/jb/dev/syncthing/syncthing/test/s2"} 1318
# HELP syncthing_model_folder_processed_bytes_total Total amount of data processed during folder syncing, per folder ID and data source (network/local_origin/local_other/local_shifted/skipped)
# TYPE syncthing_model_folder_processed_bytes_total counter
syncthing_model_folder_processed_bytes_total{folder="default",source="local_origin"} 1.01332182e+08
syncthing_model_folder_processed_bytes_total{folder="default",source="local_other"} 0
syncthing_model_folder_processed_bytes_total{folder="default",source="local_shifted"} 7.340032e+06
syncthing_model_folder_processed_bytes_total{folder="default",source="network"} 6.2819644e+07
syncthing_model_folder_processed_bytes_total{folder="default",source="skipped"} 0
syncthing_model_folder_processed_bytes_total{folder="s23",source="local_origin"} 0
syncthing_model_folder_processed_bytes_total{folder="s23",source="local_other"} 0
syncthing_model_folder_processed_bytes_total{folder="s23",source="local_shifted"} 0
syncthing_model_folder_processed_bytes_total{folder="s23",source="network"} 0
syncthing_model_folder_processed_bytes_total{folder="s23",source="skipped"} 0
# HELP syncthing_model_folder_pull_seconds_total Total time spent in folder pull iterations, per folder ID
# TYPE syncthing_model_folder_pull_seconds_total counter
syncthing_model_folder_pull_seconds_total{folder="default"} 1.8969185
syncthing_model_folder_pull_seconds_total{folder="s23"} 0
# HELP syncthing_model_folder_pulls_total Total number of folder pull iterations, per folder ID
# TYPE syncthing_model_folder_pulls_total counter
syncthing_model_folder_pulls_total{folder="default"} 1
syncthing_model_folder_pulls_total{folder="s23"} 0
# HELP syncthing_model_folder_scan_seconds_total Total time spent in folder scan iterations, per folder ID
# TYPE syncthing_model_folder_scan_seconds_total counter
syncthing_model_folder_scan_seconds_total{folder="default"} 0.022962166
syncthing_model_folder_scan_seconds_total{folder="s23"} 0.003112709
# HELP syncthing_model_folder_scans_total Total number of folder scan iterations, per folder ID
# TYPE syncthing_model_folder_scans_total counter
syncthing_model_folder_scans_total{folder="default"} 4
syncthing_model_folder_scans_total{folder="s23"} 4
# HELP syncthing_model_folder_state Current folder state
# TYPE syncthing_model_folder_state gauge
syncthing_model_folder_state{folder="default"} 0
syncthing_model_folder_state{folder="s23"} 0
# HELP syncthing_model_folder_summary Current folder summary data (counts for global/local/need files/directories/symlinks/deleted/bytes)
# TYPE syncthing_model_folder_summary gauge
syncthing_model_folder_summary{folder="default",scope="global",type="bytes"} 1.71840952e+08
syncthing_model_folder_summary{folder="default",scope="global",type="deleted"} 0
syncthing_model_folder_summary{folder="default",scope="global",type="directories"} 0
syncthing_model_folder_summary{folder="default",scope="global",type="files"} 66
syncthing_model_folder_summary{folder="default",scope="global",type="symlinks"} 0
syncthing_model_folder_summary{folder="default",scope="local",type="bytes"} 1.71840952e+08
syncthing_model_folder_summary{folder="default",scope="local",type="deleted"} 0
syncthing_model_folder_summary{folder="default",scope="local",type="directories"} 0
syncthing_model_folder_summary{folder="default",scope="local",type="files"} 66
syncthing_model_folder_summary{folder="default",scope="local",type="symlinks"} 0
syncthing_model_folder_summary{folder="default",scope="need",type="bytes"} 0
syncthing_model_folder_summary{folder="default",scope="need",type="deleted"} 0
syncthing_model_folder_summary{folder="default",scope="need",type="directories"} 0
syncthing_model_folder_summary{folder="default",scope="need",type="files"} 0
syncthing_model_folder_summary{folder="default",scope="need",type="symlinks"} 0
# HELP syncthing_protocol_recv_bytes_total Total amount of data received, per device
# TYPE syncthing_protocol_recv_bytes_total counter
syncthing_protocol_recv_bytes_total{device="I6KAH76-66SLLLB-5PFXSOA-UFJCDZC-YAOMLEK-CP2GB32-BV5RQST-3PSROAU"} 6.2888387e+07
# HELP syncthing_protocol_recv_decompressed_bytes_total Total amount of data received, after decompression, per device
# TYPE syncthing_protocol_recv_decompressed_bytes_total counter
syncthing_protocol_recv_decompressed_bytes_total{device="I6KAH76-66SLLLB-5PFXSOA-UFJCDZC-YAOMLEK-CP2GB32-BV5RQST-3PSROAU"} 6.2899593e+07
# HELP syncthing_protocol_recv_messages_total Total number of messages received, per device
# TYPE syncthing_protocol_recv_messages_total counter
syncthing_protocol_recv_messages_total{device="I6KAH76-66SLLLB-5PFXSOA-UFJCDZC-YAOMLEK-CP2GB32-BV5RQST-3PSROAU"} 499
# HELP syncthing_protocol_sent_bytes_total Total amount of data sent, per device
# TYPE syncthing_protocol_sent_bytes_total counter
syncthing_protocol_sent_bytes_total{device="I6KAH76-66SLLLB-5PFXSOA-UFJCDZC-YAOMLEK-CP2GB32-BV5RQST-3PSROAU"} 232942
# HELP syncthing_protocol_sent_messages_total Total number of messages sent, per device
# TYPE syncthing_protocol_sent_messages_total counter
syncthing_protocol_sent_messages_total{device="I6KAH76-66SLLLB-5PFXSOA-UFJCDZC-YAOMLEK-CP2GB32-BV5RQST-3PSROAU"} 499
# HELP syncthing_protocol_sent_uncompressed_bytes_total Total amount of data sent, before compression, per device
# TYPE syncthing_protocol_sent_uncompressed_bytes_total counter
syncthing_protocol_sent_uncompressed_bytes_total{device="I6KAH76-66SLLLB-5PFXSOA-UFJCDZC-YAOMLEK-CP2GB32-BV5RQST-3PSROAU"} 244095
# HELP syncthing_scanner_hashed_bytes_total Total amount of data hashed, per folder
# TYPE syncthing_scanner_hashed_bytes_total counter
syncthing_scanner_hashed_bytes_total{folder="default"} 131668
syncthing_scanner_hashed_bytes_total{folder="s23"} 0
# HELP syncthing_scanner_scanned_items_total Total number of items (files/directories) inspected, per folder
# TYPE syncthing_scanner_scanned_items_total counter
syncthing_scanner_scanned_items_total{folder="default"} 252
syncthing_scanner_scanned_items_total{folder="s23"} 8

calmh added 20 commits June 29, 2023 14:15
@calmh
all: Add Prometheus metrics endpoint
04b121b
@calmh
bearer
dca496c
@calmh
scanner
2a77eb5
@calmh
wip
7009cf1
@calmh
wip
deb2f42
@calmh
wip
7fc4354
@calmh
wip
9a9b751
@calmh
wip
102c3c3
@calmh
Merge branch 'main' into prometheus
0c7a479 
* main: (27 commits)
  gui, man, authors: Update docs, translations, and contributors
  cmd/syncthing: Mention STVERSIONEXTRA in --help output (ref syncthing#8980)
  gui, lib/api: Add possibility to feed through extra version information (syncthing#8980)
  cmd/ursrv: Update map tile URL
  cmd/strelaypoolsrv: Update map tile URL (syncthing#8985)
  cmd/stcrashreceiver: Correct parsing of current version string
  build: Update dependencies
  cmd/ursrv: Remove old, unused user movement code
  cmd/ursrv: Merge ursrv and uraggregate as subcommands
  cmd/ursrv: Remove useless static TLS cert handling
  cmd/ursrv: Embed static assets
  cmd/ursrv: Refactor to use CLI options, fewer global vars
  gui: Show xattr filter editor when send xattrs checked (fixes syncthing#8958) (syncthing#8959)
  gui, man, authors: Update docs, translations, and contributors
  build: Build Docker image for plain 32 bit arm (fixes syncthing#8973)
  lib/fs: Clarify errors for Windows filenames (fixes syncthing#8968) (syncthing#8969)
  gui: Avoid spurious comma in shared-with device list (fixes syncthing#8967) (syncthing#8970)
  cmd/ursrv: Summarize tiny fraction items into Other
  cmd/ursrv: Update distributions list
  build: Only push releases to cloud storage, and also use `latest`
  ...
@calmh
wip
236bee8
@calmh
wip
89bb36d
@calmh
wip
76f185a
@calmh
wip
4d2ffae
@calmh
wip
2a860f0
@calmh
wip
fe35d5a
@calmh
Merge branch 'main' into prometheus
d2cfb53 
* main:
  gui: Remove Twitter link from footer (syncthing#9000)
  gui, man, authors: Update docs, translations, and contributors
  lib/ur: Don't report uptime if start time is in the past (fixes syncthing#7698) (syncthing#8996)
  lib/model: use WARN for "Unexpected folder" messages (syncthing#8998)
  lib/osutil: Skip setLowPriority in Windows if already lower (fixes syncthing#6597) (syncthing#8993)
  all: replace empty slice literal with `var` (syncthing#8990)
  all: fix unused method receiver (syncthing#8988)
  all: unused parameter should be replaced by underscore (syncthing#8989)
@calmh
wip
ca41a49
@calmh
wip
745e35f
@calmh
wip
6848640
@calmh
wip
8f72f4d
@calmh calmh changed the title all: Add Prometheus-style metrics to expose some internal performance counters all: Add Prometheus-style metrics to expose some internal performance counters (fixes #5175) Jul 25, 2023
@acolomb
Copy link
Member

acolomb commented Jul 25, 2023

Can't say much about the utility or implementation here. Just a reminder to please update the API docs as well if this gets added.

@ttonin33
Copy link

ttonin33 commented Jul 30, 2023

A status info would also great for a Main Grafana Page of a Server where your can easy see if all thinks works.
Like: scanning, waiting, syncthing, ready, and error.

calmh added 4 commits August 3, 2023 08:58
@calmh
wip
243e065
@calmh
Merge branch 'main' into prometheus
9c2f1ef 
* main:
  build: Increase Go version to 1.20.7
  lib/config: Allow sharing already encrypted folder with untrusted devices (fixes syncthing#8965) (syncthing#9012)
  gui: Use case-insensive and backslash-agnostic versions filter (fixes syncthing#7973) (syncthing#8995)
  gui, man, authors: Update docs, translations, and contributors
  build: Run govulncheck (fixes syncthing#8983)
  build: Run build & tests on main branch nightly
  build: Send test logs to Grafana Loki for statistics
  all: Refactor the protocol/model interface a bit (ref syncthing#8981) (syncthing#9007)
  lib/connections: Fix building with `-tags noquic` (syncthing#9009)
  gui: Fix tooltips on buttons inside button groups (ref syncthing#7984) (syncthing#9008)
  cmd/strelaysrv: Handle accept error with debug set (fixes syncthing#9001) (syncthing#9004)
  lib/api: Fix data race in TestCSRFRequired (syncthing#9006)
  gui: Show full error for failed items (syncthing#9005)
  lib/api: Allow `Bearer` authentication style with API key (syncthing#9002)
@calmh
wip
9cf5d41
@calmh
wip
7fe7a2e
@calmh
Copy link
Member Author

calmh commented Aug 3, 2023 •
edited
Loading

@acolomb I added a script to find all metrics definitions in the source code and generate a markdown list. The intention is to use this in the docs repo in some script to make sure we have an up to date list of all metrics as they get added or removed over time. Sample output:

Package events

syncthing_events_total (counter vector)

Total number of created/forwarded/dropped events.

Package fs

syncthing_fs_operation_bytes_total (counter vector)

Total number of filesystem bytes transferred, per filesystem root and
operation.

syncthing_fs_operation_seconds_total (counter vector)

Total time spent in filesystem operations, per filesystem root and
operation.

syncthing_fs_operations_total (counter vector)

Total number of filesystem operations, per filesystem root and
operation.

Package model

syncthing_model_folder_processed_bytes_total (counter vector)

Total amount of data processed during folder syncing, per folder ID and
data source (network/local_origin/local_other/local_shifted/skipped).

syncthing_model_folder_pull_seconds_total (counter vector)

Total time spent in folder pull iterations, per folder ID.

syncthing_model_folder_pulls_total (counter vector)

Total number of folder pull iterations, per folder ID.

syncthing_model_folder_scan_seconds_total (counter vector)

Total time spent in folder scan iterations, per folder ID.

syncthing_model_folder_scans_total (counter vector)

Total number of folder scan iterations, per folder ID.

syncthing_model_folder_state (gauge vector)

Current folder state.

syncthing_model_folder_summary (gauge vector)

Current folder summary data (counts for global/local/need
files/directories/symlinks/deleted/bytes).

Package protocol

syncthing_protocol_recv_bytes_total (counter vector)

Total amount of data received, per device.

syncthing_protocol_recv_decompressed_bytes_total (counter vector)

Total amount of data received, after decompression, per device.

syncthing_protocol_recv_messages_total (counter vector)

Total number of messages received, per device.

syncthing_protocol_sent_bytes_total (counter vector)

Total amount of data sent, per device.

syncthing_protocol_sent_messages_total (counter vector)

Total number of messages sent, per device.

syncthing_protocol_sent_uncompressed_bytes_total (counter vector)

Total amount of data sent, before compression, per device.

Package scanner

syncthing_scanner_hashed_bytes_total (counter vector)

Total amount of data hashed, per folder.

syncthing_scanner_scanned_items_total (counter vector)

Total number of items (files/directories) inspected, per folder.

er-pa
er-pa approved these changes Aug 4, 2023
View reviewed changes
Copy link
Member

@er-pa er-pa left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I have no objections, looks good to me code-wise

@calmh calmh merged commit b9c08d3 into syncthing:main Aug 4, 2023
@calmh calmh deleted the prometheus branch August 4, 2023 17:57
@acolomb
Copy link
Member

acolomb commented Aug 6, 2023

@calmh Do you have some integration for your script ready for the docs repo yet? Just nagging so we don't let the API docs get out of sync again so quickly.

calmh added a commit to calmh/syncthing that referenced this pull request Aug 9, 2023
@calmh
Merge branch 'main' into multicon
99086c4 
* main:
  gui, man, authors: Update docs, translations, and contributors
  all: Add Prometheus-style metrics to expose some internal performance counters (fixes syncthing#5175) (syncthing#9003)
  build: Increase Go version to 1.20.7
  lib/config: Allow sharing already encrypted folder with untrusted devices (fixes syncthing#8965) (syncthing#9012)
  gui: Use case-insensive and backslash-agnostic versions filter (fixes syncthing#7973) (syncthing#8995)
  gui, man, authors: Update docs, translations, and contributors
  build: Run govulncheck (fixes syncthing#8983)
  build: Run build & tests on main branch nightly
  build: Send test logs to Grafana Loki for statistics
@calmh calmh added this to the v1.24.0 milestone Aug 23, 2023
@st-review st-review added the frozen-due-to-age Issues closed and untouched for a long time, together with being locked for discussion label Aug 3, 2024
@syncthing syncthing locked and limited conversation to collaborators Aug 3, 2024
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Reviewers

@er-pa er-pa er-pa approved these changes

Assignees

No one assigned

Labels

frozen-due-to-age Issues closed and untouched for a long time, together with being locked for discussion

Projects

None yet

Milestone

v1.24.0

Development

Successfully merging this pull request may close these issues.

5 participants

@calmh @acolomb @ttonin33 @er-pa @st-review