updated user-mode data collection and support for flux resource manager #133
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Signed-off-by: Karl W Schulz <karl.schulz@amd.com>
Signed-off-by: Karl W Schulz <karl.schulz@amd.com>
Signed-off-by: Karl W Schulz <karl.schulz@amd.com>
Signed-off-by: Karl W Schulz <karl.schulz@amd.com>
check during metrics push Signed-off-by: Karl W Schulz <karl.schulz@amd.com>
logging output to a file which hostname prepended Signed-off-by: Karl W Schulz <karl.schulz@amd.com>
victoria metrics back-end. Current enablement hard-coded via victoriaMode=True setting in main(). Signed-off-by: Karl W Schulz <karl.schulz@amd.com>
Signed-off-by: Karl W Schulz <karl.schulz@amd.com>
… port Signed-off-by: Karl W Schulz <karl.schulz@amd.com>
command-line setting Signed-off-by: Karl W Schulz <karl.schulz@amd.com>
Signed-off-by: Karl W Schulz <karl.schulz@amd.com>
Signed-off-by: Karl W Schulz <karl.schulz@amd.com>
…tepFile Signed-off-by: Karl W Schulz <karl.schulz@amd.com>
annotation file Signed-off-by: Karl W Schulz <karl.schulz@amd.com>
execution Signed-off-by: Karl W Schulz <karl.schulz@amd.com>
victoriametrics during a job; threading added to support flask endpoint that can be used to terminate the data collector and push final data (previous file-based termination removed). Signed-off-by: Karl W Schulz <karl.schulz@amd.com>
"shutdown" endpoint; restrict max number of go processes for victormetrics server Signed-off-by: Karl W Schulz <karl.schulz@amd.com>
Signed-off-by: Karl W Schulz <karl.schulz@amd.com>
Signed-off-by: Karl W Schulz <karl.schulz@amd.com>
Signed-off-by: Karl W Schulz <karl.schulz@amd.com>
getMetrics() method to take timestamp in millisecs directly Signed-off-by: Karl W Schulz <karl.schulz@amd.com>
(deep)copied and shipped to a separate thread to push the data. This minimizes blocking of main polling loop. Signed-off-by: Karl W Schulz <karl.schulz@amd.com>
victoriametrics; remove unused remotewrite configuration for prometheus Signed-off-by: Karl W Schulz <karl.schulz@amd.com>
victoriametrics Signed-off-by: Karl W Schulz <karl.schulz@amd.com>
names for victoriametrics settings Signed-off-by: Karl W Schulz <karl.schulz@amd.com>
Signed-off-by: Karl W Schulz <karl.schulz@amd.com>
checking for victoriametrics path; tweak shutdown timeout for exporter when using victoriametrics Signed-off-by: Karl W Schulz <karl.schulz@amd.com>
victoriametrics examples Signed-off-by: Karl W Schulz <karl.schulz@amd.com>
Signed-off-by: Karl W. Schulz <karl.schulz@amd.com>
Signed-off-by: Karl W. Schulz <karl.schulz@amd.com>
Signed-off-by: Karl W. Schulz <karl.schulz@amd.com>
Signed-off-by: Karl W. Schulz <karl.schulz@amd.com>
Signed-off-by: Karl W. Schulz <karl.schulz@amd.com>
Signed-off-by: Karl W. Schulz <karl.schulz@amd.com>
time in the Job Step panel; additional queries and transformations added to sort by the job step time (since we cannot assume the job step is an ordinal number, thank you flux). Enabled missing legend in the Average GPU Power panel. Signed-off-by: Karl W. Schulz <karl.schulz@amd.com>
Signed-off-by: Karl W. Schulz <karl.schulz@amd.com>
Signed-off-by: Karl W. Schulz <karl.schulz@amd.com>
Signed-off-by: Karl W. Schulz <karl.schulz@amd.com>
using prometheus) Signed-off-by: Karl W. Schulz <karl.schulz@amd.com>
example, account for binary name in ubuntu) Signed-off-by: Karl W. Schulz <karl.schulz@amd.com>
push-based using victoria Signed-off-by: Karl W. Schulz <karl.schulz@amd.com>
Signed-off-by: Karl W. Schulz <karl.schulz@amd.com>
Signed-off-by: Karl W. Schulz <karl.schulz@amd.com>
Signed-off-by: Karl W. Schulz <karl.schulz@amd.com>
Signed-off-by: Karl W. Schulz <karl.schulz@amd.com>
Signed-off-by: Karl W. Schulz <karl.schulz@amd.com>
Signed-off-by: Karl W. Schulz <karl.schulz@amd.com>
with victoria Signed-off-by: Karl W. Schulz <karl.schulz@amd.com>
Signed-off-by: Karl W. Schulz <karl.schulz@amd.com>
Signed-off-by: Karl W. Schulz <karl.schulz@amd.com>
Signed-off-by: Karl W. Schulz <karl.schulz@amd.com>
Signed-off-by: Karl W. Schulz <karl.schulz@amd.com>
Signed-off-by: Karl W. Schulz <karl.schulz@amd.com>
Signed-off-by: Karl W. Schulz <karl.schulz@amd.com>
Signed-off-by: Karl W. Schulz <karl.schulz@amd.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR incudes two main additions:
an updated user-mode data collection process that leverages a "push" model with VictoriaMetrics as the underlying back end as opposed to the previous "pull" model with Prometheus. This approach leverages existing data collector families as is but implements a local polling loop to query the data and cache the results. At periodic intervals (default of 5 minutes), the cached results are pushed to a VictoriaMetrics server running on the master compute node. The results can be queried the same way via a prometheus endpoint provided by Victoria and there is no change required for user-mode Grafana. Adopting a cached push model, GPU telemetry metrics can be sampled at a higher rates (~e.g. down to 10-50 milliseconds).
adds support for the Flux resource manager (in addition to SLURM). This impacts the
collector_rmsdata collector and enables job identification support (along with job steps) on systems running flux. Note that flux jobids are not ordinal integers so there is some corresponding impact to Grafana dashboard configuration.