fix: Add missing tenant label to some loki.write metrics #2297

towolf · 2024-12-18T10:50:41Z

PR Description

We noticed that in the some loki.write metrics we find the tenant label, but it's missing for loki_write_send_.+ and in those the tenant label would be useful.

Which issue(s) this PR fixes

Notes to the Reviewer

PR Checklist

CHANGELOG.md updated
Documentation added
Tests updated
Config converters updated

Re-add missing files

CLAassistant · 2024-12-18T10:50:49Z

All committers have signed the CLA.

CHANGELOG.md

dehaansa

Looks like there's at least one failing test that's validating the labels on the metrics. Clean that up and this should be good to go!

dehaansa · 2025-01-07T15:14:50Z

Still several failures in test

--- FAIL: TestClient_Handle (0.79s)
    --- FAIL: TestClient_Handle/batch_log_entries_together_honoring_the_tenant_ID_overridden_while_processing_the_pipeline_stages (0.11s)
        client_test.go:569: 
            	Error Trace:	/Users/runner/work/alloy/alloy/internal/component/common/loki/client/client_test.go:569
            	Error:      	Received unexpected error:
            	            	 # HELP loki_write_dropped_entries_total Number of log entries dropped because failed to be sent to the ingester after all retries.
            	            	 # TYPE loki_write_dropped_entries_total counter
            	            	 loki_write_dropped_entries_total{host="127.0.0.1:[49](https://github.com/grafana/alloy/actions/runs/12641290761/job/35236818571?pr=2297#step:4:50)248",reason="ingester_error",tenant="tenant-1"} 0
            	            	 loki_write_dropped_entries_total{host="127.0.0.1:49248",reason="ingester_error",tenant="tenant-2"} 0
            	            	 loki_write_dropped_entries_total{host="127.0.0.1:49248",reason="ingester_error",tenant="tenant-default"} 0
            	            	 loki_write_dropped_entries_total{host="127.0.0.1:49248",reason="line_too_long",tenant="tenant-1"} 0
            	            	 loki_write_dropped_entries_total{host="127.0.0.1:49248",reason="line_too_long",tenant="tenant-2"} 0
            	            	 loki_write_dropped_entries_total{host="127.0.0.1:49248",reason="line_too_long",tenant="tenant-default"} 0
            	            	 loki_write_dropped_entries_total{host="127.0.0.1:49248",reason="rate_limited",tenant="tenant-1"} 0
            	            	 loki_write_dropped_entries_total{host="127.0.0.1:49248",reason="rate_limited",tenant="tenant-2"} 0
            	            	 loki_write_dropped_entries_total{host="127.0.0.1:49248",reason="rate_limited",tenant="tenant-default"} 0
            	            	 loki_write_dropped_entries_total{host="127.0.0.1:49248",reason="stream_limited",tenant="tenant-1"} 0
            	            	 loki_write_dropped_entries_total{host="127.0.0.1:49248",reason="stream_limited",tenant="tenant-2"} 0
            	            	 loki_write_dropped_entries_total{host="127.0.0.1:49248",reason="stream_limited",tenant="tenant-default"} 0
            	            	 # HELP loki_write_mutated_bytes_total The total number of bytes that have been mutated.
            	            	 # TYPE loki_write_mutated_bytes_total counter
            	            	 loki_write_mutated_bytes_total{host="127.0.0.1:49248",reason="ingester_error",tenant="tenant-1"} 0
            	            	 loki_write_mutated_bytes_total{host="127.0.0.1:49248",reason="ingester_error",tenant="tenant-2"} 0
            	            	 loki_write_mutated_bytes_total{host="127.0.0.1:49248",reason="ingester_error",tenant="tenant-default"} 0
            	            	 loki_write_mutated_bytes_total{host="127.0.0.1:49248",reason="line_too_long",tenant="tenant-1"} 0
            	            	 loki_write_mutated_bytes_total{host="127.0.0.1:49248",reason="line_too_long",tenant="tenant-2"} 0
            	            	 loki_write_mutated_bytes_total{host="127.0.0.1:49248",reason="line_too_long",tenant="tenant-default"} 0
            	            	 loki_write_mutated_bytes_total{host="127.0.0.1:49248",reason="rate_limited",tenant="tenant-1"} 0
            	            	 loki_write_mutated_bytes_total{host="127.0.0.1:49248",reason="rate_limited",tenant="tenant-2"} 0
            	            	 loki_write_mutated_bytes_total{host="127.0.0.1:49248",reason="rate_limited",tenant="tenant-default"} 0
            	            	 loki_write_mutated_bytes_total{host="127.0.0.1:49248",reason="stream_limited",tenant="tenant-1"} 0
            	            	 loki_write_mutated_bytes_total{host="127.0.0.1:49248",reason="stream_limited",tenant="tenant-2"} 0
            	            	 loki_write_mutated_bytes_total{host="127.0.0.1:49248",reason="stream_limited",tenant="tenant-default"} 0
            	            	 # HELP loki_write_mutated_entries_total The total number of log entries that have been mutated.
            	            	 # TYPE loki_write_mutated_entries_total counter
            	            	 loki_write_mutated_entries_total{host="127.0.0.1:49248",reason="ingester_error",tenant="tenant-1"} 0
            	            	 loki_write_mutated_entries_total{host="127.0.0.1:49248",reason="ingester_error",tenant="tenant-2"} 0
            	            	 loki_write_mutated_entries_total{host="127.0.0.1:49248",reason="ingester_error",tenant="tenant-default"} 0
            	            	 loki_write_mutated_entries_total{host="127.0.0.1:49248",reason="line_too_long",tenant="tenant-1"} 0
            	            	 loki_write_mutated_entries_total{host="127.0.0.1:49248",reason="line_too_long",tenant="tenant-2"} 0
            	            	 loki_write_mutated_entries_total{host="127.0.0.1:49248",reason="line_too_long",tenant="tenant-default"} 0
            	            	 loki_write_mutated_entries_total{host="127.0.0.1:49248",reason="rate_limited",tenant="tenant-1"} 0
            	            	 loki_write_mutated_entries_total{host="127.0.0.1:49248",reason="rate_limited",tenant="tenant-2"} 0
            	            	 loki_write_mutated_entries_total{host="127.0.0.1:49248",reason="rate_limited",tenant="tenant-default"} 0
            	            	 loki_write_mutated_entries_total{host="127.0.0.1:49248",reason="stream_limited",tenant="tenant-1"} 0
            	            	 loki_write_mutated_entries_total{host="127.0.0.1:49248",reason="stream_limited",tenant="tenant-2"} 0
            	            	 loki_write_mutated_entries_total{host="127.0.0.1:49248",reason="stream_limited",tenant="tenant-default"} 0
            	            	 # HELP loki_write_sent_entries_total Number of log entries sent to the ingester.
            	            	 # TYPE loki_write_sent_entries_total counter
            	            	-loki_write_sent_entries_total{host="127.0.0.1:49248",tenant="tenant-1"} 2
            	            	-loki_write_sent_entries_total{host="127.0.0.1:49248",tenant="tenant-2"} 1
            	            	-loki_write_sent_entries_total{host="127.0.0.1:49248",tenant="tenant-default"} 1
            	            	+loki_write_sent_entries_total{host="127.0.0.1:49248",tenant=""} 4
            	            	 
            	Test:       	TestClient_Handle/batch_log_entries_together_honoring_the_tenant_ID_overridden_while_processing_the_pipeline_stages
--- FAIL: TestClient_StopNow (0.13s)
    --- FAIL: TestClient_StopNow/send_requests_shouldn't_be_cancelled_after_StopNow() (0.11s)
        client_test.go:714: 
            	Error Trace:	/Users/runner/work/alloy/alloy/internal/component/common/loki/client/client_test.go:714
            	Error:      	Received unexpected error:
            	            	 # HELP loki_write_dropped_entries_total Number of log entries dropped because failed to be sent to the ingester after all retries.
            	            	 # TYPE loki_write_dropped_entries_total counter
            	            	 loki_write_dropped_entries_total{host="127.0.0.1:492[50](https://github.com/grafana/alloy/actions/runs/12641290761/job/35236818571?pr=2297#step:4:51)",reason="ingester_error",tenant=""} 0
            	            	 loki_write_dropped_entries_total{host="127.0.0.1:49250",reason="line_too_long",tenant=""} 0
            	            	 loki_write_dropped_entries_total{host="127.0.0.1:49250",reason="rate_limited",tenant=""} 0
            	            	 loki_write_dropped_entries_total{host="127.0.0.1:49250",reason="stream_limited",tenant=""} 0
            	            	 # HELP loki_write_sent_entries_total Number of log entries sent to the ingester.
            	            	 # TYPE loki_write_sent_entries_total counter
            	            	-loki_write_sent_entries_total{host="127.0.0.1:49250",tenant=""} 3
            	            	+loki_write_sent_entries_total{host="127.0.0.1:49250"} 3
            	            	 
            	Test:       	TestClient_StopNow/send_requests_shouldn't_be_cancelled_after_StopNow()
    --- FAIL: TestClient_StopNow/shouldn't_retry_after_StopNow() (0.02s)
        client_test.go:714: 
            	Error Trace:	/Users/runner/work/alloy/alloy/internal/component/common/loki/client/client_test.go:714
            	Error:      	Received unexpected error:
            	            	 # HELP loki_write_dropped_entries_total Number of log entries dropped because failed to be sent to the ingester after all retries.
            	            	 # TYPE loki_write_dropped_entries_total counter
            	            	 loki_write_dropped_entries_total{host="127.0.0.1:492[52](https://github.com/grafana/alloy/actions/runs/12641290761/job/35236818571?pr=2297#step:4:53)",reason="ingester_error",tenant=""} 0
            	            	 loki_write_dropped_entries_total{host="127.0.0.1:49252",reason="line_too_long",tenant=""} 0
            	            	 loki_write_dropped_entries_total{host="127.0.0.1:49252",reason="rate_limited",tenant=""} 1
            	            	 loki_write_dropped_entries_total{host="127.0.0.1:49252",reason="stream_limited",tenant=""} 0
            	            	 # HELP loki_write_sent_entries_total Number of log entries sent to the ingester.
            	            	 # TYPE loki_write_sent_entries_total counter
            	            	-loki_write_sent_entries_total{host="127.0.0.1:49252",tenant=""} 0
            	            	+loki_write_sent_entries_total{host="127.0.0.1:49252"} 0
            	            	 
            	Test:       	TestClient_StopNow/shouldn't_retry_after_StopNow()

towolf · 2025-01-08T11:14:10Z

Yeah, sorry, I missed some spots in the tests.

Fingers crossed 🤞

fix: Add missing tenant label to some loki_write metrics

f9367e0

Re-add missing files

towolf requested a review from a team as a code owner December 18, 2024 10:50

towolf added 2 commits December 18, 2024 17:43

Actually hook up to loki_write metrics

0064049

Missed some occurences in tests

9ad49f0

towolf changed the title ~~fix: Add missing tenant label to some loki_write metrics~~ fix: Add missing tenant label to some loki.write metrics Dec 18, 2024

dehaansa reviewed Jan 3, 2025

View reviewed changes

CHANGELOG.md Outdated Show resolved Hide resolved

towolf and others added 3 commits January 3, 2025 17:14

Add tenant label to requestDuration

d9bd213

Add attribution

9bfaec4

Merge branch 'grafana:main' into add-tenant-label

02ef49f

dehaansa approved these changes Jan 6, 2025

View reviewed changes

dehaansa requested changes Jan 6, 2025

View reviewed changes

Fix the failing test

351a091

Fix remaining tests

55e4387

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

fix: Add missing tenant label to some loki.write metrics #2297

fix: Add missing tenant label to some loki.write metrics #2297

towolf commented Dec 18, 2024 •

edited

Loading

CLAassistant commented Dec 18, 2024 •

edited

Loading

dehaansa left a comment

dehaansa commented Jan 7, 2025

towolf commented Jan 8, 2025 •

edited

Loading

fix: Add missing tenant label to some loki.write metrics #2297

Are you sure you want to change the base?

fix: Add missing tenant label to some loki.write metrics #2297

Conversation

towolf commented Dec 18, 2024 • edited Loading

PR Description

Which issue(s) this PR fixes

Notes to the Reviewer

PR Checklist

CLAassistant commented Dec 18, 2024 • edited Loading

dehaansa left a comment

Choose a reason for hiding this comment

dehaansa commented Jan 7, 2025

towolf commented Jan 8, 2025 • edited Loading

towolf commented Dec 18, 2024 •

edited

Loading

CLAassistant commented Dec 18, 2024 •

edited

Loading

towolf commented Jan 8, 2025 •

edited

Loading