From e0031b0882958dd97976f4c2ebc30cf35a842965 Mon Sep 17 00:00:00 2001 From: josunect Date: Thu, 24 Oct 2024 15:55:33 +0100 Subject: [PATCH 01/11] Add tempo tunning --- .../p8s-jaeger-grafana/tracing/tempo.md | 30 +++++++++++++++++++ 1 file changed, 30 insertions(+) diff --git a/content/en/docs/Configuration/p8s-jaeger-grafana/tracing/tempo.md b/content/en/docs/Configuration/p8s-jaeger-grafana/tracing/tempo.md index 2ee66c07..261f603f 100644 --- a/content/en/docs/Configuration/p8s-jaeger-grafana/tracing/tempo.md +++ b/content/en/docs/Configuration/p8s-jaeger-grafana/tracing/tempo.md @@ -237,6 +237,36 @@ In `external_services.tracing` | Jaeger | `.internal_url = 'http://jaeger_service_url:16686/jaeger'`
`.use_grpc = false`
| `.internal_url = 'http://jaeger_service_url:16685/jaeger'`
`.use_grpc = true (Not required: by default)`
| | Tempo |
`internal_url = 'http://query_frontend_url:3200'`
`.use_grpc = false`
`.provider = 'tempo'`
| `.internal_url = 'http://query_frontend_url:3200'`
`.grpc_port: 9095`
`.provider: 'tempo'`
`.use_grpc = true (Not required: by default)`
| +### Tempo tunning + +Grafana Tempo is a powerful tool, but it can lead to performance issues when not configured correctly. +The following configuration can lead to OOM issues for simple queries in the frontend-querier component: + +```yaml +spec: + resources: + total: + limits: + memory: 2Gi + cpu: 2000m +``` + +These resources are shared between all the Tempo components. +Instead of applying the resources globally, they can be applied to each specific component, when needed: + +```yaml +spec: + template: + queryFrontend: + component: + resources: + limits: + cpu: "2" + memory: 2Gi +``` + + + ### Tempo authentication configuration The Kiali CR provides authentication configuration that will be used also for querying the version check to provide information in the Mesh graph. From 184861dcebcc5a1c13dd7871e94ce23c7df4029c Mon Sep 17 00:00:00 2001 From: josunect Date: Thu, 24 Oct 2024 16:00:46 +0100 Subject: [PATCH 02/11] Fix sintax --- .../en/docs/Configuration/p8s-jaeger-grafana/tracing/tempo.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/content/en/docs/Configuration/p8s-jaeger-grafana/tracing/tempo.md b/content/en/docs/Configuration/p8s-jaeger-grafana/tracing/tempo.md index 261f603f..d9fc8711 100644 --- a/content/en/docs/Configuration/p8s-jaeger-grafana/tracing/tempo.md +++ b/content/en/docs/Configuration/p8s-jaeger-grafana/tracing/tempo.md @@ -237,10 +237,10 @@ In `external_services.tracing` | Jaeger | `.internal_url = 'http://jaeger_service_url:16686/jaeger'`
`.use_grpc = false`
| `.internal_url = 'http://jaeger_service_url:16685/jaeger'`
`.use_grpc = true (Not required: by default)`
| | Tempo |
`internal_url = 'http://query_frontend_url:3200'`
`.use_grpc = false`
`.provider = 'tempo'`
| `.internal_url = 'http://query_frontend_url:3200'`
`.grpc_port: 9095`
`.provider: 'tempo'`
`.use_grpc = true (Not required: by default)`
| -### Tempo tunning +### Tempo tuning Grafana Tempo is a powerful tool, but it can lead to performance issues when not configured correctly. -The following configuration can lead to OOM issues for simple queries in the frontend-querier component: +The following configuration can lead to OOM issues for simple queries in the query-frontend component: ```yaml spec: From 6c75294bf2fa21f1edabf50d01545621513b4482 Mon Sep 17 00:00:00 2001 From: josunect Date: Wed, 30 Oct 2024 13:21:28 +0000 Subject: [PATCH 03/11] Add some recommendations --- .../p8s-jaeger-grafana/tracing/tempo.md | 26 ++ static/files/tempo-dashboard.json | 276 ++++++++++++++++++ 2 files changed, 302 insertions(+) create mode 100644 static/files/tempo-dashboard.json diff --git a/content/en/docs/Configuration/p8s-jaeger-grafana/tracing/tempo.md b/content/en/docs/Configuration/p8s-jaeger-grafana/tracing/tempo.md index d9fc8711..b4a3912f 100644 --- a/content/en/docs/Configuration/p8s-jaeger-grafana/tracing/tempo.md +++ b/content/en/docs/Configuration/p8s-jaeger-grafana/tracing/tempo.md @@ -239,6 +239,8 @@ In `external_services.tracing` ### Tempo tuning +#### Resources consumption + Grafana Tempo is a powerful tool, but it can lead to performance issues when not configured correctly. The following configuration can lead to OOM issues for simple queries in the query-frontend component: @@ -265,7 +267,31 @@ spec: memory: 2Gi ``` +[This Grafana Dashboard]("/static/files/tempo-dashboard.json") is available to measure the resources used in the Tempo namespace. + +#### Caching + +Tempo can use [a cache](https://grafana.com/docs/tempo/latest/operations/caching/) that acts in different levels that is used by default with Tanka and Helm deployment examples. It uses external cache, supporting Memcached and Redis. +The lower level cache has a higher hit rate, and caches bloom filters and parquet data. +The higher level caches frontend-search data. + +Optimizing the cache depends on the application usage, and can be done modifying different parameters: + +- Connection limit for MemCached: Should be increased in large deployments, as MemCached is set to 1024 by default. +- Cache size control: Should be increased when the working set is larger than the size of cache. + +#### Tune search pipeline + +There are many parameters to [tune the search pipeline](https://grafana.com/docs/tempo/latest/operations/backend_search/), some of these: + +- max_concurrent_queries: If it is too high it can cause OOM. +- concurrent_jobs: How many jobs are done concurrently. +- max_retries: When it is too high it can result in a lot of load. + +#### Dedicated attribute columns +When used the storage format vParquet3, defining [dedicated attribute columns}(https://grafana.com/docs/tempo/latest/operations/dedicated_columns/) can improve the query performance. +In order to best choose those columns (Up to 10), a good criteria is to choose attributes that contribute growing the block size (And not those commonly used). ### Tempo authentication configuration diff --git a/static/files/tempo-dashboard.json b/static/files/tempo-dashboard.json new file mode 100644 index 00000000..f76812fa --- /dev/null +++ b/static/files/tempo-dashboard.json @@ -0,0 +1,276 @@ +{ + "annotations": { + "list": [ + { + "builtIn": 1, + "datasource": { + "type": "grafana", + "uid": "-- Grafana --" + }, + "enable": true, + "hide": true, + "iconColor": "rgba(0, 211, 255, 1)", + "name": "Annotations & Alerts", + "type": "dashboard" + } + ] + }, + "editable": true, + "fiscalYearStartMonth": 0, + "graphTooltip": 0, + "id": 9, + "links": [], + "panels": [ + { + "datasource": { + "type": "prometheus", + "uid": "PBFA97CFB590B2093" + }, + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "axisBorderShow": false, + "axisCenteredZero": false, + "axisColorMode": "text", + "axisLabel": "", + "axisPlacement": "auto", + "barAlignment": 0, + "drawStyle": "line", + "fillOpacity": 0, + "gradientMode": "none", + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "insertNulls": false, + "lineInterpolation": "linear", + "lineWidth": 1, + "pointSize": 5, + "scaleDistribution": { + "type": "linear" + }, + "showPoints": "auto", + "spanNulls": false, + "stacking": { + "group": "A", + "mode": "none" + }, + "thresholdsStyle": { + "mode": "off" + } + }, + "mappings": [], + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": null + }, + { + "color": "red", + "value": 80 + } + ] + } + }, + "overrides": [] + }, + "gridPos": { + "h": 11, + "w": 22, + "x": 0, + "y": 0 + }, + "id": 2, + "options": { + "legend": { + "calcs": [], + "displayMode": "list", + "placement": "bottom", + "showLegend": true + }, + "tooltip": { + "maxHeight": 600, + "mode": "single", + "sort": "none" + } + }, + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "PBFA97CFB590B2093" + }, + "disableTextWrap": false, + "editorMode": "builder", + "expr": "rate(container_cpu_system_seconds_total{namespace=\"tempo\"}[$__rate_interval])", + "fullMetaSearch": false, + "includeNullMetadata": true, + "instant": false, + "legendFormat": "{{pod}}", + "range": true, + "refId": "A", + "useBackend": false + }, + { + "datasource": { + "type": "prometheus", + "uid": "PBFA97CFB590B2093" + }, + "disableTextWrap": false, + "editorMode": "builder", + "expr": "sum by(sum) (rate(container_cpu_system_seconds_total{namespace=\"tempo\"}[$__rate_interval]))", + "fullMetaSearch": false, + "hide": false, + "includeNullMetadata": false, + "instant": false, + "legendFormat": "__auto", + "range": true, + "refId": "B", + "useBackend": false + } + ], + "title": "Container CPU system seconds", + "type": "timeseries" + }, + { + "datasource": { + "type": "prometheus", + "uid": "PBFA97CFB590B2093" + }, + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "axisBorderShow": false, + "axisCenteredZero": false, + "axisColorMode": "text", + "axisLabel": "", + "axisPlacement": "auto", + "barAlignment": 0, + "drawStyle": "line", + "fillOpacity": 0, + "gradientMode": "none", + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "insertNulls": false, + "lineInterpolation": "linear", + "lineWidth": 1, + "pointSize": 5, + "scaleDistribution": { + "type": "linear" + }, + "showPoints": "auto", + "spanNulls": false, + "stacking": { + "group": "A", + "mode": "none" + }, + "thresholdsStyle": { + "mode": "off" + } + }, + "mappings": [], + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": null + }, + { + "color": "red", + "value": 80 + } + ] + } + }, + "overrides": [] + }, + "gridPos": { + "h": 9, + "w": 22, + "x": 0, + "y": 11 + }, + "id": 1, + "options": { + "legend": { + "calcs": [], + "displayMode": "list", + "placement": "bottom", + "showLegend": true + }, + "tooltip": { + "maxHeight": 600, + "mode": "single", + "sort": "none" + } + }, + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "PBFA97CFB590B2093" + }, + "disableTextWrap": false, + "editorMode": "builder", + "expr": "container_memory_usage_bytes{namespace=\"tempo\"} / 1048576", + "fullMetaSearch": false, + "includeNullMetadata": true, + "instant": false, + "interval": "", + "legendFormat": "{{pod}}", + "range": true, + "refId": "A", + "useBackend": false + }, + { + "datasource": { + "type": "prometheus", + "uid": "PBFA97CFB590B2093" + }, + "disableTextWrap": false, + "editorMode": "builder", + "expr": "sum(container_memory_usage_bytes{namespace=\"tempo\"} / 1048576)", + "fullMetaSearch": false, + "hide": false, + "includeNullMetadata": true, + "instant": false, + "legendFormat": "__auto", + "range": true, + "refId": "B", + "useBackend": false + } + ], + "title": "Container memory usage MB", + "type": "timeseries" + } + ], + "schemaVersion": 39, + "tags": [], + "templating": { + "list": [] + }, + "time": { + "from": "now-5m", + "to": "now" + }, + "timeRangeUpdatedDuringEditOrView": false, + "timepicker": {}, + "timezone": "browser", + "title": "Tempo resources consumption by pod", + "uid": "bdyyqh50uxwcgf", + "version": 3, + "weekStart": "" +} \ No newline at end of file From eddcfce429a12a51031dc4cc5f0b89958d52066a Mon Sep 17 00:00:00 2001 From: josunect Date: Wed, 30 Oct 2024 13:28:24 +0000 Subject: [PATCH 04/11] Add some recommendations --- .../en/docs/Configuration/p8s-jaeger-grafana/tracing/tempo.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/content/en/docs/Configuration/p8s-jaeger-grafana/tracing/tempo.md b/content/en/docs/Configuration/p8s-jaeger-grafana/tracing/tempo.md index b4a3912f..1951d377 100644 --- a/content/en/docs/Configuration/p8s-jaeger-grafana/tracing/tempo.md +++ b/content/en/docs/Configuration/p8s-jaeger-grafana/tracing/tempo.md @@ -267,7 +267,7 @@ spec: memory: 2Gi ``` -[This Grafana Dashboard]("/static/files/tempo-dashboard.json") is available to measure the resources used in the Tempo namespace. +[This Grafana Dashboard](/static/files/tempo-dashboard.json) is available to measure the resources used in the Tempo namespace. #### Caching @@ -290,7 +290,7 @@ There are many parameters to [tune the search pipeline](https://grafana.com/docs #### Dedicated attribute columns -When used the storage format vParquet3, defining [dedicated attribute columns}(https://grafana.com/docs/tempo/latest/operations/dedicated_columns/) can improve the query performance. +When used the storage format vParquet3, defining [dedicated attribute columns](https://grafana.com/docs/tempo/latest/operations/dedicated_columns/) can improve the query performance. In order to best choose those columns (Up to 10), a good criteria is to choose attributes that contribute growing the block size (And not those commonly used). ### Tempo authentication configuration From c3cdea289b1ea0956ddacd899e546a29198f0df2 Mon Sep 17 00:00:00 2001 From: josunect Date: Wed, 30 Oct 2024 13:31:44 +0000 Subject: [PATCH 05/11] Add some recommendations --- .../en/docs/Configuration/p8s-jaeger-grafana/tracing/tempo.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/content/en/docs/Configuration/p8s-jaeger-grafana/tracing/tempo.md b/content/en/docs/Configuration/p8s-jaeger-grafana/tracing/tempo.md index 1951d377..bea1bcb3 100644 --- a/content/en/docs/Configuration/p8s-jaeger-grafana/tracing/tempo.md +++ b/content/en/docs/Configuration/p8s-jaeger-grafana/tracing/tempo.md @@ -267,7 +267,7 @@ spec: memory: 2Gi ``` -[This Grafana Dashboard](/static/files/tempo-dashboard.json) is available to measure the resources used in the Tempo namespace. +[This Grafana Dashboard](/files/tempo-dashboard.json) is available to measure the resources used in the Tempo namespace. #### Caching From 71924a817fcacca5e4450838597e3144c1d148b5 Mon Sep 17 00:00:00 2001 From: josunect Date: Wed, 30 Oct 2024 14:52:07 +0000 Subject: [PATCH 06/11] add table of contents --- .../p8s-jaeger-grafana/tracing/tempo.md | 24 ++++++++++++++++++- 1 file changed, 23 insertions(+), 1 deletion(-) diff --git a/content/en/docs/Configuration/p8s-jaeger-grafana/tracing/tempo.md b/content/en/docs/Configuration/p8s-jaeger-grafana/tracing/tempo.md index bea1bcb3..87cd5298 100644 --- a/content/en/docs/Configuration/p8s-jaeger-grafana/tracing/tempo.md +++ b/content/en/docs/Configuration/p8s-jaeger-grafana/tracing/tempo.md @@ -5,6 +5,28 @@ description: > weight: 2 --- +- [Grafana Tempo Configuration](#grafana-tempo-configuration) + - [Using the Grafana Tempo API](#using-the-grafana-tempo-api) + - [Setup the Kiali CR](#set-up-the-kiali-cr) + - [Set up a Tempo Datasource in Grafana](#set-up-a-tempo-datasource-in-grafana) + - [Additional Configuration](#additional-configuration) + - [Service check URL](#service-check-url) + - [Configuration for the Grafana Tempo Datasource](#configuration-for-the-grafana-tempo-datasource) + - [Using the Jaeger frontend with Grafana Tempo tracing backend](#using-the-jaeger-frontend-with-grafana-tempo-tracing-backend) + - [Tanka](#tanka) + - [Tempo Operator](#tempo-operator) + - [Configuration table](#configuration-table) + - [Supported Versions](#supported-versions) + - [Minimal configuration for Kiali <= 1.79](#minimal-configuration-for-kiali--179) + - [Minimal configuration for Kiali > 1.79](#minimal-configuration-for-kiali--179-1) + - [Tempo tuning](#tempo-tuning) + - [Resources consumption](#resources-consumption) + - [Caching](#caching) + - [Resources consumption](#tune-search-pipeline) + - [Dedicated attribute columns](#dedicated-attribute-columns) + - [Tempo authentication configuration](#tempo-authentication-configuration) + + ## Grafana Tempo Configuration There are two possibilities to integrate Kiali with Grafana Tempo: @@ -267,7 +289,7 @@ spec: memory: 2Gi ``` -[This Grafana Dashboard](/files/tempo-dashboard.json) is available to measure the resources used in the Tempo namespace. +[This Grafana Dashboard](/files/tempo-dashboard.json) is available to measure the resources used in the **tempo** namespace. #### Caching From 274cf48ea9e38c5c98a37caa45ad3cc07ac086ee Mon Sep 17 00:00:00 2001 From: josunect Date: Wed, 30 Oct 2024 15:02:04 +0000 Subject: [PATCH 07/11] add table of contents --- .../p8s-jaeger-grafana/tracing/tempo.md | 20 +++++++++---------- 1 file changed, 10 insertions(+), 10 deletions(-) diff --git a/content/en/docs/Configuration/p8s-jaeger-grafana/tracing/tempo.md b/content/en/docs/Configuration/p8s-jaeger-grafana/tracing/tempo.md index 87cd5298..ee74b591 100644 --- a/content/en/docs/Configuration/p8s-jaeger-grafana/tracing/tempo.md +++ b/content/en/docs/Configuration/p8s-jaeger-grafana/tracing/tempo.md @@ -15,16 +15,16 @@ weight: 2 - [Using the Jaeger frontend with Grafana Tempo tracing backend](#using-the-jaeger-frontend-with-grafana-tempo-tracing-backend) - [Tanka](#tanka) - [Tempo Operator](#tempo-operator) - - [Configuration table](#configuration-table) - - [Supported Versions](#supported-versions) - - [Minimal configuration for Kiali <= 1.79](#minimal-configuration-for-kiali--179) - - [Minimal configuration for Kiali > 1.79](#minimal-configuration-for-kiali--179-1) - - [Tempo tuning](#tempo-tuning) - - [Resources consumption](#resources-consumption) - - [Caching](#caching) - - [Resources consumption](#tune-search-pipeline) - - [Dedicated attribute columns](#dedicated-attribute-columns) - - [Tempo authentication configuration](#tempo-authentication-configuration) +- [Configuration table](#configuration-table) + - [Supported Versions](#supported-versions) + - [Minimal configuration for Kiali <= 1.79](#minimal-configuration-for-kiali--179) + - [Minimal configuration for Kiali > 1.79](#minimal-configuration-for-kiali--179-1) +- [Tempo tuning](#tempo-tuning) + - [Resources consumption](#resources-consumption) + - [Caching](#caching) + - [Resources consumption](#tune-search-pipeline) + - [Dedicated attribute columns](#dedicated-attribute-columns) +- [Tempo authentication configuration](#tempo-authentication-configuration) ## Grafana Tempo Configuration From 514c446bf12be8e6934e0cc61d019de21c94b3ed Mon Sep 17 00:00:00 2001 From: Josune Cordoba <49480155+josunect@users.noreply.github.com> Date: Thu, 14 Nov 2024 06:01:49 +0000 Subject: [PATCH 08/11] Update content/en/docs/Configuration/p8s-jaeger-grafana/tracing/tempo.md Co-authored-by: Jay Shaughnessy --- .../en/docs/Configuration/p8s-jaeger-grafana/tracing/tempo.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/content/en/docs/Configuration/p8s-jaeger-grafana/tracing/tempo.md b/content/en/docs/Configuration/p8s-jaeger-grafana/tracing/tempo.md index ee74b591..5bf836dd 100644 --- a/content/en/docs/Configuration/p8s-jaeger-grafana/tracing/tempo.md +++ b/content/en/docs/Configuration/p8s-jaeger-grafana/tracing/tempo.md @@ -264,7 +264,7 @@ In `external_services.tracing` #### Resources consumption Grafana Tempo is a powerful tool, but it can lead to performance issues when not configured correctly. -The following configuration can lead to OOM issues for simple queries in the query-frontend component: +For example, the following configuration is not recommended and may lead to OOM issues for simple queries in the query-frontend component: ```yaml spec: From 59e34e48033bf1e5531c120560e6b1b87cad9353 Mon Sep 17 00:00:00 2001 From: Josune Cordoba <49480155+josunect@users.noreply.github.com> Date: Thu, 14 Nov 2024 06:02:11 +0000 Subject: [PATCH 09/11] Update content/en/docs/Configuration/p8s-jaeger-grafana/tracing/tempo.md Co-authored-by: Jay Shaughnessy --- .../en/docs/Configuration/p8s-jaeger-grafana/tracing/tempo.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/content/en/docs/Configuration/p8s-jaeger-grafana/tracing/tempo.md b/content/en/docs/Configuration/p8s-jaeger-grafana/tracing/tempo.md index 5bf836dd..eb4cead5 100644 --- a/content/en/docs/Configuration/p8s-jaeger-grafana/tracing/tempo.md +++ b/content/en/docs/Configuration/p8s-jaeger-grafana/tracing/tempo.md @@ -276,7 +276,7 @@ spec: ``` These resources are shared between all the Tempo components. -Instead of applying the resources globally, they can be applied to each specific component, when needed: +When needed, apply resources to each specific component, instead of applying the resources globally: ```yaml spec: From 0cdab707195647939b2419e5b455efe2547d7f3f Mon Sep 17 00:00:00 2001 From: Josune Cordoba <49480155+josunect@users.noreply.github.com> Date: Thu, 14 Nov 2024 06:02:24 +0000 Subject: [PATCH 10/11] Update content/en/docs/Configuration/p8s-jaeger-grafana/tracing/tempo.md Co-authored-by: Jay Shaughnessy --- .../en/docs/Configuration/p8s-jaeger-grafana/tracing/tempo.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/content/en/docs/Configuration/p8s-jaeger-grafana/tracing/tempo.md b/content/en/docs/Configuration/p8s-jaeger-grafana/tracing/tempo.md index eb4cead5..b055b472 100644 --- a/content/en/docs/Configuration/p8s-jaeger-grafana/tracing/tempo.md +++ b/content/en/docs/Configuration/p8s-jaeger-grafana/tracing/tempo.md @@ -293,7 +293,7 @@ spec: #### Caching -Tempo can use [a cache](https://grafana.com/docs/tempo/latest/operations/caching/) that acts in different levels that is used by default with Tanka and Helm deployment examples. It uses external cache, supporting Memcached and Redis. +Tempo offers multi-level [caching](https://grafana.com/docs/tempo/latest/operations/caching/) that is used by default with Tanka and Helm deployment examples. It uses external cache, supporting Memcached and Redis. The lower level cache has a higher hit rate, and caches bloom filters and parquet data. The higher level caches frontend-search data. From b5e06fb06521b405c7546798293c0f566f5201dd Mon Sep 17 00:00:00 2001 From: Josune Cordoba <49480155+josunect@users.noreply.github.com> Date: Thu, 14 Nov 2024 06:02:47 +0000 Subject: [PATCH 11/11] Update content/en/docs/Configuration/p8s-jaeger-grafana/tracing/tempo.md Co-authored-by: Jay Shaughnessy --- .../en/docs/Configuration/p8s-jaeger-grafana/tracing/tempo.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/content/en/docs/Configuration/p8s-jaeger-grafana/tracing/tempo.md b/content/en/docs/Configuration/p8s-jaeger-grafana/tracing/tempo.md index b055b472..fbf0c0db 100644 --- a/content/en/docs/Configuration/p8s-jaeger-grafana/tracing/tempo.md +++ b/content/en/docs/Configuration/p8s-jaeger-grafana/tracing/tempo.md @@ -312,7 +312,7 @@ There are many parameters to [tune the search pipeline](https://grafana.com/docs #### Dedicated attribute columns -When used the storage format vParquet3, defining [dedicated attribute columns](https://grafana.com/docs/tempo/latest/operations/dedicated_columns/) can improve the query performance. +When using the vParquet3 storage format , defining [dedicated attribute columns](https://grafana.com/docs/tempo/latest/operations/dedicated_columns/) can improve the query performance. In order to best choose those columns (Up to 10), a good criteria is to choose attributes that contribute growing the block size (And not those commonly used). ### Tempo authentication configuration