Keda prometheus trigger keeps returning empty results though present

[anipadak]

Report

The KEDA Prometheus trigger suddenly starts returning empty values. Restarting the KEDA operator pods temporarily resolves the issue, but after some time, it begins returning empty values again. however CloudWatch trigger works properly without any issues.

Keda version:

2.11.2

Scaledobject:

apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: kic-gateay-scale
  namespace: infrastructure
spec:
  scaleTargetRef:
    kind: Deployment
    name: kic-gateway
  pollingInterval: 10
  maxReplicaCount: 4
  minReplicaCount: 4
  triggers:
  - type: prometheus
    metadata:
      metricName: http_requests_total
      serverAddress: http://kube-prometheus-prometheus.observability.svc.cluster.local:9090
      query: sum(increase(http_requests_total{pod=~"kic-gateway-.*"}[2m]))
      threshold: "200"
  - type: memory
    metadata:
      type: Utilization
      value: "50"
  - type: cpu
    metadata:
      type: Utilization
      value: "60"
  advanced:
    restoreToOriginalReplicaCount: true
    horizontalPodAutoscalerConfig:
      behavior:
        scaleDown:
          stabilizationWindowSeconds: 300

Expected Behavior

successful query should return the current metric value.

But it keeps returning empty value.

Actual Behavior

Steps to Reproduce the Problem

Its happening for one environment. Not able to reproduce the same behaviour in the different environment even after having same keda config.

KEDA Version

< 2.12.0

Kubernetes Version

< 1.28

Platform

Amazon Web Services

Scaler Details

prometheus

Reference similar issue

#5432

[zroubalik]

Hi, this KEDA version is quite old. I recommend you to update it.