[processor/memory_limiter] Common confusion when configuring soft limit

[andrzej-stencel]

Is your feature request related to a problem? Please describe.

I saw it more than once that users misconfigure the Memory Limiter processor by setting the spike limit to a large value, not realizing that soft limit is calculated as hard limit - spike limit.

Recent example: https://cloud-native.slack.com/archives/C01N6P7KR6W/p1736477043653559

processors:
  memory_limiter:
    check_interval: 1s
    limit_percentage: 80
    spike_limit_percentage: 75

In this case, assuming 1000 MiB total available memory, the soft limit will be set to 1000 * (0.80 - 0.75) = 50 MiB.

I presume the user wanted spike_limit_percentage: 5 and soft limit 1000 * (0.80 - 0.05) = 750 MiB.

Describe the solution you'd like

Perhaps we can improve the documentation.

Describe alternatives you've considered

We could change the way the component is configured, but I don't think it's needed. Happy to hear others' thoughts.

Additional context

The documentation already explicitly says it:

The soft limit value will be equal to (limit_mib - spike_limit_mib).

This sentence is a bit buried inside the description of the spike_limit_mib option. It is not mentioned in the spike_limit_percentage option. Perhaps that's why not every user finds it?

[andrzej-stencel]

I've created #12064 that will hopefully eliminate the confusion. Let me know what you think.