SCAN throwing an exception unless I use a very small window.

[rdietrich-clearstreet]

I'm getting this exception, and I can't tell if it is a configuration issue or a code issue:

Garnet.common.GarnetException: Failed to write to response buffer
at Garnet.common.GarnetException.Throw(String message, LogLevel logLevel) in /Users/rdietrich/garnet/libs/common/GarnetException.cs:line 53
at Garnet.server.RespServerSession.WriteOutputForScan(Int64 cursorValue, List`1 keys, Byte*& curr, Byte* end) in /Users/rdietrich/garnet/libs/server/Resp/ArrayCommands.cs:line 979

This error points to this comment (very polite!):

                // Reaching here means that we retried SendAndReset without the RespWriteUtils.Write*
                // method making any progress. This should only happen when the message being written is
                // too large to fit in the response buffer.
                GarnetException.Throw("Failed to write to response buffer", LogLevel.Critical);

I have 100k keys in my garnet instance. I use a redis client that uses SCAN on bootup to download the keys. I figured out if I lowered my SCAN value all the way down to 10k per SCAN it works. The only problem is, I plan on having millions of keys, so scanning 10k at a time is going to be pretty slow.

I've been digging through the source code looking for either code or configuration that determines the response buffer size. I would like to increase this buffer massively so I can not worry about this. I did try using ObjectScanCountLimit, but it didn't seem to internally limit the request (I had my SCAN count in my client set to 100,000).

Any help on changing the configuration or forcing the response buffer to be larger would be helpful.

This is my current configuration:

    "EnableAOF": true,
    "AofMemorySize": "512m",
    "AofPageSize": "64m",
    "AofSizeLimit": "1g",
    "MemorySize": "4g",
    "ObjectScanCountLimit": 10000,
    "CompactionFrequencySecs": 3600,
    "CompactionType": "Lookup",
    "CommitFrequencyMS": 1000,
    "EnableStorageTier": true,
    "Recover": true,
    "EnableFastCommit": true,
    "EnableIncrementalSnapshots": true,
    "LogLevel": "Trace",
    "OnDemandCheckpoint": true