Default socket buffer/rcvbuf size

[essen]

Is your feature request related to a problem? Please describe.
In https://www.erlang.org/doc/apps/kernel/inet.html#setopts/2 we can read:

For TCP it is recommended to have val(buffer) >= val(recbuf) to avoid performance issues because of unnecessary copying.

But the default is very far from the case. I am wondering if OTP should have a saner default:

2> inet:getopts(Sock, [recbuf, buffer]).
{ok,[{recbuf,131072},{buffer,1460}]}

11> socket:getopt(Sock, {otp, rcvbuf}).
{ok,8192}
12> socket:getopt(Sock, {socket, rcvbuf}).
{ok,131072}

Describe the solution you'd like
A better default.

1460 is definitely too little. Even 8192 is very conservative, but at least it's more reasonable.

If there's no easy way to determine a good buffer size directly from OTP then I would suggest increasing the default for inet sockets to 8192. This will help avoid the throughput being abysmal. In a Cowboy benchmark where high throughput is expected I get roughly the following numbers:

• 1460 buffer: ~100 reqs/s
• 8192 buffer: ~500 reqs/s
• 131072 buffer: ~1800 reqs/s

So the benefit is clearly apparent even for a small increase to 8192.

Describe alternatives you've considered
Each application can and many do set the value. RabbitMQ for example automatically sets the buffer size to the largest size value between sndbuf (not sure why), recbuf and buffer.

But doing it dynamically implies an extra setopts call for every incoming connection so that is not ideal.

Configuring it in listen/connect calls is better but in that case it becomes more difficult to know what the ideal value is, at least for applications that may run on all kinds of different environments (RabbitMQ, Cowboy...).

[jhogberg]

Thanks for pointing this out!

There's a memory/speed trade-off here. #8229 was caused by sub-binaries taken out of these buffers counting as references to the whole buffer -- which they should since they really do keep that whole buffer alive.

We've got plans to implement binary compaction in the GC that will remedy this to an extent, but that's at least two releases away.

[RaimoNiskanen]

The default has been 1460 since the dawn of Erlang. Applications can know better and set a value. In your case it seems to be easy to set it to 8192 for all listen/connect calls.

If you set recbuf or sndbuf; buffer is set to the same, if it is smaller and not set explicitly. This is an echo of the TCP recommendation, but the same happens for UDP, where it maybe is not what you want.

I don't know if it would hurt many to change the default to 8192, but one day it may be just as bad as 1460, so it is not obvious that it is worth doing the change.

[essen]

There's a memory/speed trade-off here.

Definitely. But I'm not convinced that 1460 uses less memory than 8192 for example, except for the cases where whole network messages fit within 1460 bytes of memory. As soon as you need to receive more than that you are going to concatenate or accumulate binaries anyway, therefore allocate more memory. For example, often a single HTTP header (cookie for example) is larger than 1460 bytes, let alone all headers of an HTTP message, so the default of 1460 is likely harmful even on the memory front, because to parse that data I must create a binary and append repeatedly (more than if the buffer size was larger), which in turn results in allocations.

I don't know if it would hurt many to change the default to 8192, but one day it may be just as bad as 1460, so it is not obvious that it is worth doing the change.

I would say it can be increased then, too. But I think we are talking about very far into the future. We'll all be using socket by then, which does come with a better default.

If you set recbuf or sndbuf; buffer is set to the same, if it is smaller and not set explicitly.

It's unfortunately not that simple. :)

1> {ok, Sock} = gen_tcp:connect("google.com", 80, []).
{ok,#Port<0.5>}
2> inet:getopts(Sock, [sndbuf, recbuf, buffer]).
{ok,[{sndbuf,87040},{recbuf,131072},{buffer,1460}]}
3> inet:setopts(Sock, [{recbuf, 131072}]).
ok
4> inet:getopts(Sock, [sndbuf, recbuf, buffer]).
{ok,[{sndbuf,87040},{recbuf,262144},{buffer,131072}]}

I don't really know what's going on to be honest.

[jhogberg]

Definitely. But I'm not convinced that 1460 uses less memory than 8192 for example ...

Good point. :)

I don't really know what's going on to be honest.

Two rather surprising things:

1. If recbuf is passed alone, it also sets buffer.
2. Linux takes whatever you pass as SO_RCVBUF and doubles it to account for internal bookkeeping. The effective buffer size is roughly what you set, or conversely, half of what it reports.