We are using the kestrel default for the max number of streams per connection, which is is 100
Kestrel:
Limits:
Http2:
MaxStreamsPerConnection: 100
This limits the number of concurrent gRPC calls per connection to 100. Quite a lot of our calls last indefinitely so e.g. if you have 100 catchup subscriptions, you can't start another one on the same connection.
Unfortunately, if you try, the call waits indefinitely rather than failing.
The server tells the client what the limit is, and the client respects the limit, queueing additional calls locally until a stream for it becomes available. This is handled by the grpc implementation in the client, not the client sdk. The server is aware that a connection is at the limit, but does not know if anything is queued.
In dotnet the client automatically opens another connection when the limit is reached (kurrent-io/KurrentDB-Client-Dotnet#218) but the gRPC implementation in Java (and perhaps other platforms) does not support this (grpc/grpc-java#8252)
The heartbeat mechanism does not help here because the call never starts and is therefore not expecting any heartbeats.
@ahjohannessen recently ran into this problem during a rolling upgrade using the java client. When a node shutdown, more subscriptions were moved to an existing connection to a different node, which reached the limit.
Possible courses of action
- Bump up the limit (however the sneaky limit will still be somewhere, and having very large numbers of calls sharing a connection has downsides such as a missing packet for one call will cause all calls to wait)
- Have the server advertise the limit via a separate gRPC or HTTP API so that the client can take it into account even if its grpc library doesn't expose it
- Implement some way of having the call timeout in the clients even if queued locally
- ?
Discord discussion: https://discord.com/channels/1116060392045678642/1116060392746143747/1525041095233376266
We are using the kestrel default for the max number of streams per connection, which is is 100
This limits the number of concurrent gRPC calls per connection to 100. Quite a lot of our calls last indefinitely so e.g. if you have 100 catchup subscriptions, you can't start another one on the same connection.
Unfortunately, if you try, the call waits indefinitely rather than failing.
The server tells the client what the limit is, and the client respects the limit, queueing additional calls locally until a stream for it becomes available. This is handled by the grpc implementation in the client, not the client sdk. The server is aware that a connection is at the limit, but does not know if anything is queued.
In dotnet the client automatically opens another connection when the limit is reached (kurrent-io/KurrentDB-Client-Dotnet#218) but the gRPC implementation in Java (and perhaps other platforms) does not support this (grpc/grpc-java#8252)
The heartbeat mechanism does not help here because the call never starts and is therefore not expecting any heartbeats.
@ahjohannessen recently ran into this problem during a rolling upgrade using the java client. When a node shutdown, more subscriptions were moved to an existing connection to a different node, which reached the limit.
Possible courses of action
Discord discussion: https://discord.com/channels/1116060392045678642/1116060392746143747/1525041095233376266