Providing custom `Tracker.list` timeout

[sb8244]

Tracker.Shard uses a GenServer.call which doesn't accept any options from the caller. This call can be expensive in a large shard or one which is under heavy load.

Is it worthwhile to expose a timeout option the entire way through the function calls? My thought is to set a low timeout value when writing a listeners? function and then just assuming true if the function times out.

I'm happy to add this in but wanted to run it by the maintainers first. I could see a desire to expose this through all of the Tracker public APIs.

[michalmuskala]

The work inside the call itself is minuscule - the main part of obtaining the list is performed by the caller directly. So the only reason for timeouts there would be that the shard server is overloaded and has a very long message queue - in that case crashing callers is a very primitive back-pressure mechanism.

[sb8244]

In my use case I am willing to accept the caller timing out and then just saying "there's listeners" as a safe default. The 5 second timeout is longer than I'd want however as the timeout length directly affects time to deliver messages when the system is backed up. The timeout of the tracker for track calls will cause the client to reconnect.

Still looking at why my tracker process is backing up. I have ensured that there are no single large topics and have 8 shards which have pretty evenly distributed reduction count. However, having the ability to customize the timeout would allow me to reduce the maximum duration of message delivery in the worst case scenario of a backed up tracker shard.