task: add database query retry for read queries (not only mutations)

[fatih-acar]

Component

API Server / GraphQL

Task Description

When running concurrent GraphQL queries that fetches a cardinality many relationship with a lot of peers, we can end up in a situation where we exhaust Neo4j's client thread pool; thus throwing a TransientError to the user.

We already have the retry_db_transaction decorator that handles such transient errors but it's mostly applied to mutations. We should also use it for the many_relationship_resolver and make the retries smoother (by implementing exponential backoff retries?)

[BaptisteGi]

Yep, it would also be great if we could surface that in observability. I’m not sure what we can do on the database side to monitor the thread pool, but at the very least, it would be helpful to log or emit an alert when we hit exhaustion and enter backoff retries.

[fatih-acar]

Yep, it would also be great if we could surface that in observability. I’m not sure what we can do on the database side to monitor the thread pool, but at the very least, it would be helpful to log or emit an alert when we hit exhaustion and enter backoff retries.

We already expose a prometheus metric for that infrahub_db_transaction_retries but we are probably missing a panel in our grafana dashboards.
Also retrial already emits logs when they happen.