Non-Retryable requests when transactions are being used?

[ralfhergert]

We run our integration tests in gitlab in a dockerized environment, and since we started to use MarkLogic server transactions more we noticed our tests behaving increasingly flaky. We suspect the dockerized environment to induce some connection and availability issues. Here is a stacktrace of one of the issues:

com.marklogic.client.FailedRetryException: Service unavailable and maximum retry period elapsed: 0 seconds after 0 retries
	at com.marklogic.client.impl.OkHttpServices.sendRequestWithRetry(OkHttpServices.java:576)
	at com.marklogic.client.impl.OkHttpServices.headImplExec(OkHttpServices.java:1054)
	at com.marklogic.client.impl.OkHttpServices.headImpl(OkHttpServices.java:1016)
	at com.marklogic.client.impl.OkHttpServices.head(OkHttpServices.java:972)
	at com.marklogic.client.impl.DocumentManagerImpl.exists(DocumentManagerImpl.java:145)

The code leading into this trace is supposed to check, whether a document exists and looks like this:

final DocumentDescriptor documentDescriptor = manager.exists(docUri.toString(), transaction);

The transaction is non-null in the above statement. And as it turns out the OkHttpServices in line 1054:

return sendRequestWithRetry(requestBldr, (transaction == null), doHeadFunction, null);

renders a request using a transaction being "not retryable".

In a distributed system communication errors can happen, regardless of whether a transaction is being used or not. There should be a destinction being made depending on the type of error, whether it makes sense to repeat a request. The existence of a transaction should not make a difference here.

(We are currently using the marklogic-java-client in version 8.0.0 and the Markogic Server in 11.3.1)

[rjrudin]

Hi @ralfhergert - couple things here - first, I'll look into this particular usage of transactions with retry support.

Second - we've experienced the same flakiness with connections being dropped along with other IO exceptions in our own Docker / MarkLogic / Jenkins environment. Here's what we're doing to handle those since the 8.0 release:

• There's a not-yet-documented OkHttp interceptor that supports what I'd call "real" retry on IO exceptions - https://github.com/marklogic/java-client-api/blob/master/marklogic-client-api/src/main/java/com/marklogic/client/impl/okhttp/RetryIOExceptionInterceptor.java
• And here's an example of it being used in our test plumbing - https://github.com/marklogic/java-client-api/blob/master/marklogic-client-api/src/test/java/com/marklogic/client/test/Common.java#L36

In the interceptor comments, you can see mention of how the low-level retry support in the Java Client is based on HTTP response status codes. But of course, those don't exist in the event of an IO exception - which I know at least locally can happen when Docker runs out of memory for MarkLogic, causing MarkLogic to restart (I can reliably reproduce this when running all the ml-gradle tests locally). And this does seem specific to a Docker environment - e.g. we have performance tests using the Java Client and Flux that import / export large amounts of data and never encounter these errors.

The interceptor isn't documented, but it is a public class so you're welcome to try it out. It has definitely been helpful for our Java Client regression test suite.

[rjrudin]

Hey @ralfhergert - I did some research, and part of the challenge in understanding this is in knowing what "retry" means at the OkHttpServices layer, where it's specific to a 502, 503, or 504 being returned by MarkLogic. Since a MarkLogic REST API transaction requires host affinity, the retry is never attempted because the response code may indicate a problem with the host. This logic is repeated for other calls to the REST API when a transaction ID is present.

My hunch too is that based on your description, you may be encountering the same Docker-related IO exceptions that we encounter, and thus the response-code-based retry mechanism isn't going to help.

If you can, try out that not-yet-documented interceptor that I mentioned. It would be useful to know if that helps in your own Docker-based environment, as that would give us reason to make the interceptor part of the public API with documentation.

On a related note, I'm also going to improve the description of the underlying retry-based-on-response-code feature to make it more clear what its scope and intent is.

[ralfhergert]

Thank you, @rjrudin! We will have a look at the interceptor, you are mentioning.

Also it is good to know, that you are aware of the Docker-releated issues. One error I have seen (but I'm not sure whether it is the cause for the "Service unavailable" error here in this issue) stems from the docker network itself: from time to time the docker network reports an error when trying to resolve a hostname.

[rjrudin]

For what it's worth, I just switched yesterday from Docker Desktop to Rancher Desktop. I'll try running the full test suite for this project locally soon, without the retry interceptor enabled, and report back, I'll be curious if there's any difference in terms of dropped connections between Docker and Rancher.