OOM error with apache mina client, thousands of threads at jdk/internal/Unsafe.park(Native Method), java/lang/Object.waitImpl(Native Method)

[PreciseAyo]

Version

2.14.0

Bug description

We are observing a large number of threads in Runnable state that never finish. These threads appear to be stuck in loops or waiting indefinitely, causing resource exhaustion over time. We are seeing excessive runnable threads in NioProcessor and SSHD Session timeout handling.

Environment:
OS: Windows Server 2022
JDK: 17.0.14+7
Apache MINA: 2.0.27
Apache MINA SSHD: 2.14.0

Actual behavior

Threads remain in Runnable state for extended periods. Application performance degrades due to thread buildup. Threads remain active indefinitely, consuming CPU and memory and eventually triggering an OOM response.

Expected behavior

Threads should terminate or block appropriately after completing their tasks.

Relevant log output

Nio:

at java/lang/Object.waitlmpl(Native Method)
at java/lang/Object.wait(Object.java:251(Compiled Code))
at java/lang/Object.wait(Object.java:219(Compiled Code))
at EDU/oswego/cs/dl/util/concurrent/Semaphore.attempt(Bytecode PC:62(Compiled Code))
at EDU/oswego/cs/dl/util/concurrent/SemaphoreControlledChannel.poll(Bytecode PC:19(Compiled Code))
at - (PsSelector.java:118(Compiled Code))
at - (PsSelector.java:208(Compiled Code))
at org/apache/mina/transport/socket/nio/NioProcessor.select(NioProcessor.java:112(Compiled Code))
at org/apache/mina/core/polling/AbstractPollingloProcessor$Processor.run(AbstractPollingloProcessor.java:616(Compiled Code)
at org/apache/mina/util/NamePreservingRunnable.run(NamePreservingRunnable.java:64)
at java/util/concurrent/ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136(Compiled Code))
at java/util/concurrent/ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635(Compiled Code))
at java/lang/Thread.run(Thread.java:853(Compiled Code))

SSHD Client:
at java/util/concurrent/ConcurrentHashMap.remove(ConcurrentHashMap.java:1107(Compiled Code))
at org/apache/sshd/common/session/helpers/AbstractSession.preProcessEncodeBuffer(AbstractSession.java:1069(Compiled Code))
at org/apache/sshd/common/session/helpers/AbstractSession.encode(AbstractSession.java:1436(Compiled Code))
at org/apache/sshd/common/session/helpers/AbstractSession.resolveOutputPacket(AbstractSession.java:1156(Compiled Code))
at org/apache/sshd/common/session/helpers/AbstractSession.doWritePacket(AbstractSession.java:1164(Compiled Code))
at org/apache/sshd/common/session/helpers/KeyExchangeMessageHandler.writePacket(KeyExchangeMessageHandler.java:244(Compiled Code))
at org/apache/sshd/common/session/helpers/AbstractSession.writePacket(AbstractSession.java:1087(Compiled Code))
at org/apache/sshd/common/session/Session.writePacket(Session.java:159(Compiled Code))
at org/apache/sshd/common/session/Session.writePacket(Session.java:145(Compiled Code))
at org/apache/sshd/common/session/helpers/SessionHelper.disconnect(SessionHelper.java:1209(Compiled Code))
at org/apache/sshd/common/session/helpers/SessionHelper.checkForTimeouts(SessionHelper.java:301(Compiled Code))
at org/apache/sshd/common/session/helpers/SessionTimeoutListener.run(SessionTimeoutListener.java:84(Compiled Code))
at java/util/concurrent/Executors$RunnableAdapter.call(Executors.java:539(Compiled Code))
at java/util/concurrent/FutureTask.runAndReset(FutureTask.java:305(Compiled Code))
at java/util/concurrent/ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305(Compiled Code))
at java/util/concurrent/ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136(Compiled Code))
at java/util/concurrent/ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635(Compiled Code))
at java/lang/Thread.run(Thread.java:853(Compiled Code))

at jdk/internal/misc/Unsafe.park(Native Method)
at java/util/concurrent/locks/LockSupport.parkNanos(LockSupport.java:252(Compiled Code))
at java/util/concurrent/locks/AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:1679(Compiled Code)
at java/util/concurrent/ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1182(Compiled Code))
at java/util/concurrent/ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:899(Compiled Code))
at java/util/concurrent/ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1062(Compiled Code))
at java/util/concurrent/ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1122(Compiled Code))
at java/util/concurrent/ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635(Compiled Code))
at java/lang/Thread.run(Thread.java:853(Compiled Code))

Other information

No response

[PreciseAyo]

Any update on this item?

[garydgregory]

@PreciseAyo
Do you have a reproducer we can test with?

[tomaswolf]

A reproducer would be great, but we'd also need a full thread dump including lock information.

Also, I notice the stack traces contain EDU/oswego/cs/dl/util/concurrent/Semaphore.attempt and a lot of Compiled code for Apache MINA sshd code. This seems to be some non-standard JVM, maybe GraalVM? Plus some experimental semaphore stuff from some university lab?

Finally, it's reported against version 2.14.0. We can only investigate and fix problems on the latest version, i.e. 2.16.0. We have no resources to maintain older releases.

Maybe related: #474.

[PreciseAyo]

There is an open ticket from April that has the same issue that we are seeing in our clients. #721

We are seeing a lot of these sshd-SshClient[*]-timer-thread-1 spin up and never get cleaned up and they throw the following stack trace:
at jdk/internal/misc/Unsafe.park(Native Method) at java/util/concurrent/locks/LockSupport.parkNanos(LockSupport.java:252(Compiled Code)) at java/util/concurrent/locks/AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:1679(Compiled Code) at java/util/concurrent/ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1182(Compiled Code)) at java/util/concurrent/ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:899(Compiled Code)) at java/util/concurrent/ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1062(Compiled Code)) at java/util/concurrent/ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1122(Compiled Code)) at java/util/concurrent/ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635(Compiled Code)) at java/lang/Thread.run(Thread.java:853(Compiled Code))