TEZ-4667: Correct the test case testInMemAllocationWithJvmMaxMemory

[maheshrajus]

This test case is failing in some env when requested memory is less compared to available memory.
We need to increase the requested memory so that it will not fail.

[tez-yetus]

🎊 +1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	26m 57s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	detsecrets	0m 0s		detect-secrets was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 1 new or modified test files.
			_ master Compile Tests _
+1 💚	mvninstall	11m 56s		master passed
+1 💚	compile	0m 37s		master passed
+1 💚	checkstyle	1m 4s		master passed
+1 💚	javadoc	0m 45s		master passed
+0 🆗	spotbugs	2m 0s		tez-runtime-library in master has 241 extant spotbugs warnings.
			_ Patch Compile Tests _
+1 💚	mvninstall	0m 25s		the patch passed
+1 💚	codespell	0m 28s		No new issues.
+1 💚	compile	0m 23s		the patch passed
+1 💚	javac	0m 23s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	checkstyle	0m 12s		the patch passed
+1 💚	javadoc	0m 21s		the patch passed
+1 💚	spotbugs	1m 13s		the patch passed
			_ Other Tests _
+1 💚	unit	6m 0s		tez-runtime-library in the patch passed.
+1 💚	asflicense	0m 13s		The patch does not generate ASF License warnings.
		54m 0s

Subsystem	Report/Notes
Docker	ClientAPI=1.52 ServerAPI=1.52 base: https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-445/1/artifact/out/Dockerfile
GITHUB PR	#445
Optional Tests	dupname asflicense javac javadoc unit spotbugs checkstyle codespell detsecrets compile
uname	Linux 5a509883dafb 5.15.0-156-generic #166-Ubuntu SMP Sat Aug 9 00:02:46 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	/home/jenkins/jenkins-home/workspace/tez-multibranch_PR-445/src/.yetus/personality.sh
git revision	master / 4aa5728
Default Java	Ubuntu-21.0.8+9-Ubuntu-0ubuntu124.04.1
Test Results	https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-445/1/testReport/
Max. process+thread count	1125 (vs. ulimit of 5500)
modules	C: tez-runtime-library U: tez-runtime-library
Console output	https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-445/1/console
versions	git=2.43.0 maven=3.8.7 spotbugs=4.9.3 codespell=2.0.0
Powered by	Apache Yetus 0.15.1 https://yetus.apache.org

This message was automatically generated.

[abstractdog]

@maheshrajus: thanks for taking care of this, a few questions:

1. what's the issue that's thrown in the problematic case?
2. what was the requested/available memory in that case?
3. is it 100% sure that * 10 solves the problem in this case when + 100L failed?

[maheshrajus]

@abstractdog
The RequestSize size is coming less in test case failure scenario when compare to InMemThreshold value.

INFO [Time-limited test] impl.TestSimpleFetchedInputAllocator (TestSimpleFetchedInputAllocator.java:testInMemAllocationWithJvmMaxMemory(99)) - jvmMax: 1073741824
INFO [Time-limited test] impl.TestSimpleFetchedInputAllocator (TestSimpleFetchedInputAllocator.java:testInMemAllocationWithJvmMaxMemory(107)) - InMemThreshold: 107374184
INFO [Time-limited test] impl.SimpleFetchedInputAllocator (SimpleFetchedInputAllocator.java:(117)) - srcName: RequestedMemory=107374184, AssignedMemory=107374184, maxSingleShuffleLimit=107374184
INFO [Time-limited test] impl.TestSimpleFetchedInputAlloc¸ator (TestSimpleFetchedInputAllocator.java:testInMemAllocationWithJvmMaxMemory(115)) - RequestSize: 42949776

here RequestSize should be greater than InMemThreshold. Then only it will allocate from the DISK. So it is failing in some env only. In any case we should put more RequestSize . So currently * 10 is solving the issue.

Test sample:

    // check if requestSize is greater than maxSingleShuffleLimit
    assertTrue(requestSize > inputManager.maxSingleShuffleLimit);

    // requestSize is greater than the maxSingleShuffleLimit, so allocation is from DISK
    FetchedInput fi1 = inputManager.allocate(requestSize, compressedSize, new InputAttemptIdentifier(1, 1));
    assertEquals(FetchedInput.Type.DISK, fi1.getType());

[abstractdog]

I still feel the test case is shady in its current form, and I'm afraid this fix won't make it any clearer
having this magic:

    long requestSize = (long) (0.4f * inMemThreshold) + 100L;

requestSize should be greater than inMemThreshold, which is a lower bound, also I guess it has an upper bound constraint (jvm memory, or whatever), am I right to assume that?

could this line be clearer to comply with those constraints and work in every scenario (regardless of the jvmMax)

[maheshrajus]

@abstractdog I will make requestSize simple instead of jvm memory and "(long) (0.4f * inMemThreshold) + 100L" checks. Lower bound value already tested in existed test cases. i will put requestSize size more than inMemThreshold and test the scenario

Planning to modify like this:
long jvmMax = 1073741824L;
LOG.info("jvmMax: " + jvmMax);

float bufferPercent = 0.1f;
conf.setFloat(TezRuntimeConfiguration.TEZ_RUNTIME_SHUFFLE_FETCH_BUFFER_PERCENT, bufferPercent);
conf.setFloat(TezRuntimeConfiguration.TEZ_RUNTIME_SHUFFLE_MEMORY_LIMIT_PERCENT, 1.0f);
conf.setStrings(TezRuntimeFrameworkConfigs.LOCAL_DIRS, localDirs.getAbsolutePath());

long inMemThreshold = (long) (bufferPercent * jvmMax);
LOG.info("InMemThreshold: " + inMemThreshold);

SimpleFetchedInputAllocator inputManager = new SimpleFetchedInputAllocator(
        "srcName", UUID.randomUUID().toString(), 123, conf,
        jvmMax, inMemThreshold);

**long requestSize = inMemThreshold + 100L;**

requestSize size should be more than inMemThreshold so that it will be allocated from DISK.

[tez-yetus]

🎊 +1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 52s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	detsecrets	0m 0s		detect-secrets was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 1 new or modified test files.
			_ master Compile Tests _
+1 💚	mvninstall	12m 7s		master passed
+1 💚	compile	0m 39s		master passed
+1 💚	checkstyle	1m 4s		master passed
+1 💚	javadoc	0m 45s		master passed
+0 🆗	spotbugs	2m 0s		tez-runtime-library in master has 241 extant spotbugs warnings.
			_ Patch Compile Tests _
+1 💚	mvninstall	0m 25s		the patch passed
+1 💚	codespell	0m 28s		No new issues.
+1 💚	compile	0m 25s		the patch passed
+1 💚	javac	0m 25s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	checkstyle	0m 12s		the patch passed
+1 💚	javadoc	0m 20s		the patch passed
+1 💚	spotbugs	1m 10s		the patch passed
			_ Other Tests _
+1 💚	unit	6m 0s		tez-runtime-library in the patch passed.
+1 💚	asflicense	0m 13s		The patch does not generate ASF License warnings.
		28m 6s

Subsystem	Report/Notes
Docker	ClientAPI=1.52 ServerAPI=1.52 base: https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-445/2/artifact/out/Dockerfile
GITHUB PR	#445
Optional Tests	dupname asflicense javac javadoc unit spotbugs checkstyle codespell detsecrets compile
uname	Linux 91e2ecc61eca 5.15.0-156-generic #166-Ubuntu SMP Sat Aug 9 00:02:46 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	/home/jenkins/jenkins-home/workspace/tez-multibranch_PR-445/src/.yetus/personality.sh
git revision	master / 263c049
Default Java	Ubuntu-21.0.9+10-Ubuntu-124.04
Test Results	https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-445/2/testReport/
Max. process+thread count	1109 (vs. ulimit of 5500)
modules	C: tez-runtime-library U: tez-runtime-library
Console output	https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-445/2/console
versions	git=2.43.0 maven=3.8.7 spotbugs=4.9.3 codespell=2.0.0
Powered by	Apache Yetus 0.15.1 https://yetus.apache.org

This message was automatically generated.

[maheshrajus]

@abstractdog Can you review and approve the changes ? thanks !