Skip to content

Upgrade Nomad & Consul to LTS#2870

Draft
sitole wants to merge 4 commits into
support-more-versionsfrom
cluster-upgrade-flow
Draft

Upgrade Nomad & Consul to LTS#2870
sitole wants to merge 4 commits into
support-more-versionsfrom
cluster-upgrade-flow

Conversation

@sitole
Copy link
Copy Markdown
Member

@sitole sitole commented May 30, 2026 •
edited
Loading

The upgrade requires 3 full cluster rollouts due to a limited compatibility window between releases.
We can theoretically do the upgrade in fewer hops, but there is no official guarantee that the cluster will work with older nodes.

Releases available:

The oldest still supported Consul is 1.21.X LTS is supported until April 30, 2027 (https://developer.hashicorp.com/consul/docs/upgrade/lts).

The oldest still supported Nomad is 1.10.X LTS is supported until April 30, 2027 (https://developer.hashicorp.com/nomad/docs/release-notes).

Depends on #2868 as a prerequisite to allow customizing Nomad and Consul versions per cluster.

@cla-bot cla-bot Bot added the cla-signed label May 30, 2026
@cursor
Copy link
Copy Markdown

cursor Bot commented May 30, 2026 •
edited
Loading

PR Summary

High Risk
Rolling cluster image upgrades with mixed Nomad/Consul versions and a raw_exec cgroup behavior change can affect scheduling and workload isolation until all nodes are replaced.

Overview
This PR lands the final hop of the Nomad/Consul LTS upgrade by baking Consul 1.21.5 and Nomad 1.10.5 into the AWS and GCP cluster disk images, and aligns Packer build overrides with PACKER_* environment variables so intermediate rollout versions can be supplied without renaming ad-hoc Make variables. Nomad client config for raw_exec no longer sets no_cgroups, matching newer Nomad defaults and cgroup behavior on client nodes.

Reviewed by Cursor Bugbot for commit 56de665. Bugbot is set up for automated code reviews on this repo. Configure here.

gemini-code-assist[bot]
gemini-code-assist Bot reviewed May 30, 2026
View reviewed changes
Copy link
Copy Markdown
Contributor

@gemini-code-assist gemini-code-assist Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request updates the Packer build variables in the Makefiles for both AWS and GCP provider disk images to use prefixed environment variables, bumps the default versions of Consul to 1.19.2 and Nomad to 1.10.5, and removes the no_cgroups = true configuration from the raw_exec plugin in the Nomad run scripts. There are no review comments, and I have no additional feedback to provide.

@codecov
Copy link
Copy Markdown

codecov Bot commented May 30, 2026 •
edited
Loading

❌ 3 Tests Failed:

Tests completed Failed Passed Skipped
2697 3 2694 5
View the full list of 5 ❄️ flaky test(s) 
github.com/e2b-dev/infra/tests/integration/internal/tests/api/metrics::TestSandboxMetrics

Flake rate in main: 44.66% (Passed 730 times, Failed 589 times)

Stack Traces | 15.4s run time 
=== RUN   TestSandboxMetrics
=== PAUSE TestSandboxMetrics
=== CONT  TestSandboxMetrics
    sandbox_metrics_test.go:26: 
        	Error Trace:	.../api/metrics/sandbox_metrics_test.go:26
        	Error:      	Condition never satisfied
        	Test:       	TestSandboxMetrics
        	Messages:   	sandbox metrics not available in time
--- FAIL: TestSandboxMetrics (15.38s)

github.com/e2b-dev/infra/tests/integration/internal/tests/envd::TestCommandKillNextApp

Flake rate in main: 42.98% (Passed 731 times, Failed 551 times)

Stack Traces | 170s run time 
=== RUN   TestCommandKillNextApp
=== PAUSE TestCommandKillNextApp
=== CONT  TestCommandKillNextApp
Executing command /bin/bash in sandbox id3yuaxb49lq2g60fgar9
    process_test.go:28: Command [npx] output: event:{start:{pid:1262}}
Executing command /bin/bash in sandbox i8v4xp6sbv3cmy42qx2f7
    process_test.go:28: Command [npx] output: event:{data:{stderr:"npm"}}
    process_test.go:28: Command [npx] output: event:{data:{stderr:" "}}
    process_test.go:28: Command [npx] output: event:{data:{stderr:"WARN"}}
    process_test.go:28: Command [npx] output: event:{data:{stderr:" exec The following package was not found and will be installed: create-next-app@16.2.6\n"}}
    process_test.go:28: Command [npx] output: event:{data:{stdout:"Creating a new Next.js app in .../home/user/nextapp.\n"}}
    process_test.go:28: Command [npx] output: event:{data:{stdout:"\n"}}
    process_test.go:28: Command [npx] output: event:{data:{stdout:"Using npm.\n\nInitializing project with template: app-tw \n\n"}}
    process_test.go:28: Command [npx] output: event:{data:{stdout:"\nInstalling dependencies:\n- next\n"}}
    process_test.go:28: Command [npx] output: event:{data:{stdout:"- react\n- react-dom\n"}}
    process_test.go:28: Command [npx] output: event:{data:{stdout:"\nInstalling devDependencies:\n- @tailwindcss/postcss\n- @types/node\n- @types/react\n- @types/react-dom\n- eslint\n- eslint-config-next\n- tailwindcss\n- typescript\n\n"}}
    process_test.go:29: 
        	Error Trace:	.../tests/envd/process_test.go:29
        	Error:      	Received unexpected error:
        	            	failed to execute command npx in sandbox iaaynsw8sqjywqvu1vr2r: invalid_argument: protocol error: incomplete envelope: unexpected EOF
        	Test:       	TestCommandKillNextApp
--- FAIL: TestCommandKillNextApp (169.83s)

github.com/e2b-dev/infra/tests/integration/internal/tests/orchestrator::TestSandboxMemoryIntegrity

Flake rate in main: 57.78% (Passed 735 times, Failed 1006 times)

Stack Traces | 69s run time 
=== RUN   TestSandboxMemoryIntegrity
=== PAUSE TestSandboxMemoryIntegrity
=== CONT  TestSandboxMemoryIntegrity
    sandbox_memory_integrity_test.go:27: Build completed successfully
--- FAIL: TestSandboxMemoryIntegrity (68.98s)

github.com/e2b-dev/infra/tests/integration/internal/tests/orchestrator::TestSandboxMemoryIntegrity/tmpfs_hash

Flake rate in main: 57.90% (Passed 725 times, Failed 997 times)

Stack Traces | 198s run time 
=== RUN   TestSandboxMemoryIntegrity/tmpfs_hash
=== PAUSE TestSandboxMemoryIntegrity/tmpfs_hash
=== CONT  TestSandboxMemoryIntegrity/tmpfs_hash
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{start:{pid:1266}}
Executing command bash in sandbox ivk05vcwpsm7qwy8wr0r0 (user: root)
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{data:{stdout:"Total memory: 985 MB\nUsed memory before tmpfs mount: 190 MB\nFree memory before tmpfs mount: 794 MB\nMemory to use in integrity test (60% of free, min 64MB): 476 MB\n"}}
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{data:{stderr:"476+0 records in\n476+0 records out\n499122176 bytes (499 MB, 476 MiB) copied, 1.90549 s, 262 MB/s\n"}}
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{data:{stderr:"\tCommand being timed: \"dd if=/dev/urandom of=/mnt/testfile bs=1M count=476\"\n\tUser time (seconds): 0.00\n\tS"}}
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{data:{stderr:"ystem time (seconds)"}}
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{data:{stderr:": 1.85\n\tPerce"}}
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{data:{stderr:"nt of CPU"}}
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{data:{stderr:" this job got: 97%\n\tElapsed (wal"}}
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{data:{stderr:"l clock) "}}
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{data:{stderr:"time (h:m"}}
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{data:{stderr:"m:ss or "}}
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{data:{stderr:"m:ss): "}}
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{data:{stderr:"0:01.91\n\tAverage shared text size (kbytes): 0\n\tAverage unshared data size (kbytes): 0\n\tAverage stack size (kbytes): 0\n\tAverag"}}
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{data:{stderr:"e total size "}}
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{data:{stderr:"(kbytes)"}}
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{data:{stderr:": 0\n\tMaximum re"}}
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{data:{stderr:"sident set s"}}
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{data:{stderr:"ize (kby"}}
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{data:{stderr:"tes): 2648\n"}}
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{data:{stderr:"\tAverage"}}
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{data:{stderr:" residen"}}
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{data:{stderr:"t set size "}}
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{data:{stderr:"(kbytes):"}}
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{data:{stderr:" 0\n\tMajor (requiring I/O) page faults: 3\n\tMinor (reclaiming a frame) page faults: 343\n\tVoluntary context "}}
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{data:{stderr:"switches:"}}
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{data:{stderr:" 4\n\tInvol"}}
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{data:{stderr:"untary context"}}
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{data:{stderr:" switche"}}
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{data:{stderr:"s: 8\n\t"}}
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{data:{stderr:"Swaps: 0\n\tFile system inputs: 176\n\tFile system outputs: 0\n\tSocket messages sent: 0"}}
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{data:{stderr:"\n\tSocket messages rece"}}
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{data:{stderr:"ived: 0\n\tSig"}}
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{data:{stderr:"nals del"}}
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{data:{stderr:"ivered: "}}
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{data:{stderr:"0\n\tPage"}}
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{data:{stderr:" size (byte"}}
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{data:{stderr:"s): 4096\n"}}
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{data:{stderr:"\tExit s"}}
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{data:{stderr:"tatus: 0"}}
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{data:{stderr:"\n"}}
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{data:{stdout:"Used memory after tmpfs mount and file fill: 670 MB\n"}}
    sandbox_memory_integrity_test.go:70: Command [bash] output: event:{end:{exited:true  status:"exit status 0"}}
    sandbox_memory_integrity_test.go:70: Command [bash] completed successfully in sandbox i8rz590gtdmbxocu8ge8i
Executing command bash in sandbox i8rz590gtdmbxocu8ge8i (user: root)
    sandbox_memory_integrity_test.go:80: Command [bash] output: event:{start:{pid:1282}}
Executing command bash in sandbox io5ch9ydopmkexdz2t4yo (user: root)
    sandbox_memory_integrity_test.go:80: Command [bash] output: event:{data:{stdout:"e32282e989a3cf6793ddd39f91ac51725f30e707ed4398ca009f705d7dcf4c28\n"}}
    sandbox_memory_integrity_test.go:80: Command [bash] output: event:{end:{exited:true  status:"exit status 0"}}
    sandbox_memory_integrity_test.go:80: Command [bash] completed successfully in sandbox i8rz590gtdmbxocu8ge8i
    sandbox_memory_integrity_test.go:80: Command [bash] output: event:{start:{pid:1285}}
Executing command bash in sandbox i8rz590gtdmbxocu8ge8i (user: root)
Executing command bash in sandbox i8rz590gtdmbxocu8ge8i (user: root)
Executing command bash in sandbox i8rz590gtdmbxocu8ge8i (user: root)
Executing command bash in sandbox i8rz590gtdmbxocu8ge8i (user: root)
Executing command bash in sandbox i8rz590gtdmbxocu8ge8i (user: root)
Executing command bash in sandbox i8rz590gtdmbxocu8ge8i (user: root)
Executing command bash in sandbox i8rz590gtdmbxocu8ge8i (user: root)
Executing command bash in sandbox i8rz590gtdmbxocu8ge8i (user: root)
Executing command bash in sandbox i8rz590gtdmbxocu8ge8i (user: root)
Executing command bash in sandbox i8rz590gtdmbxocu8ge8i (user: root)
Executing command bash in sandbox i8rz590gtdmbxocu8ge8i (user: root)
Executing command bash in sandbox i8rz590gtdmbxocu8ge8i (user: root)
Executing command bash in sandbox i8rz590gtdmbxocu8ge8i (user: root)
Executing command bash in sandbox i8rz590gtdmbxocu8ge8i (user: root)
Executing command bash in sandbox i8rz590gtdmbxocu8ge8i (user: root)
Executing command bash in sandbox i8rz590gtdmbxocu8ge8i (user: root)
Executing command bash in sandbox i8rz590gtdmbxocu8ge8i (user: root)
Executing command bash in sandbox i8rz590gtdmbxocu8ge8i (user: root)
Executing command bash in sandbox i8rz590gtdmbxocu8ge8i (user: root)
Executing command bash in sandbox i8rz590gtdmbxocu8ge8i (user: root)
Executing command bash in sandbox i8rz590gtdmbxocu8ge8i (user: root)
Executing command bash in sandbox i8rz590gtdmbxocu8ge8i (user: root)
Executing command bash in sandbox i8rz590gtdmbxocu8ge8i (user: root)
Executing command bash in sandbox i8rz590gtdmbxocu8ge8i (user: root)
Executing command bash in sandbox i8rz590gtdmbxocu8ge8i (user: root)
Executing command bash in sandbox i8rz590gtdmbxocu8ge8i (user: root)
Executing command bash in sandbox i8rz590gtdmbxocu8ge8i (user: root)
Executing command bash in sandbox i8rz590gtdmbxocu8ge8i (user: root)
Executing command bash in sandbox i8rz590gtdmbxocu8ge8i (user: root)
Executing command bash in sandbox i8rz590gtdmbxocu8ge8i (user: root)
Executing command bash in sandbox i8rz590gtdmbxocu8ge8i (user: root)
Executing command bash in sandbox i8rz590gtdmbxocu8ge8i (user: root)
Executing command bash in sandbox i8rz590gtdmbxocu8ge8i (user: root)
Executing command bash in sandbox i8rz590gtdmbxocu8ge8i (user: root)
Executing command bash in sandbox i8rz590gtdmbxocu8ge8i (user: root)
Executing command bash in sandbox i8rz590gtdmbxocu8ge8i (user: root)
Executing command bash in sandbox i8rz590gtdmbxocu8ge8i (user: root)
Executing command bash in sandbox i8rz590gtdmbxocu8ge8i (user: root)
Executing command bash in sandbox i8rz590gtdmbxocu8ge8i (user: root)
Executing command bash in sandbox i8rz590gtdmbxocu8ge8i (user: root)
Executing command bash in sandbox i8rz590gtdmbxocu8ge8i (user: root)
Executing command bash in sandbox i8rz590gtdmbxocu8ge8i (user: root)
Executing command bash in sandbox i8rz590gtdmbxocu8ge8i (user: root)
Executing command bash in sandbox i8rz590gtdmbxocu8ge8i (user: root)
Executing command bash in sandbox i8rz590gtdmbxocu8ge8i (user: root)
Executing command bash in sandbox i8rz590gtdmbxocu8ge8i (user: root)
Executing command bash in sandbox i8rz590gtdmbxocu8ge8i (user: root)
Executing command bash in sandbox i8rz590gtdmbxocu8ge8i (user: root)
Executing command bash in sandbox i8rz590gtdmbxocu8ge8i (user: root)
Executing command bash in sandbox i8rz590gtdmbxocu8ge8i (user: root)
Executing command bash in sandbox i8rz590gtdmbxocu8ge8i (user: root)
Executing command bash in sandbox i8rz590gtdmbxocu8ge8i (user: root)
Executing command bash in sandbox i8rz590gtdmbxocu8ge8i (user: root)
Executing command bash in sandbox i8rz590gtdmbxocu8ge8i (user: root)
Executing command bash in sandbox i8rz590gtdmbxocu8ge8i (user: root)
Executing command bash in sandbox i8rz590gtdmbxocu8ge8i (user: root)
Executing command bash in sandbox i8rz590gtdmbxocu8ge8i (user: root)
Executing command bash in sandbox i8rz590gtdmbxocu8ge8i (user: root)
Executing command bash in sandbox i8rz590gtdmbxocu8ge8i (user: root)
Executing command bash in sandbox i8rz590gtdmbxocu8ge8i (user: root)
Executing command bash in sandbox i8rz590gtdmbxocu8ge8i (user: root)
Executing command bash in sandbox i8rz590gtdmbxocu8ge8i (user: root)
Executing command bash in sandbox i8rz590gtdmbxocu8ge8i (user: root)
Executing command bash in sandbox i8rz590gtdmbxocu8ge8i (user: root)
Executing command bash in sandbox i8rz590gtdmbxocu8ge8i (user: root)
Executing command bash in sandbox i8rz590gtdmbxocu8ge8i (user: root)
Executing command bash in sandbox i8rz590gtdmbxocu8ge8i (user: root)
Executing command bash in sandbox i8rz590gtdmbxocu8ge8i (user: root)
Executing command bash in sandbox i8rz590gtdmbxocu8ge8i (user: root)
Executing command bash in sandbox i8rz590gtdmbxocu8ge8i (user: root)
Executing command bash in sandbox i8rz590gtdmbxocu8ge8i (user: root)
Executing command bash in sandbox i8rz590gtdmbxocu8ge8i (user: root)
Executing command bash in sandbox i8rz590gtdmbxocu8ge8i (user: root)
Executing command bash in sandbox i8rz590gtdmbxocu8ge8i (user: root)
Executing command bash in sandbox i8rz590gtdmbxocu8ge8i (user: root)
Executing command bash in sandbox i8rz590gtdmbxocu8ge8i (user: root)
Executing command bash in sandbox i8rz590gtdmbxocu8ge8i (user: root)
Executing command bash in sandbox i8rz590gtdmbxocu8ge8i (user: root)
Executing command bash in sandbox i8rz590gtdmbxocu8ge8i (user: root)
Executing command bash in sandbox i8rz590gtdmbxocu8ge8i (user: root)
Executing command bash in sandbox i8rz590gtdmbxocu8ge8i (user: root)
Executing command bash in sandbox i8rz590gtdmbxocu8ge8i (user: root)
Executing command bash in sandbox i8rz590gtdmbxocu8ge8i (user: root)
    sandbox_memory_integrity_test.go:110: 
        	Error Trace:	.../tests/orchestrator/sandbox_memory_integrity_test.go:81
        	            				.../hostedtoolcache/go/1.26.3.../src/runtime/asm_amd64.s:1771
        	Error:      	Received unexpected error:
        	            	failed to execute command bash in sandbox i8rz590gtdmbxocu8ge8i: unavailable: HTTP status 502 Bad Gateway
    sandbox_memory_integrity_test.go:110: 
        	Error Trace:	.../tests/orchestrator/sandbox_memory_integrity_test.go:78
        	            				.../tests/orchestrator/sandbox_memory_integrity_test.go:110
        	Error:      	Condition never satisfied
        	Test:       	TestSandboxMemoryIntegrity/tmpfs_hash
--- FAIL: TestSandboxMemoryIntegrity/tmpfs_hash (197.94s)

github.com/e2b-dev/infra/tests/integration/internal/tests/proxies::TestSandboxAutoResumeViaProxy

Flake rate in main: 43.71% (Passed 729 times, Failed 566 times)

Stack Traces | 15.3s run time 
=== RUN   TestSandboxAutoResumeViaProxy
=== PAUSE TestSandboxAutoResumeViaProxy
=== CONT  TestSandboxAutoResumeViaProxy
Executing command apt-get in sandbox i0arwmjk17stb2m3r4kjs (user: root)
    auto_resume_test.go:97: [Status code: 502] Response body: {"sandboxId":"ik3x8eew76506g5z8926y","message":"The sandbox is running but port is not open","port":8000,"code":502}
    auto_resume_test.go:116: 
        	Error Trace:	.../tests/proxies/auto_resume_test.go:116
        	Error:      	Received unexpected error:
        	            	Get "http://localhost:3002": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
        	Test:       	TestSandboxAutoResumeViaProxy
--- FAIL: TestSandboxAutoResumeViaProxy (15.32s)

To view more test analytics, go to the Test Analytics Dashboard
📋 Got 3 mins? Take this short survey to help us improve Test Analytics.

cursor[bot]
cursor Bot reviewed May 30, 2026
View reviewed changes
Copy link
Copy Markdown

@cursor cursor Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes and found 1 potential issue.

Fix All in Cursor

❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.

Reviewed by Cursor Bugbot for commit 718e042. Configure here.

Comment thread iac/provider-aws/nomad-cluster/scripts/run-nomad.sh
plugin "raw_exec" {
config {
enabled = true
no_cgroups = true
Copy link
Copy Markdown

@cursor cursor Bot May 30, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Raw exec cgroup limits orchestrator

High Severity

Removing no_cgroups = true from the client raw_exec plugin config enables cgroup memory enforcement in Nomad 1.7+. The system orchestrator job uses raw_exec with no resources block, so Nomad’s default memory cap can OOM-kill the orchestrator after the LTS upgrade.

Additional Locations (1)
Fix in Cursor Fix in Web

Reviewed by Cursor Bugbot for commit 718e042. Configure here.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@cursor cursor[bot] cursor[bot] left review comments

@ValentaTomas ValentaTomas Awaiting requested review from ValentaTomas ValentaTomas will be requested when the pull request is marked ready for review ValentaTomas is a code owner

@jakubno jakubno Awaiting requested review from jakubno jakubno will be requested when the pull request is marked ready for review jakubno is a code owner

@dobrac dobrac Awaiting requested review from dobrac dobrac will be requested when the pull request is marked ready for review dobrac is a code owner

+1 more reviewer

@gemini-code-assist gemini-code-assist[bot] gemini-code-assist[bot] left review comments

Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

cla-signed

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@sitole