Skip to content

perf(iac): tune NFS caching and RPC parallelism for persistent volumes #2871

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
tomassrnka wants to merge 2 commits into
base: main
Choose a base branch
from
+20 −2
Open

perf(iac): tune NFS caching and RPC parallelism for persistent volumes #2871

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
b27a117
perf(iac): tune NFS caching and RPC parallelism for persistent volumes
tomassrnka May 30, 2026
9f90ac9
fixup: drop `|| true` from sunrpc sysctls
tomassrnka May 30, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 7 additions & 2 deletions iac/provider-gcp/main.tf
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -81,8 +81,13 @@ locals {
"local_lock=none", // all locks are network locks

// caching
"noac", // disable attribute caching. slower, but more reliable
"lookupcache=none", // disable lookup caching
// Bound cross-host staleness to ~1s while avoiding a Filestore RPC on every
// metadata op. Sandboxes mounting the same volume on different hosts still see
// peer writes within the attribute timer; strict coherency for shared mutable
// state should use NLM locks (already enabled via lock,local_lock=none) or an
// out-of-band signal.
"actimeo=1", // cap all attribute caches at 1 second
"lookupcache=positive", // cache hits; negative lookups still hit the server so new files appear promptly

// security
"noacl", // do not use an acl
Expand Down
13 changes: 13 additions & 0 deletions iac/provider-gcp/nomad-cluster/scripts/start-client.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -83,6 +83,19 @@ echo "$SWAPFILE none swap sw 0 0" | tee -a /etc/fstab
sysctl vm.swappiness=10
sysctl vm.vfs_cache_pressure=50

# Increase RPC slot table so nconnect can actually parallelize. Default of 2 in-flight
# RPCs per TCP connection bottlenecks metadata-heavy NFS workloads (uncached lookups,
# many concurrent sandboxes). Modprobe options apply when sunrpc loads (first NFS mount);
# the runtime sysctl handles the case where the module is already loaded.
mkdir -p /etc/modprobe.d
cat <<'EOF' >/etc/modprobe.d/sunrpc.conf
options sunrpc tcp_slot_table_entries=128 tcp_max_slot_table_entries=128
EOF
if [ -d /proc/sys/sunrpc ]; then
sysctl -w sunrpc.tcp_slot_table_entries=128
sysctl -w sunrpc.tcp_max_slot_table_entries=128
fi
Comment on lines +94 to +97

This comment was marked as low quality.

Sign in to view

Copy link
Copy Markdown
Member Author

@tomassrnka tomassrnka May 30, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Gemini is mistaken here. On modern Linux kernels these are exposed via both interfaces — they read/write the same underlying variables in net/sunrpc/xprtsock.c:

  • module parameters: /sys/module/sunrpc/parameters/tcp_{slot,max_slot}_table_entries
  • sysctls: /proc/sys/sunrpc/tcp_{slot,max_slot}_table_entries (i.e. sysctl sunrpc.tcp_slot_table_entries)

The project's own benchmark already uses the sysctl path successfully — see packages/orchestrator/cmd/simulate-nfs-traffic/experiments.go:72-75, which shells out to sysctl -w sunrpc.tcp_slot_table_entries=128 via setSysFs. That validates the path works on the kernels we run.

Also: the suggested replacement reintroduces || true, which was explicitly removed in the second commit on this branch — the [ -d ... ] guard already covers the "module not loaded" case, and silently swallowing errors past it would hide real misconfiguration.

Keeping the current implementation.


# TODO: Optimize the mount more according to https://cloud.google.com/filestore/docs/mounting-fileshares
%{ if USE_FILESTORE_CACHE }
# Configure NFS read ahead
Expand Down
Loading