spurd: user jobs fail with "Permission denied" when job script staged into a dir the executing user cannot read

[yansun1996]

What happened

Non-root batch jobs fail immediately with:

/bin/bash: /root/.spur_job_<jobid>.sh: Permission denied

The job is marked FAILED (NonZeroExitCode) and the job body never executes. The job's stdout is empty; the error only appears in the job's stderr file.

Root cause (observed)

spurd runs as root and stages the job script as .spur_job_<jobid>.sh inside the job's working directory (crates/spurd/src/executor.rs, ~line 287). The job process itself is then launched as the submitting user.

The working directory is taken from the job spec (submitter CWD, or --chdir). When that directory is writable by root but not readable/traversable by the executing user, root successfully writes the script but the user's job process cannot open it.

Two real-world directory situations both trigger this:

1. work_dir resolves to /root (mode 700). Root writes /root/.spur_job_N.sh (the file is even chmod 755), but the non-root user cannot traverse /root to reach it.
2. work_dir is an NFS path under root_squash (e.g. a managed NFS mount with root_squash). Here root is squashed to nobody and cannot create the script in the user-owned directory at all.

In executor.rs (~line 263) work_dir is accepted if create_dir_all(work_dir) succeeds for root, otherwise it falls back to /tmp. The decision is based purely on whether root can write — it never checks whether the executing user can read the resulting script.

Expected

Any user able to submit a job should have that job's script staged somewhere the executing user is guaranteed to be able to read, so the job body runs regardless of the working directory's ownership/permissions.

Repro

1. Cluster with a non-root user (uid present on the compute node), spurd running as root.

2. Submit a trivial batch job whose effective working directory is /root or an NFS root_squash path, with #SBATCH --output=/tmp/spur-%j.out and --error=/tmp/spur-%j.err (local, so the error is observable):

   #!/bin/bash
   #SBATCH --job-name=stage-repro
   #SBATCH --nodes=1
   #SBATCH --output=/tmp/spur-%j.out
   #SBATCH --error=/tmp/spur-%j.err
   echo "hello from $(id -un)"

3. Job ends FAILED; /tmp/spur-<id>.err contains /bin/bash: /root/.spur_job_<id>.sh: Permission denied.

Workaround that confirms the diagnosis: adding #SBATCH --chdir=/tmp makes the job COMPLETE, because /tmp is root-writable and world-traversable (sticky 1777), so the user can read its own staged script.

Notes

• Observed with spurd launched from a systemd unit running User=root.
• Setting HOME / WorkingDirectory on the spurd unit does not change the staging path — staging follows the job's work_dir, not spurd's environment.
• Security-adjacent caveat for whoever picks this up: the tempting admin workaround is to loosen /root (e.g. chmod 711 /root) so users can traverse it. That weakens root's home directory fleet-wide and should be avoided — the staging location should not depend on relaxing directory permissions. Also worth reviewing: root writing an executable file into a directory derived from user-controlled input (CWD/--chdir) is a pattern worth hardening against symlink/TOCTOU if a shared/world-writable work_dir is ever in play.