Gateway failure on cgroup v2: openat2 /sys/fs/cgroup/kubepods/pids.max: no such file or directory

[syseeker]

Ran the following in brev instance and got the same issue as running "ubuntu@brev-a8qqtwtmt:~$ curl -fsSL https://nvidia.com/nemoclaw.sh | bash"

ubuntu@brev-a8qqtwtmt:~$ source ~/.bashrc
ubuntu@brev-a8qqtwtmt:~$ git clone https://github.com/NVIDIA/NemoClaw.git
Cloning into 'NemoClaw'...
remote: Enumerating objects: 1785, done.
remote: Counting objects: 100% (283/283), done.
remote: Compressing objects: 100% (52/52), done.
remote: Total 1785 (delta 247), reused 235 (delta 231), pack-reused 1502 (from 2)
Receiving objects: 100% (1785/1785), 898.09 KiB | 5.65 MiB/s, done.
Resolving deltas: 100% (1051/1051), done.
ubuntu@brev-a8qqtwtmt:~$ ls
NemoClaw
ubuntu@brev-a8qqtwtmt:~$ cd NemoClaw
ubuntu@brev-a8qqtwtmt:~/NemoClaw$ ls
CONTRIBUTING.md  LICENSE   README.md    bin   install.sh  nemoclaw-blueprint  package.json    scripts           test          uv.lock
Dockerfile       Makefile  SECURITY.md  docs  nemoclaw    package-lock.json   pyproject.toml  spark-install.md  uninstall.sh
ubuntu@brev-a8qqtwtmt:~/NemoClaw$ ./install.sh 
[INFO]  === NemoClaw Installer ===
[INFO]  Node.js found: v22.22.1
[INFO]  Runtime OK: Node.js v22.22.1, npm 10.9.4
[INFO]  NemoClaw package.json found in current directory — installing from source…
npm warn deprecated node-domexception@1.0.0: Use your platform's native DOMException instead
npm warn deprecated glob@10.5.0: Old versions of glob are not supported, and contain widely publicized security vulnerabilities, which have been fixed in the current version. Please update. Support for old versions may be purchased (at exorbitant rates) by contacting i@izs.me

added 675 packages, and audited 676 packages in 19s

133 packages are looking for funding
  run `npm fund` for details

3 vulnerabilities (2 high, 1 critical)

To address all issues, run:
  npm audit fix --force

Run `npm audit` for details.

changed 1 package, and audited 3 packages in 528ms

found 0 vulnerabilities
[INFO]  Verified: nemoclaw is available at /home/ubuntu/.nvm/versions/node/v22.22.1/bin/nemoclaw
[INFO]  Running nemoclaw onboard…

  NemoClaw Onboarding
  ===================

  [1/7] Preflight checks
  ──────────────────────────────────────────────────
  ✓ Docker is running
  ✓ openshell CLI: openshell 0.0.7

  !! cgroup v2 detected but Docker is not configured for cgroupns=host.
     OpenShell's gateway runs k3s inside Docker, which will fail with:

       openat2 /sys/fs/cgroup/kubepods/pids.max: no such file or directory

     To fix, run:

       nemoclaw setup-spark

     This adds "default-cgroupns-mode": "host" to /etc/docker/daemon.json
     (preserving any existing settings) and restarts Docker.

     Detail: /etc/docker/daemon.json exists but "default-cgroupns-mode" is not set to "host"
ubuntu@brev-a8qqtwtmt:~/NemoClaw$ nemoclaw

  nemoclaw — NemoClaw CLI

  Getting Started:
    nemoclaw onboard                 Interactive setup wizard (recommended)
    nemoclaw setup                   Legacy setup (deprecated, use onboard)
    nemoclaw setup-spark             Set up on DGX Spark (fixes cgroup v2 + Docker)

  Sandbox Management:
    nemoclaw list                    List all sandboxes
    nemoclaw <name> connect          Connect to a sandbox
    nemoclaw <name> status           Show sandbox status and health
    nemoclaw <name> logs [--follow]  View sandbox logs
    nemoclaw <name> destroy          Stop NIM + delete sandbox

  Policy Presets:
    nemoclaw <name> policy-add       Add a policy preset to a sandbox
    nemoclaw <name> policy-list      List presets (● = applied)

  Deploy:
    nemoclaw deploy <instance>       Deploy to a Brev VM and start services

  Services:
    nemoclaw start                   Start services (Telegram, tunnel)
    nemoclaw stop                    Stop all services
    nemoclaw status                  Show sandbox list and service status

  Credentials are prompted on first use, then saved securely
  in ~/.nemoclaw/credentials.json (mode 600).

ubuntu@brev-a8qqtwtmt:~/NemoClaw$ openshell
OpenShell CLI tool

USAGE
  openshell <command> <subcommand> [flags]

SANDBOX COMMANDS
  sandbox:     Manage sandboxes
  forward:     Manage port forwarding to a sandbox
  logs:        View sandbox logs
  policy:      Manage sandbox policy
  provider:    Manage provider configuration

GATEWAY COMMANDS
  gateway:     Manage the gateway lifecycle
  status:      Show gateway status and information
  inference:   Manage inference configuration
  doctor:      Diagnose gateway issues

ADDITIONAL COMMANDS
  term:        Launch the OpenShell interactive TUI
  completions: Generate shell completions
  ssh-proxy:   SSH proxy (used by ProxyCommand)
  help:        Print this message or the help of the given subcommand(s)

FLAGS
  -g, --gateway <GATEWAY>
          Gateway name to operate on (resolved from stored metadata) [env: OPENSHELL_GATEWAY=]
      --gateway-endpoint <GATEWAY_ENDPOINT>
          Gateway endpoint URL (e.g. <https://gateway.example.com>). Connects directly without looking up gateway metadata [env: OPENSHELL_GATEWAY_ENDPOINT=]
  -v, --verbose...
          Increase verbosity (-v, -vv, -vvv)
  -h, --help
          Print help
  -V, --version
          Print version

EXAMPLES
  $ openshell sandbox create
  $ openshell gateway start
  $ openshell logs my-sandbox

LEARN MORE
  Use `openshell <command> --help` for more information about a command.
ubuntu@brev-a8qqtwtmt:~/NemoClaw$ ^C
ubuntu@brev-a8qqtwtmt:~/NemoClaw$ openshell term
Error:   × No active gateway.
  │ Set one with: openshell gateway select <name>
  │ Or deploy a new gateway: openshell gateway start

ubuntu@brev-a8qqtwtmt:~/NemoClaw$ nemoclaw onboard

  NemoClaw Onboarding
  ===================

  [1/7] Preflight checks
  ──────────────────────────────────────────────────
  ✓ Docker is running
  ✓ openshell CLI: openshell 0.0.7

  !! cgroup v2 detected but Docker is not configured for cgroupns=host.
     OpenShell's gateway runs k3s inside Docker, which will fail with:

       openat2 /sys/fs/cgroup/kubepods/pids.max: no such file or directory

     To fix, run:

       nemoclaw setup-spark

     This adds "default-cgroupns-mode": "host" to /etc/docker/daemon.json
     (preserving any existing settings) and restarts Docker.

     Detail: /etc/docker/daemon.json exists but "default-cgroupns-mode" is not set to "host"
ubuntu@brev-a8qqtwtmt:~/NemoClaw$ stat -fc %T /sys/fs/cgroup/
cgroup2fs

[syseeker]

Ah well, I resolve this with the following spark config. But would like to flag that this happens in brev too.

## Manual Setup (if setup-spark doesn't work)

### Fix Docker cgroup namespace

```bash
# Check if you're on cgroup v2
stat -fc %T /sys/fs/cgroup/
# Expected: cgroup2fs

# Add cgroupns=host to Docker daemon config
sudo python3 -c "
import json, os
path = '/etc/docker/daemon.json'
d = json.load(open(path)) if os.path.exists(path) else {}
d['default-cgroupns-mode'] = 'host'
json.dump(d, open(path, 'w'), indent=2)
"

# Restart Docker
sudo systemctl restart docker

Fix Docker permissions

sudo usermod -aG docker $USER
newgrp docker  # or log out and back in

[brianwtaylor]

PR #62 (merged) adds cgroup v2 preflight detection to the onboarding flow. When nemoclaw onboard detects cgroup v2 without cgroupns=host configured, it now prints a clear error message pointing users to nemoclaw setup-spark before they hit this gateway failure.

[ericksoa]

Update (2026-03-22)

This is fixed server-side in OpenShell v0.0.7+. OpenShell PR NVIDIA/OpenShell#329 (merged Mar 16) sets cgroupns_mode: host on the gateway container itself, so the Docker daemon config no longer matters.

If you're hitting this, upgrade OpenShell:

# Re-run the installer (fetches latest release)
bash scripts/install-openshell.sh

# Verify version is 0.0.7 or later
openshell --version

# Destroy the old gateway and re-onboard
nemoclaw onboard

For users on older OpenShell versions who can't upgrade immediately, PR #657 adds a preflight check that detects the cgroup v2 misconfiguration and auto-fixes it on Linux (setup-spark) and Colima (colima restart). Docker Desktop users will get instructions to update their settings manually.