Skip to content

Revise README for installation and usage instructions #5

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
seanmcconkey merged 5 commits into from
Apr 10, 2026
Merged

Revise README for installation and usage instructions #5

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
794c279
Revise README for installation and usage instructions
rjaduthie Apr 8, 2026
5b4bf17
Fix code indents and do another pass over the docs
rjaduthie Apr 8, 2026
0df8cfa
Fix heading levels in README
rjaduthie Apr 10, 2026
972ccbf
Minor tweaks to README
rjaduthie Apr 10, 2026
e1332eb
Fix RPM instructions
rjaduthie Apr 10, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
100 changes: 43 additions & 57 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,86 +1,72 @@
# Path Finder

A Rust implementation of the SKA path finder tool for authentication, locating & mounting data from the SKA storage system within a Slurm login host.
**pathFinder** is a tool for mounting SKA data on Slurm clusters without copying the data locally. Currently, it is provided as a single binary or an RPM installer.

## System Requirements
The tool allows the Scientist to specify which files, identified from the Science Gateway, they want to mount while keeping the files secure and owned by them. Two methods are planned: interactive and a workflow managed by the Science Gateway via prepareData.

For instructions on the setup & requirements for your HPC Server side environment see [pathFinder - Server Configuration](./SERVER-CONFIGURATION.md)
Features:

- OAuth2 device code flow authentication
- Data location lookup via Data Management API
- Site capabilities verification via Site Capabilities API
- Secure data mounting with proper permissions

## Installation

## Usage
For instructions on the requirements and setup for your HPC server environment, and installation of **pathFinder** itself, see the [SERVER-CONFIGURATION.md](./SERVER-CONFIGURATION.md) doc.

**Note**: The tool will automatically check if the file exists locally at the local RSE `/skadata`. If the file is not found locally, it will display the sites where the file is available and prompt you to ensure the data has been staged to your local site before mounting.
## Usage

### Mount Data

The `pathFinder` command is available to run on the CLI after logging into the Slurm login node. The command needs to be run as `sudo` because it is mounting data, users in the `pathfinder` group are granted `sudo` privileges to execute the `pathFinder` executable.

#### Usage

```
$ sudo pathFinder --help

A tool for finding SKA data paths for mounting purposes

Usage: pathFinder [OPTIONS] --namespace <NAMESPACE> --file-name <FILE_NAME>

Options:
--namespace <NAMESPACE> Namespace of the data
--file-name <FILE_NAME> Name of the data file
--no-login Do not use OAuth2 for authentication - use environment variables instead
--unmount Unmount previously mounted data instead of mounting
-h, --help Print help
```

#### OAUTH Authentication
The `pathFinder` command is available to run on the CLI after logging into the Slurm login node. The command needs to be run as `sudo` because it is mounting data, users in the `pathfinder` group are granted `sudo` privileges to execute the `pathFinder` executable.

Example using SKAIAM OAuth2 (required).
$ sudo pathFinder --help

```
$ sudo pathFinder --namespace daac --file-name simple_file.txt
A tool for finding SKA data paths for mounting purposes

Authenticating with OAuth2...
Cached tokens expired
Usage: pathFinder [OPTIONS] --namespace <NAMESPACE> --file-name <FILE_NAME>

ACTION REQUIRED:
Open this URL in a browser and authenticate: https://ska-iam.stfc.ac.uk/device?user_code=KNIBUH
Options:
--namespace <NAMESPACE> Namespace of the data
--file-name <FILE_NAME> Name of the data file
--no-login Do not use OAuth2 for authentication - use environment variables instead
--unmount Unmount previously mounted data instead of mounting
-h, --help Print help

Waiting for authentication (timeout: 5 minutes)...
Tokens cached for 3600 seconds
Authentication successful!
RSE Path for file 'simple_file.txt' in namespace 'daac': /daac/14/66/simple_file.txt
Mount verification successful: simple_file.txt is mounted at /home/sm2921/projects/daac/simple_file.txt
```
#### Authentication with SKA-IAM flow

#### Token Authentication
Example using SKA-IAM OAuth2:

Example with environment variables (for automation):
$ sudo pathFinder --namespace daac --file-name simple_file.txt

```
$ export DATA_MANAGEMENT_ACCESS_TOKEN="your_token_here"
$ export SITE_CAPABILITIES_ACCESS_TOKEN="your_token_here"
$ sudo pathFinder --namespace daac --file-name simple_file.txt
```
Authenticating with OAuth2...
Cached tokens expired

#### Unmounting Data
ACTION REQUIRED:
Open this URL in a browser and authenticate: https://ska-iam.stfc.ac.uk/device?user_code=KNIBUH

Example for unmounting a file.
Waiting for authentication (timeout: 5 minutes)...
Tokens cached for 3600 seconds
Authentication successful!
RSE Path for file 'simple_file.txt' in namespace 'daac': /daac/14/66/simple_file.txt
Mount verification successful: simple_file.txt is mounted at /home/sm2921/projects/daac/simple_file.txt

```
$ sudo pathFinder --namespace daac --file-name simple_file.txt --unmount
Unmounted simple_file.txt from /home/sm2921/projects/daac/simple_file.txt
```
#### Authentication using environment variables

#### RPM Package
Example with environment variables (e.g. for automation):

The binary and RPM are built and published at [pathFinder GitHub release](https://github.com/uksrc/pathFinder/releases).
export DATA_MANAGEMENT_ACCESS_TOKEN="your_token_here"
export SITE_CAPABILITIES_ACCESS_TOKEN="your_token_here"
sudo pathFinder --namespace daac --file-name simple_file.txt --no-login

Check your current release with :
### Unmount Data

```
dnf info pathFinder
```
Example for unmounting a file:

$ sudo pathFinder --namespace daac --file-name simple_file.txt --unmount
Unmounted simple_file.txt from /home/sm2921/projects/daac/simple_file.txt

## Development

Notes on development can be found in the [DEVELOPMENT.md](DEVELOPMENT.md) doc.
165 changes: 74 additions & 91 deletions SERVER-CONFIGURATION.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,103 +1,86 @@
# pathFinder - Server Configuration #
# Server Configuration

pathFinder is a tool for mounting SKA data on Slurm clusters without copying the data locally.
This document covers the prerequisites to setup on the underlying configuration on a Slurm cluster using a Ceph file-system and the installation of the pathFinder tool.

It allows the Scientist to specify which files, identified from the Science Gateway, they want to mount while keeping the files secure and owned by them.
## Pre-requisites

Two methods are planned, interactive and a workflow managed by the Science Gateway via prepareData.
The following requirements must be met:

This documentation covers the prerequisites to setup on the underlying configuration on a Slurm cluster using a Ceph file-system and the installation of the pathFinder tool.
- CRB Enabled
- RHEL EPEL (Extra Packages)
- BindFS
- Ceph Common

## Pre-requisites ##
Note: These are for Rocky 9.x releases and have not been tested on RHEL 10.x or Ubuntu.

The following requirements must be met.
## Server-Side Configuration

(Note these are for Rocky 9.x releases and have not been tested on RHEL 10.x or Ubuntu)
The configuration is only required on the Login node of your Slurm cluster, this assumes that all your user home directories are CephFS/NFS mount points.

- CRB Enabled
- RHEL EPEL (Extra Packages)
- BindFS
- Ceph Common
If you already have EPEL enabled you can skip the next 2 steps.

## Server Side Configuration ##
1. Enable CRB:

The configuration is only required on the Login node of your Slurm cluster, this assumes that all your user home directories are CephFS/NFS mount points.
crb status
crb enable

If you already have EPEL enabled you can skip the next 2 steps.
2. Install EPEL:

sudo dnf install epel-release
sudo dnf repolist

3. Configure your Ceph Keyring:

vi /etc/ceph/ceph.client.rucio_prod_ro.keyring

... and add your Access key:

[client.rucio_prod_ro]
key = ****************************

4. Create mountpoints, this MUST be owned by root with permissions of 550 & 700:

sudo mkdir /skadata /mnt/private_mounts/skadata
sudo chmod 550 /skadata
sudo chmod 700 /mnt/private_mounts/skadata

5. Add the following entries to the **/etc/fstab** file:

# Ceph mount
10.4.200.9:6789,10.4.200.13:6789,10.4.200.13:6789,10.4.200.17:6789,10.4.200.25:6789,10.4.200.26:6789:/volumes/_nogroup/a8af40e8-6412-44da-ad08-3731fdf19258/4945e5c2-aab7-4416-9b75-666f2af512d7 /skadata ceph name=rucio_prod_ro,x-systemd.device-timeout=30,x-systemd.mount-timeout=30,noatime,_netdev,ro,nodev,nosuid 0 2

# Bindfs mount
/mnt/private_mounts/skadata /skadata fuse.bindfs force-user=root,force-group=root 0 0

**Note**: The `/mnt/private_mounts` is used to hide the owner & group of the mounted file-system, so all files under `/skadata` are presented as `root root` for owner and group and hides the real owner **uid/gid** which would typically be the xrootd, Webdav & Storm user uid/gid.

6. Mount the /skadata mountpoints:

mount -a
systemctl daemon-reload

**Note**: We use bindfs here as well so all files under `/skadata` are presented as `root root` for owner and group and hides the real owner **uid/gid** which would typically be the xrootd, Webdav & Storm user uid/gid:

7. Add a sudoers file to control access to the pathfinder tool:

vi /etc/sudoers.d/pathFinder

...allow group `pathfinder` access to the `pathFinder` executable:

%pathfinder ALL = NOPASSWD: /usr/bin/pathfinder, /usr/bin/pathFinder

8. Add the patfinder group:

groupadd pathfinder

9. Add or update the local users to their corresponding group:

usermod -a -G pathfinder sm2921

## RPM Package installation

An RPM installation package is published on the [pathFinder release](https://github.com/uksrc/pathFinder/releases) page, check for the latest version before installing or upgrading.

Set the version and install the pathFinder package:

1. Enable CRB

```
crb status
crb enable
```

2. Install EPEL
```
sudo dnf install epel-release
sudo dnf repolist
```

3. Configure your Ceph Keyring

```
vi /etc/ceph/ceph.client.rucio_prod_ro.keyring
```
Add your Access key.
```
[client.rucio_prod_ro]
key = ****************************
```

4. Create mountpoints, this MUST be owned by root with permissions of 550 & 700.
```
sudo mkdir /skadata /mnt/private_mounts/skadata
sudo chmod 550 /skadata
sudo chmod 700 /mnt/private_mounts/skadata
```

5. Add the following entries to the **/etc/fstab** file. The `/mnt/private_mounts` is used to hide the owner & group of the mounted file-system, so all files under `/skadata` are presented as `root root` for owner and group and hides the real owner **uid/gid** which would typically be the xrootd, Webdav & Storm user uid/gid.
```
# Ceph mount
10.4.200.9:6789,10.4.200.13:6789,10.4.200.13:6789,10.4.200.17:6789,10.4.200.25:6789,10.4.200.26:6789:/volumes/_nogroup/a8af40e8-6412-44da-ad08-3731fdf19258/4945e5c2-aab7-4416-9b75-666f2af512d7 /skadata ceph name=rucio_prod_ro,x-systemd.device-timeout=30,x-systemd.mount-timeout=30,noatime,_netdev,ro,nodev,nosuid 0 2
# Bindfs mount
/mnt/private_mounts/skadata /skadata fuse.bindfs force-user=root,force-group=root 0 0
```

6. Mount the /skadata mountpoints.

Note that we use bindfs here as well so all files under `/skadata` are presented as `root root` for owner and group and hides the real owner **uid/gid** which would typically be the xrootd, Webdav & Storm user uid/gid.
```
mount -a
systemctl daemon-reload
```

7. Add a sudoers file to control access to the pathfinder tool.
```
vi /etc/sudoers.d/pathFinder
```
Using group `pathfinder` for group access for users you want to give access to.
```
%pathfinder ALL = NOPASSWD: /usr/bin/pathfinder, /usr/bin/pathFinder
```

8. Add the patfinder group.
```
groupadd pathfinder
```

9. Add or update the local users to their corresponding group.
```
usermod -a -G pathfinder sm2921
```

## pathFinder Package installation

The latest version is published at [pathFinder Release](https://github.com/uksrc/pathFinder/releases) check before installing or upgrading.

Set the version and install the pathFinder package.

```
VERSION=1.x.x
dnf upgrade https://github.com/uksrc/pathFinder/releases/download/v1.0.0/pathfinder-${VERSION}-1.x86_64.rpm
```
VERSION=1.x.y dnf upgrade https://github.com/uksrc/pathFinder/releases/download/v${VERSION}/pathfinder-${VERSION}-1.x86_64.rpm
Loading