Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
28 changes: 28 additions & 0 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -23,3 +23,31 @@ jobs:

- name: Run tests
run: cargo test

integration-test:
runs-on: ubuntu-latest
# rust:latest runs as root by default, which is required for chown to
# reassign ownership to the test user created during the integration tests.
container:
image: rust:latest

steps:
- uses: actions/checkout@v6

- name: Install test dependencies
run: |
apt-get update
apt-get install -y --no-install-recommends passwd
rm -rf /var/lib/apt/lists/*

- name: Cache dependencies
uses: actions/cache@v5
with:
path: |
~/.cargo/registry
~/.cargo/git
target
key: ${{ runner.os }}-cargo-integration-${{ hashFiles('**/Cargo.lock') }}

- name: Run integration tests
run: cargo test -- --include-ignored
8 changes: 7 additions & 1 deletion CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,13 @@ All notable changes to this project will be documented in this file.
The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).

## [Unreleased]

### Fixed

- Mounting two or more files no longer causes error when settting permissions.
- Fixed documentation when the `--help` flag is set.

## v1.0.1 (2026-04-01)

### Added
Expand All @@ -22,4 +29,3 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
### Added

- Initial Rust implementation

13 changes: 12 additions & 1 deletion Cargo.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

3 changes: 2 additions & 1 deletion Cargo.toml
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
[package]
name = "pathFinder"
version = "1.0.1"
version = "1.0.2-alpha.0"
edition = "2021"

[[bin]]
Expand All @@ -18,6 +18,7 @@ tokio = { version = "1.40", features = ["full"] }
regex = "1.10"
dirs = "5.0"
libc = "0.2"
users = "0.11.0"

[dev-dependencies]
httpmock = "0.7"
Expand Down
63 changes: 51 additions & 12 deletions DEVELOPMENT.md
Original file line number Diff line number Diff line change
@@ -1,18 +1,57 @@
# Architecture
# Development Guide

There are CI pipelines in GitHub for testing and building the executable, plus publishing an RPM on a release.

## Features
For manual testing there here are some hints:

- OAuth2 device code flow authentication
- Data location lookup via Data Management API
- Site capabilities verification via Site Capabilities API
- Secure data mounting with proper permissions
## Prerequisites

- [Rust toolchain](https://rustup.rs/) (stable)
- Docker (for running integration tests locally)

### Modules
## Building

- **main.rs** - Main path finder CLI logic
- **api_client.rs** - HTTP client for Data Management and Site Capabilities APIs
- **oauth2_auth.rs** - OAuth2 device code flow implementation with token caching
- **models.rs** - Data structures for API responses (sites, nodes, storage areas, data locations)
- **mount.rs** - Mount/unmount utility for data access
cargo build

For a release build:

cargo build --release

## Running the binary locally

cargo run -- --namespace daac --file-name pi24_test_run_1_cleaned.fits

## Testing

### Unit tests

Run the fast unit test suite (no root required, all I/O mocked):

cargo test

### Integration tests

The integration tests exercise real `chown` behaviour — they create a temporary system user (`pf_testuser`), call the actual `chown` binary, and verify ownership changes on disk. They require root on Linux.

#### Locally via Docker (recommended before pushing)

docker build -f Dockerfile.test -t pf-test .
docker run --rm pf-test

This builds a `rust:latest` container (runs as root) and executes `cargo test -- --include-ignored`.

#### Directly (if already root on a Linux machine)

cargo test -- --include-ignored

Integration tests are skipped automatically when not running as root, so it is safe to run this command in any environment.

#### In CI

The `integration-test` job in `.github/workflows/ci.yml` runs `cargo test -- --include-ignored` inside a `rust:latest` container, which provides root automatically. It runs in parallel with the regular unit test job on every push and pull request.

### Cleaning up after an interrupted integration test

If a test run is interrupted before the `pf_testuser` system user is removed, subsequent runs will panic with a clear message. Remove the leftover user with:

sudo userdel pf_testuser
28 changes: 28 additions & 0 deletions Dockerfile.test
Original file line number Diff line number Diff line change
@@ -0,0 +1,28 @@
# Integration test image.
#
# Runs as root (the default for this base image), which is required for `chown`
# to reassign ownership to the test user created during the tests.
#
# Build and run:
# docker build -f Dockerfile.test -t pf-test .
# docker run --rm pf-test
FROM rust:latest

# passwd provides useradd/userdel, needed by integration tests to create/remove
# the pf_testuser system account. Included by default in most distros but made
# explicit here for reproducibility.
RUN apt-get update \
&& apt-get install -y --no-install-recommends passwd \
&& rm -rf /var/lib/apt/lists/*

WORKDIR /app

# Fetch dependencies before copying source so this layer is cached independently.
COPY Cargo.toml Cargo.lock ./
RUN mkdir src && echo 'fn main() {}' > src/main.rs \
&& cargo fetch \
&& rm -rf src

COPY src src

CMD ["cargo", "test", "--", "--include-ignored"]
99 changes: 43 additions & 56 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,86 +1,73 @@
# Path Finder

A Rust implementation of the SKA path finder tool for authentication, locating & mounting data from the SKA storage system within a Slurm login host.
**pathFinder** is a tool for mounting SKA data on Slurm clusters without copying the data locally. Currently, it is provided as a single binary or an RPM installer.

## System Requirements
The tool allows the Scientist to specify which files, identified from the Science Gateway, they want to mount while keeping the files secure and owned by them. Two methods are planned: interactive and a workflow managed by the Science Gateway via prepareData.

For instructions on the setup & requirements for your HPC Server side environment see [pathFinder - Server Configuration](./SERVER-CONFIGURATION.md)
Features:

- OAuth2 device code flow authentication
- Data location lookup via Data Management API
- Site capabilities verification via Site Capabilities API
- Secure data mounting with proper permissions

## Installation

## Usage
For instructions on the requirements and setup for your HPC server environment, and installation of **pathFinder** itself, see the [SERVER-CONFIGURATION.md](./SERVER-CONFIGURATION.md) doc.

**Note**: The tool will automatically check if the file exists locally at the local RSE `/skadata`. If the file is not found locally, it will display the sites where the file is available and prompt you to ensure the data has been staged to your local site before mounting.
## Usage

### Mount Data

The `pathFinder` command is available to run on the CLI after logging into the Slurm login node. The command needs to be run as `sudo` because it is mounting data, users in the `pathfinder` group are granted `sudo` privileges to execute the `pathFinder` executable.

#### Usage

```
$ sudo pathFinder --help

A tool for finding SKA data paths for mounting purposes

Usage: pathFinder [OPTIONS] --namespace <NAMESPACE> --file-name <FILE_NAME>

Options:
--namespace <NAMESPACE> Namespace of the data
--file-name <FILE_NAME> Name of the data file
--no-login Do not use OAuth2 for authentication - use environment variables instead
--unmount Unmount previously mounted data instead of mounting
-h, --help Print help
```
The `pathFinder` command is available to run on the CLI after logging into the Slurm login node. The command needs to be run as `sudo` because it is mounting data, users in the `pathfinder` group are granted `sudo` privileges to execute the `pathFinder` executable.

#### OAUTH Authentication
$ sudo pathFinder --help

Example using SKAIAM OAuth2 (required).
A tool for finding SKA data paths for mounting purposes

```
$ sudo pathFinder --namespace daac --file-name simple_file.txt
Usage: pathFinder [OPTIONS] --namespace <NAMESPACE> --file-name <FILE_NAME>

Authenticating with OAuth2...
Cached tokens expired
Options:
--namespace <NAMESPACE> Namespace of the data
--file-name <FILE_NAME> Name of the data file
--no-login Do not use OAuth2 for authentication - use environment variables instead
--unmount Unmount previously mounted data instead of mounting
-h, --help Print help

ACTION REQUIRED:
Open this URL in a browser and authenticate: https://ska-iam.stfc.ac.uk/device?user_code=KNIBUH
#### Authentication with SKA-IAM flow

Waiting for authentication (timeout: 5 minutes)...
Tokens cached for 3600 seconds
Authentication successful!
RSE Path for file 'simple_file.txt' in namespace 'daac': /daac/14/66/simple_file.txt
Mount verification successful: simple_file.txt is mounted at /home/sm2921/projects/daac/simple_file.txt
```
Example using SKA-IAM OAuth2:

#### Token Authentication
$ sudo pathFinder --namespace daac --file-name simple_file.txt

Example with environment variables (for automation):
Authenticating with OAuth2...
Cached tokens expired

```
$ export DATA_MANAGEMENT_ACCESS_TOKEN="your_token_here"
$ export SITE_CAPABILITIES_ACCESS_TOKEN="your_token_here"
$ sudo pathFinder --namespace daac --file-name simple_file.txt
```
ACTION REQUIRED:
Open this URL in a browser and authenticate: https://ska-iam.stfc.ac.uk/device?user_code=KNIBUH

#### Unmounting Data
Waiting for authentication (timeout: 5 minutes)...
Tokens cached for 3600 seconds
Authentication successful!
RSE Path for file 'simple_file.txt' in namespace 'daac': /daac/14/66/simple_file.txt
Mount verification successful: simple_file.txt is mounted at /home/sm2921/projects/daac/simple_file.txt

Example for unmounting a file.
#### Authentication using environment variables

```
$ sudo pathFinder --namespace daac --file-name simple_file.txt --unmount
Unmounted simple_file.txt from /home/sm2921/projects/daac/simple_file.txt
```
Example with environment variables (e.g. for automation):

#### RPM Package
export DATA_MANAGEMENT_ACCESS_TOKEN="your_token_here"
export SITE_CAPABILITIES_ACCESS_TOKEN="your_token_here"
sudo pathFinder --namespace daac --file-name simple_file.txt --no-login

The binary and RPM are built and published at [pathFinder GitHub release](https://github.com/uksrc/pathFinder/releases).
### Unmount Data

Check your current release with :
Example for unmounting a file:

```
dnf info pathFinder
```
$ sudo pathFinder --namespace daac --file-name simple_file.txt --unmount
Unmounted simple_file.txt from /home/sm2921/projects/daac/simple_file.txt

## Development

Notes on how to build the executable, run the unit and integration tests, and local Docker-based testing can be found in the development can be found in the [DEVELOPMENT.md](DEVELOPMENT.md) doc.

Loading
Loading