s3cmdc is a containerized execution wrapper for the s3cmd utility, built using Apptainer. It is designed to interface with OpenStack Swift / Ceph Object Gateway (RGW) endpoints. The wrapper dynamically converts standard OpenStack Identity (Keystone) credentials into S3-compatible EC2 credentials at runtime, ensuring secure and reproducible interaction with object storage without requiring persistent configuration files.
To maintain strict isolation and prevent credential leakage, the following architectural principles are implemented:
- Environment Variable Inheritance: Apptainer natively injects the host operating system's environment variables into the container. OpenStack credentials are authenticated on the host, delegating interactive password prompts to the host's shell (e.g., Bash) and avoiding POSIX/dash interpreter limitations within the container.
- Dynamic Configuration Generation: Upon execution, the
%runscriptutilizespython-openstackclientto query the service catalog for the public S3 endpoint and generates temporary EC2 application credentials. These are written to an ephemerals3cmdconfiguration file (~/.s3cfg-apptainer). - Transparent Execution: The container script replaces its own process with the
s3cmdbinary via theexeccommand, passing all user-provided arguments ("$@"). This allows the container to operate identically to a locally installed binary.
- Apptainer installed on the host system.
- A valid OpenStack RC file (e.g.,
project-openrc.sh) containing standard Keystone authentication variables (OS_AUTH_URL,OS_PROJECT_NAME,OS_USERNAME, etc.).
The wrapper can be deployed either by retrieving a pre-compiled image directly via Apptainer or by executing a local build process. Both methods ensure environmental reproducibility.
To bypass the local build phase, the pre-compiled Apptainer Image Format (SIF) artifact is retrieved directly from the GitHub Container Registry using Apptainer's native ORAS (OCI Registry As Storage) transport.
# 1. Retrieve the image artifact and output it as the executable 's3cmdc'
apptainer pull s3cmdc oras://ghcr.io/pscicomp/s3cmdc:1.0.1
# 2. Assign execution permissions
chmod +x s3cmdc
# 3. Relocate to the system execution path
sudo mv s3cmdc /usr/local/bin/
(Note: Replace <namespace> and <repository> with the target GitHub identifiers.)
The container is built locally from a minimal python:3.13-slim base image to optimize resource consumption.
- Create the Definition File: Save the provided Apptainer definition as
s3cmdc.def. - Build the Image: Execute the build command to generate the executable.
apptainer build s3cmdc s3cmdc.def
- Deploy to Path:
chmod +x s3cmdc
sudo mv s3cmdc /usr/local/bin/
(If administrative privileges are unavailable, the binary can be placed in ~/.local/bin/.)
Execution is a two-step process requiring host-side authentication followed by standard s3cmd invocation.
Load the identity variables into the host environment. This step handles all necessary interactive prompts (e.g., passwords) required by the OpenStack Identity service.
source project-openrc.sh
Invoke s3cmdc followed by the desired operational subcommands and parameters. The container automatically processes the configuration and routes the request to the correct endpoint.
# List all containers (buckets)
s3cmdc ls
# Create a new container
s3cmdc mb s3://research-data-alpha
# Upload an object to the container
s3cmdc put dataset.csv s3://research-data-alpha/
# Retrieve metadata for a specific object
s3cmdc info s3://research-data-alpha/dataset.csv
If s3cmdc is executed without prior authentication, the execution is halted and the following error is returned to standard error (stderr):
Error: OpenStack authentication variables (OS_AUTH_URL, OS_PASSWORD, etc.) are missing.
Ensure the OpenStack RC file is sourced on the host before running s3cmdc.