Skip to content

Doc.Singularity.Usecases.OSG.Example

Jump to bottom
Joe Latessa edited this page May 18, 2026 · 8 revisions

Example Use-Case for Compiling JETSCAPE or X-SCAPE and Submitting jobs to the HTCondor Scheduler on the Open Science Grid (OSG) while using a Singularity Image to Satisfy the Dependencies

This use-case assumes that you already have an OSG account and have already logged in using an ssh terminal. If you don't already have an OSG account or have not already logged in, please refer to the OSG's introductory documentation: OSPool Documentation

This use-case applies to both JETSCAPE and X-SCAPE. The repository names and branch names can be adjusted as needed.

Compiling JETSCAPE or X-SCAPE on the OSG

On many HPC clusters, the execution node (where the program runs) has access to your user account's home directory. However, because the OSG is a distributed system, the execution nodes cannot directly access your home directory. It is therefore necessary to send your compiled code to an execution node along with any input files.

These first scripts jetcomp.submit and jetcomp.sh can be placed in your home directory and can be used to submit a job that will clone X-SCAPE, compile it, and return the compiled code to be used in subsequent job submissions.

jetcomp.submit

universe = vanilla
executable = jetcomp.sh

Requirements = HAS_SINGULARITY == TRUE
+SingularityImage = "/cvmfs/singularity.opensciencegrid.org/jetscape/base:stable"

# transfer X-SCAPE from home directory to the worker node
arguments = $(Cluster) $(Process)

transfer_output_files = X-SCAPE-EXE.tar.gz

OSDF_LOCATION = osdf:///ospool/apXX/data/your_osg_user_name

transfer_output_remaps = "X-SCAPE-EXE.tar.gz=$(OSDF_LOCATION)/X-SCAPE-EXE.tar.gz"

should_transfer_files = YES
when_to_transfer_output = ON_EXIT

output = job_$(Cluster)_$(Process).out
error = job_$(Cluster)_$(Process).err
log = job_$(Cluster)_$(Process).log

JobDurationCategory = "Medium"

request_memory = 8.0GB
request_disk = 8.0GB
request_cpus = 1

queue 1

  • This is the script that will be submitted to the scheduler to compile X-SCAPE.

  • The line executable = jetcomp.sh calls a script that will perform the tasks of the job. Ensure that the jetcomp.sh file is placed in your home directory along with the jetcomp.submit file.

  • The jetscape/base Docker image, which provides the dependencies needed for X-SCAPE, has already been associated with the OSG and is referenced in the above script.

  • This job's output is a compiled tar archive of the X-SCAPE code, which is specified at the line: transfer_output_files = X-SCAPE-EXE.tar.gz

  • The line OSDF_LOCATION = osdf:///ospool/apXX/data/your_osg_user_name defines the location of your "public" folder. The OSG requires large files (over 1GB) to be placed in your public folder instead of your private home directory. More details about this can be found here.

    When replacing OSDF files, updated files with the same name might not propogate immediately, and worker nodes may therefore run cached or outdated versions. It's best to use different file names when replacing files such replacing EXAMPLE_v1.tar.gz with EXAMPLE_v2.tar.gz.

    On the above line, change your_osg_user_name to your OSG user name, and change XX to the specific AP number that the OSG assigned to your login node. You can move to your OSDF directory with cd /ospool/apXX/data/your_osg_user_name/ and return to your home directory with cd /home/your_osg_user_name. Preset environment variables $HOME and $DATA may also be used to switch between the home directory and the OSDF directory.

  • The line transfer_output_remaps = "X-SCAPE-EXE.tar.gz=$(OSDF_LOCATION)/X-SCAPE-EXE.tar.gz" remaps the output file from being placed in your home directory to being placed in the public OSDF space. If you expect your output file to be under 1GB, you can omit this transfer_output_remaps line and the file will be placed in your home directory.

    Note that the OSDF is publicly accessible. See the OSG's documentation here for other important details.

  • JobDurationCategory = "Medium" specifies jobs expected to complete in under 10 hours. Use "Long" for jobs expected to complete in under 20 hours. See more details about job duration here.

  • queue 1 specifies to submit one instance of this job.

  • Submit this job to the scheduler from your home directory using this command: condor_submit jetcomp.submit

  • Check the status of your job with the command: condor_q

jetcomp.sh

#!/bin/bash

# clone the repository
echo "Cloning the X-SCAPE repository"
git clone https://github.com/JETSCAPE/X-SCAPE.git
# checkout the desired branch
cd ${_CONDOR_SCRATCH_DIR}/X-SCAPE
git checkout main
echo "X-SCAPE repository cloned and main branch checked out"

# build the external packages
echo "Building the external packages"
cd ${_CONDOR_SCRATCH_DIR}/X-SCAPE/external_packages
./get_music.sh
./get_iSS.sh
./get_freestream-milne.sh
./get_smash.sh
./get_3dglauber.sh
echo "External packages built"

# build the X-SCAPE code
echo "Building the X-SCAPE code"
cd ${_CONDOR_SCRATCH_DIR}/X-SCAPE
if [ -d "build" ]; then
  rm -rf "build"
fi
mkdir build
cd build
export SMASH_DIR="${_CONDOR_SCRATCH_DIR}/X-SCAPE/external_packages/smash/smash_code"
cmake .. -DUSE_MUSIC=ON -DUSE_ISS=ON -DUSE_FREESTREAM=ON -DUSE_SMASH=ON -DUSE_3DGlauber=ON
make -j$(nproc)
echo "X-SCAPE code built"

# create an archive of the X-SCAPE code (built on the execute node)
echo "Creating an archive of the X-SCAPE code"
cd ${_CONDOR_SCRATCH_DIR}
tar -czf X-SCAPE-EXE.tar.gz X-SCAPE
echo "Archive of the X-SCAPE code created"
  • This is the bash script that was called from the jetcomp.submit submission file. The work of this script is to clone X-SCAPE, check out a desired branch, download the external packages, build X-SCAPE, and create a tar archive of the compiled code to be returned at the end of the job.
  • This script can be modified to clone X-SCAPE or JETSCAPE or to checkout a different repository branch.

Submitting a JETSCAPE or X-SCAPE Job on the OSG

Now that the X-SCAPE code has been compiled and placed in a tar archive, that tar archive can be passed as input to subsequent jobs.

jetjob.submit

universe = vanilla
executable = jetjob.sh

Requirements = HAS_SINGULARITY == TRUE
+SingularityImage = "/cvmfs/singularity.opensciencegrid.org/jetscape/base:stable"

# transfer JETSCAPE from home directory to the worker node
OSDF_LOCATION = osdf:///ospool/apXX/data/your_osg_user_name

transfer_input_files = $(OSDF_LOCATION)/X-SCAPE-EXE.tar.gz
arguments = $(Cluster) $(Process)

should_transfer_files = YES
when_to_transfer_output = ON_EXIT

output = $(ClusterId)_$(ProcId)/output.txt
error = job_$(Cluster)_$(Process).err
log = job_$(Cluster)_$(Process).log

# Produce a per-job archive filename so multiple queued jobs don't conflict
transfer_output_files = test_out_$(ClusterId)_$(ProcId).tar.gz
transfer_output_remaps = "test_out_$(ClusterId)_$(ProcId).tar.gz = $(ClusterId)_$(ProcId)/test_out.tar.gz"

JobDurationCategory = "Medium"

request_memory = 8.0GB
request_disk = 8.0GB
request_cpus = 1

queue 2

  • This is the script that will be submitted to the scheduler to run X-SCAPE.

  • Be sure to update the OSDF_LOCATION variable just as it was done in jetcomp.submit.

  • The transfer_input_files line assumes that the tar archive of the compiled X-SCAPE code is in your OSDF space. If the tar archive of the compiled code is instead in your home directory (because it is under 1GB), you may omit the OSDF path.

  • The output, error, and log files are named using the job id variables, so multiple jobs won't overwrite files with the same name.

  • Because this example doesn't expect the archived output files to be greater than 1GB, OSDF is not specified in the transfer_output_remaps.

  • queue 2 means that two instances of this submission will run with the same input parameters. If you need each submission to have distinct input parameters, the OSG describes various approaches here.

  • Submit this job to the scheduler from your home directory using this command: condor_submit jetjob.submit

  • Check the status of your job with the command: condor_q

jetjob.sh

#!/bin/bash

# set the arguments from the submit file
cluster_id=$1
proc_id=$2

# extract tar archive
echo "Extracting the X-SCAPE code..."
cd ${_CONDOR_SCRATCH_DIR}
mv X-SCAPE-EXE.tar.gz X-SCAPE.tar.gz
tar -xzf X-SCAPE.tar.gz
rm X-SCAPE.tar.gz
echo "X-SCAPE code extracted"

# create a directory for the output
OUT_DIR="${_CONDOR_SCRATCH_DIR}/test_out_${cluster_id}_${proc_id}"
echo "Creating output directory ${OUT_DIR}..."
mkdir -p "${OUT_DIR}"
echo "Output directory created"

# set the environment variables
export SMASH_DIR="${_CONDOR_SCRATCH_DIR}/X-SCAPE/external_packages/smash/smash_code"
export JETSCAPE_DIR="${_CONDOR_SCRATCH_DIR}/X-SCAPE"

# if using certain external modules, directories that include shared library
# files may need to be added to LD_LIBRARY_PATH. Adjust the find command for
# either JETSCAPE or X-SCAPE, depending on which codebase you're using.
# You may comment out these lines if not using external modules.
ALL_LIB_PATHS=$(find "${_CONDOR_SCRATCH_DIR}/X-SCAPE" -name "*.so" -exec dirname {} \; | sort -u | tr '\n' ':')
export LD_LIBRARY_PATH="${ALL_LIB_PATHS}${LD_LIBRARY_PATH}"

# Checkout Default-tunes Repository (for PP tune)
cd ${_CONDOR_SCRATCH_DIR}
echo "Cloning the Default-tunes repository..."
git clone https://github.com/JETSCAPE/Default-tunes.git

# run the job
echo "Running the job..."
cd ${_CONDOR_SCRATCH_DIR}/X-SCAPE/build
./runJetscape ${_CONDOR_SCRATCH_DIR}/Default-tunes/publications_config/arXiv_1910.05481/jetscape_user_PP_1910.05481.xml
echo "Job completed"

# move any files beginning with test_out into the per-job output directory
echo "Moving output files into ${OUT_DIR}..."
mv ${_CONDOR_SCRATCH_DIR}/X-SCAPE/build/test_out* "${OUT_DIR}/" 2>/dev/null || true
echo "Output files moved"

# archive the per-job output directory using the cluster/proc ids
echo "Archiving the output directory..."
cd ${_CONDOR_SCRATCH_DIR}
ARCHIVE_NAME="test_out_${cluster_id}_${proc_id}.tar.gz"
tar -czf "${ARCHIVE_NAME}" "$(basename "${OUT_DIR}")"
echo "Output directory archived as ${ARCHIVE_NAME}"

# clean up
echo "Cleaning up..."
rm -rf X-SCAPE
echo "Clean up complete"

  • This is the bash script that was called from the jetjob.submit submission file. The work of this script is to extract the compiled X-SCAPE code, clone the tunes repository which includes various input configurations, run X-SCAPE, and package the desired output files in a tar archive to be returned to the user at the end of the job.

  • Below we see the output files test_out_final_state_hadrons.dat and test_out_final_state_partons.dat returned to the home directory after the run successfully finishes. 16595125_0 is a unique job id for this job.

[my_osg_user_name@apXX ~]$ tar -tvf 16595125_0/test_out.tar.gz 
drwxr-xr-x osgusers/domain users 0 2024-09-08 11:46 test_out/
-rw-r--r-- osgusers/domain users 10720017 2024-09-08 11:46 test_out/test_out_final_state_hadrons.dat
-rw-r--r-- osgusers/domain users  1855471 2024-09-08 11:46 test_out/test_out_final_state_partons.dat
[my_osg_user_name@apXX ~]$

An Option to Compile a Private Branch or to Compile Code Not Yet Pushed to a Repository

If the code you wish to compile is included in a private repository or not yet pushed, clone or otherwise transfer the source code into your OSG home directory. There you can modify any of the source files before compiling.

[my_osg_user_name@apXX JETSCAPE]$ ls
activate_jetscape.csh  AUTHORS         cmakemodules  COPYING  examples           INSTALL.txt  JetScapeDoxy.conf  README.md
activate_jetscape.sh   CMakeLists.txt  config        docker   external_packages  jail         README_LINUX.md    src

The jetcomp_alt.submit and jetcomp_alt.sh scripts (shown below) compile code that has been amended locally in the home directory.

Before submitting the compile job, create a tar archive of the JETSCAPE or X-SCAPE folder that you wish to compile. If the archive is under 1GB, it can be submitted from the home directory.

tar -czf JETSCAPE-ALT.tar.gz JETSCAPE

jetcomp-alt.submit

universe = vanilla
executable = jetcomp-alt.sh

Requirements = HAS_SINGULARITY == TRUE
+SingularityImage = "/cvmfs/singularity.opensciencegrid.org/jetscape/base:stable"

# transfer JETSCAPE or X-SCAPE from home directory to the worker node
transfer_input_files = X-SCAPE-ALT.tar.gz
arguments = $(Cluster) $(Process)

transfer_output_files = X-SCAPE-ALT-EXE.tar.gz

should_transfer_files = YES
when_to_transfer_output = ON_EXIT

output = job_$(Cluster)_$(Process).out
error = job_$(Cluster)_$(Process).err
log = job_$(Cluster)_$(Process).log

JobDurationCategory = "Medium"

request_memory = 8.0GB
request_disk = 8.0GB
request_cpus = 1

queue 1

  • The above jetcomp-alt.submit script passes the tar archive JETSCAPE-ALT.tar.gz to an execution node for compilation.

  • Because X-SCAPE-ALT.tar.gz is under 1GB, it can be passed from the home directory.

  • The compiled code, also expected to be under 1GB, is returned to the home directory as X-SCAPE-ALT-EXE.tar.gz. This archive can be passed to other job submissions to run X-SCAPE.

jetcomp-alt.sh

#!/bin/bash

# extract tar archive
mv X-SCAPE-ALT.tar.gz X-SCAPE.tar.gz
tar -xzf X-SCAPE.tar.gz
rm X-SCAPE.tar.gz

# build the X-SCAPE code
cd ${_CONDOR_SCRATCH_DIR}/X-SCAPE
# if the build directory exists, remove it
if [ -d "build" ]; then
  rm -rf "build"
fi
mkdir build
cd build
cmake ..
make

# create an archive of the X-SCAPE directory (built on the execute node)
cd ${_CONDOR_SCRATCH_DIR}
tar -czf X-SCAPE-ALT-EXE.tar.gz X-SCAPE

  • With jetcomp-alt.submit, jetcomp-alt.sh and X-SCAPE-ALT.tar.gz in your home directory, the compile job can be submitted with the command: condor_submit jetcomp-alt.submit and monitored with condor_q.

  • X-SCAPE-ALT-EXE.tar.gz, which contains the compiled code, is returned to your home directory at the end of the job.

  • jetcomp-alt.sh doesn't include any of the external modules. If your code depends on the external modules, see the above scripts from earlier in this document and ensure that environment variables and shared libraries are properly set.

Clone this wiki locally