2026 CMS PbPb Run - Foresting

README last updated: 19 May 2026

• 1) Setup for PbPb
• 2) Processing Forests
• 3) Quick Reference
  • CMSSW
  • CRAB
  • Updating "forest_CMSSW_16_X_X"
  • Updating ZDC emap
  • VOMS Certificate Setup

---

1) Setup for PbPb

1.1) Install CMSSW

cmsrel CMSSW_16_1_1
cd CMSSW_16_1_1/src
cmsenv

1.2) Add CMS Heavy Ion foresting tools

git cms-merge-topic CmsHI:forest_CMSSW_16_1_X
scram build -j4

Tip

You can add CMSHI as a remote git reference in case of updates:

git remote add cmshi git@github.com:CmsHI/cmssw.git

1.3) Clone this repository and add your remote repo

On github.com, fork this repository to make your own version. This will be used to document your forest configs.

Next, clone your forked version of this repo:

git clone git@github.com:<your_git_username>/HiForestSetupPbPbRun2026.git
cd HiForestSetupPbPbRun2026/

Finally, add the original repo as an "upstream" repo:

git remote add upstream git@github.com:jdlang/HiForestSetupPbPbRun2026.git
git fetch upstream
git pull upstream main

---

2) Processing Forests

2.1) Edit CRABConfig settings

Make a copy of the CRABConfig file with an appropriate name:

cp forest_CRABConfig_Run3_PbPb_DATA_TEMPLATE.py forest_CRABConfig_Run3_PbPb_DATA_<your_label>.py

If you want to process over a local file or a list of files, use forest_CRABConfig_Run3_PbPb_DATA_filelist_TEMPLATE.py as your template instead. Save your file(s) to a .txt file using a command like:

ls /path/to/files/*.root > filelist_<your_label>.txt
# If the miniaod files are on /eos, you MUST remove "/eos/cms" from
# the start of the paths:
sed -i "s|/eos/cms||" filelist_<your_label>.txt

Modify the input and output paths in the config (example shown below):

# INPUT/OUTPUT SETTINGS

pd = '0'
jobTag = 'LowPUpp_SpecialZeroBias' + pd
cmsswConfig = 'forest_CMSSWConfig_Run3_161X_2026PbPb_DATA.py'

isOnDAS = False
# If isOnDAS == True, use these inputs:
input = '/HIZeroBias' + pd + '/PbPb2026-PromptReco-v1/MINIAOD'
inputDatabase = 'global'
# Otherwise, use a filelist as input:
inputFilelist = 'filelist_HIZeroBias' + pd + '.txt'

output = '/store/group/phys_heavyions/' + username + '/Run3_2026ExpressForests/'
outputServer = 'T2_CH_CERN'

Explanation of variables:

• pd is the PD number of the dataset being forested.
• jobTag is a personal label for differentiating samples.
• cmsswConfig is the CMSSW config file these CRAB jobs should use.
• isOnDAS is a toggle to use either a DAS path or a list of file paths.
• Input on DAS:
  • input is the miniAOD path on CMS DAS.
  • inputDatabase is the DAS "dbs instance" that contains the files (typically 'global' or 'phys03').
• Input from file list:
  • inputFilelist list of local file paths on /eos.
• output is the path on the output server. Forested files are saved here.
• outputServer is the CMS T2 server where data will be stored.

2.2) Initialize VOMS proxy

voms-proxy-init -rfc -voms cms

Tip

Add an alias for this to ~/.bash_profile to make VOMS easier:

alias proxy='voms-proxy-init -rfc -voms cms; cp/tmp/x509up_u'$(id -u)' ~/'

This will let you initialize VOMS just by running the command: proxy

2.3) Submit CRAB jobs (may need to be from src)

cd ..
# Copy CMSSW configs to "src"
cp HiForestSetupPbPbRun2026/forest_CMSSWConfig* ..
# Run CRAB configs from "src"
crab submit -c HiForestSetupPbPbRun2026/forest_CRABConfig_Run3_<your_label>.py

2.4) Track status of CRAB jobs

You can view the status of a job with:

crab status -d CrabWorkArea/crab_<your job tag>/

Tip

Always check job status ~2-3 minutes after submitting to make sure the job has been accepted! If you see the status SUBMITREFUSED you will need to fix the config(s) and delete the job folder from CrabWorkArea/ before submitting it again.

When you (inevitably) have failed jobs, you can resubmit them with:

crab resubmit -d CrabWorkArea/crab_<your job tag>/

Optionally you can also change the requested memory or runtime for jobs when you resubmit:

crab resubmit --maxmemory 2500 --maxruntime 300 -d CrabWorkArea/crab_<your job tag>/

Warning

Requesting more than the maximum allowed memory or runtime will result in your job being refused and you will be unable to resubmit any failed jobs for that CRAB submission!

• maxmemory must not exceed 5000 (MB)!
• maxruntime must not exceed 900 (minutes)!

If you need to stop a job before it finishes, use:

crab kill -d CrabWorkArea/crab_<your job tag>/

---

3) Quick Reference

CMSSW

# Run CMSSWConfig LOCALLY:
cmsRun forest_CMSSWConfig_XXXX.py

CRAB

# Submit job:
crab submit -c <CRAB_config_file.py>

# Check job status:
crab status -d <path/to/crab_status_directory/>

# Kill a job (WARNING: this is irreversible!):
crab kill -d <path/to/crab_status_directory/>

# Resubmit failed jobs:
crab resubmit -d <path/to/crab_status_directory/>
# Resubmit with max memory and max runtime
crab resubmit --maxmemory 3000 --maxruntime 450 -d <path/to/crab_status_directory/>

Updating "forest_CMSSW_16_1_X"

This process should work for any branch of the cmshi/cmssw.git repo. The example below uses forest_CMSSW_16_1_X:

cd CMSSW_16_1_1/src/HeavyIonsAnalysis/
git config pull.rebase true
git remote add git@github.com:cmshi/cmssw.git
git fetch cmshi forest_CMSSW_16_1_X
git pull cmshi forest_CMSSW_16_1_X

# Return to src folder and recompile:
cd ..
cmsenv
scram b -j4

VOMS Certificate Setup

Obtaining Certificates

https://ca.cern.ch/ca/user/Request.aspx?template=ee2user

Use the “New Grid User Certificate” tab to get a new CERN grid. You should set a password for this, and will need to remember it.

Linux/Unix Installation

https://twiki.cern.ch/twiki/bin/view/CMSPublic/WorkBookStartingGrid#BasicGrid

To setup the certificate in your remote workspace, you should:

1. Export the certificate from your browser to a file in p12 format. You can give any name to your p12 file (in the example below the name is mycert.p12).

2. Place the p12 certificate file in the .globus directory of your home area. If the .globus directory doesn't exist, create it.

cd ~
mkdir .globus
cd ~/.globus
mv /path/to/mycert.p12 .

3. Execute the following shell commands:

rm -f usercert.pem
rm -f userkey.pem
openssl pkcs12 -in mycert.p12 -clcerts -nokeys -out usercert.pem
openssl pkcs12 -in mycert.p12 -nocerts -out userkey.pem
chmod 400 userkey.pem
chmod 400 usercert.pem

Warning

If you are new to VOMS, you will need to sign the Acceptable Usage Policy (AUP) before you are able to access files, tools, and servers secured by certificate access. Just follow instructions here to sign the CMS AUP: https://twiki.cern.ch/twiki/bin/view/CMSPublic/SWGuideLcgAccess#AUP