"replicate" option in future_om_list does not replicate rec_devs when scenarios run in tandem

[rwildermuth]

After some digging, I figured out that running multiple scenarios in a single run_SSMSE() call with scen = c("replicate", "all") in the future_om_list doesn't actually make the recruitment deviations in the OM the same across scenarios. If I provide a seed to the run_SSMSE() call and run it separately for each scenario, then I do get the same rec_dev time series.

I've saved an example of this while implementing autocorrelated recruitment to our SardineMSE github here.

I discovered this because I need to call run_SSMSE() separately for our project in order to use multiple different custom HCR functions and that function won't allow you to specify more than one 'custom_MS_source'.

Note: this isn't a problem for situations where all recruitment deviations are provided outside of SSMSE as is done here. In this case, tandem scenarios all have rec_dev time series in common.

[k-doering-NOAA]

Thanks for reporting this @rwildermuth! Am I right in that you have a work around for this issue for now?

[rwildermuth]

Correct. I've confirmed that running run_SSMSE() with a single scenario and the same seed value gives the same rec_devs forcing each time. That's what I've implemented, for example, here: https://github.com/futureseas/SardineMSE/blob/main/R/constGrowth2001OM_constGrowth2005EM_ARRec_MSE.R It's kind of messy, but at least gets the job done...

…

On Fri, Dec 16, 2022 at 6:52 AM Kathryn Doering ***@***.***> wrote: Thanks for reporting this @rwildermuth <https://github.com/rwildermuth>! Am I right in that you have a work around for this issue for now? — Reply to this email directly, view it on GitHub <#144 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ACXJHTWW5ZZB52XU3TPXIYDWNR6ZDANCNFSM6AAAAAATAJZB6I> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[k-doering-NOAA]

Replicated the bug by running the following code and looking at the Fcast_recdevs section of the ss.par file for each OM output. The Fcast_recdevs were all different, even though they should have been identical across scenarios and iterations given the rec_dev_specify$scen <- c("replicate", "all") line.

devtools::load_all() # or library(SSMSE)
new_temp_path <- file.path("mult_scenarios")
nyrs <- 6
datfile <- system.file("extdata", "models", "cod", "ss3.dat", package = "SSMSE")
# create the sample_structure
sample_struct <- create_sample_struct(dat = datfile, nyrs = nyrs)
sample_struct[["lencomp"]] <- NULL
sample_struct[["meanbodywt"]] <- NULL
sample_struct[["MeanSize_at_Age_obs"]] <- NULL

# add recruitment deviations (core issue)
template_mod_change <- create_future_om_list(list_length = 1)
rec_dev_specify <- template_mod_change[[1]]
rec_dev_specify$pars <- "rec_devs"
rec_dev_specify$scen <- c("replicate", "all")
rec_dev_specify$input$first_yr_averaging <- 1 # use same sd as from the orig model.
rec_dev_specify$input$last_yr_averaging <- 100
rec_dev_specify$input$last_yr_orig_val <- 100
rec_dev_specify$input$first_yr_final_val <- 101
rec_dev_specify$input$ts_param <- "sd"
rec_dev_specify$input$value <- NA
my_future_om_list <- list(rec_dev_specify)

# run SSMSE

  result <- run_SSMSE(
    scen_name_vec = c("scen1", "scen2"), # name of the scenario
    out_dir_scen_vec = new_temp_path, # directory in which to run the scenario
    iter_vec = c(2, 2), # run with 3 iterations each
    OM_name_vec = "cod",
    EM_name_vec = "cod", # cod is included in package data
    MS_vec = "EM", # The management strategy is specified in the EM
    custom_MS_source = NULL,
    use_SS_boot_vec = TRUE, # use the SS bootstrap module for sampling
    nyrs_vec = nyrs, # Years to project OM forward
    nyrs_assess_vec = 3, # Years between assessments
    future_om_list = my_future_om_list,
    run_EM_last_yr = FALSE,
    run_parallel = FALSE,
    sample_struct_list = list(sample_struct, sample_struct), # How to sample data for running the EM.
    seed = 12345
  )