Unable to plot results from running tasks generated by fred_make_tasks

[TerenceSperringerJr]

Oracle VM VirtualBox
Memory: 3.9 GiB
Processor: Intel Core i7-4800MQ CPU @ 2.70GHz
OS: Ubuntu 16.04 LTS (64-bit)

After running tasks & (generated by fred_make_parameters and fred_make_tasks), I try to run fred_plot –v C –b -n -w -X 10 –t "Comparing Epidemics" -k R0=1.0 -k R0=1.5 -k R0=2.0 -k R0=2.5 and receive the following errors:
.../FRED/bin/fred_csv: ERROR FINDING Pop file .../FRED/RESULTS/JOB/1/DATA/TABLES/Weekly-1.txt .../FRED/bin/fred_plot: error in get_data

Indeed, /FRED/RESULTS/JOB/1/DATA/TABLES/Weekly-1.txt doesn't seem to be generated after running tasks &

However, fred_plot seems to work fine after running:
fred_job -k test -p params -n 3
(then after finishing)
fred_plot -k test -v AR

I got as far as the fred_plot instructions at the bottom of this page:
https://github.com/PublicHealthDynamicsLab/FRED/wiki/Parameter-Sweeps

Are the flags correct in the wiki instructions or am I somehow forgetting something?
My goal is ultimately to get this working with cygwin on windows (getting stuck with sh: line 1: ### Aborted (core dumped)... errors with running tasks & but I am pretty sure Cygwin is just getting stuck trying to run perl scripts within the current C shell environment setup)

[gallowdd]

tasks is just creating a bunch of fred_jobs, correct?

Try opening it up and seeing the actual fred_job commands.

Are they working individually?

[TerenceSperringerJr]

On Ubuntu running works fine, but there are no Weekly-#.txt files appearing in /JOB/#/DATA/TABLES so plotting fails.

(EDIT: I moved the Cygwin-related matters to a new thread #45)

I ran:

$ cat tasks
TASKDIR/task.R0=1.0 > TASKDIR/task.R0=1.0.out
TASKDIR/task.R0=1.5 > TASKDIR/task.R0=1.5.out
TASKDIR/task.R0=2.0 > TASKDIR/task.R0=2.0.out
TASKDIR/task.R0=2.5 > TASKDIR/task.R0=2.5.out

$ cat TASKDIR/task.R0=1.0
#!/bin/csh -f
echo -n "task.R0=1.0 STARTED "
date
fred_delete -f -k R0=1.0
fred_job -k R0=1.0 -p PARAMDIR/params.R0=1.0 -n 10  -m 1
if ( -e collect_task_data ) then
./collect_task_data R0=1.0
endif
echo -n "task.R0=1.0 FINISHED "
date

$ fred_job -k R0=1.0 -p PARAMDIR/params.R0=1.0 -n 10 -m 1
R0=1.0 10

but it seems that /FRED/RESULTS/JOB/10/DATA/TABLES is still missing the Weekly-#.txt files

[TerenceSperringerJr]

I reproduced the steps in a mac environment and am hitting the same problem: no files are appearing in /FRED/RESULTS/JOB/1/DATA/TABLES other than VARS

$ ./tasks &
[1] 90730

$ fred_jobs
KEY = R0=1.0                    JOB =   1       STATUS = FINISHED Mon Oct 31 18:14:01 2016
KEY = R0=1.5                    JOB =   2       STATUS = FINISHED Mon Oct 31 18:14:22 2016
KEY = R0=2.0                    JOB =   3       STATUS = FINISHED Mon Oct 31 18:14:49 2016
KEY = R0=2.5                    JOB =   4       STATUS = FINISHED Mon Oct 31 18:15:20 2016

$ fred_plot -v C b -n -w -X 10 -t "Comparing Epidemics" -k R0=1.0 -k R0=1.5 -k R0=2.0 -k R0=2.5
/Users/.../PublicHealthDynamicsLab/FRED/bin/fred_csv: ERROR FINDING Pop file /Users/.../PublicHealthDynamicsLab/FRED/RESULTS/JOB/1/DATA/TABLES/Weekly-1.txt
/Users/.../PublicHealthDynamicsLab/FRED/bin/fred_plot: error in get_data

$ cd ..

$ ls
LICENSE                 bin                     input_files             tests
Makefile                doc                     populations             tutorial
README.md               etc                     set_variables.sh
RESULTS                 experiment              src

$ cd RESULTS

$ ls
ID      JOB     KEY

$ cd JOB

$ ls
1       2       3       4

$ cd 1

$ ls
DATA    META    WORK

$ cd DATA

$ ls
OUT     REPORTS TABLES

$ cd TABLES

$ ls
VARS

[TerenceSperringerJr]

I omitted -m 1 and tried running fred_job -k R0=1.0 -p PARAMDIR/params.R0=1.0 -n 10 again, but this time I saw results:

$ ls RESULTS/JOB/1/DATA/TABLES
AR-1.txt        GQ-5.txt             Prison.txt             Weekly_College-4.txt    Weekly_N-9.txt
AR-10.txt       GQ-6.txt             R-1.txt                Weekly_College-5.txt    Weekly_N.txt
AR-2.txt        GQ-7.txt             R-10.txt               Weekly_College-6.txt    Weekly_Nursing_Home-1.txt
AR-3.txt        GQ-8.txt             R-2.txt                Weekly_College-7.txt    Weekly_Nursing_Home-10.txt
AR-4.txt        GQ-9.txt             R-3.txt                Weekly_College-8.txt    Weekly_Nursing_Home-2.txt
AR-5.txt        GQ.txt               R-4.txt                Weekly_College-9.txt    Weekly_Nursing_Home-3.txt
AR-6.txt        I-1.txt              R-5.txt                Weekly_College.txt      Weekly_Nursing_Home-4.txt
AR-7.txt        I-10.txt             R-6.txt                Weekly_Cs-1.txt         Weekly_Nursing_Home-5.txt
AR-8.txt        I-2.txt              R-7.txt                Weekly_Cs-10.txt        Weekly_Nursing_Home-6.txt
AR-9.txt        I-3.txt              R-8.txt                Weekly_Cs-2.txt         Weekly_Nursing_Home-7.txt
AR.txt          I-4.txt              R-9.txt                Weekly_Cs-3.txt         Weekly_Nursing_Home-8.txt
ARs-1.txt       I-5.txt              R.txt                  Weekly_Cs-4.txt         Weekly_Nursing_Home-9.txt
ARs-10.txt      I-6.txt              RR-1.txt               Weekly_Cs-5.txt         Weekly_Nursing_Home.txt
ARs-2.txt       I-7.txt              RR-10.txt              Weekly_Cs-6.txt         Weekly_P-1.txt
ARs-3.txt       I-8.txt              RR-2.txt               Weekly_Cs-7.txt         Weekly_P-10.txt
ARs-4.txt       I-9.txt              RR-3.txt               Weekly_Cs-8.txt         Weekly_P-2.txt
ARs-5.txt       I.txt                RR-4.txt               Weekly_Cs-9.txt         Weekly_P-3.txt
ARs-6.txt       Is-1.txt             RR-5.txt               Weekly_Cs.txt           Weekly_P-4.txt
ARs-7.txt       Is-10.txt            RR-6.txt               Weekly_E-1.txt          Weekly_P-5.txt
ARs-8.txt       Is-2.txt             RR-7.txt               Weekly_E-10.txt         Weekly_P-6.txt
ARs-9.txt       Is-3.txt             RR-8.txt               Weekly_E-2.txt          Weekly_P-7.txt
ARs.txt         Is-4.txt             RR-9.txt               Weekly_E-3.txt          Weekly_P-8.txt
C-1.txt         Is-5.txt             RR.txt                 Weekly_E-4.txt          Weekly_P-9.txt
C-10.txt        Is-6.txt             S-1.txt                Weekly_E-5.txt          Weekly_P.txt
C-2.txt         Is-7.txt             S-10.txt               Weekly_E-6.txt          Weekly_Prison-1.txt
C-3.txt         Is-8.txt             S-2.txt                Weekly_E-7.txt          Weekly_Prison-10.txt
C-4.txt         Is-9.txt             S-3.txt                Weekly_E-8.txt          Weekly_Prison-2.txt
C-5.txt         Is.txt               S-4.txt                Weekly_E-9.txt          Weekly_Prison-3.txt
C-6.txt         M-1.txt              S-5.txt                Weekly_E.txt            Weekly_Prison-4.txt
C-7.txt         M-10.txt             S-6.txt                Weekly_GQ-1.txt         Weekly_Prison-5.txt
C-8.txt         M-2.txt              S-7.txt                Weekly_GQ-10.txt        Weekly_Prison-6.txt
C-9.txt         M-3.txt              S-8.txt                Weekly_GQ-2.txt         Weekly_Prison-7.txt
C.txt           M-4.txt              S-9.txt                Weekly_GQ-3.txt         Weekly_Prison-8.txt
College-1.txt   M-5.txt              S.txt                  Weekly_GQ-4.txt         Weekly_Prison-9.txt
College-10.txt  M-6.txt              VARS                   Weekly_GQ-5.txt         Weekly_Prison.txt
College-2.txt   M-7.txt              Week-1.txt             Weekly_GQ-6.txt         Weekly_R-1.txt
College-3.txt   M-8.txt              Week-10.txt            Weekly_GQ-7.txt         Weekly_R-10.txt
College-4.txt   M-9.txt              Week-2.txt             Weekly_GQ-8.txt         Weekly_R-2.txt
College-5.txt   M.txt                Week-3.txt             Weekly_GQ-9.txt         Weekly_R-3.txt
College-6.txt   Military-1.txt       Week-4.txt             Weekly_GQ.txt           Weekly_R-4.txt
College-7.txt   Military-10.txt      Week-5.txt             Weekly_I-1.txt          Weekly_R-5.txt
College-8.txt   Military-2.txt       Week-6.txt             Weekly_I-10.txt         Weekly_R-6.txt
College-9.txt   Military-3.txt       Week-7.txt             Weekly_I-2.txt          Weekly_R-7.txt
College.txt     Military-4.txt       Week-8.txt             Weekly_I-3.txt          Weekly_R-8.txt
Cs-1.txt        Military-5.txt       Week-9.txt             Weekly_I-4.txt          Weekly_R-9.txt
Cs-10.txt       Military-6.txt       Week.txt               Weekly_I-5.txt          Weekly_R.txt
Cs-2.txt        Military-7.txt       Weekly-1.txt           Weekly_I-6.txt          Weekly_RR-1.txt
Cs-3.txt        Military-8.txt       Weekly-10.txt          Weekly_I-7.txt          Weekly_RR-10.txt
Cs-4.txt        Military-9.txt       Weekly-2.txt           Weekly_I-8.txt          Weekly_RR-2.txt
Cs-5.txt        Military.txt         Weekly-3.txt           Weekly_I-9.txt          Weekly_RR-3.txt
Cs-6.txt        N-1.txt              Weekly-4.txt           Weekly_I.txt            Weekly_RR-4.txt
Cs-7.txt        N-10.txt             Weekly-5.txt           Weekly_Is-1.txt         Weekly_RR-5.txt
Cs-8.txt        N-2.txt              Weekly-6.txt           Weekly_Is-10.txt        Weekly_RR-6.txt
Cs-9.txt        N-3.txt              Weekly-7.txt           Weekly_Is-2.txt         Weekly_RR-7.txt
Cs.txt          N-4.txt              Weekly-8.txt           Weekly_Is-3.txt         Weekly_RR-8.txt
Date-1.txt      N-5.txt              Weekly-9.txt           Weekly_Is-4.txt         Weekly_RR-9.txt
Date-10.txt     N-6.txt              Weekly_AR-1.txt        Weekly_Is-5.txt         Weekly_RR.txt
Date-2.txt      N-7.txt              Weekly_AR-10.txt       Weekly_Is-6.txt         Weekly_S-1.txt
Date-3.txt      N-8.txt              Weekly_AR-2.txt        Weekly_Is-7.txt         Weekly_S-10.txt
Date-4.txt      N-9.txt              Weekly_AR-3.txt        Weekly_Is-8.txt         Weekly_S-2.txt
Date-5.txt      N.txt                Weekly_AR-4.txt        Weekly_Is-9.txt         Weekly_S-3.txt
Date-6.txt      Nursing_Home-1.txt   Weekly_AR-5.txt        Weekly_Is.txt           Weekly_S-4.txt
Date-7.txt      Nursing_Home-10.txt  Weekly_AR-6.txt        Weekly_M-1.txt          Weekly_S-5.txt
Date-8.txt      Nursing_Home-2.txt   Weekly_AR-7.txt        Weekly_M-10.txt         Weekly_S-6.txt
Date-9.txt      Nursing_Home-3.txt   Weekly_AR-8.txt        Weekly_M-2.txt          Weekly_S-7.txt
Date.txt        Nursing_Home-4.txt   Weekly_AR-9.txt        Weekly_M-3.txt          Weekly_S-8.txt
Day-1.txt       Nursing_Home-5.txt   Weekly_AR.txt          Weekly_M-4.txt          Weekly_S-9.txt
Day-10.txt      Nursing_Home-6.txt   Weekly_ARs-1.txt       Weekly_M-5.txt          Weekly_S.txt
Day-2.txt       Nursing_Home-7.txt   Weekly_ARs-10.txt      Weekly_M-6.txt          WkDay-1.txt
Day-3.txt       Nursing_Home-8.txt   Weekly_ARs-2.txt       Weekly_M-7.txt          WkDay-10.txt
Day-4.txt       Nursing_Home-9.txt   Weekly_ARs-3.txt       Weekly_M-8.txt          WkDay-2.txt
Day-5.txt       Nursing_Home.txt     Weekly_ARs-4.txt       Weekly_M-9.txt          WkDay-3.txt
Day-6.txt       P-1.txt              Weekly_ARs-5.txt       Weekly_M.txt            WkDay-4.txt
Day-7.txt       P-10.txt             Weekly_ARs-6.txt       Weekly_Military-1.txt   WkDay-5.txt
Day-8.txt       P-2.txt              Weekly_ARs-7.txt       Weekly_Military-10.txt  WkDay-6.txt
Day-9.txt       P-3.txt              Weekly_ARs-8.txt       Weekly_Military-2.txt   WkDay-7.txt
Day.txt         P-4.txt              Weekly_ARs-9.txt       Weekly_Military-3.txt   WkDay-8.txt
E-1.txt         P-5.txt              Weekly_ARs.txt         Weekly_Military-4.txt   WkDay-9.txt
E-10.txt        P-6.txt              Weekly_C-1.txt         Weekly_Military-5.txt   WkDay.txt
E-2.txt         P-7.txt              Weekly_C-10.txt        Weekly_Military-6.txt   Year-1.txt
E-3.txt         P-8.txt              Weekly_C-2.txt         Weekly_Military-7.txt   Year-10.txt
E-4.txt         P-9.txt              Weekly_C-3.txt         Weekly_Military-8.txt   Year-2.txt
E-5.txt         P.txt                Weekly_C-4.txt         Weekly_Military-9.txt   Year-3.txt
E-6.txt         Prison-1.txt         Weekly_C-5.txt         Weekly_Military.txt     Year-4.txt
E-7.txt         Prison-10.txt        Weekly_C-6.txt         Weekly_N-1.txt          Year-5.txt
E-8.txt         Prison-2.txt         Weekly_C-7.txt         Weekly_N-10.txt         Year-6.txt
E-9.txt         Prison-3.txt         Weekly_C-8.txt         Weekly_N-2.txt          Year-7.txt
E.txt           Prison-4.txt         Weekly_C-9.txt         Weekly_N-3.txt          Year-8.txt
GQ-1.txt        Prison-5.txt         Weekly_C.txt           Weekly_N-4.txt          Year-9.txt
GQ-10.txt       Prison-6.txt         Weekly_College-1.txt   Weekly_N-5.txt          Year.txt
GQ-2.txt        Prison-7.txt         Weekly_College-10.txt  Weekly_N-6.txt
GQ-3.txt        Prison-8.txt         Weekly_College-2.txt   Weekly_N-7.txt
GQ-4.txt        Prison-9.txt         Weekly_College-3.txt   Weekly_N-8.txt

So it would seem that the -m flag (which seems to be for utilizing multiprocessors) fails to create any files (other than an empty VARS) in FRED/RESULTS/JOB/#/DATA/TABLES/.

I removed the -m flag from each of the fred_job calls within the task files of TASKDIR and re-ran ./tasks &. This creates the DATA/TABLES files but it doesn't fix the issue of properly processing the -m flag option.