Running and making jobs on Cygwin

[TerenceSperringerJr]

After configuring Cygwin with a custom startup batch and an edit or two (see: #43), I was able to get as far as the fred_plot instructions at the bottom of this page:
https://github.com/PublicHealthDynamicsLab/FRED/wiki/Parameter-Sweeps

After running tasks & (generated by fred_make_parameters and fred_make_tasks), I receive the following errors:

> cat tasks
TASKDIR/task.R0=1.0 > TASKDIR/task.R0=1.0.out
TASKDIR/task.R0=1.5 > TASKDIR/task.R0=1.5.out
TASKDIR/task.R0=2.0 > TASKDIR/task.R0=2.0.out
TASKDIR/task.R0=2.5 > TASKDIR/task.R0=2.5.out

> TASKDIR/task.R0=1.0 > TASKDIR/task.R0=1.0.out
sh: line 1: 10656 Aborted                 (core dumped) /cygdrive/c/Users/.../workspace/PublicHealthDynamicsLab/FRED/FRED/bin/FRED /cygdrive/c/Users/.../workspace/PublicHealthDynamicsLab/FRED/FRED/RESULTS/JOB/9/WORK/params.R0=1.0 1 > /cygdrive/c/Users/.../workspace/PublicHealthDynamicsLab/FRED/FRED/RESULTS/JOB/9/DATA/OUT/LOG1
sh: line 1:  6072 Aborted                 (core dumped) /cygdrive/c/Users/.../workspace/PublicHealthDynamicsLab/FRED/FRED/bin/FRED /cygdrive/c/Users/.../workspace/PublicHealthDynamicsLab/FRED/FRED/RESULTS/JOB/9/WORK/params.R0=1.0 2 > /cygdrive/c/Users/.../workspace/PublicHealthDynamicsLab/FRED/FRED/RESULTS/JOB/9/DATA/OUT/LOG2
sh: line 1: 13412 Aborted                 (core dumped) /cygdrive/c/Users/.../workspace/PublicHealthDynamicsLab/FRED/FRED/bin/FRED /cygdrive/c/Users/.../workspace/PublicHealthDynamicsLab/FRED/FRED/RESULTS/JOB/9/WORK/params.R0=1.0 3 > /cygdrive/c/Users/.../workspace/PublicHealthDynamicsLab/FRED/FRED/RESULTS/JOB/9/DATA/OUT/LOG3
sh: line 1:  6396 Aborted                 (core dumped) /cygdrive/c/Users/.../workspace/PublicHealthDynamicsLab/FRED/FRED/bin/FRED /cygdrive/c/Users/.../workspace/PublicHealthDynamicsLab/FRED/FRED/RESULTS/JOB/9/WORK/params.R0=1.0 4 > /cygdrive/c/Users/.../workspace/PublicHealthDynamicsLab/FRED/FRED/RESULTS/JOB/9/DATA/OUT/LOG4
sh: line 1: 13144 Aborted                 (core dumped) /cygdrive/c/Users/.../workspace/PublicHealthDynamicsLab/FRED/FRED/bin/FRED /cygdrive/c/Users/.../workspace/PublicHealthDynamicsLab/FRED/FRED/RESULTS/JOB/9/WORK/params.R0=1.0 5 > /cygdrive/c/Users/.../workspace/PublicHealthDynamicsLab/FRED/FRED/RESULTS/JOB/9/DATA/OUT/LOG5
sh: line 1: 11844 Aborted                 (core dumped) /cygdrive/c/Users/.../workspace/PublicHealthDynamicsLab/FRED/FRED/bin/FRED /cygdrive/c/Users/.../workspace/PublicHealthDynamicsLab/FRED/FRED/RESULTS/JOB/9/WORK/params.R0=1.0 6 > /cygdrive/c/Users/.../workspace/PublicHealthDynamicsLab/FRED/FRED/RESULTS/JOB/9/DATA/OUT/LOG6
sh: line 1: 14624 Aborted                 (core dumped) /cygdrive/c/Users/.../workspace/PublicHealthDynamicsLab/FRED/FRED/bin/FRED /cygdrive/c/Users/.../workspace/PublicHealthDynamicsLab/FRED/FRED/RESULTS/JOB/9/WORK/params.R0=1.0 7 > /cygdrive/c/Users/.../workspace/PublicHealthDynamicsLab/FRED/FRED/RESULTS/JOB/9/DATA/OUT/LOG7
sh: line 1: 15084 Aborted                 (core dumped) /cygdrive/c/Users/.../workspace/PublicHealthDynamicsLab/FRED/FRED/bin/FRED /cygdrive/c/Users/.../workspace/PublicHealthDynamicsLab/FRED/FRED/RESULTS/JOB/9/WORK/params.R0=1.0 8 > /cygdrive/c/Users/.../workspace/PublicHealthDynamicsLab/FRED/FRED/RESULTS/JOB/9/DATA/OUT/LOG8
sh: line 1: 13416 Aborted                 (core dumped) /cygdrive/c/Users/.../workspace/PublicHealthDynamicsLab/FRED/FRED/bin/FRED /cygdrive/c/Users/.../workspace/PublicHealthDynamicsLab/FRED/FRED/RESULTS/JOB/9/WORK/params.R0=1.0 9 > /cygdrive/c/Users/.../workspace/PublicHealthDynamicsLab/FRED/FRED/RESULTS/JOB/9/DATA/OUT/LOG9
sh: line 1:  2392 Aborted                 (core dumped) /cygdrive/c/Users/.../workspace/PublicHealthDynamicsLab/FRED/FRED/bin/FRED /cygdrive/c/Users/.../workspace/PublicHealthDynamicsLab/FRED/FRED/RESULTS/JOB/9/WORK/params.R0=1.0 10 > /cygdrive/c/Users/.../workspace/PublicHealthDynamicsLab/FRED/FRED/RESULTS/JOB/9/DATA/OUT/LOG10

task.R0=1.0.out doesn't tell me a whole lot:

task.R0=1.0 STARTED Thu Oct 27 14:46:07 EDT 2016
/cygdrive/c/Users/.../workspace/PublicHealthDynamicsLab/FRED/FRED/bin/fred_delete: KEY = R0=1.0  JOB = 5
/cygdrive/c/Users/.../workspace/PublicHealthDynamicsLab/FRED/FRED/bin/fred_delete: /cygdrive/c/Users/.../workspace/PublicHealthDynamicsLab/FRED/FRED/RESULTS/JOB/5 deleted
R0=1.0 9
task.R0=1.0 FINISHED Thu Oct 27 14:46:14 EDT 2016

but task.R0=1.0 itself is a script that contains:

#!/bin/csh -f
echo -n "task.R0=1.0 STARTED "
date
fred_delete -f -k R0=1.0
fred_job -k R0=1.0 -p PARAMDIR/params.R0=1.0 -n 10  -m 1
if ( -e collect_task_data ) then
./collect_task_data R0=1.0
endif
echo -n "task.R0=1.0 FINISHED "
date

It seems that fred_job -k R0=1.0 -p PARAMDIR/params.R0=1.0 -n 10 -m 1 is where my troubles lie as running it produces the aborted errors above.

Update: So I tried fred_job -k R0=1.0 -p PARAMDIR/params.R0=1.0 -n 1 and still got the same error.

However, I discovered that RESULTS/JOB/12/DATA/OUT outputs an error file, err1.txt:
FRED ERROR: Sorry, could not find a county for fips = |42065|

Checking the bottom of DATA/OUT/LOG1 I find the last two lines are:

fred_open_file: opening file /cygdrive/c/Users/.../workspace/PublicHealthDynamicsLab/FRED/FRED/input_files/countries/usa/US_counties.txt for reading
FRED ERROR: Sorry, could not find a county for fips = |42065|

But opening /FRED/input_files/countries/usa/US_counties.txt and performing a search for '42065' produces 121 matches.

[TerenceSperringerJr]

https://github.com/PublicHealthDynamicsLab/FRED/wiki/Parameter-Sweeps#baseline-parameter-file

Baseline Parameter File
First create a file called, for example, params.base that contains parameters that are fixed for all jobs in the parameter sweep. The contents will depend on the purpose if the study. Here is an example:

% cat params.base

days = 100
fips = 42065

If I use a parameter instead of fips for this example (such as counties = 53033), fred_jobs will run to completion. Given the error above I'd say that the fips parameter isn't processing correctly.

Update: Actually it looks like counties = 53033 is ignored and has no effect on the used parameters according to the log.

Update: The problem seems to arise within /FRED/src/Place_List.cc lines 307 - 341

    // get population_id from fips
    char line_string[FRED_STRING_SIZE];
    char* line;
    char* city;
    char* state;
    char* county;
    char* fips;
    int found = 0;
    int fipsLength = strlen(Global::FIPS_code);
    if(fipsLength == 5) {
      FILE* fp = Utils::fred_open_file(Place_List::Counties_file);
      if(fp == NULL) {
        Utils::fred_abort("counties file |%s| NOT FOUND\n", Place_List::Counties_file);
      }
      while(fgets(line_string, FRED_STRING_SIZE - 1, fp) != NULL) {
        line = line_string;
        city = strsep(&line, "\t");
        state = strsep(&line, "\t");
        county = strsep(&line, "\t");
        fips = strsep(&line, "\n");
        // printf("city = |%s| state = |%s| county = |%s| fips = |%s|\n",
        // city,state,county,fips);
        if(strcmp(Global::FIPS_code, fips) == 0) {
          found = 1;
          break;
        }
      }
      fclose(fp);
      if(found) {
        Utils::fred_log("FOUND a county = |%s County %s| for fips = |%s|\n", county, state, fips);
        sprintf(Global::Synthetic_population_id, "%s_%s", Global::Synthetic_population_version, fips);
      } else {
        Utils::fred_abort("Sorry, could not find a county for fips = |%s|\n", Global::FIPS_code);
      }
    } else if(fipsLength == 2) {

[TerenceSperringerJr]

I uncommented the printf() instruction after the strsep() calls and it turns out the problem is Unix newline characters not accounting for Windows-styled carriage return line feed newlines. My idea is to remove any carriage return characters whenever a newline character is found.

[TerenceSperringerJr]

RESOLVED Cygwin Issue: fred_plot

This arises after correcting for #44

I removed the -m flag from each of the fred_job calls within the task files of TASKDIR and re-ran ./tasks &. This created the DATA/TABLES files, so I then tried running fred_plot again:

> fred_plot -v C -b -n -w -X 10 -t "Comparing Epidemics" -k R0=1.0 -k R0=1.5 -k R0=2.0 -k R0=2.5
Can't exec "plot-C.plt": No such file or directory at .../FRED/bin/fred_plot line 352.
fred_plot: source_file = plot-C.plt  image_file = plot-C.pdf
Can't exec "open": No such file or directory at .../FRED/bin/fred_plot line 354.

plot-C.plt was created in my current working directory (FRED/experiment which contains the params.base, config, and task files and directories mentioned earlier), however, so I retried after moving it to FRED/bin:

> fred_plot -v C -b -n -w -X 10 -t "Comparing Epidemics" -k R0=1.0 -k R0=1.5 -k R0=2.0 -k R0=2.5
Fontconfig error: Cannot load default config file
fred_plot: source_file = plot-C.plt  image_file = plot-C.pdf
Can't exec "open": No such file or directory at .../FRED/bin/fred_plot line 354.

Oddly enough a pdf that lacks labels was still created in my current working directory (FRED/experiment). I retried with a copy of the config file in FRED/bin, but apparently the results were the same so Fontconfig must be trying to load a different config file (which I assume to be for initializing internal defaults)...

[gallowdd]

So the last part is simply because there is no "open" command on linux like there is on Mac. fred_plot will create the pdf and lastly will try to open it up on a mac. If you want it to open, there is a probably a linux alternative. One such is 'xdg-open' (see: http://stackoverflow.com/questions/264395/linux-equivalent-of-the-mac-os-x-open-command)
You can eitehr alter the part of fred_plot to check the OS or you can just alias 'open' to 'xdg-open' on your machine.

[TerenceSperringerJr]

That's good to know. This seems to do the trick:

354:
355: if($^O eq "darwin") {
356: system ("open $outfile") if $display;
357: }
358: else {
359: system ("xdg-open $outfile") if $display;
360: }
361:

Hmmm, but I still need to get Fontconfig to work if these unlabeled plots are to be of any meaning.

Update: I made some modifications to fred_plot:

Added Lines 6 - 8

5: use Getopt::Long qw(:config no_ignore_case bundling);
6: use Cwd;
7:
8: my $cwd = getcwd();
9:
.
.
.
Added lines 356 - 367
355: system "chmod +x $pltfile";
356: if(system ($pltfile) != 0) {
357: print "Trying " . $cwd . "/" . $pltfile . "\n";
358: system ($cwd . "/" . $pltfile);
359: }
360: print "fred_plot: source_file = $pltfile image_file = $outfile\n";
361:
362: if($^O eq "darwin") {
363: system ("open $outfile") if $display;
364: }
365: else {
366: system ("xdg-open $outfile") if $display;
367: }

And it turns out the Fontconfig error seems to be caused in plot-C.plt (aka $pltfile), which turns out to be a generated gnuplot script. So gnuplot is having difficulty finding its default config file.

ISSUE RESOLVED: fred_plot
And now I realize the default config file couldn't be found because I've been using a localized 'partial' copy of Cygwin in an effort to package a 'runs-out-of-box' solution. Running plot-C.plt with the full installation version produced the expected output and finished the job appropriately. Please pardon my reporting this last error (though the xdg-open solution was helpful).