Need a resume option for the beamformer

[vivekvenkris]

In case of job failures due to cluster problems or wall times, we need a way to resume a processing.

If we were to do this outside skyweavercpp then, we need to synchronize the time across all file boundaries (of CB, IB and STATS files)

We could take the maximum file size (in bytes) of CB and then compute what that would correspond to for the other files, or actually take max_seconds_per_file and make MultiFileWriter make decisions based on what file it is writing.

[ewanbarr]

I would call it max_output_file_duration.

Need to be careful here as we will be giving this times in seconds and it will translate to samples. If we want to use that time again later we need to ensure that it is integer correct.

Note, we also currently round off the max file size so that it is a multiple of the blocking factor used in the pipeline:

    // Here we round the file size to a multiple of the stream prototype
    std::size_t filesize =
        std::max(1ul,
                 _config.max_output_filesize() / stream_data.size() /
                     sizeof(typename VectorType::value_type)) *
        stream_data.size();
    BOOST_LOG_TRIVIAL(debug)
        << "Maximum allowed file size = " << filesize << " bytes (+header)";

[ewanbarr]

In general does this feature actually make sense?

You could also split the input data in the nexflow jobs and process in smaller batches. Then checkpointing is irrelevant and it is just the job of the cleaver to merge files that are separated in time.

[vishnubk]

Do you mean if the observation duration is 300 seconds, then nextflow splits this into 300/X jobs and then merge later in time? True, we can have a workaround like that as long as the output filenames are unique for every run.

The impact of a failed run will be lesser this way, I can leave X as a variable for nextflow users to tune. Is everyone happy with this workaround?

[vivekvenkris]

skyweaver currently takes start time and duration in seconds to beamform, which it then round off to the nearest sample - so it is the same problem. We need then a way to also give these as samples to make sure that the subsequent runs are contiguous.

I like the idea of having constant time boundaries for each file (CB/IB/Stats), if we can achieve them unambiguously.

[ewanbarr]

My worry about this feature is that in needs lots of parts of the code to know about each other somehow and restricts how they can be implemented. All the different output chains have to output data that is guaranteed to be some multiple of the rates of all the other outputs. Wouldn't it just be safer to do this on the Python side. The logic is so much clearer to me:

1. Check all output files from the run
2. find the latest valid timestamp from the data by looking at the file sizes and header
3. Truncate the existing files back to a common end time (this might be also a multiple of the input file size for example)
4. Calculate what that end time corresponds to as the start time for the input data
5. Make a new input list
6. Start the code again with the same output directory

Doing this in the C++ side just seems difficult as we don't have any signal handling implemented and there is no guarantee anyway that when the code is stopped the stops will be clean. It is better to have the Python code verify the actual correct time to restart from based on what is actually written already.

The only thing that would need to be taken care of from the C++ side then would be to ensure that the filenames don't clash when restarting.

[coclar]

@ewanbarr suggested: "You could also split the input data in the nexflow jobs and process in smaller batches. Then checkpointing is irrelevant and it is just the job of the cleaver to merge files that are separated in time."

I don't think that works - won't there be gaps at the start/end of each batch of input files due to the incoherent de-dispersion?

Actually, I think the same problem applies to the python solution of working out last "good" end time, and starting a new batch of inputs from there.

[ewanbarr]

Yea, this is is a good point. The restart time would need to account for the delays in the system. Those delays are quite complex to work out at the moment as the coherent dedisperser causes anti-delay and the incoherent dedisperser causes delay. These aren't just based on the DM either, in the case of the coherent dedisperser it is also dependent on the optimal FFT size.

[ewanbarr]

Is this feature dead?