Disk use limitation

[stsmall]

Hi,
When SGA-ICE is run using multiple-kmer values it saves each intermediate file to the tmp directory. The cleanup option only seems to remove the files at the very end of the run. If the original fastq (not gzipped, is that a limitation of SGA-ICE or SGA?) is large and paired, so 2 40G files, saving the intermediate files becomes a space issue. For example, a kmer option of 50, 100, 150, 200 would produce 400G in intermediate files. I edited the bash file to include 'rm' to remove the intermediate files and it seems to have worked out OK. Do you see any issue with adding the 'rm' line? I added it before the "build index" where k50 was corrected to k100, then I removed k50 before the index was built on k100.

[MichaelHiller]

Hi,
Yes, you can delete intermediate files before the end of the run. Good point.

The earliest point where you can delete the _k50_fastq files is after you have corrected them with k=100. Afterwards, all k100 files are concatenated and a new index is build. So what you describe above is correct.

Best,
Michael

[stsmall]

Hi Michael,
Thanks. It seems that SGA will take a gzipped fastq, is the requirement of an gunzipped fastq file determined by SGA-ICE?
thanks,
scott

[MichaelHiller]

We have never tested running SGA on fastq.gz data. In any case, SGA-ICE just finds the .fastq or .fq files in the directory, however it could be easily extended to also find .fastq.gz.

A quick workaround might be to create .fastq links to the .fastq.gz files. This could work if SGA is still able to figure out that the input file is gzipped.

Best,
Michael