Question regarding parallelisation in IsoQuant

[andr-kun]

Hi there,

I noticed that IsoQuant provide a parallelisation option within the DatasetProcessor class (more specifically in collect_reads method) but it is not entirely clear to me if the parallelisation is done at the read/read group level or at the chromosome level.

I was wondering if you can please clarify which parallelisation strategy was used? I tried to increase the number of threads in my IsoQuant run (using aligned BAM files as input and with a single chromosome only in the BAM, FASTA and GTF files), but when running htop, it looks like only one thread is running while the others stays suspended.

Thanks and kind regards,
Andrian

[andrewprzh]

Hi @andr-kun

IsoQuant does per-chromosome parallelization - extracting reads from a sorted BAM file is the easiest (and still more or less balanced) way to do it (https://github.com/ablab/IsoQuant/blob/master/src/dataset_processor.py#L528).

I tried running IsoQuant quite a lot of time and I have not noticed anything suspicious. Could you send me the log file to check?

Best
Andrey

[andr-kun]

Hi Andrey,

Thanks for getting back to me so quickly! I am running IsoQuant as part of the scnanoseq nextflow pipeline and one of their recommendation for processing PromethION data was to increase the CPU and RAM for IsoQuant. I did so following their recommendation, but I still noticed that the IsoQuant step was still taking some time to run, even with the pipeline splitting the aligned BAM, GTF and FASTA into individual chromosome (so IsoQuant only process a single chromosome) to parallelise the analysis. I was curious why IsoQuant was taking so long so I decided to ssh to the node running the job and found out that only a single thread was active.

I unfortunately do not have a log file handy at the moment, but there are some samples being processed by the pipeline currently, and I can extract the log file once the pipeline is finished.

Is it possible at all to parallelise the processing of the reads further (rather than just by chromosome) as some of the chromosome is taking 7+ hour to finish?

Kind regards,
Andrian

[sklages]

@andr-kun - you may also take a look at I/O . This may slow down processing significantly. Actually reducing the number of cores used may speed up things in the end (I reduced the cores from 40 to 24 and finally to 12 for a bunch of isoquant quant jobs reading/writing data from/to the same storage)..

[andrewprzh]

@andr-kun

In fact, IsoQuant itself does per-chromosome parallelization. I don't remember exactly why scnanoseq implemented their own one on top, but there was some reason at that time. I think there is some discussion about this in this repo. However, there is no other parallelization mechanism, so if you run IsoQuant on a single chromosome, it will use only a single thread.

I might try implement some other parallelization approach with smaller read batches in the future, but it may take quite some time and effort. Also, my main priority at the moment is to finalize single-cell/spatial functionality within IsoQuant (including barcode calling, PCR de-duplication etc). Let me know you'd like to try it out.

There might be several reasons for a single chromosome to stand out in processing time. The most frequent one that I observed is the presence of the regions with extremely high coverage and a lot of annotated genes and splice junctions.

As was mentioned by @sklages IO may be issue as well.

Best
Andrey

[andr-kun]

Thanks @sklages and Andrey (@andrewprzh) for the replies and suggestions! I've extracted some logs from two scnanoseq runs (from separate samples). Both runs took about 3-4 hours, even though one of the sample (SLX27335) only have 1/3 the amount of read compared to the other sample.

I suspect @sklages comment about I/O being the limiting factor could potentially be the reason here as I do notice sometimes that the I/O at the HPC can be quite variable.

Since Andrey has confirmed that the parallelisation is done at the chromosome level, I will reduce the amount of CPU assigned to IsoQuant (since scnanoseq recommend assigning 30 CPU, which is not being utilised at all). Do you have any recommendations regarding the amount of RAM that should be allocated? Is the amount of RAM dependent on the chromosome being processed or on the number of reads being processed?

Thanks and kind regards,
Andrian

SLX27335_isoquant.log
SLX27338_isoquant.log

[lianov]

@andr-kun : saw this and wanted to ask to please let us know what you observe with the reduced CPU along with the memory and time used. It has been a while since we have tested this in-depth but from our older discussions we still observed an issue with scalability. If you notice that leaving the data "un-splitted" scales better now, we will be happy to revert this in scnanoseq.

One of the older issues discussing scalability is this one: #280, but we may have had additional discussions as well.

[andrewprzh]

@andr-kun

I see there are more than 23 million reads mapped to chrM, this can take quite a while.
Estimating RAM is complicated and depends on way too many factors.

Also, the version you use (3.6.1) is quite old.

@lianov thanks for referencing the old discussion, it may bring some light.

I also plan to make a new release soon with multiple performance improvements and single-cell functionality (e.g. barcode calling and PCR deduplication).

[andr-kun]

Yes, I did notice that the version of isoquant used is quite old, however that is the version utilised in the scnanoseq pipeline.

From previous discussion with @lianov, I believe there will be an update to scnanoseq this year, so hopefully the version of IsoQuant used in the pipeline will be updated then.

[andrewprzh]

Great news!

@lianov let's stay in touch, there might some new stuff beneficial for your pipeline as well.

[alagurajv]

I notice 3.11.1 is taking 12 plus hours for a sample of 50M reads. i am using primary chromosome only. What is happening. i increased memory/decreased cpu etc nothing improved in performance.

[lianov]

@andrewprzh yes agreed. We would like to improve IsoQuant's performance in scnanoseq, happy to stay in touch.

@andr-kun you are correct @atrull314 and I will be updating the pipeline in Q1/Q2. However, if the latest version of IsoQuant has shown significant improvements in time, we can do that relatively easily / fast (at least to get in in the dev branch where users can test etc.). Have you noticed any notable improvements in time on your tests with a later version of IsoQuant?

[andrewprzh]

@alagurajv

Please open an new issue and send the log file.

[andrewprzh]

@lianov

I did some performance optimizations, but they mostly affect grouped counts and RAM consumption on large datasets. Let's see if there are any reports from the users.

And I finally released beta version of barcode calling and UMI deduplication functionality.

[lianov]

@andrewprzh : just a quick update from our end. @atrull314 and I are making upgrading IsoQuant a priority during the nf-core hackathon, so we are happy to report on performance here once some of our tests go through.