snapper threading does not always work

[rbouckaert]

When a nucleotide alignment is imported in BEAUti, it runs with a single thread, but fails when running with multiple threads with a NullPointerException. This should be fixed.

[CMWbio]

Dear Remco,

Any estimation on your timeline for this fix?

Cheers,

Chris

[rbouckaert]

c1035ff may have done this.

[MarnusStoltz]

Yes, the latest SNAPPER can now handle missing data and multithreading is also working. Please let us know if any bugs pop-up.

…

On Mon, Jun 28, 2021 at 5:23 AM Remco Bouckaert ***@***.***> wrote: c1035ff <c1035ff> may have done this. — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#3 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ACHLDX7566U3X4GW73TZGYTTU7TMVANCNFSM4Z6V4EOQ> .

[davidjamesbryant]

BIFFF!!!!! ======================= Prof David Bryant Dept Mathematics and Statistics University of Otago Aotearoa New Zealand On 28/06/2021, at 8:40 PM, Marnus_Stoltz ***@***.***> wrote:  Yes, the latest SNAPPER can now handle missing data and multithreading is also working. Please let us know if any bugs pop-up.

On Mon, Jun 28, 2021 at 5:23 AM Remco Bouckaert ***@***.***> wrote: c1035ff <c1035ff> may have done this. — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#3 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ACHLDX7566U3X4GW73TZGYTTU7TMVANCNFSM4Z6V4EOQ> .

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub<https://apc01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Frbouckaert%2Fsnapper%2Fissues%2F3%23issuecomment-869492424&data=04%7C01%7Cdavid.bryant%40otago.ac.nz%7C22884f5048004fcbbbeb08d93a106edb%7C0225efc578fe4928b1579ef24809e9ba%7C1%7C0%7C637604664520709034%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=O82gOKRmFkJK41pX6WbCMhYQltE736aWMnOVk9vxsJw%3D&reserved=0>, or unsubscribe<https://apc01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FAATIGSHWOKIPCMOTNG7FB4LTVAYQ7ANCNFSM4Z6V4EOQ&data=04%7C01%7Cdavid.bryant%40otago.ac.nz%7C22884f5048004fcbbbeb08d93a106edb%7C0225efc578fe4928b1579ef24809e9ba%7C1%7C0%7C637604664520718992%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=HDQw1E9sFvC3BPsQLhErZKvJ8MnMK%2Bkz2AVYHt3PrdY%3D&reserved=0>.

[CMWbio]

Thanks very much!

[bmichanderson]

Multithreading is not working for me with BEAST 2.6.6 and SNAPPER 1.0.2 on Linux (Ubuntu 20.04) and Windows 10 using the tutorial example files (https://github.com/BEAST2-Dev/beast-docs/releases/download/v1.0/snapper-delimitation-tutorial-2021.zip).
The command:
path/to/beast -threads 6 RunA.xml
The error produced:

Error 110 parsing the xml input file

validate and intialize error: initAndValidate() failed! Index 0 out of bounds for length 0

Error detected about here:
  <beast>
      <run spec='beast.inference.PathSampler'>
          <mcmc id='mcmc' spec='MCMC'>
              <distribution id='posterior' spec='util.CompoundDistribution'>
                  <distribution id='likelihood' spec='util.CompoundDistribution'>
                      <distribution id='snapperTreeLikelihood.hemi129' spec='snapper.SnapperTreeLikelihood'>

[MarnusStoltz]

Hi Ben, Please try SNAPPER 1.0.1 - there might be an update that broke it again. Thanks Marnus

…

On Tue, Oct 5, 2021 at 12:36 PM Ben Anderson ***@***.***> wrote: Multithreading is not working for me with BEAST 2.6.6 and SNAPPER 1.0.2 on Linux (Ubuntu 20.04) and Windows 10 using the tutorial example files ( https://github.com/BEAST2-Dev/beast-docs/releases/download/v1.0/snapper-delimitation-tutorial-2021.zip ). The command: path/to/beast -threads 6 RunA.xml The error produced: Error 110 parsing the xml input file validate and intialize error: initAndValidate() failed! Index 0 out of bounds for length 0 Error detected about here: <beast> <run spec='beast.inference.PathSampler'> <mcmc id='mcmc' spec='MCMC'> <distribution id='posterior' spec='util.CompoundDistribution'> <distribution id='likelihood' spec='util.CompoundDistribution'> <distribution id='snapperTreeLikelihood.hemi129' spec='snapper.SnapperTreeLikelihood'> — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#3 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ACHLDX6XYWRDCNBAVE4Y3YLUFLILXANCNFSM4Z6V4EOQ> . Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>.

[bmichanderson]

Hi Marnus,

Using SNAPPER 1.0.1 now gives the following error:

java.lang.ClassCastException: snap.Data cannot be cast to snapper.Data
        at snapper.SnapperTreeLikelihood.initAndValidate(Unknown Source)
        at beast.util.XMLParser.initBEASTObjects(Unknown Source)
        at beast.util.XMLParser.parse(Unknown Source)
        at beast.util.XMLParser.parseFile(Unknown Source)
        at beast.app.BeastMCMC.parseArgs(Unknown Source)
        at beast.app.beastapp.BeastMain.main(Unknown Source)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at beast.app.beastapp.BeastLauncher.run(Unknown Source)
        at beast.app.beastapp.BeastLauncher.main(Unknown Source)

Error 110 parsing the xml input file

validate and intialize error: snap.Data cannot be cast to snapper.Data

Error detected about here:
  <beast>
      <run id='mcmc' spec='MCMC'>
          <distribution id='posterior' spec='util.CompoundDistribution'>
              <distribution id='likelihood' spec='util.CompoundDistribution'>
                  <distribution id='snapperTreeLikelihood.hemi129' spec='snapper.SnapperTreeLikelihood'>

[MarnusStoltz]

Hi Ben, Apologies for the late response, but if you use snapper 1.0.1 then you should change all class references from snap.data class to snapper.data in your xml. Happy to help you get it up and running if it doesn't work. Cheers Marnus

…

On Wed, Oct 6, 2021 at 4:48 AM Ben Anderson ***@***.***> wrote: Hi Marnus, Using SNAPPER 1.0.1 now gives the following error: java.lang.ClassCastException: snap.Data cannot be cast to snapper.Data at snapper.SnapperTreeLikelihood.initAndValidate(Unknown Source) at beast.util.XMLParser.initBEASTObjects(Unknown Source) at beast.util.XMLParser.parse(Unknown Source) at beast.util.XMLParser.parseFile(Unknown Source) at beast.app.BeastMCMC.parseArgs(Unknown Source) at beast.app.beastapp.BeastMain.main(Unknown Source) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at beast.app.beastapp.BeastLauncher.run(Unknown Source) at beast.app.beastapp.BeastLauncher.main(Unknown Source) Error 110 parsing the xml input file validate and intialize error: snap.Data cannot be cast to snapper.Data Error detected about here: <beast> <run id='mcmc' spec='MCMC'> <distribution id='posterior' spec='util.CompoundDistribution'> <distribution id='likelihood' spec='util.CompoundDistribution'> <distribution id='snapperTreeLikelihood.hemi129' spec='snapper.SnapperTreeLikelihood'> — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#3 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ACHLDXZBDQ7EK44BSLWW6MDUFO2G7ANCNFSM4Z6V4EOQ> . Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>.

[bmichanderson]

Hi Marnus,

I changed the RunA.xml file from the tutorial at line 71 to spec="snapper.Data" and it no longer has the error for being unable to cast the data. It gives a new error instead (no threads):

java.lang.ArrayIndexOutOfBoundsException: 30
        at snapper.SnapperLikelihoodCore.setLeafPolyFactors(Unknown Source)
        at snapper.SnapperTreeLikelihood.initAndValidate(Unknown Source)
        at beast.util.XMLParser.initBEASTObjects(Unknown Source)
        at beast.util.XMLParser.parse(Unknown Source)
        at beast.util.XMLParser.parseFile(Unknown Source)
        at beast.app.BeastMCMC.parseArgs(Unknown Source)
        at beast.app.beastapp.BeastMain.main(Unknown Source)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at beast.app.beastapp.BeastLauncher.run(Unknown Source)
        at beast.app.beastapp.BeastLauncher.main(Unknown Source)

Error 110 parsing the xml input file

validate and intialize error: 30

Error detected about here:
  <beast>
      <run id='mcmc' spec='MCMC'>
          <distribution id='posterior' spec='util.CompoundDistribution'>
              <distribution id='likelihood' spec='util.CompoundDistribution'>
                  <distribution id='snapperTreeLikelihood.hemi129' spec='snapper.SnapperTreeLikelihood'>

When trying with -threads 8 I get a similar error:

java.lang.ArrayIndexOutOfBoundsException: 30
        at snapper.SnapperLikelihoodCore.setLeafPolyFactors(Unknown Source)
        at snapper.SnapperTreeLikelihood.initAndValidate(Unknown Source)
        at beast.core.BEASTInterface.initByName(Unknown Source)
        at snapper.SnapperTreeLikelihood.initAndValidate(Unknown Source)
        at beast.util.XMLParser.initBEASTObjects(Unknown Source)
        at beast.util.XMLParser.parse(Unknown Source)
        at beast.util.XMLParser.parseFile(Unknown Source)
        at beast.app.BeastMCMC.parseArgs(Unknown Source)
        at beast.app.beastapp.BeastMain.main(Unknown Source)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at beast.app.beastapp.BeastLauncher.run(Unknown Source)
        at beast.app.beastapp.BeastLauncher.main(Unknown Source)
java.lang.RuntimeException: initAndValidate() failed! 30
        at beast.core.BEASTInterface.initByName(Unknown Source)
        at snapper.SnapperTreeLikelihood.initAndValidate(Unknown Source)
        at beast.util.XMLParser.initBEASTObjects(Unknown Source)
        at beast.util.XMLParser.parse(Unknown Source)
        at beast.util.XMLParser.parseFile(Unknown Source)
        at beast.app.BeastMCMC.parseArgs(Unknown Source)
        at beast.app.beastapp.BeastMain.main(Unknown Source)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at beast.app.beastapp.BeastLauncher.run(Unknown Source)
        at beast.app.beastapp.BeastLauncher.main(Unknown Source)

Error 110 parsing the xml input file

validate and intialize error: initAndValidate() failed! 30

Error detected about here:
  <beast>
      <run id='mcmc' spec='MCMC'>
          <distribution id='posterior' spec='util.CompoundDistribution'>
              <distribution id='likelihood' spec='util.CompoundDistribution'>
                  <distribution id='snapperTreeLikelihood.hemi129' spec='snapper.SnapperTreeLikelihood'>

[bmichanderson]

Hi,
Any progress on this bug? Any suggested alternatives?
Cheers,
Ben

[felixgrewe]

Hi,

I tried snapper 1.0.0 and 1.0.2 on Ubuntu Linux and multithreading did not work for me. A single thread (without the -threads option) worked. I would also like to know if you plan on fixing this or if there is any alternative.

Thanks,
Felix

[hwherrmann]

Hello,
I have the same issue with multi-threading SNAPPER. However, when I run it on my Mac the activity monitor shows that Beast is using 25 threads?
With -threads I get the same error messages as above. Is there a fix in sight?

[davidjamesbryant]

HI there,
I was involved in the multi-threading coding, but I'll try and have a look and see what is happening. No promises though...

The multithreading is only a linear-in-number-of-threads speedup (it evaluates different site patterns on different threads), so you can get a comparable speedup by running multiple chains in parallel.

cheers,

David.

[fcsalgado]

Hi David,

If I run multiple chains in parallel as you suggested, how can I calculate the marginal likelihood estimate to compare between models?

Thanks for your help,

Best,

Fabian