Turbsim_OpenMp issue

[DT4015]

Bug description
When running the precompiled version of turbsim_openMP to generate inflow for a 16 turbine wind farm it gives the following error message: CalcFourierCoeffs:CalcFourierCoeffs_IEC:Error in Cholesky factorization: LAPACK_SPOTRF: Leading
minor of order 131 of A is not positive definite, so Cholesky factorization could not be
completed.
The error occurred in the u-component coherence matrix at frequency 64 (8.00000E-02 Hz) Check the
input file for invalid physical properties or modify the coherence.
CalcFourierCoeffs_IEC:Error in Cholesky factorization: LAPACK_SPOTRF: Leading minor of order
131 of A is not positive definite, so Cholesky factorization could not be completed.
The error occurred in the u-component coherence matrix at frequency 127 (0.15875 Hz) Check the
input file for invalid physical properties or modify the coherence.
CalcFourierCoeffs_IEC:Error in Cholesky factorization: LAPACK_SPOTRF: Leading minor of order
95 of A is not positive definite, so Cholesky factorization could not be completed.
The error occurred in the u-component coherence matrix at frequency 253 (0.31625 Hz) Check the
input file for invali

Aborting TurbSim.

When running the same input file in the non openMP precompiled version it runs no problem

To Reproduce
I have attached the input file
Farm_4000_12.txt

OpenFAST Version
v5.0.0

**************************************************************************************************
 OpenFAST

 Copyright (C)  National Renewable Energy Laboratory
 Copyright (C)  Envision Energy USA LTD

 This program is licensed under Apache License Version 2.0 and comes with ABSOLUTELY NO WARRANTY.
 See the "LICENSE" file distributed with this software for details.
 **************************************************************************************************

 OpenFAST-v2.0.0
 Compile Info:
  - Architecture: 64 bit
  - Precision: double
  - Date: Nov 27 2018
  - Time: 17:19:38
 Execution Info:
  - Date: 11/29/2018
  - Time: 10:52:28-0700

[andrew-platt]

Thanks for bringing this to our attention! We'll take a look and find a solution.

Your screenshot of the OpenFAST version shows v2.0.0, but you state 5.0.0 in the comment. Which version are you using?

[DT4015]

Thank you! Definitely using 5.0.0, I forgot to remove the pre-attached screenshot from the bug report template

[andrew-platt]

Can you try using TurbSim from the v4.2 release and let me know if that results in an error? The LAPACK_SPOTRF routine was added in the v5.0.0 release and may be causing issues.

[DT4015]

Using Turbsim V4.1.2 i was able to make a similair grid but with a 3000m width at a 10m eresolution for 800s no problem, I'll try using turbsim 4.2 and using the same course grid I used to compare the new versions

[DT4015]

Can confirm V4.2 works without issue also!

[bjonkman]

FYI: The matrix in the Cholesky factorization is dependent on the number of grid points and grid resolution (among other quantities), so if you change those values in the input file and then run it on a different version of TurbSim, you can't necessarily conclude that one version works and the other doesn't. You should use equivalent input files for all versions.

[DT4015]

Hi bonnie, I compared the two versions of Turbsim using the same input file which I included in my initial post. In addition to that comparison i also tried to create files which I had already done and knew worked from an older release of turbsim.

[IrisMeasure]

I compiled the TurbSim_OpenMP included with OpenFAST v5.0.0, except that I reverted LAPACK_potrf back to LAPACK_pptrf. The error messages are as follows:

CalcFourierCoeffs:CalcFourierCoeffs_IEC:Error in Cholesky factorization: LAPACK_SPPTRF: Leading
minor order          641 of A is not positive definite, so Cholesky factorization could not be
completed.
The error occurred in the u-component coherence matrix at frequency 126 (0.1575 Hz) Check the
input file for invalid physical properties or modify the coherence.
CalcFourierCoeffs_IEC:Error in Cholesky factorization: LAPACK_SPPTRF: Leading minor order
387 of A is not positive definite, so Cholesky factorization could not be completed.
The error occurred in the u-component coherence matrix at frequency 3126 (3.9075 Hz) Check the
input file for invalid physical properties or modify the coherence.
CalcFourierCoeffs_IEC:Error in Cholesky factorization: LAPACK_SPPTRF: Leading minor order
513 of A is not positive definite, so Cholesky factorization could not be completed.
The error occurred in the u-component coherence matrix at frequency 3626 (4.5325 Hz) Check the
input file for invalid physical pr

Aborting TurbSim.

 CalcFourierCoeffs:CalcFourierCoeffs_IEC:Error in Cholesky factorization: LAPACK_SPPTRF: Leading
minor order          641 of A is not positive definite, so Cholesky factorization could not be
completed.
The error occurred in the u-component coherence matrix at frequency 1501 (1.8763 Hz) Check the
input file for invalid physical properties or modify the coherence.
CalcFourierCoeffs_IEC:Error in Cholesky factorization: LAPACK_SPPTRF: Leading minor order
641 of A is not positive definite, so Cholesky factorization could not be completed.
The error occurred in the u-component coherence matrix at frequency 1126 (1.4075 Hz) Check the
input file for invalid physical properties or modify the coherence.
CalcFourierCoeffs_IEC:Error in Cholesky factorization: LAPACK_SPPTRF: Leading minor order
641 of A is not positive definite, so Cholesky factorization could not be completed.
The error occurred in the u-component coherence matrix at frequency 3501 (4.3762 Hz) Check the
input file for invalid physical p


Aborting TurbSim.

This suggests that the issue may not be solely related to the xPPTRF -> xPOTRF change, but rather to numerical behavior under parallel execution.
The issue does not occur when running with OMP_NUM_THREADS=1.
The varying frequencies at which the errors occur across runs indicate non-deterministic behavior, which is consistent with floating-point differences in parallel computations.

[andrew-platt]

Thank you for checking this @IrisMeasure!

I'm unable to download your input file. Can you attach it or put it inline in a comment? I'll look into this issue.

[IrisMeasure]

@andrew-platt
Just to clarify, I used the input file provided in the original post, not a separate one.