FAST Farm: Spatial-averaged ambient wind speed in the low-resolution domain is zero

[MYMahfouz]

Hi,

I am getting this error when i am using FAST_Farm_OMP but I do not get it when I use the normal FAST_Farm version.

FARM_InitialCO:AWAE_CalcOutput:LowResGridCalcOutput:The magnitude of the spatial-averaged ambient
wind speed in the low-resolution domain associated with the wake plane at the rotor disk for
turbine #1 is zero.

• What does this error mean?

• What can be causing it?

• Why do I get it in the OMP version only?

Thank you.

BR,
Youssef

[jjonkman]

Dear @MYMahfouz,

This error means that the denominator computed in the following equation is zero, such that the ambient turbulence intensity at the rotor of turbine 1 cannot be calculated: https://openfast.readthedocs.io/en/main/source/user/fast.farm/FFarmTheory.html#equation-eq-ti. I would only expect this error to be triggered if the inflow is ill defined (such as with a zero mean wind speed) or if the wind turbine is yawed out of the wind. Can you clarify your simulation set up?

I would generally expect that the serial and OpenMP parallelized versions of FAST.Farm to calculate the same response, albeit with the OpenMP version having a faster time to solution. I'm not sure what would cause otherwise. Have you compiled FAST.Farm yourself, or are you using the precompiled FAST.Farm binaries provided by NREL?

Best regards,

[andrew-platt]

@MYMahfouz That's really interesting that moving the files to the same location fixes the issue! I think this is a slightly different issue than @bjonkman mentioned. Just to clarify, which files did you have to relocate (all files, or just the wind files)?

Thanks for the version info (commit 6b8706b75698cb1c97c6a8e3e9f42538a63fa71b is version 3.5.0). That helps me in investigating the root cause of the issue.

At the moment I don't know why OpenMP would cause a file reading issue that requires all the files be on the same path. It might take a little bit to track down the root cause. I would like to get this fixed in our next release (3.5.1) -- hopefully in the next couple of weeks.

[MYMahfouz]

@andrew-platt At the beginning I copied all files to the same path. Now I only copied the wind files to the same path and this also works fine.

Just a comment, from the example I ran before and was working with the OpenMP, The wind files where not in the same path but where only one folder up in the path. Now for my wind files in the model which didn't work, I gave an absolute path, which was a much longer string.
I will test it now with shorter path or also just one folder away from the current path.

I also didn't get any errors that the wind file was not found, and from memory usage I feel the file was read normally.

I am using a Mann box wind field if this also makes a difference I do not know.

I would be willing to test this again or help in any way I can.

Thanks.

Best regards,

[andrew-platt]

@MYMahfouz, thanks for the additional info. How many characters was the longest pathname (including the filename) to the wind files when you used the absolute path? I'm not sure why this would change the behavior when using OpenMP though.

I don't think it should matter what type of wind file it is.

Regards,
Andy

[bjonkman]

If this is a timing issue with the file unit number, I would expect the behavior to be somewhat random.

[MYMahfouz]

@andrew-platt These two paths didn't work:
../Wind/Vhub10_TI6_PL0.2_Time1500s_u.bin
D:/data/40_mahfouz/FAST_Farm_template/Wind/MannBox/Farm_9T/Windir_p00/Seed_0/Vhub10_TI6_PL0.2_Time1500s_u.bin

While this path worked all the time
Wind/Vhub10_TI6_PL0.2_Time1500s_w.bin

and also when just in the same path as inflow wind file

Best regards,
Youssef

[MYMahfouz]

Dear @jjonkman,

Sorry to bother you again.

I am trying to run this same floating wind farm now but using wind fields generated by TurbSim changing Mod_AmbWind to 3.

I am struggling to make the model work. I wanted to ask If there is a working example that I can follow similar to these examples https://github.com/OpenFAST/r-test/tree/main/glue-codes/fast-farm .

I keep getting this error and I believe my wind fields are big enough so I do not get why.

Farm_Initialize:AWAE_Init:AWAE_UpdateStates:InflowWind_CalcOutput:CalculateOutput:IfW_FlowField_Ge
tVelAcc:Grid3DField_GetCell: GF wind array boundaries violated: Grid too small in Y direction.
Y=-501.32; Y boundaries = [-480, 480]

Thanks.

Best regards,

[andrew-platt]

@MYMahfouz
Thanks for the additional observations! Your observation supports my suspicion that there is an offset that should be applied to the high resolution grid, but isn't getting applied correctly in v3.5.0. I'm not certain where in the code this would be, but have some ideas where to look.

[andrew-platt]

@MYMahfouz
I think I found the issue and posted a possible fix on my branch here: https://github.com/andrew-platt/openfast/tree/b/FF_mod_ambwind3 (this branch has a few minor fixes that will go into 3.5.1, but no input file changes from 3.5.0). Could you give this a try and see if it fixes the issue for you?

[MYMahfouz]

@andrew-platt yes this fixed the problem. Thank you.

However, I had to move the high resolution grid to have the turbine exactly in the center but only in the y-direction. This was not needed when I used v3. In my very limited experience and knowledge about FAST Farm, this increases the computational time as now the low and high resolution grid nodes no longer coincide. Is this true?

Another question and this was also the same when I tried running the same case with v3. I have to move all the .fst files to the same path of the fstf file. Is this done on purpose? Can this be avoided?

Sorry for keeping to bother you all with my questions, I am new to FAST Farm and I am trying to learn and understand.

Best regards,
Youssef

[jjonkman]

Dear @MYMahfouz,

Regarding your first question, with Mod_AmbWind = 3, the low- and high-resolution domains are independent and the grid points need not align. And because InflowWind always centers the wind domain about the tower centerline in the Y-direction, it is also important to center the high-resolution domain about the tower centerline in the Y-direction (because the domains in FAST.Farm should generally be aligned those used by InflowWind.

Best regards,

[MYMahfouz]

Dear @jjonkman,

Thanks for the explanation it is much clearer now to me.

Best regards,

[MYMahfouz]

Dear @jjonkman,

I have another question for the OpenMP, is there a way to decide the number of cores the OpenMP is using? It currently aims to use all the available cores in the machine it is running on. Is there a way to decrease this or define the number of cores before hand?

Best regards,

[andrew-platt]

I believe this is set at runtime using environment variables.

For linux: https://www.intel.com/content/www/us/en/docs/onemkl/developer-guide-linux/2023-1/set-num-threads-using-openmp-environment-variable.html

I think Windows is similar, though I'm not familiar with the syntax required there.

[rthedin]

Alternatively, if running on a cluster, you can also request less cores per task directly with the scheduler. For slurm, you can do something like srun -c18 <fastfarmbinary> <inputfile> if you want to use 18 cores. For your batch scripts, I believe you can bind process to cores in a similar way.

[MYMahfouz]

Thank you. I will try this out.

Best regards,

[MYMahfouz]

For intel visual studio command prompt this command worked for me. I will just leave it here in case someone else asks a similar question in the future.
set OMP_NUM_THREADS=9

[MYMahfouz]

Dear @jjonkman,

I am running now a couple of simulations with the MannBox same wind field as an input.

I got this error:

FARM_UpdateStates:AWAE_UpdateStates:InflowWind_CalcOutput:InflowWind_GetSpatialAverage:CalculateOu
tput:IfW_FlowField_GetVelAcc:Grid3DField_GetCell: Error: GF wind array was exhausted at 1212.1
seconds (trying to access data at 860.32 seconds).

I believe for the MannBox FAST.Farm assumes it is periodic, so why is this error happening? Also I used this windfield for the same farm just rotated to simulate different wind directions and didn't receive this error.

What should I do to fix this?

Thank you.

Best regards,

[MYMahfouz]

Dear all,

Sorry that I keep asking questions over and over, and thank you for the help and your time.

I am running a 9 turbine floating wind turbine with 16 wind directions. For every wind direction, I rotate the farm and the mooring system while my settings in inflowwind is always at zero.

I use Mann turbulent box generated wind fields.

I was running 16 wind directions one of the gave the error i mentioned above the rest of the 15 wind directions kept running normally.

I was running simulations for 4200s and all the turbines for all simulations for all wind directions got out of the high resolution grid at the same exact second, at 4132s.

I have checked all my turbines motion, and there is nothing wrong happening there. I cannot find a reason why I got the error that they all went out of the grid. I also do not know what to check. I also find it weird that all the turbines in all simulations get out of the grid at the exact same time step.

I do not know what to check or what might be causing this error.

I am currently using FAST.Farm v3.5.0 (the most recent version from the main repository), should I switch to an earlier version? Which version do you recommend?

I am looking forward to your reply. Thanks a lot. Please find the error I get below.

T1:FARM_UpdateStates:FWrap_Increment:FAST_Solution:FAST_AdvanceStates:SolveOption2c_Inp2AD_SrvD:In
flowWind_CalcOutput:CalculateOutput:IfW_FlowField_GetVelAcc:Grid4DField_GetVel:Outside the grid
bounds.
InflowWind_CalcOutput:InflowWind_GetSpatialAverage:CalculateOutput:IfW_FlowField_GetVelAcc:Grid4DF
ield_GetVel:Outside the grid bounds.
InflowWind_CalcOutput:InflowWind_GetSpatialAverage:CalculateOutput:IfW_FlowField_GetVelAcc:Grid4DF
ield_GetVel:Outside the grid bounds.
T2:FARM_UpdateStates:FWrap_Increment:FAST_Solution:FAST_AdvanceStates:SolveOption2c_Inp2AD_SrvD:In
flowWind_CalcOutput:CalculateOutput:IfW_FlowField_GetVelAcc:Grid4DField_GetVel:Outside the grid
bounds.
InflowWind_CalcOutput:InflowWind_GetSpatialAverage:CalculateOutput:IfW_FlowField_GetVelAcc:Grid4DF
ield_GetVel:Outside the grid bounds.
InflowWind_CalcOutput:InflowWind_GetSpatialAverage:CalculateOutput:IfW_FlowField_GetVelAcc:Grid4DF
ield_GetVel:Outside the grid bounds.
T3:FARM_UpdateStates:FWrap_Increment:FAST_Solution:FAST_AdvanceStates:SolveOptio

Best regards,

[andrew-platt]

This seems suspiciously like we lost the Periodic flag on the HAWC wind type during the refactoring. [Edit: after reviewing the code, it looks like it is set correctly] [Edit 2: I misunderstood that the bounds error is actually occurring in 3.4.1, so it is clearly not a refactoring issue]

As an aside, my understanding is that the Mann boxes are actually periodic in all 3 dimensions. In InflowWind we only consider the X/time dimension as periodic. Perhaps periodicity in the other dimensions for Mann box winds is something we can add later.

[MYMahfouz]

@andrew-platt I am unsure why this error happened in v3.5 with the wind field is exhausted but as I mentioned I got this with the OpenFAST standalone as well as FAST Farm v3.5.

Yes, also as far as I understand the Mann boxes are periodic in the three dimensions. It is basically like a lego cube/cuboid as far as I know. Adding the capability that the Mann box is periodic in all directions would be great as it would allow even simulating much bigger wind farms easier, as also there are memory limitations on creating the Mann boxes.

However, for the error I have now with older versions it seems that the turbines are out of the wind field in the time dimension? Why would this happen? How can I avoid these errors?

[andrew-platt]

@MYMahfouz, from the print statements and error outputs you added to your code, it is the time dimension that is causing issues. I have a working theory right now that the periodic nature of the Mann box is somehow lost when Mod_AmbWind=2.

When Mod_AmbWind=2, one instance of InflowWind is read in by the AWAE module where it is stored as 3D data (HAWCWind in v 3.4.1, or Grid3D in v 3.5). For each turbine, a small wind box around the turbine is extracted and passed as 4Dext data to that turbine (hence the IfW_4Dext_CalcOutput in the error). The 4Dext` data is not periodic, but is instead a small volume of wind data in a small time window. So if there is any error in the calculated time index (calculated based on a start time for the 4D data (single prec), the current time (double prec), and DT (double prec)), this could result in an out of bounds error. Given when in the simulation this occurs, maybe some precision differences (double vs. single) cause this index to be incorrect after running for a long time. If my theory is correct, this would explain why all turbines throw the error at the same timestep.

@MYMahfouz, what is the DT you are using for the individual turbines?

[MYMahfouz]

@andrew-platt The DT for a single turbine is 0.0378s. My FAST Farm DT is 100 times larger so it is 3.78s

What is the duration of the small window passed to the single turbines? Is it like a fixed value?

[andrew-platt]

I think we now have all this resolved with pull request #1684.