Error when running regional_esg_grid

Submitted by leuthold on Fri, 10/01/2021 - 21:04
Forums
Initialization

Total Noob with UFS here, but lots of WRF experience.  Also, I don't know if I should post here or in the SRW  forum?  Anyways, I've built everything for SRW successfully and am using the standalone scripts to run the preprocessing workflow (ufs-srweather-app/regional_workflow/ush/wrappers), but when running run_make_grid.sh/regional_esg_grid, I get a "error Invalid argument".  The message output previous looks like this...

arcx, arcy  24.0569 14.5016
For lam=    0.80 the best [smallest possible]
optimality criterion, Q, for this domain:  0.453511E-04
The corresponding optimal A and K:   0.1156  -0.3476
The corresponding m_arcx,y:  0.4167189652257E+00  0.2524146128930E+00
x and y central grid resolution in map units:  0.19473E-02  0.19567E-02
Get additional diagnostics from hgrid_ak_rr
  check netcdf status=          22
error Invalid argument
Stopped

For whatever reason, the regional_grid.nc file doesn't get created.  Any help would be appreciated as I've been stuck on this for a few days.  Thanks,

Mike

Permalink

Submitted by jbeck on Fri, 10/01/2021 - 22:00

Hey Mike,

This is the relevant block of code in regional_esg_grid.f90:

use netcdf
integer,intent(in) :: status
!
if(status /= nf90_noerr) then
  write(0,*)' check netcdf status=',status
  write(0,'("error ", a)')trim(nf90_strerror(status))
  stop "Stopped"
endif

This is the "check" subroutine which verifies the output of regional_esg_grid, so it looks like something went wrong with netCDF.  We may need to get the author of this code involved, but before doing that, can you increase the number of nodes/cores you are providing this job?  It could be a memory issue.

Thanks.

 

Permalink

Submitted by kavulich on Mon, 10/04/2021 - 04:14

Mike, can you also let us know what version of the code you are running? Is this the latest release (v1.0.1)?

Permalink

Submitted by leuthold on Mon, 10/04/2021 - 19:57

I think it's the most recent.  From ufs-srweather-app/Externals.cfg

[regional_workflow]
protocol = git
repo_url = https://github.com/NOAA-EMC/regional_workflow
tag = ufs-v1.0.1
local_path = regional_workflow
required = True

[ufs_utils]
protocol = git
repo_url = https://github.com/NOAA-EMC/UFS_UTILS
tag = ufs-v2.0.0
local_path = src/UFS_UTILS
required = True

[ufs_weather_model]
protocol = git
repo_url = https://github.com/ufs-community/ufs-weather-model
tag = ufs-v2.0.0
local_path = src/ufs_weather_model
required = True

[EMC_post]
protocol = git
repo_url = https://github.com/NOAA-EMC/EMC_post
tag = upp_v9.0.1
local_path = src/EMC_post
required = True

[externals_description]
schema_version = 1.0.0

 

 

Permalink

Submitted by leuthold on Mon, 10/04/2021 - 21:18

I tried it using 24 cores on one of the compute nodes as well as 2 nodes.  No luck.  My installation is not a Level 1 or 2, but a New Platform, so there is a (good?)chance that I've not set it up correctly.  I followed the instructions to change all the regional_workflow/scripts to use mpirun, which seems to work as the /ufs-srweather-app/bin/regional_esg_grid exe did run, as you referenced above.

thank you.

Mike

 

Permalink

Submitted by jbeck on Tue, 10/05/2021 - 15:50

Mike,

Can you check the rest of your log file for other warnings or errors that occur prior to this netCDF check?  There may be additional information above the error you ran into that show problems with the system setup, for example.

Thanks.

Permalink

Submitted by gerard.ketefian on Tue, 10/05/2021 - 18:33

Hi Mike,

To better pinpoint where in the code the error is occurring, you can try putting in some print statements before/between/after the

call check(...)

statements starting on line ~111 of regional_esg_grid.f90.  Maybe do that for the first batch of those statements and see if the problem is there; if not, continue to the next batch, etc.  I suspect it might be the very first one, i.e.

call check( nf90_create("regional_grid.nc", NF90_64BIT_OFFSET, ncid) )

maybe due to the wrong version of NetCDF.  

Gerard

Permalink

Submitted by leuthold on Fri, 10/08/2021 - 17:02

Gerard,

You were right about the version of NetCDF.  I put in a call to get the version and build date, and it's from my WRF install!

 4.1.3 of Jul  1 2020 11:48:44

I think the mistake was that I didn't reset the NETCDF env variable to the NCEPLIBS_external NetCDF.  Once I fix that, I would assume I have to start over and build NCEPLIBS again, and then ufs-srweather-app. 

Thanks a lot!

Mike

 

Permalink

Submitted by gerard.ketefian on Fri, 10/08/2021 - 17:29

Good, sounds like progress!  I would guess you're right about rebuilding NCEPLIBS, but I'm not 100% sure since I usually use a pre-compiled version of NCEPLIBS.  Give that a shot and if it doesn't work, let us know.  There are a couple of others on our team that can guide you through that part.

Gerard

Permalink

Submitted by leuthold on Fri, 10/08/2021 - 19:22

So I resolved the issues with NetCDF, but now a new problem crops up when doing 'make' in NCEPLIBS.  I'm considering deleting NCEPLIBS and NCEPLIBS-external and starting over from scratch as I must have screwed something up.

thanks.

Mike

 

-- Could NOT find JPEG (missing: JPEG_LIBRARY JPEG_INCLUDE_DIR)
CMake Error at /cm/shared/apps/cmake/gcc/3.18.0/share/cmake-3.18/Modules/FindPackageHandleStandardArgs.cmake:165 (message):
  Could NOT find Jasper (missing: JASPER_LIBRARIES JASPER_INCLUDE_DIR
  JPEG_LIBRARIES)
Call Stack (most recent call first):
  /cm/shared/apps/cmake/gcc/3.18.0/share/cmake-3.18/Modules/FindPackageHandleStandardArgs.cmake:458 (_FPHSA_FAILURE_MESSAGE)
  /cm/shared/apps/cmake/gcc/3.18.0/share/cmake-3.18/Modules/FindJasper.cmake:39 (FIND_PACKAGE_HANDLE_STANDARD_ARGS)
  CMakeLists.txt:25 (find_package)


-- Configuring incomplete, errors occurred!
See also "/ipchome/mleuthold/NCEPLIBS-ufs-v2.0.0/src/NCEPLIBS/g2/src/g2-build/CMakeFiles/CMakeOutput.log".
make[2]: *** [g2/src/g2-stamp/g2-configure] Error 1
make[1]: *** [CMakeFiles/g2.dir/all] Error 2
make: *** [all] Error 2

 

Permalink

Submitted by gerard.ketefian on Fri, 10/08/2021 - 20:48

Hi Mike,

I asked a couple of my colleagues who know more about building NCEPLIBS to take a look.  Hopefully they'll get around to it soon.  In the meantime, please let us know if starting over worked.

Thanks,
Gerard

Permalink

Submitted by kavulich on Wed, 10/13/2021 - 15:54

Mike, if you have not done so already, it may be worth re-building from scratch...if parts of the NCEPLIBS were built with a mis-match of libraries (some from NCEPLIBS-external, and some from your WRF build) then there may be complicated problems with the libraries that would be most quickly solved by a "nuke-and-rebuild" approach. 

Out of curiosity's sake, does your machine make use of modules, or was the environment for NCEPLIBS set up with the "setenv_nceplibs.sh" script?

Permalink

Submitted by leuthold on Wed, 10/13/2021 - 16:56

That is my plan, once I get some time and the power comes back on!  I live in rural Arizona and we had a big wind storm two nights ago. 

I used modules for NCEPLIBS.

Thanks for your reply,

Mike

Permalink

Submitted by leuthold on Fri, 10/29/2021 - 16:50

I was able to make some more progress as I did what you suggested and rebuilt NCEPLIBS from scratch.  I've been able to compile much of the model, but am now stuck at  FV3/CMakeFiles/fv3atm.dir/module_fcst_grid_comp.F90.

[  4%] Performing build step for 'ufs_weather_model'
[  1%] Built target ccpp
[ 54%] Built target ccppphys
[ 72%] Built target fms
[ 80%] Built target stochastic_physics
[ 83%] Built target gfsphysics
[ 83%] Built target ipd
[ 92%] Built target fv3dycore
[ 93%] Built target stochastic_physics_wrapper
[ 93%] Built target cpl
[ 96%] Built target io
[ 96%] Built target ccppdriver
[ 96%] Building Fortran object FV3/CMakeFiles/fv3atm.dir/module_fcst_grid_comp.F90.o
module_fcst_grid_comp.F90(485): #error: #if: syntax error.
make[5]: *** [FV3/CMakeFiles/fv3atm.dir/module_fcst_grid_comp.F90.o] Error 1
make[4]: *** [FV3/CMakeFiles/fv3atm.dir/all] Error 2
make[3]: *** [all] Error 2
make[2]: *** [src/ufs_weather_model/src/ufs_weather_model-stamp/ufs_weather_model-build] Error 2
make[1]: *** [src/CMakeFiles/ufs_weather_model.dir/all] Error 2
make: *** [all] Error 2

The line is

 

#if ESMF_VERSION_MAJOR >= 8

Two steps forward, one back.

Thanks.

Mike

 

Permalink

Submitted by kavulich on Fri, 10/29/2021 - 17:14

Mike,

Can you give some information about your machine? I'm especially interested in compiler versions, as this seems like an issue that might be related to older or perhaps very new compiler versions that we have not tested yet.

Permalink

Submitted by leuthold on Fri, 10/29/2021 - 17:19

Linux borah-ldm001 3.10.0-1160.42.2.el7.x86_64 #1 SMP Tue Sep 7 14:49:57 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux

[mleuthold@borah-ldm001 ufs-srweather-app-ufs-v1.0.1]$ module list
Currently Loaded Modulefiles:
 1) intel/mpi/64/2021/2.0.2883        8) bacio/2.4.1   15) ip2/1.1.2          22) sp/2.3.3
 2) intel/compiler/64/2021/2.0.2883   9) bufr/11.4.0   16) landsfcutil/2.4.1  23) w3emc/2.7.3
 3) gcc/9.2.0                        10) crtm/2.3.0    17) nceppost/dceca26   24) w3nco/2.4.1
 4) rocoto/1.3.4                     11) g2/3.4.1      18) nemsio/2.5.2       25) wgrib2/2.0.8
 5) slurm/slurm/19.05.8              12) g2tmpl/1.9.1  19) nemsiogfs/2.5.3    26) wrf_io/1.1.1
 6) nco/4.9.3                        13) gfsio/1.4.1   20) sfcio/1.4.1        27) NCEPLIBS/2.0.0
 7) cmake/gcc8/3.18.0                14) ip/3.3.3      21) sigio/2.3.2

 

  • 92 views

Copyright © 2023. All rights reserved.