Contributing to the UFS/CCPP

Hi, I wanted to introduce our Vulcan climate modeling group to this forum. We have been working with the FV3GFS component of the UFS for over a year in close collaboration with GFDL, and hope to be able to make some contributions upstream (e.g. the CCPP). These edits span several of the "core submodules" in the UFS. For example, we have added diagnostics to FV3, fixed bugs, and plugged-in prognostic machine learning paramaterizations to CCPP. We have developed this work in this repo: https://github.com/VulcanClimateModeling/fv3gfs-fortran, which loosely mimics the top-level structure of https://github.com/ufs-community/ufs-weather-model.

Some of our contributions are probably of interest to the broader community, but aspects of the UFS submodule hierarchy make it challenging to contribute upstream changes: many of the submodules don't have automated tests on github and some cannot be built/tested in isolation.

In particular, I could easily envision contributing to https://github.com/NCAR/ccpp-framework/. It has tests and clear build instructions. On the other hand, https://github.com/NCAR/ccpp-physics does not look like it can be built outside of the context of the UFS of the single column model. Therefore, testing any contribution to this repo would require forking/updating all the parent repos in sequence, before finally building triggering the cmake build from the root of https://github.com/ufs-community/ufs-weather-model. To avoid this laborious process we have turned all submodules into subtrees in our fork, and built a small test suite that checks for bit-for-bit regression of reduced resolution simulations.

How does the core development team approach testing/integration across the sub-module hierarchy? Forgive me if this is already documented someplace.

Also, are there any plans to restructure these sub-modules into independently buildable/testable units. I think that could really accelerate external contributions.

Thanks,

Noah

Hello Noah,

Welcome to the UFS Forum!

You are correct that we do not have a standalone build (nor tests) for the FV3ATM or ccpp-physics repositories and that it would be good to have that. This has been brought up a few times and is rising in priority. 

At this time we use the UFS Weather Model (UFS WM; https://github.com/ufs-community/ufs-weather-model) as our main testing and integration platform. The UFS WM has a set of regression tests that must pass before new code can be committed to the UFS WM itself or its submodules (including ccpp-physics and ccpp-framework).

Here is how we currently handle the submodules. When a PR is submitted to ccpp-physics, an associated PRs must be submitted to FV3ATM (https://github.com/NOAA-EMC/FV3ATM) with the submodule pointing to the ccpp-physics PR. At the same time, an associated PR must be submitted to https://github.com/ufs-community/ufs-weather-model with the submodule pointing to the FV3ATM PR.

Since this may be too complex for some of our contributors, once PRs to ccpp-physics or ccpp-framework are in place, often our team at the Developmental Testbed Center (DTC) takes care of the associated PRs to FV3ATM and the UFS WM. This procedure is described at the UFS WM wiki (https://github.com/ufs-community/ufs-weather-model/wiki/Making-code-changes-in-the-UFS-weather-model-and-its-subcomponents).

Regarding ccpp-physics specifically, we are undergoing a process to refine the code management and I hope that we will be able to answer your questions better once that is more mature. 

As far as additional plans, you may know that NOAA is standing up EPIC, the Earth Prediction Innovation Center, which is envisioned to be the powerhouse for the overall UFS code management and software infrastructure. We are confident that procedures will be more solid once that group has had the chance to spin up.

Please let us know if you have additional questions or suggestions.

Ligia 

 

 

Hi Noah,

It is great that you are asking these questions because they will help us better understand what the broader, non-NOAA, community needs in order to contribute. We really value community contributions (even though some of our practices may not be as friendly as could be).

The UFS Weather Model regression tests currently run on the platforms listed here. As you will see, these are NOAA HPC, the MSU Orion (which is kind of a NOAA machine but is located at a university), and the NCAR Cheyenne. We are currently developing a way to run RTs in the cloud for the purposes of enabling contributors to run their own tests but this is not ready yet. I will look for more information on the status of this development and get back to you.

Regarding the standalone build and tests you have for FV3ATM, I agree that the UFS community could benefit from what you have already developed. I will get in touch with one of the UFS Weather Model code managers for a follow up.

Regarding EPIC, the status is that we expect there will be an award in this quarter to the group that will be in charge of standing up the center. I really do not know how things will turn out, you can find more information at https://wpo.noaa.gov/Programs/EPIC. If you look at the vision statement for EPIC, you will see that its mission is to foster community involvement in developing the UFS. So, my expectation is that the mechanisms for that, ranging from communication forums to code management, will be robust. I believe EPIC will indeed be a platform for open collaboration w/ people working with the UFS.

Ligia

 

Thanks for the reply Ligia. I really just wanted to start a conversation and open a path towards concrete contributions. It sounds like some of these issues have come up before. I am just trying to gauge the appetite for public contributions, and understand how we can more tightly integrate with the community that develops the UFS, rather than just the user community. How can we help?

Here are some follow-ups:

The UFS WM has a set of regression tests that must pass before new code can be committed to the UFS WM itself or its submodules (including ccpp-physics and ccpp-framework).

Is there any way to for external PRs to trigger some of these tests or is this all on private NOAA infrastructure?

You are correct that we do not have a standalone build (nor tests) for the FV3ATM or ccpp-physics repositories and that it would be good to have that.

This would be great. Maybe we could contribute here. We already run a basic set of integration tests for FV3ATM on CircleCI and have experience configuring CI systems like github actions or Circle CI.

As far as additional plans, you may know that NOAA is standing up EPIC, the Earth Prediction Innovation Center, which is envisioned to be the powerhouse for the overall UFS code management and software infrastructure.

Is EPIC intended to replace these public forums as a place where community relevant design decisions are discussed? To be honest, I haven't heard a lot specific information about EPIC since I wasn't invited to the workshop. As an early career scientist, my perception is that this is more of a closed-doors intiative for big organizational decisions and a clearinghouse for grants than a platform for open collaboration w/ people who already have external funding to work on FV3, but I could be wrong.

Thanks again,

Noah

Hi, Noah,

Thanks for reaching out to us on making contributions to UFS WM. UFS is under development, we do hope we can build the model with features to facilitate more developers to make contributions. For your questions:

1) UFS WM has CI test set up through github actions which will be triggered by all the PRs on UFS WM repo. Due to limited cloud resources, currently the CI test only runs the gfs control test, restart test, 32bit test and control test in debug mode. We are working on adding more tests in CI test.

2) UFS model does support platforms besides NOAA R&D machines, the model can be built and run some of the low resolution tests on laptops and desktops. As of now we haven't added those platforms to our regular regression test system yet. There is some discussion on how to support non-tier1/2 platforms in regression test (laptops/desktops or platforms in research community), if you have any suggestions on this, please let us know.

3) As Ligia mentioned, there are some discussions on model sub-component unit test. We have been looking into standalone FV3 driver and FV3 write grid component driver, we are considering setting up a main driver on top of fv3 cap in fv3atm repo so people can conduct fv3atm unit test. I myself don't think this requires too much efforts. I am not sure about CCPP driver though since that requires introducing IO feature into CCPP in order to set up atmosphere state to run tests. Again, we are looking forward to hearing comments you and research community.

Thanks

Jun

I forgot to mention that we do have the input data working for the latest UFS WM on cloud. I thinking maybe we can also put a couple of low resolution canned cases, this way you may set up test cases on your own laptop/desktop from there. If you have your own HPC, sample of porting documentation can be found here

Ligia and Jun. Thanks for the additional info about your testing strategy.

Thanks for pointing out the github actions tests on the top-level repo, Jun. Those kind of simple tests (and hopefully fast) are a good model for the other repos.

We have had success running the model in docker with GNU compilers using the makefiles in the fv3atm and GFDL_atmos_cubed_sphere repos. We haven't explored the cmake build system, but overall, we haven't had any significant problems compiling the model with GNU compilers and building docker images with it.

Regarding the fv3atm tests/builds, we are targeting the main defined here https://github.com/VulcanClimateModeling/fv3gfs-fortran/blob/master/FV3/coupler_main.F90. As I said we are using the old makefiles to build "fv3.exe", but it shouldn't be too hard to switch to cmake. We then perform some very simple regression tests at C16 resolution in Circle CI. 

At the start of our collaboration, GFDL provided input data to us which we have been storing in google cloud storage. We have written a tool called fv3config to setup regression cases based on data stored in google cloud storage. fv3config allows building run directories from yaml files like this. It's a fairly robust tool at this point, and helps deal with some of the corner cases involved in setting up run directories (e.g. setting the model start time in multiple locations, changing all the flags needed to do restarts, etc).

Re ccpp-physics build/test. Did you consider using the single column model as a regression test of ccpp-physics? It seems like the most minimal build of the physics suite currently available, and could be bundled into the ccpp-physics repo. For instance, any PR to ccpp-physics could run the single column and make some plots for reviewers to examine.

Hi Noah,

For CCPP, we have indeed considered using the SCM as a simple host to exercise the PRs and obtain preliminary results. We could gather information on whether the PRs change the results of the tested suites. And if there are changes, we could get an idea of their characteristics. We just have not had the time to do that yet. Collaboration on this front would be of interest.

 

Ligia