Add a cmake or autotools build system?

[edwardhartnett]

Is there any interest in a CMake or Autotools build system?

I converted the FMS from mkmf to Autotools - I have a python script that does most of the heavy lifting.

We could also do a CMake build, which is what we use for everything else in the UFS.

We are installing this as part of the UFS at NOAA, and would greatly prefer to have a standard build system. I'm willing to help get there...

[marshallward]

We have an autotools build system which is documented here:

https://github.com/NOAA-GFDL/MOM6/blob/dev/gfdl/ac/README.md

It does not exactly follow usual autoconf builds:

• It only uses autoconf, not automake or libtool
• It still relies on mkmf to determine the dependencies
• configuration resides in a separate ac directory, rather than top-level

But overall we feel it suits our current workflows.

We would like to integrate this better into coupled builds so that is somewhere we could work together.

A CMake build was developed by @aerorahul but I believe there were a few blockers to integration (possibly FMS2 migration, but I can't recall the details). Hopefully he can provide an update here.

We would like to support both autotool and CMake builds.

[aerorahul]

A standalone MOM6 cmake build that uses the latest FMS cmake from NOAA-GFDL is still there and can be resurrected quickly enough.
I have not kept up with the updates in the MOM6 repository pertaining to config_src/ and others.
Also, I have not paid attention to the coupled applications such as with SIS2 and others as configured via MOM6-examples/src. It shouldn't be too difficult with CMake to bring that into the fold.

I had given a presentation and demo of CMake with MOM6 on this topic on July 1, 2020.

[edwardhartnett]

The point of an autotools build is to follow the usual rules. ;-)

How about we upgrade this to not use mkmf.

I am very familiar with mkmf, as I generated the FMS build. mkmf is an old-timey tool, not fit for purpose in the modern world. I understand your Fortran programmers are comfortable with it, but it also forces many bad choices on your build, and it's really holding you back. Retire mkmf, as a tool that has done a fine job for many years. It is time to move on.

The FMS team made this transition and is happy with it. I'm happy to help by giving you a working set of Makefile.am files, which I can generate with my python script.

Can we also get @aerorahul 's CMake build merged? That would be very helpful. What blocked its adoption?

Building the UFS stack with a bunch of components which don't test for their dependent libraries is unstable. We need MOM6 to keep up with our build requirements, and that means CMake and/or autotools.

[marshallward]

I'm afraid we're not quite yet ready to abandon mkmf.

We found that the automake build of FMS is much slower than makefiles generated by mkmf, and it also eliminates the burden of maintaining Makefile.am lists. Nor do I see any evidence that mkmf is holding us back, or influencing the development of the model.

So we have no plan to retire mkmf until we see evidence that it will improve our builds. I think we've produced an effective synthesis of autoconf and mkmf, and I would encourage you to look before passing judgement.

If you'd like to discuss this in person, then feel free to reach out to me outside of the issue.

[edwardhartnett]

Your automake build slowness is probably a bug in the Makefile.am files which can be found and fixed. There's no reason your make files should be faster.

mkmf is holding you back because it requires one module file in each directory, and then causes your programmers to put their F90 code into include files, instead of F90 files. At least, this was the case with FMS. This makes it harder for code analysis tools to work (for example, I never did figure out how to turn on code coverage analysis for FMS, even though that's trivial in a normal build system).

I have absolutely looked at and worked with mkmf more than I wanted to or ever plan to again. When I say that it is a legacy tool that has served beyond its time, I am not rushing to judgement. ;-)

[marshallward]

To answer the original question: Yes, we do have an autotools build, and a CMake build does exist although we do not maintain it.

There seem to be followup questions as to whether the current available options are suited to UFS, and if a UFS-compatible solution should reside in this repository, but we need more information to answer these questions.

I think discussions about the relative merits of mkmf vs automake should take a backseat for now. (Again, I invite you to reach out to me if you want to discuss our design decisions.)

Given that the scope of this issue is starting to shift in unpredictable ways, I'm going to move it to discussions where we can hash out ideas and plans. As specific issues arise, we can open new issues.

[adcroft]

We've discussed the build process quite a lot recently. One challenge is that many people think MOM6 could be built like a library but it cannot. The MOM6 object files depend on the optional software being compiled and MOM6 sits between that optional software and the coupler that the coupler also knows about. This is not how we would design things now but it is the way the original models were set up. Personally, I would like to be compatible with multiple build strategies and I do not think we will ever adopt just one. The current autotools is in ac/ partly because we don't want to promote it as the preferred solution. Nor does it address the context problem. We have indeed had a few submissions of cmake etc but the solutions have been for someone's configuration and not configuration agnostic - i.e. it didn't build some of our own configurations. UFS is only one application and adding a cmake specifically to assist the UFS stack is very much at odds with what MOM6 is about. We've had the same debate with several institutions so far. We do agree the build process can be better encapsulated but in our discussions of possible solutions the context problem always gets in the way of it working for everyone.

For what it's worth, the mkmf approach is working for everyone who doesn't have root access to upgrade/install cmake. I also think it's valuable to expose to users what the build process is doing, rather than provide a black box that just works by magic. The current instructions for building do use mkmf, but they also try to explain what is happening and you can build quite easily without mkmf, and many people do.

[edwardhartnett]

@aerorahul how do we build MOM6 now? Did we create a wrapper project with your CMake build?

@adcroft I have heard of the unusual build problems of MOM6 for some time. I don't think you will solve them with home-rolled build tools. Instead I suggest you adopt more standard solutions.

[edwardhartnett]

Your build problems can be solved in both CMake and autotools. For example, the netcdf-c build works based on the optional presence of various packages. Is that what you mean?

[adcroft]

Not quite. The build process needs to be able to bring in source code from alternative places, and MOM6 can not possibly know what all those alternative places might be. For example, we have stubs for BGC and the NCAR and GFDL BGC codes have completely unrelated source trees. We can't anticipate all the possible combinations of source codes. Only the user putting all the different source codes together can know where and what everything is and so they must provide the search paths. It therefore lives outside of MOM6.

[edwardhartnett]

I believe you guys have backed yourselves into a corner with questionable build choices. You have convinced yourselves that this is inevitable, but I remain unconvinced.

I suspect that your build problems can be readily solved, but to do that you must abandon the hubris of thinking you guys can write a better version of automake, and instead start using the standard tools correctly. That means, instead of being so original, doing things the same way everyone else does them.

Many other software packages deal with the optional presence or absence of other software. The solutions are there, if you care to use them.

Until this is resolved, UFS and other applications will have to do the heavy lifting of managing the build of MOM6. It's a shame that this software is being hampered by something as pedestrian as a poor build system.

[klindsay28]

I think that this discussion would be far more productive if it focused on internal and external requirements of MOM's build system, and pros and cons of different approaches to building, and if personal attacks were omitted.

[edwardhartnett]

I apologize if that came across as a personal attack. That was not what was intended.

I don't think that MOM6 is going to meet modern build expectations using home-rolled tools. That's an engineering judgement, formed after spending some time working with mkmf. Nor do I think it is a controversial judgement - most software engineers would say the same.

You guys let me know if you change your mind - until then I will focus on the 20 or so NOAA libraries and tools I already have to worry about building. ;-)