Support C++20 Modules

[PikachuHyA]

This is the discussion thread for the bazelbuild/proposals#354, which aims to support C++20 Modules in Bazel.
The C++20 Modules version of open-source C++20 coroutine library, async_simple (about 10 kloc), can be built with the bazel now. my fork is here: https://github.com/PikachuHyA/async_simple
Some extra tests are put here: https://github.com/PikachuHyA/bazel_cxx20_module_test
Feedback is very welcome.

[zaucy]

Instead of introducing a new attribute (module_interfaces) could the files in srcs be parsed to detect if they're modules? Maybe simply looking up if the module keyword is used anywhere or maybe clang-scan-deps already reveals this information?

Perhaps the overhead wouldn't be worth it (if at all possible.)

[PikachuHyA]

could the files in srcs be parsed to detect if they're modules?

yes.

maybe clang-scan-deps already reveals this information?

we know whether it is a module interface unit after using clang-scan-deps

The reason is that when compiling Module Interface Unit, I use a different set of actions compared to other Translation Units. It has additional outputs, such as the BMI file. Actions are constructed during the analysis phase. However, I only know in the execution phase which files are module interfaces. At this point, I cannot change the actions, such as adding a new action or modifying the output of an action.

Below is a more detailed explanation.

Currently, the BMI files are generated by two-phase compilation. If a file is a module interface, I need to compile it using two actions. The first action generates the BMI, and the second action generates the Object File.

┌────────┐ step1     ┌────────┐ step2  ┌────────┐
│bar.cppm├──────────►│bar.pcm ├───────►│bar.o   │
└────────┘           └────────┘        └────────┘

If a file is not a module interface, I only need to use one action to compile it into an Object File.

┌────────┐                             ┌────────┐
│main.cc ├────────────────────────────►│main.o  │
└────────┘                             └────────┘

Whether it is a file in the srcs or in the module_interfaces, it needs to be scanned using clang-scan-deps. After scanning, I can then know whether it is a module interface and which modules it depends on. However, scanning can only occur during the execution phase. If no attribute module_interfaces, I have to add new action in execution phase, which is not feasible.

Another way to produce BMI is one-phase compilation.
If the one-phase compilation is used, the problem is that Bazel does not support dynamic outputs.

                      ┌────────┐
                ┌────►│bar.pcm │
                │     └────────┘
                │
┌────────┐      │
│bar.cppm├──────┤
└────────┘      │
                │     ┌────────┐
                └────►│bar.o   │
                      └────────┘

The action graph is constructed during the analysis phase, but I only know in the execution phase whether the source file is a module interface or not. When declaring an action in the analysis phase, I don't know whether to consider the BMI file as an output. Files that are not declared as action outputs are not visible to other actions. This means they will not be copied out of the sandbox and cannot be used by other actions. If no attribute module_interfaces, I have to change action's output in execution phase, which is not feasible.

see also

• bazel Evaluation model
• Clang: How to produce a BMI

[zaucy]

Thanks for going into detail!

Bazel does not support dynamic outputs

A dynamic output may not be needed if the .pcm file is always produced and is empty when dealing with a non-module file. Checking if the file is empty would then have to be done at the execution phase.

[PikachuHyA]

A dynamic output may not be needed if the .pcm file is always produced and is empty when dealing with a non-module file.

sorry about that.

My understand is that if compiling a non-module file, the .pcm file will not be generated. It is not an empty file, it simply does not exist. For Bazel, if an output file is declared but the action does not actually generate it, an error will be reported.

[zaucy]

Yeah that's what I'm trying to suggest is a possibility. During execution check if source file is a non-module and produce an empty file. That way during analysis the file always exits, but may be empty.

[peakschris]

@PikachuHyA thanks for your work on this. Could you comment on what would be needed to extend this to support MSVC? Thank you.

[PikachuHyA]

hi @peakschris
I have supported build with MSVC locally with the following restrictions

• one-phase compilation
• only named modules supported
• only tested with msvc (msvc-cl). I'm not familiar with mingw and clang-cl, so only msvc-cl tested.

[peakschris]

@PikachuHyA amazing! Will this be part of the MR or could it be added in some way?

[PikachuHyA]

@peakschris

I'm sorry for the delayed response.
I wanted to finish the code so you could test it directly with MSVC before replying. However, the second patch (#22427) has taken much longer than expected, and many tests are still failing. It may take some more time before I can share the code.

Will this be part of the MR or could it be added in some way?

Yes.

I am currently working on the second patch (#22427). Once it is ready, I will make changes related to MSVC toolchains in the third patch (#22429).
ps: GCC and Clang are also supported with one-phase compilation.

Your reviews and feedback on the code are welcome. Please feel free to report any issues.