Improved Compiler API

[alex-mckenna]

Problem

Historically, clash-lib has been designed from the point of view that it has one consumer — clash-ghc. However, additional programs may be created which wish to use the compiler as a library. While they can interact with the compiler externally, by spawning a clash process then collecting the output files, it is more desirable for clash-lib to provide a suitable API.

Moreover, for some use cases spawning a clash process is not enough. If a program wishes to use an intermediate stage of the compiler, such as the netlist, then there is currently no way to do that without piecing together calls to clash-ghc and clash-lib until the information can be extracted (e.g. runToNetlistStage in clash-testsuite:lib).

Proposed Solution

I propose a new API is exposed by Clash.Driver which contains:

• a type of a Clash design's "environment". This includes ancillary information which may be used in different steps but does not change, like ClashOpts, TyConMap, CompiledPrimMap etc.
• a type of a Clash design. This may just be a BindingMap of Term
• functions to normalize both a single term and a whole design in an environment
• functions to convert a single normalized term or a normalized design in an environment into netlist.
• a function to produce a manifest from a netlist
• a function to produce HDL (and other output for EDA tools like SDC files) from a netlist

Benefits

The main advantage of this proposal is that Clash becomes more amenable to third-party development. This means users who wish to develop tooling can access the output of different stages of the compiler, making it easier to develop proof-of-concept implementations for new compiler features without needing as much familiarity with the internals of clash-lib.

This proposal also helps better set the stage for future potentially useful improvements, such as allowing Clash to start or stop compilation from a different stage. This is my initial motivation for this proposal.

Drawbacks

Even with a friendlier API for Clash.Driver, clash-lib and clash-ghc will be unstable and potentially vulnerable to change. An updated API designed for third parties to consume may be less likely to change, but users should not expect the same stability guarantees as clash-prelude gives.

---

CC @gergoerdi, whose mailing list post motivated me writing this proposal out

[leonschoorl]

Good idea.

There would have to be a part of this API in clash-ghc, the part that runs the GHC frontend and generate the Clash design "environment".
Basically a more convenient version of generateBindings
Which should preferably handle the linker issues from #1686 automatically.

[alex-mckenna]

Yeah, you're right, if someone wants to make a tool they would have to slog through the clash-ghc calls before they reach the nicer API which is less than ideal.

[gergoerdi]

My current use cases for read-back manifest files:

• Clashilator uses it to figure out the IO pins and automatically detect the clock (if there is a single one). I guess it could also give a simplified API if you don't want to reset your circuit, by resetting it for you at the beginning and then not exposing the reset line. For this use case, there is no benefit for having the manifest file on disk -- I would much rather just call a Clash function that generates HDL and gives me the manifest value, instead of roundtripping via a text file.
• clash-shake uses the manifest file to figure out what HDL files to include in the generated vendor toolchain project. TBH I am not sure if this doesn't require going through a file for Shake dependency tracking anyway; although, dependency tracking is FUBAR in clash-shake anyway so it is hard to think about this.

Something that I don't do but would like to start doing, is librarizing constraint files. Currently I have constraint files in individual projects (example), because at least for ISE, you need to list only the pins that you will actually connect (and it could be true of Vivado as well, I don't know, maybe it would cut down on warnings?). A nicer way of doing this would be to have a library of constraint files for supported boards, and then filtering them for the pins connected. I remember Kansas Lava had this, see this repo for an example of librarized UCF files.

To do this fully, I would need to move more to Clash that is currently done in external hand-written HDL files. I haven't thought about this yet -- this would require target-specific Clash files that even hook up the clock managers etc. So this is just a pie-in-the-sky idea for now.

[alex-mckenna]

Hmm, maybe it would be beneficial to have "resolved" and "unresolved" manifests. The difference being that an unresolved manifest doesn't contain file paths for generated items (but can still include hashes of generated items). This is maybe something @martijnbastiaan has an opinion on

[martijnbastiaan]

I would much rather just call a Clash function that generates HDL and gives me the manifest value, instead of roundtripping via a text file.

Agreed.

I'm also supportive of the resolved/unresolved manifests @alex-mckenna mentioned. I don't think there's a need to run normalization at all to generate port names, port types, etc.