Code generation strategies

[armanbilge]

I see this issue as having a couple parts:

1. The actual code-generation strategy.

• and specifically, what the code should look like? API patterns, etc.

2. How to incorporate this generated code and/or evolve the current library towards it in the least painful way for downstream.

I think discussion should focus around (1) while keeping (2) in mind. Perhaps over-optimistically I believe (2) if not already addressed by (1) is surmountable via type aliases, scalafixes, and other sorceries, or just plain breaking changes, assuming we can find a strategy for (1) that we are happy with.

Furthermore, my assumption is the source material for code generation would be lib.dom.d.ts. If there are any comparable alternatives that would be great to know as well.

As pointed out in #486 (comment), there is a spectrum of approaches to this:

1. Manually apply changes to the existing library from a source of auto-generated code.
2. Auto-generate the code once and manually maintain it going forward (I believe this is how the current library was born as described in Null handling #481 (comment)).
3. Continually auto-generate the codebase as updates are applied to the upstream source material.

The remaining issue is what should the generator lib.dom.d.ts => scala-js-dom do? To answer this question, it would be very helpful if we can get critical commentary on Scalably Typed's latest generated code. Is it good enough to use, and if not, why, and how can it be improved?

[armanbilge]

3. Continually auto-generate the codebase as updates are applied to the upstream source material.

If done right, (3) would increase the scope and quality of this library while decreasing the maintenance burden. Ideally, dramatically on both counts. The challenge to this scheme was described in #486 (comment):

it's volatile, small changes in the original version may easily cause binary-incompatible changes in the translated version.

With this in mind, I make the following proposal for (3). Assume we have a generator that does lib.dom.d.ts => scala-js-dom.

1. Fork the microsoft/TypeScript repo containing lib.dom.d.ts and generate scala-js-dom (is it possible to fork a single file?).
2. When changes are made to lib.dom.d.ts upstream, (automatically) open a PR to scala-js-dom to merge these changes in.
3. Maintainers review the PR, adjust as necessary to maintain binary-compatibility in scala-js-dom, and merge.
   a. Reviewing a diff to lib.dom.d.ts is likely comparable to reviewing diffs to the API reports introduced in Poor Man's MiMa #471.
   b. The review can almost certainly be helped by a properly-configured MiMa that ignores certain changes (see Enable MiMa #461 (comment)).

Personally, I quite like the ST facades (I made extensive use of a Node.js facade in typelevel/fs2#2453). I think the most annoying thing about them is that Javascript and therefore Typescript does not distinguish numerical types, so everything becomes a Double. Perhaps one way this could be mitigated is by adding custom comment-based annotations to our fork of lib.dom.d.ts indicating a more precise type for number that an enhanced ST generator could take advantage of.

[japgolly]

When changes are made to lib.dom.d.ts upstream, (automatically) open a PR to scala-js-dom to merge these changes in.

Yeah ideally we'd automate this too, eg. with a daily-run Github Action

[japgolly]

Typescript does not distinguish numerical types, so everything becomes a Double.

I'm glad you remembered this cos it was a thought I had last week and then it evaporated out my ears before I could write it down. This is an important one! Thoughts:

• annotations or similar could be a viable solution, cool idea!
• I'd prefer not to fork anything if we don't have to and we could still have this ability without forking by maintaining a separate list in scala-js-dom of type overrides or something that the code generator reads and combines with its (externally-maintained) TS source
• whatever solution we might use, it might be an awesome idea to generate it from our existing SJS code (!). That way only new API would need to be considered
• ideally though, this info would be obtainable without human intervention/maintenance. I wonder if this better numeric precision is already solved somehow in the JS/TS world, maybe annotations and type declarations using it already exist? Research required

[armanbilge]

I'd prefer not to fork anything if we don't have to

I'd be curious to hear more about why you're not crazy about this, to me this really seemed like the way to go!

I wonder if this better numeric precision is already solved somehow in the JS/TS world, maybe annotations and type declarations using it already exist

I briefly researched this and found a couple int proposals for typescript that were ultimately rejected. I would really love to find something for this too, I wonder if we can even scrape it from docs somehow.

it might be an awesome idea to generate it from our existing SJS code (!)

Yes!!! That would be a load off 😅 after watching your scalafix incantations I believe anything is possible!

[japgolly]

Thanks for creating this @armanbilge!

An important point from my pov, is that I believe we'll need a way to make ad-hoc customisations. So if we're always regenerating, then we need a way to either instruct the code generation to do the following, or have an additional post-processing step to accommodate the following

• overriding signatures of certain definitions (eg. changing a return type)
• add additional code that's not in the facades for whatever reason
• exclude certain code defined in TS that for whatever reason, we might not want in SJS
• maintain backwards-compatibility from SJS if the TS definitions change

[armanbilge]

Excellent points! I think some of these can be addressed by my idea to maintain our own fork of lib.dom.d.ts:

• overriding signatures of certain definitions: change the typescript definition in our fork
• exclude certain code defined in TS: delete that code from our fork
• maintain backwards-compatibility from SJS if the TS definitions change: revert those offending commits in our fork

add additional code that's not in the facades for whatever reason

This one depends. If it's facade code that can be auto-generated, we can just create a TS definition for it and watch it generate itself :)

If it's Scala code that needs to be written, we'll need to come up with a scheme for this. Maybe every auto-generated class/trait/object can extend a user-defined trait, where we can put in custom code? It'll be a bit of cruft, but most of these traits can be empty. I'm sure there's more elegant solutions too (e.g., a custom annotation in the TS to indicate it should extend a scala trait/class with the given FQCN).

[oyvindberg]

I'll add a bit to this discussion going forward and as time permits, but one thing that I wanted to mention straight away:

The generated code will be volatile in two dimensions. one is the evolution of (lib.*.d.ts - there are many files, depending on which standard you want), and the other is the evolution of the conversion code. I wanted to showcase this commit log where you can see the former evolution across typescript versions: https://github.com/ScalablyTyped/scala-st-std/commits/master . These commits are all generated by the same conversion code. It tracks esnext so it changes a bit more than necessary, but it essentially shows the backlog of things changed since Typescript 1.x

[armanbilge]

Thanks, this is very illustrative!! For facade binary-compatibility, I think we are primarily concerned with removed classes/objects (removed fields/methods in facades are not a problem). And also any of the usual binary-compat concerns for non-facade Scala code. I only took a quick look through that commit log but seems like most changes are of the binary-compatible sort?

[japgolly]

Note for future: If this takes off, consider using caniuse and maybe some annotations as discussed in #514

[armanbilge]

@benhutchison wrote some helpful commentary on auto-generated vs hand-maintained facades in #486 (comment). This comment particularly stuck out to me:

Also, I find generated ScalablyTyped facades often harder to make sense of than I naively expect, eg Im not sure what all these refs to pointerenter in s/std/ are and if I need to care. So I still see value in a smaller, curated interface from a usability POV.

[oyvindberg]

Just FYI, I'm tracking this particular issue in ScalablyTyped/Converter#368 .

I ended up with these "fake literals" for a bunch of reasons.

• Avoiding erasure collisions with literal types became much more difficult. With literal types you get some kind of erasure which is either the literal type or string, depending on the return type. And then you have huge hierarchies of things, and it just became hard.
• IDE tooling at the time. Haven't revisited it in a while. Performance suffered, and I wanted "ctrl-shift-space" completions to work.

Hope to find some time to revisit this

[peter-jerry-ye]

I'm new to this part of discussion, and I'm not sure what's the current status. Correct me if I'm wrong, but it seems that all the discussion suggest that the generation being based on .d.ts from typescript?

I see that Microsoft has a generator that generates their .d.ts from idl of webref; ghcjs generates their definition using webkitgtk's idl file. So should we also start from the IDL file?

[armanbilge]

Thanks for the references. It looks like IDL distinguishes between numerical types? That would be a big advantage over the Typescript definitions.

I won't have time to look into this but I am very supportive of further investigation into this strategy. If you are interested to work on this, please let me know how I can help you :)

[peter-jerry-ye]

I'm not sure if I'm capable of working anything out.

By the way, if there were a new API, should it rely on cats-effect for IO Monad or something like that as they did for ghcjs? Or should we have no dependency at all?

[armanbilge]

Ah, this project should have definitely have no dependency besides Scala.js itself. The primary goal of this library is to be a raw, unopinionated facade for the APIs.

However, building an IO-based wrapper for the DOM APIs is something I am very interested in. See armanbilge/calico#3 (comment).

[peter-jerry-ye]

Well, in that case, I would suggest that we take the approach of generating directly the API from the web IDL. And I don't think we really need to worry about the binary compatibility because that should be taken care of by the makers of specification.

As for the compatibility with this repository, I don't know if we can generate it so that it's compatible? I mean we are following the specification when we add anything, so that shouldn't be a problem? Only if there's nothing to be changed in the current design. If there's anything to be changed, then a new major version or even a new library might seem a better solution.

[oyvindberg]

The effort to build something like the ST converter again is likely an order of magnitude bigger than what you assume, if it is to generate a version of the DOM for Scala.js which 1) compiles, 2) is flexible and easy to use, 3) looks good.

And that's before we start talking about the backwards compatibility story. The DOM definitions evolve in a way which is backwards compatible for structural type systems.

ST already has the infrastructure necessary to experiment with things like:

• supply third-party data about whether numeric types should be Double or Int
• generate an IO/ZIO facade. Personally I think ZIO with a DOM requirement in Env would be awesome, since you can tag all code which touches the DOM
• in general, perform any kind of rewrite on the AST layer.

Another idea is to replace the first phase in the converter, the typescript parsing, with an IDL parser, this is the approach taken by https://github.com/Kotlin/dukat .

[3xau1o]

To succesfully map TS union and literal types, scala 3 is required, is it in consideration @sjrd?