Zero-code adoption using drop-in runtimes

[shykes]

Overview

The design of Cloak opens the possibility of zero-code adoption, where a new user gets value from Dagger without having to write any code. That would make Dagger much easier to learn and adopt, which would greatly help with adoption.

This would be made possible by 1) implementing a system of modular runtimes, and 2) developing "drop-in runtimes" capable of natively running a project's pre-existing automation without modification, including:

• Dockerfiles
• Docker Compose files
• Makefiles
• NPM scripts
• etc.

Onboarding path

The complete onboarding path could look something like this:

1. Drop in. "Dagger runs my existing automation, better than the original tools (faster, more secure, more scalable, more portable)
2. Extend. "I can extend my existing automation with new actions, using a Dagger SDK for my favorite language. The developer experience is better, and I don't lose compatibility"
3. Refactor. "My team needs better tooling. I will replace our artisanal scripts with a custom tool written in a modern language. I can do this on top of the Dagger SDK, and save time by reusing all the actions I've already written so far.
4. Modularize. "My teams needs to customize their automation without re-writing if from scratch. I show them how to use Dagger to consume my automations as an API, and create their own custom automations on top, using the language of their choice.

Design considerations

I’m assuming these runtimes will need to implement a special hook (itself running on dagger of course) that takes a source repo as input, and outputs a graphql schema exposing all the actions, artifacts and services that it found.

graph BT;
  project[Project Directory];
  subgraph "Dagger"
  cloak(["Cloak engine"])
  go("Go runtime");
  typescript("Typescript runtime");
  dockerfile("Dockerfile runtime");
  dockercompose("Docker Compose runtime");
  make("Make runtime");
  yarn("Yarn runtime");
  cloak --- go & typescript & dockerfile & dockercompose & make & yarn;
  end;
  project --- cloak;

    subgraph "Actions";
    test("test");
    lint("lint");
    deploy("deploy")
   end;

    subgraph "Services";
    www("www");
    api("api");
    db("db");
    end;

    subgraph "Artifacts";
    build("yarn/build")
    head("git/head")
    checkout("git/checkout")
    container("dockerfile/stage/main")
    buildenv("dockerfile/stage/buildenv")
    end;

 go --- deploy;
 make --- lint;
 yarn --- test & build;
 dockercompose --- www & api & db;
 dockerfile --- container & buildenv;
 
 class project secondary;

Loading

Presumably the Go & Typescript SDKs (which do require coding to use, by definition) would ship with a runtime of their own, implementing the same hooks. Perhaps their implementation will be very simple (if there is no boilerplate to generate) or become more advanced (if code and/or schema generation is involved). This matches common use of the terms “runtime” and “sdk”, as in “the JDK vX ships with the JRE vY”.

[aluzzardi]

Definitely interesting.

/cc @marcosnils -- we were briefly discussing no/low code adoption yesterday, this approach could be it (actions are available to run before you write a single line of code).

I think the gap to make this possible is smaller than it appears. I don't think we would need to generate schemas on the fly, GraphQL is flexible enough that a static schema works. For instance for make:

# List the targets
make {
  targets {
    name
  }
}

# Run `make test`
make {
  target(name: "test") {
    run {
      stdout
    }
  }
}

[shykes]

Maybe we are not talking about the same thing with the term "dynamic schema". I'm talking about having a middleware that produces a schema for cloak to consume. Whether that schema is dynamically generated is up to the middleware. Once generated and loaded by cloak, it is "static" and therefore would require no particular graphql magic, and it would not complicate the usage patterns.

[aluzzardi]

Addendum to previous comment: Actually if we have a totally generic untyped API then we probably wouldn't need a runtime at all; it could just fit in the current action model today.

Yes, that's what I was thinking. I think we can get reach mostly the same outcome with the action model without an extra layer

[shykes]

I think having regular make and yarn actions as you describe @aluzzardi will be very cool and useful, but it will not be the same outcome as what I'm describing IMO.

[marcosnils]

Yep, agree with all this. It's one of the possibilities that we spoke the other day with @aluzzardi and I particularly like that this is extremely useful in the context of improving adoption. I think we could take some inspiration from the cloud native buildpacks (https://buildpacks.io/) proejct to understand how they're currently doing this "automatic" runtime detection / configuration since they do something very similar.

[shykes]

See also #24 for a continuation of my thoughts on this :)

[sipsma]

One quick note to self and @aluzzardi about implementation details: this goal seems like it will require that we support subresolvers in user actions (i.e. fields not directly under Query can also be explicit, custom written resolvers). We weren't sure if we wanted to support those initially just because it's more to document for developers, but this all seems like a compelling enough use case that we should. Luckily it won't be a huge refactor or anything like that, just need to add a few extra blocks of code here and there to handle the extra cases introduced by this.

[aluzzardi]

Yep.

Related but I was thinking we should stop doing the "query mangling" ourselves and instead rely on the schema doing an extend Query { yarn: Yarn! } } instead. Relies on a native way to do it, less magic, opens the door for extend Filesystem { yarn: Yarn! }

That however requires sub resolvers already

[sipsma]

Related but I was thinking we should stop doing the "query mangling" ourselves and instead rely on the schema

Yes 100%, the mangling is another hacky artifact of the first draft of the switch to graphql. extend is the much better approach.

[dolanor]

Really like this idea.
It would be very compelling. I don't have any opinions on the technical side because I didn't dig in it enough to understand all of what you're talking about.

I guess specific tasks wouldn't work out of the box:

• makefile spinning up local server as dependencies
• docker-compose running e2e testing or setting up deps (does buildkit support running docker services inside it?)
• specific environment settings that wouldn't propagate in dagger by default