Compilation pipeline notes

[gnumonik]

Here are some notes I took on the compilation pipeline. They've already come in handy for me, and may be useful to to others. (I'll follow up with some discussion in subsequent posts)

Compilation Pipeline

1. Starts w/ a "psc compile" command in app/Main.hs
2. Calls app/Command/Compile.hs compile which:
   - Checks whether files corresponding to module that can be compiled exist
   - Reads the modules into raw text
   - In the Make monad (todo: figure out what that's for):
   a. Parses the modules into a CST using Language.PureScript.CST.parseModulesFromFiles
   b. Generates a filePathMap :: Map ModuleName (Either RebuildPolicy FilePath)
   c. Infers foreign modules
   d. Generates a MakeActions from the outputdir, filepathmap, foreign modules, and a (pscmUsePrefix :: Bool). Not sure what the last thing's for.
   • MakeActions contains things like the codegen function, which we should be able to replace with our own w/o much hassle
     e. Runs Language.PureScript.Make.make on the make actions and the parsed CST modules
3. Language.PureScript.Make.make:
   - Checks the module names
   - Reads a cache database (Map ModuleName CacheInfo) where CachInfo is a newtype over (Map FilePath (UTCTime, ContentHash)) from a JSON cache file.
   • I assume this is to avoid recompiling unchanged modules?
     - Sorts the modules and produces a module graph
     - Constructs a BuildPlan and a new cache database from: The MakeActions, the cacheDb, the sorted modules, and the module graph
   • TODO: Look into what a BuildPlan is
     - Filters out the list of sorted modules, yielding a list of only those modules which need to be rebuilt
     - For each module that needs to be rebuilt, ASYNCHRONOUSLY:
   • Extracts the module name
   • Determines the dependencies
   • ** Calls Language.PureScript.Make.buildModule with parameters
     • the build plan
     • the module name
     • the total module count (this is the # of modules that need to be REBUILT)
     • a list of parser warnings from the parsed module
     • (Either (NonEmpty ParserError) Module) - this is the source module with Primitive modules added as implicit imports (using Language.PureScript.AST.Declarations.importPrim)
     • The sorted dependencies (as a list of modules)
       - Waits for the async compilation to complete, collects the results & errors
       - Writes the updated cacheDb to the JSON file
       - Writes the package json (not sure what this is)
       - Generates the docs (incl for prim modules)
       - Returns a sorted ("toplogically") list of ExternsFiles
4. Language.PureScript.Make.buildModule (defined in a where binding for Language.PureScript.Make.make)
   - Logs any parser warning
   - Waits for dependencies to be built
   - Adds all dependencies to the Build Environment (presumably for typechecking)
   - Reads the Build Environment and the Build Plan Index for the current module (I think?) from the Buildplan
   - Increments the Build Plan Index counter
   - Calls rebuildModuleWithIndex (which appears to be how modules are built for the first time as well) with the parameters:
   • The MakeActions originally passed to Language.PureScript.Make.make
   • The Build Environment (Env)
   • A list [ExternsFile] of the ExternsFiles for each dependency
   • The Module (in CST form)
   • The module index & module count (again, the # of modules that need to be rebuild)
     - Returns the ExternsFile for the current module if the build succeeded
5. Language.PureScript.Make.rebuildModuleWithIndex :: (..) => MakeActions m -> Env -> [ExternsFile] -> Module -> Maybe (Int,Int) -> m ExternsFile (Make.hs 78)
   - "Applies the externs files (the dependencies) to the environment". Need to look into what exactly that entails, presumably for typechecking
   - Imports primitives into the module
   - In a Supply monad (for fresh names):
   • Desugars the module (partially, case guard desugaring takes place later) using Language.PureScript.Sugar.desugar (see that module for a list of all the desugaring transformations)
   • Typechecks the desugared module
     • IMPORTANT: Returns the CheckEnv for the module which contains a map from qualified names to types
       - Checks imported newtypes for coercibility
       - Desugars case declarations
       - Replaces all sets of mutually recursive declarations with binding groups
       - Transforms the Language.PureScript.AST.Declarations.Module Module (which is what it's been working with up to this point) into a CoreFn module
       - Optimizes the CoreFn module
       - Renames identifiers in the optimized CoreFn module
       - Generates an ExternsFile for the module being compiled (generated from the AST module not the CoreFn module)
       - Runs the ffiCodegen function (which is a field of MakeActions), which checks the FFI modules and prints them to the output directory, on the renamed CoreFn module
       - Generates docs for the module
       - Runs the codgen function (which is a field of MakeActions and ergo can be configured) on the optimized CoreFn module, the generated docs, and the Externs file for the current module
       - Returns the Externs file for the current module