Home

Welcome to the Porcelain wiki!

Nothing in this document is final. Many of the lower level details can be changed during implementation.

Project goals

Porcelain aims to provide a simple API in Elixir on top of Erlang ports. It can also augment the capabilities of Erlang ports by using the middleman program called goon.

Specifically, the features Porcelain should implement are:

• simple API for spawning external processes (wrapping Erlang port functions)

• being able to interact with "eof-driven" programs (those are programs that don't produce output before they reach EOF in the input; they don't work with bare ports)

• sending signals to external processes

Below are some advanced features that may or may not be implemented at a later stage. Those are described here just for the sake of completeness. The current goal is to produce a working library ASAP with basic functionality described above.

• chaining multiple external processes in a portable way (supporting POSIX and Windows platforms)

• integrating external programs into an Elixir supervision tree

• inspecting running OS processes

• multiplexing multiple external processes onto a single running instance of goon (to avoid doubling the total amount of external processes spawned from Elixir when using goon; also, to provide additional features by using process groups, etc.)

The API

In this section we define the Porcelain API shared by all of its drivers. If a driver doesn't support certain functions, calling those functions should immediately throw exception at run time.

Sync API

The Porcelain.exec function will spawn the given shell command or executable and will block until the OS process terminates.

• Porcelain.exec(invocation, opts \\ []) when is_binary(invocation) – spawn a shell that will execute invocation as a command

• Porcelain.exec({program, args}, opts \\ []) – spawn an OS processes running program with arguments args

If you don't want to spawn a shell, but are too lazy to type the list of arguments by hand, use Porcelain.shplit(str) function that will try to parse the given string into program name and a list of arguments according to the POSIX convention.

The opts argument is a keyword list containing the following options:

• :in – specify the way input will be passed to the external process. Possible values:

  • <iodata> – the data is fed into stdin as the sole input for the program
  • <stream> – interprets <stream> as a stream of iodata to be fed into the program
  • {:path, <string>} – path to a file to be fed into stdin
  • {:file, <file>} – <file> is a file pid obtained from e.g. File.open; the file will be read from the current position until EOF

• :out – specify the way output will be passed back to Elixir. Possible values:

  • :buffer – the whole output will be accumulated in memory and returned as one string to the caller
  • {:path, <string>} – the file at path will be created (or truncated) and the output will be written to it
  • {:append, <string>} – the output will be appended to the the file at path (it will be created first if needed)
  • {:file, <file>} – <file> is a file pid obtained from e.g. File.open; the file will be written to starting at the current position

• :err – specify the way stderr will be passed back to Elixir. Possible values are the same as for :out.

It is also possible to redirect stderr to stdout and vice versa by using the option name as the option value. For example:

Porcelain.exec(program, in: "hello", out: {:path, "output.txt"}, err: :out)

will feed the string "hello" into program and will write both stdout and stderr to output.txt.

When no options are provided, default values will be used. Thus, the following two calls are equivalent:

Porcelain.exec(program)

Porcelain.exec(program, in: "", out: :buffer, err: :buffer)

The return value will be a Porcelain.Result struct:

Porcelain.exec("echo", in: "Hello world")
#=> %Porcelain.Result{out: "Hello world\n", err: ""}

Async/Stream API

TBD later.

Implementation details

The current implementation of Porcelain requires goon to work. We are going to change that and make goon an optional component. It will be detected at run time by default, but the user should be able to override the setting: to either never use goon or to require goon to be present in $PATH.

The basic functionality that only provides a simpler API on top of ports will be implemented in the Porcelain.Driver.Simple (or just Simple) driver.

Additional functionality that includes circumventing the "eof bug" and sending signals to OS processes will be implemented in the Porcelain.Driver.Goon (or just Goon) driver.

Suggestions of better names will be appreciated.

Common functionality

The following tasks are going to be performed by all drivers, so they should be implemented in a shared module.

• splitting the shell-like invocation into program name and a list of arguments (Porcelain.shplit)
• processing and validating given options
• defining a set of Erlang port options shared accross all drivers

The Simple driver

This section describes the details of the pure Elixir implementation of the Porcelain API.

This currently describes the functionality only as far as the Sync API is concerned.

It serves as a light shim on top of ports. The main responsibilities of the driver are:

• building the invocation of Port.open
• implementing a receive loop to collect output from the program

The Goon driver

This section describes the details of interacting with the goon middleman from Elixir. See below for the description of goon's implementation.

This currently describes the functionality only as far as the Sync API is concerned.

The main responsibilities of the driver are:

• building the invocation of Port.open
• performing the handshake with goon to select the protocol for communications and perform any other required configuration (will most likely be implemented as a set of command-line flags and arguments to the goon program)
• implementing a receive loop to collect output from goon (reference)
• sending signals to the managed OS process

The middleman

Implemented in Go, goon will provide a lightweight solution for patching some holes in the Erlang port functionality, namely:

• being able to interact with "eof-driven" programs
• being able to send signals to OS processes

In terms of OS processes, the hierarchy when using the Goon driver will look as follows:

    +-----------+
    | Erlang VM |
    +-----------+
         ||
         \/
      +------+
      | Goon |
      +------+
         ||
         \/
+------------------+
| External program |
+------------------+

Arrows pointing down indicate the parent→child relationship between the OS processes.

The Go implementation will use packages from Go's stdlib to work with external processes in a portable way:

• os
• exec
• signal

The actual implementation should conform to the Goex protocol specification described below.

The Goex protocol

In order for Porcelain and goon to work together, a simple protocol is defined that is used for all communications between the two components.

In Elixir, we always do Port.open {:spawn_executable, <path to goon>}, ... and then exchange messages with the spawned instance of goon. We use simple framing to be able to pass data to the external program managed by goon with as little overhead as possible.

TODO: ...

• describe the protocol
• describe the way future extensions will be handled