Include a pex_binary into a pex_binary

[mvgijssel]

Sometimes you need to include a pex_binary into a pex_binary. For example a client pex binary which starts a docker container with a server pex binary. Or maybe you want to link an archive into the pex_binary for uploading. Including something built by Pants into another target can sometimes be handy!

Unfortunately you can't use the dependencies argument directly like:

pex_binary(
    name = "server",
    entry_point = "server.py",
)

pex_binary(
    name = "client",
    dependencies = [
        ":server",
    ],
    entry_point = "client.py",
)

Currently the dependencies argument for pex_binary only supports Python/resource types. Because the output of :server is not a Python/resource file, the output files of this target will be filtered out quietly.

For this to work, it's required that the :server target is exposed as a resource. This is possible with the experimental_wrap_as_resources rule! And because experimental_wrap_as_resources currently only supports files as an input, :server first needs to be converted to a file somehow. This can be achieved with the adhoc_tool or the shell_command rule.

A shell_command target can be implemented like this:

shell_command(
    name="server-file",
    command="exit 0",
    output_files=["server.pex"],
    execution_dependencies=[":server"],
)

It's basically a noop by executing exit 0, but the result is is that the server.pex from the :server target is now converted into a file! This file can be wrapped with the experimental_wrap_as_resources rule:

experimental_wrap_as_resources(
    name="server-resource",
    inputs=[
        ":server-file"
    ],
)

And now we have our :server target exposed as a resource called :server-resource! This can be included in a pex_binary using the dependencies argument:

pex_binary(
    name = "client",
    dependencies = [
        ":server-resource",
    ],
    entry_point = "client.py",
)

Now when running pants run //:client the :server target will be built and injected into the :client target 🎉 .

An example macro glueing this all together:

def package_resource(name, source, output_files, description=None, tags=None):
    """
    From Pants Build Slack community: https://pantsbuild.slack.com/archives/C046T6T9U/p1724752064431699

    Macro which enables exposing a package target as a resource target.
    This is useful for example wanting to include an archive inside a pex_binary,
    but can be used for any packageable target.
    """
    if tags is None:
        tags = []

    shell_name = f"{name}.command"
    resource_name = name

    # Turn the package in `source` into a File
    shell_command(
        name=shell_name,
        command="exit 0",
        workdir="/",
        output_files=output_files,
        execution_dependencies=[source],
    )

   # Turn the file in `inputs` into a Resource
    experimental_wrap_as_resources(
        name=resource_name,
        description=description,
        tags=tags,
        inputs=[
            f":{shell_name}",
        ],
    )

This macro can be used in a BUILD file as follows

pex_binary(
    name = "server",
    entry_point = "server.py",
)

package_resource(
    name = "server-pex",
    output_files = [
        "server.pex",
    ],
    source = ":server",
)

pex_binary(
    name = "client",
    dependencies = [
        ":server-pex",
    ],
    entry_point = "client.py",
)

References

• https://pantsbuild.slack.com/archives/C046T6T9U/p1724752064431699
• https://pantsbuild.slack.com/archives/C046T6T9U/p1707089355999959
• https://www.pantsbuild.org/2.21/reference/targets/experimental_wrap_as_resources

[benjyw]

Thanks for writing this up @mvgijssel !