Allow large PEX while submitting skein app #127
Conversation
| :param allow_large_pex: Creates a non-executable pex that will need to be unzipped to circumvent | ||
| python's limitation with zips > 2Gb. The file will need to be unzipped | ||
| and the entry point will be <output>/__main__.py |
There was a problem hiding this comment.
@PaulMathon are you aware Pex now supports shipping a single-file native executable that includes a python interpreter? This nets you the benefit of --layout packed for those huge apps, but embedded in a single file that knows how to unpack itself and run. The key new option is --scie {eager,lazy} and this is explained more here: https://docs.pex-tool.org/scie.html
For example (this is against a tiny app, but the point stands):
# N.B.: The `--venv` is only for reducing latency of runs 2+, this may not be applicable in a cluster deploy - I have 0 domain expertise there.
:; pex --venv --scie eager --layout packed f2-commander -c f2 -o f2.pex
# The `--layout packed` PEX is deposited at f2.pex as per usual:
:; tree -a f2.pex/
f2.pex/
├── .bootstrap
├── .deps
│ ├── Send2Trash-1.8.3-py3-none-any.whl
│ ├── f2_commander-0.1.0-py3-none-any.whl
│ ├── humanize-4.11.0-py3-none-any.whl
│ ├── linkify_it_py-2.0.3-py3-none-any.whl
│ ├── markdown_it_py-3.0.0-py3-none-any.whl
│ ├── mdit_py_plugins-0.4.2-py3-none-any.whl
│ ├── mdurl-0.1.2-py3-none-any.whl
│ ├── platformdirs-4.3.6-py3-none-any.whl
│ ├── pygments-2.18.0-py3-none-any.whl
│ ├── python_dotenv-1.0.1-py3-none-any.whl
│ ├── rich-13.9.2-py3-none-any.whl
│ ├── textual-0.83.0-py3-none-any.whl
│ ├── typing_extensions-4.12.2-py3-none-any.whl
│ └── uc_micro_py-1.0.3-py3-none-any.whl
├── PEX-INFO
├── __main__.py
└── __pex__
└── __init__.py
3 directories, 18 files
# You get a new f2 (no `.pex`) though beside it and its a native executable.
:; file f2
f2: ELF 64-bit LSB pie executable, x86-64, version 1 (SYSV), static-pie linked, BuildID[sha1]=f1f01ca2ad165fed27f8304d4b2fad02dcacdffe, stripped
:; ls -lh f2
-rwxr-xr-x 1 jsirois jsirois 33M Oct 20 10:24 f2
The f2 executable has a zip trailer you can inspect (the head is the scie-jump binary and then a PBS interpreter after that, then the app code):
:; zipinfo -1 f2
warning [f2]: 31532378 extra bytes at beginning or within zipfile
(attempting to process anyway)
.bootstrap
.deps/
.deps/textual-0.83.0-py3-none-any.whl
.deps/linkify_it_py-2.0.3-py3-none-any.whl
.deps/uc_micro_py-1.0.3-py3-none-any.whl
.deps/Send2Trash-1.8.3-py3-none-any.whl
.deps/humanize-4.11.0-py3-none-any.whl
.deps/rich-13.9.2-py3-none-any.whl
.deps/typing_extensions-4.12.2-py3-none-any.whl
.deps/platformdirs-4.3.6-py3-none-any.whl
.deps/pygments-2.18.0-py3-none-any.whl
.deps/mdit_py_plugins-0.4.2-py3-none-any.whl
.deps/markdown_it_py-3.0.0-py3-none-any.whl
.deps/mdurl-0.1.2-py3-none-any.whl
.deps/f2_commander-0.1.0-py3-none-any.whl
.deps/python_dotenv-1.0.1-py3-none-any.whl
__main__.py
__pex__/
__pex__/__init__.py
PEX-INFO
When you run the ./f2 (Note: no PATH needed, its a self-contained static binary):
:; env -i PATH= ./f2
You get:
There was a problem hiding this comment.
I should note that although f2 and f2.pex are emitted side-by-side, they are totally independent. For a running f2 app, you can just scp the single f2 file to some other host and run it there.
There was a problem hiding this comment.
I'll also note this related issue / question led me here: pex-tool/pex#2560
I have not advertised --scie eager widely, but I suspect it is useful for some applications.
No description provided.