dev: Create a Python package, build Docker images from it

[chishm]

What type of PR is this?

• feature
• dev (Internal development)

What this PR does / why we need it:

Build a Python package (sdist and wheel) that can be used to install Mealie, including the frontend SPA files. The package is then used to create the Docker images used for testing and release.

The advantage here is that the frontend code (Javascript) and backend code (Python) are both platform-independent, and so the same package can be installed into both the ARM64 and AMD64 Docker images. This saves having to build them twice.

Eventually, Mealie will be completely installable from PyPI as a Python package, as an option in addition to the Docker images. But more work is needed to achieve that goal and further PRs will follow this one. Until then, this PR stands on its own to benefit the E2E testing and CI release process, and can be merged now.

Changes in detail

The backend by default looks for the frontend SPA files in mealie/frontend (actually, the frontend sub-directory of the mealie Python package). This can still be overridden by setting STATIC_FILES.

There is a new top-level task in Taskfile.yml, py:package, that builds the Python packages. The generated frontend files are copied from frontend/dist to mealie/frontend by the task py:package:copy-frontend (called by py:package).

During the package build, a requirements.txt file is generated from Poetry's lock file. This ensures that exact pinned dependencies are installed.

I have documented the 2 ways to build Mealie in building-packages.md.

A few extra stages are added to the Dockerfile to allow setting an additional build context which holds the pre-built Python package (and associated requirements.txt). This can be done by using the docker build argument --build-context packages=dist, where dist is the directory holding a pre-built package. When this context is not set, the Dockerfile will build the package itself. When it is set, the build steps will be skipped.

The steps of the py:package task are replicated in the Dockerfile's frontend and backend build stages. This ensures that running docker-compose up will give the same result, regardless of whether the Docker image was built from a pre-built package or from scratch.

The GitHub workflows have been modified to make use of this. There is a new partial-package.yml workflow that incorporates the frontend build, adds a backend build, and stores the resulting package as an artifact for later jobs. The E2E testing workflow uses the package for its Docker build, ensuring that it gets tested. The partial-builder workflow also uses the package for its Docker image build and push, saving the effort of rebuilding the frontend and backend for each platform.

Finally, to make the Python package a bit nicer to use (not needing to modify PYTHONPATH), I've incorporated #4323 into this PR.

Which issue(s) this PR fixes:

This is a few tasks from discussion 4322.

Testing

• I successfully tested the built Python package on my development machine, outside of a Docker container.
• I used task docker:prod to build from scratch and test the Docker image.
• I used task docker:build-from-package to rebuild the Docker image (after deleting the previous one), followed by task docker:prod to test it.
• I ran the new GitHub workflow for pull requests on my fork of the mealie repository.
  • Have a look under the Artifacts header at backend-dist for the Python sdist, wheel, and requirements.

[boc-the-git]

Without wanting to speak to the overall validity of this PR (I've not looked in any details), I just wanted to note that the below is not an objective for the Mealie project:

Eventually, Mealie will be completely installable from PyPI as a Python package, as an option in addition to the Docker images.

[chishm]

Without wanting to speak to the overall validity of this PR (I've not looked in any details), I just wanted to note that the below is not an objective for the Mealie project:

Eventually, Mealie will be completely installable from PyPI as a Python package, as an option in addition to the Docker images.

I understand, but is it an anti-objective? That is, if a PR improves things overall (e.g. faster CI, or easier dev environment setup) will it be rejected just because it happens to make it easier to install Mealie from a package?