Refactor package manager handling (#1640)

Previously, the buildpack didn't track which package manager was being
used, but instead had a number of fragile file presence checks and env
vars like `SKIP_PIP_INSTALL` scattered throughout (which won't work
when the package manager isn't a binary choice).

As such, package manager handling has been refactored to allow for more
easily adding Poetry support later.

This also continues the gradual migration of code from sourced
`bin/steps/` scripts, to functions declared in `lib/`. As part of this,
I've started namespacing the lib functions using the `module::fn_name()`
pattern described in:
https://google.github.io/styleguide/shellguide.html#function-names

On the most part I've tried to keep this refactoring a no-op (aside from
fixes for a few quirks/bugs found whilst working on this). Longer term
we will be deprecating/sunsetting several features, such as the
`setup.py` fallback, using Pipenv without a lockfile, and support for
apps that have ended up with the files for multiple package managers.

GUS-W-16811078.

CHANGELOG.md

@@ -2,7 +2,12 @@
 
 ## [Unreleased]
 
-- Buildpack error messages are now more consistently formatted and use colour. ([#1639](https://github.com/heroku/heroku-buildpack-python/pull/1639))
+- Moved the SQLite3 install step prior to installing dependencies when using Pipenv. This now matches the behaviour when using pip and allows dependencies to actually use the headers. ([#1640](https://github.com/heroku/heroku-buildpack-python/pull/1640))
+- Stopped exposing the `SKIP_PIP_INSTALL` env var to `bin/post_compile` and other subprocesses when using Pipenv. ([#1640](https://github.com/heroku/heroku-buildpack-python/pull/1640))
+- Stopped creating `.heroku/python/requirements-{declared,installed}.txt` files when using pip. ([#1640](https://github.com/heroku/heroku-buildpack-python/pull/1640))
+- Stopped creating a placeholder `requirements.txt` file when an app only has a `setup.py` file and no other package manager files. Instead pip is now invoked directly using `--editable .`. ([#1640](https://github.com/heroku/heroku-buildpack-python/pull/1640))
+- Improved buildpack metrics for package manager detection and duration of install steps. ([#1640](https://github.com/heroku/heroku-buildpack-python/pull/1640))
+- Updated buildpack-generated error messages to use colour and be more consistently formatted. ([#1639](https://github.com/heroku/heroku-buildpack-python/pull/1639))
 
 ## [v256] - 2024-09-07
 

bin/compile

@@ -16,6 +16,10 @@ BUILDPACK_DIR=$(cd "$(dirname "$(dirname "${BASH_SOURCE[0]}")")" && pwd)
 source "${BUILDPACK_DIR}/bin/utils"
 source "${BUILDPACK_DIR}/lib/metadata.sh"
 source "${BUILDPACK_DIR}/lib/output.sh"
+source "${BUILDPACK_DIR}/lib/package_manager.sh"
+source "${BUILDPACK_DIR}/lib/pip.sh"
+source "${BUILDPACK_DIR}/lib/pipenv.sh"
+source "${BUILDPACK_DIR}/lib/utils.sh"
 
 compile_start_time=$(nowms)
 
@@ -97,12 +101,10 @@ cd "$BUILD_DIR"
 # - The build is executed, modifying `~/.heroku/{known-paths}`.
 # - Once the build is complete, `~/.heroku/{known-paths}` is copied back into the cache.
 
-# Create the cache directory, if it doesn't exist.
 mkdir -p "$CACHE_DIR/.heroku"
 
 # Restore old artifacts from the cache.
 mkdir -p .heroku
-
 # The Python installation.
 cp -R "$CACHE_DIR/.heroku/python" .heroku/ &>/dev/null || true
 # A plain text file which contains the current stack being used (used for cache busting).
@@ -116,10 +118,7 @@ if [[ -d "$CACHE_DIR/.heroku/src" ]]; then
 	cp -R "$CACHE_DIR/.heroku/src" .heroku/ &>/dev/null || true
 fi
 
-# The pre_compile hook. Customers rely on this. Don't remove it.
-# This part of the code is used to allow users to customize their build experience
-# without forking the buildpack by providing a `bin/pre_compile` script, which gets
-# run inline with the buildpack automatically.
+# Runs a `bin/pre_compile` script if found in the app source, allowing build customisation.
 source "${BUILDPACK_DIR}/bin/steps/hooks/pre_compile"
 
 # Sticky runtimes. If there was a previous build, and it used a given version of Python,
@@ -128,43 +127,15 @@ if [[ -f "$CACHE_DIR/.heroku/python-version" ]]; then
 	CACHED_PYTHON_VERSION=$(cat "$CACHE_DIR/.heroku/python-version")
 fi
 
-# We didn't always record the stack version. This code is in place because of that.
+# We didn't always record the stack version.
 if [[ -f "$CACHE_DIR/.heroku/python-stack" ]]; then
 	CACHED_PYTHON_STACK=$(cat "$CACHE_DIR/.heroku/python-stack")
 else
 	CACHED_PYTHON_STACK=$STACK
 fi
 
-# TODO: Move this into a new package manager handling implementation when adding Poetry support.
-# We intentionally don't mention `setup.py` here since it's being removed soon.
-if [[ ! -f requirements.txt && ! -f Pipfile && ! -f setup.py ]]; then
-	display_error <<-EOF
-		Error: Couldn't find any supported Python package manager files.
-
-		A Python app on Heroku must have either a 'requirements.txt' or
-		'Pipfile' package manager file in the root directory of its
-		source code.
-
-		Currently the root directory of your app contains:
-
-		$(ls -1 --indicator-style=slash "${BUILD_DIR}")
-
-		If your app already has a package manager file, check that it:
-
-		1. Is in the top level directory (not a subdirectory).
-		2. Has the correct spelling (the filenames are case-sensitive).
-		3. Isn't listed in '.gitignore' or '.slugignore'.
-
-		Otherwise, add a package manager file to your app. If your app has
-		no dependencies, then create an empty 'requirements.txt' file.
-
-		For help with using Python on Heroku, see:
-		https://devcenter.heroku.com/articles/getting-started-with-python
-		https://devcenter.heroku.com/articles/python-support
-	EOF
-	meta_set "failure_reason" "package-manager-not-found"
-	exit 1
-fi
+PACKAGE_MANAGER=$(package_manager::determine_package_manager "${BUILD_DIR}")
+meta_set "package_manager" "${PACKAGE_MANAGER}"
 
 # Pipenv Python version support.
 # Detect the version of Python requested from a Pipfile (e.g. python_version or python_full_version).
@@ -189,9 +160,9 @@ else
 	echo "${DEFAULT_PYTHON_VERSION}" >runtime.txt
 fi
 
-# Create the directory for .profile.d, if it doesn't exist.
+# The directory for the .profile.d scripts.
 mkdir -p "$(dirname "$PROFILE_PATH")"
-# Create the directory for editable source code installation, if it doesn't exist.
+# The directory for editable VCS dependencies.
 mkdir -p /app/.heroku/src
 
 # On Heroku CI, builds happen in `/app`. Otherwise, on the Heroku platform,
@@ -207,46 +178,58 @@ if [[ "$(realpath "${BUILD_DIR}")" != "$(realpath /app)" ]]; then
 	# Note: .heroku/src is copied in later.
 fi
 
-# Download / Install Python, from pre-build binaries available on Amazon S3.
-# This step also bootstraps pip / setuptools.
+# Download and install Python using pre-built binaries from S3.
 install_python_start_time=$(nowms)
 source "${BUILDPACK_DIR}/bin/steps/python"
 meta_time "python_install_duration" "${install_python_start_time}"
 
-# Install Pipenv dependencies, if a Pipfile was provided.
-source "${BUILDPACK_DIR}/bin/steps/pipenv"
-
-# If no requirements.txt file given, assume `setup.py develop` is intended.
-# This allows for people to ship a setup.py application to Heroku
-
-if [[ ! -f requirements.txt ]] && [[ ! -f Pipfile ]]; then
-	meta_set "setup_py_only" "true"
-	echo "-e ." >requirements.txt
-else
-	meta_set "setup_py_only" "false"
-fi
+# Install the package manager and related tools.
+package_manager_install_start_time=$(nowms)
+bundled_pip_module_path="$(utils::bundled_pip_module_path "${BUILD_DIR}")"
+case "${PACKAGE_MANAGER}" in
+	pip)
+		pip::install_pip_setuptools_wheel "${bundled_pip_module_path}"
+		;;
+	pipenv)
+		# TODO: Stop installing pip when using Pipenv.
+		pip::install_pip_setuptools_wheel "${bundled_pip_module_path}"
+		pipenv::install_pipenv
+		;;
+	*)
+		abort_internal_error "Unhandled package manager"
+		;;
+esac
+meta_time "package_manager_install_duration" "${package_manager_install_start_time}"
 
 # SQLite3 support.
-# This sets up and installs sqlite3 dev headers and the sqlite3 binary but not the
-# libsqlite3-0 library since that exists on the stack image.
+# Installs the sqlite3 dev headers and sqlite3 binary but not the
+# libsqlite3-0 library since that exists in the base image.
 install_sqlite_start_time=$(nowms)
 source "${BUILDPACK_DIR}/bin/steps/sqlite3"
 buildpack_sqlite3_install
 meta_time "sqlite_install_duration" "${install_sqlite_start_time}"
 
-# pip install
-# -----------
-
-# Install dependencies with pip (where the magic happens).
-source "${BUILDPACK_DIR}/bin/steps/pip-install"
+# Install app dependencies.
+dependencies_install_start_time=$(nowms)
+case "${PACKAGE_MANAGER}" in
+	pip)
+		pip::install_dependencies
+		;;
+	pipenv)
+		pipenv::install_dependencies
+		;;
+	*)
+		abort_internal_error "Unhandled package manager"
+		;;
+esac
+meta_time "dependencies_install_duration" "${dependencies_install_start_time}"
 
 # Support for NLTK corpora.
 nltk_downloader_start_time=$(nowms)
 sub_env "${BUILDPACK_DIR}/bin/steps/nltk"
 meta_time "nltk_downloader_duration" "${nltk_downloader_start_time}"
 
-# Support for editable installations. Here, we are copying pip–created src directory,
-# and copying it into the proper place (the logical place to do this was early, but it must be done here).
+# Support for editable installations.
 # In CI, $BUILD_DIR is /app.
 # Realpath is used to support use-cases where one of the locations is a symlink to the other.
 if [[ "$(realpath "${BUILD_DIR}")" != "$(realpath /app)" ]]; then
@@ -256,9 +239,6 @@ fi
 
 # Django collectstatic support.
 # The buildpack automatically runs collectstatic for Django applications.
-# This is the cause for the majority of build failures on the Python platform.
-# These failures are intentional — if collectstatic (which can be tricky, at times) fails,
-# your build fails.
 collectstatic_start_time=$(nowms)
 sub_env "${BUILDPACK_DIR}/bin/steps/collectstatic"
 meta_time "django_collectstatic_duration" "${collectstatic_start_time}"
@@ -304,7 +284,7 @@ fi
 cp "${BUILDPACK_DIR}/vendor/WEB_CONCURRENCY.sh" "$WEB_CONCURRENCY_PROFILE_PATH"
 cp "${BUILDPACK_DIR}/vendor/python.gunicorn.sh" "$GUNICORN_PROFILE_PATH"
 
-# Experimental post_compile hook. Don't remove this.
+# Runs a `bin/post_compile` script if found in the app source, allowing build customisation.
 source "${BUILDPACK_DIR}/bin/steps/hooks/post_compile"
 
 # Store new artifacts in the cache.

bin/report

@@ -67,6 +67,7 @@ STRING_FIELDS=(
 	failure_reason
 	nltk_downloader
 	package_manager
+	package_manager_multiple_found
 	pip_version
 	pipenv_version
 	python_version_major
@@ -81,6 +82,7 @@ ALL_OTHER_FIELDS=(
 	dependencies_install_duration
 	django_collectstatic_duration
 	nltk_downloader_duration
+	package_manager_install_duration
 	pipenv_has_lockfile
 	post_compile_hook
 	post_compile_hook_duration

bin/steps/pipenv-python-version

@@ -1,8 +1,10 @@
 #!/usr/bin/env bash
 
+# TODO: Move this to lib/ as part of the refactoring for .python-version support.
+
 # Detect Python-version with Pipenv.
 
-if [[ -f $BUILD_DIR/Pipfile ]]; then
+if [[ "${PACKAGE_MANAGER}" == "pipenv" ]]; then
 
 	if [[ ! -f $BUILD_DIR/runtime.txt ]]; then
 		if [[ ! -f $BUILD_DIR/Pipfile.lock ]]; then

bin/steps/python

@@ -166,34 +166,3 @@ else
 
 	hash -r
 fi
-
-PIP_VERSION=$(get_requirement_version 'pip')
-SETUPTOOLS_VERSION=$(get_requirement_version 'setuptools')
-WHEEL_VERSION=$(get_requirement_version 'wheel')
-
-puts-step "Installing pip ${PIP_VERSION}, setuptools ${SETUPTOOLS_VERSION} and wheel ${WHEEL_VERSION}"
-
-meta_set "pip_version" "${PIP_VERSION}"
-meta_set "setuptools_version" "${SETUPTOOLS_VERSION}"
-meta_set "wheel_version" "${WHEEL_VERSION}"
-
-# Python bundles Pip within its standard library, which we can use to install our chosen
-# pip version from PyPI, saving us from having to download the usual pip bootstrap script.
-# We have to use a glob since the bundled wheel filename contains the pip version, which
-# differs between Python versions. We also have to handle the case where there are multiple
-# matching pip wheels, since in some versions of Python (eg 3.9.0) multiple versions of pip
-# were accidentally bundled upstream. Note: This implementation relies upon `nullglob` being
-# set, which is the case thanks to the `bin/utils` that was run earlier.
-BUNDLED_PIP_WHEEL_LIST=(.heroku/python/lib/python*/ensurepip/_bundled/pip-*.whl)
-BUNDLED_PIP_WHEEL="${BUNDLED_PIP_WHEEL_LIST[0]}"
-
-if [[ -z "${BUNDLED_PIP_WHEEL}" ]]; then
-	display_error "Error: Failed to locate the bundled pip wheel."
-	meta_set "failure_reason" "bundled-pip-not-found"
-	exit 1
-fi
-
-/app/.heroku/python/bin/python "${BUNDLED_PIP_WHEEL}/pip" install --quiet --disable-pip-version-check --no-cache-dir \
-	"pip==${PIP_VERSION}" "setuptools==${SETUPTOOLS_VERSION}" "wheel==${WHEEL_VERSION}"
-
-hash -r

buildpack.toml

@@ -7,6 +7,7 @@ files = [
   "builds/",
   "etc/publish.sh",
   "spec/",
+  ".editorconfig",
   ".gitignore",
   ".rubocop.yml",
   ".shellcheckrc",

lib/kvstore.sh

@@ -1,5 +1,7 @@
 #!/usr/bin/env bash
 
+# TODO: Switch this file to using namespaced functions like `kvstore::<fn_name>`.
+
 # Taken from: https://github.com/heroku/heroku-buildpack-nodejs/blob/main/lib/kvstore.sh
 
 kv_create() {

lib/metadata.sh

@@ -1,5 +1,7 @@
 #!/usr/bin/env bash
 
+# TODO: Switch this file to using namespaced functions like `metadata::<fn_name>`.
+
 # Based on: https://github.com/heroku/heroku-buildpack-nodejs/blob/main/lib/metadata.sh
 
 source "${BUILDPACK_DIR}/lib/kvstore.sh"

lib/output.sh

@@ -1,5 +1,7 @@
 #!/usr/bin/env bash
 
+# TODO: Switch this file to using namespaced functions like `output::<fn_name>`.
+
 ANSI_RED='\033[1;31m'
 ANSI_RESET='\033[0m'