From 5a7bef54175aebaba19ab8a611c732283b325644 Mon Sep 17 00:00:00 2001 From: Keira Brooks Date: Wed, 18 Sep 2024 15:12:45 -0600 Subject: [PATCH] Python packaging and distribution (#26) * first pass at python packaging and distribution * fix formatting * fix email addresses for commit * don't worry about it * add index.rst files * add max depth * appropriate spacing * 3 spaces is the right amount of space * pull out the terminology section to its own document * fixing headings * Updated formatting and organization * update formatting and logical progression * make it make sense * Fix headings * Update based on reviewer comments and add clarity --- docs/source/index.rst | 1 + docs/source/programming_languages/index.rst | 7 + .../programming_languages/python/index.rst | 8 + .../python/packaging_and_distribution.md | 185 ++++++++++++++++++ .../python/terminology.md | 40 ++++ guideline_template.md | 4 +- 6 files changed, 244 insertions(+), 1 deletion(-) create mode 100644 docs/source/programming_languages/index.rst create mode 100644 docs/source/programming_languages/python/index.rst create mode 100644 docs/source/programming_languages/python/packaging_and_distribution.md create mode 100644 docs/source/programming_languages/python/terminology.md diff --git a/docs/source/index.rst b/docs/source/index.rst index b1cb31f..9760354 100644 --- a/docs/source/index.rst +++ b/docs/source/index.rst @@ -7,5 +7,6 @@ Welcome to the LASP Developer's Guide! :maxdepth: 1 licensing + programming_languages/index data_management/index workflows/index diff --git a/docs/source/programming_languages/index.rst b/docs/source/programming_languages/index.rst new file mode 100644 index 0000000..f05b170 --- /dev/null +++ b/docs/source/programming_languages/index.rst @@ -0,0 +1,7 @@ +Programming Languages +===================== + +.. toctree:: + :maxdepth: 1 + + python/index diff --git a/docs/source/programming_languages/python/index.rst b/docs/source/programming_languages/python/index.rst new file mode 100644 index 0000000..932cbd3 --- /dev/null +++ b/docs/source/programming_languages/python/index.rst @@ -0,0 +1,8 @@ +Python +===================== + +.. toctree:: + :maxdepth: 1 + + terminology + packaging_and_distribution diff --git a/docs/source/programming_languages/python/packaging_and_distribution.md b/docs/source/programming_languages/python/packaging_and_distribution.md new file mode 100644 index 0000000..3a3d7b6 --- /dev/null +++ b/docs/source/programming_languages/python/packaging_and_distribution.md @@ -0,0 +1,185 @@ +> **Warning:** More information is needed to complete this guideline. + +# Python Packaging and Distribution +Examples of Python packaging and distribution options and how to use them. + +## Purpose +> **Warning** Need to add an explanation of how this guideline supports DS workflows, meets internal and external +> policies, and aids in collaboration and our overall success + +## Options +The options for Python packaging and distribution that we often see used at LASP are: +- [PyPI](#packaging-for-pypi--pip-install-) +- [Conda](#packaging-for-conda--conda-install-) + +## Packaging for PyPI (`pip install`) + +### PyPI resources: + +- [PyPI Help Page](https://pypi.org/help/) + +- [Setting up a PyPI account](https://pypi.org/account/register/) + +- [Getting a PyPI access token](https://pypi.org/help/#apitoken) + + +### Built-In (`build` + `twine`) + +> **Warning**: Need to add introductory paragraph that summarizes Built-In + +#### How to use Built-In +Python Packaging User Guide: https://packaging.python.org/en/latest/ +The link below is a fairly complete tutorial. There are also instructions there for using various other build tools: +https://packaging.python.org/en/latest/tutorials/packaging-projects/ + +#### Built-In resources + +- [Python Packaging User Guide](https://packaging.python.org/en/latest/) + +#### Setuptools Example – Library Package +
+ setup.py + +``` +""" +Setup file for the science data processing pipeline. + +The only required fields for setup are name, version, and packages. Other fields to consider (from looking at other +projects): keywords, include_package_data, requires, tests_require, package_data +""" +from setuptools import setup, find_packages + +# Reads the requirements file +with open('requirements.txt') as f: + requirements = f.read().splitlines() + +setup( + name='my_py_library', + version='0.1.0', + author='Jane Doe, John Doe, This is just a str', + author_email='jane.doe@lasp.colorado.edu', + description='Science data processing pipeline for the instrument', + long_description=open('README.md', 'r').read(), # Reads the readme file + python_requires='>=3.8, <4', + url='https://some-git.url', + classifiers=[ + "Natural Language :: English", + "Topic :: Scientific/Engineering", + "Topic :: Scientific/Engineering :: Astronomy", + "Programming Language :: Python :: 3.8", + "Operating System :: MacOS :: MacOS X", + "Operating System :: POSIX :: Linux", + ], + packages=find_packages(exclude=('tests', 'tests.*')), + package_data={ + "my_py_library": [ + "some_necessary_config_data.json", + "calibration_data/*" + ] + }, + py_modules=['root_level_module_name',], + install_requires=requirements, + entry_points={ + 'console_scripts': [ + 'run-processing=my_py_library.cli:main', # package.module:function + ] + } +) +``` +
+ +### Poetry + +> **Warning**: Need to add introductory paragraph that summarizes Poetry + +[Poetry Build and Publish Docs](https://python-poetry.org/docs/cli/#build) + +How to Publish to PyPI from Poetry +``` +poetry lock +poetry install +poetry version +poetry build +PYPI_USERNAME=__token__ +PYPI_TOKEN= +poetry publish # You will be prompted for your PyPI credentials if you don't provide the environment variables +``` + +#### Poetry Project Configuration Example – Library Package + +
+ pyproject.toml + + ``` + # pyproject.toml + # See: https://python-poetry.org/docs/pyproject/ + + [tool.poetry] + name = "my_python_package" + version = "0.1.0" + description = "Science data processing library and applications for some instrument." + authors = [ # Alphabetical + "Jane Doe ", + "John Doe " + ] + + # Configure private PyPI repo to download packages + [[tool.poetry.source]] + name = "lasp-pypi" # This name will be used in the configuration to retrieve the proper credentials + url = "https://artifacts.pdmz.lasp.colorado.edu/repository/lasp-pypi/simple" # URL used to download your private packages + + # Dependency specification for core package + [tool.poetry.dependencies] + python = "^3.9" + astropy = "^4.2.1" + h5py = "^3.3.0" + numpy = "^1.21.0" + spiceypy = "^4.0.1" + lasp-packets = "1.2" + requests = "^2.26.0" + SQLAlchemy = "^1.4.27" + psycopg2 = "^2.9.2" + cloudpathlib = {extras = ["s3"], version = "^0.6.2"} + + # Development dependencies + [tool.poetry.dev-dependencies] + pytest-cov = "^2.12.1" + pylint = "^2.9.3" + responses = "^0.14.0" + pytest-randomly = "^3.10.2" + moto = {extras = ["s3"], version = "^2.2.16"} + + # Script entrypoints to put in installed bin directory + [tool.poetry.scripts] + sdp = 'my_python_package.cli:main' + + # Poetry boilerplate + [build-system] + requires = ["poetry-core>=1.0.0"] + build-backend = "poetry.core.masonry.api" + ``` +
+ +## Packaging for Conda (`conda install`) +> **Warning**: Need a volunteer to expand on Conda + +### How to install and use Conda +https://conda.io/projects/conda-build/en/latest/user-guide/tutorials/build-pkgs.html + +> Conda Develop: +> There is a conda subcommand called `conda develop`, but it is not actively maintained. The maintainers of +conda recommend using `pip install` to install an editable package in development mode. +> See: https://github.com/conda/conda-build/issues/1992 + +## Useful Links +Here are some helpful resources: + +- [Python Packaging User's Guide](https://packaging.python.org/en/latest/) +- [The Hitchhiker's Guide to Python - Packaging your Code](https://docs.python-guide.org/shipping/packaging/) +- [The Sheer Joy of Packaging](https://python-packaging-tutorial.readthedocs.io/en/latest/index.html) +- [Package Python Projects the Proper Way with Poetry](https://hackersandslackers.com/python-poetry-package-manager/) +- [Poetry Documentation](https://python-poetry.org/docs/) +- [Setuptools Documentation](https://setuptools.pypa.io/en/latest/) +- [Building conda packages from scratch](https://conda.io/projects/conda-build/en/latest/user-guide/tutorials/build-pkgs.html) + +Credit: Content taken from a Confluence guide written by Gavin Medley diff --git a/docs/source/programming_languages/python/terminology.md b/docs/source/programming_languages/python/terminology.md new file mode 100644 index 0000000..84937d3 --- /dev/null +++ b/docs/source/programming_languages/python/terminology.md @@ -0,0 +1,40 @@ +# Python Terminology + +Some Python terminology that a user might encounter, particularly when working through this Python guide. + +## Purpose +Like all programming languages, Python has some terminology that is unique to it and it is helpful to have that language +explained. This page may be updated over time so that it holds the most useful terminology to those that use this +developer's guide. + +### Library Packages vs Applications vs Scripts + +Python is a highly flexible interpreted language. This means it's easy to get started, quick to write, and easy to run. +Unfortunately, this also means it's very easy to write code that works in one context but not in others or code that is +robust but isn't designed to be inherited. + +**Package**: A `python` package, intended for redistribution, containing objects that serve as building blocks for +other developers to use. Examples are `numpy`, `pytest`, and `sqlalchemy`. + +> **Note** While most people think of a Python package as the above definition, a Python "package" is technically +> any directory containing an `__init__.py` file. + +**Application**: A python project (which may or may not be a packaged distribution) that provides specific and possibly +configurable functionality. Examples are Poetry, the AWS CLI, the Conda CLI, the Green Unicorn WSGI HTTP server, any +Django "app". + +**Script**: Pretty much anything else written in Python. One could arguably say that a Script is just a trivial Application +with little configuration or portability. Scripts are usually run with `python my_script.py argv`. They tend to be +difficult to maintain, update, or distribute. + +**Packaging Tooling**: Not only managing a local environment, but also providing tooling for developing, building, and +distributing python packages for other users. While Conda does support this use case (it's how one creates and +distributes conda packages to `conda-forge`), Poetry and `setuptools` are much easier to develop with (for PyPI) and Poetry +boasts a similar dependency resolver to Conda. One major drawback to Conda in packaging is that there is no notion of an +"editable install" so the developer is forced to build and test end user functionality (e.g. script entrypoints) over +and over instead of simply making code changes in place. + +## Useful Links +Helpful links to additional resources on the topic + +Credit: Content taken from a Confluence guide written by Gavin Medley diff --git a/guideline_template.md b/guideline_template.md index c5f567d..c99d67d 100644 --- a/guideline_template.md +++ b/guideline_template.md @@ -13,4 +13,6 @@ An explanation of the options available for this guideline (could be one or more Detailed instructions or general guidance for implementation of the guideline ## Useful Links -Helpful links to additional resources on the topic \ No newline at end of file +Helpful links to additional resources on the topic + +Credit: Content taken from a Confluence guide written by