Plan for development

[diazona]

@sjlongland I figured I'd lay out a rough plan, for clarity and so we can avoid stepping on each others' toes:

• Adding support for as many setuptools/pyproject.toml entries as we can. Some of them will be tricky (like version numbers where setuptools_scm is involved), but there are going to be a lot of straightforward ones, which is where your gist comes in handy.
• Documentation
• Testing and coverage reporting
• Linting and static checks
• Build infrastructure (automated testing, packaging, and publishing to PyPI)

If you want to "claim" any particular part of the work, I guess let me know here? That way we don't duplicate effort. We can also break these things up into specific issues in the issue tracker and assign them, but I don't know if the project needs that level of organization just yet.

FWIW I've recently been working on build infrastructure for a couple other projects, so that part is fresh on my mind... I could get that set up soon. Unless you wanted to do it, of course.

[sjlongland]

Yeah, there's going to be some tricky edge cases (like setuptools_scm). I'm thinking plug-ins like that can probably be deferred until we get the core functionality working as not all projects make use of all the plug-ins.

Mocking the setuptools infrastructure shall be a fun little job. The good news is the constructor for the command gets "fed" an instance of the distribution, so presumably we can feed it something else.

Build infrastructure I had a little exposure to, one thing I used to do with pypi uploads is digitally sign these (OpenPGP), but I note lately pypi doesn't seem to make the OpenPGP signature "visible" anywhere. I'm thinking if the CI system does the publishing, it's going to need its own key, rather than me using my own private key (these days, that's a YubiKey USB token) -- that's assuming pypi publishes the signature somewhere -- otherwise I guess the point of doing it is probably moot (and there's nothing stopping me manually downloading a package, digitally signing it, and publishing the signature somewhere for people to find).

Documentation is something I'm still learning how to do properly with Python. Never quite got the hang of Sphinx… I'm too spoiled by the likes of Doxygen in the C/C++ world that practically does everything for you. One gotcha here is that some of the Markdown libraries for docs are really lagging in terms of support for newer Sphinx versions. This might not seem to matter, but it can be a nightmare for distribution packagers. (I've picked fights with packages that wanted a specific version of sphinx many times when updating my Gentoo laptop -- despite this, Gentoo still offers the best environment I've seen for Python yet.) I'm more comfortable with Markdown than RST, but sadly, it's the latter that has the better first-class support in Sphinx.

Regarding linting and static checks, I think I've got a lot to learn. I know some of my projects target Python 3.5 to be as widely compatible as practical so much of my own code does not use type annotations, however here this is a green-fields project, and also as a tool, is not actually a dependency of any given project that might use it. It'll be installed, used to generate pyproject.toml, then hopefully, job done, the changes committed and the tool is no longer needed.

So I think we should be targetting versions of Python used in "still supported" OS releases. Debian Buster (release 10) shipped Python 3.7, which supports a lot of the newer type hint features and still receives long-term support. If we aim for that, it should practically work anywhere including quite old systems.

I think this will be an interesting project, in so far as there's a lot of little tricks to learn about making something that's properly maintainable.

[diazona]

Hmm so I messed up: Python 3.5 does support type hinting, with PEP 484 and the typing module. Maybe we should try to support 3.5, at least to start with. I put up a PR (#3) to implement that. If we find something that makes it too difficult to support 3.5 then we can drop it.

[sjlongland]

Yeah, 3.5 is probably just marginally too old for this. We can try it and see how it goes, but it was very "new" in 3.5, so might be a little primitive. Sometimes it takes them a couple of bites at the cherry before they get something that "works".

It's like for async, the minimum you can realistically support is 3.5 even though asyncio was introduced in 3.3 (and trollius backported it to 2.7). aioax25 used to clamp its minimum at 3.4 (because that still worked on many older OSes but had good async support via coroutines/yield from), but then Python 3.10 dropped coroutine, so I was forced to move to async / await, which meant moving to Python 3.5 as it was too burdensome to support both 3.4 and 3.10.

Let's try it and see how we go, but I'm fine with bumping it up a notch or two to 3.6 or 3.7 if the APIs between 3.5 and newer in the typing module are too radically different.

[diazona]

Indeed I ran into something that doesn't work in 3.5: variable annotations (PEP 526). So I just bumped it up to 3.6 in #3. If this project ever takes off and anyone really wants 3.5 support, we'll figure it out.

[sjlongland]

Yep, let's not make this any more difficult than necessary. Given the once-off use case of this tool, spinning up a temporary container off Docker or using a chroot with a newer Python installation is not a difficult thing to achieve at all.

People should be able to spin up an appropriate temporary environment, mount their work into it, run the tool to generate the pyproject.toml files they need, then commit those new files and go back to whatever older version of Python they were wanting to use. If they're building this into CI environments to auto-generate pyproject.toml from setup.py repeatedly, they're doing it backward :-).

[diazona]

Definitely. And I even thought about maybe setting up a web service that can do that: check out any project's repository, install all the dependencies, run python setup.py pyproject, and display the resulting pyproject.toml file to the user. But that's pretty far down the line.

[diazona]

I'm thinking that I can try to add a really basic test and get the whole tox/pytest workflow working, and then after that we should set up a Github Actions workflow to run tests so that it's easy to tell when a change is breaking something. Will you want to handle the workflow part @sjlongland? (if not, I'm fine doing it)

[sjlongland]

BTW: if I seem a bit slow lately, I'll apologise right up front… I had picked up a chest bug at the last emergency comms event (not COVID-19, but there's lots being 'shared' for everyone) and had intended to "rest up" this week-end, quietly working on the laptop.

One escapee cat who decided he'd go out Friday morning and disappear until Sunday morning meant I was out and about looking for him instead of recovering (or working on this project). Talk about timing.

Will be visiting the doctors today, hopefully some antibiotics and rest, and I'll be able to get into this fully. :-)

[diazona]

No worries! Sorry to hear about the illness (and the cat, although I guess he got found?), and hope you have a quick and thorough recovery. There's plenty enough work on this project to last.

[sjlongland]

Yeah, he returned on his own, and has not been let out since. Cheeky little sod. One is never sure whether to smother the little buggers in hugs and kisses or wring their necks when they do things like that. :-)

As for the illness, just talked with the doctor today -- not as bad as feared, so I'm on antibiotics now. That and keeping warm, taking it easy (physically) should mean we beat that quickly. I think cold weather + stress searching for a missing feline is not exactly a recipe for beating a chest bug. I was half expecting to be marched off for a chest X-ray and receiving a pneumonia diagnosis. (Been there, done that!)

Looking thorough the set-up, I see there's actually a lot of little tricks to learn from this project alone. I did fumble through a set-up of Github Actions for a few projects, but the set-up you have here is by far much more elegant, and I think there is much to learn about how this is structured.

I had a reasonable set-up with Travis CI on some other projects, but of course that got shut down.

I guess my problem is I'm self-taught… having only dabbled in Python prior to 2010, then doing it as a business where things like CI and automated testing was not considered a priority. (We've changed though! Now pretty much everything gets tested in CI, and I try to keep unit test coverage near 100%)

So no, the set-up progress here is great. When I get a moment, I should have a close look at the dummy script I created, and see if I can align that with the pyproject.toml schema to get this ball rolling in earnest.

[diazona]

Sounds good! To be fair, a lot of what I have in this project, including the current CI setup, was merged in from https://github.com/jaraco/skeleton, so I can't claim credit for that. But I do have some ideas of changes to make, based on experience I picked up from work and from my other open-source project. For now I'll keep working on CI and the project infrastructure and such things.

[diazona]

FYI @sjlongland (in case you didn't see it yourself) I created a bunch of issues for individual fields of pyproject.toml, just going off the type definition I added in #26. These should help us keep track of the work that needs to be done and avoid stepping on each others' toes. Any time you want to work on adding support for any of those fields, just assign the corresponding issue(s) to yourself, so I'll know that you've got those covered, and I'll do likewise if I work on any of them (although for now I am still focusing on infrastructure stuff).

Note that I've defined these issues to focus solely on static definitions, i.e. not the cases where certain fields in pyproject.toml are defined dynamically by plugins, like setuptools_scm. Like you mentioned upthread, those are more complicated and I'm not sure how they're going to break down into tasks.

[sjlongland]

Yep, I think this is a much better plan than adding them on an ad-hoc basis. This definitely helps "check off" all the different fields and ensure nothing is missed.

[diazona]

I guess at this point we have a lovely build and packaging infrastructure set up but the project doesn't really do anything 😂 so I'm thinking of implementing some of those actual features. I had been leaving them for you to get first priority on them, but if you're busy or whatever, perhaps I should go ahead with a few?

The way I see it, getting even a minimal version published may help generate some interest in the package, which isn't the biggest deal but it would be nice to see it get some use.

[sjlongland]

Yeah, I do apologise for that… there's been quite a bit going on and even this week-end… I'm rounding up equipment belonging to my workplace that's been here since the pandemic, to bring back to the new office next week.

That, and sometimes projects jump the queue out of necessity -- NitroKey V3 work was needed because I bought two intending to use them with Android, only to discover it isn't supported -- so I had to roll my sleeves up and get to work there.

Today I've got a few things to do for work (I want to test an idea for over-the-air updates before I pack the gear up), then I should have some time to devote to this. :-)

[diazona]

No worries! I know life happens and we have to work around stuff like that. I just wanted to make sure we keep in sync.