Use CI to build release binaries

[squeek502]

Currently only have a few precompiled binaries for Luvi's newer releases. We've been meaning to set this up for a while but never gotten around to it. Example of when release deploying was added to luv via travis: luvit/luv@6b901a2

Related issues: #187, #181

Here's what we had in v2.7.6: https://github.com/luvit/luvi/releases/tag/v2.7.6

• Windows Use appveyor to build/release Windows binaries #201
• Linux Use travis to build/release Linux binaries  #202
  • i686
  • x86_64
• Mac/Darwin active travis auto publish release #203
• ARM
  • aarch64 / arm64
  • armv7
  • armv6
• FreeBSD

[creationix]

Yes, this would be great if it could be automated. I think travis now has OSX guests and we have appveyor for windows, so it should be possible for most of these.

I was building freebsd and linux-686 in virtualbox images on my workstation, but I don't think those are as needed as others if it proves to be a problem.

[zhaozg]

Enough if win-x64,Mac-x64,Linux-x64 auto github-release for ci.Others are optional

[creationix]

Sure, that would be the bare minimum. Next important on the list I think would be arm32 and arm64 variants. Those platforms are getting more and more popular.

[zhaozg]

That means arm platform need manual release, I do not have news about arm free CI platform

[creationix]

Sometimes you can build them inside docker or chroots using qemu-user-static to emulate arm CPUs

[squeek502]

I'm going to start working on the Windows targets now. Will do the Linux/Mac ones after that.

[squeek502]

For Linux:

• Got something mostly working for x86_64 and i686 using travis, docker, and holy-build-box (master...squeek502:master / example release) but the resulting binaries are gigantic--need to look into why that is. EDIT: It's debug info.
  EDIT#2: Fixed and made a PR: Use travis to build/release Linux binaries  #202

For Mac:

• Right now we don't have Mac CI turned on at all, and I don't have a Mac to do any testing, so will probably need some help with this when I get around to it

For ARM:

• Funnily enough, the top google result for "travis arm" (for me) is an old article about getting Travis to cross-compile Luvit (1.0) for ARM: https://www.tomaz.me/2013/12/02/running-travis-ci-tests-on-arm.html
• It might be easier now to use Docker; more possible solutions here: Support Android / Arm travis-ci/travis-ci#3376

For FreeBSD:

• I know sourcehut.org/sr.ht offers FreeBSD CI and it should be possible to integrate that with Github in some way (seen it mentioned that it's possible before). EDIT: More info: https://man.sr.ht/tutorials/builds.sr.ht/github-integration.md

[zhaozg]

@squeek502 I'll continue to finish Mac base your code.

[zhaozg]

The PR #203 add auto publish for osx.

here is ci log. here is real effect

Notice: travis need GITHUB_OAUTH_TOKEN to do deploy.

[squeek502]

Nice, thanks @zhaozg. Should we release v2.9.3 to try out all the new automatic publishing?

EDIT: Probably makes sense to merge #204 before a new release

[squeek502]

Here goes nothing:

• https://github.com/luvit/luvi/releases/tag/v2.9.3
• https://travis-ci.org/luvit/luvi/builds/513312262
• https://ci.appveyor.com/project/racker-buildbot/luvi/builds/23468810

EDIT: Appveyor failed due to auth_token:

Error creating GitHub release: Provider setting not found or it's value is empty. If secure setting is used please check that value was encrypted (or YAML was exported) while being logged under correct account.
Parameter name: auth_token

Either I messed something up or we'll need to use an auth token from a different account than mine.

[zhaozg]

I do git tag -d v2.9.3 && git tag v2.9.3 -m "v2.9.3" && git push origin v2.9.3 --force

[squeek502]

I do git tag -d v2.9.3 && git tag v2.9.3 -m "v2.9.3" && git push origin v2.9.3 --force

I went with a different approach, and appveyor seems to be working now: https://github.com/luvit/luvi/releases/tag/v2.9.3-temp

Will clean things up once the build finishes: I'll move the Windows binaries to the v2.9.3 tag and delete the v2.9.3-temp tag. Then I'll revert c6a410e

[zhaozg]

Sorry for that, I will do nothing in next 4 hours. Please re tag v2.9.3 --force

[squeek502]

Ok, I think we're good to go. The Windows binaries have been moved to the v2.9.3 tag and still have the proper version (luvi.exe v2.9.3), and the v2.9.3-temp tag has been deleted. The v2.9.3 tag seems fine to me, please verify that it's ok. And any future releases should work correctly (I think what happened is that I accidentally had copied some whitespace with the auth_token I encrypted the first time).

Here's an example of the random build error I mentioned in #202 btw: https://travis-ci.org/luvit/luvi/jobs/513318947

[zhaozg]

Checked package download from https://github.com/luvit/luvi/releases/tag/v2.9.3 test pass.

• luvi-regular-Darwin_x86_64
• luvi-regular-Linux_i686
• luvi-regular-Linux_x86_64
• luvi-regular-Windows-amd64.exe
• luvi-regular-Windows-amd64.lib
• luvi-regular-Windows-ia32.exe
• luvi-regular-Windows-ia32.lib
• luvi-tiny-Darwin_x86_64
• luvi-tiny-Linux_i686
• luvi-tiny-Linux_x86_64
• luvi-tiny-Windows-amd64.exe
• luvi-tiny-Windows-ia32.exe

[squeek502]

Checked package download from https://github.com/luvit/luvi/releases/tag/v2.9.3 test pass.

Tested the rest except the Windows .lib files. Everything seems good.

[zhaozg]

to @squeek502
I think change https://github.com/luvit/luvi/blob/master/.travis.yml#L47 to https://github.com/luvit/luvi/blob/master/.travis.yml#L69, and remove https://github.com/luvit/luvi/blob/master/.travis.yml#L48-L49 will also do publish with out docker,

[squeek502]

Using docker & holy-build-box has portability benefits: https://github.com/phusion/holy-build-box

[creationix]

I've had good experience using docker combined with qemu-user-static to build stuff in arm containers on x86 kernels/hosts

[squeek502]

Looks like we can use Github actions to build arm binaries. @truemedian is doing just that here:

https://github.com/truemedian/luvit-bin/blob/main/.github/workflows/manual.yml#L154-L196

[truemedian]

My approach in luvit-bin for ARM binaries is.. not the best, nor would I recommend it. It requires running a full system vm on github's actions machines, which turns a 2-3 minute build process into a 30-40 minute build process

[squeek502]

Surely it's better than nothing. If we don't want to take the hit for every CI, we could only build arm for tagged releases.

[truemedian]

I decided to take another stab at this for building just luvi (this can be rather easily extended to both lit and luvit if need be).

ARM + OpenSSL binaries take the longest, so thats what these times are going to based off of.

from scratch: 37m 31s.
full ccache: 9m 56s.

Surprisingly, this is about on par with the current CI setup in terms of speed (when the ccache is full, having to compile slows down the process massively for ARM builds)

This allows for multiple things:

• All (or most of) the CI in one place.
• Running tests on PRs to prevent serious breakages.
• +armv6/armv7 builds with "minimal effort"

A few things of note:

• arm64v8 (aarch64) is currently unbuildable due to some issue in arm64 luajit (anyone want to pin it down? its a segfault / "attempt to compare two table values", possibly related)
• no FreeBSD support

[SinisterRectus]

I think we're ready to release 2.15.0, which will use Actions to build the release binaries.

Following this, I believe we can close #284, #275, and #279 and address #238, #266 and #280.

[Bilal2453]

Honestly, one thing I wanted to change before releasing 2.15.0 is updating miniz to latest master branch, but I am having issues building and testing it so cba at the moment

[Bilal2453]

Should be ready now, miniz and LPeg have been updated to latest.

[SinisterRectus]

Indeed. As for the remaining checkboxes:

We now support aarch64 / arm64, but I guess we have to drop arm6 and arm7?

FreeBSD is supported but not with Actions.

[truemedian]

We don't necessarily have to drop armv6 and armv7, however there are very few existing and maintained projects aimed towards wide glibc support. I didn't include them in my CI overhaul because I couldn't find a nice solution towards the maintenance of such an environment.

However, if one exists then it would be relatively trivial to add it to the build process.

[zhaozg]

I didn't include them in my CI overhaul because I couldn't find a nice solution towards the maintenance of such an environment.

Try download arm cross build chains from https://developer.arm.com/downloads/-/gnu-a.

[truemedian]

There are two problems there:

We need to build with an old enough version of glibc that it will work on practically any distro using luvi.

We cannot cross compile. CMake does not understand the difference between host and target binaries. This distinction is required to build Luajit, and our build process is not suited to cross compile OpenSSL.

[zhaozg]

Yes, We need to build cmake toolchains config for that, I think consider it as long time tasks, not done that in 2.15.0

[truemedian]

Just took a look around and https://github.com/vmactions/freebsd-vm seems it may be a decent fit for building FreeBSD binaries, the same organization also has similar actions for openbsd, solaris and dragonfly if we'd like to support those

[Bilal2453]

The rest of remaining architectures are of lower-priority to a lot of people, I believe. Supporting arm64 is the most important one beside the other major architectures because of the new ARM chips, I am not sure if Raspberry Pi is arm64 or armv7 but that could potentially influence the decision.

We can probably take a similar approach for armv6 and armv7, having QEMU do its magic.

I think it is fine to have a luvi release as of the current state.