Improve build performance on Travis

[ioquatix]

I've implemented https://travis-ci.org/ioquatix/vips-thumbnail which includes travis tests.

The build time on travis is pretty horrendous. Is there something we can do to improve this? Precompiled binaries? Also, is it possible to install without sudo? That also improves performance a bit.

[jcupitt]

I think you need to cache the vips build, so just build the vips lib once, then reuse that binary for all your rvm tests.

https://docs.travis-ci.com/user/migrating-from-legacy/#How-Do-I-Cache-Dependencies-and-Directories%3F

I'll see if I can get the main ruby-vips repo to do this.

[ioquatix]

Have you done this for ruby-vips?

[jcupitt]

No, but it should be simple.

(edit: stupid kb, posted too soon)

[jcupitt]

It seems to be working, it's reusing the vips binary now:

https://travis-ci.org/jcupitt/ruby-vips/jobs/213787127

https://github.com/jcupitt/ruby-vips/tree/use-travis-containers

What do you think?

[jcupitt]

I've merged to master, and got php-vips doing this as well.

[jcupitt]

Also, I like your thumbnailer. You're right, it should get quicker with vips_thumbnail(), when 8.5 is done.

[ioquatix]

I like what you did, and I think I improved it a bit. I put all vips specific environment in install file.

https://github.com/ioquatix/vips-thumbnail/blob/master/install-vips.sh

Invoked using

before_script:
  - source install-vips.sh

[jcupitt]

I'm not sure that will work, will it? The gem will need GI_TYPELIB_PATH to run, for example, and it'll need PKG_CONFIG_PATH to build, so they'll need to be in the global env table.

My idea was to share install-vips.sh with php-vips, so I wanted to have all the config in the .travis.yml file. If you have the config there it's easy to add things like test against various vips versions too.

[ioquatix]

It works.

That's why it use source. It works in the current "shell instance" as it were rather than executing a separate shell.

As much as possible should be in install-vips.sh, otherwise it's cumbersome to manage .travis.yml IMHO.

[jcupitt]

Ah OK, I hadn't noticed the source, duh.

I like having all the config stuff at the top so it's easy to see. And global env vars should be defined in env: global: (imo), it's easy to miss them if they are set elsewhere.

I moved some other stuff into install-vips.sh, does that look a bit cleaner?

[ioquatix]

The only thing I'd be inclined to put in .travis.yml would be things which I'd want to vary on the build matrix, so perhaps:

- VIPS_VERSION_MAJOR=8
- VIPS_VERSION_MINOR=4
- VIPS_VERSION_MICRO=5

Everything else should go in install-vips.sh. Keeping .travis.yml as light as possible makes the most sense to me. Think about if you moved to a different CI platform - you'd want to reuse as much as possible. But, in any case, it's up to you.

[ioquatix]

I thin w.r.t. versions, it would be even better to just write VIPS_VERSION=8.4.5 as that approach is more common IMHO.

[ioquatix]

I realised something: every single thing that depends on vips, needs to use this script. It's really heavy for travis, and it's really heavy for things which depend on vips.

Specifically, each project needs to:

• Declare all the appropriate packages/addons (may change which would have a significant maintenance burden).
• Include the install-vips.sh script and maintain it.
• The build process is so long that each project needs to have a cache.

What we should do is try to get a decent version of vips added to travis whitelist.

@BanzaiMan just wondering if you are able to provide some advice here. I'll try to kick off the whitelist process. @jcupitt is there a debian/ubuntu package for libvips? Is it up to date?

[ioquatix]

Is this the current package for trusty ubuntu? http://packages.ubuntu.com/trusty/libvips-dev - It seems a bit old.

We need to follow the procedure here: https://github.com/travis-ci/apt-package-whitelist

I take it the package we want is libvips-dev?

[ioquatix]

There is a more recent package here: https://launchpad.net/ubuntu/+source/vips/8.4.5-1 perhaps we can use a PPA to get the latest versions of the packages available on Travis?

[jcupitt]

Hi, sure, a PPA would be great.

There have been quite a few attempts to get vips whitelisted for travis, I guess you've seen. For example:

travis-ci/apt-source-safelist#68

And many others.

[ioquatix]

That's odd, when I searched for it I didn't get any results. It's almost like there are too many open issues.

There is a PR here: travis-ci/apt-source-safelist#137 but it's a bit out of date now.

[felixbuenemann]

You could also pupload travis compatible linux binaries to an S3 bucket and use that for builds. The build could either be a tarball unpacked to a local directory with all the dependencies or a deb-package that still requires the dependencies like libpng to be installed in the system.