Building Orbit can be extremely painful - How can we make it better?

[pierricgimmig]

The amount of effort to build Orbit can be overwhelming. I have built Orbit on many different machines, on both Windows and Linux, over the past few years, and things rarely go smoothly. I still find myself regularly dedicating hours, if not days, to building or debugging build issues. I'm wondering if we should change paradigm so that contributors could be up and running in reasonable time.

Some of the pain points:

• Build time: Building Orbit for the first time takes a lot of time. It's still unclear to me why we can't use binaries from Conan instead of having to compile common dependencies. For example, why do I have to compile openssl on Windows using msvc2019? That must be one of the most common configurations out there. Or, should we ever have to compile LLVM? What about Grpc or abseil? Shouldn't we have pre-built binaries for common compiler configurations?

• Build is brittle: As it turns out, I can't compile openssl on my machine (Can't compile Windows - openssl/1.1.1s #4663). I have compiled it before, but I guess my setup is now slightly different, and the esoteric perl script that is part of the build process for that lib is buggy.

• Conan cache hides build issues: If I have successfully built all the dependencies once, then I can rely on my locally cached binaries for subsequent builds. This hides issues where some external dependency no longer compiles for X reason. We then have a "it works on my machine" problem that makes Orbit maintainers unaware of build issues.

• We are depending on things we don't control: Maybe a Conan recipe relies on a code package hosted on a website that is no longer reachable, maybe we have exceeded the number monthly downloads, maybe a recipe was updated and a bug was introduced, etc, etc. This means that even if we have a nice CI badge stating that the build is green, there is no guarantee that a potential contributor will be able to compile Orbit, even if they are on the most common configuration. We are offloading the responsibility of reliability to Conan, so far with mixed results.

• Expanding on the previous point, we have historically put a lot of effort making sure that every commit compiles, it's far from obvious that this would be the case given that we don't control our dependencies.

Those issues are a massive barrier to entry to even experienced contributors, let alone new ones eager to build Orbit for the first time.

What can we do to minimize time-to-breakpoint™ , the time it takes for a new contributor to build, launch and break in the Orbit code? Perhaps providing a development docker containers with preinstalled compiler and dependencies? It seems like hosting pre-built binaries for all dependencies of common configurations would already go a long way.

[beckerhe]

Hi Pierric,
I totally see your point. And we also saw the problem - that's why we made it possible to build Orbit with system dependencies on Ubuntu 22.04 LTS. The current build scripts don't use that workflow yet, but the CI does, and presubmits verify that you can build Orbit on Ubuntu 22.04.

This of course didn't take Windows into account. And I still struggle to see the use-case for a Windows build as long as we don't have Windows profiling support. Do you have any plans here?

If not one option would be to entirely drop conan support and the Windows build and only support the build with system dependencies. This would also be very close to your container suggestion. Because any standard Ubuntu 22.04 LTS container would be a suitable development container. This could also run in WSL if developing on Windows was desired.

I also have some thoughts on how to make the Conan workflow better - but let's first agree on a future for the Windows build.

[pierricgimmig]

Hi Henning, thanks for your input. It's great to see that the Linux build is getting much better btw, thanks for that.

I think the Windows build should be maintained, yes. In some ways it is already working. We already have sampling profiling that works on Windows through ETW, as well as manual instrumentation. We also have the option of connecting a Windows client to a Linux profiling session, which is a great feature, and even works for WSL. Also, I did spend a bit of time extracting the original Windows dynamic instrumentation into it's own library in the past few days. I don't think I'm far from a working solution.

Is there something preventing us from having prebuilt binaries for Windows dependencies for let's say VS2019 and VS2022?

[beckerhe]

Hi Pierric, great to hear that you're making progress on the Windows profiling side.

Is there something preventing us from having prebuilt binaries for Windows dependencies for let's say VS2019 and VS2022?

With Conan it's not that easy unfortunately. You have to set up and maintain your own Artifactory server for that - you can't just upload the files to somewhere. (Uploading to "Github Packages" would be ideal, but they don't support Conan)

I still think that the prebuilts on Conan's public repo is our best shot - if we stick with Conan. There are a few things we can do to ensure we use actually existing packages.

1. Currently we build relwithdebinfo by default. We should change that to release because there are no prebuilts for relwithdebinfo.
2. We have a diamond in our dependency graph. grpc and libssh2 both require openssl. So conan requires us to disambiguate which version of openssl we wanna use and then compiles both dependencies with the explicitly defined version of openssl. Ideally we always wanna use the openssl version required by the grpc package because it's far bigger than libssh2 and we wanna avoid recompiling that. Not sure if it's possible to tell Conan "please always use the openssl version required by the grpc package", but I will check.
3. We might also need to tweak the build script to always use the latest version of the Conan recipes because outdated recipes (from a previous build of Orbit) might not have prebuilts anymore on the public Conan repo server. Also there are regularly new versions of the grpc package uploaded to Conan-Center which update the required version of openssl - whenever there is a security fix release. Ideally we wanna use the latest version here and still have prebuilts.

Another option might be to switch to some other package management system, but I haven't explored that yet. Unfortunately we can't provide premade Docker containers for development - at least not for Windows. The Visual Studio license doesn't allow distributing Docker containers with MSVC installed. And since the main problem is Windows development (as far as I understand it), this won't help.

[pierricgimmig]

Thanks for the info @beckerhe, I think those are good areas to explore. Cheers!

[phaedon]

I'm curious whether adding Bazel support (and fetching dependencies from the BCR / Bazel Central Repository) is on the roadmap, given that this is a Google-supported library?