Removal of the vendor folder - open discussion

[ZyanKLee]

This idea has been voiced once or twice already.

What are the pro's and con's of that?

• How can we make sure, supply chain attacks will be discovered?
• How can we avoid getting screwed over by some dependency maintainer in a foul mood converts their project into protest-ware?

On the one hand we already forked two projects into our org, but I'm not too keen on doing that for 6-7 github-hosted and 10+ gioui.org-hosted golang deps as well.
Can we avoid it?

[Dadadah]

We already have the go.mod file to keep track of what versions of the dependencies we're using, and I don't see much of a benefit by uploading the vendor folder. I also don't believe that dependencies being deleted is a real issue; we can probably find duplicates of a lost dependency in most cases (eg. a toml library, there are multiple available.) We can mitigate it by reducing external dependencies and be precise with our dependency choices.

[TheDukeofErl]

I initially had relatively strong feelings about this in terms of seeing the vendor folder remove but have cooled down a bit after reading up a bit on some of the history behind it, in which people seem to have very strong opinions in that the vendor code should be committed.

Pros

• Drastically lowers the total lines of code committed to the repository: by a large margin, most of the code in the repo is vendor code, bringing the repo to ~450k lines. After eliminating most of the c/ code and vendor/ code, the codebase ends up at a quite spritely 4000 lines (both of these numbers are off my memory from when I was doing code inspections so they should just be ballpark, I'm sure they won't be exact).
• Allows devs to use local, already grabbed vendor code via go mod: this probably really isn't that big an issue in reality.

Cons

• Open to supply chain attacks: this should be mitigated by go.sum. I don't even want to think about the possibility of more protest-ware... I'm just lumping it in with this since it is a supply chain attack, it's just extra frustrating.
• Risk of dependencies being deleted: I can't think of a great way to get around this one. If a project that is relied on is deleted or deprecated it probably has to be swapped out for something no matter what, even if we have it in the vendor folder or otherwise copied, but it would really put a fire under one's proverbial butt. Like @Dadadah suggested, depending on the dependency there are probably other options that can be swapped in.

I think I still sit on the remove the vendor folder side of things but can really go either way. At a bare minimum, a go mod verify should probably be added somewhere to the Makefile to avoid issues with the vendor folder. At the very least then it will be being checked against go.mod and go.sum. We can also choose to strategically fork things that may be at risk, though predicting the future is often in vain.

I might come back around and edit this or add replies as I think and learn more about this. Up til now I haven't worked with Go at all so I'm doing a fair share of learning as I go through issues, code, and history.

[ZyanKLee]

Seems like we have a consensus here. At least no one supporting the other side spoke up for it.

What do you see to be required for the removal?
Simply purging the folder seems to be enough, right?

@OldiLo if you would un-conflict and rebase your PR #269, I think we are now at the point to discuss it.

[Dadadah]

To delete the vendor folder you only need to add it to .gitignore and remove the folder from git. Go mod is automatically processed during go test and go build

[lawl]

Everything is vendored because i'm strongly against npm style left-pad dependency management.

I think people take on dependencies way too lightly. And as already mentioned here, supply chain attacks are not a theoretical problem.

It's quite possible that i swing too far to the other side of npm-style dependencies, but my philosophy is that if you take one on, you must consider that code (and all its dependencies, recusively) as your liability. (Exceptions confirm the rule, std-lib etc.)

(I'm still not interested in picking this back up, but just an explaination why everything is vendored. I also dont understand how you vendor something without committing the vendor folder, but i also dont smoke whatever bestpractices(tm) people come up with all the time.)

[lawl]

Some more thoughts: the go.sum contains the hashes, so from this point of view it should not matter as long as you trust the hash function. But vendoring allows you to actually easily diff everything that changes when you update a dependency. Also it makes it psychologocally clear that you too, are now responsible for this code, because you're shipping it.

Last but not least: you dont need internet to build it, everything's there, just type make.

So whatever you do, this was not an accident, and i still stand by that decision and would do it the exact same way again.

Hiding code in a different repo doesn't mean you're not still shipping it to YOUR users.

[ZyanKLee]

Hey @lawl,

thanks a lot for explaining your view of that issue. Those are all valid points and I fully understand them.

As a devops operations engineer I came across several events in the recent past where we were suddenly on high alert because some npm packages were being compromised and we needed to make sure we do not use them (also not in proxy / as sub-dependency) and to find a way to avoid such issues for the future.

Therefore I really understand the issue and share the worry about something malicious creeping in through a dependency chain somewhere.

One option we may have, is to have a look at a dependency repository like Nexus3 or Artifactory. Up till now I only looked at them from a corporation point of view and I do not know if they offer special deals for OSS projects.

On the other hand those big diffs you posted, where someone may have hidden malware, were mostly about the vendor folder, right? That is definitely a con for vendoring the deps.

[lawl]

On the other hand those big diffs you posted, where someone may have hidden malware, were mostly about the vendor folder, right?

No its not. Go checks that the vendor folder and go.sum match. Deleting it achieves nothing in that sense, if you trust sha256 or whatever go.sum uses you're guaranteed the same code.

And even if it didnt, you could delete it, let go refetch it, and then diff it against whats committed here. You should get an empty diff or somethings obviously wrong. That would be a stupid way to hide since its trivial to spot.

The issue is that i had reviewed that diff for obvious maliciousness. You cant trust that review anymore, and i don't want to do it again.

The vendor debate is entirely unrelated to that, because you just close your eyes of that problem, doesn't mean its not there. Vendor or not doesn't matter.

So in a way this is an argument FOR vendoring. Because now you see the problem.

[AXDOOMER]

I would like to install Snyk on this repository when I'll get write access to this organization. It will help us prevent supply chain attacks because this tool will surveil our dependencies for anything malicious, which will prevent us from releasing a binary that contains malicious code if a supply chain attack is known. It will also automatically update vulnerable dependencies to a fixed version. I believe it will eliminate most worries we can have if vendors is removed.

[RihanArfan]

I think we shouldn't commit vendor but definitely rely on GitHub Actions much more than is currently used. For commits, automatically running go mod verify can easily be done.

[RihanArfan]

There are plenty of companies out there focusing on security in open source. https://deepsource.io/ and https://docs.snyk.io/products/snyk-open-source/language-and-package-manager-support/snyk-for-golang is something that can be used for free on OSS repos like this too.

[ZyanKLee]

I do not understand why we would use go mod verify when we are not committing the vendor folder? In that case go build in our actions would download the deps on every build anew.
So we can be quite sure they match what is in go.mod and go.sum, right?

[TheDukeofErl]

Does Go automatically run go mod verify or something equivalent when doing a build? If it doesn't it may be worth including in the Makefile yet as a way to ensure the someone doing a local build has good code cached locally (e.g. ~/go/pkg/mod/).

There probably isn't really much an advantage for things on Github itself once code is committed assuming the vendor folder is removed.

[aaomidi]

I'm a strong believer in vendoring dependencies. For what it's worth I used to be against it but my experience working @letsencrypt made me change my view on this.

There's a few advantages of vendoring and very little of not doing so from what I understand. Advantages:

1. Updating dependencies doesn't happen opaquely. Validating and doing a review on dependency updates becomes part of the normal code review cycle. You effectively allow yourself to utilize the same code review system to extend to reviewing third party dependencies.

2. It adds a cost to adding dependencies. This may seem like a disadvantage, however if you consider third party dependencies as part of the whole package you're shipping then it makes sense to be a bit more strict about what dependencies and dependency chains we may be introducing.

3. (Nit) it allows for faster builds for people building from source. You don't need to reach out to various sources to get the entire package.

Disadvantages:

1. Repo size.

2. Increase in "chore" commit sizes to keep stuff up to date.

3. The need to do go mod verify during builds, etc.

I don't think the repo size will matter for a project of this scale. It probably won't go through so much churn to make it unmaintainable.

Since I use this software effectively daily, I'd be fine helping with the maintenance churn/PR churn if there is a need :)

[ZyanKLee]

If we remove the vendor folder we still need to review changes being done to dependencies when they are updated to be sure they were not tampered with.

So we will not be spared from this tedious work.

The only thing changing is: we will not do that work in our repository, but in their original repos.

I've finally come to the conclusion to go with the removal. It at least clears the repo up some more.

[mackshot]

I fully agree with your point of view. Removing the folder does not mean shutting the eyes to the changes in the dependencies. When updating the dependency it is important not only to check, whether everything still works as expected it is also important to check whether there is something "bad" going on or not and whether there are new interesting features :)
The dependencies are similar for me as a git submodule... you should take care of all changes in a git submodule as well, but separate the code to a different repository where discussions, changes and reviews can be done.
Edit: As others before mentioned I suggest to add /vendor to .gitignore

[Technetium1]

I want to add the submodules are unpleasant, and I genuinely hate them every time I've been forced to interact with them. I'm not the only one... In light of this, I would like to recommend literally anything but submodules.

[pikdum]

How can we make sure, supply chain attacks will be discovered?
How can we avoid getting screwed over by some dependency maintainer in a foul mood converts their project into protest-ware?

I use https://www.mend.io/free-developer-tools/renovate/ (https://github.com/marketplace/renovate) at my day job to help with this. It's basically a better Dependabot, and makes a PR for dependency updates with things like adoption rate, success rate, and a link to the code diff. No experience with using it for Go, but probably works well as long as dependencies are pinned.

[The-Munk]

A very different approach I didn't see discussed would be shipping vendor with release versions. These versions could be branches and these branches could fetch and extract a tarball containing that version/branches Vendor folder. The Head branch would be a rolling release, fetching the latest, being unstable. When it comes time to push a new release version, the diffs get audited.

So head would have vendor as submodules,
Each version branch would have a tarball of vendor - a static vendor folder you're comfortable shipping for that release version
The head branch is just for development, and end users should use the latest version - a full tarball they can build offline

This way, you don't really have to worry quite as much about what you're shipping, as what you're shipping will all have been checked and approved except the bleeding edge head branch, which regular end users should be steered away from using.

Maybe none of that would work for reasons I don't understand. Just thought I'd throw in my 2 cents as an outsider having stumbled upon this little issue and become vaguely interested. Take it for what it's worth

[polarathene]

Any security audit/concerns belonging to the upstream dependencies should be handled upstream. It's not beneficial to projects to all independently do their own internally. So long as the source of the dependencies is being trusted, you update for the project and leverage checksums as your trust factor. It's not going to get much better than that?

Vulnerabilities/exploits can exist long before they're caught. Committing them into the project in full doesn't magically protect from that, you can get changes that reviewers cannot confidently assess (or lack the awareness of such) where such issues can slip through or introduce the exploit in a non-obvious manner.

I couldn't imagine personally reviewing every single software update on my linux system line by line for example, you draw the line somewhere.

---

As for a copy, we have forks/mirrors and other services that archive/snapshot such. I strongly disagree about dependencies being part of the main projects history. They're a different kind of change, probably better suited with git LFS or a separate branch if not repo if insisting on it.

For cloning locally to build, you've got to do a shallow clone otherwise you're pulling in a lot more weight lugged around from vendor/ which is redundant. You're also placing trust in a third-party (the cloned project), not the upstream, that no tampering is done, that security updates are addressed quickly and properly (do you verify every Go project you'd use that they're carrying over the same fix via diff lines, or is it better to confirm with package management from a central source?). It doesn't seem to scale well as a consumer and I'm pretty sure I've witnessed breakage/issues when a Go project had a chain of dependencies on each other with each manually managing their own copies in git.

In the CI, you can leverage caching between runs. Cache can persist for a duration before it's dropped, and be downloaded if needed, should you need to fallback to it due to issues with upstream.

---

At least I'm having a hard time seeing worthwhile pragmatic benefits of having the source of dependencies committed.

As a user of a project, it doesn't instil more confidence in the project being trustworthy, more likely the opposite. I understand that opinion can be different when you're the one performing the review, or it's an internal product, but I've seen that go wrong too.

[jvalecillos]

I stumbled upon the vendor folder while trying to build the app today, and I found this discussion and I wanted to share my 2 cents.

I hope this doesn’t get into a war of opinions since I see that people already have strong positions on the matter for something that should be actually trivial. But I wanted to share my point of view since I think this is a super nice project and really helpful tool.

First of all, I think there’s an over-concern for the security or thrust on the dependencies but there’s virtually no difference between having them committed in the vendor folder versus just using go.mod in terms of security. It is the same thing because thanks to the go.sum hashes you will download the very same version all the time.

Every time you update a dependency it is the job of the maintainers to check what changed between versions, and if there’s a vulnerability introduced in a dependency you just simply stick to the older version (pinned by the version in go.mod and in turn verified by the hash in go.sum) and wait until the upstream version is updated (or fork the project if you are not patient). Moreover, the repo owner already activated dependabot on the repository you will get security updates as soon there’s a new CVA. On top of that, there are even more mainstream projects using the same dependencies/infrastructure (and programming language) for which this is not a problem. Think about Docker, Kubernetes, Prometheus, Terraform, etc. and put this project in perspective.

Second, go mod is the official tool from the programming language to add, maintain and update dependencies, why to reinvent the wheel? There is an argument of being able to build the application offline, but at that point you already cloned the project from a repository in GitHub so you already need the Internet. In addition to the half of users using this project for internet calls (oh!, the irony). Besides, you really need an offline version of the application that's what the releases are for. You can simultaneously add a binary release or a source release with the vendor folder included.

[aaomidi]

< there’s virtually no difference between having them committed in the vendor folder versus just using go.mod in terms of security.

The files you download are the same, however, when its vendored in you actually have to go through codereview rather than just bumping a version number.

Second, go mod is the official tool from the programming language to add, maintain and update dependencies, why to reinvent the wheel?

go mod vendor is also officially supported by Go. That argument doesn't work here.

Google itself vendors its dependencies internally :)

[jvalecillos]

when its vendored in you actually have to go through codereview rather than just bumping a version number.

You should never update the dependencies and later inspect what changed between them, as your comment seems to suggest. On the contrary, you are supposed to do it the other way around: if you observe if there's a new version on the dependency that brings any benefit or patches a security issue, only then, you bump the dependency in the go.mod file. In other words, you are always in control of when and how you update your dependencies

go mod vendor is also officially supported by Go

vendor is a sub-command or parameter of go mod that tells the tool to download the dependencies in the vendor directory. It is not a different tool if that's what you are trying to say. Therefore, my argument still stands.