Windows Containers Foreign Layer Discussion

[NAWhitehead]

Recently, we posted a blog announcing the update to the End User License Agreement for Windows Container Base Images. We had the initial plan to remove Foreign Layers from the base images, however, we received feedback on how Foreign Layers provides benefit to many developers. Through further investigation, we found that many users would be negatively impacted by removing Foreign Layers without other fixes.

Below are some of the negative/beneficial impacts if we keep as is or remove the Foreign Layer:

Removal Impacts:

• Concerning building pipelines in open source, developers will have to pull and push the image back to their repository
• Developers that work from home may not have good internet connection which can exponentially increase the push time
• Possible increase in storage costs

Removal benefits:

• Ecosystem friendly/in-line with Linux
• No longer will need to utilize the non-distributable artifact flag
• Air gapped user will be able use the base images in their own ACR (Azure Container Registry)

No Removal Benefits:

• Push times will maintain at current levels
• The image is from Microsoft (until image signing is solved)

Currently we are investigating solutions that can either mitigate the increase in push times or mitigate the drawbacks to Foreign Layers. We are seeking more outside opinions around this topic. Are Foreign Layers beneficial to you? What possible mitigations are out there?

[imjasonh]

Removal Impacts:

Concerning building pipelines in open source, developers will have to pull and push the image back to their repository
Developers that work from home may not have good internet connection which can exponentially increase the push time
Possible increase in storage costs

I think it's worth noting that all of these could be mitigated by having smaller images. Or even being more careful about layering.

When layers were foreign it wasn't as crucial to care about these best practices, but if they become regular layers it may be worth considering them.

[NAWhitehead]

I do agree that smaller images sizes could mitigate this however, that would be a long-term goal which could leave us waiting for a while before removing foreign layers would be practical or even an option.

[imjasonh]

More benefits of removing non-distributable layers:

• Layers that the user has pushed to a registry in their cloud provider (e.g., GCR, ECR) will get better download latency to the VM pulling the image, since the blob will come from GCS or S3 instead of mcr.microsoft.com. I could attempt to measure this if this is useful.
• Users should be able to narrow the scope of their firewall rules to disallow requests to mcr.microsoft.com; they presumably already depend on being able to pull from their cloud's container image registry.

[NAWhitehead]

I agree that without foreign layers that locality ad firewall scope will be improved. This was permitted before the licensing change and it's achievable without removing foreign layers (limited to the same customer) now uses the flag, however it's not perfect as that changes the digest of the layers.
I would like to get your opinion on the increase to push and pull times when using the base images. Its important to note that many developers do not work onsite anymore and may not have the most stable internet connection as well. From simple tests of using an image without foreign layers we saw push times increase that were originally a few seconds to several minutes. I can elaborate further here if you have questions. Do you see an issue with push times increase, and possible storage cost increase?

[NAWhitehead]

Thank you for sharing the benefits I do really appreciate the feedback.

[imjasonh]

I agree that the increase in storage and latency from having these be regular layers is something worth considering, but as in my previous comment I think the solution to that concern is for those layers to become smaller, or take better advantage of layering, than they do today.

I understand that when they were foreign layers these weren't really concerns, so maybe some unhygienic habits were permissible, and undoing those decisions will take work. I don't think these should be considered blockers though, unless that work is actively being done, and clear exit criteria are set. Otherwise, experience says nothing will ever get done, and they'll remain foreign layers forever, despite no longer needing them due to previous licensing restrictions.

Suddenly having to push a 2GB(!!) layer when it was previously foreign is painful, but luckily that will only have to be done when the layer changes. If the layer changes often, there's an opportunity to split the fast-changing parts of the filesystem into separate layers from slower-changing parts. Folks have been doing this successfully on Linux-based images for years, so there's a fair bit of built-up experience and tooling in the community. I'd be happy to chat about ways we can identify slower-changing parts and split those into separate layers, if it unblocks this work.

[NAWhitehead]

Thank you for the reply.
I think it would be very beneficial for me to hear about "identifying slower-changing parts and split them into separate layers." Understanding that better will help me understand the possible short-term mitigations.

[SteveLasker]

To add to @imjasonh's comments, all registries support de-dupping. It's on the first push to the registry that needs to upload the base layers. The same issue works for home users. Whether the base layers come from MAR, or they come from their registry, they still need to be pulled locally. Through local caching, they're only pulled once.
When pushing, they should only see a layer push for the first time that image is put into the registry. ACR and other registry support de-duping across repositories, so this is only a pain for a brand new registry.
While we'd all like to see smaller layers, we all know how difficult dieting is, and windows has already done a tremendous amount of image reduction. We could likely be better about new versions, minimizing churning base layers, but I'd still suggest we're not balancing the complexity with the e2e usability of keeping the layers with the image, by default.
As Microsoft starts shipping signed images, this also becomes complex as the signature is of the manifest, which either does or doesn't include foreign layers.