Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update concurrent-ruby to latest #2795

Closed
justinstoller opened this issue Dec 6, 2023 · 1 comment
Closed

Update concurrent-ruby to latest #2795

justinstoller opened this issue Dec 6, 2023 · 1 comment

Comments

@justinstoller
Copy link
Member

justinstoller commented Dec 6, 2023

Issue

Puppet Agent removed a workaround for concurrent-ruby < 1.1.9 that is required to prevent request threads from leaking memory. This was intended to be paired with upgrading the version of concurrent-ruby shipped across the product to 1.1.10. However, because of packaging constraints some products were upgraded to 1.1.9 and some to 1.1.10, and in the confusion Puppet Server wasn't upgraded at all.

This affects users running Puppet Server with versions of the Puppet Agent from 7.25.0 & 8.1.0 onwards. The Puppet Server version doesn't matter, but those versions of the Puppet Agent were shipped in Platform and PE releases from this summer and autumn.

Describe the Solution You Would Like

Puppet Server should not leak memory.

To resolve the issue we should update concurrent-ruby to be inline with the versions shipped in the platform. At least 1.1.10, and preferably latest (1.2.2 at this time).

Describe Alternatives You've Considered

According to the concurrent-ruby maintainers the issue should be resolved post 1.1.9, however the underlying structure that was doing the leaking remains until a major refactor removed it in 1.2.0. There's some concern that we may still leak objects until that major refactor. However, we don't have conclusive evidence either way. We had done some tests that showed 1.1.10 did not leak, but since this issued escaped, we will consider that evidence inconclusive until we do an RCA.

Alternatively, we could revert the Puppet Agent change, but that change is required to take up 1.2.0+ of concurrent-ruby.

Additional Context & Workaround

  • Users may resolve this issue on deployed systems by manually updating the concurrent-ruby gem, either through the package type and the puppetserver_gem provider, or by running puppetserver gem install --no-document -v 1.2.2 concurrent-ruby.
  • Users should not need to remove the older version of concurrent-ruby, but if they wish to they can run gem uninstall -i /opt/puppetlabs/server/data/puppetserver/vendored-jruby-gems concurrent-ruby to do so.
  • We need to have this fix in all of our products' source trees before the holidays for releases early next year.
  • Bolt was the primary reason for only upgrading to 1.1.9 in part of the platform, and Puppet itself was only tested with 1.1.10, so we need to make sure that Bolt's reason for not upgrading was merely conservative packaging pins and that neither Bolt nor Puppet usage of concurrent-ruby break on an upgrade to 1.2.2 (latest). - Initial code review doesn't show any API breakages, and initial testing hasn't discovered any issues, but we are continuing on both fronts.
  • See Ship latest concurrent-ruby with runtime projects puppet-runtime#775 for discussion on what versions go in which branch.
justinstoller added a commit to justinstoller/puppetserver that referenced this issue Dec 6, 2023
justinstoller added a commit to justinstoller/puppetserver that referenced this issue Dec 7, 2023
joshcooper added a commit that referenced this issue Dec 7, 2023
(#2795) Update concurrent-ruby to latest to fix memory leak
@justinstoller
Copy link
Member Author

This was resolved in December of 2023.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

1 participant