You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Puppet Agent removed a workaround for concurrent-ruby < 1.1.9 that is required to prevent request threads from leaking memory. This was intended to be paired with upgrading the version of concurrent-ruby shipped across the product to 1.1.10. However, because of packaging constraints some products were upgraded to 1.1.9 and some to 1.1.10, and in the confusion Puppet Server wasn't upgraded at all.
This affects users running Puppet Server with versions of the Puppet Agent from 7.25.0 & 8.1.0 onwards. The Puppet Server version doesn't matter, but those versions of the Puppet Agent were shipped in Platform and PE releases from this summer and autumn.
Describe the Solution You Would Like
Puppet Server should not leak memory.
To resolve the issue we should update concurrent-ruby to be inline with the versions shipped in the platform. At least 1.1.10, and preferably latest (1.2.2 at this time).
Describe Alternatives You've Considered
According to the concurrent-ruby maintainers the issue should be resolved post 1.1.9, however the underlying structure that was doing the leaking remains until a major refactor removed it in 1.2.0. There's some concern that we may still leak objects until that major refactor. However, we don't have conclusive evidence either way. We had done some tests that showed 1.1.10 did not leak, but since this issued escaped, we will consider that evidence inconclusive until we do an RCA.
Alternatively, we could revert the Puppet Agent change, but that change is required to take up 1.2.0+ of concurrent-ruby.
Additional Context & Workaround
Users may resolve this issue on deployed systems by manually updating the concurrent-ruby gem, either through the package type and the puppetserver_gem provider, or by running puppetserver gem install --no-document -v 1.2.2 concurrent-ruby.
Users should not need to remove the older version of concurrent-ruby, but if they wish to they can run gem uninstall -i /opt/puppetlabs/server/data/puppetserver/vendored-jruby-gems concurrent-ruby to do so.
We need to have this fix in all of our products' source trees before the holidays for releases early next year.
Bolt was the primary reason for only upgrading to 1.1.9 in part of the platform, and Puppet itself was only tested with 1.1.10, so we need to make sure that Bolt's reason for not upgrading was merely conservative packaging pins and that neither Bolt nor Puppet usage of concurrent-ruby break on an upgrade to 1.2.2 (latest). - Initial code review doesn't show any API breakages, and initial testing hasn't discovered any issues, but we are continuing on both fronts.
Issue
Puppet Agent removed a workaround for concurrent-ruby < 1.1.9 that is required to prevent request threads from leaking memory. This was intended to be paired with upgrading the version of concurrent-ruby shipped across the product to 1.1.10. However, because of packaging constraints some products were upgraded to 1.1.9 and some to 1.1.10, and in the confusion Puppet Server wasn't upgraded at all.
This affects users running Puppet Server with versions of the Puppet Agent from 7.25.0 & 8.1.0 onwards. The Puppet Server version doesn't matter, but those versions of the Puppet Agent were shipped in Platform and PE releases from this summer and autumn.
Describe the Solution You Would Like
Puppet Server should not leak memory.
To resolve the issue we should update concurrent-ruby to be inline with the versions shipped in the platform. At least 1.1.10, and preferably latest (1.2.2 at this time).
Describe Alternatives You've Considered
According to the concurrent-ruby maintainers the issue should be resolved post 1.1.9, however the underlying structure that was doing the leaking remains until a major refactor removed it in 1.2.0. There's some concern that we may still leak objects until that major refactor. However, we don't have conclusive evidence either way. We had done some tests that showed 1.1.10 did not leak, but since this issued escaped, we will consider that evidence inconclusive until we do an RCA.
Alternatively, we could revert the Puppet Agent change, but that change is required to take up 1.2.0+ of concurrent-ruby.
Additional Context & Workaround
puppetserver gem install --no-document -v 1.2.2 concurrent-ruby
.gem uninstall -i /opt/puppetlabs/server/data/puppetserver/vendored-jruby-gems concurrent-ruby
to do so.The text was updated successfully, but these errors were encountered: