Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

PipelineBusV2 block shutdown #16657

Open
kaisecheng opened this issue Nov 7, 2024 · 1 comment · May be fixed by #16682
Open

PipelineBusV2 block shutdown #16657

kaisecheng opened this issue Nov 7, 2024 · 1 comment · May be fixed by #16682
Assignees
@yaauie
Labels
bug int-shortlist status:work-in-progress

Comments

@kaisecheng
Copy link
Contributor

kaisecheng commented Nov 7, 2024 •
edited by andsel
Loading

Logstash version: 8.15.3

Pipeline to pipeline has upgraded to use PipelineBusV2. Logstash is unable to shutdown in 8.15.3 while downgrade to 8.14.3 has no such issue.

jstack found deadlock

Found one Java-level deadlock:
=============================
"pipeline_1":
  waiting to lock monitor 0x00007f5fd00062d0 (object 0x000000069bcb4fa8, a java.util.concurrent.ConcurrentHashMap$Node),
  which is held by "pipeline_2"

"pipeline_2":
  waiting to lock monitor 0x00007f5fe8001730 (object 0x00000006b3233ea0, a java.util.concurrent.ConcurrentHashMap$Node),
  which is held by "pipeline_3"

"pipeline_3":
  waiting to lock monitor 0x00007f6000002cd0 (object 0x0000000695d3fcf0, a org.jruby.gen.LogStash$$Plugins$$Builtin$$Pipeline$$Input_934266047),
  which is held by "Converge PipelineAction::StopAndDelete<pipeline_4>"

"Converge PipelineAction::StopAndDelete<pipeline_4>":
  waiting to lock monitor 0x00007f5fe8001730 (object 0x00000006b3233ea0, a java.util.concurrent.ConcurrentHashMap$Node),
  which is held by "pipeline_3"
Java stack information for the threads listed above 
"pipeline_1":
	at java.util.concurrent.ConcurrentHashMap.compute(java.base@21.0.4/ConcurrentHashMap.java:1931)
	- waiting to lock <0x000000069bcb4fa8> (a java.util.concurrent.ConcurrentHashMap$Node)
	at org.logstash.plugins.pipeline.PipelineBusV2.unregisterSender(PipelineBusV2.java:63)
	at java.lang.invoke.LambdaForm$DMH/0x00007f60a8fb2c00.invokeInterface(java.base@21.0.4/LambdaForm$DMH)
	at java.lang.invoke.LambdaForm$MH/0x00007f60a90ac800.invoke(java.base@21.0.4/LambdaForm$MH)
	at java.lang.invoke.DelegatingMethodHandle$Holder.delegate(java.base@21.0.4/DelegatingMethodHandle$Holder)
	at java.lang.invoke.LambdaForm$MH/0x00007f60a86ff800.guard(java.base@21.0.4/LambdaForm$MH)
	at java.lang.invoke.DelegatingMethodHandle$Holder.delegate(java.base@21.0.4/DelegatingMethodHandle$Holder)
	at java.lang.invoke.LambdaForm$MH/0x00007f60a86ff800.guard(java.base@21.0.4/LambdaForm$MH)
	at java.lang.invoke.Invokers$Holder.linkToCallSite(java.base@21.0.4/Invokers$Holder)
	at usr.share.logstash.logstash_minus_core.lib.logstash.plugins.builtin.pipeline.output.RUBY$method$close$0(/usr/share/logstash/logstash-core/lib/logstash/plugins/builtin/pipeline/output.rb:43)
	at usr.share.logstash.logstash_minus_core.lib.logstash.plugins.builtin.pipeline.output.RUBY$method$close$0$__VARARGS__(/usr/share/logstash/logstash-core/lib/logstash/plugins/builtin/pipeline/output.rb:42)
	at java.lang.invoke.DirectMethodHandle$Holder.invokeStatic(java.base@21.0.4/DirectMethodHandle$Holder)
	at java.lang.invoke.LambdaForm$MH/0x00007f60a86e8c00.invokeExact_MT(java.base@21.0.4/LambdaForm$MH)
	at org.jruby.internal.runtime.methods.CompiledIRMethod.call(CompiledIRMethod.java:139)
	at org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:112)
	at org.jruby.ir.targets.indy.InvokeSite.failf(InvokeSite.java:802)
	at org.jruby.ir.targets.indy.InvokeSite.failf(InvokeSite.java:816)
	at java.lang.invoke.LambdaForm$DMH/0x00007f60a86fd400.invokeVirtual(java.base@21.0.4/LambdaForm$DMH)
	at java.lang.invoke.LambdaForm$MH/0x00007f60a8ab5000.invoke(java.base@21.0.4/LambdaForm$MH)
	at java.lang.invoke.Invokers$Holder.linkToCallSite(java.base@21.0.4/Invokers$Holder)
	at usr.share.logstash.logstash_minus_core.lib.logstash.plugin.RUBY$method$do_close$0(/usr/share/logstash/logstash-core/lib/logstash/plugin.rb:98)
	at java.lang.invoke.LambdaForm$DMH/0x00007f60a955f000.invokeStatic(java.base@21.0.4/LambdaForm$DMH)
	at java.lang.invoke.Invokers$Holder.invokeExact_MT(java.base@21.0.4/Invokers$Holder)
	at org.jruby.internal.runtime.methods.CompiledIRMethod.call(CompiledIRMethod.java:152)
	at org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:148)
	at org.jruby.internal.runtime.methods.DynamicMethod.call(DynamicMethod.java:212)
	at org.jruby.RubyClass.finvoke(RubyClass.java:598)
	at org.jruby.runtime.Helpers.invoke(Helpers.java:708)
	at org.jruby.RubyBasicObject.callMethod(RubyBasicObject.java:349)
	at org.logstash.config.ir.compiler.OutputStrategyExt$SimpleAbstractOutputStrategyExt.close(OutputStrategyExt.java:270)
	at org.logstash.config.ir.compiler.OutputStrategyExt$AbstractOutputStrategyExt.doClose(OutputStrategyExt.java:137)
	at org.logstash.config.ir.compiler.OutputDelegatorExt.close(OutputDelegatorExt.java:121)
	at org.logstash.config.ir.compiler.AbstractOutputDelegatorExt.doClose(AbstractOutputDelegatorExt.java:75)
	at org.logstash.config.ir.compiler.AbstractOutputDelegatorExt$INVOKER$i$0$0$doClose.call(AbstractOutputDelegatorExt$INVOKER$i$0$0$doClose.gen)
	at org.jruby.internal.runtime.methods.JavaMethod$JavaMethodN.call(JavaMethod.java:841)
	at org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:90)
	at org.jruby.RubySymbol$SymbolProcBody.yieldInner(RubySymbol.java:1513)
	at org.jruby.RubySymbol$SymbolProcBody.doYield(RubySymbol.java:1528)
	at org.jruby.runtime.BlockBody.yield(BlockBody.java:108)
	at org.jruby.runtime.Block.yield(Block.java:189)
	at org.jruby.RubyArray.each(RubyArray.java:1981)
	at java.lang.invoke.LambdaForm$DMH/0x00007f60a86f4c00.invokeVirtual(java.base@21.0.4/LambdaForm$DMH)
	at java.lang.invoke.LambdaForm$MH/0x00007f60a8708800.invoke(java.base@21.0.4/LambdaForm$MH)
	at java.lang.invoke.LambdaForm$MH/0x00007f60a86fbc00.reinvoke(java.base@21.0.4/LambdaForm$MH)
	at java.lang.invoke.LambdaForm$MH/0x00007f60a86fc000.guard(java.base@21.0.4/LambdaForm$MH)
	at java.lang.invoke.LambdaForm$MH/0x00007f60a86fbc00.reinvoke(java.base@21.0.4/LambdaForm$MH)
	at java.lang.invoke.LambdaForm$MH/0x00007f60a86fc000.guard(java.base@21.0.4/LambdaForm$MH)
	at java.lang.invoke.Invokers$Holder.linkToCallSite(java.base@21.0.4/Invokers$Holder)
	at usr.share.logstash.logstash_minus_core.lib.logstash.java_pipeline.RUBY$method$shutdown_workers$0(/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:484)
	at java.lang.invoke.DirectMethodHandle$Holder.invokeStatic(java.base@21.0.4/DirectMethodHandle$Holder)
	at java.lang.invoke.Invokers$Holder.invokeExact_MT(java.base@21.0.4/Invokers$Holder)
	at org.jruby.internal.runtime.methods.CompiledIRMethod.call(CompiledIRMethod.java:152)
	at org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:148)
	at org.jruby.internal.runtime.methods.DynamicMethod.call(DynamicMethod.java:212)
	at org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:456)
	at org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:195)
	at org.jruby.ir.interpreter.InterpreterEngine.processCall(InterpreterEngine.java:346)
	at org.jruby.ir.interpreter.StartupInterpreterEngine.interpret(StartupInterpreterEngine.java:66)
	at org.jruby.ir.interpreter.InterpreterEngine.interpret(InterpreterEngine.java:76)
	at org.jruby.internal.runtime.methods.MixedModeIRMethod.INTERPRET_METHOD(MixedModeIRMethod.java:164)
	at org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:151)
	at org.jruby.internal.runtime.methods.DynamicMethod.call(DynamicMethod.java:212)
	at org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:456)
	at org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:195)
	at org.jruby.ir.interpreter.InterpreterEngine.processCall(InterpreterEngine.java:346)
	at org.jruby.ir.interpreter.StartupInterpreterEngine.interpret(StartupInterpreterEngine.java:66)
	at org.jruby.ir.interpreter.Interpreter.INTERPRET_BLOCK(Interpreter.java:118)
	at org.jruby.runtime.MixedModeIRBlockBody.commonYieldPath(MixedModeIRBlockBody.java:136)
	at org.jruby.runtime.IRBlockBody.call(IRBlockBody.java:66)
	at org.jruby.runtime.IRBlockBody.call(IRBlockBody.java:58)
	at org.jruby.runtime.Block.call(Block.java:144)
	at org.jruby.RubyProc.call(RubyProc.java:354)
	at org.jruby.internal.runtime.RubyRunnable.run(RubyRunnable.java:111)
	at java.lang.Thread.runWith(java.base@21.0.4/Thread.java:1596)
	at java.lang.Thread.run(java.base@21.0.4/Thread.java:1583)
"pipeline_2":
	at java.util.concurrent.ConcurrentHashMap.compute(java.base@21.0.4/ConcurrentHashMap.java:1931)
	- waiting to lock <0x00000006b3233ea0> (a java.util.concurrent.ConcurrentHashMap$Node)
	at org.logstash.plugins.pipeline.PipelineBusV2$AddressStateMapping.mutate(PipelineBusV2.java:137)
	at org.logstash.plugins.pipeline.PipelineBusV2.lambda$unregisterSender$5(PipelineBusV2.java:64)
	at org.logstash.plugins.pipeline.PipelineBusV2$$Lambda/0x00007f60a9554000.accept(Unknown Source)
	at java.lang.Iterable.forEach(java.base@21.0.4/Iterable.java:75)
	at org.logstash.plugins.pipeline.PipelineBusV2.lambda$unregisterSender$6(PipelineBusV2.java:64)
	at org.logstash.plugins.pipeline.PipelineBusV2$$Lambda/0x00007f60a946bc18.apply(Unknown Source)
	at java.util.concurrent.ConcurrentHashMap.compute(java.base@21.0.4/ConcurrentHashMap.java:1940)
	- locked <0x000000069bcb4fa8> (a java.util.concurrent.ConcurrentHashMap$Node)
	at org.logstash.plugins.pipeline.PipelineBusV2.unregisterSender(PipelineBusV2.java:63)
	at java.lang.invoke.LambdaForm$DMH/0x00007f60a8fb2c00.invokeInterface(java.base@21.0.4/LambdaForm$DMH)
	at java.lang.invoke.LambdaForm$MH/0x00007f60a90ac800.invoke(java.base@21.0.4/LambdaForm$MH)
	at java.lang.invoke.DelegatingMethodHandle$Holder.delegate(java.base@21.0.4/DelegatingMethodHandle$Holder)
	at java.lang.invoke.LambdaForm$MH/0x00007f60a86ff800.guard(java.base@21.0.4/LambdaForm$MH)
	at java.lang.invoke.DelegatingMethodHandle$Holder.delegate(java.base@21.0.4/DelegatingMethodHandle$Holder)
	at java.lang.invoke.LambdaForm$MH/0x00007f60a86ff800.guard(java.base@21.0.4/LambdaForm$MH)
	at java.lang.invoke.Invokers$Holder.linkToCallSite(java.base@21.0.4/Invokers$Holder)
	at usr.share.logstash.logstash_minus_core.lib.logstash.plugins.builtin.pipeline.output.RUBY$method$close$0(/usr/share/logstash/logstash-core/lib/logstash/plugins/builtin/pipeline/output.rb:43)
	at usr.share.logstash.logstash_minus_core.lib.logstash.plugins.builtin.pipeline.output.RUBY$method$close$0$__VARARGS__(/usr/share/logstash/logstash-core/lib/logstash/plugins/builtin/pipeline/output.rb:42)
	at java.lang.invoke.DirectMethodHandle$Holder.invokeStatic(java.base@21.0.4/DirectMethodHandle$Holder)
	at java.lang.invoke.LambdaForm$MH/0x00007f60a86e8c00.invokeExact_MT(java.base@21.0.4/LambdaForm$MH)
	at org.jruby.internal.runtime.methods.CompiledIRMethod.call(CompiledIRMethod.java:139)
	at org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:112)
	at org.jruby.ir.targets.indy.InvokeSite.failf(InvokeSite.java:802)
	at org.jruby.ir.targets.indy.InvokeSite.failf(InvokeSite.java:816)
	at java.lang.invoke.LambdaForm$DMH/0x00007f60a86fd400.invokeVirtual(java.base@21.0.4/LambdaForm$DMH)
	at java.lang.invoke.LambdaForm$MH/0x00007f60a8ab5000.invoke(java.base@21.0.4/LambdaForm$MH)
	at java.lang.invoke.Invokers$Holder.linkToCallSite(java.base@21.0.4/Invokers$Holder)
	at usr.share.logstash.logstash_minus_core.lib.logstash.plugin.RUBY$method$do_close$0(/usr/share/logstash/logstash-core/lib/logstash/plugin.rb:98)
	at java.lang.invoke.LambdaForm$DMH/0x00007f60a955f000.invokeStatic(java.base@21.0.4/LambdaForm$DMH)
	at java.lang.invoke.Invokers$Holder.invokeExact_MT(java.base@21.0.4/Invokers$Holder)
	at org.jruby.internal.runtime.methods.CompiledIRMethod.call(CompiledIRMethod.java:152)
	at org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:148)
	at org.jruby.internal.runtime.methods.DynamicMethod.call(DynamicMethod.java:212)
	at org.jruby.RubyClass.finvoke(RubyClass.java:598)
	at org.jruby.runtime.Helpers.invoke(Helpers.java:708)
	at org.jruby.RubyBasicObject.callMethod(RubyBasicObject.java:349)
	at org.logstash.config.ir.compiler.OutputStrategyExt$SimpleAbstractOutputStrategyExt.close(OutputStrategyExt.java:270)
	at org.logstash.config.ir.compiler.OutputStrategyExt$AbstractOutputStrategyExt.doClose(OutputStrategyExt.java:137)
	at org.logstash.config.ir.compiler.OutputDelegatorExt.close(OutputDelegatorExt.java:121)
	at org.logstash.config.ir.compiler.AbstractOutputDelegatorExt.doClose(AbstractOutputDelegatorExt.java:75)
	at org.logstash.config.ir.compiler.AbstractOutputDelegatorExt$INVOKER$i$0$0$doClose.call(AbstractOutputDelegatorExt$INVOKER$i$0$0$doClose.gen)
	at org.jruby.internal.runtime.methods.JavaMethod$JavaMethodN.call(JavaMethod.java:841)
	at org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:90)
	at org.jruby.RubySymbol$SymbolProcBody.yieldInner(RubySymbol.java:1513)
	at org.jruby.RubySymbol$SymbolProcBody.doYield(RubySymbol.java:1528)
	at org.jruby.runtime.BlockBody.yield(BlockBody.java:108)
	at org.jruby.runtime.Block.yield(Block.java:189)
	at org.jruby.RubyArray.each(RubyArray.java:1981)
	at java.lang.invoke.LambdaForm$DMH/0x00007f60a86f4c00.invokeVirtual(java.base@21.0.4/LambdaForm$DMH)
	at java.lang.invoke.LambdaForm$MH/0x00007f60a8708800.invoke(java.base@21.0.4/LambdaForm$MH)
	at java.lang.invoke.LambdaForm$MH/0x00007f60a86fbc00.reinvoke(java.base@21.0.4/LambdaForm$MH)
	at java.lang.invoke.LambdaForm$MH/0x00007f60a86fc000.guard(java.base@21.0.4/LambdaForm$MH)
	at java.lang.invoke.LambdaForm$MH/0x00007f60a86fbc00.reinvoke(java.base@21.0.4/LambdaForm$MH)
	at java.lang.invoke.LambdaForm$MH/0x00007f60a86fc000.guard(java.base@21.0.4/LambdaForm$MH)
	at java.lang.invoke.Invokers$Holder.linkToCallSite(java.base@21.0.4/Invokers$Holder)
	at usr.share.logstash.logstash_minus_core.lib.logstash.java_pipeline.RUBY$method$shutdown_workers$0(/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:484)
	at usr.share.logstash.logstash_minus_core.lib.logstash.java_pipeline.RUBY$method$shutdown_workers$0$__VARARGS__(/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:475)
	at java.lang.invoke.DirectMethodHandle$Holder.invokeStatic(java.base@21.0.4/DirectMethodHandle$Holder)
	at java.lang.invoke.LambdaForm$MH/0x00007f60a86e8c00.invokeExact_MT(java.base@21.0.4/LambdaForm$MH)
	at org.jruby.internal.runtime.methods.CompiledIRMethod.call(CompiledIRMethod.java:139)
	at org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:112)
	at org.jruby.ir.targets.indy.InvokeSite.failf(InvokeSite.java:802)
	at org.jruby.ir.targets.indy.InvokeSite.failf(InvokeSite.java:816)
	at java.lang.invoke.LambdaForm$DMH/0x00007f60a86fd400.invokeVirtual(java.base@21.0.4/LambdaForm$DMH)
	at java.lang.invoke.LambdaForm$MH/0x00007f60a8ab5000.invoke(java.base@21.0.4/LambdaForm$MH)
	at java.lang.invoke.Invokers$Holder.linkToCallSite(java.base@21.0.4/Invokers$Holder)
	at usr.share.logstash.logstash_minus_core.lib.logstash.java_pipeline.RUBY$method$run$0(/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:209)
	at usr.share.logstash.logstash_minus_core.lib.logstash.java_pipeline.RUBY$method$run$0$__VARARGS__(/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:189)
	at java.lang.invoke.DirectMethodHandle$Holder.invokeStatic(java.base@21.0.4/DirectMethodHandle$Holder)
	at java.lang.invoke.LambdaForm$MH/0x00007f60a86e8c00.invokeExact_MT(java.base@21.0.4/LambdaForm$MH)
	at org.jruby.internal.runtime.methods.CompiledIRMethod.call(CompiledIRMethod.java:139)
	at org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:112)
	at org.jruby.ir.targets.indy.InvokeSite.failf(InvokeSite.java:802)
	at org.jruby.ir.targets.indy.InvokeSite.failf(InvokeSite.java:816)
	at java.lang.invoke.LambdaForm$DMH/0x00007f60a86fd400.invokeVirtual(java.base@21.0.4/LambdaForm$DMH)
	at java.lang.invoke.LambdaForm$MH/0x00007f60a8ab5000.invoke(java.base@21.0.4/LambdaForm$MH)
	at java.lang.invoke.Invokers$Holder.linkToCallSite(java.base@21.0.4/Invokers$Holder)
	at usr.share.logstash.logstash_minus_core.lib.logstash.java_pipeline.RUBY$block$start$1(/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:146)
	at java.lang.invoke.DirectMethodHandle$Holder.invokeStatic(java.base@21.0.4/DirectMethodHandle$Holder)
	at java.lang.invoke.LambdaForm$MH/0x00007f60a8718800.invoke(java.base@21.0.4/LambdaForm$MH)
	at java.lang.invoke.LambdaForm$MH/0x00007f60a80b9c00.invokeExact_MT(java.base@21.0.4/LambdaForm$MH)
	at org.jruby.runtime.CompiledIRBlockBody.callDirect(CompiledIRBlockBody.java:141)
	at org.jruby.runtime.IRBlockBody.call(IRBlockBody.java:64)
	at org.jruby.runtime.IRBlockBody.call(IRBlockBody.java:58)
	at org.jruby.runtime.Block.call(Block.java:144)
	at org.jruby.RubyProc.call(RubyProc.java:354)
	at org.jruby.internal.runtime.RubyRunnable.run(RubyRunnable.java:111)
	at java.lang.Thread.runWith(java.base@21.0.4/Thread.java:1596)
	at java.lang.Thread.run(java.base@21.0.4/Thread.java:1583)
"pipeline_3":
	at org.logstash.plugins.pipeline.PipelineBusV2$AddressStateMapping.lambda$mutate$0(PipelineBusV2.java:148)
	- waiting to lock <0x0000000695d3fcf0> (a org.jruby.gen.LogStash$$Plugins$$Builtin$$Pipeline$$Input_934266047)
	at org.logstash.plugins.pipeline.PipelineBusV2$AddressStateMapping$$Lambda/0x00007f60a8f6df88.apply(Unknown Source)
	at java.util.concurrent.ConcurrentHashMap.compute(java.base@21.0.4/ConcurrentHashMap.java:1940)
	- locked <0x00000006b3233ea0> (a java.util.concurrent.ConcurrentHashMap$Node)
	at org.logstash.plugins.pipeline.PipelineBusV2$AddressStateMapping.mutate(PipelineBusV2.java:137)
	at org.logstash.plugins.pipeline.PipelineBusV2.lambda$unregisterSender$5(PipelineBusV2.java:64)
	at org.logstash.plugins.pipeline.PipelineBusV2$$Lambda/0x00007f60a9554000.accept(Unknown Source)
	at java.lang.Iterable.forEach(java.base@21.0.4/Iterable.java:75)
	at org.logstash.plugins.pipeline.PipelineBusV2.lambda$unregisterSender$6(PipelineBusV2.java:64)
	at org.logstash.plugins.pipeline.PipelineBusV2$$Lambda/0x00007f60a946bc18.apply(Unknown Source)
	at java.util.concurrent.ConcurrentHashMap.compute(java.base@21.0.4/ConcurrentHashMap.java:1940)
	- locked <0x000000069bd0d9a8> (a java.util.concurrent.ConcurrentHashMap$Node)
	at org.logstash.plugins.pipeline.PipelineBusV2.unregisterSender(PipelineBusV2.java:63)
	at java.lang.invoke.LambdaForm$DMH/0x00007f60a8fb2c00.invokeInterface(java.base@21.0.4/LambdaForm$DMH)
	at java.lang.invoke.LambdaForm$MH/0x00007f60a90ac800.invoke(java.base@21.0.4/LambdaForm$MH)
	at java.lang.invoke.LambdaForm$MH/0x00007f60a86ff400.reinvoke(java.base@21.0.4/LambdaForm$MH)
	at java.lang.invoke.LambdaForm$MH/0x00007f60a86ff800.guard(java.base@21.0.4/LambdaForm$MH)
	at java.lang.invoke.LambdaForm$MH/0x00007f60a86ff400.reinvoke(java.base@21.0.4/LambdaForm$MH)
	at java.lang.invoke.LambdaForm$MH/0x00007f60a86ff800.guard(java.base@21.0.4/LambdaForm$MH)
	at java.lang.invoke.Invokers$Holder.linkToCallSite(java.base@21.0.4/Invokers$Holder)
	at usr.share.logstash.logstash_minus_core.lib.logstash.plugins.builtin.pipeline.output.RUBY$method$close$0(/usr/share/logstash/logstash-core/lib/logstash/plugins/builtin/pipeline/output.rb:43)
	at usr.share.logstash.logstash_minus_core.lib.logstash.plugins.builtin.pipeline.output.RUBY$method$close$0$__VARARGS__(/usr/share/logstash/logstash-core/lib/logstash/plugins/builtin/pipeline/output.rb:42)
	at java.lang.invoke.DirectMethodHandle$Holder.invokeStatic(java.base@21.0.4/DirectMethodHandle$Holder)
	at java.lang.invoke.LambdaForm$MH/0x00007f60a86e8c00.invokeExact_MT(java.base@21.0.4/LambdaForm$MH)
	at org.jruby.internal.runtime.methods.CompiledIRMethod.call(CompiledIRMethod.java:139)
	at org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:112)
	at org.jruby.ir.targets.indy.InvokeSite.failf(InvokeSite.java:802)
	at org.jruby.ir.targets.indy.InvokeSite.failf(InvokeSite.java:816)
	at java.lang.invoke.LambdaForm$DMH/0x00007f60a86fd400.invokeVirtual(java.base@21.0.4/LambdaForm$DMH)
	at java.lang.invoke.LambdaForm$MH/0x00007f60a8ab5000.invoke(java.base@21.0.4/LambdaForm$MH)
	at java.lang.invoke.Invokers$Holder.linkToCallSite(java.base@21.0.4/Invokers$Holder)
	at usr.share.logstash.logstash_minus_core.lib.logstash.plugin.RUBY$method$do_close$0(/usr/share/logstash/logstash-core/lib/logstash/plugin.rb:98)
	at java.lang.invoke.LambdaForm$DMH/0x00007f60a955f000.invokeStatic(java.base@21.0.4/LambdaForm$DMH)
	at java.lang.invoke.Invokers$Holder.invokeExact_MT(java.base@21.0.4/Invokers$Holder)
	at org.jruby.internal.runtime.methods.CompiledIRMethod.call(CompiledIRMethod.java:152)
	at org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:148)
	at org.jruby.internal.runtime.methods.DynamicMethod.call(DynamicMethod.java:212)
	at org.jruby.RubyClass.finvoke(RubyClass.java:598)
	at org.jruby.runtime.Helpers.invoke(Helpers.java:708)
	at org.jruby.RubyBasicObject.callMethod(RubyBasicObject.java:349)
	at org.logstash.config.ir.compiler.OutputStrategyExt$SimpleAbstractOutputStrategyExt.close(OutputStrategyExt.java:270)
	at org.logstash.config.ir.compiler.OutputStrategyExt$AbstractOutputStrategyExt.doClose(OutputStrategyExt.java:137)
	at org.logstash.config.ir.compiler.OutputDelegatorExt.close(OutputDelegatorExt.java:121)
	at org.logstash.config.ir.compiler.AbstractOutputDelegatorExt.doClose(AbstractOutputDelegatorExt.java:75)
	at org.logstash.config.ir.compiler.AbstractOutputDelegatorExt$INVOKER$i$0$0$doClose.call(AbstractOutputDelegatorExt$INVOKER$i$0$0$doClose.gen)
	at org.jruby.internal.runtime.methods.JavaMethod$JavaMethodN.call(JavaMethod.java:841)
	at org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:90)
	at org.jruby.RubySymbol$SymbolProcBody.yieldInner(RubySymbol.java:1513)
	at org.jruby.RubySymbol$SymbolProcBody.doYield(RubySymbol.java:1528)
	at org.jruby.runtime.BlockBody.yield(BlockBody.java:108)
	at org.jruby.runtime.Block.yield(Block.java:189)
	at org.jruby.RubyArray.each(RubyArray.java:1981)
	at org.jruby.RubyArray$INVOKER$i$0$0$each.call(RubyArray$INVOKER$i$0$0$each.gen)
	at org.jruby.internal.runtime.methods.JavaMethod$JavaMethodZeroBlock.call(JavaMethod.java:561)
	at org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:90)
	at org.jruby.ir.instructions.CallBase.interpret(CallBase.java:548)
	at org.jruby.ir.interpreter.InterpreterEngine.processCall(InterpreterEngine.java:363)
	at org.jruby.ir.interpreter.StartupInterpreterEngine.interpret(StartupInterpreterEngine.java:66)
	at org.jruby.internal.runtime.methods.MixedModeIRMethod.INTERPRET_METHOD(MixedModeIRMethod.java:128)
	at org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:115)
	at org.jruby.ir.targets.indy.InvokeSite.failf(InvokeSite.java:802)
	at org.jruby.ir.targets.indy.InvokeSite.failf(InvokeSite.java:816)
	at java.lang.invoke.LambdaForm$DMH/0x00007f60a86fd400.invokeVirtual(java.base@21.0.4/LambdaForm$DMH)
	at java.lang.invoke.LambdaForm$MH/0x00007f60a8ab5000.invoke(java.base@21.0.4/LambdaForm$MH)
	at java.lang.invoke.Invokers$Holder.linkToCallSite(java.base@21.0.4/Invokers$Holder)
	at usr.share.logstash.logstash_minus_core.lib.logstash.java_pipeline.RUBY$method$run$0(/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:209)
	at usr.share.logstash.logstash_minus_core.lib.logstash.java_pipeline.RUBY$method$run$0$__VARARGS__(/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:189)
	at java.lang.invoke.DirectMethodHandle$Holder.invokeStatic(java.base@21.0.4/DirectMethodHandle$Holder)
	at java.lang.invoke.LambdaForm$MH/0x00007f60a86e8c00.invokeExact_MT(java.base@21.0.4/LambdaForm$MH)
	at org.jruby.internal.runtime.methods.CompiledIRMethod.call(CompiledIRMethod.java:139)
	at org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:112)
	at org.jruby.ir.targets.indy.InvokeSite.failf(InvokeSite.java:802)
	at org.jruby.ir.targets.indy.InvokeSite.failf(InvokeSite.java:816)
	at java.lang.invoke.LambdaForm$DMH/0x00007f60a86fd400.invokeVirtual(java.base@21.0.4/LambdaForm$DMH)
	at java.lang.invoke.LambdaForm$MH/0x00007f60a8ab5000.invoke(java.base@21.0.4/LambdaForm$MH)
	at java.lang.invoke.Invokers$Holder.linkToCallSite(java.base@21.0.4/Invokers$Holder)
	at usr.share.logstash.logstash_minus_core.lib.logstash.java_pipeline.RUBY$block$start$1(/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:146)
	at java.lang.invoke.DirectMethodHandle$Holder.invokeStatic(java.base@21.0.4/DirectMethodHandle$Holder)
	at java.lang.invoke.LambdaForm$MH/0x00007f60a8718800.invoke(java.base@21.0.4/LambdaForm$MH)
	at java.lang.invoke.LambdaForm$MH/0x00007f60a80b9c00.invokeExact_MT(java.base@21.0.4/LambdaForm$MH)
	at org.jruby.runtime.CompiledIRBlockBody.callDirect(CompiledIRBlockBody.java:141)
	at org.jruby.runtime.IRBlockBody.call(IRBlockBody.java:64)
	at org.jruby.runtime.IRBlockBody.call(IRBlockBody.java:58)
	at org.jruby.runtime.Block.call(Block.java:144)
	at org.jruby.RubyProc.call(RubyProc.java:354)
	at org.jruby.internal.runtime.RubyRunnable.run(RubyRunnable.java:111)
	at java.lang.Thread.runWith(java.base@21.0.4/Thread.java:1596)
	at java.lang.Thread.run(java.base@21.0.4/Thread.java:1583)
"Converge PipelineAction::StopAndDelete<pipeline_4>":
	at java.util.concurrent.ConcurrentHashMap.compute(java.base@21.0.4/ConcurrentHashMap.java:1931)
	- waiting to lock <0x00000006b3233ea0> (a java.util.concurrent.ConcurrentHashMap$Node)
	at org.logstash.plugins.pipeline.PipelineBusV2$AddressStateMapping.mutate(PipelineBusV2.java:137)
	at org.logstash.plugins.pipeline.PipelineBusV2.tryUnlistenOrphan(PipelineBusV2.java:115)
	at org.logstash.plugins.pipeline.PipelineBusV2.unlistenBlocking(PipelineBusV2.java:100)
	- locked <0x0000000695d3fcf0> (a org.jruby.gen.LogStash$$Plugins$$Builtin$$Pipeline$$Input_934266047)
	at org.logstash.plugins.pipeline.PipelineBusV2.unlisten(PipelineBusV2.java:86)
	at java.lang.invoke.LambdaForm$DMH/0x00007f60a8fb2c00.invokeInterface(java.base@21.0.4/LambdaForm$DMH)
	at java.lang.invoke.LambdaForm$MH/0x00007f60a90ac800.invoke(java.base@21.0.4/LambdaForm$MH)
	at java.lang.invoke.LambdaForm$MH/0x00007f60a86ff400.reinvoke(java.base@21.0.4/LambdaForm$MH)
	at java.lang.invoke.LambdaForm$MH/0x00007f60a86ff800.guard(java.base@21.0.4/LambdaForm$MH)
	at java.lang.invoke.LambdaForm$MH/0x00007f60a86ff400.reinvoke(java.base@21.0.4/LambdaForm$MH)
	at java.lang.invoke.LambdaForm$MH/0x00007f60a86ff800.guard(java.base@21.0.4/LambdaForm$MH)
	at java.lang.invoke.Invokers$Holder.linkToCallSite(java.base@21.0.4/Invokers$Holder)
	at usr.share.logstash.logstash_minus_core.lib.logstash.plugins.builtin.pipeline.input.RUBY$method$stop$0(/usr/share/logstash/logstash-core/lib/logstash/plugins/builtin/pipeline/input.rb:77)
	at usr.share.logstash.logstash_minus_core.lib.logstash.plugins.builtin.pipeline.input.RUBY$method$stop$0$__VARARGS__(/usr/share/logstash/logstash-core/lib/logstash/plugins/builtin/pipeline/input.rb:76)
	at java.lang.invoke.DirectMethodHandle$Holder.invokeStatic(java.base@21.0.4/DirectMethodHandle$Holder)
	at java.lang.invoke.LambdaForm$MH/0x00007f60a86e8c00.invokeExact_MT(java.base@21.0.4/LambdaForm$MH)
	at org.jruby.internal.runtime.methods.CompiledIRMethod.call(CompiledIRMethod.java:139)
	at org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:112)
	at org.jruby.ir.targets.indy.InvokeSite.failf(InvokeSite.java:802)
	at org.jruby.ir.targets.indy.InvokeSite.failf(InvokeSite.java:816)
	at java.lang.invoke.LambdaForm$DMH/0x00007f60a86fd400.invokeVirtual(java.base@21.0.4/LambdaForm$DMH)
	at java.lang.invoke.LambdaForm$MH/0x00007f60a8ab5000.invoke(java.base@21.0.4/LambdaForm$MH)
	at java.lang.invoke.Invokers$Holder.linkToCallSite(java.base@21.0.4/Invokers$Holder)
	at usr.share.logstash.logstash_minus_core.lib.logstash.inputs.base.RUBY$method$do_stop$0(/usr/share/logstash/logstash-core/lib/logstash/inputs/base.rb:102)
	at usr.share.logstash.logstash_minus_core.lib.logstash.inputs.base.RUBY$method$do_stop$0$__VARARGS__(/usr/share/logstash/logstash-core/lib/logstash/inputs/base.rb:99)
	at java.lang.invoke.DirectMethodHandle$Holder.invokeStatic(java.base@21.0.4/DirectMethodHandle$Holder)
	at java.lang.invoke.LambdaForm$MH/0x00007f60a86e8c00.invokeExact_MT(java.base@21.0.4/LambdaForm$MH)
	at org.jruby.internal.runtime.methods.CompiledIRMethod.call(CompiledIRMethod.java:139)
	at org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:112)
	at org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:446)
	at org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:92)
	at org.jruby.RubySymbol$SymbolProcBody.yieldInner(RubySymbol.java:1513)
	at org.jruby.RubySymbol$SymbolProcBody.doYield(RubySymbol.java:1528)
	at org.jruby.runtime.BlockBody.yield(BlockBody.java:108)
	at org.jruby.runtime.Block.yield(Block.java:189)
	at org.jruby.RubyArray.each(RubyArray.java:1981)
	at java.lang.invoke.LambdaForm$DMH/0x00007f60a86f4c00.invokeVirtual(java.base@21.0.4/LambdaForm$DMH)
	at java.lang.invoke.LambdaForm$MH/0x00007f60a8708800.invoke(java.base@21.0.4/LambdaForm$MH)
	at java.lang.invoke.LambdaForm$MH/0x00007f60a86fbc00.reinvoke(java.base@21.0.4/LambdaForm$MH)
	at java.lang.invoke.LambdaForm$MH/0x00007f60a86fc000.guard(java.base@21.0.4/LambdaForm$MH)
	at java.lang.invoke.LambdaForm$MH/0x00007f60a86fbc00.reinvoke(java.base@21.0.4/LambdaForm$MH)
	at java.lang.invoke.LambdaForm$MH/0x00007f60a86fc000.guard(java.base@21.0.4/LambdaForm$MH)
	at java.lang.invoke.Invokers$Holder.linkToCallSite(java.base@21.0.4/Invokers$Holder)
	at usr.share.logstash.logstash_minus_core.lib.logstash.java_pipeline.RUBY$method$stop_inputs$0(/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:468)
	at usr.share.logstash.logstash_minus_core.lib.logstash.java_pipeline.RUBY$method$stop_inputs$0$__VARARGS__(/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:466)
	at java.lang.invoke.DirectMethodHandle$Holder.invokeStatic(java.base@21.0.4/DirectMethodHandle$Holder)
	at java.lang.invoke.LambdaForm$MH/0x00007f60a86e8c00.invokeExact_MT(java.base@21.0.4/LambdaForm$MH)
	at org.jruby.internal.runtime.methods.CompiledIRMethod.call(CompiledIRMethod.java:139)
	at org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:112)
	at org.jruby.ir.targets.indy.InvokeSite.performIndirectCall(InvokeSite.java:735)
	at org.jruby.ir.targets.indy.InvokeSite.invoke(InvokeSite.java:680)
	at java.lang.invoke.LambdaForm$DMH/0x00007f60a86de800.invokeVirtual(java.base@21.0.4/LambdaForm$DMH)
	at java.lang.invoke.LambdaForm$MH/0x00007f60a870b000.invoke(java.base@21.0.4/LambdaForm$MH)
	at java.lang.invoke.LambdaForm$MH/0x00007f60a86f5c00.reinvoke(java.base@21.0.4/LambdaForm$MH)
	at java.lang.invoke.LambdaForm$MH/0x00007f60a86f6000.guard(java.base@21.0.4/LambdaForm$MH)
	at java.lang.invoke.LambdaForm$MH/0x00007f60a86f5c00.reinvoke(java.base@21.0.4/LambdaForm$MH)
	at java.lang.invoke.LambdaForm$MH/0x00007f60a86f6000.guard(java.base@21.0.4/LambdaForm$MH)
	at java.lang.invoke.LambdaForm$MH/0x00007f60a86f5c00.reinvoke(java.base@21.0.4/LambdaForm$MH)
	at java.lang.invoke.LambdaForm$MH/0x00007f60a86f6000.guard(java.base@21.0.4/LambdaForm$MH)
	at java.lang.invoke.LambdaForm$MH/0x00007f60a86f5c00.reinvoke(java.base@21.0.4/LambdaForm$MH)
	at java.lang.invoke.LambdaForm$MH/0x00007f60a86f6000.guard(java.base@21.0.4/LambdaForm$MH)
	at java.lang.invoke.LambdaForm$MH/0x00007f60a86f5c00.reinvoke(java.base@21.0.4/LambdaForm$MH)
	at java.lang.invoke.LambdaForm$MH/0x00007f60a86f6000.guard(java.base@21.0.4/LambdaForm$MH)
	at java.lang.invoke.LambdaForm$MH/0x00007f60a86f5c00.reinvoke(java.base@21.0.4/LambdaForm$MH)
	at java.lang.invoke.LambdaForm$MH/0x00007f60a86f6000.guard(java.base@21.0.4/LambdaForm$MH)
	at java.lang.invoke.LambdaForm$MH/0x00007f60a86f5c00.reinvoke(java.base@21.0.4/LambdaForm$MH)
	at java.lang.invoke.LambdaForm$MH/0x00007f60a86f6000.guard(java.base@21.0.4/LambdaForm$MH)
	at java.lang.invoke.LambdaForm$MH/0x00007f60a86f5c00.reinvoke(java.base@21.0.4/LambdaForm$MH)
	at java.lang.invoke.LambdaForm$MH/0x00007f60a86f6000.guard(java.base@21.0.4/LambdaForm$MH)
	at java.lang.invoke.LambdaForm$MH/0x00007f60a86f5c00.reinvoke(java.base@21.0.4/LambdaForm$MH)
	at java.lang.invoke.LambdaForm$MH/0x00007f60a86f6000.guard(java.base@21.0.4/LambdaForm$MH)
	at java.lang.invoke.LambdaForm$MH/0x00007f60a86f5c00.reinvoke(java.base@21.0.4/LambdaForm$MH)
	at java.lang.invoke.LambdaForm$MH/0x00007f60a86f6000.guard(java.base@21.0.4/LambdaForm$MH)
	at java.lang.invoke.Invokers$Holder.linkToCallSite(java.base@21.0.4/Invokers$Holder)
	at usr.share.logstash.logstash_minus_core.lib.logstash.java_pipeline.RUBY$method$shutdown$0(/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:456)
	at usr.share.logstash.logstash_minus_core.lib.logstash.java_pipeline.RUBY$method$shutdown$0$__VARARGS__(/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:448)
	at java.lang.invoke.DirectMethodHandle$Holder.invokeStatic(java.base@21.0.4/DirectMethodHandle$Holder)
	at java.lang.invoke.LambdaForm$MH/0x00007f60a86e8c00.invokeExact_MT(java.base@21.0.4/LambdaForm$MH)
	at org.jruby.internal.runtime.methods.CompiledIRMethod.call(CompiledIRMethod.java:139)
	at org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:112)
	at org.jruby.ir.targets.indy.InvokeSite.performIndirectCall(InvokeSite.java:735)
	at org.jruby.ir.targets.indy.InvokeSite.invoke(InvokeSite.java:657)
	at java.lang.invoke.LambdaForm$DMH/0x00007f60a86e3800.invokeVirtual(java.base@21.0.4/LambdaForm$DMH)
	at java.lang.invoke.LambdaForm$MH/0x00007f60a8721800.invoke(java.base@21.0.4/LambdaForm$MH)
	at java.lang.invoke.LambdaForm$MH/0x00007f60a86e2c00.reinvoke(java.base@21.0.4/LambdaForm$MH)
	at java.lang.invoke.LambdaForm$MH/0x00007f60a86e3000.guard(java.base@21.0.4/LambdaForm$MH)
	at java.lang.invoke.LambdaForm$MH/0x00007f60a86e2c00.reinvoke(java.base@21.0.4/LambdaForm$MH)
	at java.lang.invoke.LambdaForm$MH/0x00007f60a86e3000.guard(java.base@21.0.4/LambdaForm$MH)
	at java.lang.invoke.LambdaForm$MH/0x00007f60a86e2c00.reinvoke(java.base@21.0.4/LambdaForm$MH)
	at java.lang.invoke.LambdaForm$MH/0x00007f60a86e3000.guard(java.base@21.0.4/LambdaForm$MH)
	at java.lang.invoke.Invokers$Holder.linkToCallSite(java.base@21.0.4/Invokers$Holder)
	at usr.share.logstash.logstash_minus_core.lib.logstash.pipeline_action.stop_and_delete.RUBY$block$execute$1(/usr/share/logstash/logstash-core/lib/logstash/pipeline_action/stop_and_delete.rb:30)
	at java.lang.invoke.DirectMethodHandle$Holder.invokeStatic(java.base@21.0.4/DirectMethodHandle$Holder)
	at java.lang.invoke.LambdaForm$MH/0x00007f60a8786800.invoke(java.base@21.0.4/LambdaForm$MH)
	at java.lang.invoke.LambdaForm$MH/0x00007f60a86e2c00.reinvoke(java.base@21.0.4/LambdaForm$MH)
	at java.lang.invoke.LambdaForm$MH/0x00007f60a86e3000.guard(java.base@21.0.4/LambdaForm$MH)
	at java.lang.invoke.Invokers$Holder.linkToCallSite(java.base@21.0.4/Invokers$Holder)
	at usr.share.logstash.logstash_minus_core.lib.logstash.pipelines_registry.RUBY$method$terminate_pipeline$0(/usr/share/logstash/logstash-core/lib/logstash/pipelines_registry.rb:192)
	at usr.share.logstash.logstash_minus_core.lib.logstash.pipelines_registry.RUBY$method$terminate_pipeline$0$__VARARGS__(/usr/share/logstash/logstash-core/lib/logstash/pipelines_registry.rb:184)
	at java.lang.invoke.DirectMethodHandle$Holder.invokeStatic(java.base@21.0.4/DirectMethodHandle$Holder)
	at java.lang.invoke.LambdaForm$MH/0x00007f60a86e8c00.invokeExact_MT(java.base@21.0.4/LambdaForm$MH)
	at org.jruby.internal.runtime.methods.CompiledIRMethod.call(CompiledIRMethod.java:139)
	at org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:112)
	at org.jruby.ir.targets.indy.InvokeSite.performIndirectCall(InvokeSite.java:725)
	at org.jruby.ir.targets.indy.InvokeSite.invoke(InvokeSite.java:657)
	at java.lang.invoke.LambdaForm$DMH/0x00007f60a86e3800.invokeVirtual(java.base@21.0.4/LambdaForm$DMH)
	at java.lang.invoke.LambdaForm$MH/0x00007f60a875d800.invoke(java.base@21.0.4/LambdaForm$MH)
	at java.lang.invoke.LambdaForm$MH/0x00007f60a86ff400.reinvoke(java.base@21.0.4/LambdaForm$MH)
	at java.lang.invoke.LambdaForm$MH/0x00007f60a86ff800.guard(java.base@21.0.4/LambdaForm$MH)
	at java.lang.invoke.Invokers$Holder.linkToCallSite(java.base@21.0.4/Invokers$Holder)
	at usr.share.logstash.logstash_minus_core.lib.logstash.pipeline_action.stop_and_delete.RUBY$method$execute$0(/usr/share/logstash/logstash-core/lib/logstash/pipeline_action/stop_and_delete.rb:29)
	at java.lang.invoke.DirectMethodHandle$Holder.invokeStatic(java.base@21.0.4/DirectMethodHandle$Holder)
	at java.lang.invoke.LambdaForm$MH/0x00007f60a884ac00.invoke(java.base@21.0.4/LambdaForm$MH)
	at java.lang.invoke.LambdaForm$MH/0x00007f60a86ff400.reinvoke(java.base@21.0.4/LambdaForm$MH)
	at java.lang.invoke.LambdaForm$MH/0x00007f60a86ff800.guard(java.base@21.0.4/LambdaForm$MH)
	at java.lang.invoke.LambdaForm$MH/0x00007f60a86ff400.reinvoke(java.base@21.0.4/LambdaForm$MH)
	at java.lang.invoke.LambdaForm$MH/0x00007f60a86ff800.guard(java.base@21.0.4/LambdaForm$MH)
	at java.lang.invoke.Invokers$Holder.linkToCallSite(java.base@21.0.4/Invokers$Holder)
	at usr.share.logstash.logstash_minus_core.lib.logstash.agent.RUBY$block$converge_state$1(/usr/share/logstash/logstash-core/lib/logstash/agent.rb:386)
	at java.lang.invoke.DirectMethodHandle$Holder.invokeStatic(java.base@21.0.4/DirectMethodHandle$Holder)
	at java.lang.invoke.LambdaForm$MH/0x00007f60a8718800.invoke(java.base@21.0.4/LambdaForm$MH)
	at java.lang.invoke.LambdaForm$MH/0x00007f60a80b9c00.invokeExact_MT(java.base@21.0.4/LambdaForm$MH)
	at org.jruby.runtime.CompiledIRBlockBody.callDirect(CompiledIRBlockBody.java:141)
	at org.jruby.runtime.IRBlockBody.call(IRBlockBody.java:64)
	at org.jruby.runtime.IRBlockBody.call(IRBlockBody.java:58)
	at org.jruby.runtime.Block.call(Block.java:144)
	at org.jruby.RubyProc.call(RubyProc.java:354)
	at org.jruby.internal.runtime.RubyRunnable.run(RubyRunnable.java:111)
	at java.lang.Thread.runWith(java.base@21.0.4/Thread.java:1596)
	at java.lang.Thread.run(java.base@21.0.4/Thread.java:1583)

Found 1 deadlock.

Workaround

PipelineBusV1 is not subject to this issue, and can be selected by adding the following to config/jvm.options:

# Use PipelineBusV1 to avoid possibility of deadlock during shutdown
# See https://github.com/elastic/logstash/issues/16657
-Dlogstash.pipelinebus.implementation=v1
@kaisecheng
Copy link
Contributor Author

kaisecheng commented Nov 8, 2024 •
edited by andsel
Loading

StopAndDelete try to lock A and then lock B
pipeline_3 try to lock B and then lock A

A is PipelineInput
B is AddressStateMapping.mapping

yaauie added a commit that referenced this issue Nov 8, 2024
@yaauie
Add Known Issue for 8.15.x releases and PipelineBusV2
921e603 
#16657
@yaauie yaauie mentioned this issue Nov 8, 2024
Merged
yaauie added a commit that referenced this issue Nov 11, 2024
@yaauie @karenzone
Add known issue for PipelineBusv2 to 8.15.x release notes (#16661)
e17ad0c 
* Add Known Issue for 8.15.x releases and PipelineBusV2

#16657

* Sync 8.15.4 with others

* Apply suggestions from code review

Co-authored-by: Karen Metts <35154725+karenzone@users.noreply.github.com>

* Propagate suggestions to other entries

* Apply suggestions from code review

formatting:
- plus-to-continue only in lists,
- consistency of linebreaks before next section

Co-authored-by: Karen Metts <35154725+karenzone@users.noreply.github.com>

---------

Co-authored-by: Karen Metts <35154725+karenzone@users.noreply.github.com>
@yaauie yaauie mentioned this issue Nov 12, 2024
Merged
3 tasks
This was referenced Nov 18, 2024
Open
Open
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@yaauie yaauie

Labels
bug int-shortlist status:work-in-progress
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Forwardport PR #16671 to main: PipelineBusV2 deadlock proofing elastic/logstash
3 participants
@yaauie @robbavey @kaisecheng