there are 150+ services ,skywalkling oap query request is slow and somtimes timout while the storage choose es #12739

lpq131004 · 2024-11-01T08:56:06Z

lpq131004
Nov 1, 2024

i have change some params as the flowing picture shows,but it dosen't works.Queries are still slow and always report response time out error :

some skywalking oap error log

2024-11-01 08:37:02,379 org.apache.skywalking.library.elasticsearch.bulk.BulkProcessor 87 [ElasticSearch BulkProcessor] ERROR [] - [9.7.0-138715f] flush data to ES failure:

java.util.concurrent.CompletionException: com.linecorp.armeria.client.ResponseTimeoutException

at java.util.concurrent.CompletableFuture.encodeThrowable(Unknown Source) ~[?:?]

at java.util.concurrent.CompletableFuture.completeThrowable(Unknown Source) ~[?:?]

at java.util.concurrent.CompletableFuture$UniAccept.tryFire(Unknown Source) ~[?:?]

at java.util.concurrent.CompletableFuture.postComplete(Unknown Source) ~[?:?]

at java.util.concurrent.CompletableFuture.completeExceptionally(Unknown Source) ~[?:?]

at com.linecorp.armeria.common.stream.AggregationSupport.lambda$aggregate$2(AggregationSupport.java:130) ~[armeria-1.25.0.jar:?]

at java.util.concurrent.CompletableFuture.uniHandle(Unknown Source) ~[?:?]

at java.util.concurrent.CompletableFuture$UniHandle.tryFire(Unknown Source) ~[?:?]

at java.util.concurrent.CompletableFuture.postComplete(Unknown Source) ~[?:?]

at java.util.concurrent.CompletableFuture.completeExceptionally(Unknown Source) ~[?:?]

at com.linecorp.armeria.common.stream.DeferredStreamMessage.lambda$delegate$1(DeferredStreamMessage.java:179) ~[armeria-1.25.0.jar:?]

at java.util.concurrent.CompletableFuture.uniHandle(Unknown Source) ~[?:?]

at java.util.concurrent.CompletableFuture.uniHandleStage(Unknown Source) ~[?:?]

at java.util.concurrent.CompletableFuture.handle(Unknown Source) ~[?:?]

at com.linecorp.armeria.common.stream.DeferredStreamMessage.delegate(DeferredStreamMessage.java:175) ~[armeria-1.25.0.jar:?]

at com.linecorp.armeria.common.stream.DeferredStreamMessage.lambda$delegateOnCompletion$0(DeferredStreamMessage.java:148) ~[armeria-1.25.0.jar:?]

at java.util.concurrent.CompletableFuture.uniHandle(Unknown Source) ~[?:?]

at java.util.concurrent.CompletableFuture$UniHandle.tryFire(Unknown Source) ~[?:?]

at java.util.concurrent.CompletableFuture.postComplete(Unknown Source) ~[?:?]

at java.util.concurrent.CompletableFuture.complete(Unknown Source) ~[?:?]

at com.linecorp.armeria.client.retry.RetryingClient.handleRetryDecision(RetryingClient.java:469) ~[armeria-1.25.0.jar:?]

at com.linecorp.armeria.client.retry.RetryingClient.lambda$handleResponseWithoutContent$6(RetryingClient.java:358) ~[armeria-1.25.0.jar:?]

at java.util.concurrent.CompletableFuture.uniHandle(Unknown Source) ~[?:?]

at java.util.concurrent.CompletableFuture.uniHandleStage(Unknown Source) ~[?:?]

at java.util.concurrent.CompletableFuture.handle(Unknown Source) ~[?:?]

at java.util.concurrent.CompletableFuture.handle(Unknown Source) ~[?:?]

at com.linecorp.armeria.client.retry.RetryingClient.handleResponseWithoutContent(RetryingClient.java:356) ~[armeria-1.25.0.jar:?]

at com.linecorp.armeria.client.retry.RetryingClient.lambda$handleResponse$9(RetryingClient.java:426) ~[armeria-1.25.0.jar:?]

at java.util.concurrent.CompletableFuture$UniAccept.tryFire(Unknown Source) ~[?:?]

at java.util.concurrent.CompletableFuture.postComplete(Unknown Source) ~[?:?]

at java.util.concurrent.CompletableFuture.complete(Unknown Source) ~[?:?]

at com.linecorp.armeria.common.logging.DefaultRequestLog$RequestLogFuture.completeLog(DefaultRequestLog.java:1479) ~[armeria-1.25.0.jar:?]

at com.linecorp.armeria.common.logging.DefaultRequestLog.completeSatisfiedFutures(DefaultRequestLog.java:431) ~[armeria-1.25.0.jar:?]

at com.linecorp.armeria.common.logging.DefaultRequestLog.updateFlags(DefaultRequestLog.java:419) ~[armeria-1.25.0.jar:?]

at com.linecorp.armeria.common.logging.DefaultRequestLog.endResponse0(DefaultRequestLog.java:1412) ~[armeria-1.25.0.jar:?]

at com.linecorp.armeria.common.logging.DefaultRequestLog.endResponse0(DefaultRequestLog.java:1381) ~[armeria-1.25.0.jar:?]

at com.linecorp.armeria.common.logging.DefaultRequestLog.endResponse(DefaultRequestLog.java:1367) ~[armeria-1.25.0.jar:?]

at com.linecorp.armeria.client.HttpResponseWrapper$1.run(HttpResponseWrapper.java:311) ~[armeria-1.25.0.jar:?]

at com.linecorp.armeria.internal.common.CancellationScheduler.invokeTask(CancellationScheduler.java:484) ~[armeria-1.25.0.jar:?]

at com.linecorp.armeria.internal.common.CancellationScheduler.lambda$init0$2(CancellationScheduler.java:123) ~[armeria-1.25.0.jar:?]

at com.linecorp.armeria.common.DefaultContextAwareRunnable.run(DefaultContextAwareRunnable.java:45) ~[armeria-1.25.0.jar:?]

at io.netty.util.concurrent.PromiseTask.runTask(PromiseTask.java:98) ~[netty-common-4.1.100.Final.jar:4.1.100.Final]

at io.netty.util.concurrent.ScheduledFutureTask.run(ScheduledFutureTask.java:153) ~[netty-common-4.1.100.Final.jar:4.1.100.Final]

at io.netty.util.concurrent.AbstractEventExecutor.runTask(AbstractEventExecutor.java:173) ~[netty-common-4.1.100.Final.jar:4.1.100.Final]

at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:166) ~[netty-common-4.1.100.Final.jar:4.1.100.Final]

at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:470) ~[netty-common-4.1.100.Final.jar:4.1.100.Final]

at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:416) ~[netty-transport-classes-epoll-4.1.100.Final.jar:4.1.100.Final]

at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) ~[netty-common-4.1.100.Final.jar:4.1.100.Final]

at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[netty-common-4.1.100.Final.jar:4.1.100.Final]

at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ~[netty-common-4.1.100.Final.jar:4.1.100.Final]

at java.lang.Thread.run(Unknown Source) [?:?]

Caused by: com.linecorp.armeria.client.ResponseTimeoutException

at com.linecorp.armeria.client.ResponseTimeoutException.get(ResponseTimeoutException.java:38) ~[armeria-1.25.0.jar:?]

at com.linecorp.armeria.internal.common.CancellationScheduler.invokeTask(CancellationScheduler.java:470) ~[armeria-1.25.0.jar:?]

... 12 more

2024-11-01 08:37:09,720 org.apache.skywalking.oap.server.core.storage.PersistenceTimer 151 [pool-7-thread-2] ERROR [] - [9.7.0-138715f] com.linecorp.armeria.client.ResponseTimeoutException

java.util.concurrent.CompletionException: com.linecorp.armeria.client.ResponseTimeoutException

at java.util.concurrent.CompletableFuture.encodeThrowable(Unknown Source) ~[?:?]

at java.util.concurrent.CompletableFuture.completeThrowable(Unknown Source) ~[?:?]

at java.util.concurrent.CompletableFuture$UniAccept.tryFire(Unknown Source) ~[?:?]

at java.util.concurrent.CompletableFuture.postComplete(Unknown Source) ~[?:?]

at java.util.concurrent.CompletableFuture.completeExceptionally(Unknown Source) ~[?:?]

at com.linecorp.armeria.common.stream.AggregationSupport.lambda$aggregate$2(AggregationSupport.java:130) ~[armeria-1.25.0.jar:?]

at java.util.concurrent.CompletableFuture.uniHandle(Unknown Source) ~[?:?]

at java.util.concurrent.CompletableFuture$UniHandle.tryFire(Unknown Source) ~[?:?]

at java.util.concurrent.CompletableFuture.postComplete(Unknown Source) ~[?:?]

at java.util.concurrent.CompletableFuture.completeExceptionally(Unknown Source) ~[?:?]

at com.linecorp.armeria.common.stream.DeferredStreamMessage.lambda$delegate$1(DeferredStreamMessage.java:179) ~[armeria-1.25.0.jar:?]

at java.util.concurrent.CompletableFuture.uniHandle(Unknown Source) ~[?:?]

at java.util.concurrent.CompletableFuture.uniHandleStage(Unknown Source) ~[?:?]

at java.util.concurrent.CompletableFuture.handle(Unknown Source) ~[?:?]

at com.linecorp.armeria.common.stream.DeferredStreamMessage.delegate(DeferredStreamMessage.java:175) ~[armeria-1.25.0.jar:?]

at com.linecorp.armeria.common.stream.DeferredStreamMessage.lambda$delegateOnCompletion$0(DeferredStreamMessage.java:148) ~[armeria-1.25.0.jar:?]

at java.util.concurrent.CompletableFuture.uniHandle(Unknown Source) ~[?:?]

at java.util.concurrent.CompletableFuture$UniHandle.tryFire(Unknown Source) ~[?:?]

at java.util.concurrent.CompletableFuture.postComplete(Unknown Source) ~[?:?]

at java.util.concurrent.CompletableFuture.complete(Unknown Source) ~[?:?]

at com.linecorp.armeria.client.retry.RetryingClient.handleRetryDecision(RetryingClient.java:469) ~[armeria-1.25.0.jar:?]

at com.linecorp.armeria.client.retry.RetryingClient.lambda$handleResponseWithoutContent$6(RetryingClient.java:358) ~[armeria-1.25.0.jar:?]

at java.util.concurrent.CompletableFuture.uniHandle(Unknown Source) ~[?:?]

at java.util.concurrent.CompletableFuture.uniHandleStage(Unknown Source) ~[?:?]

at java.util.concurrent.CompletableFuture.handle(Unknown Source) ~[?:?]

at java.util.concurrent.CompletableFuture.handle(Unknown Source) ~[?:?]

at com.linecorp.armeria.client.retry.RetryingClient.handleResponseWithoutContent(RetryingClient.java:356) ~[armeria-1.25.0.jar:?]

at com.linecorp.armeria.client.retry.RetryingClient.lambda$handleResponse$9(RetryingClient.java:426) ~[armeria-1.25.0.jar:?]

at java.util.concurrent.CompletableFuture$UniAccept.tryFire(Unknown Source) ~[?:?]

at java.util.concurrent.CompletableFuture.postComplete(Unknown Source) ~[?:?]

at java.util.concurrent.CompletableFuture.complete(Unknown Source) ~[?:?]

at com.linecorp.armeria.common.logging.DefaultRequestLog$RequestLogFuture.completeLog(DefaultRequestLog.java:1479) ~[armeria-1.25.0.jar:?]

at com.linecorp.armeria.common.logging.DefaultRequestLog.completeSatisfiedFutures(DefaultRequestLog.java:431) ~[armeria-1.25.0.jar:?]

at com.linecorp.armeria.common.logging.DefaultRequestLog.updateFlags(DefaultRequestLog.java:419) ~[armeria-1.25.0.jar:?]

at com.linecorp.armeria.common.logging.DefaultRequestLog.endResponse0(DefaultRequestLog.java:1412) ~[armeria-1.25.0.jar:?]

at com.linecorp.armeria.common.logging.DefaultRequestLog.endResponse0(DefaultRequestLog.java:1381) ~[armeria-1.25.0.jar:?]

at com.linecorp.armeria.common.logging.DefaultRequestLog.endResponse(DefaultRequestLog.java:1367) ~[armeria-1.25.0.jar:?]

at com.linecorp.armeria.client.HttpResponseWrapper$1.run(HttpResponseWrapper.java:311) ~[armeria-1.25.0.jar:?]

at com.linecorp.armeria.internal.common.CancellationScheduler.invokeTask(CancellationScheduler.java:484) ~[armeria-1.25.0.jar:?]

at com.linecorp.armeria.internal.common.CancellationScheduler.lambda$init0$2(CancellationScheduler.java:123) ~[armeria-1.25.0.jar:?]

at com.linecorp.armeria.common.DefaultContextAwareRunnable.run(DefaultContextAwareRunnable.java:45) ~[armeria-1.25.0.jar:?]

at io.netty.util.concurrent.PromiseTask.runTask(PromiseTask.java:98) ~[netty-common-4.1.100.Final.jar:4.1.100.Final]

at io.netty.util.concurrent.ScheduledFutureTask.run(ScheduledFutureTask.java:153) ~[netty-common-4.1.100.Final.jar:4.1.100.Final]

at io.netty.util.concurrent.AbstractEventExecutor.runTask(AbstractEventExecutor.java:173) ~[netty-common-4.1.100.Final.jar:4.1.100.Final]

at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:166) ~[netty-common-4.1.100.Final.jar:4.1.100.Final]

at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:470) ~[netty-common-4.1.100.Final.jar:4.1.100.Final]

at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:416) ~[netty-transport-classes-epoll-4.1.100.Final.jar:4.1.100.Final]

at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) ~[netty-common-4.1.100.Final.jar:4.1.100.Final]

at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[netty-common-4.1.100.Final.jar:4.1.100.Final]

at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ~[netty-common-4.1.100.Final.jar:4.1.100.Final]

at java.lang.Thread.run(Unknown Source) [?:?]

Caused by: com.linecorp.armeria.client.ResponseTimeoutException

at com.linecorp.armeria.client.ResponseTimeoutException.get(ResponseTimeoutException.java:38) ~[armeria-1.25.0.jar:?]

at com.linecorp.armeria.internal.common.CancellationScheduler.invokeTask(CancellationScheduler.java:470) ~[armeria-1.25.0.jar:?]

... 12 more

2024-11-01 08:37:09,721 org.apache.skywalking.oap.server.core.storage.PersistenceTimer 91 [pool-8-thread-1] ERROR [] - [9.7.0-138715f] Extract data and save failure.

java.util.concurrent.CompletionException: com.linecorp.armeria.client.ResponseTimeoutException

at java.util.concurrent.CompletableFuture.encodeThrowable(Unknown Source) ~[?:?]

at java.util.concurrent.CompletableFuture.completeThrowable(Unknown Source) ~[?:?]

at java.util.concurrent.CompletableFuture$UniAccept.tryFire(Unknown Source) ~[?:?]

at java.util.concurrent.CompletableFuture.postComplete(Unknown Source) ~[?:?]

at java.util.concurrent.CompletableFuture.completeExceptionally(Unknown Source) ~[?:?]

at com.linecorp.armeria.common.stream.AggregationSupport.lambda$aggregate$2(AggregationSupport.java:130) ~[armeria-1.25.0.jar:?]

at java.util.concurrent.CompletableFuture.uniHandle(Unknown Source) ~[?:?]

at java.util.concurrent.CompletableFuture$UniHandle.tryFire(Unknown Source) ~[?:?]

at java.util.concurrent.CompletableFuture.postComplete(Unknown Source) ~[?:?]

at java.util.concurrent.CompletableFuture.completeExceptionally(Unknown Source) ~[?:?]

at com.linecorp.armeria.common.stream.DeferredStreamMessage.lambda$delegate$1(DeferredStreamMessage.java:179) ~[armeria-1.25.0.jar:?]

at java.util.concurrent.CompletableFuture.uniHandle(Unknown Source) ~[?:?]

at java.util.concurrent.CompletableFuture.uniHandleStage(Unknown Source) ~[?:?]

at java.util.concurrent.CompletableFuture.handle(Unknown Source) ~[?:?]

at com.linecorp.armeria.common.stream.DeferredStreamMessage.delegate(DeferredStreamMessage.java:175) ~[armeria-1.25.0.jar:?]

at com.linecorp.armeria.common.stream.DeferredStreamMessage.lambda$delegateOnCompletion$0(DeferredStreamMessage.java:148) ~[armeria-1.25.0.jar:?]

at java.util.concurrent.CompletableFuture.uniHandle(Unknown Source) ~[?:?]

at java.util.concurrent.CompletableFuture$UniHandle.tryFire(Unknown Source) ~[?:?]

at java.util.concurrent.CompletableFuture.postComplete(Unknown Source) ~[?:?]

at java.util.concurrent.CompletableFuture.complete(Unknown Source) ~[?:?]

at com.linecorp.armeria.client.retry.RetryingClient.handleRetryDecision(RetryingClient.java:469) ~[armeria-1.25.0.jar:?]

at com.linecorp.armeria.client.retry.RetryingClient.lambda$handleResponseWithoutContent$6(RetryingClient.java:358) ~[armeria-1.25.0.jar:?]

at java.util.concurrent.CompletableFuture.uniHandle(Unknown Source) ~[?:?]

at java.util.concurrent.CompletableFuture.uniHandleStage(Unknown Source) ~[?:?]

at java.util.concurrent.CompletableFuture.handle(Unknown Source) ~[?:?]

at java.util.concurrent.CompletableFuture.handle(Unknown Source) ~[?:?]

at com.linecorp.armeria.client.retry.RetryingClient.handleResponseWithoutContent(RetryingClient.java:356) ~[armeria-1.25.0.jar:?]

at com.linecorp.armeria.client.retry.RetryingClient.lambda$handleResponse$9(RetryingClient.java:426) ~[armeria-1.25.0.jar:?]

at java.util.concurrent.CompletableFuture$UniAccept.tryFire(Unknown Source) ~[?:?]

at java.util.concurrent.CompletableFuture.postComplete(Unknown Source) ~[?:?]

at java.util.concurrent.CompletableFuture.complete(Unknown Source) ~[?:?]

at com.linecorp.armeria.common.logging.DefaultRequestLog$RequestLogFuture.completeLog(DefaultRequestLog.java:1479) ~[armeria-1.25.0.jar:?]

at com.linecorp.armeria.common.logging.DefaultRequestLog.completeSatisfiedFutures(DefaultRequestLog.java:431) ~[armeria-1.25.0.jar:?]

at com.linecorp.armeria.common.logging.DefaultRequestLog.updateFlags(DefaultRequestLog.java:419) ~[armeria-1.25.0.jar:?]

at com.linecorp.armeria.common.logging.DefaultRequestLog.endResponse0(DefaultRequestLog.java:1412) ~[armeria-1.25.0.jar:?]

at com.linecorp.armeria.common.logging.DefaultRequestLog.endResponse0(DefaultRequestLog.java:1381) ~[armeria-1.25.0.jar:?]

at com.linecorp.armeria.common.logging.DefaultRequestLog.endResponse(DefaultRequestLog.java:1367) ~[armeria-1.25.0.jar:?]

at com.linecorp.armeria.client.HttpResponseWrapper$1.run(HttpResponseWrapper.java:311) ~[armeria-1.25.0.jar:?]

at com.linecorp.armeria.internal.common.CancellationScheduler.invokeTask(CancellationScheduler.java:484) ~[armeria-1.25.0.jar:?]

at com.linecorp.armeria.internal.common.CancellationScheduler.lambda$init0$2(CancellationScheduler.java:123) ~[armeria-1.25.0.jar:?]

at com.linecorp.armeria.common.DefaultContextAwareRunnable.run(DefaultContextAwareRunnable.java:45) ~[armeria-1.25.0.jar:?]

at io.netty.util.concurrent.PromiseTask.runTask(PromiseTask.java:98) ~[netty-common-4.1.100.Final.jar:4.1.100.Final]

at io.netty.util.concurrent.ScheduledFutureTask.run(ScheduledFutureTask.java:153) ~[netty-common-4.1.100.Final.jar:4.1.100.Final]

at io.netty.util.concurrent.AbstractEventExecutor.runTask(AbstractEventExecutor.java:173) ~[netty-common-4.1.100.Final.jar:4.1.100.Final]

at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:166) ~[netty-common-4.1.100.Final.jar:4.1.100.Final]

at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:470) ~[netty-common-4.1.100.Final.jar:4.1.100.Final]

at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:416) ~[netty-transport-classes-epoll-4.1.100.Final.jar:4.1.100.Final]

at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) ~[netty-common-4.1.100.Final.jar:4.1.100.Final]

at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[netty-common-4.1.100.Final.jar:4.1.100.Final]

at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ~[netty-common-4.1.100.Final.jar:4.1.100.Final]

at java.lang.Thread.run(Unknown Source) [?:?]

Caused by: com.linecorp.armeria.client.ResponseTimeoutException

at com.linecorp.armeria.client.ResponseTimeoutException.get(ResponseTimeoutException.java:38) ~[armeria-1.25.0.jar:?]

at com.linecorp.armeria.internal.common.CancellationScheduler.invokeTask(CancellationScheduler.java:470) ~[armeria-1.25.0.jar:?]

Answered by wu-sheng

Nov 1, 2024

We would not update that, because that is storage level setup. You should remove all indices and templates.

View full answer

wu-sheng · 2024-11-01T12:21:22Z

wu-sheng
Nov 1, 2024
Collaborator

com.linecorp.armeria.client.ResponseTimeoutException

This is a sever side timeout, which usually means your elasticsearch can't support that load.

BTW, SW_STORAGE_ES_CONCURRENT_REQUESTS=50 doesn't make sense. We only set persistent timer thread number as 2

    # The number of threads used to prepare metrics data to the storage.
    prepareThreads: ${SW_CORE_PREPARE_THREADS:2}

Too many threads only makes things worse.

14 replies

wu-sheng Nov 1, 2024
Collaborator

Are you setting a very long TTL and query metrics through a long time range? I noticed you have 43 indices, but we should only have 4 per day.

wu-sheng Nov 1, 2024
Collaborator

The query performance relies on many things. Such as, CPU and memory of elasticsearch datanode, the number of indices of your query, and number of docs in each index.

And you also have 45 shards? How is that possible?

Are you using a very old OAP version?

lpq131004 Nov 1, 2024
Author

and i try to change certerain index shards like the follow picture,but after i restart oap and es ,the new index is still using the default setting

wu-sheng Nov 1, 2024
Collaborator

We would not update that, because that is storage level setup. You should remove all indices and templates.

Answer selected by lpq131004

lpq131004 Nov 1, 2024
Author

The query performance relies on many things. Such as, CPU and memory of elasticsearch datanode, the number of indices of your query, and number of docs in each index.

And you also have 45 shards? How is that possible?

Are you using a very old OAP version?

we are using 9.7.0

wu-sheng Nov 1, 2024
Collaborator

We never provided 45 shards config. Then, it is on your side change.

lpq131004 Nov 1, 2024
Author

we set es save 7 days data, everday will create an indice like metrics-all-YYMMDD,and i don't know why there are some demo indices

wu-sheng Nov 1, 2024
Collaborator

I also saw metricbeat has 6.2m docs, this is a huge index. 5.9g, and you can see others sw_* is not that large.

Also, I can see you have something sw_records-all-xxxx has replicas, which means you double the data.

Your elasticsearch server seems to include many different things from different skywalking. Can't tell why.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

there are 150+ services ,skywalkling oap query request is slow and somtimes timout while the storage choose es #12739

{{title}}

{{editor}}'s edit

{{editor}}'s edit

Replies: 1 comment 14 replies

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

Select a reply

there are 150+ services ,skywalkling oap query request is slow and somtimes timout while the storage choose es #12739

lpq131004 Nov 1, 2024

Replies: 1 comment · 14 replies

wu-sheng Nov 1, 2024 Collaborator

wu-sheng Nov 1, 2024 Collaborator

wu-sheng Nov 1, 2024 Collaborator

lpq131004 Nov 1, 2024 Author

wu-sheng Nov 1, 2024 Collaborator

lpq131004 Nov 1, 2024 Author

wu-sheng Nov 1, 2024 Collaborator

lpq131004 Nov 1, 2024 Author

wu-sheng Nov 1, 2024 Collaborator

lpq131004
Nov 1, 2024

Replies: 1 comment 14 replies

wu-sheng
Nov 1, 2024
Collaborator

wu-sheng Nov 1, 2024
Collaborator

wu-sheng Nov 1, 2024
Collaborator

lpq131004 Nov 1, 2024
Author

wu-sheng Nov 1, 2024
Collaborator

lpq131004 Nov 1, 2024
Author

wu-sheng Nov 1, 2024
Collaborator

lpq131004 Nov 1, 2024
Author

wu-sheng Nov 1, 2024
Collaborator