Skip to content

Commit

Permalink
[Chore](job)Provides configuration of job execution queue size
Browse files Browse the repository at this point in the history
When dealing with a large number of tasks, the default execution queue size is 1024. This can lead to tasks being dropped if the queue becomes full. To address this, you can add the parameters `insert_task_queue_size` and `mtmv_task_queue_size` in the `fe.conf` configuration file. These parameters must be set to a power of 2. Keep in mind, increasing this value is recommended only when thread resources are limited; otherwise, you should consider increasing the number of task execution threads.
  • Loading branch information
CalvinKirs committed Oct 22, 2024
1 parent 11e6560 commit 476f162
Show file tree
Hide file tree
Showing 2 changed files with 2 additions and 2 deletions.
Original file line number Diff line number Diff line change
Expand Up @@ -45,7 +45,7 @@ public void run(Timeout timeout) {
return;
}
if (!dispatchDisruptor.publishEvent(this.job)) {
log.warn("dispatch timer job failed, job id is {}, job name is {}",
log.warn("dispatch timer job failed, queue maybe full. job id is {}, job name is {}",
this.job.getJobId(), this.job.getJobName());
}
} catch (Exception e) {
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -168,7 +168,7 @@ public void schedulerInstantJob(T job, TaskType taskType, C context) throws JobE
for (AbstractTask task : tasks) {
if (!taskDisruptorGroupManager.dispatchInstantTask(task, job.getJobType(),
job.getJobConfig())) {
throw new JobException("dispatch instant task failed, job id is "
throw new JobException("dispatch instant task failed, queue maybe full, job id is "
+ job.getJobId() + ", task id is " + task.getTaskId());
}
log.info("dispatch instant job, job id is {}, job name is {}, task id is {}", job.getJobId(),
Expand Down

0 comments on commit 476f162

Please sign in to comment.