[DSIP-73] Add dolphinscheduler-task-executor module to unify the task execution logic #16790
+5,876
−7,408
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
ruanwenjun
requested review from
caishunfeng,
SbloodyS and
Gallardot
as code owners
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
There was a problem hiding this comment.
CodeQL found more than 20 potential problems in the proposed changes. Check the Files changed tab for more details.
ruanwenjun
force-pushed
the
dev_wenjun_implementLogicTaskSupportPasueKill
branch
from
d3942d4
to
9abfb14
Compare
ruanwenjun
force-pushed
the
dev_wenjun_implementLogicTaskSupportPasueKill
branch
4 times, most recently
from
ac13593
to
e3f5a95
Compare
ruanwenjun
force-pushed
the
dev_wenjun_implementLogicTaskSupportPasueKill
branch
14 times, most recently
from
2c71278
to
2830a68
Compare
ruanwenjun
force-pushed
the
dev_wenjun_implementLogicTaskSupportPasueKill
branch
from
fced835
to
b31f567
Compare
ruanwenjun
force-pushed
the
dev_wenjun_implementLogicTaskSupportPasueKill
branch
7 times, most recently
from
93402c9
to
a8c7311
Compare
|
ruanwenjun
force-pushed
the
dev_wenjun_implementLogicTaskSupportPasueKill
branch
from
a8c7311
to
d1fa92b
Compare
ruanwenjun
force-pushed
the
dev_wenjun_implementLogicTaskSupportPasueKill
branch
5 times, most recently
from
95ea2da
to
2f72848
Compare
SbloodyS
reviewed
Nov 22, 2024
| @@ -34,4 +34,5 @@ This document records the incompatible updates between each version. You need to | |||
| * Deprecated upgrade code of 1.x and 2.x ([#16543])(https://github.com/apache/dolphinscheduler/pull/16543) | |||
| * Remove the `Data Quality` module ([#16794])(https://github.com/apache/dolphinscheduler/pull/16794) | |||
| * Remove the `registry-disconnect-strategy` in `application.yaml` ([#16821])(https://github.com/apache/dolphinscheduler/pull/16821) | |||
| * Remove `exec-threads` in worker's `application.yaml`, please use `physical-task-config`;Remove `master-async-task-executor-thread-pool-size` in master's `application.yaml`, please use `logic-task-config` ([#16790])(https://github.com/apache/dolphinscheduler/pull/16790) | |||
There was a problem hiding this comment.
Suggested change
| * Remove `exec-threads` in worker's `application.yaml`, please use `physical-task-config`;Remove `master-async-task-executor-thread-pool-size` in master's `application.yaml`, please use `logic-task-config` ([#16790])(https://github.com/apache/dolphinscheduler/pull/16790) | |
| * Remove `exec-threads` in worker's `application.yaml`, please use `physical-task-config`. Remove `master-async-task-executor-thread-pool-size` in master's `application.yaml`, please use `logic-task-config` ([#16790])(https://github.com/apache/dolphinscheduler/pull/16790) |
Comment on lines
+105
to
+125
| // public AsyncTaskExecuteFunction getAsyncTaskExecuteFunction() throws MasterTaskExecuteException { | ||
| // List<Map<String, String>> parameterGroup = generateParameterGroup(); | ||
| // | ||
| // if (parameterGroup.size() > dynamicParameters.getMaxNumOfSubWorkflowInstances()) { | ||
| // log.warn("the number of sub process instances [{}] exceeds the maximum limit [{}]", parameterGroup.size(), | ||
| // dynamicParameters.getMaxNumOfSubWorkflowInstances()); | ||
| // parameterGroup = parameterGroup.subList(0, dynamicParameters.getMaxNumOfSubWorkflowInstances()); | ||
| // } | ||
| // | ||
| // // if already exists sub process instance, do not generate again | ||
| // List<WorkflowInstance> existsSubWorkflowInstanceList = | ||
| // subWorkflowService.getAllDynamicSubWorkflow(workflowInstance.getId(), taskInstance.getTaskCode()); | ||
| // if (CollectionUtils.isEmpty(existsSubWorkflowInstanceList)) { | ||
| // generateSubWorkflowInstance(parameterGroup); | ||
| // } else { | ||
| // resetProcessInstanceStatus(existsSubWorkflowInstanceList); | ||
| // } | ||
| // return new DynamicAsyncTaskExecuteFunction(taskExecutionContext, workflowInstance, taskInstance, this, | ||
| // commandMapper, | ||
| // subWorkflowService, dynamicParameters.getDegreeOfParallelism()); | ||
| // } |
There was a problem hiding this comment.
Please remove unnessnary comment.
There was a problem hiding this comment.
We can remove this class after #16591, so I don't fix the exist problem in this class.
...org/apache/dolphinscheduler/server/master/engine/system/event/GlobalMasterFailoverEvent.java
Outdated
Show resolved
Hide resolved
SbloodyS
reviewed
Nov 22, 2024
.../main/java/org/apache/dolphinscheduler/server/master/engine/WorkflowEventBusFireWorkers.java
Outdated
Show resolved
Hide resolved
ruanwenjun
force-pushed
the
dev_wenjun_implementLogicTaskSupportPasueKill
branch
8 times, most recently
from
fc9e2d6
to
22c6fd5
Compare
ruanwenjun
force-pushed
the
dev_wenjun_implementLogicTaskSupportPasueKill
branch
from
22c6fd5
to
ecf5ece
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Purpose of the pull request
close #16619
Brief change log
Verify this pull request
This pull request is code cleanup without any test coverage.
(or)
This pull request is already covered by existing tests, such as (please describe tests).
(or)
This change added tests and can be verified as follows:
(or)
Pull Request Notice
Pull Request Notice
If your pull request contain incompatible change, you should also add it to
docs/docs/en/guide/upgrede/incompatible.md