Fix race condition in ThreadJob.waitForRun #659

ThreadJob.waitForRun() suffers from a race condition. Between checking
for a blocking job with a conflicting scheduling rule and starting to
wait on the state lock of that blocking job to get notified when the
scheduling rule is released, the blocking job may have already changed
its state and acquired a different scheduling rule, such that no
conflict exists anymore. Since this condition is not reevaluated but
waiting on the lock object is started without a timeout, the blocked job
unnecessarily waits for the initially blocking job to change its state
again, i.e., to finish its execution.

One relevant situation in which the scheduling rule of a job can change
is if the job is an internally used ThreadJob. They are started whenever
a rule is acquired on the JobManager and are reused for the same thread.
When the same thread acquires multiple rules, one after another, the
rule of the ThreadJob may effectively change. This, for example, happens
during a workspace build operation, which changes the rule for two
times. Thus, if the blocking job in ThreadJob.waitForRun() is a
ThreadJob, it may change its rule, in particular when that job performs
a build operation.

With this change, the scheduling rules of blocking and blocked thread
are checked for conflicts again before starting to wait on the state
lock in the blocked job. This leads to a reevaluation of the block
condition in case the scheduling rule of the blocking job has changed in
between and avoids the job to unnecessarily block until the blocking job
finishes.

Fixes #659.

resources/tests/org.eclipse.core.tests.resources/src/org/eclipse/core/tests/internal/builders/ParallelBuildChainTest.java

@@ -141,6 +141,17 @@ public void testIndividualProjectBuilds_ProjectRelaxedRule() throws Exception {
 		});
 	}
 
+	@Test
+	public void testIndividualProjectBuilds_WithManyProjects_ProjectRelaxedRule() throws Exception {
+		int numberOfParallelBuilds = 60;
+		var longRunningProjects = createMultipleTestProjects(numberOfParallelBuilds, BuildDurationType.LONG_RUNNING,
+				RuleType.CURRENT_PROJECT_RELAXED);
+		executeIndividualFullProjectBuilds(numberOfParallelBuilds, () -> {
+			assertBuildsToStart(getAllProjects());
+			assertMinimumNumberOfSimultaneousBuilds(longRunningProjects.size());
+		});
+	}
+
 	@Test
 	public void testWorkspaceBuild_NoConflictRule() throws Exception {
 		int numberOfParallelBuilds = 3;

runtime/bundles/org.eclipse.core.jobs/src/org/eclipse/core/internal/jobs/ThreadJob.java

@@ -15,8 +15,14 @@
 
 import java.util.List;
 import org.eclipse.core.internal.runtime.RuntimeLog;
-import org.eclipse.core.runtime.*;
-import org.eclipse.core.runtime.jobs.*;
+import org.eclipse.core.runtime.Assert;
+import org.eclipse.core.runtime.IProgressMonitor;
+import org.eclipse.core.runtime.IStatus;
+import org.eclipse.core.runtime.OperationCanceledException;
+import org.eclipse.core.runtime.Status;
+import org.eclipse.core.runtime.jobs.ISchedulingRule;
+import org.eclipse.core.runtime.jobs.Job;
+import org.eclipse.core.runtime.jobs.LockListener;
 
 /**
  * Captures the implicit job state for a given thread.
@@ -317,11 +323,19 @@ private static ThreadJob waitForRun(final ThreadJob threadJob, IProgressMonitor
 						// The actual exit conditions are listed above at the beginning of
 						// this while loop
 						int state = blockingJob.getState();
-						//ensure we don't wait forever if the blocker is waiting, because it might have yielded to me
-						if (state == Job.RUNNING && canBlock) {
-							blockingJob.jobStateLock.wait();
-						} else if (state != Job.NONE) {
-							blockingJob.jobStateLock.wait(250);
+						// Check that blockingJob has not acquired a different, non-conflicting
+						// scheduling rule since checking for conflicts by JobManager. This can
+						// particularly happen if blockingJob is a ThreadJob that is reused for the
+						// same thread across the acquisition of different rules (see ThreadJob::recycle
+						// and ImplicitJob::newThreadJob) via JobManager::beginRule.
+						if (state != Job.NONE && blockingJob.isConflicting(threadJob)) {
+							// ensure we don't wait forever if the blocker is waiting, because it might have
+							// yielded to me
+							if (state == Job.RUNNING && canBlock) {
+								blockingJob.jobStateLock.wait();
+							} else {
+								blockingJob.jobStateLock.wait(250);
+							}
 						}
 					} catch (InterruptedException e) {
 						// This thread may be interrupted via two common scenarios. 1) If