Fix scoring bug, properly handeling nan values

[fsschneider]

When computing our benchmark scores, we want to "ignore" runs on a base workload, if the submission doesn't hit the target on the held-out workload. This is implemented here:

algorithmic-efficiency/scoring/performance_profile.py

Lines 322 to 328 in c465e25

	# For each held-out workload if variant target was not hit set submission to inf
	for workload in df.keys():
	if workload not in BASE_WORKLOADS:
	# If variants do not have finite score set base_workload score to inf
	base_workload = get_base_workload_name(workload)
	df[base_workload] = df.apply(
	variant_criteria_filter(base_workload, workload), axis=1)

However, the variant_criteria_filter() only checks for np.inf values (

algorithmic-efficiency/scoring/performance_profile.py

Lines 245 to 257 in c465e25

	def variant_criteria_filter(base_workload, variant_workload):
	def filter(x):
	try:
	if x[variant_workload] == np.inf:
	return np.inf
	else:
	return x[base_workload]
	except KeyError as e:
	print(x.keys())
	raise e
	return filter

).
But another invalid score that can occur is a nan. This happens, e.g. when running OOM. In this case, the base workload score should also be ignored.

This PR fixes this issue. To properly do so, it also needs to load the list of held-out workloads (to drop all other workload variants that have only been computed for the baseline).

[github-actions]

MLCommons CLA bot All contributors have signed the MLCommons CLA ✍️ ✅