QUERY: "Change" column not populated when running asv compare

[philipc2]

Is the "Change" column supposed to be blank when running asv compare? I was under the impression that this would include the difference in the benchmark values between the two results.

Below is a screenshot of a couple entries from when I ran asv compare.

Here is the full output: UXARRAY/uxarray#831 (comment)

[HaoZeke]

Yup, this is expected for the Ratios observed. Essentially (maybe worth documenting) the mark in the Change column comes from here:

asv/asv/commands/compare.py

Lines 293 to 330 in ee3b187

	if (version_1 is not None and version_2 is not None and
	version_1 != version_2):
	# not comparable
	color = 'lightgrey'
	mark = 'x'
	elif time_1 is not None and time_2 is None:
	# introduced a failure
	color = 'red'
	mark = '!'
	worsened = True
	elif time_1 is None and time_2 is not None:
	# fixed a failure
	color = 'green'
	mark = ' '
	improved = True
	elif time_1 is None and time_2 is None:
	# both failed
	color = 'default'
	mark = ' '
	elif _isna(time_1) or _isna(time_2):
	# either one was skipped
	color = 'default'
	mark = ' '
	elif _is_result_better(time_2, time_1,
	ss_2.get(benchmark), ss_1.get(benchmark),
	factor, use_stats=use_stats):
	color = 'green'
	mark = '-'
	improved = True
	elif _is_result_better(time_1, time_2,
	ss_1.get(benchmark), ss_2.get(benchmark),
	factor, use_stats=use_stats):
	color = 'red'
	mark = '+'
	worsened = True
	else:
	color = 'default'
	mark = ' '

For significantly worse results, for example:

Change	Before [0b43769a] <master~1>	After [0b600d8f]	Ratio	Benchmark (Parameter)
+	62.9±0.5ns	1.00±0s	1.59285e+07	benchmarks.time_myfunc

In most cases, ' ' means nothing terribly wrong has happened.

[philipc2]

@HaoZeke

That makes sense! #1421 will make it much more clear what the expected behavior is moving forward!

Thank you for the clarification!