monai.metrics.MeanDice() giving wrong results?

[YerePhy]

I'm computing DICE metrics over a set of tensors. These tensors have shape (N=1)(C=2)HWD. The problem is that monai.metrics.compute_meandice and a custom implementation of DICE are giving different results than monai.metrics.MeanDice.

My implementations:

 
class CustomDice(torch.nn.Module):

    def __init__(self, include_background=False, squeeze_batch=False):
        super().__init__()
        self.include_background = include_background
        self.squeeze_batch = squeeze_batch

    def forward(self, y_pred, y_true):
        if self.squeeze_batch:
            y_pred = y_pred.squeeze(0)
            y_true = y_true.squeeze(0)
        if not self.include_background:
            y_pred = y_pred[1:]
            y_true = y_true[1:]

        self.value = 2 * torch.sum(y_true * y_pred, dim=[-1,-2,-3]) /  torch.sum(y_true**2 + y_pred**2, dim=[-1,-2,-3])
        return self.value
    
    def aggregate(self):
        return self.value

class ComputeDice(torch.nn.Module):

    def __init__(self, include_background=False, squeeze_batch=False):
        super().__init__()
        self.include_background = include_background

    def forward(self, y_pred, y_true):
        
        self.value = m.compute_meandice(y_pred, y_true, include_background=self.include_background)
        return self.value
    
    def aggregate(self):
        return self.value

Instantiations:

metrics = {
    "COMPUTE_DICE": ComputeDice(include_background=False),
    "CUSTOM_DICE": CustomDice(include_background=False, squeeze_batch=True),
    "DICE": m.DiceMetric(include_background=False, reduction="mean")
}

I'm computing the metrics as follows:

# metric_ is a value of metrics Dict
metric_(y_pred, y_true)
value = metric_.aggregate()

Results:

[rijobro]

Are you using metric_.reset() for the last one?

[YerePhy]

Sorry for wasting your time... However, it would be nice to be a little more explicit about this in the documentation since it is hidden in the example of monai.metrics.CumulativeIterationMetric and briefly documented in monai.metrics.Cumulative.

Thanks @rijobro

[rijobro]

@Nic-Ma do you think you could have a look at this one? (I'm on annual leave.)

[Nic-Ma]

Hi @rijobro , sure, it would be useful to enhance the documentation, we already received this request multiple times.

Thanks.

[Nic-Ma]

The example doc is covered in this PR: #4660.

Thanks.

[Sinamhd9]

@Nic-Ma I might have a similar problem that couldn't figure out. Would you please help? I am not sure why I am getting different results from DiceMetric and compute_meandice. See the code below. I am breaking the loop after first example, so the get_buffer method of dice metric is giving out this number [0.8170, 0.4066, 0.4384] for the three classes. Compute mean dice is giving me 208 numbers and nans. The shapes of labels and outputs are [3, 208, 230, 172]. Ignore empty in both cases are set to True, so the nans are produced where ground truth is empty. When I do average of these 208 numbers (torch.nanmean), the result is different than the above three numbers. It is [0.7121, 0.2461, 0.2598]. So my question is how exactly this array is being reduced to those three numbers above?

Code:

dice_metric = DiceMetric(include_background=True, reduction='none',ignore_empty=True)

with torch.no_grad():
    for val_data in val_loader:
        val_input = val_data['image'].to(device)
        val_label = val_data['label'].to(device)
        val_output = inference(val_input)
        val_output = [post_trans(i) for i in decollate_batch(val_output)]
        dice_metric(y_pred=val_output, y=val_label)
        metric = compute_meandice(val_output[0],val_label[0], ignore_empty=True,include_background=True)
        break
        

[Sinamhd9]

@Nic-Ma I might have a similar problem that couldn't figure out. Would you please help? I am not sure why I am getting different results from DiceMetric and compute_meandice. See the code below. I am breaking the loop after first example, so the get_buffer method of dice metric is giving out this number [0.8170, 0.4066, 0.4384] for the three classes. Compute mean dice is giving me 208 numbers and nans. The shapes of labels and outputs are [3, 208, 230, 172]. Ignore empty in both cases are set to True, so the nans are produced where ground truth is empty. When I do average of these 208 numbers (torch.nanmean), the result is different than the above three numbers. So my question is how exactly this array is being reduced to those three numbers above?

Code:

dice_metric = DiceMetric(include_background=True, reduction='none',ignore_empty=True)

with torch.no_grad():
    for val_data in val_loader:
        val_input = val_data['image'].to(device)
        val_label = val_data['label'].to(device)
        val_output = inference(val_input)
        val_output = [post_trans(i) for i in decollate_batch(val_output)]
        
        dice_metric(y_pred=val_output, y=val_label)
        metric = compute_meandice(val_output[0],val_label[0], ignore_empty=True,include_background=True)
        break

[Nic-Ma]

Hi @Sinamhd9 ,

Here is the logic that MONAI metrics compute the mean:
https://github.com/Project-MONAI/MONAI/blob/dev/monai/metrics/utils.py#L75
@KumoLiu can share you more details if you still face any issues.

Thanks.

[Sinamhd9]

@Nic-Ma Thanks for your reply.
@KumoLiu The problem still persists. I do reduction step by step the results are different. , I see f[nans] = 0. When I do this on the output of compute mean_dice, it gives way worse results comparing to the buffer of DiceMetric. To put it simply, the output I get from the compute mean dice, is a tensor with three elements, the first one (for the first class) is this array below. Whether I set the nans to 0 or 1, then use mean, or whether I use mean only on non-nan values, (torch.nanmean), it will never become 0.8170, the result of DiceMetric.

tensor([   nan,    nan,    nan,    nan,    nan,    nan,    nan,    nan,    nan,
           nan,    nan,    nan,    nan,    nan,    nan,    nan,    nan,    nan,
           nan,    nan,    nan,    nan,    nan,    nan,    nan,    nan,    nan,
           nan,    nan,    nan,    nan,    nan,    nan,    nan,    nan,    nan,
           nan,    nan,    nan,    nan,    nan,    nan,    nan, 0.3977, 0.4706,
        0.6078, 0.6516, 0.6659, 0.7067, 0.7331, 0.7053, 0.7791, 0.7853, 0.7704,
        0.8574, 0.8641, 0.8663, 0.8892, 0.8968, 0.8931, 0.8888, 0.8839, 0.9171,
        0.9071, 0.8869, 0.8840, 0.8818, 0.8848, 0.8983, 0.8829, 0.8507, 0.8248,
        0.8244, 0.8312, 0.7786, 0.7763, 0.7706, 0.7418, 0.6996, 0.7120, 0.7021,
        0.7162, 0.6866, 0.5866, 0.5088, 0.2899, 0.0000, 0.0000, 0.0000,    nan,
           nan,    nan,    nan,    nan,    nan,    nan,    nan,    nan,    nan,
           nan,    nan,    nan,    nan,    nan,    nan,    nan,    nan,    nan,
           nan,    nan,    nan,    nan,    nan,    nan,    nan,    nan,    nan,
           nan,    nan,    nan,    nan,    nan,    nan,    nan,    nan,    nan,
           nan,    nan,    nan,    nan,    nan,    nan,    nan,    nan,    nan,
           nan,    nan,    nan,    nan,    nan,    nan,    nan,    nan,    nan,
           nan,    nan,    nan,    nan,    nan,    nan,    nan,    nan,    nan,
           nan,    nan,    nan,    nan,    nan,    nan,    nan,    nan,    nan,
           nan,    nan,    nan,    nan,    nan,    nan,    nan,    nan,    nan,
           nan,    nan,    nan,    nan,    nan,    nan,    nan,    nan,    nan,
           nan,    nan,    nan,    nan,    nan,    nan,    nan,    nan,    nan,
           nan,    nan,    nan,    nan,    nan,    nan,    nan,    nan,    nan,
           nan,    nan,    nan,    nan,    nan,    nan,    nan,    nan,    nan,
           nan], device='cuda:0')

[KumoLiu]

Hi @Sinamhd9, compute_meandice need input be one-hot format and first dim is batch and will return a result with shape [batch_size, num_classes]. So maye check your data and label's shape.

[Sinamhd9]

Thank you so much @KumoLiu! Problem solved! Explaining here if anyone has the same problem, my val_output and val_labels were lists with one element with the size (3, 208, 230, 172). You can pass these list to DiceMetric but cannot pass lists (should pass tensors) to compute_meandice. So I was passing val_output[0], and val_label[0] to compute meandice, however I should have made the val_input and val_output to be a tensor with size (1, 3, 208, 230, 172) instead. Now the results of dice_metric.get_buffer() (DiceMetric) and metric (compute_ meandice) match.

[RY-97]

I also encountered the same problem. When I ran the following code, some categories were nan.

Can you help to solve it?