Multiple class output

[Czaki]

If I correctly go through the code currently only object segmentation is supported and there is no support for assigning objects to one of the predefined class?

If I'm wrong could you point me how to do this. Or maybe it is possible to somehow modify code to achieve this?

[constantinpape]

Can you specify a bit more what exactly you want to do?
Do you want to do:

• semantic segmentation (i.e. segment pixels into classes, like background vs. cyoplasm vs. nucleus vs. mitochondrion)
• semantic instance segmentation (i.e. segment pixels into objects and classes, like mito1, mito2, mito3, ..., nucleus1, nucleus2, nucleus3, ...)

Both options are in principle supported, semantic segmentation is fairly easy, semantic instance segmentation is a bit more complex (and there are potentially multiple options for tackling it).
If you can share an example image that would also be helpful :).

[Czaki]

In general, I have 2D (2 channels) data from experiments where I classify objects into one of 11 classes (I also have labeling where each object has its number). I would like to train the network first to reproduce the output of my current algorithm (deterministic), then manually reduce bugs and retrain the net.

So it looks like the first option (translate to the second one could be done with connected components algorithm in case of my data).

I base on sample code from the readme. (I just noticed that I do not think about label_transform). Did I should just remove lable_transform and train, or should I add something else?

Sample data (channel axis is 0):

Ecoli neu_180 min 2 4E-1N 003.zip

[constantinpape]

So it looks like the first option (translate to the second one could be done with connected components algorithm in case of my data).

Yes, indeed, this looks like you can approach it with semantic segmentation.

I base on sample code from the readme. (I just noticed that I do not think about label_transform). Did I should just remove lable_transform and train, or should I add something else?

The simplest approach is to convert the labels to a one-hot encoding, then you can use almost the same code as in the sample code from the readme; you just need to change the number of output channels in your network to the number of classes. To get the one hot encoding you can use https://github.com/constantinpape/torch-em/blob/main/torch_em/transform/label.py#L150, as class_ids just pass the number of classes.

For some background: the default code will use a dice coefficient based loss, which requires a one-hot encoded target. You could also use cross entropy loss (from pytorch directly), which would not need the one-hot encoded target. But the dice loss is more robust against sparse classes (i.e. small fraction of foreground pixels) and usually works better for microscopy data.

Let me know if you run into any issues.

[Czaki]

Hi, Thanks for your response. I get nice-looking results.

I try to reconstruct labeling by using argmax and it highlights problems that I observe when looking at separate channels of output. The objects which do not belong to e given class are highlighted so it produce artifacts and requires much more sophisticated postprocessing.

When I change the number of learning steps from 2000 to 10000 I do not see significant changes.

Target segmentation:

Current segmentation:

I try to play with the initial_number of features, and depth but do not see significant changes.

May you have some suggestions? Or I just should run much more epochs?

[constantinpape]

In general the biggest question here is what kind of data you're training the network on, and if the task you want to learn is feasible given the data and labels. This is all very data dependent, so without any real knowledge about the data and/or labels here are some general observations and things I observe in the screenshots:

• Is the image you show from a separate test set that was used neither for training (train_loader) (very important!) nor for validation (val_loader) (not so important but still good practice, as long as you have enough data)
• How much training data do you have? Is it manually annotated, or produced by some other method (I think you said so somewhere). If produced by another method, then how does it discern classes that look very similar (see next bullet point). In general: the more training data the better, and it probably only makes sense to train a larger model and for more iterations if you have enough training data (what exactly is "enough" is hard to say a priori).
• Some of the classes look very similar (judging from the screenshots, where I ofc I can only really see shapes): orange, blue and the smaller roundish instances of gray and green. Is there anything in the signal that reliably enables distinguishing them.
• There seems to be almost no predictions for green. This could indicate that this class is very underrepresented in the training data and you either need more training data with it (preferable!) or introduce some weighting to the loss to make this class more important.
• Also, the "spongy" instance of green in the target segmentation appears to have almost no signal (judging from the predicted segmentation where most of it is not predicted). Is there any underlying signal and it's just much weaker or is there an error in the target segmentation?

[Czaki]

This is all very data dependent, so without any real knowledge about the data and/or labels here are some general observations and things I observe in the screenshots

These data are different stages of converting neutrophils to neutrophils net base on time and concentration of ecola bacteria. So images from different time points contain a different proportion of different class representatives. I could share more details, but on a more private channels (ex. mail, napari zulip chat etc)

• How much training data do you have? Is it manually annotated, or produced by some other method

92 images in train set and 30 in test set, annotated using a deterministic algorithm. It is to low?

• Some of the classes look very similar (judging from the screenshots, where I ofc I can only really see shapes): orange, blue and the smaller roundish instances of gray and green. Is there anything in the signal that reliably enables distinguishing them.

Data has two channels, so not only the shape, detected from the main channel, is important. But as these are

• Also, the "spongy" instance of green in the target segmentation appears to have almost no signal (judging from the predicted segmentation where most of it is not predicted).

This is the unspecific binding of the probe (added to segmentation for faster predicting potentially problematic images).

I see that maybe I do not select enough good images. I will add the next screenshot tomorrow.

[Czaki]

Sorry for the extended response. Each time I start writing the next question and need to fully formulate it, I find the next step that needs to be done. Finally, I got things working, and it looks nice.

Did you know a way to get universal information about the number of input channels from the model exported to bioimage io format?