Erros in annotations

[mazatov]

@fmagera , I just noticed quite a few errors in annotation files which is very concerning.

Do you have an estimate of what error rate we should be expecting in the annotations? Both in the classes and detected and pixel values? It, first of all, endangers the training of the segmentation model but also sets an upper bound on the accuracy so it would be nice to know.

Here's a set of examples I found by eye just in the first 50 images of the test set!

in 00020
The image is annotated to have Goal left post left while it clearly doesn't.

in 00031 :
'Small rect. right top' is missing.
"Big rect. right main" has 4 points? I didn't anything put circle that can have 4 points. Is there a guarantee that when a non-circle entity has more than 2 points that the first and the last are extremities as is assumed in the evaluator?

in 00039:
'Goal right post left', 'Big rect. left main', 'Small rect. left main': on a clearly right view image.

Some of the errors come as quite a shock because marking left-side classes on a majority right-side images is easily filterable. That raises two questions: 1) How often are the wrong classes reported in the annotations? 2) How accurate are the pixel locations? The mistakes in pixel locations are hard to spot manually, but given that we have a high rate of mistakes in classes I have to assume there are mistakes in pixels as well.

[fmagera]

Hi,

Yes indeed there are some errors in the annotation files, it has been manually annotated so yes, humans usually produce mistakes. Based on the errors you're reporting, there are about 1% classes errors, which does not seem very concerning to us.

The pixels locations are sometimes inaccurate, and this is exactly why we consider several thresholds of pixel distance between predictions and annotations.

[mazatov]

Yes indeed there are some errors in the annotation files, it has been manually annotated so yes, humans usually produce mistakes. Based on the errors you're reporting, there are about 1% classes errors, which does not seem very concerning to us.

To be honest I think the error rate is much higher than 1% which is why I am so concerned. From the errors that I've noticed, and I was only checking the main camera view, it looked more like 5%. In 20 images I checked in the test dataset, I found 3 that had a few classes with mistakes.

As I mentioned above, one of the errors was also non-circle entities having more than 2 points. Since you know the nature of the labeling process and software, do you think, when those mistakes happen that the first and the last points are still extremities?

The pixels locations are sometimes inaccurate, and this is exactly why we consider several thresholds of pixel distance between predictions and annotations.

That makes sense. I haven't found glaring pixels mistakes. It looks like the majority of the mistakes come from classes. So for 20 pixel threshold I think there wouldn't be many mistakes. But because of the mistakes in classes, I wouldn't be surprised if ~95% accuracy is the upper bound of the challenge.

[fmagera]

The lines were annotated with polylines, so the annotators that put several points were doing it sequentially and their goal was probably to draw a polyline that covered perfectly the pitch lines. Of course I can't guarantee you that you won't find an example with unordered points, but from my knowledge of the annotation tool, this is rather a good initiative.