Label leakage in ECtHR dataset

[schnpwr]

I observed that actual ECHR article numbers are present in the text of case descriptions. For example, in ECtHR B dataset, in the test split, I found 178 instances where at least one gold-standard label is explicitly mentioned in the case description. Can anyone confirm whether such article mentions were masked while computing the accuracy numbers? If yes, how were they masked? I feel that if the data is used as it is, it will give an unfair advantage to the classifiers.

[iliaschalkidis]

Hi @schnpwr,

Thanks for bringing this to our attention, it's a really interesting issue.

In the ECtHR A task, this is not an issue in my opinion, since the labeling accounts for violated articles that are only verbatim mentioned in the court decision -the last part of the court's opinion-, which is not part of the input (facts). So, references to the allegedly violated articles (a superset of the violated ones, i.e., the set of violated articles is a subset of those) do not leak the labeling, since the involved parties, their legal representation, and the judges all know which are the articles at stake.

In the ECtHR B task, it is a potential issue, since the labeling refers to the allegedly violated articles, i.e., those that are considered/discussed in court in order to decide if they have been violated or not. In this case, the potential labeling leakage should be considered case by case, since in many cases the input legal case is a re-trial or appeal in some sense, which means that the court has already considered the applicant's legal case in the past, and there were grounds for reconsideration, which means that the original allegedly violated articles are part of the facts. In any case, I agree that masking this information would improve the experimental setup since it's hard to distinguish whether this information leads to label leakage or not.

It would be interesting to manually review these cases to identify what is the role of these references. Could you provide the IDs or some snippets? Have you already considered experiments w/ and w/o masking?

[schnpwr]

Thank you @iliaschalkidis for your response.

I simply checked the regex below:
'Article (2|3|5|6|8|9|10|11|14|P1-1)\b'

And checked if the Article number matched in the group 1 is also present as one of the labels. In test split, found 178 such instances. Some examples are IDs 0, 13, 14, 20, 21.. here these are indexes starting from 0.

I haven't done any experiments with masking yet. But thinking of changing the text matched to the above regex to 'Article XX'. Any better ideas for masking?

Thanks and Regards,

• Sachin