Removed incorrect not-entailed examples in VerbNet
The method to recast VerbNet annotations/examples into NLI created an issue in some examples. Here we provide a brief example and below we include statistics about the change.
Example
Consider the sentence Jackie chased the thief .
which comes from the example in VN class chase-51.6. The semantics for this example from VN are motion(during(E), Agent) motion(during(E), Theme)
indicating that both the Agent
, i.e. Jackie
, and the Theme
, i.e. the thief
moved. To create negative, i.e. not-entailed
examples, we swapped thematic roles in our templates.
Consequently, this example sentence would then incorrectly label the example where the hypothesis would be The thief moved
as not-entailed
. In this version/release of the dataset, we removed such examples
Stats
train | dev | test | |
---|---|---|---|
unique contexts | 23 | 6 | 2 |
all unique pairs | 1386 | 143 | 160 |
bad pairs | 43 | 11 | 4 |
% bad pairs | 3.10 | 7.69 | 2.50 |
------------------ | ------- | ------ | ------ |
# v0.1 examples | 1441 | 154 | 164 |
# v0.1.1 examples | 1398 | 143 | 160 |