Skip to content

Removed incorrect not-entailed examples in VerbNet

Compare
Choose a tag to compare
@azpoliak azpoliak released this 09 May 20:00
· 3 commits to master since this release

The method to recast VerbNet annotations/examples into NLI created an issue in some examples. Here we provide a brief example and below we include statistics about the change.

Example

Consider the sentence Jackie chased the thief . which comes from the example in VN class chase-51.6. The semantics for this example from VN are motion(during(E), Agent) motion(during(E), Theme) indicating that both the Agent, i.e. Jackie, and the Theme, i.e. the thief moved. To create negative, i.e. not-entailed examples, we swapped thematic roles in our templates.

Consequently, this example sentence would then incorrectly label the example where the hypothesis would be The thief moved as not-entailed. In this version/release of the dataset, we removed such examples

Stats

train dev test
unique contexts 23 6 2
all unique pairs 1386 143 160
bad pairs 43 11 4
% bad pairs 3.10 7.69 2.50
------------------ ------- ------ ------
# v0.1 examples 1441 154 164
# v0.1.1 examples 1398 143 160