An Inference-Based Approach to Recognizing Entailment.ppt

1、An Inference-Based Approach to Recognizing Entailment,Peter Clark and Phil Harrison Boeing Research and Technology Seattle, WA,Outline,BLUE (our system) Description Good and bad examples on RTE5 Performance and ablations on RTE5 Reflections The Knowledge Problem The Reasoning Problem,BLUE (Boeing La

2、nguage Understanding Engine),Logic Representation,Bag-of-Words Representation,T,H,YES/NO,YES,UNKNOWN,UNKNOWN,WordNet DIRT,BLUE (Boeing Language Understanding Engine),Parse, generate logic for T and H See if every clause in H subsumes part of T Use DIRT and WordNet,Logic Representation,Bag-of-Words R

3、epresentation,T,H,YES/NO,YES,UNKNOWN,UNKNOWN,WordNet DIRT,BLUE (Boeing Language Understanding Engine),Logic Representation,Bag-of-Words Representation,T,H,YES/NO,YES,UNKNOWN,UNKNOWN,WordNet DIRT,T: A black cat ate a mouse. H: A mouse was eaten by an animal.,1. The Logic Module: Generating a Represen

4、tation,(DECL (VAR _X1 “a“ “cat“ (AN “black“ “cat“) (VAR _X2 “a“ “mouse“) (S (PAST) _X1 “eat“ _X2),“cat“(cat01),“black“(black01),“eat“(eat01),“mouse“(mouse01),modifier(cat01,black01),subject(eat01,cat01),object(eat01,mouse01).,“A black cat ate a mouse.”,Parse + Logical form,Initial Logic,cat#n1(cat01

5、),black#a1(black01), mouse#n1(mouse01),eat#v1(eat01),color(cat01,black01),agent(eat01,cat01),object(eat01,mouse01).,Final Logic,1. The Logic Module: Lexico-Semantic Inference,Computing subsumption (= entailment),subject(eat01,cat01), object(eat01,mouse01), mod(cat01,black01),“by”(eat01,animal01), ob

6、ject(eat01,mouse01),T: A black cat ate a mouse,H: A mouse was eaten by an animal,1. The Logic Module: Lexico-Semantic Inference,Subsumption,subject(eat01,cat01), object(eat01,mouse01), mod(cat01,black01),“by”(eat01,animal01), object(eat01,mouse01),T: A black cat ate a mouse,H: A mouse was eaten by a

7、n animal,WordNet,also,Inference with DIRT,T: A black cat ate a mouse,IF X eats Y THEN X chews Y,IF X eats Y THEN X digests Y,T: A black cat ate a mouse. The cat is black.The cat digests the mouse. The cat chewed the mouse. The cat swallows the mouse,With Inference,T: A black cat ate a mouse,IF X eat

8、s Y THEN X digests Y,H: An animal digested the mouse.,Subsumes,IF X eats Y THEN X chews Y,T: A black cat ate a mouse. The cat is black.The cat digests the mouse. The cat chewed the mouse. The cat swallows the mouse,H entailed!,BLUE (Boeing Language Understanding Engine),WordNet DIRT,Logic Representa

9、tion,Bag-of-Words Representation,T,H,YES/NO,YES,UNKNOWN,UNKNOWN,Ignore syntactic structure: Use bag of words for T and H See if every word in H subsumes one in T Use DIRT and WordNet,BLUE (Boeing Language Understanding Engine),WordNet DIRT,Logic Representation,Bag-of-Words Representation,T,H,YES/NO,

10、YES,UNKNOWN,UNKNOWN,REPRESENTATION, black cat eat mouse , mouse digest animal ,subsumes?,T: A black cat ate a mouse. H: A mouse was digested by an animal.,Bag of Words Inference, black cat eat mouse , mouse digest animal ,subsumes?,T: A black cat ate a mouse. H: A mouse was digested by an animal.,T,

11、H,Bag of Words Inference, black cat eat mouse , mouse digest animal ,T: A black cat ate a mouse. H: A mouse was digested by an animal.,T,H,WordNet,Bag of Words Inference, black cat eat mouse , mouse digest animal ,T: A black cat ate a mouse. H: A mouse was digested by an animal.,T,H,DIRT,IF X eats Y

12、 THEN X digests Y,“eat”,“digest”,Bag of Words Inference, black cat eat mouse , mouse digest animal ,T: A black cat ate a mouse. H: A mouse was digested by an animal.,T,H,H entailed!,Outline,BLUE (our system) Description Good and bad examples on RTE5 Performance and ablations on RTE5 Reflections The

13、Knowledge Problem The Reasoning Problem,The Good,T: Ernie Barneswas an offensive linesman. H: Ernie Barnes was an athlete.,#191 (BLUE got this right),via WordNet: linesman#n1 isa athlete#n1,T: hijacking of a Norwegian tankerby Somali pirates H: Somali pirates attacked a Norwegian tanker.,#333 (BLUE

14、got this right),via DIRT: IF X hijacks Y THEN Y is attacked by X.,T: Charles divorced Diana H: Prince Charles was married to Princess Diana.,Pilot H26 (BLUE got this right),via DIRT: IF X divorces Y THEN X marries Y.,The Good (Cont),HEADLINE: EU slams Nepalese kings dismissal T: The EUpresidency cal

15、led for democracy. H: There has been acall for democracy in Nepal,Pilot H142 (BLUE got this right),via use of HEADLINE as context (and WordNet Nepalese/Nepal),T: Crippa diedafter he atedeadlywild mushrooms H: Crippa was killed by a wild mushroom.,#78 (BLUE got this right),via DIRT: IF X dies of Y TH

16、EN Y is killed by X,The Bad,T: Venus Williams triumphed overBartoli H*: Venus Williams was defeated byBartoli,#407 (BLUE got this wrong, predicting YES),via (bad rule in) DIRT: IF Y wins over X THEN X defeats Y.,T: PepsiCo acquired Star Foods H: PepsiCo holds Star Foods,#219 (BLUE got this right, bu

17、t for nonsensical reasons),via DIRT: IF X acquires Y THEN X sells Yand: IF Y sells Xs business THEN Y holds Xs tongue and WordNet: “tongue” isa “food”,T: even if Iceland offered Fischer citizenship. H*: Iceland granted Bobby Fischer citizenship.,Pilot H29 (BLUE got this wrong, predicting YES),BLUE d

18、oes not recognize the hypothetical blocks entailment,The Bad (Cont),T: Slumdog Millionaire director Danny Boyle. H: The movie “Slumdog Millionaire” has been directed by Danny Boyle.,#157 (BLUE got this wrong, predicting UNKNOWN),(unable to conclude “movie” in H),T: the oath taken by the 115 electors

19、 H*: The cardinals electing the pope,“115 is a cardinal” (!),Pilot H75 (BLUE got this wrong, predicting YES),Outline,BLUE (our system) Description Good and bad examples on RTE5 Performance and ablations on RTE5 Reflections The Knowledge Problem The Reasoning Problem,Results: Main Task,Pipeline works

20、 best,Logic,Bag,Logic,Bag,Results: Main Task,Pipeline works best,Logic,Bag,Logic,Bag,Logic alone is worse than bag alone Only decides 29% of cases, but does well (64%) on these,Ablation Studies,Ablation Studies,Ablation Studies,WordNet is significantly helping,Ablation Studies,WordNet is significant

21、ly helping DIRT is barely helpingRules are noisy ( 50% are bad)Applicability is low ( 10%-15%) most RTE problems are outside DIRTs scope,Ablation Studies,WordNet is significantly helping DIRT is barely helping Parsing is barely helpingExtracting syntactic structure is very error-proneSemantic relati

22、onships usually persist from T to H and H* Non-entailment caused by other factors,“Semantic Continuity”,How important is semantic (hence syntactic) structure?,T: Boyle directed Slumdog Millionaire H*: Slumdog Millionaire directed Boyle NOT entailed,IF T and H are “sensible” AND T and H are consisten

23、t with world knowledge AND T and H are topically similar THEN this heavily constrains the variability in possible semantic (hence syntactic) relationships, reduced discriminatory power of semantic/syntactic analysis,“Semantic Continuity” Conjecture:,but this kind of example is unusual in RTE!,Outlin

24、e,BLUE (our system) Description Good and bad examples on RTE5 Performance and ablations on RTE5 Reflections The Knowledge Problem The Reasoning Problem,What are the Long Term Challenges?,The Knowledge Problem Still missing a lot of world knowledge,Need to know flood: water.overflowing onto normally

25、dry land bank: sloping landbesidewater,?,What are the Long Term Challenges?,The Knowledge Problem Still missing a lot of world knowledge,Need to know flood: water.overflowing onto normally dry land bank: sloping landbesidewater,includes,What are the Long Term Challenges?,The Knowledge Problem Still

26、missing a lot of world knowledge The Reasoning Problem Finding some path from T to H is error-prone,What are the Long Term Challenges?,The Knowledge Problem Still missing a lot of world knowledge The Reasoning Problem Finding some path from T to H is error-prone,T: Venus Williams triumphed over Bart

27、olito win H*: Venus Williams was defeated byBartoli,BUT: evidence against H: triumph=defeat, and defeat is antisymmetric World Knowledge: “win” implies defeat (not defeated by) Better: look at multiple reasoning pathsfind the “best”, consistent subset of implications,IF Y triumphs over X THEN X defe

28、ats Y,Wrong,#407,Venus Williams triumphed over Bartoli to win,“T” text:,Williams triumphed over Bartoli,Venus Williams triumphed over Bartoli to win,Williams won,Williams was defeated by Bartoli,Williams triumphed,Williams lost to Bartoli,Williams defeated someone,Williams had a victory,“T” text:,Wi

29、lliams triumphed over Bartoli,Venus Williams triumphed over Bartoli to win,Williams won,Williams was defeated by Bartoli,Williams triumphed,Williams lost to Bartoli,Williams defeated someone,Williams had a victory,“T” text:,“H” text:,Was Williams defeated?,Answer: No!,What is the overall scene?,Will

30、iams triumphed over Bartoli,Venus Williams triumphed over Bartoli to win,Williams won,Williams was defeated by Bartoli,Williams triumphed,Williams lost to Bartoli,Williams defeated someone,Williams had a victory,“T” text:,“H” text:,Williams was defeated?,Answer: No!,What is the overall scene?,Answer

31、Williams triumphed over Bartoli,Venus Williams triumphed over Bartoli to win,Williams won,Williams was defeated by Bartoli,Williams triumphed,Williams lost to Bartoli,Williams defeated someone,Williams had a victory,“T” text:,a step towards text “understanding”,Summary,BLUE: Pipeline of logical representation + bag-of-words Reasoning with WordNet and DIRT Performance ok (above the median) Ablations: WordNet helps a lot DIRT and parsing barely helped Two big challenges: Knowledge need lots more Reasoning need search for coherence, not a single path,Thank you!,