Skip to content

Instantly share code, notes, and snippets.

@So-Cool

So-Cool/Readme.md

Last active May 27, 2017 16:23
Show Gist options
  • Save So-Cool/43587e5745cc46ce837895d7f62a7f82 to your computer and use it in GitHub Desktop.
Save So-Cool/43587e5745cc46ce837895d7f62a7f82 to your computer and use it in GitHub Desktop.
Download ZIP
AISB hackathon -- story comprehension
Raw
Readme.md

Automated story comprehension

This notebook shows how to extract background knowledge and relations from a story text. To this end, we use Stanford's NLP parser available to download here.

We parsed the following story (story.txt file):

Mary was sleeping.
Her phone rang.
She was annoyed.
Mary answered the phone.
Ann told the good news to Mary.

with

java -cp "*" -Xmx2g edu.stanford.nlp.pipeline.StanfordCoreNLP -annotators tokenize,ssplit,pos,lemma,ner,parse,dcoref -file story.txt

This resulted in an xml file story.txt.xml that contains all the necessary data to extract relevant information from the story.

The next step is to extract this information and represent it as Prolog facts. To this end, we wrote the Python code included below.

Raw
story.txt
Mary was sleeping.
Her phone rang.
She was annoyed.
Mary answered the phone.
Ann told the good news to Mary.
Raw
story.txt.xml
<?xml version="1.0" encoding="UTF-8"?>
<?xml-stylesheet href="CoreNLP-to-HTML.xsl" type="text/xsl"?>
<root>
<document>
<sentences>
<sentence id="1">
<tokens>
<token id="1">
<word>Mary</word>
<lemma>Mary</lemma>
<CharacterOffsetBegin>0</CharacterOffsetBegin>
<CharacterOffsetEnd>4</CharacterOffsetEnd>
<POS>NNP</POS>
<NER>PERSON</NER>
<Speaker>PER0</Speaker>
</token>
<token id="2">
<word>was</word>
<lemma>be</lemma>
<CharacterOffsetBegin>5</CharacterOffsetBegin>
<CharacterOffsetEnd>8</CharacterOffsetEnd>
<POS>VBD</POS>
<NER>O</NER>
<Speaker>PER0</Speaker>
</token>
<token id="3">
<word>sleeping</word>
<lemma>sleep</lemma>
<CharacterOffsetBegin>9</CharacterOffsetBegin>
<CharacterOffsetEnd>17</CharacterOffsetEnd>
<POS>VBG</POS>
<NER>O</NER>
<Speaker>PER0</Speaker>
</token>
<token id="4">
<word>.</word>
<lemma>.</lemma>
<CharacterOffsetBegin>17</CharacterOffsetBegin>
<CharacterOffsetEnd>18</CharacterOffsetEnd>
<POS>.</POS>
<NER>O</NER>
<Speaker>PER0</Speaker>
</token>
</tokens>
<parse>(ROOT (S (NP (NNP Mary)) (VP (VBD was) (VP (VBG sleeping))) (. .))) </parse>
<dependencies type="basic-dependencies">
<dep type="root">
<governor idx="0">ROOT</governor>
<dependent idx="3">sleeping</dependent>
</dep>
<dep type="nsubj">
<governor idx="3">sleeping</governor>
<dependent idx="1">Mary</dependent>
</dep>
<dep type="aux">
<governor idx="3">sleeping</governor>
<dependent idx="2">was</dependent>
</dep>
<dep type="punct">
<governor idx="3">sleeping</governor>
<dependent idx="4">.</dependent>
</dep>
</dependencies>
<dependencies type="collapsed-dependencies">
<dep type="root">
<governor idx="0">ROOT</governor>
<dependent idx="3">sleeping</dependent>
</dep>
<dep type="nsubj">
<governor idx="3">sleeping</governor>
<dependent idx="1">Mary</dependent>
</dep>
<dep type="aux">
<governor idx="3">sleeping</governor>
<dependent idx="2">was</dependent>
</dep>
<dep type="punct">
<governor idx="3">sleeping</governor>
<dependent idx="4">.</dependent>
</dep>
</dependencies>
<dependencies type="collapsed-ccprocessed-dependencies">
<dep type="root">
<governor idx="0">ROOT</governor>
<dependent idx="3">sleeping</dependent>
</dep>
<dep type="nsubj">
<governor idx="3">sleeping</governor>
<dependent idx="1">Mary</dependent>
</dep>
<dep type="aux">
<governor idx="3">sleeping</governor>
<dependent idx="2">was</dependent>
</dep>
<dep type="punct">
<governor idx="3">sleeping</governor>
<dependent idx="4">.</dependent>
</dep>
</dependencies>
<dependencies type="enhanced-dependencies">
<dep type="root">
<governor idx="0">ROOT</governor>
<dependent idx="3">sleeping</dependent>
</dep>
<dep type="nsubj">
<governor idx="3">sleeping</governor>
<dependent idx="1">Mary</dependent>
</dep>
<dep type="aux">
<governor idx="3">sleeping</governor>
<dependent idx="2">was</dependent>
</dep>
<dep type="punct">
<governor idx="3">sleeping</governor>
<dependent idx="4">.</dependent>
</dep>
</dependencies>
<dependencies type="enhanced-plus-plus-dependencies">
<dep type="root">
<governor idx="0">ROOT</governor>
<dependent idx="3">sleeping</dependent>
</dep>
<dep type="nsubj">
<governor idx="3">sleeping</governor>
<dependent idx="1">Mary</dependent>
</dep>
<dep type="aux">
<governor idx="3">sleeping</governor>
<dependent idx="2">was</dependent>
</dep>
<dep type="punct">
<governor idx="3">sleeping</governor>
<dependent idx="4">.</dependent>
</dep>
</dependencies>
</sentence>
<sentence id="2">
<tokens>
<token id="1">
<word>Her</word>
<lemma>she</lemma>
<CharacterOffsetBegin>19</CharacterOffsetBegin>
<CharacterOffsetEnd>22</CharacterOffsetEnd>
<POS>PRP$</POS>
<NER>O</NER>
<Speaker>PER0</Speaker>
</token>
<token id="2">
<word>phone</word>
<lemma>phone</lemma>
<CharacterOffsetBegin>23</CharacterOffsetBegin>
<CharacterOffsetEnd>28</CharacterOffsetEnd>
<POS>NN</POS>
<NER>O</NER>
<Speaker>PER0</Speaker>
</token>
<token id="3">
<word>rang</word>
<lemma>ring</lemma>
<CharacterOffsetBegin>29</CharacterOffsetBegin>
<CharacterOffsetEnd>33</CharacterOffsetEnd>
<POS>VBD</POS>
<NER>O</NER>
<Speaker>PER0</Speaker>
</token>
<token id="4">
<word>.</word>
<lemma>.</lemma>
<CharacterOffsetBegin>33</CharacterOffsetBegin>
<CharacterOffsetEnd>34</CharacterOffsetEnd>
<POS>.</POS>
<NER>O</NER>
<Speaker>PER0</Speaker>
</token>
</tokens>
<parse>(ROOT (S (NP (PRP$ Her) (NN phone)) (VP (VBD rang)) (. .))) </parse>
<dependencies type="basic-dependencies">
<dep type="root">
<governor idx="0">ROOT</governor>
<dependent idx="3">rang</dependent>
</dep>
<dep type="nmod:poss">
<governor idx="2">phone</governor>
<dependent idx="1">Her</dependent>
</dep>
<dep type="nsubj">
<governor idx="3">rang</governor>
<dependent idx="2">phone</dependent>
</dep>
<dep type="punct">
<governor idx="3">rang</governor>
<dependent idx="4">.</dependent>
</dep>
</dependencies>
<dependencies type="collapsed-dependencies">
<dep type="root">
<governor idx="0">ROOT</governor>
<dependent idx="3">rang</dependent>
</dep>
<dep type="nmod:poss">
<governor idx="2">phone</governor>
<dependent idx="1">Her</dependent>
</dep>
<dep type="nsubj">
<governor idx="3">rang</governor>
<dependent idx="2">phone</dependent>
</dep>
<dep type="punct">
<governor idx="3">rang</governor>
<dependent idx="4">.</dependent>
</dep>
</dependencies>
<dependencies type="collapsed-ccprocessed-dependencies">
<dep type="root">
<governor idx="0">ROOT</governor>
<dependent idx="3">rang</dependent>
</dep>
<dep type="nmod:poss">
<governor idx="2">phone</governor>
<dependent idx="1">Her</dependent>
</dep>
<dep type="nsubj">
<governor idx="3">rang</governor>
<dependent idx="2">phone</dependent>
</dep>
<dep type="punct">
<governor idx="3">rang</governor>
<dependent idx="4">.</dependent>
</dep>
</dependencies>
<dependencies type="enhanced-dependencies">
<dep type="root">
<governor idx="0">ROOT</governor>
<dependent idx="3">rang</dependent>
</dep>
<dep type="nmod:poss">
<governor idx="2">phone</governor>
<dependent idx="1">Her</dependent>
</dep>
<dep type="nsubj">
<governor idx="3">rang</governor>
<dependent idx="2">phone</dependent>
</dep>
<dep type="punct">
<governor idx="3">rang</governor>
<dependent idx="4">.</dependent>
</dep>
</dependencies>
<dependencies type="enhanced-plus-plus-dependencies">
<dep type="root">
<governor idx="0">ROOT</governor>
<dependent idx="3">rang</dependent>
</dep>
<dep type="nmod:poss">
<governor idx="2">phone</governor>
<dependent idx="1">Her</dependent>
</dep>
<dep type="nsubj">
<governor idx="3">rang</governor>
<dependent idx="2">phone</dependent>
</dep>
<dep type="punct">
<governor idx="3">rang</governor>
<dependent idx="4">.</dependent>
</dep>
</dependencies>
</sentence>
<sentence id="3">
<tokens>
<token id="1">
<word>She</word>
<lemma>she</lemma>
<CharacterOffsetBegin>35</CharacterOffsetBegin>
<CharacterOffsetEnd>38</CharacterOffsetEnd>
<POS>PRP</POS>
<NER>O</NER>
<Speaker>PER0</Speaker>
</token>
<token id="2">
<word>was</word>
<lemma>be</lemma>
<CharacterOffsetBegin>39</CharacterOffsetBegin>
<CharacterOffsetEnd>42</CharacterOffsetEnd>
<POS>VBD</POS>
<NER>O</NER>
<Speaker>PER0</Speaker>
</token>
<token id="3">
<word>annoyed</word>
<lemma>annoy</lemma>
<CharacterOffsetBegin>43</CharacterOffsetBegin>
<CharacterOffsetEnd>50</CharacterOffsetEnd>
<POS>VBN</POS>
<NER>O</NER>
<Speaker>PER0</Speaker>
</token>
<token id="4">
<word>.</word>
<lemma>.</lemma>
<CharacterOffsetBegin>50</CharacterOffsetBegin>
<CharacterOffsetEnd>51</CharacterOffsetEnd>
<POS>.</POS>
<NER>O</NER>
<Speaker>PER0</Speaker>
</token>
</tokens>
<parse>(ROOT (S (NP (PRP She)) (VP (VBD was) (ADJP (VBN annoyed))) (. .))) </parse>
<dependencies type="basic-dependencies">
<dep type="root">
<governor idx="0">ROOT</governor>
<dependent idx="3">annoyed</dependent>
</dep>
<dep type="nsubjpass">
<governor idx="3">annoyed</governor>
<dependent idx="1">She</dependent>
</dep>
<dep type="auxpass">
<governor idx="3">annoyed</governor>
<dependent idx="2">was</dependent>
</dep>
<dep type="punct">
<governor idx="3">annoyed</governor>
<dependent idx="4">.</dependent>
</dep>
</dependencies>
<dependencies type="collapsed-dependencies">
<dep type="root">
<governor idx="0">ROOT</governor>
<dependent idx="3">annoyed</dependent>
</dep>
<dep type="nsubjpass">
<governor idx="3">annoyed</governor>
<dependent idx="1">She</dependent>
</dep>
<dep type="auxpass">
<governor idx="3">annoyed</governor>
<dependent idx="2">was</dependent>
</dep>
<dep type="punct">
<governor idx="3">annoyed</governor>
<dependent idx="4">.</dependent>
</dep>
</dependencies>
<dependencies type="collapsed-ccprocessed-dependencies">
<dep type="root">
<governor idx="0">ROOT</governor>
<dependent idx="3">annoyed</dependent>
</dep>
<dep type="nsubjpass">
<governor idx="3">annoyed</governor>
<dependent idx="1">She</dependent>
</dep>
<dep type="auxpass">
<governor idx="3">annoyed</governor>
<dependent idx="2">was</dependent>
</dep>
<dep type="punct">
<governor idx="3">annoyed</governor>
<dependent idx="4">.</dependent>
</dep>
</dependencies>
<dependencies type="enhanced-dependencies">
<dep type="root">
<governor idx="0">ROOT</governor>
<dependent idx="3">annoyed</dependent>
</dep>
<dep type="nsubjpass">
<governor idx="3">annoyed</governor>
<dependent idx="1">She</dependent>
</dep>
<dep type="auxpass">
<governor idx="3">annoyed</governor>
<dependent idx="2">was</dependent>
</dep>
<dep type="punct">
<governor idx="3">annoyed</governor>
<dependent idx="4">.</dependent>
</dep>
</dependencies>
<dependencies type="enhanced-plus-plus-dependencies">
<dep type="root">
<governor idx="0">ROOT</governor>
<dependent idx="3">annoyed</dependent>
</dep>
<dep type="nsubjpass">
<governor idx="3">annoyed</governor>
<dependent idx="1">She</dependent>
</dep>
<dep type="auxpass">
<governor idx="3">annoyed</governor>
<dependent idx="2">was</dependent>
</dep>
<dep type="punct">
<governor idx="3">annoyed</governor>
<dependent idx="4">.</dependent>
</dep>
</dependencies>
</sentence>
<sentence id="4">
<tokens>
<token id="1">
<word>Mary</word>
<lemma>Mary</lemma>
<CharacterOffsetBegin>52</CharacterOffsetBegin>
<CharacterOffsetEnd>56</CharacterOffsetEnd>
<POS>NNP</POS>
<NER>PERSON</NER>
<Speaker>PER0</Speaker>
</token>
<token id="2">
<word>answered</word>
<lemma>answer</lemma>
<CharacterOffsetBegin>57</CharacterOffsetBegin>
<CharacterOffsetEnd>65</CharacterOffsetEnd>
<POS>VBD</POS>
<NER>O</NER>
<Speaker>PER0</Speaker>
</token>
<token id="3">
<word>the</word>
<lemma>the</lemma>
<CharacterOffsetBegin>66</CharacterOffsetBegin>
<CharacterOffsetEnd>69</CharacterOffsetEnd>
<POS>DT</POS>
<NER>O</NER>
<Speaker>PER0</Speaker>
</token>
<token id="4">
<word>phone</word>
<lemma>phone</lemma>
<CharacterOffsetBegin>70</CharacterOffsetBegin>
<CharacterOffsetEnd>75</CharacterOffsetEnd>
<POS>NN</POS>
<NER>O</NER>
<Speaker>PER0</Speaker>
</token>
<token id="5">
<word>.</word>
<lemma>.</lemma>
<CharacterOffsetBegin>75</CharacterOffsetBegin>
<CharacterOffsetEnd>76</CharacterOffsetEnd>
<POS>.</POS>
<NER>O</NER>
<Speaker>PER0</Speaker>
</token>
</tokens>
<parse>(ROOT (S (NP (NNP Mary)) (VP (VBD answered) (NP (DT the) (NN phone))) (. .))) </parse>
<dependencies type="basic-dependencies">
<dep type="root">
<governor idx="0">ROOT</governor>
<dependent idx="2">answered</dependent>
</dep>
<dep type="nsubj">
<governor idx="2">answered</governor>
<dependent idx="1">Mary</dependent>
</dep>
<dep type="det">
<governor idx="4">phone</governor>
<dependent idx="3">the</dependent>
</dep>
<dep type="dobj">
<governor idx="2">answered</governor>
<dependent idx="4">phone</dependent>
</dep>
<dep type="punct">
<governor idx="2">answered</governor>
<dependent idx="5">.</dependent>
</dep>
</dependencies>
<dependencies type="collapsed-dependencies">
<dep type="root">
<governor idx="0">ROOT</governor>
<dependent idx="2">answered</dependent>
</dep>
<dep type="nsubj">
<governor idx="2">answered</governor>
<dependent idx="1">Mary</dependent>
</dep>
<dep type="det">
<governor idx="4">phone</governor>
<dependent idx="3">the</dependent>
</dep>
<dep type="dobj">
<governor idx="2">answered</governor>
<dependent idx="4">phone</dependent>
</dep>
<dep type="punct">
<governor idx="2">answered</governor>
<dependent idx="5">.</dependent>
</dep>
</dependencies>
<dependencies type="collapsed-ccprocessed-dependencies">
<dep type="root">
<governor idx="0">ROOT</governor>
<dependent idx="2">answered</dependent>
</dep>
<dep type="nsubj">
<governor idx="2">answered</governor>
<dependent idx="1">Mary</dependent>
</dep>
<dep type="det">
<governor idx="4">phone</governor>
<dependent idx="3">the</dependent>
</dep>
<dep type="dobj">
<governor idx="2">answered</governor>
<dependent idx="4">phone</dependent>
</dep>
<dep type="punct">
<governor idx="2">answered</governor>
<dependent idx="5">.</dependent>
</dep>
</dependencies>
<dependencies type="enhanced-dependencies">
<dep type="root">
<governor idx="0">ROOT</governor>
<dependent idx="2">answered</dependent>
</dep>
<dep type="nsubj">
<governor idx="2">answered</governor>
<dependent idx="1">Mary</dependent>
</dep>
<dep type="det">
<governor idx="4">phone</governor>
<dependent idx="3">the</dependent>
</dep>
<dep type="dobj">
<governor idx="2">answered</governor>
<dependent idx="4">phone</dependent>
</dep>
<dep type="punct">
<governor idx="2">answered</governor>
<dependent idx="5">.</dependent>
</dep>
</dependencies>
<dependencies type="enhanced-plus-plus-dependencies">
<dep type="root">
<governor idx="0">ROOT</governor>
<dependent idx="2">answered</dependent>
</dep>
<dep type="nsubj">
<governor idx="2">answered</governor>
<dependent idx="1">Mary</dependent>
</dep>
<dep type="det">
<governor idx="4">phone</governor>
<dependent idx="3">the</dependent>
</dep>
<dep type="dobj">
<governor idx="2">answered</governor>
<dependent idx="4">phone</dependent>
</dep>
<dep type="punct">
<governor idx="2">answered</governor>
<dependent idx="5">.</dependent>
</dep>
</dependencies>
</sentence>
<sentence id="5">
<tokens>
<token id="1">
<word>Ann</word>
<lemma>Ann</lemma>
<CharacterOffsetBegin>77</CharacterOffsetBegin>
<CharacterOffsetEnd>80</CharacterOffsetEnd>
<POS>NNP</POS>
<NER>PERSON</NER>
<Speaker>PER0</Speaker>
</token>
<token id="2">
<word>told</word>
<lemma>tell</lemma>
<CharacterOffsetBegin>81</CharacterOffsetBegin>
<CharacterOffsetEnd>85</CharacterOffsetEnd>
<POS>VBD</POS>
<NER>O</NER>
<Speaker>PER0</Speaker>
</token>
<token id="3">
<word>the</word>
<lemma>the</lemma>
<CharacterOffsetBegin>86</CharacterOffsetBegin>
<CharacterOffsetEnd>89</CharacterOffsetEnd>
<POS>DT</POS>
<NER>O</NER>
<Speaker>PER0</Speaker>
</token>
<token id="4">
<word>good</word>
<lemma>good</lemma>
<CharacterOffsetBegin>90</CharacterOffsetBegin>
<CharacterOffsetEnd>94</CharacterOffsetEnd>
<POS>JJ</POS>
<NER>O</NER>
<Speaker>PER0</Speaker>
</token>
<token id="5">
<word>news</word>
<lemma>news</lemma>
<CharacterOffsetBegin>95</CharacterOffsetBegin>
<CharacterOffsetEnd>99</CharacterOffsetEnd>
<POS>NN</POS>
<NER>O</NER>
<Speaker>PER0</Speaker>
</token>
<token id="6">
<word>to</word>
<lemma>to</lemma>
<CharacterOffsetBegin>100</CharacterOffsetBegin>
<CharacterOffsetEnd>102</CharacterOffsetEnd>
<POS>TO</POS>
<NER>O</NER>
<Speaker>PER0</Speaker>
</token>
<token id="7">
<word>Mary</word>
<lemma>Mary</lemma>
<CharacterOffsetBegin>103</CharacterOffsetBegin>
<CharacterOffsetEnd>107</CharacterOffsetEnd>
<POS>NNP</POS>
<NER>PERSON</NER>
<Speaker>PER0</Speaker>
</token>
<token id="8">
<word>.</word>
<lemma>.</lemma>
<CharacterOffsetBegin>107</CharacterOffsetBegin>
<CharacterOffsetEnd>108</CharacterOffsetEnd>
<POS>.</POS>
<NER>O</NER>
<Speaker>PER0</Speaker>
</token>
</tokens>
<parse>(ROOT (S (NP (NNP Ann)) (VP (VBD told) (NP (DT the) (JJ good) (NN news)) (PP (TO to) (NP (NNP Mary)))) (. .))) </parse>
<dependencies type="basic-dependencies">
<dep type="root">
<governor idx="0">ROOT</governor>
<dependent idx="2">told</dependent>
</dep>
<dep type="nsubj">
<governor idx="2">told</governor>
<dependent idx="1">Ann</dependent>
</dep>
<dep type="det">
<governor idx="5">news</governor>
<dependent idx="3">the</dependent>
</dep>
<dep type="amod">
<governor idx="5">news</governor>
<dependent idx="4">good</dependent>
</dep>
<dep type="dobj">
<governor idx="2">told</governor>
<dependent idx="5">news</dependent>
</dep>
<dep type="case">
<governor idx="7">Mary</governor>
<dependent idx="6">to</dependent>
</dep>
<dep type="nmod">
<governor idx="2">told</governor>
<dependent idx="7">Mary</dependent>
</dep>
<dep type="punct">
<governor idx="2">told</governor>
<dependent idx="8">.</dependent>
</dep>
</dependencies>
<dependencies type="collapsed-dependencies">
<dep type="root">
<governor idx="0">ROOT</governor>
<dependent idx="2">told</dependent>
</dep>
<dep type="nsubj">
<governor idx="2">told</governor>
<dependent idx="1">Ann</dependent>
</dep>
<dep type="det">
<governor idx="5">news</governor>
<dependent idx="3">the</dependent>
</dep>
<dep type="amod">
<governor idx="5">news</governor>
<dependent idx="4">good</dependent>
</dep>
<dep type="dobj">
<governor idx="2">told</governor>
<dependent idx="5">news</dependent>
</dep>
<dep type="case">
<governor idx="7">Mary</governor>
<dependent idx="6">to</dependent>
</dep>
<dep type="nmod:to">
<governor idx="2">told</governor>
<dependent idx="7">Mary</dependent>
</dep>
<dep type="punct">
<governor idx="2">told</governor>
<dependent idx="8">.</dependent>
</dep>
</dependencies>
<dependencies type="collapsed-ccprocessed-dependencies">
<dep type="root">
<governor idx="0">ROOT</governor>
<dependent idx="2">told</dependent>
</dep>
<dep type="nsubj">
<governor idx="2">told</governor>
<dependent idx="1">Ann</dependent>
</dep>
<dep type="det">
<governor idx="5">news</governor>
<dependent idx="3">the</dependent>
</dep>
<dep type="amod">
<governor idx="5">news</governor>
<dependent idx="4">good</dependent>
</dep>
<dep type="dobj">
<governor idx="2">told</governor>
<dependent idx="5">news</dependent>
</dep>
<dep type="case">
<governor idx="7">Mary</governor>
<dependent idx="6">to</dependent>
</dep>
<dep type="nmod:to">
<governor idx="2">told</governor>
<dependent idx="7">Mary</dependent>
</dep>
<dep type="punct">
<governor idx="2">told</governor>
<dependent idx="8">.</dependent>
</dep>
</dependencies>
<dependencies type="enhanced-dependencies">
<dep type="root">
<governor idx="0">ROOT</governor>
<dependent idx="2">told</dependent>
</dep>
<dep type="nsubj">
<governor idx="2">told</governor>
<dependent idx="1">Ann</dependent>
</dep>
<dep type="det">
<governor idx="5">news</governor>
<dependent idx="3">the</dependent>
</dep>
<dep type="amod">
<governor idx="5">news</governor>
<dependent idx="4">good</dependent>
</dep>
<dep type="dobj">
<governor idx="2">told</governor>
<dependent idx="5">news</dependent>
</dep>
<dep type="case">
<governor idx="7">Mary</governor>
<dependent idx="6">to</dependent>
</dep>
<dep type="nmod:to">
<governor idx="2">told</governor>
<dependent idx="7">Mary</dependent>
</dep>
<dep type="punct">
<governor idx="2">told</governor>
<dependent idx="8">.</dependent>
</dep>
</dependencies>
<dependencies type="enhanced-plus-plus-dependencies">
<dep type="root">
<governor idx="0">ROOT</governor>
<dependent idx="2">told</dependent>
</dep>
<dep type="nsubj">
<governor idx="2">told</governor>
<dependent idx="1">Ann</dependent>
</dep>
<dep type="det">
<governor idx="5">news</governor>
<dependent idx="3">the</dependent>
</dep>
<dep type="amod">
<governor idx="5">news</governor>
<dependent idx="4">good</dependent>
</dep>
<dep type="dobj">
<governor idx="2">told</governor>
<dependent idx="5">news</dependent>
</dep>
<dep type="case">
<governor idx="7">Mary</governor>
<dependent idx="6">to</dependent>
</dep>
<dep type="nmod:to">
<governor idx="2">told</governor>
<dependent idx="7">Mary</dependent>
</dep>
<dep type="punct">
<governor idx="2">told</governor>
<dependent idx="8">.</dependent>
</dep>
</dependencies>
</sentence>
</sentences>
<coreference>
<coreference>
<mention representative="true">
<sentence>1</sentence>
<start>1</start>
<end>2</end>
<head>1</head>
<text>Mary</text>
</mention>
<mention>
<sentence>2</sentence>
<start>1</start>
<end>2</end>
<head>1</head>
<text>Her</text>
</mention>
<mention>
<sentence>3</sentence>
<start>1</start>
<end>2</end>
<head>1</head>
<text>She</text>
</mention>
<mention>
<sentence>4</sentence>
<start>1</start>
<end>2</end>
<head>1</head>
<text>Mary</text>
</mention>
<mention>
<sentence>5</sentence>
<start>7</start>
<end>8</end>
<head>7</head>
<text>Mary</text>
</mention>
</coreference>
<coreference>
<mention representative="true">
<sentence>2</sentence>
<start>1</start>
<end>3</end>
<head>2</head>
<text>Her phone</text>
</mention>
<mention>
<sentence>4</sentence>
<start>3</start>
<end>5</end>
<head>4</head>
<text>the phone</text>
</mention>
</coreference>
</coreference>
</document>
</root>
Display the source blob
Display the rendered blob
Raw
story_comprehension.ipynb
{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Automated story comprehension #\n",
"This notebook shows how to extract background knowledge and relations from a story text. To this end, we use Stanford's NLP parser available to download [here](https://nlp.stanford.edu/software/lex-parser.html#Download).\n",
"\n",
"We parsed the following story (`story.txt` file):\n",
"```\n",
"Mary was sleeping.\n",
"Her phone rang.\n",
"She was annoyed.\n",
"Mary answered the phone.\n",
"Ann told the good news to Mary.\n",
"```\n",
"\n",
"with\n",
"```\n",
"java -cp \"*\" -Xmx2g edu.stanford.nlp.pipeline.StanfordCoreNLP -annotators tokenize,ssplit,pos,lemma,ner,parse,dcoref -file story.txt\n",
"```\n",
"This resulted in an xml file `story.txt.xml` that contains all the necessary data to extract relevant information from the story.\n",
"\n",
"The next step is to extract this information and represent it as Prolog facts. To this end, we wrote the Python code included below."
]
},
{
"cell_type": "code",
"execution_count": 1,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
"# Import libraries\n",
"from xml.etree.ElementTree import parse as xml_parse\n",
"from pprint import pprint"
]
},
{
"cell_type": "code",
"execution_count": 2,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
"# Read in the xml file with the parsed story\n",
"e = xml_parse(\"story.txt.xml\").getroot()"
]
},
{
"cell_type": "code",
"execution_count": 3,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
"# path_tags is a list of tuples: [(tag, attribute, attribute_id), ...]\n",
"def get_xml_struct(root, path_tags):\n",
" current = root\n",
" for tag, attribute, attribute_value in path_tags:\n",
" for c in current:\n",
" if c.tag == tag:\n",
" if attribute is not None and attribute_value is not None:\n",
" if c.attrib.get(attribute) == attribute_value:\n",
" current = c\n",
" break\n",
" else:\n",
" current = c\n",
" break\n",
" return current\n",
"\n",
"def get_attrib_value(root, dep_type, dependent=True):\n",
" identifiers = {}\n",
" for c in root:\n",
" if c.tag == \"dep\":\n",
" if c.attrib.get(\"type\", \"\") == dep_type:\n",
" for d in c:\n",
" if dependent and d.tag == \"dependent\":\n",
" return d.attrib.get(\"idx\", \"\")\n",
" identifiers[d.tag] = d.attrib.get(\"idx\", \"\")\n",
" return identifiers\n",
"\n",
"def get_token_lemma(tokens, token_id):\n",
" for t in tokens:\n",
" if t.tag == \"token\" and t.attrib.get(\"id\", \"\") == token_id:\n",
" for i in t:\n",
" if i.tag == \"lemma\":\n",
" return i.text\n",
" return \"none\""
]
},
{
"cell_type": "code",
"execution_count": 4,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Sentence ids:\n",
"['1', '2', '3', '4', '5']\n"
]
}
],
"source": [
"# get sentences ids\n",
"sentence_ids = []\n",
"t = get_xml_struct(e, [(\"document\", None, None), (\"sentences\", None, None)])\n",
"for c in t:\n",
" id = c.attrib.get(\"id\")\n",
" if id is not None:\n",
" sentence_ids.append(id)\n",
"\n",
"print \"Sentence ids:\"\n",
"print sentence_ids"
]
},
{
"cell_type": "code",
"execution_count": 5,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"{'1': {'dobj': 'none', 'root': 'sleep', 'nmod': 'none', 'nsubj': 'mary'}, '3': {'dobj': 'none', 'root': 'annoy', 'nmod': 'none', 'nsubj': 'she'}, '2': {'dobj': 'none', 'root': 'ring', 'nmod': 'none', 'nsubj': 'phone'}, '5': {'dobj': 'news', 'root': 'tell', 'nmod': 'mary', 'nsubj': 'ann'}, '4': {'dobj': 'phone', 'root': 'answer', 'nmod': 'none', 'nsubj': 'mary'}}\n"
]
}
],
"source": [
"sentence_dependencies = {}\n",
"for sid in sentence_ids:\n",
" sentence_structure = {\"root\":\"none\", \"nsubj\":\"none\", \"dobj\":\"none\", \"nmod\":\"none\"}\n",
" # get basic-dependencies\n",
" basicd = get_xml_struct(e, [(\"document\", None, None), (\"sentences\", None, None), \\\n",
" (\"sentence\", \"id\", sid), (\"dependencies\", \"type\", \"basic-dependencies\")\n",
" ]\n",
" )\n",
" # get lemmas\n",
" tokens = get_xml_struct(e, [(\"document\", None, None), (\"sentences\", None, None), \\\n",
" (\"sentence\", \"id\", sid), (\"tokens\", None, None)\n",
" ]\n",
" )\n",
"\n",
" for i in [\"nsubj\", \"nsubjpass\"]:\n",
" it = get_token_lemma(tokens, get_attrib_value(basicd, i)).lower()\n",
" if it != \"none\":\n",
" sentence_structure[\"nsubj\"] = it\n",
" for i in [\"root\", \"dobj\", \"nmod\"]:\n",
" it = get_token_lemma(tokens, get_attrib_value(basicd, i)).lower()\n",
" if it != \"none\":\n",
" sentence_structure[i] = it\n",
" sentence_dependencies[sid] = sentence_structure\n",
"\n",
"print sentence_dependencies"
]
},
{
"cell_type": "code",
"execution_count": 6,
"metadata": {},
"outputs": [],
"source": [
"predicates_string = \"\"\n",
"for i in sorted(sentence_dependencies.keys()):\n",
" si = sentence_dependencies[i]\n",
" si[\"time\"] = i\n",
" \n",
" x = \"s(1) :: %(root)s(%(nsubj)s, %(dobj)s, %(nmod)s) at %(time)s.\" % si\n",
" \n",
" x = x.lower()\n",
" predicates_string += x + \"\\n\""
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Processing output ##\n",
"`xml` parsing gives us the following logical representation of the story:"
]
},
{
"cell_type": "code",
"execution_count": 7,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"s(1) :: sleep(mary, none, none) at 1.\n",
"s(1) :: ring(phone, none, none) at 2.\n",
"s(1) :: annoy(she, none, none) at 3.\n",
"s(1) :: answer(mary, phone, none) at 4.\n",
"s(1) :: tell(ann, news, mary) at 5.\n"
]
}
],
"source": [
"print predicates_string,"
]
},
{
"cell_type": "code",
"execution_count": 8,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
"# Save the rules to the `extracted_knowledge.pl` file\n",
"with open(\"extracted_knowledge.pl\", \"w\") as rules_file:\n",
" rules_file.write(predicates_string)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": []
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 2",
"language": "python",
"name": "python2"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 2
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython2",
"version": "2.7.13"
}
},
"nbformat": 4,
"nbformat_minor": 2
}
Raw
story_extracted_knowledge.pl
s(1) :: sleep(mary, none, none) at 1.
s(1) :: ring(phone, none, none) at 2.
s(1) :: annoy(she, none, none) at 3.
s(1) :: answer(mary, phone, none) at 4.
s(1) :: tell(ann, news, mary) at 5.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment