Friday, 20 January 2017

An Algorithm to determine the context


The purpose of this note is to propose an algorithm to automatically detect the context of a given text.

Context means "1. The part of a text or statement that surrounds a particular word or passage and determines its meaning.
2. The circumstances in which an event occurs; a setting." according to Free Dictionary.

I used 3 texts as example for the experiment.  These are fairytales named "The Princess and the Pea"
and "The Little Match-Seller" of Hans Christian Andersen, "The Frog Prince" of Grimm Brothers.

My algorithm is as follows:

1- Determine each sentence that each word is used in:

Little===>0,1,3,5,5,8,12,14,18,20,27,32,35,37,39,40,40
Match-Seller===>0
It===>0,2,4,4,10,13,16,16,17,17,18,18,19,20,25,25,30,31,34
was===>0,0,10,10,13,19,20,26,27,30,35,35,38,39,41
terribly===>0

2-Determine which two words are used in the same sentences of the text:
across==>3,27     (PS. ARS =across is used in the 3. and 27. sentences)
ctx(0)= 3           (PS. ARS = ctx stands for shared context)
host=60 ctxList=(3,27) hostCTX=3 across .............street (3)...............dest=61
host=60 ctxList=(3,27) hostCTX=3 across .............avoid (3)...............dest=62
host=60 ctxList=(3,27) hostCTX=3 across .............two (3,11)...............dest=63
host=60 ctxList=(3,27) hostCTX=3 across .............carriages (3)...............dest=64
host=60 ctxList=(3,27) hostCTX=3 across .............rolling (3)...............dest=65
host=60 ctxList=(3,27) hostCTX=3 across .............along (3,8)...............dest=66
host=60 ctxList=(3,27) hostCTX=3 across .............terrible (3)...............dest=68
host=60 ctxList=(3,27) hostCTX=3 across .............rate (3)...............dest=69
ctx(1)= 27
host=60 ctxList=(3,27) hostCTX=27 across .............goose (10,26,27,37)...............dest=134
host=60 ctxList=(3,27) hostCTX=27 across .............down (11,27,31)...............dest=147
host=60 ctxList=(3,27) hostCTX=27 across .............wonderful (19,27)...............dest=211
host=60 ctxList=(3,27) hostCTX=27 across .............its (25,27)...............dest=234
host=60 ctxList=(3,27) hostCTX=27 across .............still (27,41)...............dest=255
host=60 ctxList=(3,27) hostCTX=27 across .............more (27,30)...............dest=256
host=60 ctxList=(3,27) hostCTX=27 across .............jumped (27)...............dest=257
host=60 ctxList=(3,27) hostCTX=27 across .............dish (27)...............dest=258
host=60 ctxList=(3,27) hostCTX=27 across .............waddled (27)...............dest=259
host=60 ctxList=(3,27) hostCTX=27 across .............floor (27)...............dest=260
host=60 ctxList=(3,27) hostCTX=27 across .............knife (27)...............dest=261
host=60 ctxList=(3,27) hostCTX=27 across .............fork (27)...............dest=262
host=60 ctxList=(3,27) hostCTX=27 across .............breast (27)...............dest=263

3- Determine the sentences that are common for any two words:
------------------------>contextIDLIST
===============>>>1=Little 6=cold sentence=The Little Match-Seller

It was terribly cold and nearly dark on the last evening of the old year, and the snow was falling fast.

===============>>>1=Little 11=last sentence=The Little Match-Seller

It was terribly cold and nearly dark on the last evening of the old year, and the snow was falling fast.
===============>>>1=Little 12=evening sentence=The Little Match-Seller

...
===============>>>1=Little 6=cold sentence= In the cold and the darkness, a poor little girl, with bare head and naked feet, roamed through the streets.
===============>>>1=Little 22=poor sentence= In the cold and the darkness, a poor little girl, with bare head and naked feet, roamed through the streets.
===============>>>1=Little 23=girl sentence= In the cold and the darkness, a poor little girl, with bare head and naked feet, roamed through the streets.
===============>>>1=Little 27=naked sentence= In the cold and the darkness, a poor little girl, with bare head and naked feet, roamed through the streets.
===============>>>1=Little 28=feet sentence= In the cold and the darkness, a poor little girl, with bare head and naked feet, roamed through the streets.
...

4-Count the freq of two words(word couples) that are used in/per the text.

------------------------>countFreqContextIdentifications

freq=6===============>1=Little 6=cold sentence=0
The Little Match-Seller

It was terribly cold and nearly dark on the last evening of the old year, and the snow was falling fast.

freq=6===============>1=Little 6=cold sentence=1
 In the cold and the darkness, a poor little girl, with bare head and naked feet, roamed through the streets.

freq=3===============>1=Little 22=poor sentence=1
 In the cold and the darkness, a poor little girl, with bare head and naked feet, roamed through the streets.

freq=5===============>1=Little 23=girl sentence=1
 In the cold and the darkness, a poor little girl, with bare head and naked feet, roamed through the streets.

The output displays the related sentences together with the frequency of the word couple.

The results for the three chosen texts, fairytales follows:

The Little Match-Seller
***********************
freq=6===============>1=Little 6=cold sentence=0
Little===>0,1,3,5,5,8,12,14,18,20,27,32,35,37,39,40,40
cold===>0,1,5,8,12,13,14,28,39
freq=5===============>1=Little 23=girl sentence=1
girl===>1,5,20,27,35,39
freq=3===============>1=Little 22=poor
freq=3===============>1=Little 28=feet
freq=3===============>1=Little 70=One
freq=3===============>183=match 192=wall

The text of the fairy tales follows:

0===>The Little Match-Seller

It was terribly cold and nearly dark on the last evening of the old year, and the snow was falling fast.<===
1===> In the cold and the darkness, a poor little girl, with bare head and naked feet, roamed through the streets.<===
2===> It is true she had on a pair of slippers when she left home, but they were not of much use.<===
3===> They were very large, so large, indeed, that they had belonged to her mother, and the poor little creature had lost them in running across the street to avoid two carriages that were rolling along at a terrible rate.<===
4===> One of the slippers she could not find, and a boy seized upon the other and ran away with it, saying that he could use it as a cradle, when he had children of his own.<===
5===> So the little girl went on with her little naked feet, which were quite red and blue with the cold.<===
6===>

     In an old apron she carried a number of matches, and had a bundle of them in her hands.<===
7===> No one had bought anything of her the whole day, nor had any one given here even a penny.<===
8===> Shivering with cold and hunger, she crept along; poor little child, she looked the picture of misery.<===
9===> The snowflakes fell on her long, fair hair, which hung in curls on her shoulders, but she regarded them not.<===
10===>

     Lights were shining from every window, and there was a savory smell of roast goose, for it was New-year's eve - yes, she remembered that.<===
11===> In a corner, between two houses, one of which projected beyond the other, she sank down and huddled herself together.<===
12===> She had drawn her little feet under her, but she could not keep off the cold; and she dared not go home, for she had sold no matches, and could not take home even a penny of money.<===
13===> Her father would certainly beat her; besides, it was almost as cold at home as here, for they had only the roof to cover them, through which the wind howled, although the largest holes had been stopped up with straw and rags.<===
14===>

     Her little hands were almost frozen with the cold.<===
15===> Ah!<===
16===> perhaps a burning match might be some good, if she could draw it from the bundle and strike it against the wall, just to warm her fingers.<===
17===>

     She drew one out - "scratch!" how it sputtered as it burnt!<===
18===> It gave a warm, bright light, like a little candle, as she held her hand over it.<===
19===> It was really a wonderful light.<===
20===> It seemed to the little girl that she was sitting by a large iron stove, with polished brass feet and a brass ornament.<===
21===> How the fire burned!<===
22===> and seemed so beautifully warm that the child stretched out her feet as if to warm them, when, lo!<===
23===> the flame of the match went out, the stove vanished, and she had only the remains of the half-burnt match in her hand.<===
24===>

     She rubbed another match on the wall.<===
25===> It burst into a flame, and where its light fell upon the wall it became as transparent as a veil, and she could see into the room.<===
26===> The table was covered with a snowy white table-cloth, on which stood a splendid dinner service, and a steaming roast goose, stuffed with apples and dried plums.<===
27===> And what was still more wonderful, the goose jumped down from the dish and waddled across the floor, with a knife and fork in its breast, to the little girl.<===
28===> Then the match went out, and there remained nothing but the thick, damp, cold wall before her.<===
29===>

     She lighted another match, and then she found herself sitting under a beautiful Christmas-tree.<===
30===> It was larger and more beautifully decorated than the one which she had seen through the glass door at the rich merchant's.<===
31===> Thousands of tapers were burning upon the green branches, and colored pictures, like those she had seen in the show-windows, looked down upon it all.<===
32===> The little one stretched out her hand towards them, and the match went out.<===
33===>

     The Christmas lights rose higher and higher, till they looked to her like the stars in the sky.<===
34===> Then she saw a star fall, leaving behind it a bright streak of fire.<===
35===> "Some one is dying," thought the little girl, for her old grandmother, the only one who had ever loved her, and who was now dead, had told her that when a star falls, a soul was going up to God.<===
36===>

     She again rubbed a match on the wall, and the light shone round her; in the brightness stood her old grandmother, clear and shining, yet mild and loving in her appearance.<===
37===>

     "Grandmother," cried the little one, "O take me with you; I know you will go away when the match burns out; you will vanish like the warm stove, the roast goose, and the large, glorious Christmas-tree."

     And she made haste to light the whole bundle of matches, for she wished to keep her grandmother there.<===
38===> And the matches glowed with a light that was brighter than the noon-day, and her grandmother had never appeared so large or so beautiful.<===
39===> She took the little girl in her arms, and they both flew upwards in brightness and joy far above the earth, where there was neither cold nor hunger nor pain, for they were with God.<===
40===>

     In the dawn of morning there lay the poor little one, with pale cheeks and smiling mouth, leaning against the wall; she had been frozen to death on the last evening of the year; and the New-year's sun rose and shone upon a little corpse!<===
41===> The child still sat, in the stiffness of death, holding the matches in her hand, one bundle of which was burnt.<===
42===>

     "She tried to warm herself," said some.<===
43===>

     No one imagined what beautiful things she had seen, nor into what glory she had entered with her grandmother, on New-year's day.<===

The results for the other test texts:
The Princess and the Pea
************************
freq=5===============>4=Princess 23=real l sentence=0
Once upon a time there was a prince who wanted to marry a princess; but she would have to be a real princess.
freq=5===============>4=Princess 23=real sentence=4
 So he came home again and was sad, for he would have liked very much to have a real princess.
freq=5===============>4=Princess 23=real sentence=11
 And yet she said that she was a real princess.
freq=5===============>4=Princess 23=real sentence=17
 "I have scarcely closed my eyes all night. Heaven only knows what was in the bed, but I was lying on something hard, so that I am black and blue all over my body. It's horrible!"

     Now they knew that she was a real princess because she had felt the pea right through the twenty mattresses and the twenty eider-down beds.
freq=5===============>4=Princess 23=real sentence=18


     Nobody but a real princess could be as sensitive as that.
freq=5===============>4=Princess 23=real sentence=19


     So the prince took her for his wife, for now he knew that he had a real princess; and the pea was put in the museum, where it may still be seen, if no one has stolen it.

The Frog Prince
***************
freq=11===============>3=Frog 10=princess sentence=5
     Whilst she was speaking, a frog put its head out of the water, and said, "Princess, why do you weep so bitterly?"
freq=11===============>3=Frog 10=princess sentence=7
     "What nonsense," thought the princess, "this silly frog is talking! He can never even get out of the spring to visit me, though he may be able to get my ball for me, and therefore I will tell him he shall have what he asks".
freq=11===============>3=Frog 10=princess sentence=10
     As soon as the young princess saw her ball, she ran to pick it up; and she was so overjoyed to have it in her hand again, that she never thought of the frog, but ran home with it as fast as she could.
freq=11===============>3=Frog 10=princess sentence=11
     The frog called after her, "Stay, princess, and take me with you as you said,"