Advanced Information Systems for sharing information and Knowledge about the Environment
Home
Organizers
Call for abstracts
Event extraction task
Invited speakers
Program
Registration
Venue

Task evaluation

We developed a representation of events with relations to participants that is independent of the KYOTO system. Event-participant relations are presented as simple triplets, which consists of:
  • a relation;
  • a list of word token identifiers that express the event;
  • a list of word token identifiers that express a participant.
Word tokens identifiers are taken from the representation of the text documents in the KYOTO Annotation Format or KAF (see www.kyoto-project.eu and the documentation below for more details). Typically, a single event has multiple participants. Each participant relation is converted to a separate triplet.
-------------------------------
Go to : Appendix I: Results per test file
Go to : Appendix II: Code book for annotating KYOTO events
-------------------------------
An example file with annotations and triplets was provided in the task-data. In this file, we present the following triplet:
<triplet id="76" relation="patient">
	<eventids>
		<event id="w1349"/> 
		<event id="w1350"/>
	</eventids>
	<participantids>
		<participant id="w1352"/>
		<participant id="w1353"/>
		<participant id="w1354"/>
		<participant id="w1355"/> 
		<participant id="w1356"/> 
		<participant id="w1357"/> 
		<participant id="w1358"/>
		<participant id="w1359"/>
	</participantids>
</triplet>
The tokens in the triplet refer to the following segment in the corresponding KAF file:
  <wf wid="w1348" sent="59" page="5">the</wf>
  <wf wid="w1349" sent="59" page="5">long-term</wf>
  <wf wid="w1350" sent="59" page="5">improvement</wf>
  <wf wid="w1351" sent="59" page="5">of</wf>
  <wf wid="w1352" sent="59" page="5">the</wf>
  <wf wid="w1353" sent="59" page="5">quality</wf>
  <wf wid="w1354" sent="59" page="5">of</wf>
  <wf wid="w1355" sent="59" page="5">the</wf>
  <wf wid="w1356" sent="59" page="5">Bay</wf>
  <wf wid="w1357" sent="59" page="5">and</wf>
  <wf wid="w1358" sent="59" page="5">its</wf>
  <wf wid="w1359" sent="59" page="5">rivers</wf>
We provided 3 KAF files for evaluation by November 23rd . Participants were asked to extract events from these files that are relevant for the environment domain. They could participate in this task by processing the test documents (represented in the KyotoAnnotationFormat) and converting the output to the triplet format. The results had to be delivered by December 15th.
We created an annotation tool with which relations can be encoded for any text as a gold-standard. A PhD students trained in semantics but not involved in the KYOTO project annotated the 3 files manually. The annotation guidelines are given in the appendix. The gold-standard (GS) consists of 3 documents for which 15 different semantic relations have been used. The most dominant relations are done-by, patient and has-state. See table-2 for an overview of the relations encoded. In total 256 triplets have been encoded in the gold-standard for the 3 documents on environmental news.
The evaluation module was made available as well. It calculates precision, recall and F-measure for any file with triplets in relation to a gold-standard triplets. The results of each test file are merged using micro-average calculation:
  • Recall R = Nr. correct/Nr. of gold-standard triplets
  • Precision P = Nr. correct/Nr. of system triplets
  • F-measure = 2*(P.R/P+R)
When evaluating, we only consider system triplets that propose event-identifiers that match the event-identifiers of gold-standard. Furthermore, event-identifiers and participant-identifiers do not need to match fully. A partial match of one identifier is sufficient for a match. Duplicate triplets are only counted once and we calculate the average number of event-identifiers and participant-identifiers for each triplet file. The latter is needed to prevent systems from including all identifiers of the file in a triplet so that it always matches. None of the participating systems used extensively identifiers to manipulate the evaluation.
Two groups outside KYOTO participated in the task (AST from Taiwan and KAIST from Korea). KYOTO participated with a generic system that was not adapted to the domain and we used a baseline system without semantic processing. The results are shown in the next table:
TABLE 1: Overall results KYOTO Event mining task
As a baseline, we created a patient relation between any constituent in a sentence, without any semantic processing. This generated 14,902 triplets of which 1,815 triplets overlap with the events encoded in the gold-standard (triplets in scope).
The KYOTO system has the highest F-measure (combining recall with precision): 26%. The KAIST system has the highest precision (57%) and KYOTO has the highest recall (23%). KAIST limited itself to only done-by and patient relations using the subject and object relations with verbs. Their system only considers clear structures that can be interpreted with higher precision. KAIST using semantic processing to choose the appropriate role for subjects and objects. KYOTO uses a similar semantic processing but considers all possible structures and relations. Likewise, KYOTO tries to represent all the implications encoded in the text. Table-2 gives an overview of the results for each relation.
TABLE 2: Results separated by relation
Another interesting aspect of the results are the interpretations of simple-cause-of, done-by, patient and result-of. The PhD that created the gold-standard is not familiar with the semantic processing of KYOTO nor with the ontology. As a trained linguist, she followed subject and object relations more closely to choose between done-by and patient, more similar to the KAIST style of interpretation. So "water flows" is coded by both as done-by but by KYOTO as a patient since water cannot control the process. Furthermore, the PhD used done-by also for processes in subject position, which according to KYOTO are simple-cause-of relations. So the style of coding of the PhD is closer to the way KAIST is extracting relations.
To see the effect of the choice of relation, we re-evaluated the output ignoring the relation, i.e. the relation was always considered correct. The results are shown in Table-3.
TABLE 3: Overall results while ignoring the relations
Obviously, all systems now perform better but the difference in F-measure between KAIST and KYOTO doubled. Remarkably this is due to the increase in recall rather than precision.
To solve the subject-object bias of relations, it would be better to use role-features rather than role-labels, e.g.:
Features
SAs self affected by the change +/-
CCs controlling the change +/-
OAs other affecting by change +/-

water flowss: SA+, CC-, OA-
water intrudess: SA+, CC-, OA+
fish eat algaes: SA+, CC+, OA+
algea kill plants: SA-, CC-, OA+
man kills fishs: SA-, CC+, OA+
This gets closer to Dowty's prototype agents and prototype patients but exactly makes explicit the implications of the roles.
All the evaluation data can be downloaded:s

Appendix I: Results per test file


Appendix II: Code book for annotating KYOTO events

1. Introduction
This documents describes the annotation process for creating a gold-standard annotation for the KYOTO project (http://www.kyoto-project.eu). This gold-standard is used to evaluate the system output generated by KYOTO. The KYOTO system extracts events from text, including their participants and time and place indicators. Participants are connected to events through role-relations. A fixed set of role relations is used in KYOTO, which is a subset of the relations defined in DOLCE-Lite. We will first introduce the annotation tool. More elaborate documentation is included in the package. Next we describe the annotation procedure through a number of examples and cases.


2. Annotation tool
We developed an annotation tool that can read a KAF annotated file and allows to assign any set of tags to the word -tokens. The tags can be adapted and organized hierarchically. The annotation tool can be downloaded from:https://kyoto.let.vu.nl/~kyoto/files/kafannotator/kafannotator_v.0.1.zip
Documentation is included in the package. The annotator is a stand-alone Java program that runs on any platform. It reads a KAF file and displays the text in a column with 6 subsequent levels for annotation. Any set of tags can be created to annotate word-tokens. The next screen dump shows a fragment of the annotation for creating the gold-standard triplets in KYOTO. The first column shows the token identifiers, the second column shows the word tokens and the third column the terms of word lemmas. Column four shows the part-of-speech and column 5 the synset associated with the lemma (only shown if there is a single synset).

The Tag1, Tag3 and Tag4 columns are used to assign tags. Tag2, Tag4 and Tag6 columns are used to assign an EVENT-SCOPE tag to tags that belong to the same event. To connect tagged tokens to the same EVENT-SCOPE, you should select all the rows that are tagged in the previous levels and assign the EVENT-SCOPE tag these rows. So for Tag1, assign EVENT-SCOPE in Tag2, Tag3 use EVENT-SCOPE in Tag4 and for Tag5 use EVENT-SCOPE in Tag6.
As can be seen in this example, word-tokens can get up to 3 tags that belong to 3 different EVENT-SCOPE. These 3 levels have shown to be sufficient for encoding almost all possible relations. A configuration file can be used to indicate which tags represent an event in the triplets and which tag is used to bind tags together. All other tags are used to label participants. For the above annotation, EVENT-SCOPE in Tag2, Tag4 and TAG6 is used to bind tags and EVENT in TAG1, TAG3 and TAG5 are used for the event identifiers in the triplets. All other tags in TAG1, TAG3 and TAG5 indicate participants. The tags for the participants are used to represent the relations between the event and the participant in each triplet.
The tag set for KYOTO is stored in: resources/KybotTagSet.txt. Load the tag set from the menu File/Load TAG set. The full tag set is shown below. The tags are subdivided for relations that apply to endurants (physical objects and substances) and perdurants (processes and states). The relations are a subset of the relations defined in the KYOTO ontology, which is an extension of the DOLCE-Lite ontology:

Relations for endurants (objects and substances)

endurant#patient

Objects or substances (endurants) that undergo a change

endurant#done-by

Objects or substances (endurants) that do the change

endurant#instrument

Objects or substances (endurants) used as an instrument to make a change

endurant#use-of

Objects or substances (endurants) used as a resource for making a change

endurant#participant

Objects or substances (endurants) somehow involved (super class of the previous roles)

endurant#has-state

Object or substances to which the state expressed as an EVENT applies, for static EVENTS only

endurant#motion#path-of

Object or substance that is the path along which a motion takes place

endurant#motion#destination-of

Object or substance that is the target of a motion

endurant#motion#source-of

Object or substance from which a motion departed

endurant#generic-location

Object or substance that represents a location at which an event takes place

endurant#product-of

Object or substance that comes into existence through an event

endurant#part-of

Object or substance that is part of another object or substance

Relations for perdurants

perdurant#simple-cause-of

Process (perdurant) that leads to or causes another process or state

perdurant#result-of

Process or state that is the result of another process or state

perdurant#state-of

State that applies to a process (not that states that apply to an object are represented as EVENT)

perdurant#purpose-of

anything that represents the purpose of an event or object

perdurant#goal-of

results of events that are intentional

LOCATION

named entity location associated with an EVENT

TIME

named entity time associated with an EVENT

The annotation is saved in a separate file and can be loaded on top of a KAF file that is loaded. The tag file can be converted to a triple file at any moment using the triplet export menu below the file menu. The first time you save the tagging (File/Save tagging), the program will ask for a file name to save the data. It is good practice to select the original KAF file and extend it with ".tag" in the same directory, e.g. data/myfile.kaf and data/myfile.kaf.tag. In that way, the tag file is stored with the KAF file to which it belongs. When loading a tag file (File/Load Tag file), the program looks in the same location as the KAF file and you can find the corresponding tag file next to it.


3 How to annotate word tokens
Sentences in KAF are marked in the table with different colours. Within a sentence, we can find more than one event or property that is expressed involving one or more participants. Furthermore, every event and property can be connected to an explicit LOCATION or TIME (which should be a named-entity, i.e. a place known in GeoNames or a specific date). These can be named entities in the same sentence or in preceding or following sentences. So usually, the event/property and participants are within the same sentence but TIME and LOCATION can be within or outside the sentence.
The general procedure is as follows:
  1. Select word-tokens that denote a event and tag the tokens as EVENT. These can be nouns, verbs or adjectives;
  2. Select word-tokens that denote a participant in the EVENT: these can be nouns or verbs. Assign a tag that relates the participant to the EVENT. Use the endurant tags for participants that are endurants and use perdurant tags for participants that are perdurants;
  3. Scan the sentence and if necessary to surrounding sentences for named-entities denoting TIME or LOCATIONs. Note that these have to be instances, e.g. New York, and not generic locations such as "near the coast". The latter are related as a participant with generic-location relation.
The EVENT can be a dynamic process expressed by a verb or a noun, e.g. "increase" or "growth", but it can also be a static property that is expressed by an adjective or noun: e.g. "small fish", "poor water quality":
The
population - patient
increased/grew - EVENT

The
increasing/growing - EVENT
population - patient

The
increase/growth - EVENT
of
the
population - patient

Population - patient
increase/growth - EVENT

small - EVENT
fish - has-state

poor - EVENT
water quality - has-state
Also the static properties are thus tagged as EVENT.
There is one peculiarity of the triplet representation. Since only an event can be the element to which another element is related, endurant-to-endurant relations need to be represented as event-participant relations. This only applies to the part-of relation. In this case the part is the participant and the whole to which it belongs needs to be encoded as an event. Example:
upper area - part-of
of
the bay - EVENT
It is also possible to assign properties or states to perdurants. In that case the perdurant is the EVENT and the property or state is related to the EVENT as state-of. Example:
algae - done-by
completely - state-of
block - EVENT
sunlight - patient
The important rule is: THINK SMALL! Events and participants can be expressed by small structures. Furthermore, the same word-tokens can be related to multiple events/properties and small events can be chained together, e.g.:
the pollution - EVENT simple-cause-of
of
the bay - patient
by rain - done-by
causes
fish - patient
kills - EVENT
In this examples, we have two events each with their own scope and participants. The first event is also linked to the second through a simple-cause-of or result-of relation. The details of the first event are not represented in the second event.