KYOTO Logo
Knowledge Yielding Ontologies
for Transition-based Organization
  • Increase font size
  • Default font size
  • Decrease font size
Evaluation

Evaluation - Evaluation of semantic search

E-mail Print PDF
Article Index
Evaluation
Evaluation of Module
Evaluation of facts
Evaluation of semantic search
Evaluation of the knowledge editor
All Pages
Evaluation of semantic search

 

The facts extracted by the Kybots are indexed and searchable in a cross-lingual semantic index. To evaluate the usefulness of these facts, we carried out a benchmark evalauation of the search indexes and an end-user evaluation. In this evaluation, we compare the search on Kybot facts with a standard text search solution. The benchmark evaluation checks to what extend the kybot facts represent all the information in the database. We compared the recall of the semantic search in the Kybot system with the recall of a standard text search. This showed that 35% of the coverage of a standard system is achieved. The extraction of the kybots is thus a good step towards a full text representation but still there is room for improvement. The details of the benchmark can be found in the KYOTO deliverable D09.5.

For the end-user evaluation, we had 21 users (students and environmentalists) that had to find answers to questions using the standard retrieval system, a mash-up fact retrieval system and the semantic search in the kybot facts. The evaluation showed no significant difference in quality and performance but the users had major difficulties understanding and appreciating the innovative features of the semantic search in the fact index. Some less-conservative users on the other hand exactly liked these features. The details are discussed in the KYOTO deliverable D09.6.

 



 

ICT-211423 - 2008 © Kyoto Consortium