Project Goal
The primary goal of the project was to verbalise OWL 2 DL ontologies into Afrikaans sentences, using the grammar based approach. This was achieved by creating a system to verbalise ontologies into Afrikaans, leveraging the capabilities of Grammatical Framework and the OWL API.
Design
This sections is a brief outline of how the grammar based approach was implemented.
The system was developed in an iterative manner. The goal of each iteration was to produce a complete horizontal view of the system, this means that each iteration was able to read in an ontology and output some kind of verbalisation. Each iteration then added extra features or included a major refactor, examples of this include adding additional axioms, adding the translation functionality and making pre-processing more robust.
Every iteration resulted in a program which worked completely in a horizontal manner, even if all the functionality wasn’t available yet. An agile approach was deemed necessary due to the evolving nature of the project and necessity for responsiveness to change. It also allowed for multiple informal evaluations to take place in order to increase the overall quality of the final system.
The complete commit history is available on github.
Figure 1: Architecture diagram showing various parts of system.
The various parts of the system are discussed in more depth in the paper. The major parts are: reading in the input file (ontology) by integrating with OWL api, pre-processing the input and then generating the gf grammar file necessary to verbalise the sentences. These are then verbalised using grammatical framework.
Results & Discussion
Results
2 Evaluations were performed, these are discussed in depth in the report. The final evaluation will be discussed below in order to make sense of the results. The final evaluation consisted of comparing the two approaches based on how well they verbalised 6 ontologies. The ontologies used are available under the downloads page, or here. First, the Grammar based approach system was used to verbalise the ontologies, Table 1 summarises the results for this, while Table 2 summarises the results for the template based approach.
Table 1: Table showing final evaluation results for the Grammar based approach
Table 2: Table showing final evaluation results for the Template based approach
The grammar based approach was outperformed by the template based approach in terms of axiom coverage for 3/6 test ontologies. It was also slower for 5/6 of the ontologies.
The results show that the template based approach performed better in terms of axiom coverage (97.1% vs 93.32%) and time taken (2.01s vs 2.32s). than the grammar based approach.
The template and grammar based approach had the same quality of verbalisations for AfricanWildlife. University and Wine. The Template based approach scored better in the ComputerScience, Pizza and Stuff ontologies, which lead to an average quality of 4.4 versus the 3.83 of the grammar based approach.
Taking a closer look at the ontologies and where missing axioms came from gives us a better understanding of the system. The only axiom which wasn’t verbalised for Pizza was one which contained an ObjectMaxCardinalityOf. This appears to no longer be working after the addition of the translation functionality. The integer parameter is no longer being fulfilled as all integers are currently being treated as strings, as a side effect of the translation. This bug affects all cardinality constraints, not just MaxCardinality.
Due to the shortcomings which were brought to light by the final evaluation, some bug fixes were made in order to ensure that the project goal of verbalising all OWL 2 DL axioms, was met. Another informal evuluation was run which simply involved running the Grammar based approach against the previously used 6 sample ontologies. The axiom coverage has been summarised in table 3:
Table 3. Axiom coverage per ontology including new changes
As can be seen, the average increased dramatically from 93.32 to 98.74. This increase means that the Grammar based approach ends up having higher axiom coverage than the template based approach with 98.74 and 97.1 respectively. The post bug fix axiom coverage is now greater than or equal to the template based approach for all ontologies. The ontology with the lowest axiom coverage for the final evaluation (80.88) also scored the lowest in terms of quality of verbalisation for the template based approach. It is not infeasible that an axiom coverage increase from 80.88 to 98.53 could increase the quality of verbalisation for this ontology.
Discussion
As can be seen above, the template based approach outperformed grammar based approach on almost all measured metrics in the final evaluation. It should be noted however that the grammar based approach includes a lot more functionality than the template based approach and as a result spread its focus out across the different parts instead of simply on the quality of verbalisation and coverage of complex axioms. Some of this functionality includes the translation module as well as generating the GF files dynamically. This functionality also increases the time taken parameter due to the fact that more background steps are occurring.
The post bug fix axiom coverage ended up being very high with only 7 axioms in total not being verbalised. This is a huge improvement from the preliminary evaluation which had less than 80% axiom coverage instead of the final 98.74%. The maximum time taken increased from 4.07 in the preliminary evaluation to 5.46 in the final evaluation. While the final evaluations time taken is less than the template based approach, it is still troubling that the maximum time taken is this high. It is hard to imagine a user waiting for almost 6 seconds for the program to run without thinking that something has gone wrong. This figure is definitely a shortcoming of the project but at the same time is also a function of the extra functionality discussed above.
As briefly mentioned above, it is hoped that the bug fixes would increase the average quality of verbalisation for the poorly performing ontologies. While it isn’t clear that this would happen it is within the realm of possibility. It seems unintuitive that the grammar based approach scored a lower quality of verbalisation score than the template based approach and that should be seen as a shortcoming of the project implementation.
The ease with which the bugs which were highlighted in the final evaluation could be fixed was a result of the initial time investment which was used to learn GF. Now that a familiarity has been built up making additional changes becomes more and more trivial.
The aim of the final evaluation: To determine whether the system was robust enough to verbalise OWL 2 DL ontologies was achieved, an extremely high axiom coverage of 98.74% was satisfactory in order to achieve this goal.