We present the findings from the human evaluation. Five isiZulu speakers with different levels of language proficiency (first, second and third language speakers) were recruited to join the questionnaire, which resulted in a total of 50 responses over 10 descriptions.
A Stacked Chart Comparing the Proportion of Ratings and the Total Exact Matches (out of 5) of Each Formula; and Displays the Total Percentage of Each Rating and the Total Exact Matches for the 50 Responses.
68% of all responses were rated as understandable.
26% were unsure of a description’s understandability.
6% of all responses disagreed on a description’s understandability.
The expressions that received disagreeing or unsure responses, and low exact matches, contained minus, integral or square root operators.
The total number
of exact matches across all descriptions is 25 out of 50 (50%).
Based on a
human evaluation of the generated text, we conclude that our template-based realisation method
produced text that was averagely perceived as understandable among isiZulu
speakers; and thus, is an appropriate technique for building a Content MathML-to-Text NLG system.
Although the descriptions were perceived as understandable on average, the results also show that a formula could only be accurately typed out 50% of the time. This was due to a consistent problem among participants with the minus, square root and
integral templates. Hence, these three templates will require improvements to their terminology.
created with
Website Builder Software .