Results

This table gives an indication of each participant's understanding of the descriptions.

The green cells indicate that participants 2, 3, 4, 5 mostly perceived the descriptions as understandable, while the yellow cell indicates that participant 1 mostly felt unsure about the understandability of the descriptions.

The results suggest that an average participant percieved the descriptions as understandable and the average number of exact matches per participant is 5 (± 1.58).

Individual Participant's Performance

Table Comparing Each Participant’s Most Common Rating and the Total Number of Exact Matches Between Their Input Formula and the Expected Formula, and the Overall Average Rating and Matches between Participants.

Individual Description's Performance

A Stacked Chart Comparing the Proportion of Ratings and the Total Exact Matches (out of 5) of Each Formula; and Displays the Total Percentage of Each Rating and the Total Exact Matches for the 50 Responses.

The expressions that performed the best included those with power, equals, multiplication, plus and divide operators.
The descriptions for the complex expressions y=mx+c and (xyz)^3 were rated as understandable by 100% of the responses, which coincides with all five participants typing out the formulae correctly.

The descriptions that did not perform as well, were those that contained square root, integral and minus operators. The description for the expression (x-y)/2 was interpreted as (y-x)/2 (reversed order of operands for the minus sign) by 3 of the 5 participants. The same issue with the minus operator occurred with the expression x-(y/2), it was understood as (y/2)-x by 4 out of 5 participants and received zero exact matches.

Descriptions containing a square root received mixed ratings and few exact matches. The expression sqrt(a-b)/2c could not be answered correctly by any of the participants. The L1 speaker was the only participant able to distinguish between sqrt(a)+b and sqrt(a+b). One participant, an L2 speaker, commented that they were confused by the wording ‘Umaziphinde’, in the square root description.

There was a single description containing an integral, int(x+1), which was answered correctly only by the L1 speaker. This description received the most mixed ratings, with a different rating from each of the five participants.

Findings

There were two types of issues uncovered by the formula input section of the evaluation; suspected problems with terminology
for the square root and integral templates; and issues with the order of operands for the minus template.

Asking participants to type out the expression’s formula was a way to determine if a participant’s perceived understandability matches with their actual understandability. Although the participants perceived the descriptions as understandable, the results show that on average a participant was only able to accurately replicate the formulae 50% of the time.
A good example of why this check was necessary is seen in the stacked chart; 80% of participants agreed that the description for (x-y)/2 was understandable; however, three participants wrote down the same incorrect formula. Since there was a consistent error across these three participants, it’s likely that the error came from the semantics of the template, and not a misunderstanding from the participants.

A recommendation for going forward is that the templates for the minus, square root, and integral operators require improvements; we consider that during the corpus creation stage, collecting more output samples from a wider range of isiZulu speakers and then basing our templates off the most popular responses, could be a more comprehensive approach to building templates, as shown by (Ferres and Sepúlveda, 2011) in their template-based system.