Introduction
Adaptive Learning systems are educational applications that provide learner's with personalised learning materials, that are tailored to their knowledge on a certain topic. To create an effective Adaptive Learning system, multiple components are required to work in tandem to provide a learner with a positive experiance, as well as to successfully identify and bridge any knowledge gaps. The first of these components, is the Automatic Question Generation (AQG). An AQG simply automatically generates questions for a learner based on their learner knowledge model, which stores a learner's ability for each previously administered topic. The AQG is an important component in the entire Adaptive Learning system as it continously assesses a learner's knowledge on a topic, to ensure that the system has an updated and accurate representation of their abilities for adaptability purposes. The AQG works by generating questions relating to the learner's three worst subjects, to ensure that their progress is accurately monitored. This ensures that the Adaptive System is able to identify whether the learning materials are helping the learner better understand the topic, or if further learning materials for that topic should be produced as the learner is not currently receiving an adequete amount of learning materials. Furthermore, the AQG also generates questions pertaining to a learner's best concpets, as this will also ensure that the learner thoroughly understands that subject.
Research Aims
The aim of this research is to analyze existing AQG algorithms, and do then devise a new AQG algorithm that seeks to improve the grammaticality, understandability, adaptability, and suitability of the generated questions. The new AQG algorithm should smoothly integrate into an Adaptive Learning system, allowing the system to constantly evaluate and receive an updated version of a learner's knowledge model.
Methodology
The AQG component followed a light-weight design, by not requiring a large amount of information to generate adaptive questions for a learner. This was a signifcant consideration, as the reviewed literature made it clear that existing AQG algorithms struggled with being light-weight and often required an immense amount of information prior to any adaptability aspects being enforced. Thus, the AQG exclusively utilises the learner knowledge model for the adaptivity aspect. The AQG was designed using a template-based approach, whereby question temmplates were manually created and then are randomly populated using an ontology during run-time. All of the generated questions will then be stored into two large question banks, one for standard questions and one for difficult questions. Once all of the question templates have been generated, the AQG will then identify a learner's three worst concepts, according to the learner knowledge model, and then adaptively select questions from the question banks that relate to any of the learner's three worst concepts.
Evaluation
The AQG was evaluated using two different forms of evaluation, systematic and user evaluation. The systematic evaluation aimed to identify whether the questions were accurately adapting to a learner's knowledge model, as well as identify exactly how many questions were generated per iteration. The user evaluation aimed to identify whether the AQG was successful in achieving grammaticality, understandabiliy, adaptability, and suitability. The user evaluation was conducted by allowing selected individuals to interact with the AQG, whilst using a manually created and standardized learner knowledge model. Thereafter, the user would then receive a survey which they would complete by rating certain aspects of the AQG.
Results
The results of the evaluation showcased that learners highly rated the generated questions. The surveys which the learners completed showed that learners felt that the questions were grammatically correct, understandable and did adapt to the learner knowledge model. The results showed that no learner felt any question to be grammatically incorrect. We understand that the generated questions were understandable as 80% of learners felt that the questions were appropriate for their level of understanding, as well as 60% of learners felt confident in answering the questions. However, the reason for 40% of people not feeling confident, is not due to the structuring of the question but rather of the question content. Thus, this does not signal a cause for concern. Furthermore, 60% of learners could sense that they were on a clear and tailored learning path. The reason of 40% of learners not sensing this, is due to them receiving one or two irrelevant questions. However, this is expected as this is how the system introduces new concepts to a learner. However, the results regarding the suitability of the generated questions produced results that required further investigation. Only 20% of learners felt that the questions were suitable to their difficulty tolerance. Thus, most learners felt that the generated questions were too easy.
Conclusions
The systematic evaluation showed that the questions adapted to the learner knowledge model and followed a specific and tailored learning path. This assumption was further corroborated by the user evaluation whereby participants could clearly identify that they were on a specific and tailored learning path. Our evaluation showed that the questions were both grammatically correct and understandable. These are promising results as several existing AQG algorithms struggled from the generation of questions that were either not grammatically correct or difficult to understand. However, we were also able to conclude that the suitability of the questions generated by the AQG were not suitable to all participants. Although there were different question templates that had different diffculty levels, participants of the survey did not find the diffcult questions to be diffcult. They were still able to answer the questions easily as long as they knew what the food item was. Our evaluation showed that the inclusion of the AQG in an adaptive learning system would be of benefit and would improve the entire adaptive learning system. This is true for ontologies that are smaller in size, as our AQG does not require a large amount of information to adaptively generate questions for a learner.
Future Work
The development and implementation of the AQG was successful however, the evaluations did identify the potential for future work that would improve the AQG. As a template-based approach was followed, newer templates could be introduced that varied in difficulty and assessed different concepts. This would add further variation to the already diverse set of questions. Furthermore, the size of the ontology could be increased, as this will ensure that more questions are able to be generated for more foods.