CoSMo Studio

What is CoSMo?

CoSMo is a content selection modelling language aimed at fulfilling the first step in the NLG pipeline, which is content selection. Therefore, CoSMo is intended to be the language for the constructors in Abstract Wikipedia. The language consists of elements, connectors and adornment icons that can be represented both graphically and textually and can be combined to form a constructor. Elements and adornment icons can also be populated with items from Wikidata and Wikifunctions.

What is CoSMo Studio?

CoSMo Studio is an intuitive tool where users can build and manage constructors in the correct CoSMo syntax.

Aims

Dynamically Building Constructors

Enable users to build constructors graphically by combining CoSMo elements, assigning Wikidata items to them, and optionally adding functions from Wikidata.

Generation of Textual Constructors

After building a constructor, enable users to generate their textual version in the correct CoSMo syntax.

Syntax Validation

Validate the syntax of user-made constructors to ensure compliance with the CoSMo syntax, as defined in the CoSMo documentation.

Support for Multilingualism

Allow for the generation of textual constructors in a wide variety of languages, where the constructor labels can be generated in as many languages as is stored in Wikidata.

Constructor Management

Allow users to save, modify, and delete their own constructors. Additionally, enable users to combine their own constructors with those made by other users, accessed through a central repository.

Implementation

Software & Architecture

CoSMo Studio was implemented using the client-server architecture. The React client is responsible for managing user interactions and displaying data to the user. The server handles API requests from the client, performs CRUD operations on the database, and sends responses back to the client. The MySQL database stores items modelled after Wikidata and Wikifunction items. A MySQL database was designed to store dummy data, modelled after items in Wikidata and Wikifunctions. For integration with Abstract Wikipedia, the server can be modified to communicate directly with Wikidata and Wikifunctions. The node-based editor for graphically building constructors was built using the ReactFlow library.

cosmo-architecture

Methodology

The system was developed using an iterative approach, where features were designed, implemented, and tested in repeated cycles, each lasting approximately 14 days. With each iteration, the system evolved, gradually incorporating the necessary functionalities and refining its overall performance.

Design Iteration One

Gathered system requirements

Developed first iteration of prototype

Modelled UML diagram to visually map the flow of activities and processes across features

➜
Development Iteration One & Two

Developed feature for graphically building constructors

Developed feature for generating the textual version of constructors

➜
Design Iteration Two

Developed second iteration of prototype

➜
Development Iteration Three & Four

Developed syntax validation mechanism and feature for support of multiligualism

Developed constructor management feature

➜
Testing and Evaluation

Software testing

Usability testing

User acceptance testing

➜
Development Iteration Five

Enhanced system usability based on user feedback to produce the final product

How it Works

The following video provides a demonstration of CoSMo Studio and its core features.

Testing and Results

Usability Testing

To assess the usability of the constructor building tool, four English-speaking users were asked to complete a task-based evaluation, a custom survey and a System Usability Scale (SUS) survey. Participants were selected using snowball sampling.

Task-based Evaluation

Users were assigned a series of tasks to complete, including both simple navigation tasks e.g. (log in, navigate to a tab) and more complex tasks, such as building a constructor from a given sentence.The time to complete and success rate for each task were recorded per user.


Results

Users completed simple tasks quickly, suggesting they are intuitive and well-supported by the system. Longer times for complex tasks indicate higher cognitive demands, but it's important to note that participants were non-experts with limited experience in CoSMo. Despite this, the 2-3 minute average for complex tasks is reasonable. The task success rate was also high, with only 3 small errors discovered across all users.

graph1

Custom Survey

Users completed a custom survey on the usability and navigability of the interface and rated both aspects on a scale of 1 to 10.

Results

Both the navigability and ease of use ratings were high, with average scores of 8.75 and 8.25, indicating a generally positive user perception of the system’s usability and ease of navigation. From this survey, several areas for improvement were identified, such as improved connector selection with radio buttons, enhanced navigation with added buttons etc. These improvements were implemented and can be seen in the final product, as shown in the demonstration.

SUS Survey

Users completed a SUS survey, a 10-question questionnaire designed to measure perceived usability.

Results

The average SUS score was 72.5, falling within the "acceptable" to "good" range, showing that users consider the system usable but with some potential for enhancement.

User Acceptance Testing

Two Microsoft Teams meetings were held with two known Wikipedia contributors, recommended by the project supervisor, to demonstrate the completed system. The presentation included a walk-through of each page and feature, highlighting its purpose. Following the presentation, an interview with each contributor provided feedback on the system’s usefulness.


Results

The first Wikipedia contributor found the interface clean, quick, and learnable, highlighting its potential for Abstract Wikipedia but suggesting further enhancements, such as building constructors from sentences, saving them in multiple languages, and providing a clear user manual. The second contributor also found the interface intuitive and useful for building constructors, noting that some training may be necessary for users unfamiliar with CoSMo. Both contributors acknowledged the potential usefulness of the tool in the context of Abstract Wikipedia.

Conclusion

CoSMo Studio has successfully achieved the project’s objectives through a meticulous and iterative development process, guided by feedback from both user evaluations and user acceptance tests. The system has met the requirements gathered by enabling users to intuitively build, modify and combine CoSMo constructors through the use of a graphical node-based editor, and with syntax validation to ensure constructors are built in the correct CoSMo syntax. Furthermore, the system enables users to generate the textual version of a given constructor with support for multilingualism. Finally, the system carefully tracks user access to enable users to save, modify and delete constructors.