Individual contribution / Umr Karriem
Connecting configurable model evaluation to a public leaderboard for exploring South African language performance.
From prompt templates to comparable results.
Our work brings together the prompt-based evaluation pipeline and the MzansiBench website. The system centralises resources that were scattered across repositories and makes evaluation results easier to explore.
We evaluated models across 12 NLP tasks and 15 datasets, focusing on South Africa’s nine low-resource Niger-Congo B languages alongside English and Afrikaans.
Make evaluation reproducible and results accessible.
Configure, evaluate, convert and explore.
Evaluation pipeline
Version-controlled YAML files define prompts, dataset fields and scoring. LM Evaluation Harness performs inference and scoring, with command-line arguments or a terminal interface used to configure runs.
Raw scores are converted to CSV and then combined into the JSON data used by the leaderboard.
Website
Users switch between tables and charts, and between summary rankings, task breakdowns and detailed prompt results. Filters include model, architecture, dataset, language and task group.
New results are submitted for administrator review before appearing on the leaderboard.
Validation approach: unit tests, an AfroBench reproduction using a pinned harness version and matched prompt settings, model comparisons, and a user study with five NLP postgraduate students.
Strong agreement with AfroBench under matched conditions.
MzansiBench closely reproduced AfroBench’s results under matched evaluation conditions. Gemma 2 9B scores fell within AfroBench’s reported prompt-variability range across all six compared datasets.
| Model | LR average (%) |
|---|---|
| Gemma 2 9B | 43.1 |
| Gemma 1.1 7B | 30.5 |
| AfroLLaMa 8B | 27.1 |
| LLaMa 3 8B | 26.9 |
LR averages summarise per-task scores for available low-resource language results, using the best-performing prompt. Model coverage differs: only Gemma 2 9B was evaluated on MasakhaPOS and the three machine-translation datasets.
A foundation for broader South African evaluation.
MzansiBench joins datasets, prompts, evaluation code and public result comparison in one modular framework. Matched reproduction checks support the pipeline’s correctness, while model results reveal substantial performance differences between high-resource and low-resource languages.
More datasets, models and task coverage are needed, especially for underrepresented languages. Clearer filtering and configuration guidance remain important usability improvements.
Umr’s research documents.