Prompt-based
Evaluate through prompts
Zero-shot and few-shot templates evaluate decoder-only and encoder-decoder models through LM Evaluation Harness.
University of Cape Town / Honours 2026
MzansiBench
Benchmarking LLMs for
South African languages.
A unified path from South African language datasets to model evaluation and public result comparison.
Context
Making uneven coverage visible.
South African NLP resources are scattered across repositories. Some languages are evaluated across many tasks; others have very limited coverage. This makes consistent, reproducible comparisons difficult.
MzansiBench brings datasets, evaluation workflows and benchmark results together in a standardised system focused on South Africa’s 11 official written languages.
Our aims
- Standardise evaluation Provide a common framework for South African language benchmarks.
- Support complementary approaches Evaluate models through prompting and supervised fine-tuning.
- Make results accessible Centralise datasets and compare model performance through an interactive leaderboard.
The system
Two pipelines. One benchmarking system.
A Python evaluation library connects a centralised dataset repository to an interactive website. Both pipelines produce structured results for comparison.
Fine-tuned
Adapt with labelled data
Encoder-only models are fine-tuned using dataset-specific preprocessing and training presets within a shared workflow.
Website & leaderboard
Explore the results
Compare models, tasks and languages using tables and charts, with summary rankings and detailed result views.
Coverage
Many languages. Different kinds of capability.
The benchmark combines understanding, reasoning and generation tasks. Availability depends on the dataset and evaluation pipeline.
Text classification
Token labelling
Questions & reasoning
Text generation
South Africa’s 11 official written languages
- Englisheng
- Afrikaansafr
- isiXhosaxho
- isiZuluzul
- isiNdebelenbl
- siSwatissw
- Sesothosot
- Setswanatsn
- Sepedinso
- Tshivendaven
- Xitsongatso
Project outcomes
A common workflow, validated against existing work.
8 / 8
Fine-tuning datasets tested end to end
All eight supported datasets completed the workflow. Fine-tuned evaluation results were within a reasonable range of comparable published baselines overall.
Matched setup
Prompt-based results reproduced
MzansiBench closely reproduced AfroBench’s published results under matched evaluation conditions, within its reported prompt variability across the six compared datasets.
Together, the pipelines and leaderboard provide a unified benchmark for South African language evaluation. The results also expose persistent gaps in model performance and language coverage.
The team
View our individual contributions.
Mustafaa von Doorze
Fine-tuned evaluation pipeline
VDRMUS001@myuct.ac.za
Umr Karriem
Prompt-based evaluation & website
KRRUMR001@myuct.ac.za
Supervisor: Prof. Francois Meyer · francois.meyer@uct.ac.za
Project resources
Read the work behind the benchmark.
Explore our project documents and individual research, or open the MzansiBench application.