8 / 8
End-to-end functional tests
Loading, preprocessing, training, evaluation and result generation completed for every supported dataset.
← Project overview
Individual contribution / Mustafaa von Doorze
Fine-tuned evaluation
A standardised workflow for preparing datasets, fine-tuning encoder models and recording reproducible evaluation results.
Overview
One workflow for different supervised tasks.
South African datasets differ in input structure, preprocessing and training requirements. The shared fine-tuning pipeline retains these dataset-specific differences while standardising model training, evaluation and result generation.
Eight datasets cover sentiment, topic and intent classification, hate-speech detection, natural language inference, named entity recognition and part-of-speech tagging.
Aims
Consistency, comparability and usability.
- Support different South African datasets and supervised task types through a common, reproducible workflow.
- Compare results with published fine-tuned baselines where comparable experiments are available.
- Evaluate configuration, information clarity and result interpretation through a user study.
Methodology
Separate dataset preparation from the shared runner.
-
Configure the experiment
Select the model, datasets, languages and seeds. A central registry resolves valid runs and combines dataset-specific training presets with user overrides.
-
Prepare the data
Validate splits and labels, tokenise inputs and apply task-specific processing. AfriXNLI trains on English XNLI before target-language evaluation; token tasks align labels to subword tokens.
-
Train and evaluate
Use Python, PyTorch and Hugging Face tooling. Train on the training split, select the model using validation data, then evaluate on held-out test data.
-
Record and repeat
Save metrics, configuration, seeds and runtime information in JSON. Aggregate repeated runs using means and standard deviations.
View the full fine-tuning activity diagram
Results
All eight datasets completed the workflow.
7 / 8
Broadly consistent comparisons
Seven dataset-level comparisons were reasonably close to published baselines.
5 / 5
Positive ease-of-use ratings
All five user-study participants rated the pipeline as easy to use. The small study was exploratory.
| Dataset | Metric | Published | MzansiBench |
|---|---|---|---|
| AfriHate | Macro F1 | 83.46 | 81.11 |
| AfriSenti | Accuracy | 47.53 | 48.74 |
| AfriXNLI | Accuracy | 65.79 | 66.91 |
| INJONGO | Accuracy | 87.29 | 90.65 |
| MasakhaNER-X | Entity F1 | 86.72 | 79.43 |
| MasakhaNEWS | Weighted F1 | 91.13 | 92.16 |
| MasakhaPOS | Token accuracy | 86.33 | 88.29 |
| SIB-200 | Accuracy | 71.85 | 71.92 |
Scores average retained seeds, then comparable model-language combinations. Collapsed or failed seeds are excluded; these are dataset-specific metrics, not a single interchangeable score.
Conclusion
Standardisation with dataset-specific flexibility.
The pipeline supports consistent evaluation across eight datasets while preserving their preprocessing and training needs. Recorded configurations and repeated seeds make variation easier to inspect and experiments easier to reproduce.
Exact baseline reproduction remains sensitive to seeds and incomplete experimental details. Broader coverage still depends on new datasets for underrepresented languages and tasks.
Resources
Mustafaa’s research documents.