SABC2TXT

Methods

1. Evaluation Metrics

Word Error Rate

The Word Error Rate (WER) measures the accuracy of a speech recognition system by calculating errors in word substitution, deletion, or insertion, divided by the total words in the correct sentence. \[ \text{WER} = \frac{\text{substitutions} + \text{deletions} + \text{insertions}}{\text{total number of words}} \]

Levenshtein distance

The Levenshtein distance measures the number of changes required to make one word sequence match another, and in the project, it was applied at both word and character levels. \[ \text{LD} = \text{substitutions} + \text{deletions} + \text{insertions} \]

2. Training database preparation

There are no pre-existing language or acoustic models for isiXhosa. Therefore, a new language and acoustic model must be trained using CMU SphinxTrain's command-line tools. SphinxTrain is tailored to develop speech recognition systems for any language with sufficient acoustic data and is compatible with the CMUSphinx recognizer.

2.1 Building a language model

Using the CMUCLMTK toolkit from CMUSphinx, an isiXhosa language model was developed based on the 56-hour NCHLT isiXhosa Speech Corpus and supplemented with the Lwazi isiXhosa TTS corpora from SADiLaR. After assembling a complete reference text, a unique vocabulary file was made, leading to the final ARPA format language model.

2.2 Phonetic dictionary and G2P model training

The isiXhosa phonetic dictionary file was obtained from the NCHLT-inlang pronunciation dictionaries from SADiLaR, which are broad phonemic transcriptions for 15,000 common words in 11 different languages. The phonetic dictionary contains the phonetic representation of all the words contained in the transcription files, allowing the decoder to understand how to pronounce each word.

The NCHLT-inlang isiXhosa dictionary was incomplete, lacking some words from the transcription file. To address this, Sequitur G2P was employed to train a grapheme-to-phoneme model, which generated phonetic representations for missing words using existing dictionary entries. After three training iterations, the final model filled in the dictionary gaps.

2.3 Training database structure

In the process of training an acoustic model, a training database is required. The database provides the data needed to create an acoustic model that extracts statistics from speech. The database was divided into two sections: a training section and a testing section. The testing section was about 1/10th of the total data size and did not exceed more than 4 hours of audio. The database prompts with post-processing contained the following database structure:

3. Acoustic model training

Acoustic model training occurred in two phases. Initially, it solely used the NCHLT isiXhosa Speech Corpus, resulting in a limited word recognition. The second phase integrated data from Lwazi isiXhosa TTS corpora, improving the model. During training, alignment errors between audio and transcription arose, requiring forced alignment and prompt filtering. The initial model had a WER of 30.1% and an SER of 33.2% as depicted in the table below. The second phase marginally improved the WER, but the SER remained consistent.

Iteration	Error rate evaluation	Error	Transcription errors
Iteration 1	Sentence Error Rate (SER)	33.2%	920/2770
	Word Error Rate (WER)	30.1%	2442/8107
Iteration 2	Sentence Error Rate (SER)	33.2%	920/2770
	Word Error Rate (WER)	30.2%	2442/8107

4. Android app integration and implementation

The app, adapted for isiXhosa speech recognition, initializes its recognizer upon launch. It continuously listens for the activation keyword "ubuntu". Once this keyword is detected, the app transitions into continuous recognition mode, capturing and processing audio input. After the audio input concludes, the app displays the transcribed version of the audio on the screen. A demonstration of how the Android app works can be found below.

5. Experimental Design

Data preprocessing

• Audio segmented into shorter chunks using pydub.
• Artificial city-traffic background noise added at varying levels.
• Casual isiXhosa conversations cleaned and normalized.

Different hardware

• Transcriptions tested on Android emulator and various phones (Samsung Galaxy A72, Samsung Galaxy Pocket, Xiaomi Redmi 6A).

Transcription procedure

• Audio played to the app for transcription.
• Each audio transcribed three times for accuracy, results averaged.