Traditional Lithuanian spelling is based on the set of 32 graphemes: a, ą, b, c, č, d, e, ę, ė, f, g, h, i, į, y, j, k, l, m, n, o, p, r, s, š, t, u, ū, ų, v, z, ž that includes 9 diacritic symbols.2²

Linguistic entity, like a grapheme or word written according to Lithuanian orthography is given in italics. International Phonetic Alphabet (IPA) based phonetic transcription is enclosed within square brackets. SAMPA-LT based allophonic transcription is given in plain text.

The considerations above imply that G2P conversion of Lithuanian is quite complex. G2P converter that relies on a word spelling and grapheme rewrite rules (Greibus et al., 2017; Lileikytė et al., 2018), henceforth referred to as a shallow G2P converter, is incapable of resolving ambiguities related to vowel duration, syllable stress, and syllable boundaries and consequently is incapable of producing detailed and consistent allophone sequences. Only G2P converter making use of supplementary pronunciation dictionaries (Skripkauskas and Telksnys, 2006) or of accentuation algorithms (Norkevičius et al., 2005; Kazlauskienė et al., 2010), henceforth referred to as a knowledge-rich G2P converter,4⁴

Grapheme-to-allophone converter would be a more appropriate name.

The problem of finding the best word to sub-word unit mapping for the applications of Lithuanian ASR was first addressed by Raškinis and Raškinienė (2003), followed by Šilingas (2005), Laurinčiukaitė and Lipeika (2007), Gales et al. (2015), Greibus et al. (2017), Lileikytė et al. (2018), and Ratkevicius et al. (2018).

All abovementioned studies have used very different ASR setups (see Table 2). First, different proprietary speech corpora were used for ASR system training and evaluation (Laurinčiukaitė et al., 2006; Harper, 2016; Laurinčiukaitė et al., 2018). Second, ASR setups were based on different acoustic modelling techniques, such as monophone HMM system (Šilingas, 2005; Ratkevicius et al., 2018), triphone HMM system (Raškinis and Raškinienė, 2003; Šilingas, 2005; Laurinčiukaitė, 2008; Greibus et al., 2017), or hybrid HMM – neural network models (Gales et al., 2015; Lileikytė et al., 2018). Third, different evaluation methodologies were used. Raškinis and Raškinienė (2003), Laurinčiukaitė and Lipeika (2007), Ratkevicius et al. (2018) and this study prefer accuracy estimation through cross-validation, whereas other studies estimate recognition accuracy on a held-out data, an approach that is less computation intensive. Fourth, different evaluation criteria were used. Studies differ by comparing PER (Šilingas, 2005), WER (Raškinis and Raškinienė, 2003; Šilingas, 2005; Laurinčiukaitė and Lipeika, 2007; Gales et al., 2015; Lileikytė et al., 2018; Ratkevicius et al., 2018), and sentence error rate (Greibus et al., 2017). Fifth, ASR setups incorporated different language models such as word loops (Raškinis and Raškinienė, 2003; Šilingas, 2005; Laurinčiukaitė and Lipeika, 2007; Ratkevicius et al., 2018), word n-grams (Gales et al., 2015; Lileikytė et al., 2018), command lists (Greibus et al., 2017), and phone n-grams (this study).

⁵

Actual/maximum term-weighted value is used to evaluate keyword spotting performance.

Though word to grapheme mappings investigated by different studies are quite similar, word to phoneme mappings are different and mostly incompatible across studies. Each study makes its own choices about whether to and how to represent stress, duration, palatalization, affricates, diphthongs and mixed diphthongs in a phonemic lexicon (see Table 3). Laurinčiukaitė and Lipeika (2007) go beyond word to phoneme mappings and investigate word to sub-word unit mappings, where sub-words may be phonemes, syllables and pseudo-syllables.

⁷

Lexicon includes allophones to represent differently stressed variants of all vowels and diphthongs (✚), or only diphthongs ai, au, ei, ui (✓). Lexicon ignores the opposition of stressed vs. non-stressed sounds (O).

Given such a variety of the experimental setups it is not surprising that different studies came to different and even opposite conclusions. For instance, Raškinis and Raškinienė (2003) achieved the best WER by the word to phoneme mapping that ignored stress and preserved palatalization (see Table 3, ABC phonemic lexicon), whereas (Šilingas, 2005) achieved best WER by preserving stress and ignoring palatalization (see Table 3, BFR6 phonemic lexicon). Greibus et al. (2017) achieved best SER by ignoring both stress and palatalization. Gales et al. (2015) found that grapheme-based system outperforms phoneme-based system, whereas Šilingas (2005), Greibus et al. (2017) and Lileikytė et al. (2018) came to an opposite result. Laurinčiukaitė and Lipeika (2007) found that mapping into a mixture of phonemes and syllable-like units improves WER.

Incompatible conclusions are partially due to the limitations of the experimental setups. Some findings are based on a small training corpus (Raškinis and Raškinienė, 2003; Ratkevicius et al., 2018) or on a limited carefully selected held-out data (Šilingas, 2005). Other studies (Greibus et al., 2017; Lileikytė et al., 2018) are testing limited word-to-phoneme mappings due to the usage of a shallow G2P converter which is unable to produce allophone-rich phonemic transriptions. Conclusions of many studies are dependent on a single (though generally state-of-the-art at the time of investigation) acoustic modelling technique. Finally, recognition accuracies obtained by the majority of studies are not “pure” indicators of performance of different word to sub-word mappings as they are strongly influenced by different amounts of linguistic constraints embedded into ASR setups. For instance, Greibus et al. (2017) restrict their language model (LM) to a command list, where commands share 271 unique word types, and Ratkevicius et al. (2018) restrict their LM to a 10-digit word loop.

In this study, we have adopted an experimental approach common to other similar studies (Raškinis and Raškinienė, 2003; Šilingas, 2005). It consists of defining some phonemic lexicon which serves as a reference point. Thereafter, reductions of this lexicon are derived by elimination of various phonological properties (e.g. stress, palatalization) or by splitting compound phonetic units (e.g. diphthongs, affricates) into sub-parts and measuring the performance of the ASR system for every reduced lexicon. Our reference phonemic lexicon consists of 130 allophones (henceforth referred as to “detailed” lexicon). It is presented in Table 4 using SAMPA-LT (Raškinis et al., 2003) encoding.

We have compared the “detailed” lexicon against 5 reduced phonemic lexicons and one graphemic lexicon in order to answer the questions about what is the best approach to: •

Stress modelling (present vs. absent),

The process by which different phonemic and graphemic transcriptions were obtained is described in Fig. 1. First, word-level ortographic transcription was processed by the knowledge-rich G2P converter (Kazlauskienė et al., 2010) resulting in allophone sequence that observes intra-word sound assimilation rules. Thereafter optional sound assimilation rules were applied at word boundaries in an automatic way on a basis of a maximum-likelihood criterion. This resulted in “detailed” phone-level transcription encoded by SAMPA-LT symbols that served as our reference word-to-phoneme mapping. Reduced phonemic transcriptions were derived from the “detailed” transcription by subjecting it to one or more editing operations (see Fig. 1).

Graphemic transcription was obtained from the word-level ortographic transcription by means of a few editing operations that encoded graphemes by SAMPA-LT symbols. This encoding was necessary in order to harmonize phonemic and graphemic transcriptions for their comparison at a later stage (see 3.6). Graphemic lexicon may look like a phonemic one, but this is a false impression. Graphemic transcription was not subjected to sound assimilation rules, and the changes in graphemic transcriptions are simple and mostly reversible transliterations.

Our experiments were based on a 50-hour speech corpus that was compiled at Vytautas Magnus University. The corpus consisted of 50 speakers (25 males and 25 females) each reading book excerpts for approximately 1 hour.17¹⁷

Silent segments make 15–20% of the corpus depending on the speaker.

We built multiple ASR systems based on different phonemic/graphemic lexicons and tried to estimate their accuracies via a cross-validation technique. The cross-validation round consisted of training an ASR system (building acoustic and phone-level language models) on the speech and transcripts of 49 speakers and testing system accuracy on the speech of the held-out (or test) speaker. Full leave-one-out (or 50-fold) cross-validation was costly in terms of computational time. Instead, we approximated it with a “pessimistic” 10-fold cross-validation scheme. We call it “pessimistic” (with respect to the leave-one-out cross-validation) because of the inclusion of the most problematic speakers into the test set. The selection procedure clustered all speakers into 5 clusters of comparable size and selected 2 worst rated speakers per cluster for inclusion into the test set ( 2×5=10 in total).18¹⁸

Speaker rating was determined on the basis of WER obtained in our earlier recognition and adaptation experiments (Rudžionis et al., 2013). Speaker clustering was based on the feature set that included insertion, deletion and substitution errors.

It is reasonable to expect that a certain phonemic lexicon performs better when coupled with some particular acoustic modelling technique. In order to investigate this relationship and to assess the possible limitations of our conclusions we have built and compared the ASR systems based on the acoustic models of 7 different types19¹⁹

We have used an open-source Kaldi ASR toolkit (Povey et al., 2011a) for training and evaluating all ASR systems. Some other techniques, mostly discriminative training approaches, have been tried but not described in this paper, because their accuracy estimates correlated with the results of non-discriminative training.

We aim to build an experimental setup such that the ASR system is stripped from its lexical and grammatical knowledge (list of words of a language and probabilities associated to word sequences) that influences recognition accuracy, so that the accuracy of the ASR system reflects the performance of the word to sub-word unit mappings under investigation. It should be noted that phonotactic knowledge cannot be eliminated from our comparisons because it makes an integral part of an acoustic (starting from triphones) model. If we take a triphone acoustic model, extract the list of all triphones, and make a fully-connected triphone network that respects adjacency constraints (i.e. triphone a-x+y is connected to every triphone x-y+b in the list, where a, b, x, y denote any sub-word unit of the lexicon) we obtain a phone 3-gram with the uniform probability distribution over the outgoing links. It represents the set of categorial phonotactic constraints embedded into an acoustic model. Let’s call it the categorial phone 3-gram. Table 5 compares perplexities of the categorial and probabilistic phone-level n-grams.

We have taken the categorial phone 3-gram as our baseline decoding setup. In addition, we performed decoding experiments with phone 3-grams and 4-grams to observe how additional probabilistic phonotactic knowledge affected the ASR performance.21²¹

We did not perform decoding with phone 1-grams and 2-grams because their decoding accuracies are hard to interpret. On the one hand, phone 1-grams and 2-grams are under-constrained with respect to the categorial phonotactic constraints integral to the triphone acoustic model, and, consequently, the decoder is forced to synthesize triphones that violate phonotactic constraints of the language. On the other hand, 1-grams and 2-grams are more constrained by probabilistic knowledge than the categorial 3-gram by taking advantage of statistics of the training corpus.

Decoding with categorial 3-grams, probabilistic 3-grams and 4-grams exploited phonotactic but not lexical or syntactic-semantic knowledge, so we believe that our comparisons were independent from the lexical content of the training/evaluation data.

To summarize, our experimental investigation consisted of building 7 (phonemic/graphemic lexicons) × 7 (acoustic model types) × 10 (speaker-specific cross-validation rounds) = 490 different acoustic models and performing 490 (acoustic models) × 3 (phone-level language models) = 1470 decoding experiments in total.

We have used Phone Error Rate (PER) criterion to compare the performances of different ASR setups. It was calculated according to: (1) PER=S+I+DN100%, where S, I and D denote substitution, insertion and deletion errors respectively, and N is the total number of phones/graphemes in the test data. S, I and D estimates were extracted from automatic alignments of recognized and reference transcriptions.

Automatic alignment of recognized and reference transcriptions was preceded by the transcription normalization step. This step consisted of projecting every individual phoneme/grapheme onto a symbol or a sequence of symbols over the normalized alphabet. Without projecting lexicons of different sizes into the common lexicon the comparison would be biased against allophone-rich ASR setups as they naturally tend to result in more substitution errors than ASR setups based on reduced lexicons. Moreover, without normalization, substitution errors involving, e.g. stressed vs. unstressed or palatalized vs. non-palatalized allophones of the same phoneme, will look like equally important as phoneme substitutions.

Normalized alphabet contained 27 symbols (a b d e E: f g G x i i: j k l m n o p r s S t u u: v z Z). It represented the intersection of all investigated lexicons, i.e. it contained symbols that were common to all lexicons. Other allophone units were projected into this alphabet by eliminating their phonetic properties or by spliting compound units (affricates, diphthongs, fronted back vowels) into the sequences of 2 or 3 symbols.22²²

For instance, graphemic lexicon lacks the symbol o: (see Table 1 and bottom part of Fig. 1), so o: is not included into the normalized alphabet, and all phonemic lexicons are projecting o: → o. The “no affricates” lexicon lacks affricates ts, tS, dz, dZ, so other lexicons are projecting affricates into a sequence of two symbols.

The process of projecting phonemic and graphemic transcripts into scoring transcripts over the normalized alphabet was realized by 4 steps:

Though grapheme-based and phoneme-based reference transcriptions are mapped to transcriptions over the same normalized alphabet, they are not identical. For instance, Lithuanian word džiaugsis (will rejoice) is mapped to d Z e u k s i s (phonemic) and d Z i a u g s i s (graphemic) over the same normalized alphabet. The difference stems from the fact that phonemic transcriptions by definition are transcriptions subjected to sound assimilation rules.24²⁴

The original word spelling cannot be restituted neither from a phonemic, nor from a graphemic transcription expressed over the normalized alphabet due to the one-to-many mapping, e.g. a t S i u: could be restituted as ačiū (thanks), ačių, atšiū, atšių, ąčiū, ąčių, ątšiū, ątšių (nonsense words).