A translation is a challenging and creative act. A Machine Translation (MT) (Dorr et al., 1999; Bungum and Gambäck, 2010) can ease the work of a translator, or even replace it as a rough translation, or as a draft which serves as an aid to the translation. Nowadays, Statistical Machine Translation (SMT) (Lopez, 1993; Specia, 2010; Bungum and Gambäck, 2010) is by far the most studied and used MT method (Albat, 2007). SMT was based originally on single words, but has now progressed to the level of word sequences, called phrases. Currently the most successful SMT approach is phrase-based translation (Specia, 2010).

Translations in SMT are generated on the basis of statistical models, i.e. translation and language models, where different models’ weights provide various translations. The translation model is used to translate words or phrases from the source language to the target language text, and the language model ensures that the translated text is more fluent. These models know nothing about each other, so the problem appears to be how to find a set of weights that would provide the best translation quality. This problem can be regarded as an optimization problem. Optimization refers to the process of finding the optimal models’ weights, where the optimal weights are those which maximize the translation quality.

The translation quality is measured using translation error metrics, and the Bilingual Evaluation Understudy (BLEU) (Papineni et al., 2002) metric is one of the more popular and inexpensive automated metrics for achieving a high correlation with human judgments of quality (Callison-Burch et al., 2006). It is worth noting that MT evaluation is a complex problem, and that methods such as BLEU are not without criticism.

The quality of MT depends on the languages. For some language pairs, SMT brings good results, especially if the target language is English. The morphological richness of languages has a direct impact on the quality. Significantly lower quality is obtained for language pairs when translating from English into a morphologically rich language. Agglutinative target languages (Hungarian or Turkish) are even more problematic for statistical approaches. Although the approach proposed in this paper is general, our research was done on a difficult language pair where one language is highly analytical (English) and the other, morphologically rich (Slovenian) (Sepesy Maučec and Brest, 2010).

Numerous algorithms exist for solving general optimization problems with real valued numbers. One of these algorithms is the Differential Evolution algorithm (DE) (Storn and Price, 1997; Price et al., 2005; Neri and Tirronen, 2010; Das et al., 2016), which is a simple and effective algorithm for global optimization. It has been proved to be efficient at solving different optimization problems involving real valued numbers which interact non-linearly with each other (Das and Suganthan, 2011; Das et al., 2011; Zhou et al., 2011; Bošković et al., 2011; Glotić and Zamuda, 2015; Mlakar et al., 2014; Bošković and Brest, 2016). The DE algorithm is an evolutionary based algorithm where each individual from the population is described as a vector of models’ weights.

Currently, the more popular way to find optimal models’ weights is to use Minimum Error Rate Training (MERT) (Och, 2003; Bertoldi et al., 2009) and, in this paper, we used the jDE (Brest et al., 2006a) algorithm where each individual has its own crossover rate and scale factor. The authors in Brest et al. (2006a), Zhang and Sanderson (2009) observed through experiments that the efficiency of the DE algorithm is improved when control parameters respond to the evolution with a self-adapting mechanism. This enables the jDE algorithm to solve our problem more efficiently and reduce the number of main control parameters. Translations are then evaluated using the BLEU metric.

Recently, Evolutionary Algorithms have attracted increasing attention for enhancing the performance of Natural Language Processing (NLP) (Bungum and Gambäck, 2010) techniques. NLP is a field concerned with the interaction between computer and human using natural language – spoken (Du Bois et al., 2005; Kasparaitis and Anbinderis, 2014) and written (Koehn, 2005; Steinberger et al., 2006). In this paper, the focus is on the written language.

The SMT system should produce translations in a reasonable time. Some MT applications are working almost in real-time. Since the training and optimization processes are both a part of an offline training where we have a static corpus and no time constraints, the training time is not so relevant. Once the SMT system is built it is ready to use, and then the actual translating depends mostly on the size of the text which is to be translated.

The main goal of this paper is to find the optimal models’ weights. The work presented in this paper is different from previous studies in several aspects. Firstly, this is the first study of using the jDE algorithm to optimize models’ weights within SMT. We believe that the self-adaptive nature of the jDE algorithm in comparison to a DE algorithm improves the efficiency of the optimization algorithm. Secondly, the evaluation is usually based on only one optimizer run (Koehn et al., 2009; Bojar et al., 2015). In this paper, each optimizer (MERT, MIRA, DE, and jDE) was run many times, and the results were compared statistically to meet the conventional significance level.

The remainder of the paper is organized as follows. Section 2 presents some background on the related work. Our experiment is described in Section 3, and the results are presented in Section 4, along with the statistical analysis using a MultEval (Clark et al., 2011) tool. We conclude this paper with Section 5 where we give our opinion about the obtained results and future work.

The availabilities of linear models and discriminative optimization algorithms have been a huge boon to SMT, allowing this field to move beyond the constraints of generative noisy channels (Och and Ney, 2002). The ability to optimize these models according to an error metric has become a standard assumption in SMT, due to the wide-spread adoption of MERT. The problems with MERT can be addressed through the use of surrogate loss functions. The Margin Infused Relaxed Algorithm (MIRA) (Watanabe et al., 2007; Chiang et al., 2008, 2009; Cherry and Foster, 2012; Hasler et al., 2011) employs a structured hinge loss. In order to improve generalization, the average of all weights seen during learning is used on unseen data. Chiang et al. (2008) took advantage of the MIRA to modify each update to suit SMT better. Pairwise Ranking Optimization (PRO) (Hopkins and May, 2011) aims to handle large feature sets inside the traditional MERT architecture. This architecture is desirable, as most groups have infrastructures to n-best decode their tuning sets in parallel. A simple approach of using Evolutionary Algorithms in SMT was shown in our previous work (Dugonik et al., 2014).

SMT deals with mapping sentences in one natural language (source) into another natural language (target). This process can be represented as a stochastic process. There are many SMT variants, depending on how the translation is modelled. Commonly, we are translating the text sentence by sentence. We want to find the best possible translation e∗ out of all possible translations e for a given source sentence f. The system selects the translation with the highest probability P(e|f) . Applying the Bayes rule, the probability P(e|f) is decomposed into probabilities P(f|e) , P(e) and P(f) : (1) e∗=argmaxeP(f|e)·P(e)P(f)=argmaxeP(f|e)·P(e). The denominator P(f) does not influence argmax and can be disregarded.

This approach has three major aspects:

SMT systems usually decompose entire sentences into a sequence of strings called phrases. These phrases are not linguistic phrases but phrases found using statistical methods from a corpus. The SMT system looks for general patterns (n-grams) which appear in everyday language. An n-gram is a contiguous sequence of n items from a given sequence of text or speech. In the phrase-based model, the source sentence f is broken down into I phrases fi¯ , and each source phrase fi¯ is translated into a target phrase ei¯ . From the produced translations it can be seen that the target sentence is not fluent, hence the idea to introduce weights and scale the contribution of each model: (2) e∗=argmaxe ∏i=1IP(fi¯|ei¯)λ1·P(e)λ2.

We can generalize the setup of the SMT system to many different models, and we can scale the contribution of each of them: (3) e∗=argmaxe= ∏i=1rhi(e,f)λi, where h1,…,hr are the models of a search algorithm, e.g. translation model, language model, reordering model, word penalty, etc., r denotes the number of models, and λi,…,λr are models’ weights. The weights are scaling factors, and are optimized with a loss function which evaluates the translation quality, for example, the BLEU evaluation metric. These models are trained separately and then combined, assuming that they are independent of each other. But the contributions of different models influence each other. The problem is to find a set of weights that will provide the best translation quality: (4) e∗(λi,…,λr)=argmaxeexp ∑i=1rλi·loghi(e,f).

The area of possible weight settings is too large for the exploration of all possible values. Usually a tuning set is used to optimize weights. The simplest method is to try out with a large number of possible settings and pick what works best. Assuming we wish to optimize our decoder’s BLEU score, the natural objective of learning would be to find such λ=λi,…,λr that the BLEU score is maximal.

Optimization refers to the process of finding the optimal weights for this linear model where optimal weights are those which maximize the translation quality on the tuning set. During decoding, the decoder scores translations using a linear model. The features of this linear model are the probabilities of multiple models. Each feature contributes information over one aspect of the characteristics of a good translation, e.g. the language model ensures that the translation is more fluent. Each feature can be given a weight that sets its importance. We see the problem as an optimization problem that will be tackled using the jDE algorithm.

In our experiment, we built two SMT systems for translating from Slovenian to English and vice versa. All of the codes and data necessary to begin work on an SMT system are available as a public source, Moses toolkit (Koehn et al., 2007), and the freely available JRC-Acquis parallel corpora (Steinberger et al., 2006) used as a benchmark in the SMT community (Koehn et al., 2009).

Moses toolkit is an open-source toolkit for SMT which contains the SMT decoder and a wide variety of tools for training, tuning and applying the system to many translation tasks. The SMT system is frequency-based, where frequencies are trained on translated texts that are preprocessed and collected into a parallel corpus. Parallel corpora vary in size tremendously. Most of the language pairs, for example, Finnish to Irish, will have a far smaller parallel corpora available. Parallel corpora exist for all European languages and for many other pairs, such as Mandarin to English. However, one of the major challenges faced is the scarce availability of parallel corpora, so we need some methods for creating parallel corpora automatically and efficiently because manual creation of a large parallel corpus can be very costly in terms of effort and time. Currently, parallel corpora are an object of interest.

The used JRC-Acquis corpus must not be seen as a legal reference corpus. Instead, the purpose of the JRC-Acquis is to provide a large parallel corpus of documents for (computational) linguistics research purposes. To align the sentences in a source and language text automatically, we used the HunAlign aligner (Varga et al., 2005). After successful aligning, we selected 700,000 sentences which were used in our experiment. The exact size of the corpora is shown in Table 1. The used corpus was tokenized, lowercased, and sentences longer than 80 words were removed. It is important to obtain a representative sample as much as is possible. The translation quality of neighbouring sentences correlates positively, therefore, we chose sentences from different parts of the corpus to create the training and test sets.

The training set was divided further into training and tuning sets. Sentences shorter than 8 and longer than 60 words were removed from the tuning and test sets. The final sizes of all sets are seen in Table 2. The language model is estimated from a monolingual corpora, typically using relative frequency estimates which are then smoothed. For languages such as English, typically, billions and more words are used. Deploying such large models can pose significant engineering challenges. This is because the language model can easily be so large that it will not fit into the memory of conventional machines. Also, the language model can be queried millions of times when translating sentences, which precludes storing it on disk.

For each language (Slovenian and English) we built language and translation models. The language model was a 5-gram language model with improved Kneser-Ney smoothing using the IRST Language Modeling (IRSTLM) (Federico et al., 2008) toolkit. The translation models were built using grow-diag-final-and alignment from GIZA++ (Och and Ney, 2000). We also extended both SMT systems with four advanced models: The distortion model, the lexicalized reordering model (msd-bidirectional-fe reordering), the word and phrase penalty models. Each SMT system had six models and 14 weights:

The translation quality is considered to be the correspondence between a machine and professional human (reference) translation. There are many metrics, i.e. BLEU, Translation Error Rate (TER) (Snover et al., 2006), Word Error Rate (WER) (Saon et al., 2006), etc. The more popular metric in SMT is the BLEU metric, because it is quick, inexpensive, language-independent, and one of the first metrics to achieve a high correlation with human judgments. The central idea behind BLEU is that the closer a machine translation is to a reference translation, the better it is. The primary task in the BLEU metric is to compare the n-grams of the machine translation with the n-grams of the reference translation and count the number of matches which are position-independent. The foundation of the BLEU metric is the modified n-gram precision measure. This captures two aspects of the translation: Adequacy and fluency. The unigram scores are found to account for how much the information is retained (adequacy), and the longer n-gram scores account for the fluency of the translation, i.e. if the target language is English, to what extent it reads like “good” English. The BLEU metric’s output is always a real-valued number between 0 and 1. This value indicates how similar the machine and reference translations are. Values closer to 1 represent the more similar texts, however, few machine translations will attain a score of 1 because, in that case, the machine translation must be identical to the reference translations.

Weights in SMT systems can affect the quality of the translations significantly. In this paper, the two phrase-based SMT systems were built successfully, and their weights were optimized using the jDE algorithm. They were tested on the unseen set, and the results were comparable. However, based on extensive experiments, the jDE algorithm obtained better BLEU scores compared to the state-of-the-art algorithms MERT, MIRA and the DE algorithm. The jDE algorithm achieved the best BLEU scores of 60.57 for the Slovenian to English SMT system and 51.95 for the English to Slovenian SMT system, followed by MERT and the DE algorithm, and MIRA. We are confident of the promising characteristics of algorithms based on Differential Evolution and good attributes of the jDE self-adaptive mechanism.

Recently, the neural machine translation has emerged as a new paradigm in MT. It also has many parameters that could be optimized to yield better performance.