The superiority of artificial neural networks (ANNs) in various tasks (classification, pattern identification, prediction, etc.) has led researchers to focus much of their efforts on the study of the functioning of their components from a theoretical perspective, see Higham and Higham (2019), Smale et al. (2010). It is well known that ANNs have a high capacity for learning, the effectiveness of which depends on many factors. Amongst them, the problem complexity influences to a high degree the ANN performance, which depends not only on the ANN architecture, but also on the accurate and sufficient training data and the efficiency that datasets show throughout the process. Training in recurrent neural networks (RNNs) – ANNs that stand out for their high capacity for learning and recognition of temporal patterns – depends to a large extent on size, type and structure of the selected training sets (Chen, 2006; Zhang and Suganthan, 2016; Zapf and Wallek, 2021): this is such a central point that decisively influences both the speed and the ability to learn. During the training phase, where the unknown parameters are to be determined, the quality and learning capacity of the selected training datasets are of key importance (Mirjalili et al., 2012).

The main objective of this paper is to provide a robust methodology to select optimal training datasets (those with the highest learning capacity) that can be used in any context to maximise the performance of the trained models. This methodology has been designed to run in parallel with RNN learning so that, while the RNN learning evolves progressively only with the optimal training inputs, the data noise gradually disappears. This has a positive impact on the quality of the RNN results while minimising the likelihood of occurrence of overfitting.1 The key idea in the design of the mathematical structure that supports this selection is to define a binary relation that gives preference to those datasets with higher estimator abilities by using Utility theory. A second contribution of our work is to have designed our methodology based on tools that have not been used previously for this. This novelty lies in showing Markov Networks (MNs), widely known as generative models (Gordon and Hernandez-Lobato, 2020), as models with a real discriminative capacity when used in conjunction with the Von Newman Morgenstern theorem (VNM theorem), a cornerstone of Game Theory with an extensive background also in Decision Theory (Machina, 1982; Delbaen et al., 2011). In order to faithfully model the RNN reality, we have used the dynamic version of the MNs (TD-MRFs). It is worth noting the versatility of our proposal, which can be also applied to other data-driven methodologies provided that they are regulated by dynamical systems.

Markov Random Fields (MRFs) are also known as Markov Networks (MNs) in those contexts that require highlighting the undirected graph condition (Dynkin, 1984). MRF-type graphical models have experienced a resurgence in recent years. In its origins, they exclusively performed functions related to image processing such as restoration or reconstructing. Later works such as García Cabello (2021) or Wang et al. (2022) have acknowledged their high predictive capability due to the equivalence between MRFs and Gibbs distributions, which provides an explicit expression of the prior likelihood after appropriate choice of the energy functions. MRF solutions are widely regarded as generative models as opposed to discriminative approaches, more related to tasks which involve classification.

Regarding the literature review, the selection of optimal training sets has not been studied in a general framework so far. To this author’s knowledge, this is the first analysis that aims to provide guidance for a general context. Published papers have studied this issue either only in contexts of ANN classification tasks or in very precise scenarios (electrical, financial or chemical engineering) taking advantage of their specific techniques. Within the first category, the genetic algorithm (GA) is widely used as a tool to create high-quality training sets as a the first step in designing robust ANN classifiers, see Reeves and Taylor (1998), Reeves and Bush (2001) or more recently, the paper (Nalepa et al., 2018). Disadvantages of using GA, apart from slowness, include that it is computationally expensive and too sensitive to the initial conditions. In our proposal, however, the calculation of the probabilities associated with the utility (i.e. efficiency as estimators) of the inputs is very simple and therefore does not add computational cost.

Within the second category, in the paper (Zapf and Wallek, 2021), the authors made a comparison between existing methods in the area of chemical process modelling in order to split a training set from a given data set. In Wong et al. (2016), the authors proposed a data selection for statistical machine translations, based on recursive neural networks which can learn representations of bilingual sentences. The paper (Fernandez Anitzine et al., 2012) analyses through a very context-specific instrument (ray-tracing) the ANN optimal selection of training set in the context of predicting the received power/path loss in both outdoor and indoor links. In Kim (2006), authors propose a GA approach for ANN instance selection for financial data mining.

As for the use of MNs/MRFs (prior probability) for problems which involve probability a posteriori, in the literature the terms “MNs/MRFs” and “discriminative” appear together only and exclusively to refer to discriminative random fields (DRFs) or equivalently conditional random fields (CRFs), both type of random fields which provide by definition a posterior probability.

The rest of the paper is structured as follows: preliminaries of Section 2 include basic knowledge on preference relations and VNM theorem, MNs and RNN functioning. Section 3 structures the steps to be followed to reach a solution to the proposed problem. The design of an abstract TD-MRF-based framework is performed in Section 4 which will subsequently allow the computation of prior probabilities associated with the VNM theorem. A TD-MRF structure for the input sets is also provided here. In Section 5, the expected utility theorem is applied after proving that the conditions for doing so are met. Section 6 highlights (and proves) the main results of our work. In Section 7, an example of the method application is developed. Section 8 finally concludes the paper.

In this section we will develop the theoretical framework that will capacitate Markov Network-methodology to perform discriminative tasks based on prior probability and the VNM Theorem 2.1. Specifically, our aim is to design a mathematical model which enables MNs to identify the optimal RNN training sets, i.e. those that produce better estimates in forecasting taks. For a better understanding, we will make reference to a real example of a RNN forecasting process:

Recall that, in the networks as a whole, Recurrent Neural Networks, RNNs, stand out for their predictive abilities based on their potential in processing temporal data. Thus, we are facing some time-dependent learning process by using RNNs where the superscript t denotes time in all cases (no distinction will be made between matrices and their transpose in order to avoid confusion between t transpose and t time).

As is well known, in RNN prediction tasks, training sets are fed with temporal sequences composed with pairs of previous data of the form (input, labels) with the objective of predicting the future, referred to as future target value, y. Thus, ${X^{t}}$ is the time-varying variable which needs to be estimated, ${Z^{t}}$ is the set of RNN outputs and ${Y^{t}}$ the set of labels (i.e. past target values for training). In terms of the illustrative example: to predict the electrity price one month ahead (target value y, $t=+1$), the training sets are made up of temporal sequences $({x_{i}^{t}},{y_{i}^{t}})$ whose first component contains different factors that influence electricity prices (fuel prices, transmission and distribution costs, weather conditions, etc.) and whose second component is the electrity price, both ${x_{i}^{t}}$, ${y_{i}^{t}}$ in site i and prior to present time ($t=-9,\dots ,-2,-1,0$).

Let $\textit{In}$ be the (time-dependent) input RNN dataset formed by n vectors: each of them contains d features: ${x_{i}^{j,t}}\in \mathbb{R}$, $i=1,\dots $, $n;j=1,\dots ,d$: ${Z^{t}}[\textit{In}]$ will stand for the set of outputs which is generated after the completed RNN process corresponding the the set of inputs $\textit{In}$.

The objective is to select from the n inputs in $\textit{In}$, $D\lt n$ vectors that will make up the training set: the selection criterion is to choose those which produce the best estimates in the RNN learning process. The choice will be conducted by only using prior probability –instead of posterior– and the VNM Theorem 2.1.

We will list the steps to be taken:

Here, we first design an abstract graph-based framework that shall provide a model for a dynamic context of k sites and r filters for data discrimination (corresponding to r cliques) for which it will be shown that it is an TD-MRF under certain mild conditions. Such TD-MRF will be the core in the computation of prior probabilities $po[{X^{t}}={x_{i}^{t}}]$ of the VMN Theorem 2.1.

Let $\mathcal{S}$ be a set of k sites ${s_{i}}$, $\mathcal{S}=\{{s_{i}}|i=1,2,\dots ,k\}$ such that the variable may take different values depending on the site, ${X_{{s_{i}}}^{t}}$. For each site ${s_{i}}$, the variable ${X_{{s_{i}}}^{t}}$ can be disaggregated into a set of d characteristics which fully describes ${X_{{s_{i}}}^{t}}$ in the instant of time t: ${X_{{s_{i}}}^{t}}=({X_{{s_{i}}}^{1,t}},{X_{{s_{i}}}^{2,t}},\dots ,{X_{{s_{i}}}^{d,t}})$, with lower case ${x_{{s_{i}}}^{t}}=({x_{{s_{i}}}^{1,t}},{x_{{s_{i}}}^{2,t}},\dots ,{x_{{s_{i}}}^{d,t}})$ for the feature vector (i.e. a numerical vector corresponding to a realisation of the variable). We shall use indistinctly ${x_{{s_{i}}}^{t}}=({x_{{s_{i}}}^{1,t}},{x_{{s_{i}}}^{2,t}},\dots ,{x_{{s_{i}}}^{d,t}})$ or ${x_{i}^{t}}=({x_{i}^{1,t}},{x_{i}^{2,t}},\dots ,{x_{i}^{d,t}})$ either for upper and lower case. In a compact form, ${x_{i}^{j,t}}\in \mathbb{R},i=1,\dots ,k;j=1,\dots ,d$.

Remember that two random variables are equivalent if they have identical distribution. Then, we will define the edges by equivalently defining the neighbourhood $\mathcal{N}$ of a site: $\mathcal{N}({X_{{s_{i}}}^{t}})=\{{X_{{s_{j}}}^{t}}|\hspace{0.2778em}{X_{{s_{j}}}^{t}},{X_{{s_{i}}}^{t}}\hspace{0.2778em}\text{are equivalent}\}=\{{X_{{s_{j}}}^{t}}|P[{X_{{s_{j}}}^{t}}\leqslant x]=P[{X_{{s_{i}}}^{t}}\leqslant x]\hspace{0.1667em}\forall x\}$.

Recall that marginal distribution is also known as prior probability in contrast with the posterior distribution (the conditional one). From the former definition, sites in the same neighbourhood have the same prior probability. Moreover,

The following theorem proves then that the DGM defined in Definition 4.1 is a TD-MRF by equivalently showing that the Markov condition:

Insofar as the distribution function is the tool used to make estimates, previous Theorem 4.4 provides a joint measure of how close the variable is to taking a particular value.

Each clique has its own common estimation function:

In discrimination/classification works, the commonly used probability is the conditional or posterior probability $P[{X^{t}}=x|y]=\frac{1}{Z}{\textstyle\prod _{\begin{array}{l}\text{cliques}\hspace{2.5pt}{C_{j}}\\ {} \hspace{1em}j=1\end{array}}^{r}}{e^{-{\psi _{C}}({X^{t}}=x|y)}}$ where x stands for RNN input, y represents the corresponding target value and ${\psi _{C}}({X^{t}}=x|y)$ means that the energy function ${\psi _{C}}$ only assigns to those inputs x that correspond to y.

In this section, we will further develop equation (3), by giving the explicit calculation of $u({x_{i}^{t}})$, $\forall i$, t. To do so, we will again apply the VMN theorem to the secondary lottery.

Recall that in the preceding sections we have considered a second lottery which arises naturally when we test how good the ouput ${z_{i}^{t}}\in {Z^{t}}[\textit{In}]$ is as estimate. If M is the threshold below which the loss function is considered acceptable, the secondary lottery is whether the loss function $\text{loss}({z_{i}^{t}})\lt M$ or not. With the intuitive notation of the paper (Machina, 1982), our secondary lottery is a probability distribution function over the interval $[0,M]$ (the set of all lotteries over $[0,M]$ is denoted in Machina (1982) by $D[0,M]$) and the preference functional $V(\cdot )$ on $D[0,M]$ in our case is $\text{loss}(\cdot )$. Note that Definition 3.1 of preference over the set ${Z^{t}}[\textit{In}]$ can be additionally considered over $\textit{In}$ for the deterministic assignment ${x_{i}^{t}}\to {z_{i}^{t}}$: for ${x_{i}^{t}},{x_{j}^{t}}\in \textit{In}$ where the loss function is considered in a squared-form, $\text{loss}({z_{i}^{t}})={({z_{i}^{t}}-{y_{i}})^{2}}$ which in the context or real positive numbers is equivalent to $\text{loss}({z_{i}^{t}})=|{z_{i}^{t}}-{y_{i}}|$. The objective here is to prove that this binary relation satisfies the necessary conditions that enable the application of the Von Neumann-Morgenstern (VNM) theorem.

First of all, it has to be shown that it is a preference relation:

Moreover, former preference relation satisfies the following conditions:

Therefore, the preference relation stated in Definition 3.1 verifies the conditions required for the application of the VNM Theorem 2.1. Thus, there exists an utility function $u:\textit{In}\to [0,1]$ which quantifies the preferences: ${x_{1}^{t}}\succeq {x_{2}^{t}}$ iff $u({x_{1}^{t}})\geqslant u({x_{2}^{t}})$. Moreover, the Von Neumann-Morgenstern Expected Utility theorem also provides guidance for computing the utility of the set of inputs through the explicit formula given in equation (3):

The objective of this section is to highlight (after proving) the main results of our proposal. First of all, we define the level of efficiency of an input set $\textit{In}$, $\text{Eff}[\textit{In}]$. Thus, next Theorem 6.2 proves the existence of a formula for determining the efficiency of a training set while Theorem 6.3 gives an explicit expression for computing $\text{Eff}[\textit{In}]$.

This section is aimed at developing an example of the method application. In order to make a choice between two real data sets, their level of efficiency will be computed. As stated in Definition 6.1, the level of efficiency of an input set $\text{Eff}[\textit{In}]$ measures the learning capacity of its inputs on the basis of the preference relation of Definition 3.1 that prioritises those inputs whose outputs have a lower associated loss (better estimates). To compute $\text{Eff}[{I_{n}}]$ we shall apply the formula proved in Theorem 6.3 which follows from Theorem 6.2, in which the existence of an utility function u which quantifies the preference relation was proved. This is where p is the probability that $|{z_{i}^{t}}-y|\lt M,\forall {z_{i}^{t}}$ for a given threshold M below which the loss function is considered acceptable.

The data sets we shall use here contain prices (€/1 kg) over time for the most common olive oil varieties. These data are available on the websites of the Government of Spain:

https://www.mapa.gob.es/es/agricultura/temas/producciones-agricolas/aceite-oliva-y-aceituna-mesa/Evolucion_precios_AO_vegetales.aspx,

where prices are published on a weekly basis. Specifically, we will consider data sets corresponding to weeks 28/2023 and 39/2022 (the reasons for this choice will be explained later). We shall thus apply the above formula taking into account the following considerations. On one hand, note that the choice of the threshold M will depend on the context. In the olive oil market scenario, assuming a deviation from the olive oil price of 10% is acceptable. Hence, $M=0.5$. On the other hand, the value of the parameter p will depend on several market features which vary over time depending on physical (rainfall, pollen levels$\dots \hspace{0.1667em}$) and socio-economic (Government regulations) circumstances. When real prices are subject to unusual fluctuations (due to changes in the aforementioned circumstances), such prices used in training tasks will deviate more from the real ones. In consequence, the probability p, such that $|{z_{i}^{t}}-y|\lt M,\forall {z_{i}^{t}}$, will decrease. Either way, it should be noticed that p is known as soon as the value of M is fixed, but it is not the same for all input sets.

From the above formula, since M is known (and therefore so is p), we must focus on computing $po[{X^{t}}={x_{i}^{t}}]$ for each input ${x_{i}^{t}}$. To that end, we will follow Fig. 2 of the TD-MRF operational process given in Section 4.1. Since each input ${x_{i}^{t}}$ may be disaggregated into d features ${x_{i}^{k,t}}$ ($k=1,\dots ,d$), ${x_{i}^{t}}=({x_{i}^{1,t}},{x_{i}^{2,t}},\dots ,{x_{i}^{d,t}})$, the same is true for the corresponding probability $po[{X^{t}}={x_{i}^{t}}]=po[{x_{i}^{t}}]$, which is $po[{x_{i}^{t}}]=(po[{x_{i}^{1,t}}],po[{x_{i}^{2,t}}],\dots ,po[{x_{i}^{d,t}}])$.

We assume that the energy functions ψ corresponding to the r cliques are of Gaussian type, i.e. ${\psi _{{C_{j}}}}=\frac{{([.]-{\mu _{j}})^{2}}}{{\sigma _{j}^{2}}}$, $j=1,\dots ,r$, since Gaussian distributions are suitable in the olive oil scenario. Actually, given that Gaussian distributions portray those data sets whose majority of elements revolves around the centre, energy functions of Gaussian type are particularly suited for goods whose price takes values in an interval of small length and do not suffer very sharp price variations.

In order to achieve our goal, each input ${x_{i}^{t}}$ must be first processed in each clique ${\mathcal{C}_{j}},j=1,\dots ,r$, through the corresponding clique potential, whose result is Once the processing in the cliques has been completed, the required probability is obtained as The computation of “the partition function” Z entails certain difficulties in practice. For this reason, we shall adopt the view commonly taken in the literature that $Z=1$.

From the TD-MRF structure proved in Theorem 4.4, cliques gather those random variables with identical prior and posterior distribution (see Remark 4.6). This theoretical description fits with the specialist major retailers in the olive oil context. In this practical case, the level of efficiency shall be computed through ${P_{{C_{j}}}}[{X^{t}}={x_{i}^{t}}]$ of cliques ${C_{j}},\hspace{0.1667em}j=1,\dots ,7$ supported by the information given in Table 8 below (source: https://www.olimerca.com/precios/tipoInforme/3).

As discussed before, there are multiple factors (physical and socio-economic) that influence the price. Such factors are the features ${x_{i}^{k,t}}$ ($k=1,\dots ,d$) of each input ${x_{i}^{t}}=({x_{i}^{1,t}},{x_{i}^{2,t}},\dots ,{x_{i}^{d,t}})$. For simplicity, we focus on just one of them in order to choose the two input sets ${\textit{In}_{1}},{\textit{In}_{2}}$: the hydrographic index, which shows the average rainfall over a certain period of time. In this line, ${\textit{In}_{1}}$ is an input set of prices (€/1 kg) under severe drought conditions (and therefore, with unusual fluctuations in price as product shortages raise the prices3) while ${\textit{In}_{2}}$ is an input set of prices which correspond to a usual period of rainfall: With this choice of ${\textit{In}_{1}}$ and ${\textit{In}_{2}}$, it is to be expected that the inputs in ${\textit{In}_{1}}$ will produce worse estimators (higher associated loss) since such inputs reflect prices with unusual fluctuations (therefore with higher deviation from the mean). Hence, it is is to be expected that $\text{Eff}[{\textit{In}_{2}}]\gt \text{Eff}[{\textit{In}_{1}}]$.

The computation of level of efficiency ${\textit{In}_{1}}$ is supported by the information provided in Tables 1–7.

From the information provided by the above tables, finally the required probability is computed (see Table 9).

$\text{Eff}[{\textit{In}_{1}}]=p\cdot (po[{X^{t}}=3.16]+po[{X^{t}}=5.92]+po[{X^{t}}=6.26]+po[{X^{t}}=6.52]+po[{X^{t}}=7.10])=0.23\cdot (2.09102\text{E}-12+2.55039\text{E}-74+3.7242\text{E}-101+1.7143\text{E}-124+2.4066\text{E}-185)=0.23\cdot 2.09102\text{E}-12=$

Similarly, the computation of ${\textit{In}_{2}}$ is supported by the information provided in Tables 10–16.

Finally, the required probability is computed (see Table 17).

$\text{Eff}[{\textit{In}_{2}}]=p\cdot (po[{X^{t}}=2.71]+po[{X^{t}}=3.78]+po[{X^{t}}=3.80]+po[{X^{t}}=3.86]+po[{X^{t}}=3.96])=0.57(3.24825\text{E}-30+0.268808808+0.337851876+0.53631732+0.549630567+1.692608571)=$.

Hence, as expected (since the inputs of ${\textit{In}_{1}}$ reflect prices with unusual fluctuations and therefore with higher deviation from the mean) $\text{Eff}[{\textit{In}_{2}}]\gt \text{Eff}[{\textit{In}_{1}}]$.

This paper deals with the selection of optimal training sets (those that have a higher capacity as estimators) in Recurrent Neural Networks under prediction tasks (or pattern recognition with time series as inputs), although this may also apply to other data-driven models regulated by dynamic systems. Our objective is to fill the existing gap of clear guidelines to follow for selecting optimal training sets in a general context.

We design here a novel methodology to select optimal training data sets that can be used in any context. The key idea, which underpins the design of the mathematical structure that supports the selection, is a binary relation that gives preference to inputs with higher estimator abilities. A second novelty of our approach is to use dynamic tools that have not been used previously for this purposes: dynamic Markov Networks, which are widely regarded as generative models, successfully compute the prior probabilities involved in the formula for calculating the degree of efficiency of the training set (Theorem 6.3), derived from application of the Von Neumann-Morgenstern Theorem 2.1. It is precisely the VMN theorem the instrument that confers discriminative capacity to the MNs: in this work we show that the preference relation that we define between inputs of a training set (inputs with higher learning capacities are preferred in the sense that the error function takes lower values) fulfils the necessary hypotheses to derive the existence of a simple formula for the calculation of the utility (efficiency) of a training set.

The simplicity of this calculation allows it to be carried out in parallel with the learning process without adding computational cost. Thus the optimal sets are selected as the learning process evolves, therefore the data noise gradually disappears which decreases the likelihood of overfitting occurring.

Declarations of interest: none.